i 
Zambia Management
and Leadership Academy
Final Evaluation Report 
October 2014

Acknowledgments 
The BroadReach Institute for Training and Education (BRITE) would like to acknowledge the Ministry of 
Health (MoH) and the Ministry of Community Development, Mother and Child Health (MCDMCH) for making 
this evaluation possible. We sincerely thank the provincial and district health teams for their cooperation 
and feedback during the data collection process. We would also like to thank all health professionals who 
participated in the study and all Management Specialists under the Zambia Integrated Systems 
Strengthening Program (ZISSP) for the logistical support rendered to the evaluation team. Much appreciation 
also goes to the United States Agency for International Development (USAID), the Merck Foundation, and 
Abt Associates through the Zambian Integrated Services Strengthening Program (ZISSP) for their continued 
support. 
Finally, we would like to acknowledge the research team led by Dr. Wilbroad Mutale for the coordination of 
data collection, analysis of data and the development of this evaluation report. 
This document reports the research team’s findings and does not necessarily represent the views of BRITE, Abt 
Associates, the ZISSP project, the Merck Foundation, or the US Agency for International Development
.
Contents 
Acronyms .......................................................................................................................................... i
Executive Summary ............................................................................................................................ii
1 Background and Rationale ..........................................................................................................1
1.1 Program Logic Model and Theory of Change ............................................................................... 2
1.2 Overview of the ZMLA Program ................................................................................................... 3
1.3 Evaluation Objectives and Questions ........................................................................................... 8
2 Methodology ..............................................................................................................................9
2.1 Study Design ................................................................................................................................. 9
2.2 Target Population ......................................................................................................................... 9
2.3 Data Sources ................................................................................................................................. 9
2.3.1 Routine Data Collection Tools .............................................................................................. 9
2.3.2 Management and Leadership Survey ................................................................................... 9
2.3.3 Workplace Climate Survey.................................................................................................. 10
2.3.4 Qualitative Data Collection Tools ....................................................................................... 10
2.4 Data Management ...................................................................................................................... 11
2.4.1 Quantitative Data Analysis ................................................................................................. 11
2.4.2 Qualitative Data Analysis .................................................................................................... 11
2.5 Study Limitations ........................................................................................................................ 12
2.6 Ethical Consideration .................................................................................................................. 12
3 Results .....................................................................................................................................13
3.1 Contribution to Improved Management and Leadership Skills ................................................. 13
3.1.1 Knowledge Gaps and Knowledge Gained ........................................................................... 13
3.1.2 Leadership and Management Skills .................................................................................... 14
3.1.3 Job Motivation .................................................................................................................... 16
3.2 Organisational Benefits Resulting from ZMLA Trainings ............................................................ 17
3.2.1 Changes in the Workplace Climate ..................................................................................... 17
3.3 Overall Conduct of ZMLA ............................................................................................................ 20
3.3.1 Participant Commitment .................................................................................................... 20
3.3.2 Content and Delivery of the Course ................................................................................... 21
Case Studies ........................................................................................................................................ 22
3.3.3 Mentorship ......................................................................................................................... 22
3.3.4 Engagement of Stakeholders .............................................................................................. 23
3.3.5 Other Management and Leadership Courses ..................................................................... 24
3.3.6 Sustainability ...................................................................................................................... 24
4 Conclusion ................................................................................................................................26
5 Recommendations .................................................................................................................... 27
Annex 1: Case Study Score Card ........................................................................................................28
Annex 2: Participant Feedback Form ................................................................................................. 30
Annex 3: Leadership and Management Tool .....................................................................................32
Annex 4: Workplace Climate Tool .....................................................................................................36
Annex 5: Paired Samples Test (Mean Differences for Leadership and Management Tool) ...................41
Annex 6: Percentage of Trainees Feeling Adequately Trained in Management & Leadership (Crosstab and 
Chi-square Test) ...............................................................................................................................43
Annex 7: MLA Survey Themes ..........................................................................................................44
Annex 8: Phase II Work Climate Survey Mean Scores Before and After ZMLA Training .......................45
Annex 9 : Phase II Work Climate Survey Independent Samples Test ...................................................46
Annex 10: Work Climate Survey Themes ...........................................................................................48
Annex 11: Study Information Sheet ..................................................................................................49
Annex 12: Informed Consent Form ...................................................................................................50
Annex 13: ZMLA Participant IDI Guide ..............................................................................................51
Annex 14: Provincial Medical Officer and District Medical officer IDI .................................................53
Annex 15: Key stakeholder’s IDI .......................................................................................................55
Annex 16: IDI Guide for Health facility managers hosting Case ..........................................................56
Annex 17: IDI Guide for Case study participant .................................................................................57
Annex 18: IDI Guide for Program implementers/MOH/MCDMCH ......................................................58
Annex 19: IDI guide for NIPA representative .....................................................................................59
Annex 20: List of Anonymized Key Informants ..................................................................................60
Annex 21: Nvivo Coding Framework for Qualitative Data ..................................................................62
Annex 22: Statement of Work ..........................................................................................................63
List of Tables 
Table 1: Recruitment and Completion Dates for ZMLA Implementation Phases ........................................................... 4
Table 2: Course Components and Structure .................................................................................................................. 5
Table 3: Summary of Curriculum Content ..................................................................................................................... 5
Table 4: Trainee Demographic Characteristics and Attendance .................................................................................... 7
Table 5: Evaluation Objectives and Study Questions .................................................................................................... 8
Table 6: Summary of Routine Data Collection Tools ..................................................................................................... 9
Table 7: Public Health System Level and Job Categories of Key Informants Interviewed. ............................................10
Table 8: Phase I Case Study Scores...............................................................................................................................14
Table 9: Demographic Characteristics of Leadership and Management Survey Respondents ......................................14
Table 10: Work Climate Survey Respondent Characteristics ........................................................................................18
Table 11: Participant Perceptions of the Quality and Relevance of Workshops ...........................................................21
Table 12: Participants Perceptions of the Quality and Relevance of Training Modules ................................................21
List of Figures 
Figure 1: ZMLA Logic Model .......................................................................................................................................... 3
Figure 2: ZMLA Implementation Locations ................................................................................................................... 4
Figure 3: Knowledge Scores Pre & Post Workshops .....................................................................................................13
Figure 4: Self-Assessment of ZMLA Trainee Management and Leadership Readiness ..................................................15
Figure 5: Percentage Change in Management and Leadership Self Confidence ............................................................15
Figure 6: Phase I Job Motivation following ZMLA Training ...........................................................................................17
Figure 7: Phase II Job Motivation following ZMLA Training ..........................................................................................17
Figure 8: Evaluation of Work Climate Survey ...............................................................................................................18
i 
Acronyms 
APAS Annual Performance Appraisal System
ART Anti-Retroviral Therapy
BRITE BroadReach Institute for Training and Education
CDF Constituency Development Fund
DHIS District Health Information System
DHMT District Health Management Team
DSA Daily Subsistence Allowance
EMO(N)C Emergency Obstetric (and Neonatal) Care
GRZ Government of the Republic of Zambia
HMIS Health Management Information System
HR Human Resources
ICT Information and Communication Technology
IDI In-depth Interview
M&E Monitoring and Evaluation
M&L Management and Leadership
MCDMCH Ministry of Community Development, Maternal, and Child Health
MNCH Maternal, Neonatal, and Child Health
MOH Ministry of Health
MP Member of Parliament
MS Management Specialist
MSH Management Science for Health
NGOs Non-Governmental Organisations
NIPA National Institute for Public Administration
PA Performance Assessment
PADESA Pan African Development Education for Southern Africa
REACT The Response to Accountable Priority Setting for Trust in Health Systems Project
RBF Result Based Financing
RHC Rural Health Center
QI/QA Quality Improvement or Assurance
SMART Specific, Measurable, Attainable, Relevant, Time bound 
SPSS Statistical Package for Social Sciences
USAID United States Agency for International Development
WHO World Health Organisation
ZISSP Zambia Integrated Systems Strengthening Program
ZMLA Zambia Management and Leadership Academy
ZPCT Zambia Prevention Care and Treatment program
ii
Executive Summary 
The Zambia Management and Leadership Academy (ZMLA) training program was designed to support 
government efforts to equip healthcare managers with the knowledge and skills to lead, own, and transform 
the health sector. With funding from USAID and the Merck Foundation, ZMLA is led by the Ministry of Health 
(MOH) and the Ministry of Community Development, Mother and Child Health (MCDMCH). The program is 
also technically supported by the Zambia Integrated-Health Systems Strengthening Project (ZISSP)1
 and its 
subcontractor the BroadReach Institute for Training and Education (BRITE). ZMLA is recognized and 
accredited by the National Institute for Public Administration (NIPA). Presented below are the ZMLA project 
evaluation objectives and findings: 
Evaluation Objective 1: To measure the extent to which the ZMLA program has contributed to improved 
management and leadership skills among trainees. 
Key findings: Trainees reported being adequately trained with the proportion of trainees feeling ready to 
manage and lead being 43% before the training compared to 98% after being trained (p<0.000 at the 5% 
significance level). Average case study scores showed that all case study implementation teams could apply 
ZMLA tools to public health projects post ZMLA training. Evaluation results also showed that ZMLA increased 
trainee confidence to perform management tasks, and trainees reported increased job motivation. 
Evaluation Objective 2: To evaluate organisational benefits resulting from the ZMLA training. 
Key findings: Qualitative analysis from interviewees showed that their work environments improved in the 
following aspects: more organized process for the annual appraisal system, better shared vision and 
coordination in the workplace; shorter and more effective meetings; better understanding and appreciation 
of roles in teams; improved delegation and handover practices; more appreciation of finance processes; and 
improved reporting practices. Quantitative findings from the work climate survey showed that the most 
improved aspect of the work environment was the resolution of human resource issues. 
Evaluation Objective 3: To describe the overall conduct of the ZMLA program and useful lessons for future 
program scale-up activities and implementation. 
Key findings: Trainees reported that the short, intensive trainings held near their work sites were minimally 
disruptive to their work. ZMLA participants reported that they found the ZMLA syllabus practical and tailored 
to the local health system and environment. Overall, the quantitative and qualitative findings demonstrated 
that participants valued the training with 98% of about 500 participants reporting that they would 
recommend ZMLA to their peers. Stakeholders from NIPA, the community and policymakers at both 
ministries responsible for health reported a desire for the program to continue as an important component 
of health systems strengthening. 
Evaluation Objective 4: To evaluate the extent to which the ZMLA training contributed to improved service 
delivery. 
Key findings: The study design which is highly qualitative did not allow for objective evaluation of the impact 
of ZMLA training on service delivery as initially envisioned. This objective was therefore not achieved. 
 
1
 ZISSP is led and managed by Abt Associates Incorporated. 
iii
Conclusion: The management and leadership self-confidence survey and case study scores showed that 
ZMLA contributed to improved leadership and management confidence and skills. A significant proportion of 
ZMLA trainees reported that the program had improved their job motivation and they had observed changes 
in the resolution of human resource issues in their workplace. The greatest perceived improvements in the 
workplace were reported by participants whose senior managers were also trained in ZMLA. However, 
about half of the teams trained had difficulties in the following areas: developing model of care frameworks 
in supply chain management, planning for delegation and developing job descriptions. Despite participants 
appreciating and applying monitoring and evaluation concepts successfully, only half of all district level case 
study projects were implemented or monitored. Mentorship and follow up of case study projects after 
group mentorship sessions was reported as weak. Despite these challenges, the current ZMLA model of 
training was reported as being practical and minimally disruptive for trainees and their employers. Most 
participants reported that they would recommend the training to colleagues. In sum, top leadership in the 
ministries responsible for health, participants, mentors, and trainers openly appreciated the program and 
reported that ZMLA should continue. 
Recommendations: 
 It is recommended that the ministries responsible for health (MoH and MCDMCH) should compel 
health program managers and supervisors at all levels of the organisational structure to undergo the 
ZMLA training. 
 The ministries responsible for health (MoH and MCDMCH) with support from partners should seek 
to support the institutionalization of ZMLA with local academic institutions such as the National 
Institute for Public Administration (NIPA) and University of Zambia. District institutions should 
budget for ZMLA tuition fees for managers in need of management and leadership skills through 
their annual action plans. In the long term, ZMLA should be included in clinical training courses. For 
clinicians with known ambitions of managing health institutions, ZMLA should be offered as an 
optional in-service training course at eligible training institutions across the country. 
 Going forward, BRITE should design a more robust evaluation approach to effectively track the 
impact of ZMLA trainings. Experimental or quasi-experimental approaches such as clustering and 
randomization to intervention and control groups for instance could demonstrate attribution of 
ZMLA activities to changes in service delivery. Such knowledge would be needed to inform scale up 
of such a program across the country. 
 As BRITE is the originator of the curriculum, BRITE should revisit ZMLA content relating to the 
development of models of care in supply chain management, delegation and the development of 
health worker job descriptions. More time should be spent during training sessions to explain these 
concepts. Additionally, mentorship should emphasize more on the concepts that case study teams 
struggle to grasp and apply. Going forward, the program should improve the follow-up and 
mentorship of participants after group mentorship sessions to support case study development. 
 Although a key result area for ZMLA was to develop mentorship and coaching capacity in the 
ministries responsible for health, mentors should only be allowed to mentor colleagues after they 
themselves have completed the program. 
 In order to translate theory into practice, NIPA should request that participants implement case 
study projects in order to graduate. Trainees should be encouraged to leverage off other health 
initiatives and programs in their districts to support their case studies. 
1 
1 Background and Rationale 
Effective leadership and management can improve health service delivery.2
 Research in both the private and 
public sector demonstrates that leadership and management skills are essential ingredients of high 
performance. 3
 According to the World Health Organisation, leadership and governance are one of the six 
key building blocks for an effective health system.4
A critical challenge that Zambia currently faces is that district health teams are often led by newly qualified 
medical doctors whose pre-service training emphasizes clinical skills over management and leadership skills. 
The dissolution of civil service induction programs and the restructuring of the Central Board of Health in the 
recent history of Zambia has also led to many health managers across all professions being unprepared for 
their roles in management. 
Although the National Institute of Public Administration (NIPA) once offered a higher diploma in Health 
Management to the public service, few institutions or managers have ever enrolled for the course. The NIPA 
health diploma succeeded the European funded Pan African Development Education program for Southern 
Africa (PADESA). Once the donor funding through PADESA concluded, many new managers in the health 
system could not attend the course. This was attributed to the relative costs or expense of attending the 
diploma program, long distances to the training college and lengthy duration of the course. Due to a critical 
shortage of health workers in the health system, many health mangers could not afford to leave their work 
stations for extended periods of time. 
Recognizing these constraints, in 2011 the Zambian Ministry of Health (MOH) developed the Governance 
and Management Capacity Building (GMCB) Strategic Plan (2012-2016) whose overarching goal was to 
improve health sector governance environment that is Result oriented, Accountable and Transparent. 
Therefore, the Zambia Management and Leadership Academy (ZMLA) created to support the realization of 
the government’s capacity building plan to equip healthcare managers with the knowledge and skills to lead, 
own, and transform the health sector. 
ZMLA is led by MOH and the Ministry of Community Development, Mother and Child Health (MCDMCH). The 
program receives technical support from the Zambia Integrated Health Systems Strengthening Project 
(ZISSP) through its subcontractor BroadReach Institute for Training and Education (BRITE). The ZMLA training 
program is recognized and accredited by NIPA. 
 
2
 McKinsey & Company. Develop Leaders, Deliver Change. 2011 
3
 McKinsey & Company. Based on analysis of Organisational Health Survey database (N = 60,000). 2009.
4
The other five building blocks: a well-performing workforce; a well-functioning health-information system; equitable 
access to essential medical products, vaccines, and technologies; and adequate financing. Everybody’s business: 
Strengthening health systems to improve health outcomes, World Health Organisation, Geneva, Switzerland, 2007.
2 
1.1 Program Logic Model and Theory of Change 
The ZMLA logic model (as shown in Figure 1) was adapted from Donald and James Kirkpatrick's5
learning and 
training evaluation theory to conceptualize the pathway through which the program produces desired 
results. The model suggests that through exposure to program activities (workshops, mentoring, case 
studies), program trainees should: 
1. Demonstrate increased knowledge 
2. Be more confident in carrying out management and leadership functions 
3. Experience improved job motivation 
4. Improve management skills and behaviours that increase daily work performance 
Complementing the logic model is the McLeroy et al (1988)6
 socio-ecological model of behavioural change 
which suggests that behaviour is also determined by the following factors that led to the justification of 
recruiting cross functional7
 trainees: 
 Personal factors: Characteristics of individuals which include knowledge, attitude, behaviour, self￾perceptions etc. included in this the developmental history of individuals 
 Formal and informal social networks and social support systems: These include friends, families, and 
work systems 
 Institutional factors: These include social institutions, rules, and other organisational characteristics 
 Community factors: Relationships among organisations, institutions, individuals, cultural systems, beliefs 
 Public laws and policies. 
In addition, the ZMLA logic model was informed by work done by the Management Sciences for Health 
(MSH). 8
In MSH’s conceptual framework, after gaining confidence and skills in management and leadership, 
trainees should exhibit improvements in: 
1. Problem analysis, 
2. Use of data in decision making, 
3. Strategic planning 
4. Running of meetings, 
5. Supervising and 
6. Motivating employees. 
The MSH model postulates that as a result of these changes in trainees’ work practices, the work climate at 
trainees’ work places should improve and as a result, program performance and service delivery should also 
improve. For example, evidence of the success of such assumptions has been reported in Kenya for instance. 
 
5
Donald Kirkpatrick and James Kirkpatrick (2006). Evaluating Training Programs: Four Levels. Berrett-Koehler; third 
edition.
6 McLeroy, K. R., Bibeau, D., Steckler, A. and Glanz, K. (1988) An ecological perspective on health promotion programs. 
Health Education Quarterly, 15, 351–377. 
7
 Cross functional means that training cohorts reflect the diversity of a functional team across professions, hierarchy 
and community stakeholders. 
8
 Nancy LeMay and Alison Ellis. Evaluating leadership development and organisational performance. Management 
Sciences for Health, 2006 
3 
The six month leadership development program implemented in 67 management teams in Kenya showed 
sustained increases in health coverage at district level six months after the training program was completed.9
 
Figure 1: ZMLA Logic Model 
1.2 Overview of the ZMLA Program 
Established in 2010, ZMLA is an intensive and comprehensive management and leadership course tailored to 
meet the needs of health professionals and leaders working in the Zambian health system. The ZMLA 
program was designed to contribute to the improvement of management skills in order to increase the use 
of quality health services within 27 ZISSP target districts. More specifically, the ZMLA program aimed to: 
 Strengthen the capacity of provincial health level staff to mentor or coach district and facility staff 
 Support sustainable skills to mobilize and manage human and financial resources 
 Improve budget execution 
 Improve coordination and synergy of reproductive, child and malaria health services. 
 Improve management and leadership skills at the district and facility level 
The ZMLA training and mentorship sessions were delivered by a team of four to five trainers from BRITE, 
ZISSP, NIPA and MOH nominated experienced health managers. Each ZMLA training cohort had 20 to 30 
participants. Participants were selected and invited by ZISSP management specialists in consultation with the 
two ministries (MoH and MCHMCH) based on their profession and position in their management team or 
community. Once selected it was mandatory for participants to complete at least 75% of the course. Each 
training cohort was targeted to represent the diversity of professional groups, regional institutions, health 
facilities and the communities that ought to work as a team in a particular region. 
 
9
 Seims et al. Strengthening management and leadership practices to increase health-service delivery in Kenya: an 
evidence-based approach. Human Resources for Health. 2012, 10:25. 
4 
27 Target Districts
As of September 2014, the program had enrolled 767 trainees from all levels of the health system (central, 
provincial, district, hospital and community). The training program was implemented over three phases 
described below: 
 Training phases allowed the program to keep trainer to participant ratios consistent and manageable. This 
allowed ample time for support, follow up and mentorship of participants in between training sessions. 
Table 1: Recruitment and Completion Dates for ZMLA Implementation Phases 
Enrollment and
Completion Dates
Number of 
Trainees Enrolled
Number of Cohorts Trained
(20 – 30 in each cohort)
Phase I Oct 2011 – Jun 2013 474 18
Phase II Dec 2013 – June 2014 177 9
Phase III Jul 2014 – Dec 2014 116 4
The program was initially designed to be implemented over one year for each cohort. By the final phase of 
implementation, this time was halved. Participants from phase one of implementation were trained for a 
minimum of 12 months which extended to a maximum of 18 months for some participants. The first phase 
of programme implementation took longer than anticipated due to a number of unforeseen circumstances 
which included inter-alia; (i) fairly long MoH approval processes for the mentorship design and approach (ii) 
long periods of curriculum development and validation processes and (iii) periods when trainings could not 
be conducted due to competing national events such as planning or performance assessments. By the 
second phase of implementation, adjustments to the scheduling of course reduced the course length to just 
six months. The first phase of the program was completed in June 2013, the second phase concluded a year 
later in June 2014. The third phase of 116 trainees is expected to conclude in December 2014. 
Figure 2: ZMLA Implementation Locations 
5 
The ZMLA training approach combines three components – workshops, case study development and 
mentorship (Table 2). All three training components work synergistically to introduce simple management 
and leadership tools to trainees and encourage them to apply these tools in their day-to-day duties, medium 
term projects, and local initiatives. 
Table 2: Course Components and Structure 
Component Structure Objective
Workshops Four workshops lasting two to three days each and held 
quarterly in phase I and every other month in phase II/ III.
Build technical knowledge in a broad range of 
management and leadership core management areas. 
Case Study On-going small group (5-8) practical problem-solving 
activities at a health facility over six months 
Apply management and leadership skills learned in 
workshops to real-life settings in order to embed skills.
Mentorship Small group sessions of four to six members following each 
workshop, each lasting two days. Group sessions were 
supplemented by occasional individual one on one sessions 
with mentors held in person or by phone or email as and 
when mentees request them. 
Provide opportunities for personalization of workshop 
training, specific problem-solving assistance, and 
relationship-building with experienced managers.
To motivate participants and encourage full participation, the ZMLA course was accredited by NIPA. To 
attain NIPA certification with a higher diploma in management and leadership, participants were required to 
have attended four workshop sessions, four group mentorship sessions and successfully completed post 
training quizzes and case studies. The content of didactic training workshops is described in Table 3 below: 
Table 3: Summary of Curriculum Content
Delivery of ZMLA took into account a number of factors such as duration of the program, time taken per 
module away from participant’s work stations; ensuring adult learning methodologies were deployed 
Workshop Content delivered over 2-2.5 day for each Workshop
1 Module 1: Problem Definition
Problem definition
Strategic and operating planning
Strategic planning frameworks
Relevance of strategic planning to the 
organisation
Problem Analysis Tools
Prioritization
Critical Thinking and Pressure Testing
Critical Path
Module 2: Strategic Planning
Basics of supply chain management
Value chains and implementation frameworks
Model of care 
Developing implementation plans and Work plans
Internal and external stakeholder identification and 
analysis
Bottlenecks and the Marginal Budgeting for Bottlenecks 
Planning process in Zambia
Performance assessments
Zambia planning resources (tools and guidelines)
2 Module 3: Project Management 
Fundamentals
Organisational structures and charts
Definition of a project, project management 
and project manager role
Defining project success 
Leadership versus. Management
Running and participating in meetings 
Delegation
Decision rights
Teamwork
Providing useful feedback
Change management
3 Module 4: HR Management
Employee lifecycle
Recruitment and retention strategies 
Incentive systems and motivation tools
HR Development
Performance management – supervision, 
discipline and appraisal)
Module 5: Finance Management
Financial management overview
Budgeting, forecasting, and reporting
Cost management
Financial statements and reporting
4 Module 6: Strategic Information 
management
M&E as management tool
M&E approach
M&E metrics 
Building information culture
Using and maintaining information systems
6 
including group approaches to facilitate shared learning. In essence, ZMLA strived to balance the need to 
provide in-service training with the need to limit the absence of health workers in facilities. This was to 
sustain continuity in health service provision and not compound challenges the health system is already 
facing. Therefore to reduce the time spent by participants away from their workstations, each ZMLA 
workshop session was designed to take no longer than two-and-a-half days. 
Following each workshop, group mentorship was provided over an additional two days. As outlined in the 
subcontract between BRITE and ZISSP, 30% of ZISSP management specialist time was to be allocated to 
mentorship and follow up of trainees. To ensure ZMLA practices were supported and sustained within 
district and facility teams, ZMLA trained 47 senior managers from provincial and central MOH offices. These 
mentors were chosen and seconded by the MOH Directorate of Technical Support Services and were often 
high ranking individuals within the MOH management structure at provincial and central level. Mentors 
were taught how to mentor and coach colleagues on how to apply ZMLA tools to their case study and in 
their work setting, especially during key management cycles of: support supervision, performance 
assessment and planning. 
During the course of the training, participants in need of additional support could access one-on-one 
mentorship via phone, email and where possible, in person. Participants were free to contact any of the 
trainers and mentors any time after scheduled training and mentorship sessions whenever they needed 
support. 
At the start of the training program, participants were divided into groups of five to eight and tasked with 
designing a case study to address a particular public health issue using ZMLA tools. Case studies achieve the 
dual benefit of allowing teams to develop solutions that can be implemented in their work places while 
furthering skill development. The graduation of participants was only dependent on course attendance and 
submission of satisfactory case study project proposals, not necessarily the implementation of the case 
study. 
All ZMLA case study teams were tasked to identify and design solutions to a pertinent public health 
challenge in their districts or regions. The case study development process supported participants in their 
quest to find ways to improve service delivery in innovative and practical ways. The topics some case-studies 
sought to address included for instance; finding ways to increase the supply of medicines, commodities and 
equipment in a health facility; improve clinical staff to patient ratios; improve standard operation 
procedures; improve facility transportation; or increase community outreach. Each case study group was 
assigned a mentor to support them with the process of deciding which problems were most pertinent. 
Mentors also guided participants in assessing the feasibility of solutions generated using ZMLA tools. 
By design, phase one (I) ZMLA recruits were limited to developing case studies in 18 poorly performing 
maternal health delivery facilities. Teams in the first phase of implementation were asked to focus on 
maternal child health as a thematic focus in order to contribute to the government’s program to reduce 
maternal mortality in 2010.10 Trainees were expected to use district health maternal health indicators to 
 
10 In May 2009 the African Union launched the Campaign on Accelerated Reduction of Maternal Mortality in Africa 
(CARMMA) to trigger concerted and increased action towards improving maternal and new-born health and survival 
across the continent. As such, CARMMA is not a new initiative; rather, it is derived from the key priority areas enshrined 
in the AU Policy Framework for the promotion of Sexual and Reproductive Health and Rights in Africa (2005) and the 
Maputo Plan of Action (2006). The main objective of CARMMA is to expand the availability and use of universally 
7 
identify the worst performing health facility in their region (since participants are skilled clinicians, they were 
asked to use DHIS data to justify which facility their team would focus on in a succinct problem statement). 
Teams were given the freedom to use maternal health indicators that they collected routinely to justify 
which facility to focus on. They were further asked to use ZMLA tools to analyse root causes of the facility’s 
challenges, and design interventions that would improve maternal health services in that targeted facility. 
From a 2013 performance evaluation report,11 it was reported that among Phase 1 case study teams, only 
those that were comprised of district level health teams were in a position to implement their case studies 
without external funding. The nine teams which had a majority of participants from provincial or national 
institutions were far less likely to successfully implement their team’s case studies. This was because health 
facility sites which were the focus of the case study development process were not in their supervisory field 
of influence given their position in the health system. 
Building on the lessons learnt from the challenges of case study implementation in the first phase, in 
subsequent phases, case studies were not limited to maternal health themes or the worst performing site in 
the region. Instead, participants were free to focus on any pressing public health issue in their institution or 
region. Phase II case studies focused on a variety of major health challenges including but not limited to: 
reproductive health, malaria, HIV, TB, trauma and immunization at facility, regional or policy level. 
Table 4: Trainee Demographic Characteristics and Attendance 
Demographic Characteristics Total Trainees
Enrolled
Attendance
4 Workshop sessions only 4 Workshop + 4 mentorships 
Gender Female 222 71% 63%
Male 545 72% 63%
Recruitment 
Phase
Phase I (Enrolled between 2011-12) 474 82% 68%
Phase II (Enrolled in 2013) 177 91% 91%
Phase III* (Enrolled in 2014) 116 Ongoing( Ongoing
Job Category Administrative 197 68% 59%
Managerial 96 68% 58%
Technical Medical 224 74% 66%
Technical Non-Medical 132 69% 65%
Technical Public Health 81 83% 70%
Traditional Leader 9 56% 56%
Not Specified 28 89% 61%
System Level National 56 54% 39%
Provincial 199 83% 68%
District 357 67% 61%
Facility 146 78% 72%
Community 9 56% 56%
Residence Rural 464 72% 61%
Urban 303 72% 67%
Total 767 72% 63%
By September 2014, 552 trainees received a ZMLA workshop certificate of attendance after attending all 
four workshops (72% of 767 enrolled). By September 2014, 68 % of phase I trainees and 91% of phase II 
 
accessible quality health services, including those related to sexual and reproductive health that are critical for the 
reduction of maternal mortality.
11 2013 ZMLA Performance Evaluation Report. 
8 
trainees received a NIPA higher diploma in Management and Leadership after completing all four workshops 
and the related mentorship sessions plus having developed case-studies (Table 4 above). 
1.3 Evaluation Objectives and Questions 
The overall objective of ZMLA is to provide trainees with management and leadership skills that can be used 
in their day-to-day duties to improve service delivery. The program was evaluated to assess the extent to 
which intended program outcomes were achieved. The specific evaluation objectives and research questions 
are summarized in Table 5 below. 
Table 5: Evaluation Objectives and Study Questions 
Objectives Questions
To measure the extent to which the ZMLA 
program has contributed to improved 
management and leadership skills among 
program trainees
How has the program affected trainees’ management and leadership knowledge and skills? 
How has the program affected trainees’ management and leadership confidence and job 
motivation? 
How did trainees’ management practices change since ZMLA?
To evaluate organisational benefits that 
resulted from ZMLA trainings
How has the workplace environment changed since ZMLA training was implemented? I.e. 
in what ways has the program affected staff as individuals, their teams, and the 
organisation as whole? 
What organisational challenges did ZMLA training address?
What specific processes have individual trainees and workplaces established to address 
persistent organisational challenges after undergoing the ZMLA training?
Is there noticeable improvement in the development and implementation of action plans, 
performance management, etc., to address both organisational and service delivery 
challenges? 
To what extent was content from the formal training shared with other staff members who 
did not directly participate in the program?
To describe the overall conduct of ZMLA 
program and identify lessons useful for 
future program scale-up activities and 
implementation.
Is the format, content, and delivery of training, mentorship, and case studies relevant to 
the trainees? 
What aspects of ZMLA were most valued by trainees? 
What aspects did not have the desired outcome, were not useful, or need to be otherwise 
strengthened? 
How was the ZMLA program implemented in each phase, and was it implemented 
according to plan? (If not, what was changed?) 
How does the program compare (in terms of content, approach,) to other Management 
and Leadership training programs?
To evaluate the extent to which ZMLA 
training contributed to improved service 
delivery.
To what extent has ZMLA affected service delivery at case study sites?
This report highlights the successes and challenges of the ZMLA program based on the evaluation of Phase I 
and II implementation conducted between June 2014 and August 2014. 
9 
2 Methodology 
2.1 Study Design 
The evaluation employed a cross-sectional mixed study approach. Both qualitative and quantitative methods 
were used to address the evaluation objectives, and to triangulate findings. 
2.2 Target Population 
Responses from ZMLA trainees from the first two phases of implementation were included in the study to 
gain perspectives on the outcomes of the program so far. Stakeholders were interviewed to gain in-depth 
insights on the ZMLA program across all phases of implementation. These perspectives were from; 
participants, trainers, key NIPA figures, BRITE team leaders, MOH directors and ZISSP team leads. 
2.3 Data Sources 
2.3.1 Routine Data Collection Tools
Routine data collection tools were used to monitor ZMLA progress and program quality as described in Table 
6. Knowledge quizzes were designed to test all trainees’ knowledge of several concepts before and after 
being trained and how much knowledge was gained. Participant feedback forms were collected after each 
training session. All case study plans developed by participants were submitted to NIPA and reviewed to
assess participants’ ability to use ZMLA tools (Annex 1: Case Study Score Card, page 28). 
Table 6: Summary of Routine Data Collection Tools 
Data Collection Tool Key Evaluation 
Objectives Addressed
Type Notes on Data Collection 
Pre- and Post￾Workshop Knowledge 
Quizzes
(not presented here)
Objective 1 
Management and 
leadership knowledge 
and skills
Quantitative Ten question quizzes were administered to all 
trainees before each workshop and repeated after 
each workshop. 
Case Study Scoring 
Cards
(page 28)
Objective 1 
Management and 
leadership knowledge 
and skills
Quantitative NIPA assesses participants’ case study 
assignments against a standardized scoring sheet. 
To mitigate bias, two assessors score each 
assignment. 
Workshop and 
Mentorship Evaluation 
Feedback Forms
(page 30)
Objective 4
Program format, 
content, and delivery
Quantitative Administered to all trainees after each workshop.
2.3.2 Management and Leadership Survey 
The management and leadership survey was used to measure changes in confidence and job motivation 
among ZMLA trainee respondents. The tool was administered twice to 164 phase I and 280 phase II ZMLA 
trainees: at the beginning and at the conclusion of the training. The self-perception confidence questions in 
the survey were premised on Albert Bandura’s12 concept of self-efficacy which posits that a person who feels 
greater confidence in his or her ability to perform a specific behaviour is more likely to successfully perform 
that behaviour. The management and leadership tool (page 32) used a: 
 
12 Bandura, A. (1997). Self-efficacy: The exercise of control. New York: W.H. Freeman. 
10
 10-point Likert scale to measure responses to statements relating to management capacities, and a 
 5-point Likert scale to measure responses to statements relating to job-satisfaction and commitment to 
the organisation. 
Participant identity cards (IDs) were used in the questionnaire in order to pair perceptions per participant 
response before and after each training session. 
2.3.3 Workplace Climate Survey 
The workplace climate survey was implemented in December 2013 and again in August 2014. Data from the 
survey are analysed and presented in this report to complement and triangulate responses to qualitative 
probes. The survey was only administered to the worksites of phase II trainees that were trained from 
December 2013 to June 2014. Phase I participants did not fill in the workplace climate survey before they 
started ZMLA, making it inappropriate to use data collected from phase I to measure perceived changes 
without recollection biases. Phase I data was thus not used in this evaluation. 
The workplace climate survey was designed to measure existing conditions and changes in the workplace 
environment as scored before and after the course. Given that district health teams were more likely to be 
saturated with managerial trainees, only district community health office staffs were sampled. The survey 
targeted everyone at a workplace, regardless of exposure to the ZMLA course. 
The survey was self-administered with a ten-point Likert scale measuring personal perceptions of the work 
environment (page 36). The survey was administered at the start of the ZMLA trainings and shortly after the 
conclusion of the training (one to two months after the conclusion of trainings) in nine randomly selected 
districts community health offices. Unlike the management and leadership survey, there was no pairing of 
responses from the start and end of the training and the data set had no individual identifiers to track 
unique respondents. 
2.3.4 Qualitative Data Collection Tools 
Qualitative data was collected from 70 key informant interviews (Table 7), including 62 ZMLA trainees and 8 
ZMLA implementers (team leaders, mentors and trainers). The selection of respondents was purposeful and 
based on respondents’ positions within their organisations and their experience of the ZMLA program. 
Efforts were made to ensure that key informants from all stakeholders were represented, these are 
described below and in Annex 20: List of Anonymized Key Informants, page 60. All the provincial health 
offices were visited for key informant interviews. Districts community health offices were selected based on 
the implementation of case studies in phase I and the locations of the districts randomly selected for the 
phase II work climate survey mentioned above. The probes used for qualitative interviews are presented in 
Annex 13 and Annex 14. 
Table 7: Public Health System Level and Job Categories of Key Informants Interviewed. 
Public Health Systems Level Job Category or Perspective Number
District Case Study Respondent (part of case study 
team) 5
Clinical Care Specialist 1
District Medical Officer 9
Finance Officer 1
Health Management Information Officer 1
Hospital Administrator 1
Human Resource Officer 2
Medical Superintendent 1
NGO Coordinator 1
Planner 2
Public Health Officer 2
Sub Total 24
Health Facility Health Facility Manager 4
Nursing Officer 2
Sub Total 6
11
Public Health Systems Level Job Category or Perspective Number
District Case Study Respondent (part of case study 
team) 5
Clinical Care Specialist 1
District Medical Officer 9
Finance Officer 1
Health Management Information Officer 1
Hospital Administrator 1
Human Resource Officer 2
Medical Superintendent 1
NGO Coordinator 1
Planner 2
Public Health Officer 2
Sub Total 24
Health Facility Health Facility Manager 4
Nursing Officer 2
Sub Total 6
National MOH Director 1
Trainer 2
ZMLA Implementer 5
Sub Total 7
Provincial Clinical Care Specialist 1
Finance 3
Human Resource Officer 2
Management Specialist 8
Nursing Officer 2
Planner 3
Provincial Medical Officer 8
Principal Dental Therapist 1
Public Health 2
Sub Total 30
Grand Total 70
2.4 Data Management 
Quantitative data were entered and managed in Microsoft Access and SPSS. Data were cleaned and all 
outliers were validated by checking against hard copy entries. Qualitative data were recorded using digital 
recorders, downloaded and backed up at the end day of interviews. Transcription was done by trained 
research assistants. The lead consultant and the co-consultant checked all transcribed data to ensure quality. 
2.4.1 Quantitative Data Analysis 
Quantitative data were exported to SPSS Version 19 for analysis. Simple frequencies were used to explore 
the data. The basic unit of analysis was individual participants and ZMLA trained workplaces. Chi-squared 
tests were used to test the significance of changes in the proportion of respondents that felt more or less 
confident about leadership and management roles after ZMLA training. Chi-squared tests were also used to 
test the statistical significance of any changes in the proportion of respondents who felt that the work 
climate was conducive for productivity post ZMLA training. T-tests were used to assess the significance of 
observed differences between sample means of respondent self-scores pooled across management, 
leadership and work climate scores before and after training. For both Chi-squared and t-tests, were set at 
the 95% confidence level and 5% significance level. 
2.4.2 Qualitative Data Analysis 
Transcripts were exported to NVivo 10 for analysis. Coding was done by the main lead consultant. Content 
analysis and data coding followed pre-determined themes based on the research questions formulated from 
the evaluation scope of work. 
12
2.5 Study Limitations 
Where qualitative data was solely relied upon, the findings of this study are not generalizable. Additionally, 
self-reported Likert scales used in all the surveys are affected by a tendency of participants to report higher 
scores at baseline or before trainings, causing a ceiling effect that limits the upper scores if improvements 
higher than 10 or 5 were perceived. 
Based on the logic framework for the ZMLA program which hypothesized that if well applied and 
implemented, the ZMLA course should lead to improved service delivery in the health sector. However, the 
evaluation study design did not include a comparison group nor did it cluster randomize the training districts 
to control and implementation groups. It was not possible to exclusively attribute any noted changes in 
service delivery to ZMLA. Due to this study design limitation, this evaluation did not attempt to assess the 
impact of ZMLA training on service delivery. 
2.6 Ethical Consideration 
The study obtained permission from the University of Zambia Biomedical Ethics Committee and the Ministry 
of Health. All participants who responded to the MLA survey, work climate survey and qualitative probes 
were asked to provide written consent. For routine data, written consent was not sought from participants 
for module quizzes and training feedback forms although participants were communicated to verbally during 
training sessions that feedback was optional and no personal identifiers were required on the form. 
Participants were also told that the feedback would be used to inform amendments to and refinement of the 
course. All data were stored securely in a password protected Access database, and where analysed and 
presented, were anonymized. 
13
3 Results 
Presented below are the key results of the evaluation study according to each of the evaluation objectives. 
As indicated in the study methodology, findings from various data sources were used to validate and 
minimize research bias. The findings have been arranged according the following three (3) key objectives; 
 Contribution to improved management and leadership skills 
 Organisational benefits resulting from ZMLA trainings 
 Overall conduct of ZMLA and lessons learnt 
3.1 Contribution to Improved Management and Leadership Skills 
The first objective of the evaluation was to measure the extent to which ZMLA contributed to improved 
management and leadership skills among program participants. These aspects were assessed using pre and 
post training knowledge tests and the management and leadership confidence survey described in the 
methods section. 
3.1.1 Knowledge Gaps and Knowledge Gained 
Trainees completed quizzes to assess their individual knowledge of leadership and management prior to and 
following each of the four workshops. The quizzes were graded out of a total score of ten. In total, 1998 
quizzes were analysed. Error! Reference source not found. illustrates that ZMLA trainees were able to retain 
ew information from the training and improve their understanding of the specific skills taught in the 
workshops.
On average, knowledge levels increased by 2.1 points (38%, p<0.05 at 5% significance level) after each 
workshop. The most significant change in terms of gain in knowledge was noted for workshops 1 and 2, by 
3.0 (63%) and 1.9 (29%) points respectively. Workshops 4 showed the least change in skills score 1.3 (19%) 
points. 
Figure 3: Knowledge Scores Pre & Post Workshops 
Furthermore, to supplement individual knowledge assessments through pre and post- test quizzes, the 
program required participants to demonstrate their understanding of various ZMLA concepts by applying 
them to practical case studies. Table 8 shows participants’ scores from NIPA case study assessments. 
0 2 4 6 8 10
Workshop 1: Problem Definition and Strategic
Planning Frameworks
Workshop 2: Project Management Fundamentals
Workshop 3: HR Management and Finance
Management
Workshop 4: Strategic Information Management
Average Across all Workshops
Knowledge Scores Increased Following Each Workshop
Pre-Workshop
Average
Post-Workshop
Average
p<0.05 for all 
differences between 
pre and pos t test 
14
The results indicated that participants performed best in the area of creating relevant and sound monitoring 
and evaluation frameworks (93%), but had difficulties in developing a model of care illustration (58%), 
delegation plan (56%), and job descriptions (50%). 
Table 8: Phase I Case Study Scores
Scoring Factors Avg. Score Total Points 
Possible
Average %
of Points 
Obtained
M&E Framework 27.9 30 93%
Direct and cross-cutting functions listed 4.1 5 83%
SMART objectives developed 3.9 5 78%
Work plan/ Gantt Chart 3.8 5 75%
Solutions identified and prioritized 3.7 5 74%
Strategic planning and problem tools have been applied to ID root causes 3.7 5 73%
Decisions rights matrix 3.6 5 73%
Organisational structure for the project 3.6 5 72%
Budget 3.6 5 72%
Model of care illustration 2.9 5 58%
Delegation Plan 5.6 10 56%
Job description clearly defined for all key players in the organisational structure 5.0 10 50%
TOTAL 71.4 95 75%
N 72
See Annex 1: Case Study Score Card, page 28. Phase II case study scores were not complete at the time of the 
evaluation. 
3.1.2 Leadership and Management Skills 
To understand how the training improved individual leadership and management skills, both qualitative and 
quantitative methods were used. The demographics of the respondent’s to the management and leadership 
survey are presented below (Table 9). 
Table 9: Demographic Characteristics of Leadership and Management Survey Respondents 
Demographic Characteristic Phase I Phase II Grand Total
Public Health Systems Level Community 1 1 2
District 101 104 205
Facility 51 45 96
National 18 3 21
Provincial 109 11 120
Residence Rural 82 101 183
Urban 198 63 261
Gender Female 85 48 133
Male 195 116 311
Job Category Administrative 66 46 112
Managerial 26 24 50
Technical Medical 94 56 150
Technical Non-Medical 45 29 74
Technical Public Health 30 8 38
Traditional Leader 1 1 2
Not Categorized 18 18
Total: 280 164 444
A comparison of the average self-rated scores from 444 management and leadership survey responses 
before ZMLA and after ZMLA training showed an increase in the proportion of participants that felt 
adequately trained to undertake management and leadership tasks (F – Phase I: 63% before to 99% after, 
Phase II : 43% before to 98% after ). 
15
Figure 4: Self-Assessment of ZMLA Trainee Management and Leadership Readiness 
Note: Annex 6: Percentage of Trainees Feeling Adequately Trained in Management & Leadership (Crosstab and Chi￾square Test) page 43. 
In addition to participants feeling adequately trained after undergoing the training, an assessment of self￾perceived improvement in undertaking a range of management and leadership tasks significantly improved 
(p<0.05 at the 5% significance level). F highlights self-perceived percentage changes in mean scores for six 
broad management and leadership tasks (See Annex 5: Paired Samples Test (Mean Differences for 
Leadership and Management Tool). 
Figure 5: Percentage Change in Management and Leadership Self Confidence 
63%
43%
99% 98%
0%
25%
50%
75%
100%
Phase I (n =
274)
Phase II (n =
156)
Proportion of respondents feeling adequately trained to 
undertake management and leadership tasks pre and 
post training 
Before the course
After the course
15%
17%
18%
20%
22%
24%
19%
20%
19%
19%
20%
28%
25%
21%
0% 10% 20% 30%
Ability to manage human
resources
Ability to manage finance
Ability to build teams and
manage change
Ability to manage and use
information strategicaly
Ability to manage programmes
Ability to problem solve
Composite Score
% increase in self confidence to manage and lead after completing 
ZMLA training. 
Phase 1
(n=280)
Phase 2
(n=164)
p<0.05 at the 5% 
significant level for 
all differences 
between proportions 
observed. 
p<0.05 at the 5% 
significant level for 
all differences 
between proportions 
observed. 
16
The greatest changes were noted under the themes ‘problem solving’, ‘planning’ and ‘managing programs’. 
The differences across all themes were statistically significant for all variables in the leadership and 
management tool (Annex 5: Paired Samples Test (Mean Differences for Leadership and Management Tool)
and Annex 7: MLA Survey Themes, page 44).
Qualitative findings complemented the above quantitative results. More than half of trainee respondents 
(34 of 62) interviewed in-depth reported that they felt more motivated and that the training had improved 
the way they worked. 
“I think ZMLA is addressing what I would call the core problem from my perspective, from what we 
see in health sector, because the challenge really revolves around leadership and management. It 
comes down to the most efficient ways of using resources and someone has to have the leadership 
roles clearly outlined. I think it gives you the tools to break down the problem and see what solutions 
you can come up with at your level”. - ZMLA Trainee and mentor
As an example, some trainees felt more confident when preparing their annual action plans – captured as an 
‘ability to problem solve’ in the management and leadership survey. Using ZMLA tools on planning, some 
trainees reported that they no longer copied and pasted plans from previous years but developed original 
action plans with competence and efficiency. Similarly, many trainee respondents (39 of 62) felt better 
equipped to ‘plan’ as expressed by one trainee:
“…If you look at what we do, it’s mainly to do with planning. Like right now we are doing planning, 
we are developing our work plans, we look at the medium term expenditure framework going into 
the strategic plans…all those things that we were learning in ZMLA….,now we are not scared to go 
for planning…we now know how it should be done and my team is ready.” ZMLA trainee, Nchelenge.
3.1.3 Job Motivation 
To measure changes in job motivation before and after undergoing the ZMLA training, a series of questions 
were asked in the management and leadership survey. 
The focus of the assessment was around (i) commitment to their career in the organisation, (ii) optimism 
about future success in the organisation, (ii) degree to which the respondent enjoyed tackling new 
challenges arising in work, (iv) and degree to which the respondent felt positive and upbeat most of the time 
when working. F and F show the mean differences and associated confidence intervals for a 5-point Likert 
scale trainee responses before and after training. Commitment to a career in their organisation and feelings 
of positivity when at work were the most notable changes in trainee job motivation. These findings were 
consistent across phase I and II. 
Qualitative findings showed that some key informants felt that the ZMLA training contributed to their 
decision to stay in the public health service. The reasons given by just over half of the trainees (32 of 62) was 
that they now felt empowered and confident to be managers in the health sector and that they felt more 
motivated with skills and knowledge gained from the ZMLA. 
“Job motivation has improved, of course may be not in terms of remuneration but in terms of 
motivation, because when you know what you are supposed to do it becomes a motivation on its 
own.” ZMLA trainee, Mansa 
17
Figure 6: Phase I Job Motivation following ZMLA Training 
Figure 7: Phase II Job Motivation following ZMLA Training 
Note: Annex 5: Paired Samples Test (Mean Differences for Leadership and Management Tool) page 41. 
3.2 Organisational Benefits Resulting from ZMLA Trainings 
The second objective of the evaluation was to evaluate any organisational benefits that resulted from ZMLA 
trainings. This section presents results of the quantitative workplace climate survey (before and after 
training) supported by qualitative findings. 
3.2.1 Changes in the Workplace Climate 
In evaluating the organisational benefits that resulted from ZMLA trainings, quantitative data from the 
workplace climate survey were grouped thematically and presented below in F. Qualitative themes derived 
from in-depth interviews were used to complement quantitative findings. The characteristics of the phase II 
work climate respondents are presented in Table 10 below. 
 1.00 2.00 3.00 4.00 5.00
Feel positive and up most of the time when working
Enjoy taking on new challenges that arise in their…
Optimistic about the future success of their organization
Optimistic about own future success with the…
Feel more committed to a career with the organization
Phase II: Changes in Job Motivation Before and After ZMLA Training
Pre-ZMLA Post-ZMLA
 1.00 2.00 3.00 4.00 5.00
Feel positive and up most of the time when working
Enjoy taking on new challenges that arise in their…
Optimistic about the future success of their organization
Optimistic about own future success with the…
Feel more committed to a career with the organization
Phase I : Changes in Job Motivation Before and After ZMLA Training
Pre-ZMLA Post-ZMLA
p<0.001 at the 5% significant level for all differences between 
proportions observed except future success of organisation
p<0.001 at the 5% significant level for all differences 
between proportions observed except future success of 
organisation
18
Table 10: Work Climate Survey Respondent Characteristics 
Demographic Characteristic Pre-ZMLA Post-ZMLA
District Medical Office Chilubi 7 6
Chongwe 7 10
Lundazi 6 6
Masaiti 16 9
Nchelenge 7 6
Shangombo 12 2
Sinazongwe 9 8
Zambezi 9 11
Job Category Administrative 26 20
Manegerial 9 7
Technical (clinical) 23 15
Technical (non-clinical) 6 9
Technical (public health) 9 7
ZMLA/None ZMLA ZMLA Participant 47 36
Non ZMLA Participant 26 22
Gender Female 13 14
Male 60 44
Total 73 58
Note: Administrative cadres included finance, administration and HR officers. Technical non-clinical category included 
planners and strategic information cadres. Technical public health included public health officers, health promotion 
officers, surveillance officers and environmental health officers. 
Survey results show that respondents perceived improvements in the workplace environment, which were 
most prominent for human resource management issues, and the least prominent for ethics and 
accountability. The calculated before and after percentage change for work environment themes ranged 
from 5.8% to 13.4% as shown in F, but were only statistically significant for the resolution of HR issues 
(p<0.05 at 5% significance level). 
Figure 8: Evaluation of Work Climate Survey 
Note: For detailed analyses on individual items analysed in the workplace survey, please refer to: 
 Annex 8: Phase II Work Climate Survey Mean Scores Before and After ZMLA Training, 
 Annex 9 : Phase II Work Climate Survey Independent Samples Test
 Annex 10: Work Climate Survey Themes. 
0.6%
2.1%
4.6%
5.8%
6.4%
7.4%
7.5%
9.0%
13.4%
0.0% 4.0% 8.0% 12.0% 16.0%
Ethics and Accountability
Teamwork and Coordination
Development of HR
Supervision and feedback processes
Communication Levels
Performance Management of HR
Management of meetings
Data use, Problem analysis and solving skills
Resolution of HR Issues
% Change in Trainees Perception of their Work Climate Before and After ZMLA 
19
HR Performance Management 
The survey showed that ZMLA had improved trainees’ understanding of the internal annual performance 
appraisal system (APAS). The ZMLA human resource module covered the use and application of APAS in 
detail. Qualitative findings revealed perspectives from informants who shared that their managers started 
applying APAS in their workplaces after ZMLA training. Many trainees who came from districts or provinces 
where the top managers were not ZMLA trained complained of delays in being appraised and wished their 
managers also trained in ZMLA. 
“We are now doing APAS, though not all staff are trained. With the few who were trained through 
ZMLA, we have been helping others to make individual work plans. There were a number of staff who 
were not confirmed but now some of them have been confirmed after completing their APAS.”ZMLA 
trainee Chongwe 
Respondents explained that ZMLA had made it easy to understand APAS. In one district health team visited, 
it was reported that all employees had commenced the use of APAS. It was observed by the evaluation team 
that workplaces that had not fully implemented APAS had managers that had not undergone ZMLA training. 
Meetings 
Qualitative interviews reported improvements in the meeting culture and a greater appreciation for the 
importance of meetings. Almost all interviewees confirmed that the manner in which they now conducted 
meetings had improved after the ZMLA training. The workplace climate survey findings show that the way 
meetings were conducted in the workplace was amongst the top three work environment themes that were 
most improved post-ZMLA training. Meetings were brief and to the point and included action items and 
responsible individuals. 
“…..We have changed how we hold meetings. What we have started doing is that before a 
management meeting, we ask for departmental inputs …then set up an agenda with time allocation 
for each agenda item. If you want something discussed, it has to be sent early so that it is included in 
the agenda. We also circulate the agenda before each meeting. At the end of the meeting there are 
action points with responsible people clearly shown… this was not the case before ZMLA” ZMLA 
trainee, Chongwe 
Shared Vision, Teamwork and Coordination 
Interviewees reported that a shared vision was more emphasized following ZMLA training. Trainees had a 
greater understanding of their role and how it supported the shared vision. Improvements in management 
accountability and ability to delegate were also noted. 
“One of the things that has changed is [that there is] at least more coordination between the 
members at the district health office here and more importantly, being able to document whatever 
we discuss in management and having action points, and following those up. We have reached a 
point now where almost everyone knows about management and you can easily delegate.” ZMLA 
trainee, Nchelenge
Improvement in shared vision, team work and coordination seemed to have improved more in work places 
where the overall manager and his/her were trained in ZMLA. 
“Training of top leadership was extremely key. It was key because it also made sure that the 
leadership understands what’s happening. I think it puts the whole system of the health sector into 
perspective and I think it was really strategic to train provincial medical offices and clinical care 
specialists ……indeed the people that may not have been trained in the concepts may not really 
appreciate them” ZMLA mentor and trainee, Solwezi. 
20
Financial Management and Accountability 
Although financial management and accountability was not explicitly covered in the workplace climate 
survey, qualitative interviews indicated that for some work places, appreciation of finance systems among 
trainees improved post ZMLA training. In some work places, trainees who were not financial managers 
appreciated the role of the accountant more and the need to adhere to financial regulations as shared 
below: 
“The ZMLA training drew participants from all departments. The biggest challenge we had was 
managing the human resource, the people from different units. As accounts unit, I think we are very 
knowledgeable with financial regulations, but that was not the case with people from other 
departments. After the ZMLA training we are seeing great improvement and compliance to financial 
regulations, there is now strict adherence to budgets. People here were not very comfortable with 
the bureaucracy we have in accounts, but after the training they appreciate and they understand the 
importance of observing these controls.” ZMLA trainee, Kasama
Managers were said to have been made aware that finance documentation should be made available to 
support payout of funds before they could append their signatures. There were less audit queries in one site 
visited: 
“There is great improvement and this is evidenced by the reduction in the number of audit queries. So 
our financial management has greatly improved, audit queries have gone down.” ZMLA trainee, 
Chipata 
However, quantitative analysis of the workplace climate survey showed that ethics and accountability 
showed the least improvement after ZMLA trainings across all ZMLA worksites.
3.3 Overall Conduct of ZMLA 
To assess the aspects of the training program which were most useful or require amendment, analysis of 
routinely collected participant feedback data was triangulated with themes generated from key informant 
interviews. Key themes and aspects of the program arising from the analysis provided perspectives on 
participant commitment, the quality of workshop sessions, mentorship activities, case study experiences, 
levels of stakeholder engagement and implications for sustainability of the program. 
3.3.1 Participant Commitment 
Participants’ commitment to ZMLA was not as high in the first phase as in the second phase. This was, in 
part, because ZMLA was not well known to MOH management teams, and it was not originally designed to 
be NIPA certified. To reduce phase I attrition through NIPA certification, qualitative findings suggest that 
ZMLA became better known and participants’ enthusiasm for and commitment to the course increased. 
“In the initial stages when they didn’t know what the training entailed, there was a bit of resistance 
and some people couldn’t complete the course. Now everyone wants to get ZMLA training because 
they have discovered that you get the knowledge and a certificate that you can use elsewhere” 
Course implementer, Northern Province 
Attrition from as much as 32% in the first phase was reduced to just 8% in the second phase of 
implementation. The reduction of dropout rates may also have been due to the reduced length of the 
21
program by merging workshop and mentorship sessions into the same week rather than spacing sessions 
four weeks apart as was originally designed. 
From all interviews conducted, it was apparent that ZMLA was popular among participants. One reason 
given across all cadres was that ZMLA was filling an important gap in leadership and management 
knowledge and skills among health workers. The course was said to increase self-confidence in leaders and 
provided them with practical tools to address everyday challenges. 
The fact that the course was officially certified by NIPA, made it even more valuable for participants. They 
admitted that they could use the knowledge gained in their current role, but also to apply for promotion 
within their current organisation or elsewhere. The popularity of the course among health workers who had 
not undergone ZMLA training increased when NIPA started awarding diplomas. Participants who had 
dropped out of the course in phase one came back to complete the course in phase two when they realized 
that they would get a diploma at the end of the ZMLA course. 
“Yes, we had a situation where some people would be given a certificate of participation because 
they missed out sessions and they wanted to upgrade themselves to a higher diploma, so they came 
to join the second phase” Course implementer 
3.3.2 Content and Delivery of the Course 
More than 2000 feedback responses from post training evaluation forms showed that over 95% of all 
respondents either agreed or strongly agreed that the training content was relevant to daily work. Similarly 
over 90% of respondents found the workshop delivery approach and materials to be of high quality. Most 
importantly, almost 99% of all respondents reported that they would recommend the course to a colleague 
(Table 11 and Note: *Indicates higher levels of non-response.
Table 12). Quantitative data analysis of feedback forms showed that the most appreciated aspect of the 
training was project management, human resource management and strategic information management. 
These findings were consistent with the management, leadership and workplace survey findings previously 
presented. 
 Table 11: Participant Perceptions of the Quality and Relevance of Workshops 
Statement % Who Agree or 
Strongly Agree
Total Number of 
Responses (n)
I would recommend this workshop to a colleague. 98.8% 2007
The knowledge and skills taught in the modules will be useful in my daily work. 97.9% 2012
The level of module content was appropriate. 97.4% 2017
The examples provided in modules were relevant and appropriate. 95.9% 2023
The workshop format was conducive to a positive learning experience. 95.2% 1991
The materials for the case study were appropriate and well-prepared 90.7% 1672*
Note: *Indicates higher levels of non-response.
Table 12: Participants Perceptions of the Quality and Relevance of Training Modules 
Module % Rating Quality as “Good” or 
“Very Good”
% Rating Relevance as 
“Good” or “Very Good”
Total Number of 
Respondents (n)
1. Problem Definition 94.1% 96.9.5% 459
2. Strategic Planning Frameworks 87.6% 95.0% 508
3. Project Management Fundamentals 98.1% 99.5% 413
4. HR Management 97.9% 98.4% 571
5. Finance Management 94.9% 96.2% 564
6. Strategic Information Management 96.7% 98.5% 391*
22
Note: *Two cohorts of feedback forms (~60) were misplaced in the field during implementation hence responses are 
lower. 
Case Studies 
Although only 4 out of 17 case study groups implemented their plans from the first phase of 
implementation, the majority of respondents felt that implementation of case studies was crucial in 
ensuring trainees actually translate ZMLA management tools and techniques into everyday practice. Case 
studies were seen as an opportunity to apply knowledge in real and practical settings. Most case studies 
were designed to address actual problems which needed solutions. Participants recognized that case studies 
were in line with the call for operational research which the Ministry of Health is advocating. 
“I think for me…. it was very good. They taught us how to identify the problems, how to [come up 
with] objectives, activities and the GANTT chart. ….we don’t do them every time, it’s mostly clinical 
issues… Some of these things like how to do a problem statement, it was very good, also how to do 
[be] SMART, it was very good and well done. It also taught some of us some computer [skills] as I 
didn’t know for example how to do the GANTT chart, but after the ZMLA I can do it nicely, but also 
the case study for me was very good, you easily identify a problem and come up with a study.” 
Qualitative interviews highlighted several factors that contributed to non- implementation of case studies in 
the first phase, this included conceptual weaknesses in the case study approach, composition of case study 
teams and miscommunication on the practicality and funding of such case studies. 
“The mix of group members for case studies was crucial. In the first cohort we noticed that the mixing 
of Central, Provincial and district participants did not work well when it came to coordinating 
activities. In later trainings, case study group members were coming from the same district. We have 
since noticed better coordination and team work in implementing case studies” Course implementer
Subsequent phases of trainees were made aware in advance that their case studies would not be funded by 
ZMLA and were asked to look for ways and resources to implement their own case studies. Some phase II 
participants interviewed reported that they had managed to implement their case studies despite the 
limited time that had lapsed since completing their training. For example, key informants from Lundazi were 
among few participants who claimed that they had implemented case studies despite the shortened course 
duration in phase two. 
“Here in Lundazi, we have done very well. All the ZMLA trainees have implemented their studies…We 
had to review our action plans again, so we could include some of the problems identified through 
our case studies.” ZMLA trainee, Lundazi 
3.3.3 Mentorship 
While the intention of the mentorship strategy was good, it also had shortfalls and challenges. ZMLA 
provided training to all potential mentors. This was aimed at generating a pool of MOH mentors that would 
support the ZMLA trainees from within the Ministry of Health. 
Long-term mentorship and follow up faced major challenges and affected ZMLA trainees from all phases. 
Some mentors only received partial training and did not grasp the ZMLA concepts to the same extent as 
those who attended all four Trainer of Mentor (ToM) sessions. Because a third of mentors seconded by the 
Directorate of Technical Support were high ranking, many of them did not have time to complete the course 
or mentor. It was therefore clear that about half of the mentors trained were not able or available to 
23
mentor others. Subsequently, such mentors were not invited to provide mentorship, and a smaller more 
flexible team of mentors was eventually established. Before the core group of mentors could be established, 
the mentorship approach in phase I experienced some teething challenges. 
Some mentors did very well and learned through active supervision, while others struggled. Twenty one (21) 
mentors who trained as mentors and had also completed the full ZMLA course as trainees felt better 
equipped to be mentors. Mentors who were not trained in ZMLA first felt that it would have been better for 
them to complete the course and receive similar qualifications as their students before becoming mentors. 
Unlike in the second phase of implementation, mentors were not strictly assigned to particular training 
cohorts in the first phase. This meant that participant would receive advice from as many as three different 
mentors in the course of training. This resulted in reports of conflicting advice from different mentors given 
to participants as shared by one participant from Nyimba which contributed to mistrust and confusion 
during the mentorship process. 
“There was this time when we had already done about 70% of the case study and this person (mentor) 
comes in and says this is not the way it should be done. So that was more counterproductive and so we 
had to start afresh defining the problem before coming up with solutions.” ZMLA trainee, Nyimba
As outlined in the subcontract between BRITE and ZISSP, 30% of management specialist time was to be 
allocated to mentorship and follow up of trainees. However, qualitative findings suggest that this was 
dependent on his or her individual commitment. Management specialists noted limited mentorship funding 
as a major obstacle to mentorship follow-ups, especially for trainees in areas far from provincial 
headquarters. 
Ministry of Health mentors from the national level also complained that they felt that they had let down 
trainees as they also could not follow up groups that they were supposed to mentor. It was clear that 
trainees would have loved to see more of their mentors, both from provincial and national level, in their 
workplaces and during case study implementation. In the program design, one on one mentorships were 
envisioned to be conducted face to face by ZISSP Management Specialists. As shared by both MoH and ZISSP 
mentors, face to face mentorships were not feasible. Most contact between trainees and mentors was 
limited to phone calls and e-mails. 
The absence of local mentorship was also associated with diminishing effects of ZMLA. 
“The mentorship was a bit weak, there is need to improve in this area… it is a good idea but it has not 
worked well. I have not been in touch with my mentors after the group mentorship...you know we need 
guidance in the field” ZMLA trainee, Mansa
3.3.4 Engagement of Stakeholders 
Apart from ordinary participants or ZMLA students, other key stakeholders (relevant to the health sector) 
who directly or indirectly contributed to the smooth implementation of ZMLA included representatives from 
Non-Governmental Organisations (NGOs), the church and traditional leaders. Their views during interviews 
were unanimous, for instance, they all indicated that ZMLA course had allowed them to work better with 
both ministries responsible for health (MoH and MCDMCH). For example, the study indicates that there was 
better coordination and synergy in Eastern province where the Catholic Church, a local traditional leader and 
MOH embarked on a joint intervention to improve cervical cancer screening. The chief used his position to 
educate the community while the church used church services to share information on the importance of 
cervical cancer screening. The two ministries (MOH and MCDMCH) carried out cervical cancer screening. 
They had since noticed improvement in cervical cancer screening in target communities. 
24
“We had a problem of low coverage for cervical cancer screening. We sat down with our stakeholder 
from the Catholic Church and the local chief. Both of them were part of the ZMLA training. We 
divided roles whereby the church was telling members about the importance of cervical cancer 
screening and the chief did the same to his subjects. We have since observed an increase in the 
number of women coming for cervical cancer screening” ZMLA trainee, Chipata
Many MOH or MCDMCH staff who attended training sessions together with traditional leaders, NGO and 
church representatives appreciated and valued their contributions which were outside the public service but 
critical for the health sector. However, the main concern from these key stakeholders was that they were 
not usually consulted on the course timings and these often clashed with their programs. They also felt that 
the ZMLA program was more tailored to public health and medical interventions with no content targeted at 
the private sector. It was also noted that the number of stakeholders invited for trainings was low. This was 
attributed to the limited spaces which were given priority to MOH/MCDMCH staffs by course organizers. 
The importance of buy in from top leadership 
Our findings showed that provinces and districts where the top manager had undergone ZMLA training were 
more likely to practice and apply ZMLA principles. In places where the PMO and DMO had undergone ZMLA, 
efforts were made to send staff not trained in ZMLA to subsequent phases of trainings. Participants whose 
team leaders where not trained in ZMLA felt that their new skills were not fully utilized in their team as 
shared by one respondent: 
“What pains is that I have gotten this extra skill but nobody here is able to recognize it…I wish our 
boss had done ZMLA course, he will appreciate my skills much more” ZMLA trainee, Mansa 
3.3.5 Other Management and Leadership Courses 
Most trainees (55 of 62) reported that they never attended other management courses besides the ZMLA. 
However, a few (generally older) trainees had attended courses operated by NIPA, PADESA, and follow on 
Zambia Prevention Care and Treatment program (ZPCT II) on leadership and management in the past. Those 
trainees noted the major differentiator of the ZMLA is that it is well adapted and tailored to the current 
Zambian health context. The other strength was that ZMLA targeted management teams rather than 
individuals. It was run in short modules and did not take health workers away from their workplace for too 
long. This was in contrast to the course which was run by NIPA and PADESA which targeted individual health 
workers and was a full-time course requiring health workers to leave their work stations. 
“The PADESA and NIPA course was very long and was more theoretical. This was very different from 
ZMLA which is very practical and hands-on. It takes the form of adult learning with opportunity for 
mentorship and group learning” Course implementer 
“I attended a ZPCT governance workshop for managers … it was more on governance but for the 
ZMLA, we went straight into even how to do the frameworks properly….how to line up your 
objectives. We also learnt how to make up SMART objectives without having to crack your head 
throughout the night. There is just a way it flows.” ZMLA trainee, Nchelenge 
3.3.6 Sustainability 
This study looked at ways in which the ZMLA course could be made more sustainable. It was clear from the 
responses that most participants wanted ZMLA to continue in its current form and even extend it to other 
cadres at health facility and community levels. Unfortunately, the course was perceived as a free donor 
supported program. Over half of the respondents expected to be paid allowances for attending the course. 
Interestingly, a majority of respondents (65 of 70) admitted that the way the course was run was expensive 
25
and unsustainable. Yet no discussions were openly held with participants or districts on what they could 
contribute to the cost of running the course. 
It was suggested that costs could be cut by using government infrastructure rather than private lodges to 
deliver the program. However, respondents felt that the course must remain as practical and short as it is 
and be done in the districts to avoid taking health workers away from the workplaces for a long time. 
Trainees also felt that the mentorship component should continue, even if the course was handed over to 
NIPA. They suggested institutionalization of ZMLA as was the case with PADESA and the NIPA health 
management course. NIPA was one institution that most respondents felt had the capacity to run such a 
course followed by the University of Zambia. 
Willingness to pay 
A total of 69 out of 70 respondents interviewed suggested that more staff were likely to pay for the ZMLA 
course if the program was made mandatory by MoH/MCDMCH or even the Health Professions Council 
before assuming management or leadership roles. In addition, ZISSP Management Specialists in the 
provinces indicated that they received frequent requests on ZMLA about how one could enroll, suggesting 
that ZMLA certification is in demand. 
“It is possible to make individuals pay for ZMLA, but the catch is to make it mandatory for certain 
positions. This will force everyone to get the ZMLA qualification; it all depends on government 
commitment” ZMLA mentor, Mansa 
It was suggested by almost all respondents who were participants that if the ZMLA course was made a 
mandatory for staff taking up leadership roles, this will be a motivation for those aspiring for leadership 
positions to pay for the course either through their institutions or personally. When asked, most (47 out 70) 
participants believed that the course could be supported through inclusion in local action plans to include 
ZMLA in their annual action plans with clear budget lines. 
During interviews, the interviewee probed further and asked individual trainees how much they would be 
willing to pay for the ZMLA course as a tuition fee contribution. The amount suggested seemed to depend 
much on the health worker cadre. Medical doctors suggested the highest figures. Amounts suggested ranged 
from 1,500 to 30,000 ZMK for the whole course. NIPA students currently pay 3,500 to 7,000 ZMK for a level 
diploma course. An alternative perspective from some respondents was that the introduction of tuition fees 
and self-sponsorship should be done with caution. There is a risk that tuition fees could leave out those most 
in need of ZMLA training. 
26
4 Conclusion 
The management and leadership self-confidence survey results and case study scores show that ZMLA 
contributed to health workers perceptions of improved leadership and management skills. In addition, a 
significant proportion of ZMLA trainees reported that the ZMLA program had improved their job motivation 
levels. In terms of course content and knowledge retention, the evaluation results indicate that participants 
performed best in the area of creating relevant and sound monitoring and evaluation frameworks. 
Monitoring and evaluation was one of the most appreciated modules and was correctly applied in most case 
study project plans graded. Respondents also perceived positive changes in the resolution of Human 
Resource (HR) issues in their workplace. The greatest perceived improvements in the workplace were 
reported by participants whose senior managers were also trained in ZMLA. 
However, about half of the teams trained had difficulties in developing model of care frameworks in supply 
chain management, planning for delegation and developing job descriptions. Despite participants 
appreciating and applying monitoring and evaluation concepts successfully, only half of all district level case 
study projects were implemented. Even participants from teams that managed to implement their case 
studies reported that they did not monitor their projects. Furthermore, the mentorship and follow up of 
case study projects was reported as weak. Due to evaluation study limitations, the impact of case studies on 
service delivery could not be assessed. 
The current model of conducting training and mentorship workshops near trainee facilities and in combined 
sessions is practical and minimally disruptive for trainees and their employers. Most participants reported 
that they would recommend the training to colleagues and other cadres at health facility or community 
level. In summation, top leadership in the ministries responsible for health, participants, mentors, and 
trainers interviewed openly appreciated the program and reported that they would like ZMLA to continue. 
27
5 Recommendations 
In response to the conclusions presented above, the recommendations to ZMLA program implementers are 
presented below. 
 It is recommended that the ministries responsible for health (MoH and MCDMCH) should compel 
health program managers and supervisors at all levels of the organisational structure to undergo the 
ZMLA training. 
 The ministries responsible for health (MoH and MCDMCH) with support from partners should seek 
to support the institutionalization of ZMLA with local academic institutions such as the National 
Institute for Public Administration (NIPA) and University of Zambia. District institutions should 
budget for ZMLA tuition fees for managers in need of management and leadership skills through 
their annual action plans. In the long term, ZMLA should be included in clinical training courses. For 
clinicians with known ambitions of managing health institutions, ZMLA should be offered as an 
optional in-service training course at eligible training institutions across the country. 
 Going forward, BRITE should design a more robust evaluation approach to effectively track the 
impact of ZMLA trainings. Experimental or quasi-experimental approaches such as clustering and 
randomization to intervention and control groups for instance could demonstrate attribution of 
ZMLA activities to changes in service delivery. Such knowledge would be needed to inform scale up 
of the program across the country. 
 As BRITE is the originator of the curriculum, BRITE should revisit ZMLA content relating to the 
development of models of care in supply chain management, delegation and the development of 
health worker job descriptions. More time should be spent during training sessions to explain these 
concepts. Additionally, mentorship should emphasize more on the concepts that case study teams 
struggle to grasp and apply. 
 Going forward, the ZMLA program should improve the follow-up and mentorship of participants 
after group mentorship sessions to support case study development. 
 Although a key result area for ZMLA was to develop mentorship and coaching capacity in the 
ministries responsible for health, mentors should only be allowed to mentor colleagues after they 
themselves have completed the program. 
 In order to translate theory into practice, NIPA should request that participants implement case 
study projects in order to graduate. Trainees should be encouraged to leverage off other health 
initiatives and programs in their districts to support their case studies. 
28
Annex 1: Case Study Score Card 
Training Location of Cohort:_________________ Total Score: _______ /100 
Project Title: ________________________________________________________________________ 
Participant ID: _______________________________________________________________________ 
Please score each case study based on the listed factors (in brackets) below to prove the level of 
application of taught ZMLA tools. A score of 0 means that factors are not evident and 5 factors are 
most present. 
# Scoring Factors To Consider By Content Taught In ZMLA Score: 0 1 2 3 4 5 Score
Strategic planning and problems tools have been applied to ID root causes
(atomization diagram, PESTLE, McKinsey 7S, SWOT/internal and external, 5Ps)
Solutions identified and prioritized
(Well numbered solutions, solutions result from problem analysis, feasibility 
assessment given resources – use of matrix or 80/20 or CEA)
Model of care illustrated
(presence of flow diagram, well-articulated direct functions)
Direct and Cross cutting functions listed (either in diagram or in a table)
(Indirect objectives: Finance, HR, Governance, M&E, etc,
Direct objectives :Address problems from analysis/prioritisation )
SMART Objectives developed
(Specific, measureable, achievable, relevant and time bound. They must be 
directly as a result of findings in the problem analysis - penalise by 1 if not)
Work plan / Gantt Chart/ Critical Path Evident
(Listed goal, objectives, activities, period of implementation, critical path 
must be evident in the shaded part of the gantt chart or done elsewhere)
Responsible People or Assigned Roles in Gantt chart relate to Organisational 
Structure?
Organisational Structure for the Project
(Could be normal work organogram with relevant staff, or matric reporting 
structure, depending)
29
Job description clearly defined for all key players in the organisational 
structure (job title, job summary, key responsible areas, key reporting lines, 
qualifications)
Meeting schedule developed
(list of suggested dates
Decisions rights matrix developed with all stakeholders considered
# 0 1 2 3 4 5 Score
Change management or Risk Analysis Evidence: 
Change management process considered.
Budget – all activities costed for:
(clearly presented, all activities costed, unit costs listed, 
quantities listed, and cost effective – value for money, realistic 
and prioritized)
Subtotals for the budget used and allocated amounts realistic 
and able to achieve goal?
M and E Framework:
(Is the framework clearly presented?)
Inputs and Outputs are linked to the activities
Outcome and Impact Indicators linked to Objectives and Goal of 
the Project?
Reporting Frequency achievable
Data Sources Identified
Data Collection Methods or Tools clearly listed
Total Score:
Additional comments: 
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________ 
30
Annex 2: Participant Feedback Form 
Name of Participant (optional) 
A. Module and Case Study Evaluation
This workshop was intended to enhance management and leadership knowledge and skills, and this form will 
let us know how well we achieved our goal. Please evaluate the modules and case study on Quality and 
Relevance according to the rating scale shown above by circling the number that corresponds to your opinion. 
Furthermore, we are very interested in your feedback, if you have additional comments, please speak with us.
Training Component Quality Relevance
Module 3 - Program and Project Management Fundamentals and 
Organisational Structure
1 2 3 4 5 1 2 3 4 5
Case Study 1 2 3 4 5 1 2 3 4 5
B. Presenter Evaluation 
Please evaluate each presenter in the following areas by circling the number which indicates your level of 
satisfaction. Use the rating scale shown above
Presenter: Knowledge/Expertise
Ability to explain
key concepts
Ability to answer 
questions clearly
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Rating Scale
1=Very Poor 2=Poor 3=Average 4=Good 5=Very Good
31
C. Content Evaluation
Please indicate your level of agreement with the following statements pertaining to the workshop 
content. 
Rating Scale 1 = Strongly Disagree 2 = Disagree 3 = Neither Agree/Disagree 4 = Agree 5 = Strongly Agree
1. The level of the module content was appropriate. 1 2 3 4 5
2. The examples provided in modules were relevant and appropriate. 1 2 3 4 5
3. The knowledge and skills taught in the modules will be useful in my daily 
work.
1 2 3 4 5
4. The materials for the case study were appropriate and well-prepared. 1 2 3 4 5
5. The case study provided an opportunity to apply knowledge and skills 
covered in the modules.
1 2 3 4 5
6. The case study provided key insights that I can apply within my daily work. 1 2 3 4 5
7. The workshop format was conducive to a positive learning experience. 1 2 3 4 5
8. There was adequate time for discussion, questions and answers. 1 2 3 4 5
9. I would recommend this workshop to a colleague. 1 2 3 4 5
D. Additional Feedback 
1. Please provide one example of something you will do differently as a result of this workshop.
2. What would you change about this workshop to make it more useful?
3. What was the most useful concept, skill, framework, or tool that you learned?
4. Any additional comments?
32
Annex 3: Leadership and Management Tool
PART I: 
System level: 
□ National, 
□ Provincial, 
□ District, 
□ Facility 
□ Other (Specify)_____________________
Name of Organisation________________________________(e.g. Mwinilunga DMO, 
Solwezi PMO or FHI360)
Province Name:___________________________________________________________
District Name:____________________________________________________________ 
Highest education qualification:_______________________________________________
Profession:_______________________________________________________________
Current Job Title:__________________________________________________________
Number of years served in the current Job:________________ (Years)
Participant ID: _____________________ Date: ___________________________
Pre-training □ Post-training □
33
PART II: 
Instructions: Following is a list of different leadership and management tasks/activities. For each one, rate 
how confident you are in performing the task as of now. Rate your degree of confidence by recording a 
number from 0 to 10, using the scale given below. If you feel that a statement is not relevant to you and/or 
your job, you may write “N/A” next to the statement in place of a rating.
0 1 2 3 4 5 6 7 8 9 10
Cannot do at all Moderately certain can do Certain can do
How confident are you in your own ability to perform each of the following tasks? Self-Rating 
(0-10, or N/A)
Provide constructive feedback on a regular basis, in a way that helps those I supervise 
improve their performance.
Use non-monetary strategies (such as praise, public recognition, and reminding staff of the 
importance/value of their work) to motivate those I supervise.
Identify staff development needs and work with them to plan appropriate trainings, 
mentoring opportunities, or other ways of addressing the needs. 
Chair productive and efficient meetings that begin and end as scheduled.
Identify the most important “root” or underlying causes of specific challenges within my 
unit/department that affect the healthcare system (such as high maternal mortality, or ART 
non-adherence).
Prioritize among possible solutions/interventions to address healthcare challenges, to identify 
and implement those that will have the highest impact with fewer resources.
Engage and maintain effective communication with internal and external stakeholders 
(including other departments or levels within the MOH, community members/organisations, 
NGOs, and other ministries) when planning and implementing strategies to address 
healthcare challenges.
Develop detailed work plans that specify timelines, milestones, and roles/responsibilities of 
specific people, to address specific challenges or achieve certain goals (such as increasing use 
of family planning services, improving access to EMCOR, etc.).
Regularly check in on progress, and hold members of my organisation/unit accountable for 
following through on objectives and activities from our work plan or Action Plan.
Ensure that projects I oversee are carried out within the allotted budget, time and resources.
Promote teamwork and collaboration among staff and different units in my organisation.
34
How confident are you in your own ability to perform each of the following tasks? Self-Rating 
(0-10, or N/A)
Communicate well with staff, to ensure that they understand the overall picture (strategic 
vision) of our unit and are informed about any changes that may be introduced to our 
organisation.
Follow up to facilitate prompt resolution of staff HR issues, such as promotion and training.
Ensure that staffs reporting to me have up-to-date job descriptions, with which they are 
familiar, and understand reporting lines within our unit/department.
Take appropriate corrective action to address staff performance problems as soon as I am 
aware of them. 
Identify and apply cost-effective approaches to maximize use of the organisation’s resources.
Create realistic program or project budgets based on historical data, current cost information, 
and other relevant information sources.
Effectively interpret and use the data available to me (such as from HMIS, performance 
assessment reports, and other sources) to guide planning, decision-making, and quality 
improvement.
Create reports, charts and graphs which succinctly and effectively communicate relevant data 
to stakeholders.
Analyse the delivery model for health services to identify the major gaps and bottlenecks 
affecting the quality of care and health outcomes.
Identify and implement necessary changes in delivery models for health services to address or 
correct gaps and bottlenecks affecting quality of care and health outcomes.
Use lead times, inventories, and other supply logistics information to ensure that stock levels 
of critical supplies remain adequate at all times.
35
PART III: 
Instructions: For the next set of statements, please think about your current situation at work. Circle the 
number (from 1-5) that best describes how you feel, using the scale given below. 
1 2 3 4 5
Strongly disagree Disagree Neither agree/ disagree Agree Strongly agree
I feel I have been adequately trained to meet the management and leadership challenges I 
encounter in my position. 
1 2 3 4 5
I feel positive and up most of the time I am working. 1 2 3 4 5
I enjoy taking on new challenges that arise in my position. 1 2 3 4 5
I am optimistic about the future success of my organisation. 1 2 3 4 5
I am optimistic about my own future success with the organisation. 1 2 3 4 5
I feel more committed to a career with the organisation this year than I did a year ago. 1 2 3 4 5
36
Annex 4: Workplace Climate Tool 
Workplace climate survey questioner 
PART I: 
System level: 
 □ National, □ Provincial, □ District, □ Facility □ Other (Specify)_____________________
Name of Organisation________________________________(e.g. Mwinilunga DMO, 
Solwezi PMO or FHI360)
Province Name:___________________________________________________________
District Name:____________________________________________________________ 
Highest education qualification:_______________________________________________
Profession:_______________________________________________________________
Number of years served in the current workplace:________________ (Years)___________
Current Job Title:__________________________________________________________
Number of years served in the current Job:________________ (Years)
Have you been selected to participate in the Zambia Management and Leadership Academy (ZMLA)? 
  Yes  No  Don’t Know
Has your supervisor been selected to participate in the Zambia Management and Leadership Academy (ZMLA)?
  Yes  No  Don’t Know
Gender  Female  Male
37
PART II: 
Instructions:
For each statement below please circle the number that best corresponds to your response to the statement given. 
In this office employees understand the organisational structure and 
reporting lines of their unit/department, and how their job functions 
relates to overall departmental objectives and goals
Strongly Disagree
1 2 3 4 5 6 7 
8 9 10
Strongly 
Agree
For most meetings called for in this office, agendas are circulated to all before the meeting.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
For most meetings called for in this office, minutes are circulated to all soon after the meeting, indicating follow-up 
items.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
The leadership here keeps staff well informed about what is going on with the organisation.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office, cooperation and teamwork between staff in different units is encouraged.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office, we are encouraged to use data to guide decision-making, priority-setting, and planning.
38
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office, we are encouraged to analyse problems carefully to understand root causes before deciding on 
solutions.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office formal individual performance appraisals are routinely conducted on an annual basis.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office supervisors provide constructive feedback to their assistants on a regular basis, to help improve job 
performance.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
My contributions at work are acknowledged and appreciated.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
My supervisor works with me to identify my training needs and ensure I get the training or mentorship I need to do 
my job effectively.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
39
At this office, when staffs attend trainings, effort is made to ensure that they apply what they have learned back at 
the job site.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office supervisors delegate challenging assignments to assistants, which helps them to develop their skills 
and expertise.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office when giving special assignments, supervisors clearly communicates expectations at the beginning and 
checks in on progress, without ‘micromanaging.’ 
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office supervisors or unit leader regularly monitor progress and holds every staff accountable for following 
through on assigned tasks related to work plans.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office supervisors do everything in their power to help resolve HR issues (such as confirmation) in a timely 
manner.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In this office supervisors take appropriate corrective action when an employee is not performing well.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
40
In this office supervisors maintains a high standard of ethics and accountability.
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
In general do you feel your work environment is better today than it was 9 months ago? Before yourself or 
workmates underwent the ZMLA training
Strongly Disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
If any, please specify the positive changes you have observed in your work place in past 9 months associated to 
ZMLA training 
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________ 
What other improvement in your workplace climate do you wish to see that have not been implemented by 
management and work colleagues 
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
________________________________________________________________________________________
41
Annex 5: Paired Samples Test (Mean Differences for Leadership and 
Management Tool) 
Mean differences for leadership and management Self 
confidence 95% CI
Mean Std. 
Deviation
Std. Error 
Mean Lower Upper
P 
value
Ability to manage and use information strategically 
+1.450 1.664 0.132 
1.189 
 
1.710 0.000
Ability to manage finance 
+1.263 1.383 0.108 
1.049 
 
1.477 0.000
Ability to manage human resources 
+1.164 1.477 0.115 
0.936 
 
1.392 0.000
Ability to manage programs 
 
+1.563 1.753 0.139 
1.288 
 
1.837 0.000
Ability to problem solve 
+1.624 1.493 0.119 
1.389 
 
1.860 0.000
Ability to build teams and manage change 
+1.363 1.485 0.116 
1.134 
 
1.592 0.000
Mean differences for leadership and management Self-confidence (Detailed Variable List) Mean 
difference
Std. 
Deviation
Std. 
Error 
Mean
95% CI 
P 
Lower Upper value
Provide constructive feedback on a regular basis, in a way that helps those I supervise improve their 
performance
+1.236 1.632 0.163 0.621 1.721 0.00
.Use non-monetary strategies (such as praise, public recognition, and reminding staff of the 
importance/value of their work) to motivate those I supervise. 
+1.174 1.984 .159 0.859 1.489 .000
.Identify staff development needs and work with them to plan appropriate trainings, mentoring 
opportunities, or other ways of addressing the needs. 
+1.556 2.246 .187 1.186 1.926 .000
Chair productive and efficient meetings that begin and end as scheduled. +1.115 1.809 .149 0.821 1.409 .000
Identify the most important “root” or underlying causes of specific challenges within my 
unit/department that affect the healthcare system (such as high maternal mortality, or ART non￾adherence). -
+1.526 1.884 .162 1.205 1.847 .000
Prioritize among possible solutions/interventions to address healthcare challenges, to identify and 
implement those that will have the highest impact with fewer resources. 
+1.476 1.928 .159 1.162 1.790 .000
.Engage and maintain effective communication with internal and external stakeholders (including other 
departments or levels within the MOH, community members/organisations, NGOs, and other ministries) 
when planning and implementing strategies to address 
+1.411 1.856 .154 1.107 1.715 .000
Develop detailed work plans that specify timelines, milestones, and roles/responsibilities of specific 
people, to address specific challenges or achieve certain goals (such as increasing use of family planning 
services, improving access to EMCOR, e
+1.713 2.131 .174 1.370 2.057 .000
.Regularly check in on progress, and hold members of my organisation/unit accountable for following 
through on objectives and activities from our work plan or Action Plan
+1.669 1.980 .160 1.354 1.984 .000
Ensure that projects I oversee are carried out within the allotted budget, time and resources. +1.059 1.739 .141 0.781 1.338 .000
Promote teamwork and collaboration among staff and different units in my organisation. +0.968 1.559 .125 0.721 1.215 .000
Communicate well with staff, to ensure that they understand the overall picture (strategic vision) of our 
unit and1 are informed about any changes that may be introduced to our organisation
+1.1935 1.7732 .1424 0.9122 1.4749 .000
.Follow up to facilitate prompt resolution of staff HR issues, such as promotion and training. +1.318 2.008 .194 0.933 1.703 .000
.Ensure that staff reporting to me have up-to-date job descriptions, with which they are familiar, and 
understand reporting lines within our unit/department
+1.603 2.151 .188 1.231 1.975 .000
Take appropriate corrective action to address staff performance problems as soon as I am aware of 
them
+1.3676 1.7881 .1533 1.0644 1.6709 .000
.Identify and apply cost-effective approaches to maximize use of the organisation’s resources. +1.135 1.628 .131 0.877 1.394 .000
.Create realistic program or project budgets based on historical data, current cost information, and 
other relevant information sources. 
+1.190 1.939 .160 0.874 1.506 .000
42
.Effectively interpret and use the data available to me (such as from HMIS, performance assessment 
reports, and other sources) to guide planning, decision-making, and quality improvement.
+1.348 1.708 .137 1.077 1.619 .000
.Create reports, charts and graphs which succinctly and effectively communicate relevant data to 
stakeholders. 
+1.565 2.117 .175 1.220 1..910 .000
Analyse the delivery model for health services to identify the major gaps and bottlenecks affecting the 
quality of care and health outcomes. 
+1.875 1.950 .162 1.554 2.196 .000
Identify and implement necessary changes in delivery models for health services to address or correct 
gaps and bottlenecks affecting quality of care and health outcomes. 
+1.730 2.024 .173 1.388 2.072 .000
.Use lead times, inventories, and other supply logistics information to ensure that stock levels of critical 
supplies remain adequate at all times. 
+1.667 1.934 .170 1.330 2.004 .000
I feel I have been adequately trained to meet the management and leadership challenges I encounter in 
my position. 
+1.486 1.232 .101 1.286 1.687 .000
Job Motivation
I feel positive and up most of the time I am working. +0.595 1.016 .083 0.430 0.760 .000
I enjoy taking on new challenges that arise in my position. +0..401 .791 .065 0.272 0.530 .000
I am optimistic about the future success of my organisation. +0.177 .897 .074 0.031 0.323 .018
I am optimistic about my own future success with the organisation. Organisation. +0.306 .911 .075 0.158 0.455 .000
I feel more committed to a career with the organisation this year than I did a year ago. +0.581 .962 .079 0.425 0..737 .000
43
Annex 6: Percentage of Trainees Feeling Adequately Trained in 
Management & Leadership (Crosstab and Chi-square Test) 
Crosstab (Phase II)
Pre/Post￾Pre-test Post-test Total
I feel I have been adequately 
trained to meet the management 
and leadership challenges I 
encounter in my position. 
Disagree/Neutral Count 89 3 92
% within Pre/Post- 57.4% 1.9% 29.6%
Agree Count 66 153 219
% within Pre/Post- 42.6% 98.1% 70.4%
Total Count 155 156 311
% within Pre/Post- 100.0% 100.0% 100.0%
Chi-Square Tests
Value df Asymp. Sig. (2-sided) Exact Sig. (2-sided) Exact Sig. (1-sided)
Pearson Chi-Square 1.150E2a
1 .000
Continuity Correctionb
112.302 1 .000
Likelihood Ratio 136.629 1 .000
Fisher's Exact Test .000 .000
Linear-by-Linear Association 114.581 1 .000
N of Valid Casesb
311
a. 0 cells (.0%) have expected count less than 5. The minimum expected count is 45.85.
b. Computed only for a 2x2 table
Crosstab (Phase I)
Survey type
Pre-test Post-test Total
I feel I have been adequately 
trained to meet the management 
and leadership challenges I 
encounter in my position.
Disagree/Neutral Count 102 4 106
% within Survey type 37.5% 1.5% 19.4%
Agree Count 170 270 440
% within Survey type 62.5% 98.5% 80.6%
Total Count 272 274 546
% within Survey type 100.0% 100.0% 100.0%
Chi-Square Tests
Value df Asymp. Sig. (2-sided) Exact Sig. (2-sided) Exact Sig. (1-sided)
Pearson Chi-Square 1.133E2a
1 .000
Continuity Correctionb
111.033 1 .000
Likelihood Ratio 135.803 1 .000
Fisher's Exact Test .000 .000
Linear-by-Linear Association 113.118 1 .000
N of Valid Casesb
546
a. 0 cells (.0%) have expected count less than 5. The minimum expected count is 52.81.
b. Computed only for a 2x2 table
44
Annex 7: MLA Survey Themes 
Themes Code
Ability to manage and use information strategically 1
Ability to manage finance 2
Ability to manage human resources 3
Ability to manage programs 4
Ability to problem solve 5
Ability to build teams and manage change 6
Code Variable Variable Description
1
Pre_Qst18 Effectively interpret and use the data available to me (such as from HMIS, performance assessment reports, 
and other sources) to guide planning, decision-making, and quality improvement.
1 Pre_Qst19 Create reports, charts and graphs which succinctly and effectively communicate relevant data to stakeholders.
2 Pre_Qst10 Ensure that projects I oversee are carried out within the allotted budget, time and resources.
2 Pre_Qst16 Identify and apply cost-effective approaches to maximize use of the organisation’s resources.
2
Pre_Qst17 Create realistic program or project budgets based on historical data, current cost information, and other 
relevant information sources.
2
Pre_Qst22 Use lead times, inventories, and other supply logistics information to ensure that stock levels of critical 
supplies remain adequate at all times.
3
Pre_Qst1 Provide constructive feedback on a regular basis, in a way that helps those I supervise improve their 
performance.
3
Pre_Qst2 Use non-monetary strategies (such as praise, public recognition, and reminding staff of the importance/value 
of their work) to motivate those I supervise.
3
Pre_Qst3 Identify staff development needs and work with them to plan appropriate trainings, mentoring opportunities, 
or other ways of addressing the needs.
3 Pre_Qst11 Promote teamwork and collaboration among staff and different units in my organisation.
3 Pre_Qst13 Follow up to facilitate prompt resolution of staff HR issues, such as promotion and training.
3
Pre_Qst14 Ensure that staffs reporting to me have up-to-date job descriptions, with which they are familiar, and 
understand reporting lines within our unit/department.
3 Pre_Qst15 Take appropriate corrective action to address staff performance problems as soon as I am aware of them.
4
Pre_Qst7 Engage and maintain effective communication with internal and external stakeholders (including other 
departments or levels within the MOH, community members/organisations, NGOs, and other ministries) when 
planning and implementing strategies to address healthcare challenges.
4
Pre_Qst8 Develop detailed work plans that specify timelines, milestones, and roles/responsibilities of specific people, to 
address specific challenges or achieve certain goals (such as increasing use of family planning services, 
improving access to EMoC, etc.).
5
Pre_Qst5 Identify the most important “root” or underlying causes of specific challenges within my unit/department that 
affect the healthcare system (such as high maternal mortality, or ART non-adherence).
5
Pre_Qst6 Prioritize among possible solutions/interventions to address healthcare challenges, to identify and implement 
those that will have the highest impact Analyse resources.
5
Pre_Qst20 Analyse the delivery model for health services to identify the major gaps and bottlenecks affecting the quality 
of care and health outcomes.
5
Pre_Qst21 Identify and implement necessary changes in delivery models for health services to address or correct gaps and 
bottlenecks affecting quality of care and health outcomes.
6 Pre_Qst4 Chair productive and efficient meetings that begin and end as scheduled.
6
Pre_Qst9 Regularly check in on progress, and hold members of my organisation/unit accountable for following through 
on objectives and activities from our work plan or Action Plan.
6
Pre_Qst12 Communicate well with staff, to ensure that they understand the overall picture (strategic vision) of our unit 
and are informed about any changes that may be introduced to our organisation.
45
Annex 8: Phase II Work Climate Survey Mean Scores Before and After ZMLA 
Training 
Pre Training
Post Training
N Mean Std. 
Deviation +/- CI N Mean Std. 
Deviation +/- CI % 
Change
In this office employees understand the organisational 
structure and reporting lines of their unit/department, 
and how their job functions relates to overall 
departmental objectives and goals
73 7.56 2.40 0.56 58 8.02 2.17 0.57 6.03%
For most meetings called for in this office, agendas are 
circulated to all before the meeting. 73 7.18 2.89 0.68 58 7.45 2.51 0.66 3.76%
For most meetings called for in this office, minutes are 
circulated to all soon after the meeting, indicating 
follow-up items.
73 5.40 2.89 0.67 58 6.07 2.69 0.71 12.45%
The leadership here keeps staff well informed about 
what is going on with the organisation.
73 7.52 2.40 0.56 58 8.00 2.43 0.64 6.38%
In this office, cooperation and teamwork between 
staff in different units is encouraged. 73 8.26 2.23 0.52 58 8.14 2.25 0.59 -1.48%
In this office, we are encouraged to use data to guide 
decision-making, priority-setting, and planning. 73 7.58 2.72 0.64 58 8.31 2.31 0.61 9.70%
In this office, we are encouraged to analyse problems 
carefully to understand root causes before deciding on 
solutions.
73 7.48 2.46 0.57 58 8.10 2.24 0.59 8.34%
In this office formal individual performance appraisals 
are routinely conducted on an annual basis. 73 5.95 2.79 0.65 58 6.74 2.84 0.75 13.39%
In this office supervisors provide constructive 
feedback to their assistants on a regular basis, to help 
improve job performance.
73 6.75 2.56 0.60 58 7.36 2.31 0.61 9.01%
My contributions at work are acknowledged and 
appreciated. 73 7.67 2.44 0.57 58 7.88 2.30 0.61 2.71%
My supervisor works with me to identify my training 
needs and ensure I get the training or mentorship I 
need to do my job effectively.
73 6.97 2.91 0.68 58 7.14 2.68 0.71 2.37%
At this office, when staff attends trainings, effort is 
made to ensure that they apply what they have 
learned back at the job site.
73 7.16 2.48 0.58 58 7.59 2.22 0.58 5.89%
In this office supervisors delegate challenging 
assignments to assistants, which helps them to 
develop their skills and expertise.
73 7.29 2.57 0.60 58 7.69 2.02 0.53 5.52%
In this office when giving special assignments, 
supervisors clearly communicates expectations at the 
beginning and checks in on progress, without 
micromanaging. 
73 6.99 2.41 0.56 58 7.17 2.54 0.67 2.66%
In this office supervisors or unit leader regularly 
monitor progress and holds every staff accountable 
for following through on assigned tasks related to 
workplans.
73 7.30 2.45 0.57 58 7.12 2.46 0.65 -2.47%
In this office supervisors do everything in their power 
to help resolve HR issues (such as confirmation) in a 
timely manner.
73 6.30 3.12 0.73 58 7.41 2.59 0.68 17.65%
In this office supervisors take appropriate corrective 
action when an employee is not performing well. 73 7.11 2.58 0.60 58 7.67 2.12 0.56 7.92%
In this office supervisors maintains a high standard of 
ethics and accountability. 73 7.97 2.32 0.54 58 7.72 2.29 0.60 -3.12%
Overall 73 7.14 2.59 0.60 58 7.53 2.39 0.63 5.57%
46
Annex 9 : Phase II Work Climate Survey Independent Samples Test 
Independent Samples Test (Equal 
variances assumed)
Levene's Test for 
Equality of Variances t-test for Equality of Means 95% CI
F Sig. t df
P￾value
Mean 
Difference
Std. Error 
Difference Lower Upper
Data use, Problem analysis and solving 
skills 1.150 0.285 1.711 129 0.089 0.680 0.397 -0.106 1.465
Teamwork and Coordination 0.521 0.472 0.457 129 0.648 0.167 0.364 -0.554 0.887
Management of meetings 2.285 0.133 1.143 129 0.255 0.471 0.412 -0.344 1.286
Communication Levels 0.663 0.417 1.130 129 0.260 0.479 0.424 -0.360 1.319
Performance Management of HR 
(Appraisal and Recognition of 
contributions) 
0.196 0.659 1.286 129 0.201 0.502 0.390 -0.270 1.274
Supervision and feedback processes 0.040 0.842 1.020 129 0.310 0.397 0.390 -0.373 1.168
Development of HR (Delegation and 
Training to develop skills and enhance 
productivity)
3.246 0.074 0.868 129 0.387 0.330 0.380 -0.422 1.081
Resolution of HR Issues 0.168 0.683 1.609 129 0.110 0.796 0.495 -0.183 1.775
Ethics and Accountability 0.695 0.406 0.121 129 0.904 0.045 0.369 -0.685 0.775
Independent Samples Test (Equal 
variances assumed) detailed variable list
Levene's Test for 
Equality of 
Variances t-test for Equality of Means 95% CI
F Sig. t df P-value
Mean 
Differenc
e
Std. 
Error 
Differen
ce Lower Upper
In this office employees understand the 
organisational structure and reporting 
lines of their unit/department, and how 
their job functions relates to overall 
departmental objectives and goals
1.631 0.204 1.126 129 0.262 0.4556 0.40473 -0.34516 1.25636
For most meetings called for in this office, 
agendas are circulated to all before the 
meeting.
1.413 0.237 0.562 129 0.575 0.27019 0.4806 -0.68069 1.22108
For most meetings called for in this office, 
minutes are circulated to all soon after 
the meeting, indicating follow-up items.
0.484 0.488 1.364 129 0.175 0.67171 0.49248 -0.30267 1.64608
The leadership here keeps staff well 
informed about what is going on with the 
organisation.
0.663 0.417 1.13 129 0.26 0.47945 0.42419 -0.35982 1.31872
In this office, cooperation and teamwork 
between staff in different units is 
encouraged.
0.296 0.587 -0.311 129 0.757 -0.12234 0.39391 -0.90171 0.65703
In this office, we are encouraged to use 
data to guide decision-making, priority￾setting, and planning.
1.997 0.16 1.639 129 0.104 0.735 0.44839 -0.15214 1.62214
In this office, we are encouraged to 
analyse problems carefully to understand 
root causes before deciding on solutions.
0.995 0.32 1.5 129 0.136 0.624 0.41602 -0.19911 1.4471
In this office formal individual 
performance appraisals are routinely 
conducted on an annual basis.
0.168 0.683 1.609 129 0.11 0.79617 0.49483 -0.18286 1.77521
In this office supervisors provide 
constructive feedback to their assistants 
on a regular basis, to help improve job 
performance.
1.244 0.267 1.41 129 0.161 0.60864 0.43168 -0.24545 1.46274
My contributions at work are 
acknowledged and appreciated. 0.023 0.881 0.497 129 0.62 0.20808 0.41908 -0.62109 1.03724
My supervisor works with me to identify 
my training needs and ensure I get the 0.864 0.354 0.334 129 0.739 0.16533 0.49433 -0.81272 1.14338
47
training or mentorship I need to do my 
job effectively.
At this office, when staff attend trainings, 
effort is made to ensure that they apply 
what they have learned back at the job 
site.
0.701 0.404 1.011 129 0.314 0.42182 0.41731 -0.40384 1.24749
In this office supervisors delegate 
challenging assignments to assistants, 
which helps them to develop their skills 
and expertise.
3.913 0.05 0.976 129 0.331 0.40198 0.4119 -0.41297 1.21693
In this office when giving special 
assignments, supervisors clearly 
communicates expectations at the 
beginning and checks in on progress, 
without ‘micromanaging.’ 
0.336 0.563 0.429 129 0.669 0.18611 0.43398 -0.67253 1.04476
In this office supervisors or unit leader 
regularly monitor progress and holds 
every staff accountable for following 
through on assigned tasks related to 
workplans.
0.178 0.673 -0.419 129 0.676 -0.18068 0.43128 -1.03399 0.67263
In this office supervisors do everything in 
their power to help resolve HR issues 
(such as confirmation) in a timely manner.
4.949 0.028 2.182 129 0.031 1.11242 0.50976 0.10385 2.121
In this office supervisors take appropriate 
corrective action when an employee is 
not performing well.
1.895 0.171 1.34 129 0.183 0.56282 0.42012 -0.2684 1.39404
In this office supervisors maintains a high 
standard of ethics and accountability. 0.08 0.778 -0.612 129 0.542 -0.24846 0.40608 -1.0519 0.55497
48
Annex 10: Work Climate Survey Themes 
Themes Code
Data use, Problem analysis and solving skills 1
Teamwork and Coordination 2
Management of meetings 3
Communication Levels 4
Performance Management of 5
Supervision and feedback processes 6
Development of HR 7
Resolution of HR Issues 8
Ethics and Accountability 9
Code Variable Variable Description
1 Qst6 In this office, we are encouraged to use data to guide decision-making, priority-setting, and planning.
1 Qst7 In this office, we are encouraged to analyse problems carefully to understand root causes before deciding on solutions.
2 Qst1 In this office employees understand the organisational structure and reporting lines of their unit/department, and how their job 
functions relates to overall departmental objectives and goals
2 Qst5 In this office, cooperation and teamwork between staff in different units is encouraged.
3 Qst2 For most meetings called for in this office, agendas are circulated to all before the meeting.
3 Qst3 For most meetings called for in this office, minutes are circulated to all soon after the meeting, indicating follow-up items.
4 Qst4 The leadership here keeps staff well informed about what is going on with the organisation.
5 Qst8 In this office formal individual performance appraisals are routinely conducted on an annual basis.
5 Qst10 My contributions at work are acknowledged and appreciated.
6 Qst9 In this office supervisors provide constructive feedback to their assistants on a regular basis, to help improve job performance.
6 Qst14 In this office when giving special assignments, supervisors clearly communicates expectations at the beginning and checks in on 
progress, without ‘micromanaging.’
7 Qst11 My supervisor works with me to identify my training needs and ensure I get the training or mentorship I need to do my job 
effectively.
7 Qst12 At this office, when staffs attend trainings, effort is made to ensure that they apply what they have learned back at the job site.
7 Qst13 In this office supervisors delegate challenging assignments to assistants, which helps them to develop their skills and expertise.
8 Qst16 In this office supervisors do everything in their power to help resolve HR issues (such as confirmation) in a timely manner.
9 Qst15 In this office supervisors or unit leader regularly monitor progress and holds every staff accountable for following through on 
assigned tasks related to work plans.
9 Qst17 In this office supervisors take appropriate corrective action when an employee is not performing well.
9 Qst18 In this office supervisors maintain a high standard of ethics and accountability.
49
Annex 11: Study Information Sheet 
Title: Zambia Management and Leadership Academy (ZMLA) Performance Evaluation Study 
Introduction: In line with the Government of Zambia’s efforts to promote better health system management and in recognition of 
current skill gaps among current managers in the health sector workforce, Zambia Integrated Systems Strengthening Project (ZISSP) 
through the leadership of BroadReach Institute for Education and Training (BRITE) has been implementing the Zambia Management 
and Leadership Academy program since October 2011. 
With this background, it’s important to take a critical review of the program focusing on ZMLA participants and other Key MOH 
management staff and stakeholders. 
Study Justification and Purpose of Study: It is currently unknown whether the ZMLA approach has had any effect on participant’s 
management, leadership, motivation and general workplace climate in the public health sector. This evaluation study will help us 
understand whether there are any changes in the desired behavioural outcomes of the ZMLA approach. Furthermore, the 
evaluation exercise provides an opportunity to document implementation lessons for future program design and scale-up activities. 
Study Procedures: You are being asked to take part in this evaluation study because you have participated in the ZMLA program or 
you are a manager in the MOH/MCDMCH or a key stakeholder. If you agree to be in the study, you will be asked questions that 
relate to key objectives of the ZMLA program. 
Potential Risks and Discomforts: There are no physical risks involved in this study. Although some survey questions are about 
confidence that is ability to undertake Management & Leadership tasks and job motivation. Your views on these aspects will be kept 
confidential and will not be shared with either your supervisors or subordinates. You may refuse to answer any question or stop 
participating at any time. 
Benefits: There are no direct anticipated benefits to study participants. However, it is believed in the long run the results of the 
evaluation will feed back into program design and improvements for the future which could benefit the health system and the 
general public 
Confidentiality: Information collected on this survey will be reviewed by the study team and a summary will be shared with the 
Ministry of Health, including the District Health Office that oversees your health facility. 
Voluntary participation: Participation in the evaluation study is entirely voluntary. It is your choice whether or not to take part in the 
study. If you choose not to participate, this will not have any bearing on your job or any work-related evaluations or reports. You may 
change your mind and stop participating at any time after signing the consent form. 
This study has been reviewed and approved by the Ministry of Health/MCDMCH and University of Zambia Biomedical Research 
Ethics Committee whose task is to ensure that research participants are protected from harm. If you have any questions regarding 
this study please contact the following: 
Dr.J.C Munthali, Chairperson, University of Zambia Biomedical Research Ethics Committee, Ridgeway Campus, Lusaka, Telephone: 
260-1-256067, E-mail: unzarec@unza.zm or
Dr. Denis Mulenga, Deputy Director Technical Support Services, Ministry of Health, Ndeke House, Directorate of Technical Support 
Services, P.O. Box 30205 Lusaka, Zambia, Cell: 260-977-793830 
Dr Wilbroad Mutale, the lead consultant for this evaluation study Cell 0967780284 
Name of Researcher/person getting consent…………………………………………………..
Signature of Researcher/person getting consent………………………………………………
Date:……./………./……….. 
50
Annex 12: Informed Consent Form 
Statement of Consent 
I have read the study information sheet for “Zambia Management and Leadership Academy (ZMLA) Performance Evaluation Study,” 
or it has been read to me, and I understand the purpose of the study, procedures, and potential risks. I understand that my 
participation is voluntary, and I can withdraw from the study at any time. I have had the opportunity to ask questions and any 
questions I had, have been answered to my satisfaction. 
I give my consent to participate in this study.
Name of study participant (print): ________________________________________________________
Signature (or thumb print):________________________________ Date: ________________________
Name of witness (print): _____________________________________________________
Signature:___________________________________________ 
Date: __________________________
51
Annex 13: ZMLA Participant IDI Guide 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: 
Key Informant ID: Completed ZMLA :Yes /No
Introductory remarks
Thank you for agreeing to talk with us, and for your participation in this evaluation.
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us about your experience with the program, having gone through it, hence 
the interview. 
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
Experience with the ZMLA program
We would like to ask about your experience with the ZMLA program
How did you find the ZMLA program? 
After undergoing the MLA program do you feel adequately trained to meet your current and future 
management and leadership responsibilities? PROBE: What key positive qualities have you acquired from 
this training? If not, why not? 
Does your supervisor support your training? 
To what extent do you feel you have been encouraged by your supervisors to apply the knowledge that you 
gained. 
To what extent do your colleagues and workmates encourage you to apply the knowledge you have acquired 
through ZMLA? 
What would you say has changed in the way you were working before and after you attended the ZMLA 
courses? 
Components of the ZMLA 
What do you feel were the most important things you learned from the ZMLA course? PROBE for the 
following: 
Which skills, tools, or concepts have you used in your work? 
Which have been most helpful to you? 
Which skills or concepts were difficult to apply? What made them difficult? 
Were there any management or leadership topics or skills not addressed by ZMLA that you need training on? 
If so, what are they? 
52
The ZMLA Program was divided into three parts, the learning sessions, the case study and the mentorship 
sessions. 
Which of these three components did you find most useful and why? 
Which of these three components did you find least useful and why? 
What would you change about each of these sessions to make them better? (Probe for what changes would 
be made with reference to the learning sessions, case study and mentorship) 
Your future with your current employer 
I would like to get your views about your commitment to your career with your organisation 
(MOH/MCDMCH) now that you have completed the ZMLA program. 
Do you intend to stay or leave your current employers in the near future? 
Has your attending ZMLA affected your decision to stay or leave your organisation? 
Mentorship and follow-up
Did you ever meet with your team and your mentor at your place of work? How did this compare to the 
mentorship sessions that happened immediately following the workshops? Which format did you find to be 
most useful? Why? 
- Did you ever contact a mentor individually? Why or why not? 
- How helpful was the interaction with your mentor? 
ZMLA vs other management programs 
Have you ever attended any other management course? If yes, which one/s 
PROBE: Would you be kind to highlight the key differences with the ZMLA 
- Based on your experience, what are the strengths of the program compared to other parallel capacity 
building programs in the MoH/MCDMCH? 
Would you recommend the ZMLA program to a colleague? Why or why not? 
Would you allow someone you supervise to attend? Why or why not? 
Workplace and ZMLA 
In your opinion what specific organisational challenges did the ZMLA training address in your workplace? 
PROBE which specific processes did you establish to address organisational challenges? 
How has the workplace environment changed since ZMLA training was implemented? i.e. in what ways has 
the program affected staff as individuals, their teams, and the organisation as a whole? PROBE: For overall 
changes noticed on the way things run in the workplace or most significant changes or results e.g. systematic 
approach to meetings and follow up actions, delegation processes, financial management practices, 
implementation of APAS, strategic information management practices, quality of reports etc. 
Are there any deliberate efforts that were employed to share ZMLA skills with other staff members who did 
not participate in the training program? PROBE how this was done 
Service delivery and ZMLA 
Have you seen any changes arising as a result of the ZMLA program with regards to improvements in service 
delivery in the health sector where you operate? 
Sustainability - Do you think the ZMLA program should continue after the current funding ends? What 
mechanisms do you think should be put in place make sure the program is sustainable? 
53
Annex 14: Provincial Medical Officer and District Medical officer IDI 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: Highest qualification
Attended ZMLA Yes/No 
Key Informant ID: 
Introductory remarks
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being the PMO/DMO. 
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
Knowledge about the ZMLA program 
Have you heard about the ZMLA program? If Yes, (If no explain) 
Do you know anyone who has attended the ZMLA program in your Organisation? 
What are your thoughts on how the ZMLA program is structured? 
How adequately do you think the program prepares participants to handle their roles at work better within 
the health sector? 
What are the main changes, if any, that you have observed as a result of the program? What, in your view, is 
the value addition that the ZMLA program offers? 
What would you consider to be the main challenges with the ZMLA program as it is currently offered? What 
improvements would you propose? 
How adequately do you think ZMLA addresses current challenges in management and leadership within the 
health sector? 
Are there areas you feel should receive more attention? 
How much has ZMLA involved stakeholders in its design and implementation? (Probe – Government, CSOs, 
donors) 
How involved have other stakeholders been in ZMLA? 
What would you say about the coordination amongst stakeholders? 
Other than ZMLA course, what other management and leadership trainings are you aware of that help 
prepare health workers for roles in management and leadership? (PROBE: MOH/MCDMCH courses, Other 
courses e.g. MBA,) (IF YES ,GO TO 11, IF NO SKIP TO 12) 
54
Do you think that the ZMLA program is different from others being offered? Could you please give us more 
details on the differences you have noticed? 
Workplace and ZMLA 
In your opinion what specific organisational challenges did the ZMLA training address in your workplace? 
PROBE which specific processes did you establish to address organisational challenges? 
How has the workplace environment changed since ZMLA training was implemented? i.e. in what ways has 
the program affected staff as individuals, their teams, and the organisation as a whole? PROBE: For overall 
changes noticed on the way things run in the workplace or most significant changes or results e.g. systematic 
approach to meetings and follow up actions, delegation processes, financial management practices, 
implementation of APAS, strategic information management practices, quality of reports etc. 
Are there any deliberate efforts that have been employed to share ZMLA skills with other staff members 
who did not participate in the training program? PROBE how this is being done. 
 Service delivery and ZMLA 
Have you seen any changes arising as a result of the ZMLA program with regards to improvements in service 
delivery in the health sector where you operate? 
Sustainability 
Do you think the ZMLA program should continue after the current funding ends? How can it be made 
sustainable? What mechanisms do you think should be put in place? 
55
Annex 15: Key stakeholder’s IDI
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: Highest qualification
Attended ZMLA Yes/No 
Key Informant ID: 
Introductory remarks
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being a key stakeholder 
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
Have you heard about the ZMLA program? (If Yes, proceed with interview, if no explain about ZMLA and only 
proceed if respondent has some understanding about ZMLA) 
Have you or any of your colleagues attended the ZMLA program? (PROBE: How many) 
In your view has ZMLA helped those trained or not? (PROBE: How) 
Has your Organisation seen any benefit from the ZMLA program?(PROBE: which ways the Organisation has 
benefited) 
Is your Organisation working or collaborating with ZMLA program?(PROBE: how this is being done and the 
benefits of collaboration) 
Is your Organisation running any other leadership/Management course/s: (EXPLORE: Type and length, 
certification) 
If yes to 6, what would you say are the major differences between the course you run and ZMLA? 
Would your Organisation be willing to work with ZMLA to extend the program beyond the current funding? 
(PROBE: How e.g cost sharing, curriculum sharing and resource sharing) 
What challenges have you faced in working with ZMLA program? 
What improvements would you like to see in the ZMLA program? 
56
Annex 16: IDI Guide for Health facility managers hosting Case 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: Highest qualification
Attended ZMLA Yes/No 
Key Informant ID: 
Introductory remarks
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being a local health facility manager. 
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
Have you heard about the ZMLA program? (If NO EXPLAIN) 
Have you attended the ZMLA Course? 
This facility was used as a case study site for the ZMLA trainees. What problems were identified in this health 
facility? (Check with the list and PROBE for those not mentioned) 
For each problem identified, what activities were planned and what activities were actually implemented to 
improve performance? 
Have you noticed any improvements in the targeted areas? (for each intervention PROBE for changes in 
coverage and obtain any written evidence e.g graphs or figures) 
If there were no improvements in some targeted areas PROBE for reasons? If some things were not 
implemented what were the reasons? 
Did the trainee achieve their planned goals and coverage? 
What were the challenges of implementing case studies in this site? 
57
Annex 17: IDI Guide for Case study participant 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: 
Highest qualification
Attended ZMLA Yes/No 
Key Informant ID: 
Introductory remarks 
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being a ZMLA trainee. 
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
How far have you gone with the case study? 
What was the focus of the case study? 
Do you have a soft copy or hard copy of the case report? (Get a copy if available) 
What were your objectives? (LOOK at the report if available) 
Have you implemented your case study? (PROBE extent of implementation and limitations) 
What were your indicators and targets? 
Which indicators and targets were achieved? 
Can you attribute these improvements to the case study? (Why do you say so?) 
What were the main challenges and limitations in implementing your case study? 
What additional support would you have wished to get during the implementation of the case study? 
How did the case report consolidate the theory learnt in the ZMLA course? 
Did your mentors support you during your case study? (How?) 
58
Annex 18: IDI Guide for Program implementers/MOH/MCDMCH 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: 
Highest qualification
Key Informant ID: 
Introductory remarks 
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being a ZMLA implementer.
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
How long have you been involved in the implementation of ZMLA? 
What is your specific role? 
Have you participated in the teaching of the Course? 
How do you describe the overall participation in the course? (Diversity, gender and commitment) 
How do you describe the reception of the course by various stakeholders (Participants. MOH/MCDMCH, 
partners and other NGOs, training institutions) 
Are you happy with the way the course is structured and the contents? 
 What would you do differently if you were to redesign the course again? 
What have been the major changes or modification to the course from the way it was originally designed? 
What necessitated the changes? 
What have been the major successes? 
What have been the major challenges? 
Do you think the course is achieving its objectives? If not why if YES How? 
The course is divided into three parts, theory, mentorship and case study. Which component is better 
organized and which one is not (PROBE for reasons for good or bad performance) 
What suggestion would you give for the course to be strengthened? 
What suggestion do you have for the ZMLA course to be sustainable? 
59
Annex 19: IDI guide for NIPA representative 
Province: District: Organisation: 
Date: Sex: Position: 
Interviewer: Highest qualification
Key Informant ID: 
Introductory remarks
Thank you for agreeing to talk with us, and for your participation in this evaluation. 
The purpose of this evaluation is to assess the Zambia Management and Leadership Academy (ZMLA) 
Program, its successes as well as the challenges it is facing and how these can be overcome. We believe that 
you are in a good position to tell us your thoughts about the program, being a key partner in the 
implementation of ZMLA.
We anticipate the interview will last about an hour or less and appreciate any information you can provide. 
Your answers to the questions we will ask are completely confidential and the information you give us will be 
reported without names. Your participation is voluntary and you can refuse to answer any question or all the 
questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any 
benefit or consequence. We appreciate your assistance in this matter.
We would like to record this discussion for data analysis purposes.
Do you have any questions?
When did NIPA start working with the ZMLA program? 
Why did NIPA come on board? 
How have you found the collaboration with BRITE/ZISSP? (Benefits, areas of improvements) 
What is the role of NIPA on ZMLA? 
What is the difference of ZMLA compared to other management courses offered by NIPA? 
How do you rate the performance of those trained through the ZMLA Course? (PROBE: How many have 
graduated? 
What are the major causes of failure to graduate? 
 What are the challenges in supporting ZMLA students? (PROBE: Logistics, marking, man power, certification 
process etc) 
In your opinion, should the course be extended beyond the current funding or not? 
What is the additional benefit of ZMLA? 
How can the course be improved further? (PROBE: Structure, content, implementation, certification, etc.) 
60
Annex 20: List of Anonymized Key Informants 
S/N PROVINCE DISTRICT POSITION
1 LUSAKA CHONGWE MENTOR 
HOSPITAL ADMINISTRATOR
MEDICAL SUPRITENDENT
LUSAKA PRINCIPAL PLANNER
2 CENTRAL KABWE CLINICAL CARE SPECIALIST
MANAGEMENT SPECIALIST
KAPIRI CLINICAL CARE OFFICER
SKOWA COORDINATOR-STAKEHOLDER
CASE STUDY-MPUNDE PARTCIPANT
ENVIRONMENTAL HEALTH TECHNICIAN
3 EASTERN CHIPATA ACTING PROVINCIAL MEDICAL OFFICER
STAKEHOLDER
MANAGEMENT SPECIALIST
PROVINCIAL NURSING OFFICER
LUNDAZI DISTRICT MEDICAL OFFICER
ACCOUNTANT
NYIMBA ACTING DISTRICT MEDICAL OFFICER
PLANNER
LUANGWA DISTRICT MEDICAL OFFICER
IN-CHARGE KANSINSA(CASE STUDY SITE)
DISTRICT HEALTH INFORMATION OFFICER
CASE STUDY- PARTCIPANT
4 LUAPULA MANSA PROVINCIAL MEDICAL OFFICER
PRINCIPAL PLANNER
ACCOUNTANT
ACTING DISTRICT MEDICAL OFFICER
MANAGEMENT SPECIALIST
PROVINCIAL PLANNER(MOF)-STAKEHOLDER
NCHELENGE DISTRICT MEDICAL OFFICER
PUBLIC HEALTH OFFICER
5 NORTHERN KASAMA PROVINCIAL MEDICAL OFFICER
FINANCE & ADMIN OFFICER ZPCTII-STAKEHOLDER
ACTING NURSING OFFICER-HOSPITAL
SENIOR ACCOUNTANT-PHO
SENIOR NURSING OFFICER-HOSPITAL
MANAGEMENT SPECIALIST
MBALA DISTRICT MEDICAL OFFICER
FORMER IN-CHARGE CASE STUDY SITE
CASE STUDY PARTCIPANT
6 MUCHINGA MPIKA PROVINCILA MEDIC AL OFFICER
7 COPPERBELT NDOLA ACTING PROVINCIAL MEDICAL OFFICER
ASSITANT HUMAN RESOURCE OFFICER-PROVINCE
MANAGEMENT SPECIALIST
LUANSHYA ASSITANT HUMAN RESOURCE OFFICER-PROVINCE-DISTRICT
ASSISTANT HR-CASE STUDY
DISTRICT MEDICAL OFFICER
CASE STUDY PARTICIPANT
61
HEALTH FACLTY MANAGER
8 NORTH-WESTERN MANAGEMENT SPECIALIST
DMO
SENIOR HR-PHO
PMO
9 WESTERN MONGU MANAGEMENT SPECIALIST
PRINCIPAL DENTAL THERAPIST
PRINCIPAL NURSING OFFICER
FORMER PROVINCIAL MEDICAL OFFICER-MENTOR
KALABO PLANNER
DISTRICT MEDICAL OFFICER
HEALTH FACLTY MANAGER
10 SOUTHERN LIVINGSTONE PROVINCIAL HEALTH EDUCATION OFFICER
MANAGEMENT SPECIALIST
ACTING PROVINCIAL MEDICAL OFFICER
KALOMO HEALTH FACLTY MANAGER
IMPLEMENTORS HEAD QUARTERS DEPUTY DIRECTOR TECHNICAL SERVICES
NIPA COURSE COORDINATOR
COURSE DIRECTOR
ZISSP MANAGEMENT SPECIALIST TEAM LEADER
QUALITY IMPROVEMENT TEAM LEADER
BRITE COUNTRY DIRECTOR
SENIOR PROGRAM MANAGER
62
Annex 21: Nvivo Coding Framework for Qualitative Data
A. Management skills 
B. Specific skills or tools useful 
C. Topics not covered 
D. Changes noticed following ZMLA 
E. Supportive environment/Any resistance 
F. components useful 
G. Difficult tools or concepts 
 Workshop recommendations 
 Case study recommendations 
 Mentorship recommendations 
H. Recommendations general 
I. ZMLA Management and leadership roles 
 Differences with ZMLA 
 Recommend program to colleague 
 Attended any other management 
 Allow Supervise to attend 
J. APAS 
K. Motivation and commitment 
L. Services delivery and quality changes 
M. ZMLA and current employment 
N. Mentorship 
O. ZMLA and Other management programs 
P. Challenges 
Q. Workplace and ZMLA: Changes noted: 
 Communication 
 Strategic information 
 Quality reports 
 Finance management 
 Delegation 
 APAS implementation 
 Meeting approaches 
R. ZMLA sustainability 
 Willingness to pay for ZMLA 
 Suggestions: How can ZMLA made sustainable 
S. Case study: 
 How far with the case study 
 Knows about ZMLA 
 Attended 
 Activities 
 Objectives 
 Targets and indicators 
 What problems or focus 
 Improvements 
 Reason no improvement 
 Funding
 Planned goals achieved 
 Challenges 
T. Other nodes
63
Annex 22: Statement of Work 
SCOPE OF WORK 
BRITE seeks a suitably qualified consult or consulting team to perform an end of project impact evaluation 
of MLA in Zambia. MLA started in 2010 and will conclude all project implementation in July 2014. BRITE 
anticipates that the consultant will conduct the evaluation over a period of 60 days beginning June 16th
2014 at the latest. 
PURPOSE 
The evaluation findings will be shared with implementing partner organisations, the host government and 
other relevant national stakeholders by the last quarter of 2014 to maximize opportunities to leverage 
other donor resources to support scale up of proven health system strengthening interventions. Lessons 
learnt from the ZMLA districts will be communicated to the Zambian public health service to inform future 
leadership and management improvement efforts. The evaluation must have a sound methodological 
design, high quality documentation and presentation of findings by independent, experienced and 
objective evaluators. 
OBJECTIVES OF THE EVALUATION 
The final project impact evaluation of the Management and Leadership Academy (MLA), will seek to 
determine the extent of project performance, and provide critical information to guide future MLA project 
design and implementation. The evaluation is also intended for accountability and learning purposes, and 
will ascertain the extent to which the BroadReach Institute for Training and Education (BRITE) fulfilled its 
planned objectives after implementing MLA. More explicitly, the evaluation should: 
i. Describe the overall conduct of ZMLA program and identify lessons useful for future program scale￾up activities and implementation. 
ii. Measure the extent to which the ZMLA program has contributed to improved management and 
leadership skills among program participants 
iii. Evaluate organisational benefits that resulted from ZMLA training 
iv. Evaluate the extent to which ZMLA training contributed to improved service delivery. 
PLACE OF PERFORMANCE 
The selected consultant may provide remote technical expertise on the evaluation design, data 
analysis on report compilation. However, preparatory meetings, data collection and data cleaning 
will take pace in Zambia. 
WORK REQUIREMENTS 
As part of the documentation of the ZMLA Project, the consultant will be responsible for performing the 
following tasks over two months. Each phase requires sign off from BRITE before the consultant can 
continue to the next phase: 
Kickoff: 
- The consultant will manage meetings with respective stakeholders to plan for the evaluation and the 
relevant evaluation approach 
- Upon concluding the work scoping, create and present detailed evaluation plan with clear timelines. 
64
Research and Analysis Phase: 
- Manage a data collection team and ensure data quality for analysis. 
- Manage complex regression analysis to control for confounding factors. 
- Calculate measure of statistical inference
- Document analytical steps through STATA/SAS coding 
Reporting and Dissemination Phase: 
- Write, edit and publish the final evaluation report 
- Be available to disseminate information to partners and answer questions arising from the report. 
QUALIFICATION REQUIREMENTS 
The following skills and experience need to be represented in the consultant and/or in a consulting team. 
- Advanced research experience and must either have a PHD level qualification in Public Health or 
Management or Public Administration or must be currently enrolled for one at a reputable academic 
institution. 
- At least five years of experience leading research projects or evaluations. 
- Advanced skills in qualitative and quantitative analysis. 
- Over five years of experience in working in a developing country public health setting. 
- Must have experience writing technical documents and/or have published article in high impact 
international journals. 
EXPECTED OVERALL RESULTS AND OUTCOMES 
This section sets forth results (outcomes of the successful consultant’s performance) requirements, and 
performance standards (minimum standards that the Successful Consultant must meet) that must be met 
to BRITE’s satisfaction. The Final Evaluation Report shall be evidence-based and respond to the Key 
questions and evaluation areas outlined below. The key questions listed below are not exhaustive. 
Consultants are strongly encouraged to propose additional or alternate questions, but the study should at 
a minimum answer the following: 
EVALUATION QUESTIONS
Key evaluation question: What is MLA’s contribution to improving organisational oversight and 
functionality to improve health service delivery in ZISSP target districts?
Specific Impact Evaluation Questions: 
1. To what extent did MLA contribute to overall organisational management functionality? 
 What proportion of teams experienced changes in the organisational behaviours around: Meetings, 
planning, supervision, information management, time management, discipline and financial 
oversight? 
 What proportion of recruited organisational teams had improved action plans and budgets? 
2. To what extent did MLA contribute to overall improvement in service delivery: 
65
 What proportion of health teams showed improved timeliness of program implementation? 
 What proportion of the teams that implemented case studies led to improved service delivery 
according to action plans? 
Based on the broad evaluation question posed above, a number of questions have already been proposed by 
the monitoring and evaluation team based in Zambia. Table 1 below links the proposed research or 
evaluation objectives to a list (but not exhaustive) of questions that team has already started developing 
according to the MLA evaluation plan that should be reviewed and refined by the consultant. 
Evaluation Objective Evaluation Questions
i. To describe the overall 
conduct of ZMLA program and 
identify lessons useful for future 
program scale-up activities and 
implementation.
 Is the format, content, and delivery of training, mentorship, and case 
studies relevant to the trainees? 
 How does the programme compare (in terms of content, approach, etc) 
to other Management and Leadership training programmes? 
 What aspects of ZMLA were most valued by participants? 
 What aspects did not have the desired outcome, were not useful, or 
need to be otherwise strengthened? 
 How was the ZMLA program implemented in each phase, and was it 
implemented according to plan? (If not, what was changed?) 
ii. To measure the extent to 
which the ZMLA program has 
contributed to improved 
management and leadership skills 
among program participants
 How has the programme affected participants’ management and 
leadership knowledge and skills? 
 How has the programme affected participants’ management and 
leadership confidence and job motivation? 
 How did participants’ management practices change since ZMLA?
iii. To evaluate organisational 
benefits that resulted from ZMLA 
training 
 How has the workplace environment changed since ZMLA training was 
implemented? i.e. in what ways has the programme affected staff as 
individuals, their teams, and the organisation as whole? 
 What organisational challenges did ZMLA training address?
 What specific processes have individual participants and workplaces 
established to address persistent organisational challenges after 
undergoing the ZMLA training?
 Is there noticeable improvement in the development and 
implementation of action plans, to address both organisational and 
service delivery challenges? 
 To what extent was content from the formal training shared with other 
staff members who did not directly participate in the programme?
iv. iv. To evaluate the extent to 
which ZMLA training contributed to 
improved service delivery. 
 To what extent have service delivery indicators at case study project 
sites improved (percentage)
EVALUATION DESIGN AND METHODS 
The evaluation design and methods to be used must be explained in detail and be appropriate and of 
sufficient rigor to produce valid results. 
The successful Consultant’s evaluation team is expected to use available data from the baseline and 
subsequent annual organisational performance assessments, action plans and financial reports to compare 
the level of managerial practices and service delivery before and after the initiation of MLA. Routine health 
facility data reported through the national Health Management Information System (HMIS) presents an 
66
opportunity to compare health outcomes before and after the implementation of project interventions in the 
various regions of ZMLA implementation. 
The successful Consultant’s evaluation team is expected to employ a variety of quantitative and qualitative 
data collection methods to gather the evidence required to adequately answer the evaluation questions. 
The successful Consultant’s evaluation team will also be expected to conduct a quantitative survey and focus 
group discussions with health workers working in MLA supported district health facilities to gain an 
understanding of their experiences with project interventions. 
Limitations to the evaluation, with particular attention to the limitations associated with the evaluation 
methodology (selection bias, recall bias, observer bias, Hawthorne effect, etc.), must be disclosed in the 
evaluation report. 
The successful Consultant’s evaluation team will be expected to:
 Employ research techniques that ensure internal validity of study results (such as random sampling 
and triangulation). 
 Use statistical analysis to counter the effect of confounding factors where changes in service delivery 
are noted, such as regression analysis. 
 Utilize social science methods and tools that reduce the need for evaluator-specific judgments. 
 Identify and collect data on variables corresponding to inputs, outputs, outcomes and impacts. 
 Employ standardized recording, maintenance of records from the evaluation and perform data 
analysis with advanced statistical software to calculate statistical degrees of inference (e.g. focus 
group transcripts - NVIVO, data analysis and p-values - STATA). 
 Produce evaluation findings that are based on facts, evidence and data. This precludes relying 
exclusively upon anecdotes, hearsay and unverified opinions. Findings should be specific, concise 
and supported by quantitative and/or qualitative information that is reliable, valid and generalizable 
(with degrees of inference). 
EXPECTED TEAM COMPOSITION AND INDEPENDENCE 
The successful Consultant, co- managed by BRITE Zambia, will take the lead on the evaluation and 
submit a work plan within two weeks of accepting the offer. The successful Consultant’s evaluation 
team should be comprised of two or three key personnel who possess among them management 
expertise, monitoring and evaluation expertise, public health sector expertise and expertise in advanced 
statistical analysis. The successful Consultant’s evaluation team must demonstrate familiarity with the 
strengthening management systems to improve health care delivery and the associated measurement 
process. At least one of the three key personnel should be a senior level individual with strong 
knowledge and experience of working in the African public health sector. The Successful Consultant 
must clearly define roles and responsibilities of different team members highlighting the overall team 
lead, evaluation methodology expert, management expert, etc. 
The Consultant is expected to work closely with BRITE and MOH in setting up a team that will 
efficiently and objectively measure the quality of care levels in MLA supported districts in Zambia. This 
exercise will draw upon past experiences in assessing quality of care and use existing resources to 
whatever extent possible, while maintaining a high level of objectivity and independence. 
67
REPORTS AND DELIVERABLES 
The Successful Consultant shall submit reports, deliverables or outputs subject to the deadlines as laid out by 
the consultant in their own work plan. The Successful Consultant will also be responsible for submitting the 
following: 
1. Signed statements attesting to a lack of conflict of interest or describing an existing or potential conflict 
of interest relative to the project being evaluated by each evaluation team member. 
2. Evaluation Design: The consultant will review and improve upon the BRITE impact evaluation to associate 
the ZMLA activities to selected health service delivery indicators in each of the ZMLA target districts. The 
research design will build upon the draft evaluation plan that has been developed by BRITE and ABT 
Associates which will include a rigorous regression analysis to counter well understood confounding 
factors such as financing levels, local leadership changes and presence of other partners. The consultant 
will be required to review the data collection methodology and sampling procedures suggested by BRITE. 
3. Development of Research tools: Based on the Evaluation objectives and proposed broad evaluation 
questions, the consultant will develop relevant interview guides or questionnaires to measure both ZMLA 
intended behavioural and health outcomes/impact. 
4. Report Outline/Data Analysis Plan: based on evaluation questions and broad evaluation questions the 
consultant will from the onset provide an analysis plan and share STATA/SAS outputs and code with the 
BRITE monitoring and evaluation team for records and publishing purposes. 
5. Training of data collectors and supervision of field data collection: with support from BRITE team the 
lead consultant will lead the process of training data collectors and assistant consultant in all data 
collection and methodological processes required for a successful implementation of the evaluation 
study. Furthermore, the consultant will lead all field data collection supervision activities to ensure 
maximum quality assurance to methodological adherence. 
6. Electronic Data sets and Detailed KIs interview transcripts: All data and records from the evaluation 
must be submitted to BRITE in an easily readable and organized electronic format. 
7. Final Evaluation Report: The final report will be provided to the BRITE/Zambia in electronic form within 
15 business days following receipt of comments from BRITE. The evaluation report must address all 
evaluation questions included in the scope of work. It must represent a thoughtful, well-researched and 
well organized effort to address the evaluation purpose. 
The final report will be provided to the BRITE/Zambia in electronic form within 15 business days 
following receipt of comments from BRITE. The evaluation report must address all evaluation 
questions included in the scope of work. It must represent a thoughtful, well-researched and well 
organized effort to address the evaluation purpose. Readers must have sufficient information about 
the body of evidence and how information was gathered to make a judgment as to its reliability, 
validity and generalizability. The final report should not exceed 30 pages (excluding appendices) and 
must include the following sections: 
 An executive summary: 3-5 pages that summarizes the key points (project purpose and background, 
key evaluation questions, methods, findings, and recommendations)
 Background information on the project 
 Purpose of evaluation 
 Evaluation team: must be described with particular reference to the existence or lack thereof real or 
potential conflicts of interest relative to the project being evaluated
68
 Evaluation methods: must be explained in detail and limitations associated with the evaluation 
methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) 
must be disclosed in the report
 Evaluation findings: must be presented as analysed facts, evidence and data and not based on 
anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and 
supported by strong quantitative or qualitative evidence. When applicable, include statements 
regarding any significant unresolved differences of opinion on the part of funders, implementers 
and/or members of the evaluation team.
 Recommendations: need to be supported by a specific set of findings and must be action-oriented, 
practical and specific, with defined responsibility for the action 
 The final scope of work, evaluation tools and sources of information must be properly identified and 
listed in annex 
All data and records from the evaluation must be submitted to BRITE in an easily readable and 
organized electronic format along with the final report. 
OVERARCHING ELEMENTS AND IMPLEMENTATION MODALITIES 
Building Local Capacity: The Consultant shall, to the maximum extent possible, use Zambian staff, technical 
experts or consultants, and institutions in carrying out the evaluation of the MLA project under the resulting 
Task Order. 
Geographical Coverage: The project has a critical focus on improving the quality health care managed by 18 
district health team hospitals and Rural Health Centres (RHC) through the development of the case studies 
during training and mentorship. 
Gender Considerations: Equity should be addressed with a focus on gender. The evaluation should provide 
more details on the effect and results of the project interventions on men, women, girls and boys. 
Audience: The primary audiences for the evaluation report shall be the MERCK Foundation, USAID and the 
Government of Zambia, Development Partners, Implementing Partners and potential donors. 
Any discrepancies involving completion of tasks or disagreement between BRITE and the chosen vendor will 
be referred to both parties for review and discussion. 
ACCEPTANCE 
Approved by: 
_____________________________________________ Date: _____________________ 
<Approver’s Name>
<Approver’s Title>