EVALUATION OF THE LEADERSHIP DEVELOPMENT PROGRAM FOR THE MINISTRY OF HEALTH, Nicaragua (2001 – 2003) Nancy Vollmer LeMay M&L Monitoring and Evaluation Unit Nubia Herrera Socorro Talavera Nicaraguan Consultants Kate Waldman M&L Programs Unit June 2004 Management and Leadership Program Management Sciences for Health Boston, MA 02130 Telephone: (617) 524 7766 www.msh.org/mandl This report was made possible through support provided by the US Agency for International Development, Office of Population and Reproductive Health, under the terms of Cooperative Agreement Number HRN-A-00-00-00014-00. The opinions expressed herein are those of the author(s) and do not necessarily reflect the views of the US Agency for International Development. EVALUATION OF THE LEADERSHIP DEVELOPMENT PROGRAM FOR THE MINISTRY OF HEALTH, NICARAGUA (2001 – 2003) June 2004 Nancy Vollmer LeMay M&L Monitoring and Evaluation Unit Nubia Herrera Socorro Talavera Nicaraguan Consultants Kate Waldman M&L Programs Unit Management and Leadership Program (M&L) Cooperative Agreement No. HRN-A-00-00-00014-00 Management Sciences for Health (MSH) 891 Centre Street Boston, MA 02130 ACKNOWLEDGEMENTS The evaluation of the Nicaraguan leadership development program could not have taken place without the technical and logistical support of the MSH/Nicaragua team and a number of M&L staff members. We are thankful for the high-quality work performed by the MSH/Nicaragua staff that cooperated in the evaluation process and the skilled consultants who carried out the qualitative data collection. Our special thanks go to Barry Smith, Claritza Morales, Alba Luz Solorzano, Carla Martinez, Eduardo de Trinidad, Manuel Rodriguez, Olga Montalvan, Sarah Johnson, Alison Ellis, Tim Allen and Cary Perry, for their helpfulness and support. We also thank the Central level participants, SILAIS management teams, the Municipal management teams and the numerous municipal participants of the SILAIS of Boaco, Matagalpa, Jinotega, Masaya, Rivas, Madriz and Estelí for their willingness to participate in this evaluation and provide valuable contextual and narrative information. Finally, we are grateful for the support of the Nicaraguan MOH and for the enthusiasm and vision of the champion of the leadership development program, Violeta Barreto. i TABLE OF CONTENTS EXECUTIVE SUMMARY ....................................................................................................................................... iii 1. Background ........................................................................................................................................................1 1.1. Context........................................................................................................................................................1 1.2. Program Overview .....................................................................................................................................2 2. Evaluation Objectives and Methodology .........................................................................................................3 2.1. Objectives ...................................................................................................................................................3 2.2. Methodology...............................................................................................................................................3 3. Findings...............................................................................................................................................................7 3.1. The municipal level leadership development process.................................................................................7 3.1.1. Program description...............................................................................................................................7 3.1.2. Delivery and replication of the program................................................................................................9 3.1.3. Municipal action plans.........................................................................................................................11 3.1.4. Perceived improvements in leadership capacity ..................................................................................14 3.2. Leadership development at the central level ............................................................................................16 3.2.1. The approach and process....................................................................................................................16 3.2.2. The central level action plan................................................................................................................17 3.2.3. Perceived improvements in leadership capacity ..................................................................................18 3.3. Leadership development at the SILAIS level ............................................................................................19 3.3.1. The approach and process....................................................................................................................19 3.3.2. SILAIS Action plans............................................................................................................................19 3.3.3. Perceived improvements in leadership capacity ..................................................................................20 3.4. Organizational climate.............................................................................................................................21 3.4.1. Description of the PAHO organizational climate instrument ..............................................................21 3.4.2. Application of the PAHO climate instrument......................................................................................22 3.4.3. Overall change in municipal climate scores by SILAIS and program phase.......................................23 3.4.4. Climate results by municipality and program phase ............................................................................26 3.4.5. Comparison of climate dimensions and sub-dimensions at the municipal level..................................30 3.4.6. Climate results by SILAIS...................................................................................................................34 3.5. Leadership in the context of health services.............................................................................................35 3.5.1. Perceived effect of the program on services ........................................................................................35 3.5.2. Observations on service data ...............................................................................................................36 3.6. Sustainability of leadership processes and capacities..............................................................................37 3.6.1. Continuity of the leadership program ..................................................................................................37 3.6.2. Maintenance of climate levels over time .............................................................................................39 3.7. Observations on the internal reliability of the PAHO climate instrument................................................40 4. Conclusions.......................................................................................................................................................42 5. Recommendations ............................................................................................................................................45 Appendix 1: Scope of Work for the Evaluation............................................................................................47 Appendix 2: Interview Guides......................................................................................................................56 Guía de Entrevista para el Grupo Focal .............................................................................................................56 Guía de Entrevista Individual para Participantes ...............................................................................................58 Guía de Entrevista Individual para el Equipo de Dirección del Municipio........................................................60 ii Guía de Entrevista Individual para el Equipo de Dirección del SILAIS............................................................62 Guía de Entrevista Individual para el Nivel Central MINSA.............................................................................64 Appendix 3: Key Questions for the Evaluation............................................................................................66 Appendix 4: PAHO Climate Instrument.......................................................................................................68 Appendix 5: PAHO Climate Definitions ......................................................................................................72 Appendix 6: Municipal Level Climate Data.................................................................................................75 Appendix 7: Cost Analysis of Leadership Program .....................................................................................78 iii EXECUTIVE SUMMARY The Management and Leadership (M&L) Program of Management Sciences for Health (MSH) and the USAID-funded PROSALUD bilateral project implemented by MSH, together with the Nicaraguan Ministry of Health (MOH), launched a leadership development program in 2001 to strengthen the capacity of health managers and personnel at the municipal and SILAIS levels. The program was implemented in phases over a period of three years and was scaled up to involve 63 municipalities, 7 SILAIS and the central level of the MOH. A total of 1,978 managers and staff were trained in the leadership development program over the three years. The objectives of the leadership development program included: − Improve organizational climate in participating municipalities and SILAIS offices − Develop leadership capacities among the municipal and SILAIS management teams and key central level managers − Develop and publish the MOH Leadership Module consisting of self-instructional units intended for implementation at the municipal level − Advance the institutionalization of the leadership development program within the MOH An evaluation of the program was conducted by the M&L Program in the spring of 2004 using both quantitative and qualitative methodologies. The objectives of the evaluation were to: 1. Document the content and approach used by the leadership development program in Nicaragua as well as the intended and unintended results of each phase of the program. 2. Assess the relationship between program inputs, implementation of improvement plans, organizational climate results, and the performance of health services. Identify elements of the program content, delivery, and follow-up support associated with the results achieved. 3. Assess the extent to which leadership practices and processes, organizational climate levels and health service performance have been sustained in the post-intervention period for the first and second program phases. 4. Document the scale-up of the program, its costs, and the degree of institutionalization. 5. Analyze the internal reliability of the Pan American Health Organization (PAHO) organizational climate tool used by the Nicaragua leadership development program. Results of the evaluation indicate the following: The leadership development program achieved its primary outcome: improved organizational climate at the municipal and SILAIS levels. Participating municipalities prioritized the weakest of the climate sub-dimensions in their action plans and succeeded in improving these areas over others that needed perhaps less attention. An analysis of municipal climate scores suggests that while the impact of the leadership training and follow-up activities on organizational climate at the municipal level was minimal in the first phase, the program had a greater and measurable effect in the second and third phases. This is a logical outcome given that the first phase served mainly as a pilot to develop and perfect the program materials and iv process which were then successfully carried out in the second and third phases. Repeated applications of the climate tool are needed to determine whether or not changes in municipal climate are sustainable beyond the period of the leadership development intervention. The municipal leadership development program also resulted in notable changes in behavior and certain leadership competencies, including improved communication between supervisors and subordinates across both levels. While the program succeeded in achieving broad coverage, it was difficult to provide sufficient follow-up and monitoring of the replication activities in the municipalities and the efforts to implement the action plans, particularly after the training of facilitators had ended. At the central level, the program was still ongoing at the time of the evaluation. Nevertheless, results indicate that central level participants had already made important strides in addressing the challenge of aligning the National Health Plan with the Health Care Model. The program has also resulted in improved working relationships and coordination between divisions/departments and new lines of communication within programs/departments. In terms of the cost of the leadership program and its scale-up over a period of three years (including municipal, SILAIS and central levels), an analysis of the program’s financial data based on M&L costs revealed the following: • The number of participants increased by 51% from Phase 1 to Phase 2. From Phase 2 to Phase 3, the number of participants increased by 344%. • The total cost of the first phase of the program was $70,606. This was slightly higher than the cost of the second phase, which came to $54,023. The third phase, where the most expansion occurred, had a total cost of $339,506. • The cost per participant decreased by 51% from Phase 1 to Phase 2, but increased by 41% from Phase 2 to Phase 3. • In the Phase 3, Central level training and facilitator training at the SILAIS and municipal levels were the most costly, accounting for 35% and 40% respectively of the total expenses. Participant training at the municipal level accounted for a much smaller percentage (11%) of the total due to the fact that MSH facilitators were not used. • The average cost per participant over the three years of the program was $241. This includes level of effort (LOE), travel/per diem, training costs, and monitoring and mentoring costs. Also included are appropriate health/sick/vacation (HSV), overhead, and Allocable Cost Factor (ACF) rates. It is expected that as the program continues to spread and individual SILAIS or municipalities start to take on some of the program costs, the costs incurred by MSH will decrease. In spite of this, there will be a minimum cost associated with the maintenance, monitoring and evaluation of the program. This minimum cost can more accurately be determined as the program continues, but will likely remain under $250 per participant if the model set forth in the third phase is followed. Important factors that will contribute to the continuation of the program and its progress towards institutionalization include: v • Broad acceptance and ownership of the program by the MOH • Leadership modules, complete with facilitator and participant materials, have been published and adopted by the MOH • The climate assessment instrument has been accepted by the MOH and incorporated as an indicator in the Fully Functional Service Delivery Point (FFSDP) monitoring process • There is a broad base of facilitators at all levels who are capable of replicating the training • The training approach is low cost and in-service (self-learning) allowing for widespread and rapid scale up • There have been successful results to date in terms of improving organizational climate at the municipal level • The leadership development program responds to a felt need on the part of the MOH Two objectives in the evaluation scope of work were not completed fully. Potential relationships between municipal climate outcomes and service statistics were not included in this report for several reasons. First, the leadership development program was designed to improve organizational climate (as the outcome measure) and did not intend to affect health services in any direct way. Without a logical or programmatic link between the leadership program, climate levels and health services, no associations could be made. Second, the available data (service statistics) on health services were insufficient to perform an analysis of relationships between the program and health service outcomes. Instead, this report proposes designing a prospective study in the future, preferably an operations research design, in order to address this question. In this context, the service delivery indicators could be chosen and tracked from the beginning of the program in coordination with participating municipalities. This would serve two purposes: a) to analyze the root causes of poor services and then design the program interventions to address these, and b) to respond to municipal priorities for improving health services rather than measuring a standard set of indicators. The scope of work for this evaluation also included testing the internal reliability of the PAHO organizational climate tool used to collect outcome data for the program because while the tool was field tested, validity information on the instrument is not available. Initial factor analyses were inconclusive due to a particular nesting of the data that could not handled by the statistical software available to the M&L Programs Monitoring and Evaluation Unit. Further testing of the tool’s internal reliability will be performed later in 2004 with the assistance of a statistical consultant and a new statistical package. Meanwhile, a number of observations on the face validity of the instrument were obtained through interviews and focus groups with facilitators and municipal level participants. Results from these interviews show that the tool appears to lack full face validity. The majority of municipal participants, including those from the third phase, expressed difficulty interpreting various items in the tool and felt that some did not fit their reality at the local level. These difficulties call into question the reliability of the climate data collected by this program, at least during the first phase. However, it is very likely that results from the second and third application of the tool are closer to the true measure of climate because by this time the MSH/Nicaragua and MOH facilitators were more familiar with the instrument and better equipped to explain its use and clarify the meaning of the items for respondents. In addition, throughout the program, the MSH/Nicaragua team made several vi modifications to improve the tool particularly regarding the phrasing of certain items and calculation errors in the Excel data entry sheet. A number of recommendations are made for the municipal, SILAIS and central levels regarding the continuity of the leadership development program, including recommendations to address weaker aspects of the program such as ongoing monitoring and follow-up support. Nicaragua Leadership Development Program Evaluation 1 1. Background 1.1. Context For the past several years, the Ministry of Health (MOH) of Nicaragua has been in a process of modernization and decentralization of the health sector. This has required the development of leadership and management capacity at all levels of the organization in order to address the many challenges brought on by external and internal pressures. These include economic crises, political change, growing demands and needs of clients as well as bureaucratic lethargy and increasing apathy among health personnel. As one response to this need, the Management and Leadership (M&L) Program of Management Sciences for Health (MSH) and the PROSALUD1 project, together with the MOH, launched a leadership development program in 2001 to strengthen the capacity of health managers and personnel at the municipal and SILAIS2 levels (see section 1.2 for an overview of the leadership development program). PROSALUD designed the leadership development program to complement the strategy of the Fully Functional Service Delivery Point (FFSDP) which included a set of criteria for monitoring and evaluating the participating health units. The municipal leadership development program was developed to operationalize the FFSDP criteria for leadership. The leadership development program was implemented in phases over a period of three years and was scaled up to involve 63 municipalities, 7 SILAIS and the central level of the MOH. An evaluation of the first phase of the leadership development program was conducted by the M&L Program in the fall of 2002 to assess the organizational climate outcomes prior to the beginning of the second phase. Following completion of the second phase, a systematization of results from both the first and second phases was conducted in May-June 2003. While this current study is part of the series of in-depth evaluations on the topic of Developing Managers Who Lead, it serves as a final evaluation of the third phase of the Nicaragua leadership development program. The purpose of this in-depth evaluation is to document the approach of the Nicaragua leadership development program and assess the outcomes associated with organizational climate, service delivery improvements, and sustainability of leadership capacity following the completion of three years of leadership development with the Ministry of Health (MOH). This evaluation responds to a common set of key questions intended to provide substantive learning for the M&L Program and the leadership development program in Nicaragua and to contribute to a larger synthesis of M&L leadership evaluations. While the work that MSH and M&L have conducted in Nicaragua has included both management and leadership interventions, this study focuses solely on the leadership development program. 1 PROSALUD was the MSH bilateral in Nicaragua funded by USAID from April 1999 to June 2003. Through this bilateral, MSH developed Fully Functional Service Delivery Points (FFSDP) in 55 health units in three departments. 2 The term SILAIS refers to the administrative division at the departmental level of the MOH. Nicaragua Leadership Development Program Evaluation 2 1.2. Program Overview The Nicaragua leadership development program was implemented in three phases over a period of three years 3 (see Table 1 below). Following an initial leadership dialogue conducted by the M&L Program with the MOH and USAID Mission officials in February 2001, the first phase of the program was carried out in 13 municipalities in the SILAIS of Matagalpa, Jinotega and Boaco from July 2001 to August 2002. Twelve of these were prioritized municipalities where PROSALUD had been implementing the FFSDP program since 1999. The second phase reached the remaining 16 municipalities of the same three SILAIS from November 2002 to June 2003. The third phase was conducted from July 2003 to February 2004 in 34 municipalities of four additional SILAIS (Masaya, Madriz, Estelí and Rivas) plus the management teams of all seven SILAIS, and 56 key managers from central level MOH directorates and departments. The third phase of the program was conducted as part of the larger multi-component Leadership and Management for Health Project implemented by the M&L Program with field support funding from the USAID Mission. This program began in April 2003 and builds on the several years of service delivery strengthening interventions in Nicaragua conducted under the PROSALUD bilateral project. Table 1. The Leadership Development Program: Participants and Phases Number of municipalities per phase SILAIS First Phase 2001-2002 Second Phase 2002-2003 Third Phase 2003-2004 Total Matagalpa 6 9 - 15 Jinotega 5 3 - 8 Boaco 2 4 - 6 Masaya - - 9 9 Madriz - - 9 9 Estelí - - 6 6 Rivas - - 10 10 Total 13 16 34 63 The leadership development program was initially intended to strengthen the skills of health managers at the municipal level to select and address organizational challenges. In its third phase, the program was expanded to include leadership development for SILAIS and Central level managers. The basic methodology of the program, consistent throughout all phases of implementation, is based on the leadership development model which includes three key aspects: identification and resolution of a real organizational challenge which is facilitated through feedback and support. 3 In January 2004, the project expanded to include a fourth phased of leadership development for the Managua SILAIS, the Ministry of the Family and the National Social Security Institute, but due to the timing of the evaluation, this component is not included. Nicaragua Leadership Development Program Evaluation 3 The objectives of the leadership development program evolved over the course of the three year program and in the final year included the following: ¾ Improve organizational climate in participating municipalities and SILAIS offices ¾ Develop leadership capacities among the municipal and SILAIS management teams and key central level managers ¾ Develop and publish the MOH Leadership Module consisting of self-instructional units intended for implementation at the municipal level ¾ Advance the institutionalization of the leadership development program within the MOH 2. Evaluation Objectives and Methodology 2.1. Objectives The objectives of the current evaluation were to: • Document the content and approach used by the leadership development program in Nicaragua as well as the intended and unintended results achieved by each phase of the program. • Assess the relationship between program inputs, implementation of improvement plans, organizational climate results and the performance of health services (in terms of coverage and use of services). Identify elements of the program content, delivery and follow-up support associated with the results achieved. • Assess the extent to which leadership practices and processes, organizational climate levels and health service performance have been sustained in the post-intervention period for the first and second program phases. • Document the scale-up of the program, its costs, and the degree of institutionalization achieved. • Analyze the internal reliability of the Pan American Health Organization (PAHO) organizational climate tool used by the Nicaragua leadership development program. The full scope of work for the evaluation is provided in Appendix 1. 2.2. Methodology This study was based on both quantitative and qualitative methodologies with information from the following sources: 1. Review of project documents including learning modules, training plans, monitoring reports, evaluation and synthesis reports from the first and second phases of the leadership development program, and action plans developed by the municipal, SILAIS and Central level participants (available for the second and third phases of the program). Nicaragua Leadership Development Program Evaluation 4 2. Review of results achieved through the implementation of the improvement plans at the municipal, SILAIS and central levels. 3. Analysis of the relationship between organizational climate and the performance of services using pre- and post-intervention climate data and available service statistics provided by the MOH. This analysis included: • Review of pre- and post-intervention climate data for each program phase as well as methods used to apply the climate tools • Analysis of data from the reapplication of the PAHO organizational climate tool in municipalities from the first and second phases to measure the maintenance of organizational climate levels in the post intervention period. • Review of selected pre- and post-intervention service delivery results and analysis of the relationship between these results and organizational climate for the first and second program phases • Comparison of pre- and post-intervention service statistics from participating municipalities and a comparison area during a similar time period 4. Individual interviews and focus groups with participants and facilitators from each level of the MOH and individual interviews with members of the MSH/Nicaragua team and other key informants, as follows: Municipal level − Focus groups with program participants and non-participants in 18 municipalities and one hospital4 − Individual interviews with the Municipal Director, municipal level facilitators and selected participants in 18 municipalities and one hospital SILAIS level − Individual interviews with the SILAIS Director (where possible) or Sub-Director, departmental (SILAIS) facilitators and participating staff Central level − Individual interviews with 10 managers and program participants from 8 departments MSH team − Individual interviews with the program coordinators and facilitators from MSH/Nicaragua 4 Two focus groups were conducted in each of the phase 1 and 2 municipalities, one with health personnel who had directly participated in the replication of the training modules and one with health personnel who had not participated. Exceptions to this rule included San Lorenzo (phase 1) and Santa Lucia (phase 2) where only one focus group with participants was conducted because the majority of the health personnel had already participated in the training replications. In phase 3 municipalities almost all health personnel were involved in the training; therefore one focus group with program participants was conducted in each. Nicaragua Leadership Development Program Evaluation 5 Selection criteria for interviews and focus groups 1. The following criteria were used to select the municipalities included in the evaluation: Project phase and geographic location: Municipalities from each SILAIS and each phase of the program were selected, but with greater representation from Phases 2 and 3. Because the first phase essentially served as a pilot to develop and improve the learning modules and the training and replication process, only one municipality was chosen per SILAIS (total of three municipalities) for this phase. Climate results according to the PAHO organizational climate tool: Municipalities were selected to match the distribution of climate results in the universe of 63 participating municipalities. Approximately two thirds were higher performing municipalities where aggregate climate levels improved over baseline (more than 5% over baseline) and one third were lower performing municipalities where aggregate climate levels either declined or showed little change (less than 5%) over baseline. 2. Respondents for the participant and non-participant focus groups at the municipal level were selected according to the following criteria: • Mix of health personnel from both the municipal health center and several health posts • Mix of professional profiles including physicians, nurses, auxiliary nurses, technicians, and support staff (cleaning or cooking staff) • No members of the municipal management team or supervisory staff 3. Respondents for individual interviews at the municipal level were selected based on the following criteria: • At least one health worker from the health center who had participated in the majority of the leadership development workshops • At least one health worker from a health post who had participated in the majority of the leadership development workshops 4. Respondents for individual interviews at the SILAIS level were selected based on the following criteria: • Manager or staff member who had participated in the majority of the leadership development workshops • Facilitators (2) of the leadership development program at the SILAIS level 5. Respondents for individual interviews at the central level were selected by the MSH/Nicaragua team according to the following criteria: Nicaragua Leadership Development Program Evaluation 6 • Manager or staff member who participated in the majority of the workshops • Key informant for the MOH and the program In total, the evaluation team conducted 109 individual interviews with health staff and managers at the municipal, SILAIS and central levels, and 28 focus groups in 18 municipalities and one hospital. Seventeen focus groups were with program participants and nine were with non￾participants. In total, 309 respondents participated in the evaluation and are displayed in the following table. Table 2. Evaluation Participants by Location and Program Phase Number of Participants SILAIS Municipality Program Phase Individual interviews Focus Group (Participants) Focus Group (Non-participants) Boaco SILAIS office 1 & 2 2 n/a n/a San Lorenzo 1 8 11 n/a Santa Lucía 2 4 8 n/a San José de los Remates 2 2 6 6 Matagalpa SILAIS office 1 & 2 4 n/a n/a Muy Muy 1 3 3 7 San Isidro 2 4 9 7 San Dionisio 2 2 4 5 Matagalpa Municipality 2 4 8 10 Ciudad Dario 2 4 6 8 Jinotega SILAIS office 1 & 2 4 n/a n/a Jinotega Municipality 1 4 2 9 Concordia 2 4 5 7 Yali 2 4 7 7 Madriz SILAIS office 3 3 n/a n/a San Lucas 3 4 7 n/a Palacagüina 3 4 10 n/a Estelí SILAIS office 3 4 n/a n/a San Juan de Dios Hospital 3 4 10 n/a San Nicolás 3 4 6 n/a Rivas SILAIS office 3 3 n/a n/a San Jorge 3 4 7 n/a Rivas Municipality 3 4 8 n/a Masaya SILAIS office 3 4 n/a n/a Monimbó (Masaya Sur) 3 4 9 n/a Tisma 3 4 8 n/a MOH central level 3 10 n/a n/a TOTAL 109 134 66 Guides for the individual interviews and focus groups (Appendix 2) were developed according to the key questions in the M&L Program’s scope of work for in-depth evaluations on the topic of Developing Managers Who Lead (Appendix 3). The individual and group interviews were conducted by one member of the M&L Monitoring and Evaluation Unit and two local consultants from Nicaragua. All interviews were taped and later transcribed to preserve the actual words and expressions of the respondents and special care was taken to maintain the anonymity of each person. Nicaragua Leadership Development Program Evaluation 7 3. Findings 3.1. The municipal level leadership development process 3.1.1. Program description The Nicaragua leadership development program was designed to prepare managers and health workers at the municipal level to assume greater responsibilities and new roles within the context of health sector reform and decentralization. In Nicaragua, the MOH has three administrative levels: central, departmental (SILAIS) and municipal. At the municipal level, health service delivery is composed of a health center in the municipal capital and several health posts in outlying areas. The municipal management team is based in the health center. PROSALUD, through the FFSDP activity, worked with the 12 poorest municipalities of the Jinotega, Matagalpa and Boaco departments. The FFSDP framework included criteria for developing, monitoring and evaluating service delivery points. The municipal leadership development program was developed to operationalize the FFSDP criteria for leadership. At the same time that PROSALUD was developing the leadership development program as part of the FFSDP activity, MSH was awarded the cooperative agreement for the M&L Program and looked to PROSALUD as a partner for one of the early leadership development field tests. M&L had developed a conceptual framework for leadership which it then tested during a two-day dialogue in February 2001 with USAID Mission representatives, central level ministry officials, key SILAIS staff from Boaco, Jinotega, Matagalpa, and MSH/PROSALUD project staff. The primary organizational challenge identified during the dialogue was low motivation among health workers. The lack of motivation among health personnel at decentralized levels was associated with the need to improve organizational climate, which became the central objective of the leadership development program. The first phase of the program was carried out over a 15-month period with the set of 12 municipalities in three SILAIS (Boaco, Jinotega and Matagalpa) already supported by MSH through the PROSALUD bilateral (an additional non-PROSALUD municipality also participated in the program for a total of 13 municipalities). This first phase essentially served to develop and pilot the elements of the leadership development program. The program was meant to complement the management development and technical training and support provided to the municipalities by PROSALUD. The second and third phases were both implemented over a period of six to seven months allowing less time between learning units and climate assessments. The second phase extended the program to the remaining 16 municipalities in the three original SILAIS whereas the third phase expanded coverage to all 34 municipalities in four new SILAIS (Estelí, Madriz, Masaya, and Rivas). The basic programmatic components for the municipal level during all three phases included the following: 1. Baseline and follow-up assessments of organizational climate in all participating municipalities and among SILAIS staff. Nicaragua Leadership Development Program Evaluation 8 2. Delivery of six self-learning units on leadership development for municipal and SILAIS directors and facilitators 3. Replication of the six units with the remaining staff in the participating municipalities by the municipal facilitators supported by the SILAIS facilitators. 4. Development and implementation of action plans designed to address a challenge related to organizational climate detected through the baseline climate assessment. 5. Technical assistance and follow-up by PROSALUD/M&L staff (first phase) and SILAIS facilitators (second and third phases) in the application of concepts learned and the implementation of the action plan. The self-learning units were designed and developed based on results from the baseline application of the PAHO organizational climate tool and results of the discussions on leadership topics with MOH staff. The learning units were designed to provide municipal managers and health staff with the necessary knowledge and skills to respond to the organizational challenges identified through the climate survey. The learning units used the same self instructional training methodology – Evaluation-Training-Planning (ETP) – that PROSALUD had perfected during the FFSDP activity. According to this methodology, trained facilitators use structured modules to facilitate four-hour weekly training sessions. The modules consist of three to six learning units each. Using these units, participants review information from their own context, reflect on their experiences, are enriched conceptually by the training materials, and develop a plan to apply their learning prior to the next training session. During the next session, progress in the implementation of the plan is reviewed by the facilitators and the cycle begins again. The leadership module was one of many modules developed by PROSALUD for the FFSDP activity. The six learning units in the leadership module developed in the first program phase included: • Organizational Climate as an Institutional Challenge • Leadership in Health Institutions • Self-Knowledge and Interpersonal Communication • Coaching • Negotiation • Effective Teams The M&L Leading and Managing framework is covered in the second unit and forms the basis for the explanation of leadership in context of the health system. The first phase of the leadership development program served to validate the learning units which resulted in important revisions to the content prior to the launch of the second phase. In addition, the MOH requested that organizational culture be incorporated into the first unit. As a result, the topics of the learning units in the second and third phases included: • Organizational Culture and Climate • Analysis of Organizational Climate • Leadership in Health Institutions • Self-Knowledge and Interpersonal Communication • Coaching • Negotiation Nicaragua Leadership Development Program Evaluation 9 Over the past year, the six revised learning units of the leadership module were reviewed, finalized and published by MSH and the MOH as the official module for municipal level leadership development within the ministry. This is a substantial achievement and important step towards the institutionalization of the leadership development program within the MOH. 3.1.2. Delivery and replication of the program In the first phase, M&L facilitators delivered the six instructional units directly to municipal directors and facilitators (most often selected members of the municipal management teams) during a series of six workshops. In the second and third phases, MSH/Nicaragua facilitators delivered the learning units to municipal facilitators and were often accompanied by SILAIS facilitators and/or central level human resources staff. The units employ a self-learning strategy; each included four hours of instruction. Assignments were given at the conclusion of the unit for review at the start of the next workshop. In the first phase, the workshops were conducted every few months, depending somewhat on the availability of the M&L facilitators and on interruptions caused by national elections which led to changes in ministry staff in many places. By the second and third phases, the six workshops were generally conducted in one month intervals. The units included a structured manual for facilitators and materials for participants. Using this training format and the materials provided at the workshops, the municipal and SILAIS facilitators were expected to replicate the instructional units with their own personnel. The program delivery was designed with the intention of developing an extended cadre of SILAIS and municipal facilitators capable of replicating and scaling up the program, an important part of the scale-up and institutionalization strategy of the leadership program. Replication of the learning modules for staff at the municipal level was not particularly routine – at times one to two weeks elapsed between units and other times a few months lagged between them. These interruptions were often due to staff absences, a competing program of activities organized by the SILAIS or central levels, disease outbreaks and immunization campaigns. Facilitators tried different methods to accommodate staff availability and work schedules. These included condensing the delivery of the units into less than the allotted four hours of instruction or splitting the unit into two separate two-hour sessions. In accordance with the design of the leadership development program, facilitators in the first and second phases replicated the learning units with the remaining members of the municipal management teams (usually consisting of six to ten staff members including the Director, administrator, epidemiologist, head of training, and supervisors from each program/area) and sometimes with the extended technical committee which includes the management team and a representative from each health post. Across the board, participants and facilitators appreciated both the structure and content of the learning units. Most found the content engaging, inspiring and useful for their professional and personal lives and many commented on consulting the learning units for help when faced with difficulties on the job. Focus groups with non-participants from municipalities in the first and second phases were conducted to determine whether they had benefited from any trickle down or trickle over from Nicaragua Leadership Development Program Evaluation 10 municipal staff who had participated in the leadership program. Non-participants tended to have little knowledge about the program in general and leadership specifically. Their colleagues who had promised to integrate them into the process rarely followed through. As a result, the benefits of the program in the first and second phases primarily resided within the municipal management team or extended technical committee. Comments from non-participants about this situation included: • “We don’t know how they chose who would be trained.” • “All we know is they chose a few health workers and sent them there [to the replications] and now leadership remains there among them and they haven’t told the rest of the staff about it.” • “We heard that they went to a workshop on leadership but here, have they passed it on to the rest of the health workers? No.” Even though the program did not intend for the replications to include more than the municipal management team in the first and second phases, many municipal directors mentioned their desire to have included more health staff in the replications. They cited the lack of resources as the primary obstacle to the widespread application of the learning units with other municipal staff who had not participated in the replication. Funds are needed for photocopying the learning units, paying for the obligatory refreshments and at times providing transport for personnel. A few municipalities have looked for creative ways to overcome these limitations by integrating leadership development into the continuing education program which tends to be a budgeted and permanent process for staff development at the municipal level. In the third phase, the MSH/Nicaragua project team changed the delivery strategy and provided financial support for all health personnel to attend the replication of the learning units in an effort to broaden the participation in and benefits of the program. Even in the third phase, when sufficient funds were available to include all municipal health staff in the replications, municipal managers noted that they were still plagued by a lack of funds which compromised their ability to provide ongoing follow-up to participants and guarantee the continuity of the leadership development program once MSH funding ended. As a result of the program’s replication and scale-up strategy, a total of 1,978 managers and staff were trained by the leadership development program over three years. The following table displays this increase in program coverage. Nicaragua Leadership Development Program Evaluation 11 Table 3. Expansion of participants by program phase Number of participants per phase Participant type First Phase 2001-2002 Second Phase 2002-2003 Third Phase 2003-2004 Total Facilitators (municipal and SILAIS) 32 40 129 201 Participants (municipal and SILAIS) 183 284 1245 1712 Central level participants - - 65 65 Total 215 324 1439 1978 Municipalities were generally visited once by MSH/Nicaragua (and sometimes SILAIS or central level staff) early during the replication of the modules to ensure quality and provide guidance on the use of the learning materials. However, ongoing monitoring of quality throughout the replication process was not provided. Municipal facilitators agreed they would have appreciated greater support and guidance during the replications to confirm they had a full grasp of the material since the content was new to most of them. Some felt they were left on their own to implement and monitor the replications. 3.1.3. Municipal action plans During the first and second learning units of the leadership module, the performance improvement process guides participants to identify their key organizational challenges based on results of their baseline climate assessment. Participants then develop municipal action plans to address the selected challenges. These action plans were generally developed by the municipal facilitators during the initial training workshops (for facilitators) as an assignment between learning units. They selected the top several (this ranged between two and six) climate sub￾dimensions with the weakest scores, analyzed the root causes of the weaknesses, and developed activities to address these causes. They were expected to share the content of the plan with the rest of the municipal health staff following the training session. In practice, however, the municipal facilitators tended only to keep their municipal management team informed. Staff members outside the municipal management team were rarely informed, particularly in the first and second phases. Most participants interviewed who did not belong to the management team knew that a plan existed but were unfamiliar with the content of the plan, its status or results achieved. And most believed the plan was the responsibility of the management team to develop and implement. The majority of non-participants were not aware a plan even existed. For example, a non-participant from the second phase commented: “There is still little communication between the leadership, the administration and the rest of the health workers. We live isolated from what happens with them. To know the results of the plan we first need them to clarify who is the focus of the plan and what purpose it serves.” It appears this situation had improved by the third phase and feedback on the status of the plans was more commonly provided to at least the technical committee and in many cases to the rest of Nicaragua Leadership Development Program Evaluation 12 the municipal staff. Health staff in the third phase tended to be more aware of the content of their municipal action plan as well as the status of the climate measures which formed the basis of the plans. The difference may be both a function of time (the third phase ended only recently and therefore the information was fresher) and an improved project design. Many more health staff participated in the replication of the learning units in the third phase which provided municipal and SILAIS facilitators a convenient vehicle for sharing information rather than relying on regular staff meetings or special staff assemblies. Municipal directors and facilitators generally claimed to have achieved broad participation among health staff in the implementation of the action plan. However, this depends on the definition of participation. Across all phases, most of the health staff interviewed explained that the plan is managed and led by the management team. They “participate” when notified that a specific activity has been scheduled by the municipal facilitators, but they have little active responsibility in carrying out activities in the plan. Only in a few cases – San Lorenzo is most notable – have the extended management teams formed working groups responsible for conducting activities, such as the regular maintenance of the health center information board (mural) with updated materials and information. In terms of the content of the municipal action plans, several commonalities were noted among the selected challenges and the proposed activities. Municipalities in all three phases tended to focus on similar challenges: exchange of information (communication), recognition of contributions, responsibility, and stimulation of excellence were the most commonly selected challenges. They also tended to propose similar activities to address these challenges, likely because the plans were developed and shared with other participants during the initial training workshops for municipal facilitators. The commonly proposed activities included: − more regular technical committee meetings and/or general staff assemblies to increase communication between management and staff − public information boards (murals) for information sharing − posting of organigrams and MOH internal norms − posting of staff schedules in health centers to publicize whereabouts of staff − more systematic performance reviews − periodic training sessions for municipal staff − certificates for recognition of performance These activities, found in most of the action plans available for review (none were available from phase 1 municipalities; the majority were available from phase 2 and about half from phase 3) were fairly generic. Except for new activities such as informational boards, certificates of recognition, and in one case a banner given to health units for meeting performance goals, most of the actions in the plans were things that the municipal management team had at one time or another done (e.g. technical meetings, performance reviews, posting of norms, etc). According to both participants and management team members, the difference was in the way the management team conducted the activity. As a result of the leadership program, these rather ordinary activities tended to be more systematic, more participatory (especially the all staff meetings), and more transparent. For the first time, municipal staff felt more included in decisions made by the management team and thought that their opinions mattered. These Nicaragua Leadership Development Program Evaluation 13 perceived changes, and those presented in section 3.1.4 below, help to explain the measurable changes in climate discussed in section 3.4 of this report. The majority of the municipal facilitators reviewed the progress of their action plans on a regular basis with the municipal management teams during the course of the program. They also claimed to have shared this information with the rest of the municipal health staff during regular staff meetings or staff assemblies, but most participants stated that after the first few months of the program, they were no longer informed about the status of these plans. Even in these cases where municipal staff was not up-to-date on the content or progress of the action plan, they tended to note that members of the management team communicated in a more regular and open way with municipal staff. Follow-up and monitoring of the municipal plans from outside the municipality by SILAIS authorities or MSH/Nicaragua staff was more sporadic rather than systematic. In the first phase, MSH tended to combine technical support visits for the FFSDP activity with follow-up on the leadership action plans. From the point of view of the municipalities, this arrangement worked well because they were able to address jointly the leadership and management issues of their health centers. In the second and third phases, however, leadership development was no longer implemented within the context of the PROSALUD FFSDP activity. Rather, it became a stand￾alone project as the overall PROSALUD project came to an end. Municipalities in these phases were generally visited once during the leadership development program to monitor the quality of the replication training and review progress in the implementation of their action plans. It was too costly for MSH, with limited staff resources, to provide ongoing monitoring and follow-up to such a broad coverage of municipalities. When follow-up did occur, in the majority of cases, only the municipal director and facilitators were contacted during the visit; health personnel rarely participated. This explains why most health personnel were unaware of any monitoring or follow-up that had taken place by the SILAIS or MSH/Nicaragua. There was general agreement among the facilitators and directors that the focus of the follow-up visit was to ensure that the training materials for the replications had been delivered and to record the number of activities completed. Barriers to the implementation of certain activities were also discussed and suggestions made. But technical assistance on leadership competencies or strategies that could be used to enhance the implementation of the plans was usually not provided. Municipal facilitators tended to feel that the substance of feedback during the follow-up visits was not of great value. Nevertheless, they did receive direct support on the municipal plans during the training sessions for municipal facilitators. They reviewed their progress at the beginning of each new learning unit and received feedback from the MSH facilitators which they felt was valuable and applicable to their situations. The responsibility for providing follow-up to the municipal facilitators after the completion of the six workshops (and their replication at the local level) was left in the hands of the SILAIS, but has rarely occurred. A municipal facilitator noted a commonly described situation: “During the facilitator workshops we received a lot support from MSH… and when we were implementing the replications we received support one time from someone from MSH; afterwards we have essentially been alone Nicaragua Leadership Development Program Evaluation 14 to implement this at the municipal level. They agreed that the SILAIS would do a round of supervisions, but to be honest, the SILAIS was never with us during any of the replications.” Another municipal facilitator explained that monitoring ended after the delivery of the units was completed: “the things in our plan have discontinued because of the lack of monitoring, we need monitoring now that this person from PROSALUD is no longer here who used to come and ask how things were going…” Implementation of the action plans has varied across participating municipalities. Municipalities in all three phases tended to begin implementing the activities in their plan during the leadership development program itself, but follow through with the plans was mixed once the six learning units had been delivered and replicated. A review of action plans with managers from the municipalities and a hospital visited during the evaluation revealed less than half had completely implemented their plans. Reasons given for the lack of follow-through included: lack of monitoring by MSH/Nicaragua or the SILAIS; interference of other priorities like health campaigns; lack of widespread knowledge of the plans among municipal health staff; and lack of financial resources to fund certain activities, such as those meant to recognize staff performance with diplomas or other costly methods. On the other hand, several of the municipalities visited (one from the first phase in Boaco and three from the second phase in Matagalpa) have already developed a second action plan to provide follow-up to their progress achieved and/or address additional deficiencies in climate. If the municipal action plans are considered to be an important piece of the leadership program, greater attention and follow-up should be provided to support their implementation. 3.1.4. Perceived improvements in leadership capacity Members of the municipal management teams and program participants from all phases tended to note an improvement in the performance of the health personnel (including those from the municipal health centers and the outlying health posts) who had participated in the leadership development program. These include greater cooperation and willingness to take on assignments; increased motivation, enthusiasm and interest in their work; and better coordination among staff in order to complete their particular tasks. Health personnel feel more involved in the activities of the municipality, tend to work in teams more often and are more concerned about fulfilling their municipal indicators and targets than ever before. Participants generally defined the overall objective of their municipal action plans as improving interpersonal relationships which they recognized as one of the main problems affecting their health centers and health posts. In one focus group, a participant commented: "As we have little capacity to negotiate and to communicate with each other, the challenge is to try to eliminate conflicts between us and if there are, to try through the modules [learning units] to improve these struggles.” As a result, health workers and management staff both noticed increased solidarity in their work units and a greater inclination to work together towards common goals. In the words of one participant, "we no longer function as islands as we were before." Another commented “we Nicaragua Leadership Development Program Evaluation 15 have improved the communication in our team with the purpose of solving problems together.” Other relevant comments include: • “Before we [at the health center] would ask if they would pay us per diem to make supervision visits to the health posts – we would say if they don’t pay us we won’t go. Now no, now we are more conscientious and we help out whenever a health post needs us. If health post needs extra support we will go there to fill in. For example the Inferno health post only has two medical resources, a doctor and auxiliary nurse, and covers very remote areas and so when they ask for help… we form a team of 10 -12 people [from the health center] to go support the health post…” • “The program has served to sensitize us to whatever little thing that happens in the health unit; when someone asks us something, we don’t stay seated, we help that person, everyone collaborates…” • “We are now more accepting; we can accept constructive criticism and can work together.” However, these improvements are not spread uniformly across municipalities. In a few of the municipalities visited, participants described a climate that was tense and unproductive where labor relations between health workers and the management staff either had not improved or had deteriorated. These responses tended to coincide with the results of the climate assessment in these municipalities in which climate had not changed or had declined (climate data are provided in next section). In these municipalities, the performance of participants has not changed; rather they tend to continue to work with the same attitudes and behaviors as before. Some health personnel expressed this result as follows: • “There is still mistreatment by supervisors; they should use better communication to eliminate a series of problems between colleagues.” • “Supposedly the people who received the workshops should have made some change but this change hasn’t happened; the people are the same or worse.” In terms of municipal supervisors, in general municipal staff noted important modifications in the behavior of their supervisors and a notable improvement in communication styles between supervisors and subordinates. The most common and relevant changes include: supervisors tend to be more supportive and understanding of staff needs, more communicative and accessible with staff, more invested in municipal programs, and less afraid to recognize or expose their own shortcomings. They tend to listen to the opinions of the staff and take them into consideration. In addition, there is better recognition of staff performance now that “in supervision visits they don’t only see the bad” and they are more likely to criticize or reprimand staff in private. All of these noted changes involve the leadership competencies covered in the learning units. Likewise, supervisors tended to perceive changes in their own performance. For example, they are less afraid of change, strive to motivate health personnel to improve the quality of their work and have tried to be more approachable to staff. They believe the leadership development Nicaragua Leadership Development Program Evaluation 16 program provided the tools to help them understand their role as leaders and to better manage their health units by supporting team building processes. Importantly they have learned to listen, reflect, recognize their own errors, negotiate, and delegate functions. Regarding their performance as leaders, municipal directors and facilitators appreciated the guidance and feedback they received from MSH/Nicaragua during the facilitator workshops. However most claim that now that the workshops have ended, they have received little feedback from their corresponding SILAIS or from central level representatives to reinforce the changes they have made in their leadership behaviors or styles. 3.2. Leadership development at the central level 3.2.1. The approach and process In the third phase of the program (July 2003 to present), a separate leadership development program was designed and delivered for the SILAIS and Central levels which consisted of the following key components: 1. Leadership Dialogue to define challenges facing the MOH and identify individual leadership competencies 2. Four two-day leadership development workshops for central and SILAIS staff 3. Final evaluation workshop 4. Coaching and follow-up provided by MSH/Nicaragua Sixty-one central level managers and staff participated in the central level program, the majority of whom (54%) attended all training activities. Attendance at each workshop ranged from 46 to 50 participants. As a strong sign of support, the Minister and Vice-Minister at times attended some of the workshops. The objective of this leadership development program was to address the challenge of aligning the National Health Plan (2003-2008) with the Health Care Model5 . Initially participants had defined these issues as two separate challenges but eventually they saw the need and benefits of aligning the two as one overriding challenge. Participants interviewed explained the need to address this challenge stemmed from: • “The need to integrate the distinct divisions because they were all dispersed and working separately.” • “We couldn’t continue working each person in his/her area with no connections.” • “The search for a better way to organize the reforms.” • “The MOH was in the process of leadership and modernization and we needed to stimulate all of the activities that we intended to carry out to succeed with the transformation, break existing paradigms and have positive attitude towards change…” The topics covered in the workshops were based on the assessment of leadership practices and competencies and the selection of those deemed necessary to address the selected institutional challenge: change management, strategic thinking, communication and negotiation. 5 The Health Care Model establishes the operational plans for the MOH and identifies the different types and organization of services needed to achieve the objectives in the national plan Nicaragua Leadership Development Program Evaluation 17 3.2.2. The central level action plan Throughout the program, participants developed a plan to guide the integration of the two main strategies of the MOH – the National Health Plan and Health Care Model – in order to better respond to the needs of the Nicaraguan population. The need to align both processes was identified as a priority challenge for the MOH and will serve as “our bearing for the next several years.” Addressing this challenge means involving and gaining consensus of all divisions and departments at the central level within an organizational culture of administrative and operational disorganization and disarray. To date, participants at the central are still in the process of implementing the plan. To coordinate and guide the internal processes needed to ensure its implementation, participants have formed four working groups with the following designations: Coordination/Integration, Communication, Monitoring, and Motivation/Feedback. Members of these groups describe their difficulties in meeting regularly due to the lack of time, multiple competing tasks and work overload caused by what they call "the constant activism that we experience which tends to override our planned agendas.” Barriers to the implementation of the plan itself include: the isolated working styles between the departments; resistance to change and insufficient communication between departments and divisions; and the lack of feedback on the process by participants to other members of their divisions or departments. Nevertheless, participants have made notable progress on the plan, especially given the difficult context of the central level and the complicated nature of their selected challenge. Progress worth mentioning includes: • Development and consensus on the conceptual framework for the National Health Plan and a finalized Health Care Model • Development and consensus on a format and joint timeline for integrating the Health Plan and Health Care Model • Establishment of a permanent Technical Leadership Commission which is responsible for decision making and follow-up for the central-level leadership process • Organization of the four leadership working groups mentioned above • Coordination meetings held between the technical committees responsible for the Health Plan and Health Care Model These advances correspond to the four stages identified for the implementation of the plan: − Coordination of the Health Plan and Health Care Model technical committees − Development of key documents − Technical discussion and consensus on documents produced − Dissemination, communication and ownership of the process by the MOH Because numerous activities remain in progress and achieving the challenge is paramount to the MOH, the central level program was extended through June 2004 with continued coaching and follow-up by MSH/Nicaragua. Nicaragua Leadership Development Program Evaluation 18 3.2.3. Perceived improvements in leadership capacity Participants interviewed at the central level mentioned numerous changes brought about by their participation in the leadership development program. Among the most salient are better coordination and information sharing between the heads of divisions and departments who participated in the program; increased involvement of the staff of these departments/divisions in decisions; and a more positive work atmosphere and better distribution of tasks among staff within their immediate work group or program. Participants also point to more frequent communication on the progress of program activities and broader ministerial issues between the directors who participated in the program and their staff. • “We are more focused on the integration of processes, avoiding conflict, and including more staff.” • “There is always someone who directs, but he/she is more open now." • "We have more meetings with staff teams to improve communication." • “The [central] technical committee has expanded and other directors have started to attend; before it was only open to the highest level staff and now people from different teams [divisions] are attending.” In terms of changes in their leadership behaviors and styles, directors and supervisors interviewed point to creating more harmonious working relationships with other departmental heads, sharing information with their staff on a more regular basis, and including more staff in decision-making. Expressions of these changes include: • “I listen to what others say and try to understand them.” • “The leadership program has helped me to delegate functions and decentralize activities; our agendas are formed by consensus now.” • “Before I only wanted to do the work myself and I forgot that there were others around me with the same capacities who could do the work, and this helped me to decentralize activities and delegate more of the work in the department.” • “The processes [of the leadership development program] help us to work with data and evidence instead of individual perceptions and mistaken assumptions.” In terms of changes in the performance of their staff, the directors and supervisors generally characterized these as "subtle." One explained that "it is not easy to change the paradigm, even more when there is a custom of how things are done…and we do not make a maximum effort to change." This observation was confirmed by other participants interviewed who maintained that they have experienced little change in the way they coordinate or communicate with each other, at least within individual departments. Department staff has for the most part continued their isolated manner of "working by program… there is no integral vision, every one does what belongs to him/her; we are trying, we are taking steps, but we have not achieved it [change]". One comment which was echoed by several staff participants: "to have a radical change you needed to train all staff." Likewise another noted that “little by little they [staff] have been integrated into the processes…but the few of us here alone will not be able to do it.” Nicaragua Leadership Development Program Evaluation 19 3.3. Leadership development at the SILAIS level 3.3.1. The approach and process During the first phase, SILAIS facilitators received the same content as the municipal facilitators and replicated it among their management staff. The same three SILAIS and facilitators were involved in the second phase and most continued the process already begun in their SILAIS. In the third phase, a program focused specifically on the SILAIS level was initiated which mirrored the central level leadership development intervention described above. In this phase, 5-6 participants from the original three SILAIS (Boaco, Jinotega and Matagalpa) as well as the four additional SILAIS in phase 3 (Estelí, Madriz, Masaya and Rivas) were included. Among the seven SILAIS in the third phase, only Masaya trained their entire staff in the leadership module. Boaco and Madriz trained half the SILAIS staff while Matagalpa, Jinotega, Estelí and Rivas included one third. Participants explained that the main limitations to including all staff in the replication workshops were lack of financing and competing work schedules. The replication workshops took place after the SILAIS facilitators returned from their leadership training. These workshops were directed mainly at the SILAIS management teams and heads of departments. The intention was “first to unify the management team so it could work together in a coordinated manner rather than separately like it had been doing”, because the SILAIS leadership development program “is a little more focused on the management team and this experience will then be transmitted to the rest of the SILAIS staff.” 3.3.2. SILAIS Action plans In the first and second phases of the program, the SILAIS facilitators and management teams developed action plans based on the results of their climate assessments, but in general they were not implemented fully by the SILAIS or followed carefully by MSH/Nicaragua. During the third phase, however, the program placed a greater emphasis on the SILAIS action plans. The challenges selected by SILAIS participants in the third phase included: SILAIS Selected challenge (Phase 3) Boaco Restore the motivation to work in teams within a hostile environment Jinotega Improve the performance of administrative staff in the SILAIS office and in the municipalities according to the new functions and standards Matagalpa Improve the exchange of information between members of the management team, program heads and the remaining staff in the SILAIS office Estelí Improve organizational climate with regard to communication Madriz Improve the participation of SILAIS staff thereby affecting the exchange of information. Masaya Improve communication processes between the first and second levels of care Rivas Systematic monitoring system that identifies problems and orients the development of interventions to address improving health services Nicaragua Leadership Development Program Evaluation 20 The action plans were developed based on an analysis of the needs, problems and weaknesses in the different SILAIS departments and programs: “each participant brought from his/her office the goals, workplan, and common problems and then they [from each SILAIS] selected what was common among all of them.” They also considered statistical data from the health programs and results from their respective baseline climate assessment. Progress in implementing the action plans has varied across SILAIS and all are still in the process of implementation. Commonly noted barriers to implementation included the lack of time, prioritization of other activities, rotation of directors, and a lack of consistent follow-up by the SILAIS facilitators. Participants tended to characterize the difficulties as follows: • “Implementation occurred during the courses but we have done nothing since.” • “No one provided follow-up after the program.” • “During the workshops they shared information about the plan [with staff], afterwards they tried to implement but nothing advanced.” Each plan contains a set of indicators to measure achievement of the challenge, but data on these indicators were not available from any SILAIS. SILAIS facilitators generally were not using the indicators but rather tended to monitor the completion of activities in the plan during regular management meetings. In terms of providing feedback to staff on the progress of the action plans and the results of the climate assessments, similar patterns were noted at the SILAIS levels as the municipal levels. Little feedback was provided by the SILAIS facilitators except to the management team. The rest of the SILAIS staff tended to be unfamiliar with the plans and progress of the program. Again, the plan resided within the management team, and most often was seen as the responsibility of the facilitators to implement. 3.3.3. Perceived improvements in leadership capacity The replication of the leadership units and implementation of the action plans at the SILAIS level are mainly associated with more open communication, both horizontal and vertical; better defined roles and functions of personnel; and more consistent use of team work. Staff members interviewed tended to be more motivated to complete their work load and less likely to cling to the status quo of old working arrangements and relationships. As a result, they are more likely to work in teams and to avoid conflict. For example: • “Information sharing is now more fluid between directors and staff… and the work atmosphere is less tense.” • “We look for new arrangements to get the work done.” • “We work together in teams more often to accomplish goals.” Participants also noted behavior changes among their supervisors, including: • “Discretion about reprimanding staff members in public.” Nicaragua Leadership Development Program Evaluation 21 • “The director is more involved in the different programs and concerned about maintaining organizational climate levels.” • “My supervisor now has the capacity to be self-critical.” In terms of external follow-up and support, most participants refer to guidance provided by MSH/Nicaragua during the workshops and monitoring visits made at the beginning of the program; however, from the central level “we have not received any assistance or monitoring.” Nevertheless, a director of one SILAIS explained where he derives feedback on his performance: “The fact that we are constantly evaluating organizational climate has in some way served as my source of feedback.” 3.4. Organizational climate This section examines the main outcome of the leadership development program: improvement in organizational climate in participating municipalities and SILAIS of the MOH. Climate is a measure of the perceptions of the organization's members in relation to objective reality: the goals, structure, and functioning of the organization. Individual perceptions of that reality, and their reactions in terms of expectations, needs, and desires, determine the level of employee motivation and satisfaction. 3.4.1. Description of the PAHO organizational climate instrument Organizational climate was assessed through the use of an instrument designed and field tested by the Pan American Health Organization (PAHO) Subregional Project for the Development of Management Skills in the Health Services. This tool is designed to be administered to health managers and their staff, regardless of the level of the organization. The instrument contains a set of 80 randomly-arranged items (behavioral statements) that reflect individual perceptions about the organization and the individual’s immediate workplace. Participants respond to each of the statements, indicating whether or not (yes/no) the statement applies to their particular organizational milieu or immediate supervisor (depending on the statement). The 80 items are divided into four critical areas or dimensions of climate which are intended to provide a comprehensive portrait of the organization: Leadership: the influence exercised by certain individuals, particularly managers, over the behavior of others in order to achieve certain results Motivation: an individual’s intentions and expectations within their organizational milieu Reciprocity: the give-and-take relationship between the individual and the organization Participation: the involvement and contribution of individuals and formal and informal groups to the achievement of objectives in the organization Each dimension of climate is based, in turn, on different sub-dimensions. The sub-dimensions represent a consolidation of five items from the tool, and the climate dimension is the Nicaragua Leadership Development Program Evaluation 22 consolidation of four sub-dimensions. Due to the particular design of the instrument, the scoring system has many levels. Each of the 80 items receives a “yes” or “no” response with a corresponding value of one or zero points, respectively. Missing data are excluded from the scoring. A score for each item is calculated by summing the responses together to form a subtotal which is then divided by the number of valid responses. The range of possible values for each item score is between zero and one (i.e., a proportion). Five item scores or proportions are then summed together to form a total of 16 sub-dimensions scores, each with a range of zero to five. The four sub-dimension scores are then averaged together to form four dimension scores, each with a range of zero to five. A total climate score is calculated as the average of the four dimension scores. This final climate score again has a range between zero and five. Developers of the PAHO tool suggest that given the range of possible scores, a score of three would represent a median or target score for each dimension and for the overall climate score. Table 4. Dimensions and Sub-dimensions of the PAHO climate instrument Dimensions (4) Sub-dimensions (16) Leadership • Management • Stimulation of excellence • Promotion of teamwork • Conflict resolution Motivation • Personal fulfillment • Recognition of contributions • Responsibility • Suitability of working conditions Reciprocity • Dedication to work • Stewardship of institutional assets • Compensation • Equity Participation • Commitment to productivity • Harmonization of interests • Exchange of information • Involvement in change Appendices 4 and 5 contain a copy of the full climate instrument as well as definitions of the dimensions and their related sub-dimensions. 3.4.2. Application of the PAHO climate instrument Climate levels at the municipal level and in most of the 7 SILAIS were assessed using the PAHO instrument at the beginning and end of each phase of the leadership development program. Climate was not measured at the central level. Program facilitators from the municipal and SILAIS levels were instructed by MSH/Nicaragua staff to use a minimum sample of 19 respondents selected randomly from each area (municipality or SILAIS). In most cases, this sample represented one quarter or more of the health workers in the municipality or SILAIS. In some of the smaller municipalities, the facilitators chose to apply the survey to all health staff in their municipality. Nicaragua Leadership Development Program Evaluation 23 The sample size was selected based on MSH/Nicaragua’s familiarity with the principles of Lot Quality Assurance Sampling (LQAS). The most commonly used LQAS sample is 19 respondents per lot or supervision area (unit of analysis) which provides a simple classification of coverage results within a lot (in terms of reaching the coverage target or not) when analyzed as a binomial. The data can also provide coverage results with a 95% confidence interval when analyzed as a proportion derived from data aggregated from four or more lots. In the case of the climate assessment, the municipality or SILAIS was considered to be the lot for sampling purposes. Although the climate data are collected as yes/no binomial responses to the 80 items in the survey, they are not analyzed as a binomial but rather converted to proportions and analyzed as interval data for each municipality. According to the LQAS methodology, at least four lots are normally needed to provide a sufficient sample size for calculating proportions. For the application of the climate survey, respondents were randomly selected in each municipality and SILAIS, usually according to the staff list; however, the universe for selecting the sample of 19 differed. At the municipal level, for example, in some cases only members of the extended technical committee were sampled or only health staff from the municipal health center. In other cases, the universe included all health staff in the municipality and in still other cases, the entire municipality staff was sampled, including cleaning and security personnel. At the SILAIS level the sampling universe differed as well. Some SILAIS included only extended management staff members in the sample selection and others included all staff members from the SILAIS. The instrument was applied by municipal and SILAIS facilitators at two points during the program: between the first and second training module and following the last training module. Different strategies were used to apply the survey. For example, sometimes facilitators were able to take advantage of full staff meetings where all health personnel were present, and in other cases facilitators applied the survey during regular visits to health posts or they made special visits to collect the data from selected respondents. Before applying the instrument, facilitators provided respondents with a brief overview of the purpose of measuring climate and the scope of the instrument. MSH/Nicaragua provided municipal and SILAIS managers with a diskette containing an Excel data entry spreadsheet with a number of scoring formulas that generated a graphical representation of results once all data were entered. The data were entered into the worksheets either by the municipal or SILAIS facilitators and later submitted to MSH/Nicaragua. Electronic copies of the data are kept at the SILAIS level and with the MSH/Nicaragua team. In cases where municipal managers have computers, most have electronic copies of the data as well. 3.4.3. Overall change in municipal climate scores by SILAIS and program phase As a backdrop to the presentation of climate results throughout the following sections, it is important to keep in mind the sample sizes available for the analysis of climate data. The following table displays the number of participating municipalities (plus hospitals in the third phase) per SILAIS and program phase. Nicaragua Leadership Development Program Evaluation 24 Phase SILAIS No. municipalities (and hospitals) Total Boaco 2 1 Jinotega 5 Matagalpa 6 13 Boaco 4 2 Jinotega 3 Matagalpa 9 16 Esteli 8 Madriz* 10 Masaya 10 3 Rivas 10 38 Total n 67 *Hospital Madriz was excluded from data analysis due to missing baseline data; therefore the sample available for analysis for this SILAIS is n=9 Results of the analysis of organizational climate data in the following sections are first presented at a macro level and then by individual program phase. The following graph displays the percent change in overall climate scores across participating municipalities in each phase of the program. Data for Phase 3 include three of the four participating hospitals (Hospital Madriz is excluded). Climate data from participating SILAIS are presented separately (in the next section). It is important to keep in mind that a different group of municipalities participated in each of the three phases of the leadership development program. Therefore this graph simply presents the global change in climate scores produced by each phase, showing a trend of increasing positive change with each subsequent phase. Using both SPSS Version II and Excel 2000, significance tests were performed on the mean climate scores (baseline and follow-up) from the set of municipalities in each of the three program phases in the above graph. The Wilcoxon Rank Sum was used for Phases 1 and 2 because of the small sample sizes in both groups whereas the Paired Difference t-test was used Percent change in climate levels by program phase 0.60% 11.63% 14.01% 0% 2% 4% 6% 8% 10% 12% 14% 16% Phase 1 Phase 2 Phase 3 7/2001 - 6/2002 11/2002 - 6/2003 7/2003 - 1/2004 Nicaragua Leadership Development Program Evaluation 25 for municipalities in phase 3. Results indicate that differences in baseline and follow-up climate scores among municipalities in the first phase were not significant (p=.85, n=13). In contrast, results suggest that changes between baseline and follow-up climate levels in both the second (p=.02, n=16) and third phases (p<.0001, n=37) were significant. The data used for this analysis are presented in Appendix 6. The graph above presents the distribution of the changes in each program phase broken down by the participating SILAIS in each phase. The majority of change in climate levels in both Phase 1 and Phase 2 is attributed to a single SILAIS: Boaco in Phase 1 and Jinotega in Phase 2. In Phase 3, all SILAIS contributed to the overall change, led by Rivas and Masaya. The next section displays climate scores by individual municipality in each SILAIS for each of the three program phases. Percent change in climate levels by SILAIS and program phase 20% 1% 3% 41% 8% 7% 11% 17% 20% -6% -10% 0% 10% 20% 30% 40% 50% Boaco Jinotega Matagalpa Boaco Jinotega Matagalpa Estelí Madriz Masaya Rivas Phase 1 Phase 2 Phase 3 Nicaragua Leadership Development Program Evaluation 26 3.4.4. Climate results by municipality and program phase Baseline and follow-up climate results at the municipal level are presented below for all three SILAIS in the first phase of the program. The time difference between baseline and follow-up measures in this phase was 14-15 months. These data show mixed results across the 13 participating municipalities. Climate levels improved in six municipalities, with a range of 3% to 33% improvement over baseline levels (San Lorenzo, Bocay, Jinotega, Paiwas, Pantasma, and La Dalia). The most notable improvements belong to San Lorenzo (33% increase). Four of these municipalities achieved a final climate value higher than the target level 3 (San Lorenzo, Jinotega, La Dalia, and Paiwas). The final value for climate in Muy Muy is also greater than 3, but in this case, the municipality began the program with the second highest climate score (3.44) and showed a slight decline (3%) in the follow-up survey, finishing at 3.35 points. In the remaining seven municipalities, climate scores declined from baseline levels by a range of -28% to -3% except for two cases (Camoapa and Rancho Grande) which showed no change in climate overall. Baseline and followup climate scores: Phase 1 municipalities 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 La Dalia Muy Muy Paiwas Rancho Grande Rio Blanco Waslala Bocay Cua Jinotega Pantasma Wiwili Camoapa San Lorenzo Matagalpa Jinotega Boaco Jun-01 Aug-02 Nicaragua Leadership Development Program Evaluation 27 Baseline and follow-up climate results at the municipal level are presented below for all three SILAIS in the second phase of the program. The difference between baseline and follow-up applications of the climate survey in this phase was 6-7 months. In this phase, results are positive in the majority of the 16 municipalities, with a range of -22% to 74% change over baseline. The most notable improvements occurred in six municipalities: Esquipulas (18%), Darío (42%), Sebaco (18%), San Isidrio (18%), San Rafael del Norte (74%) and Yalí (45%). All but five municipalities (Terrabona, Matagalpa, Condordia, San Dionisio and San Ramon) maintained their climate levels above the target level 3. Only three municipalities showed a decline in climate levels: San Dionisio, San Ramon (Matagalpa SILAIS) and San Jose de los Remates (Boaco SILAIS), with a range of -22% to -3%. The third phase of the program included 34 municipalities and teams from four hospitals (two in Esteli, one in Madriz, one in Rivas), in addition to participants from the SILAIS and central levels (addressed in an earlier section). The third phase ran from July 2003 to January 2004 (with final workshops in February 2004 to review progress and climate data), providing a 6-7 month stretch between baseline and follow-up measures. This timeframe is similar to that used during the second phase of the program. Baseline and followup climate scores: Phase 2 municipalities 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 Matiguas Esquipulas Dario San Isidrio Sebaco San Dionisio San Ramon Terrabona Matagalpa S. Rafael Norte Yali Concordia Boaco Santa Lucia Teustepe San Jose Remates Matagalpa Jinotega Boaco Jan-03 Jun-03 Nicaragua Leadership Development Program Evaluation 28 The data for Phase 3 are displayed by individual SILAIS due to the large number of participating municipalities in each. The graphs in this next section display changes in climate levels for municipalities and participating hospitals together because they received the same leadership intervention. The following graph displays municipal climate data from the SILAIS Estelí. Climate levels in the SILAIS Estelí improved in all municipalities and the two hospitals with a range of .39% to 27%. The least amount of change over baseline levels occurred in the Hospital San Juan de Dios (.39%) and La Trinidad municipality (.66%), whereas San Nicolas (14%) and Condega (27%) experienced the most notable improvements. Although Condega achieved the greatest amount of change over baseline levels, this municipality has still not reached the target level of 3. The percent increase over baseline levels across all municipalities and two participating hospitals in SILAIS Estelí was 7%. In Madriz, the baseline climate assessment was conducted one month later than the other SILAIS in Phase 3, allowing a total of 6 months between baseline and follow-up measures. Baseline data are not available for the Madriz hospital, so this facility was excluded from the analysis. The following graph displays municipal climate data from the SILAIS Madriz. Baseline and followup climate in Phase 3 municipalities and hospitals: SILAIS Esteli 0.0 1.0 2.0 3.0 4.0 5.0 Condega Estelí La Trinidad Pueblo Nuevo San Juan de Limay San Nicolás Hosp. La Trinidad Hosp. San Juan de Dios Jul-03 Jan-04 Nicaragua Leadership Development Program Evaluation 29 Climate scores in Madriz started higher and ended highest among all SILAIS in Phase 3. Improvements are noted in six of the nine municipalities, with the greatest amount of increase concentrated in four municipalities: Yalaguina (13%), Totogalpa (17%), Palacuaguina (27%) and La Sabana (55%). Although climate declined in three municipalities (San Jose Cusmapa, San Lucas and San Jan del Río Coco) and showed only minimal positive change in Telpaneca, follow-up climate levels for all municipalities remained above the target level 3. The overall percent change over baseline levels for the SILAIS Madriz is 11%. The following graph displays municipal climate data from the SILAIS Masaya. Baseline and followup climate in Phase 3 municipalities: SILAIS Madriz 0.0 1.0 2.0 3.0 4.0 5.0 La Sabana Palacaguina San Jose Cusmapa San Juan Río Coco San Lucas Somoto Telpaneca Totogalpa Yalaguina Aug-03 Jan-04 Baseline and followup climate in Phase 3 municipalites: SILAIS Masaya 0.0 1.0 2.0 3.0 4.0 5.0 Catarina La Concepción Masatepe Masaya Norte Masaya Sur Nandasmo Nindirí Niquinohomo San Juan de Oriente Tisma Jul-03 Jan-04 Nicaragua Leadership Development Program Evaluation 30 Baseline climate scores in Masaya were the lowest among the SILAIS in Phase 3 and follow-up levels were also the lowest. The change in climate scores in Masaya varied across the 10 municipalities, with declines in three municipalities (Catarina, La Concepción and Nandasmo). As in the previous SILAIS, notable improvements in climate were primarily concentrated in a few (five) municipalities: Nindirí (19%), Niquinohomo (32%), Masaya Norte (40%), San Juan de Oriente (42%) and Masatepe (49%). The overall percent increase over baseline levels for the SILAIS is 17%, yet follow-up climate levels across the majority of municipalities tended to hover around the means score of 3 or less. Climate levels across all municipalities in Masaya are generally lower than in other SILAIS in Phase 3. Data for the final SILAIS in Phase 3, Rivas, are presented in the following graph. Rivas showed the greatest percent increase in climate scores (42%) over baseline levels among the four SILAIS in Phase 3. Climate levels improved in all but one municipality (Altagracia). The most prominent changes occurred in four municipalities: Potosí (21%), Tola (29%), San Jorge (69%), and Belén (70%). Climate levels in three municipalities still remain below the target level 3: San Juan, Tola and Hospital Rivas. 3.4.5. Comparison of climate dimensions and sub-dimensions at the municipal level The analysis of overall climate levels in the preceding sections tends to mask changes occurring in the dimension and sub-dimension levels. It is important to mention that, with the exception of a few municipalities where no sub-dimensions improved in Phase 1, those municipalities that showed declines in overall climate levels still achieved improvements in individual sub￾dimensions. These achievements are not evident in an analysis of the overall climate score. The graphs and table below present the overall change in the four climate dimensions and 16 sub￾dimensions over the life of the program. Baseline and followup climate in Phase 3 municipalities and hospital: SILAIS Rivas 0.0 1.0 2.0 3.0 4.0 5.0 Altagracia Belen Buenos Aires Cardenas Moyagalpa Potosi San Jorge San Juan Tola Hospital Rivas Jul-03 Jan-04 Nicaragua Leadership Development Program Evaluation 31 Table 5. Change in climate dimensions over life of program Dimension Baseline Follow-up % change Leadership 3.31 3.49 5.34% Motivation 2.71 3.01 11.32% Reciprocity 2.92 3.12 6.63% Participation 2.63 2.96 12.57% These data reveal an improvement in each of the four dimensions of organizational climate across the three phases of the leadership program. The two weakest dimensions of climate reflected the greatest degree of improvement: Motivation (13%) and Participation (11%). Leadership and Reciprocity were the strongest of the dimensions at the start of the program in 2001 and by 2004 had changed to a lesser degree over these baseline levels. At the time of the evaluation, Leadership remained the strongest dimension of climate, followed by Reciprocity. Changes over the three year period by individual climate sub-dimension are displayed in the following graph. Percent change in climate dimensions over life of program (2001 - 2004) 0% 2% 4% 6% 8% 10% 12% 14% Leadership Motivation Reciprocity Participation Nicaragua Leadership Development Program Evaluation 32 While all sub-dimensions improved over the life of the program, the above graph reveals different levels of improvement across the sub-dimensions of climate. Seven sub-dimensions showed a greater than 10% improvement across the three program phases and all belong to either the Motivation or Participation dimensions: “Recognition of contributions,” “Responsibility,” “Adequate working conditions,” Commitment to productivity,” “Harmonization of interests,” “Exchange of information,” “Involvement in change.” These were among the nine weakest sub￾dimensions at the beginning of the leadership development program. As a result, most participating municipalities selected one or more of these weaker sub-dimensions as the primary focus of their climate improvement plans. The remaining two weak sub-dimensions (belonging to the Reciprocity Dimension) showed little improvement: “Compensation” and “Equity”. However, these were rarely chosen by municipalities as a priority for their action plans. Why these dimensions were not commonly chosen is not clear from the interviews conducted for this evaluation. The two sub-dimensions that changed the least were “Promotion of teamwork” and “Conflict resolution.” Importantly, the majority of municipal management teams and participants interviewed for this evaluation requested future leadership training on these two topics. The following table displays overall information on the sub-dimensions and changes broken down by program phase. Percent change in 16 climate sub-dimensions over life of program 0% 5% 10% 15% 20% 25% Management Encouragement of excellence Promotion of teamw ork Conflict resolution Personal fulfillment Recognition of contributions Responsibility Adequate w ork conditions Dedication to w ork Stew ardship of property Compensation Equity Commitment to productivity Harmonization of interests Exchange of information Involvement in change Leadership Motivation Reciprocity Participation Nicaragua Leadership Development Program Evaluation 33 Table 6. Overall change in climate sub-dimensions and percent change by program phase Climate Percent change Dimensions Climate Sub-dimensions Overall Baseline Overall Follow-up Phase 1 Phase 2 Phase 3 Management 3.09 3.40 4% 17% 8% Encouragement of excellence 3.58 3.84 1% 11% 9% Promotion of teamwork 3.65 3.79 0% 11% 1% Leadership Conflict resolution 2.94 3.06 1% 5% 6% Personal fulfillment 3.48 3.75 0% 11% 12% Recognition of contributions 2.02 2.49 23% 20% 27% Responsibility 2.75 3.06 7% 11% 17% Motivation Adequate working conditions 2.54 2.89 15% 7% 21% Dedication to work 3.27 3.54 8% 4% 13% Stewardship of institutional property 3.08 3.34 1% 8% 17% Compensation 2.61 2.83 0% 11% 13% Reciprocity Equity 2.67 2.88 2% 7% 15% Commitment to productivity 2.99 3.33 5% 15% 14% Harmonization of interests 2.55 2.84 2% 16% 19% Exchange of information 2.31 2.72 13% 20% 19% Participation Involvement in change 2.65 3.09 9% 19% 21% In the first phase, most of the change was concentrated in four sub-dimensions: “Involvement in change,” “Adequate working conditions,” “Exchange of information,” and “Recognition of contributions.” The latter two were commonly selected as priorities for the municipal action plans from this phase. In Phase 2, changes were spread across all sub-dimensions with particular emphasis on “Recognition of contributions” and the four Participation sub-dimensions. Again this is related to the priorities of the municipal action plans. Also in Phase 2, the “Management” sub￾dimension manifested a surprising level of change even though this was not commonly selected for action planning by the majority of municipalities. Nevertheless, interviews with members of municipal management teams revealed a greater sense of capability in terms of meeting objectives and managing their teams. In Phase 3, improvements in climate were spread more evenly across all sub-dimensions related to Participation, Motivation and Reciprocity, which could reflect the greater diversity in the priorities and content selected for the municipal action plans. Nicaragua Leadership Development Program Evaluation 34 3.4.6. Climate results by SILAIS The following graph displays available climate results by the SILAIS administrative units across the three program phases. The data are incomplete because in the second phase, only SILAIS Matagalpa applied the climate instrument, and in the third phase, data from the baseline climate assessment are not available from SILAIS Madriz. Among the SILAIS participating in Phase 1, Boaco experienced a significant improvement in climate (p<.0001), while Jinotega showed essentially no change, and climate levels in Matagalpa declined. In the third phase, a similar pattern is evident where Masaya improved significantly over baseline levels (p<.0001), whereas Estelí showed minor improvement, and Rivas declined. Matagalpa succeeded in improving climate levels by 15% during the second phase which was a significant change over the Phase 2 baseline (p<.0001), but this result was still below the original baseline level in Phase 1. The greatest amount of change was noted in two Motivation sub-dimensions (“Adequate working conditions” and “Recognition of contributions”) followed by all four Participation sub￾dimensions (“Involvement in change,” “Exchange of information,” “Harmonization of interests,” and “Commitment to productivity”). The least amount of change occurred in “Responsibility,” “Teamwork”, and “Personal Fulfillment” sub-dimensions. These patterns are similar to those seen at the municipal level. Climate levels in selected SILAIS: Phases 1, 2 and 3 0.00 1.00 2.00 3.00 4.00 5.00 Boaco Jinotega Matagalpa Matagalpa Esteli Masaya Rivas Phase 1 Phase 2 Phase 3 Pre Post Nicaragua Leadership Development Program Evaluation 35 3.5. Leadership in the context of health services 3.5.1. Perceived effect of the program on services The original design of the leadership development program in Nicaragua did not target improvements in health services. The objective of the process at the municipal level (and subsequently with the SILAIS) was to improve organizational climate. This stemmed from an initial leadership dialogue during which staff motivation was selected as the primary challenge the MOH wanted to address through leadership development. Nevertheless, at the municipal and SILAIS levels, participants usually identified the purpose of the leadership development program as two-fold: to improve the climate of the organization and contribute to improving the quality of services. And these staff tended to believe that the program has achieved both. Managers, supervisors and facilitators commonly associate the leadership development program with improvements in quality of care and coverage of the health programs prioritized by the MOH, particularly reproductive health of women and adolescents. For example, from the point of view of one SILAIS Director: “The most important achievement we have had is the reduction in perinatal mortality and this was achieved as a result of various actions. For example we managed to construct in record time a maternity waiting home where we involved the general public and MOH health workers, and we also had to sensitize municipal directors to use the waiting homes, and all of this has to do with the mix of the quality program (the Quality Assurance Project [QAP]) with organizational climate, the leadership of municipal directors and working together in teams.” At the municipal level, participants explained that the program had helped them improve the quality of care they provided, particularly regarding their behavior towards clients and their coordination with other health workers within the municipality. Some of the more relevant examples included: • “We are trying harder to achieve our targets… now there is better communication between extension workers, community leaders and health workers… they come and report any new pregnant woman in the community and we go out to find her.” • “Within the referral system communication has improved… patients are taken [from the health post] to the health center and before they only transported the patients and we couldn’t give them follow-up because they didn’t send back the counter-referral slip and now they are sending the counter-referral slip so we know what happened to the patient.” • “We try to understand the patient… to listen in order to give better care… we have improved in this…now we always give more time to the patient whenever we don’t have too high of a demand.” • “Our relationships with our patients have improved because one thing is we come to the health unit and we try to leave our personal problems at home.” Nicaragua Leadership Development Program Evaluation 36 • “We have seen changes in coverage because by providing better quality of care we now have more clients who are seeking the care we offer…people are coming more, the treatment of patients by health personnel has changed, child health visits have increased, medical consultations have increased, more women are coming in – before they did not come… it is always full here.” • “We’re trying harder now because we know that they [municipal level] are going to come and measure us against the [service] indicators and so we are looking for ways to improve the system…they compare us to other health units to stimulate competition.” 3.5.2. Observations on service data There is a great deal of interest in investigating the link between climate levels and health services in the context of M&L’s leadership development programs. The scope of work for this evaluation proposed analyzing the relationship between the coverage and use of selected health services and organizational climate levels at the municipal level. However there are problems with this analysis given the design of the leadership development program and the type of data available. The Nicaragua leadership development program did not intend to affect the coverage or use of health services in any direct way. From the outset, the outcome measure for this program was improved organizational climate at the municipal level. As a result the program was designed to address the root causes of poor climate. Because of this design, there is no logical or programmatic link improving organizational climate and improving health service outcomes. To evaluate any associations between climate and services at this point would be an unconventional retrospective analysis. Furthermore, the available data on health services are insufficient to perform any analysis of relationships between the leadership development program and health service outcomes. Data on health services come from the MOH health services database (SIGPRO01) which are subject to all the caveats associated with service statistics. For example, the behavior of the indicators, in particular the fluctuations in coverage rates, is partially an artifact of the data and a function of how they were collected and aggregated in the database rather than a reflection of true variation in coverage. A simple analysis of the relationship between the leadership development program and health services using the available data would be methodologically flawed. Associating a particular impact or outcome indicator in a particular place and time period with a program intervention implemented in the same place and time would be difficult to defend. A more appropriate way to answer this question would be a carefully designed prospective study, preferably an operations research design. In this context, the service delivery indicators could be chosen and tracked from the beginning of the program in coordination with participating municipalities. This would serve two purposes: a) to analyze the root causes of poor services and then design the program interventions to address these, and b) to respond to municipal priorities for improving health services rather than measuring a standard set of indicators across all municipalities. As a result, contrary to the proposed scope of work, no data on health services are presented in this report. Nicaragua Leadership Development Program Evaluation 37 3.6. Sustainability of leadership processes and capacities 3.6.1. Continuity of the leadership program As part of its strategy for scale up and institutionalization, the leadership development program has achieved several key elements that will contribute to its continuation and progress towards sustainability: • Leadership modules, complete with facilitator and participant materials, have been published and adopted by the MOH • The climate instrument has been accepted by the MOH and the climate score is included as a leadership indicator in the newly developed guide for monitoring health centers and health posts; the biannual application of the climate instrument will soon be a national norm in order to comply with this indicator • A broad base of facilitators have been trained and are capable of replicating the learning materials • The use of the in-service (self-learning) training approach for widespread and rapid scale up • There is buy-in and support from key central level officials • The leadership development program is a response to a felt need on the part of the MOH One SILAIS Director explained the positive effects of having integrated the leadership development program with the quality program (the QAP project in Nicaragua): “the entire leadership program has been developed in conjunction with the quality management program and the implementation of the FFSDP monitoring guides which is what has contributed to its success. The leaders now have in their hands the power to make decisions for change.” At the central level, there was consensus among participants interviewed that the leadership development program is indeed necessary to gradually improve the leading and managing capacity of the Ministry. One respondent alluded to the strategy of “planting a thousand seeds:” “If management staff used the tools they [MSH] gave us in their daily administration, the sum of those leaders working towards their objectives will have a large multiplier effect on the overall improvement of this institution.” But in order to guarantee continuity of the program at all levels, participants cited the need for improved follow-up to maintain the focus on developing and implementing action plans to address organizational challenges, monitor the outcomes as well as strengthen the team of facilitators at each level. Some respondents noted that the prospects of institutionalizing the program depend fundamentally on political will at the highest levels and the presence of someone who “is supporting this 100%”. However others note that “in the MOH we have so few economic resources and institutionalizing a program depends not only on will but also on funds.” They worry that the Ministry does not have the economic capacity to assume the program because “there is no budget for training, the reproduction of materials is always done through a [donor￾funded] project.” Despite these limitations, a key central level official asserted that “...we’re looking at how we can arrange to have the rest of the staff attend the leadership course because it helps the Nicaragua Leadership Development Program Evaluation 38 individual so much and also improves our services for the public.” This person added that the program is an “initiative that will continue through the [central-level] working groups/ commissions” now that MSH/Nicaragua has provided the principal tools. The cost of the leadership development program is another important indicator of its potential for institutionalization. The basic costs of program implementation and scale up were analyzed and documented by Kate Waldman during this evaluation. This analysis demonstrates the cost efficiency of the program, based on information from M&L’s Monthly Expenditure Reports and financial information provided by the MSH/Nicaragua office. The in-kind contributions of program participants from the Ministry of Health, SILAIS, and municipalities, such as their time, are not included in this analysis as the data were too difficult to collect. The analysis of costs is first broken down by program phase and then a more in-depth analysis of Phase 3 is provided for each participant level. The analysis of the data revealed the following: • The number of participants increased by 51% from Phase 1 to Phase 2. From Phase 2 to Phase 3, the number of participants increased by 344%. • The total cost of the first phase of the program (2001-2002) was $70,606. This was slightly higher than the cost of the second phase (2002-2003), which came to $54,023. The third phase (2003-2004), where the most expansion occurred, had a total cost of $339,506. • The cost per participant decreased by 51% from Phase 1 to Phase 2, but increased by 41% from Phase 2 to Phase 3. • The average cost per participant over the course of the three years of the leadership development program was $241. This amount includes all of the level of effort, travel/per diem, training costs (meeting space rental, food/lodging for participants, and materials), and monitoring and mentoring costs associated with the program. Also included are appropriate health/sick/vacation of MSH staff, overhead, and the Allocable Cost Factor6 rates. The total cost of the program fluctuated from year to year, but the cost per participant decreased from the first phase to the second and increased from the second phase to the third. This increase in costs can be attributed to the tremendous increase in number of participants during Phase 3, which required hiring additional staff by MSH and conducting monitoring of the replication of the program. Phase 3 is a good example of future program costs to MSH, in that it involved training at all levels and monitoring of the program. The data show that Central level training and SILAIS and municipal level training are the most costly, respectively accounting for 35% and 40% of the total expenses. Municipal level participant training accounts for a much smaller percentage (11%) of the total due to the fact that MSH facilitators are not used. While costs will continue to 6 Allocable Cost Factor (ACF): This percentage covers certain project costs that benefit the entire M&L Program, including such expenses as rent and utilities, recruitment, general equipment and office expenses, as well dedicated M&L support staff including finance officers and contracts officers. Individual M&L projects receive support in activities ranging from budgeting to monthly monitoring of expenditures, producing financial and other reports, required USAID reporting, contracting and procuring services. Nicaragua Leadership Development Program Evaluation 39 fluctuate based on the number of future participants and the level of training, it can be expected that the model set forth in Phase 3 is an accurate way to anticipate future costs. Furthermore, as the program continues to spread and individual SILAIS or municipalities start to take on some of the program costs, the costs incurred by MSH will likely decrease. The complete report on the cost analysis of the leadership development program is provided in Appendix 7. 3.6.2. Maintenance of climate levels over time Data from the subsequent reapplications of the climate tool (after the follow-up application) are not yet available from most municipalities because they have only just begun to systematically apply the PAHO climate tool. However, to date at least 12 municipalities (11 from Matagalpa and one from Boaco) from Phase 1 and Phase 2 of the leadership development program have applied the climate tool a third time. San Lorenzo reapplied the tool in October 2003 and the municipalities in Matagalpa reapplied it in February-March 2004. Although the sample is small, it provides a glimpse of the trends related to the maintenance of climate levels over time. Table 7. Overall climate levels over time in 12 municipalities from Phase 1 and Phase 2 n Mean score Range Time 1 12 3.040 2.55 - 3.46 Time 2 12 3.159 2.19 - 4.46 Time 3 12 3.093 2.16 - 4.42 At a broad level, the comparison of overall scores reveals a slight decline in climate levels between the second and third application of the climate tool; however time 3 levels still remained above baseline. The graph below presents these data by municipality. Overall climate scores over time: Phase 1 and 2 municipalities 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 Waslala Rio Blanco Rancho Grande San Lorenzo Matiguas Esquipulas Dario San Isidrio San Dionisio San Ramon Terrabona Matagalpa Phase 1 Phase 2 Time 1 Time 2 Time 3 Nicaragua Leadership Development Program Evaluation 40 The above graph shows that the four Phase 1 municipalities tended to improve or maintain climate levels between the follow-up (post-program) application of the climate tool and the subsequent (third) reapplication. In contrast, the majority of Phase 2 municipalities tended to improve climate levels between baseline and follow-up applications and then did not maintain these gains, with declining levels in the third application. The two exceptions are Matagalpa which showed a steady improvement over time between baseline and the third application, and San Dionisio which declined between baseline and follow-up and then improved in the third application (yet the results still did not improve over baseline levels). When reviewing these data, it is important to keep in mind that the time frames between the baseline, follow-up, and third applications of the climate tool were different for Phase 1 and Phase 2. The third application of the climate tool in February-March 2004 occurred about 18 months after the end of Phase 1 (August 2002) and 7-8 months after the end of Phase 2 (June 2003). San Lorenzo (a Phase 1 municipality) reapplied the tool in October 2003, approximately 14 months after the end of the program. More data is necessary from other municipalities in order to adequately record the trends and draw conclusions regarding the maintenance of climate levels in the period following the leadership development program. 3.7. Observations on the internal reliability of the PAHO climate instrument The PAHO organizational climate instrument was developed in Latin America specifically for application within the context of a health system. It was field tested, but reliability and validity information on the instrument are not available. Therefore the scope of work for this evaluation proposed to test the reliability of the instrument using the available raw climate data from the program. (A validity study would require a separate methodology.) Using SPSS, an exploratory Factor Analysis with Varimax rotation was performed on both the baseline and follow-up data from municipalities in all phases of the program. The purpose of a factor analysis is to identify the underlying structure of the data and determine which variables in the instrument cluster. The factor analysis resulted in 22 factors with an eigenvalue greater than one representing 62% of the total variance. The first identified factor is the strongest and accounts for the majority of the variance. There appears to be a strong relationship between the 80 items in the instrument and this first strong factor, which may indeed represent organizational climate. However, there is little discernable relationship between the items of each sub-dimension in the instrument and the remaining 21 factors. From these data it is unclear whether the instrument measures the climate sub-dimensions or dimensions defined by the instrument. The absence of clear results is most likely due to a particular nesting of the data according to the subgroups surveyed (i.e., the municipalities) which cannot be handled by classic factor analysis or by SPSS software. A multi-level factor analysis is a more appropriate approach and will be necessary to adequately explore the clustering of the data and reliability of the instrument. This analysis will be performed in the M&L Program’s PY5 and will be available from the M&L Monitoring and Evaluation Unit. Since SPSS cannot perform multi-level modeling, the analysis is pending the acquisition of new statistical software (MPlus) by the M&L M&E unit and the completion of a training session in June that will provide the skills to perform the multi-level analysis. Nicaragua Leadership Development Program Evaluation 41 Meanwhile, a number of observations on the face validity of the instrument were obtained through interviews and focus groups with facilitators and participants. Throughout the program, the MSH/Nicaragua team made several modifications to the climate tool between the first and second phases of the program, particularly regarding the phrasing of certain items and calculation errors in the Excel data entry sheet. After the second phase they made additional changes to two items that still appear problematic. Based on interviews with municipal level participants from all phases of the program, it appears that the tool still lacks full face validity. The great majority, including third phase participants, expressed difficulty interpreting the items and felt that some did not fit their reality at the local level. For example one participant explained that “some questions were clear and others very confusing… you just had to guess.” As a result, the instrument could potentially be subject to misinterpretation or manipulation by respondents, either intentionally or not. The most common problems mentioned included: • The length of the instrument causes fatigue among respondents – some choose to complete it randomly. • Misuse of the instrument to “get back at my supervisor by answering ‘No’ to every question.” • Uncertainty over whether they were evaluating the municipal director, their immediate supervisor, or the organization in general. • Misunderstanding of items that led respondents to request help from their supervisor to interpret the questions which could have biased the outcome. • Ambiguity of certain items in the instrument leading to multiple interpretations. Finally, the particular application of the climate instrument may have introduced a sampling bias: a minimum sample of 19 (sometimes greater) was used in each municipality but the universe for the sample differed. Some took a random sample of 19 from within the expanded technical committee only, others took a random sample of 19 from all health workers in the entire municipality, and still others took a random sample of 19 from all workers (including guards and cleaning staff) in the municipality. The potential of a sampling bias could be avoided by providing better instructions to municipal and SILAIS staff who are responsible for applying the tool. The difficulties in the interpretation and application of the climate tool call into question the reliability of the climate data collected by this program, at least during the first phase. However, it is very likely that results from the second and third application of the tool are closer to the true measure of climate because by this time the MSH/Nicaragua and MOH facilitators were more familiar with the instrument and better equipped to explain its use to respondents and clarify the meaning of the items. Some respondents were randomly sampled again during second or third time application and as a result were more familiar with the items in the tool. Thus the problems noted in face validity are likely to diminish with time as the instrument becomes a part of routine municipal-level monitoring and as respondents become more familiar with the purpose of the tool and how to interpret the individual items. Nicaragua Leadership Development Program Evaluation 42 4. Conclusions After three years of implementation, the Nicaragua leadership development program has succeeded in training nearly 2,000 managers and health workers at the municipal, SILAIS, and central levels in a process of leadership development. The program employed the already tested MOH methodology of in-service training for health staff using a self-learning approach which was delivered through a network of facilitators developed at each level of the system. This approach, and the high quality materials provided, was key to the rapid scale-up and ownership of the program by the MOH. The project design was developed during the first year of implementation and improved each subsequent year incorporating the learning and experience of the project team. This demonstrates the programmatic agility of the project team and their responsiveness to the needs of the MOH. Municipal and SILAIS levels: The leadership development program achieved its main expected outcome: improved organizational climate at the municipal and SILAIS levels. An analysis of municipal climate scores suggest that while the impact of the leadership training and follow-up activities on organizational climate at the municipal level was minimal in the first phase, the program had a greater and measurable effect in the second and third phases. This is a logical outcome given that the first phase served mainly as a pilot to develop and perfect the program materials and process which were then successfully carried out in the second and third phases. The review of climate dimensions and sub-dimensions further support these conclusions. The data show that the program succeeded in its objectives: the municipalities prioritized the weakest of the climate sub-dimensions in their action plans and succeeded in improving these areas over others that needed perhaps less attention. The leadership development program also resulted in notable changes in behavior and improved use of leadership competencies such as communication between supervisors and subordinates across both municipal and SILAIS levels. Health workers also noted a greater proclivity to work together and to put more effort into their jobs. These improvements have generally persisted in the period following the program among Phase 1 and Phase 2 municipalities; even though in many cases the processes or activities defined in the municipal action plans are no longer being implemented or were never implemented. This points to the sustainability of the adoption and integration of the leadership practices and competencies provided by the program. However, whether or not the persistence of these leadership behaviors translates into the maintenance of climate levels in the period following the program is not clear from the available data. Third generation climate data is needed from additional municipalities in order to adequately assess the longer-term trends in climate scores. Nevertheless, despite the fact that developing and using communication channels was a commonly selected leadership challenge and an important focus of the leadership process at all levels, there is still a limited flow of information outside the top management levels. In the first and second phases, primarily members of the management teams and the broader technical committees were included in the replication of the learning units by the local facilitators. Information sharing did indeed improve within these groups while they were part of the process. Nicaragua Leadership Development Program Evaluation 43 But each phase has been characterized by the limited dissemination of progress and results to the rest of municipal or SILAIS staff, particularly in the period after the replications ended. Even in the third phase where all municipal staff was included in the replications, information tended to remain within the management teams. Similarly, the majority of municipal and SILAIS staff did not participate in designing or implementing the action plan. It was most often designed by the facilitators and some members of the management team for the municipality or SILAIS during the training of facilitators. Activities included in the action plan were generally areas that the management team needed to do or change. Input from staff was sometimes solicited but the plan primarily belonged to the management staff. Likewise, the facilitators or members of the management teams were mainly responsible for implementation of the action plans. While the plans were often not fully implemented (and in some cases not implemented at all) they nevertheless tended to help the municipal teams focus on the challenge on hand: improving the particular sub-dimension of climate that was weakest for that municipality. In addition, while most municipal plans included fairly generic or ordinary activities, what seemed important in terms of affecting climate was the new way these activities were carried out as a result of the leadership program. Members of the management team tended to solicit greater input from municipal staff in decisions affecting the municipality and activities that often had been haphazard in the past (especially staff meetings) tended to become more regular and participatory. According to the M&L approach to leadership development, a team or workgroup follows a particular process to work together on identifying and addressing a challenge which tends to improve the climate in that particular team or workgroup. In Nicaragua, for the purposes of the leadership intervention, the team was defined as the management team, whereas for the purposes of measuring climate, the team was defined as the entire staff of the municipality rather than the management team, which is where one would expect to see results according to the logic of the M&L approach. It is possible that changes in climate were diluted when measured at the municipal level and would be stronger if measured solely within the management team that participated in the intervention. The climate results show that climate levels generally improved in the absence of broad staff participation (i.e. municipal or SILAIS staff) in the efforts to define and address the challenge. This task was mainly limited to the management teams at the both municipal and SILAIS levels. While municipal and SILAIS staff did participate in activities programmed in the action plan, they did not own the plan or its results. They generally participated only when specific activities were organized for them by the management teams. And when asked about the results related to the plan, staff most often pointed to behavioral or attitudinal changes of their supervisors which were not necessarily programmed in an action plan. This points to the important influence of the manager or supervisor on the climate levels in a given group (in this case the municipality). While the program succeeded in achieving broad coverage, it was difficult to provide sufficient follow-up and monitoring of the replication activities in the municipalities and the efforts to implement the action plans, particularly after the training of facilitators had ended. On the whole, the SILAIS facilitators did not assume this responsibility either. The lack of systematic follow-up and monitoring during and after the replications is likely related to the varied progress Nicaragua Leadership Development Program Evaluation 44 in carrying out municipal action plans and the limited continuation of leadership activities by the municipalities once the replications ended. If leadership development is to become a sustainable process within the MOH, one obstacle that will need to be overcome is the vision of leadership as a program versus a process. Both health workers and management staff tended to identify leadership as a series of trainings or replication workshops tied to a program. They considered the program to have ended once the replications were completed. And for these individuals, once the program ended the leadership processes were no longer sustainable. Thus for many, leadership has not yet become a continual process or routine way of approaching the challenges on the job. As a result, in order to sustain the advances made in leadership, they mainly see the need for more training. It is clear that leadership development has not yet been recognized as an ongoing process such that it can continue without external technical and financial support. Nevertheless, there are several important factors that serve as positive indicators for the continuation and sustainability of the leadership development program. These include: − Broad acceptance and ownership of the program by the MOH − Incorporation of the climate assessment as an indicator in the FFSDP monitoring process − Integration of leadership practices and competencies with the quality program in several SILAIS as a more comprehensive approach to improving services − Low cost in-service training approach with high quality leadership module adapted to the local level − Cadre of trained facilitators at all levels − Successful results shown to date in terms of improving climate levels at the municipal level In terms of the costs of the leadership program and its scale-up, an analysis of the project’s financial data revealed the cost per participant decreased from the first phase to the second and increased substantially from the second to the third phase. This increase in costs can be attributed to the tremendous increase in number of participants during Phase 3, which required hiring additional staff by MSH/Nicaragua and monitoring of the replication of the program. As the program continues to spread and individual SILAIS or municipalities start to take on some of the program costs, it is likely that the costs incurred by MSH will decrease. In spite of this, there will be a minimum cost associated with the maintenance, monitoring and evaluation of the program. This minimum cost can more accurately be determined as the program continues, but will likely remain under $250 per participant if the model used in Phase 3 is followed. In order to minimize program costs, the replication of the program by SILAIS and municipal facilitators is the best way to keep costs low. Central level: Compared to the municipal level, the leadership development program at the central level is quite young and yet has achieved a great deal in a short time. Although the program is currently ongoing, participants have already made important progress in addressing the challenge of Nicaragua Leadership Development Program Evaluation 45 aligning the National Health Plan with the Health Care Model. The program has also resulted in improved working relationships and coordination between divisions/departments and new lines of communication within programs/departments. Although these changes may not yet be widespread due to the limited participation in the program, any change at the central level is a success given the established paradigms of paternalism and deep-seated resistance to change. It is important to note that there is broad acceptance of the program at the central level which extends all the way to the Minister as well as a desire to continue the processes they have started even if they are on their own, that is, when M&L support and involvement ends. Achieving the central level program objective (linking the National Health Plan with the Health Care Model) will likely have widespread implications within the health system. The National Health Plan defines the vision, objectives and strategies of the health sector whereas the Health Care Model establishes the operational plans and identifies the different types and organization of services needed to achieve the objectives in the national plan. Therefore alignment of the Plan and Model will allow the MOH to better meet its service goals and the needs of the population. In summary, the design and delivery of the leadership development program in Nicaragua played upon the strengths of the health system – a well organized administrative system with skilled staff at different levels capable of replicating the process within their respective units. The model used in the program is the right one for the Nicaraguan context. The coverage of the program was broad and of high quality, and as a result, a large number of MOH staff is now primed to take the leadership process to the next level of addressing health service outcomes. To the credit of the MSH/Nicaragua team, they have already designed a new leadership module for the next phase of leadership development which focuses on improving health services through management and leadership development. 5. Recommendations For leadership development at the Municipal and SILAIS levels: ♦ MSH should continue supporting the MOH to institutionalize the leadership development program in order to guarantee the continuity of the program. As a parallel strategy, the MOH could consider incorporating leadership development into the continuing education sessions at the municipal level (as some individual municipalities have already begun to do on their own) and ensure that staff who did not participate in the leadership development program is involved in these activities. ♦ The MOH should extend the leadership training to the remainder of staff who have not participated, especially those from the first and second phase municipalities, and at the same time offer refresher training to those who have already participated. Likewise, it should encourage broader and more effective participation of staff outside the management team in the development and implementation of action plans. ♦ The MOH should provide more systematic support and follow-up in the implementation of action plans and replication of the learning units, and continue to provide support after the replications end to ensure continuation of processes. If the municipal action plans are considered to be an important piece of the leadership program, greater attention and follow- Nicaragua Leadership Development Program Evaluation 46 up should be provided to support their implementation. The SILAIS and municipal health authorities will need to allocate sufficient funds and personnel to ensure effective and systematic follow-up. ♦ MSH and the MOH should ensure that future applications of the leadership development program at the municipal level that address service delivery challenges and outcomes also include adequate measures of these outcomes. ♦ MSH and the MOH should revisit the climate instrument for usability and suitability at the local levels. The analysis of internal reliability of the tool should be finalized. Consideration should also be given to incorporating the validation of the tool as part of the current leadership development program in the Managua SILAIS. ♦ MSH and the MOH should continue collecting climate data from participating municipalities in order to investigate the maintenance of climate levels over time following the conclusion of the leadership training. ♦ MSH should consider ways of involving municipal and SILAIS participants in LeaderNet7 that do not depend on internet access, in order to provide ongoing support and motivation to participants once their training has ended. This is especially important given the large numbers of participants trained and the lack of ongoing follow-up provided to date. ♦ MSH should design an operations research study for any future phases of the municipal level program in order to measure associations between the leadership intervention, climate outcomes and impact in terms of health services. ♦ MSH should develop indicators to measure leadership competencies that complement the M&L leadership process indicators (which measure leadership practices) as well as the measurement of climate. For the MOH regarding continuity at the Central level: ♦ Extend the leadership training to a greater number of central level managers and staff, emphasizing the participation of more staff members from the different divisions/ departments of the central Ministry. ♦ Provide ongoing follow-up and monitoring of the central level leadership processes to ensure the completion of the current action plan. ♦ Carry out systematic and routine applications of the climate instrument within each division / department of the central Ministry. 7 LeaderNet is a USAID-supported virtual community of practice made up of graduates of M&L leadership development programs. LeaderNet was initially launched in Latin America and is now expanding to other parts of the world to support M&L leadership alumni. Nicaragua Leadership Development Program Evaluation 47 Appendix 1: Scope of Work for the Evaluation Management and Leadership Project (M&L) Management Sciences for Health (MSH) In-depth Evaluation Study Protocol Nicaragua Leadership Development Program 2001 – 2004 Purpose The purpose of this in-depth evaluation is to document the approach of the Nicaragua Leadership Development Program and assess the outcomes associated with organizational climate, service delivery improvements, and sustainability of leadership capacity following the completion of three years of leadership development with the Ministry of Health (MOH). This study will build on the findings from an evaluation of immediate outcomes conducted in the fall of 2002 after the first year of the leadership development program in Nicaragua and a systematization of results conducted in May – June 2003 following the program’s second year. It is part of a series of M&L evaluations on the subject of Developing Managers Who Lead which respond to a common set of key questions intended to provide substantive learning for the M&L program in Nicaragua and M&L’s wider knowledge management activities. Background Several years ago the Ministry of Health (MOH) of Nicaragua entered a process of modernization and decentralization. This has required the development of leadership and management capacity at all levels of the organization in order to address the many challenges provoked by external and internal pressures such as economic crisis, political change, growing demands and needs of clients as well as bureaucratic lethargy and increasing apathy among health personnel. As one response to this need, M&L and the PROSALUD project, together with the MOH, launched a leadership development program in 2001 to strengthen the capacity of health managers and personnel in 63 municipalities, 7 SILAIS and the central level over a period of three years. The leadership program was implemented in three phases (see table below). Following a leadership dialogue in February 2001, the first phase of the program took place from July 2001 to June 2002 in 13 municipalities in the SILAIS of Matagalpa, Jinotega and Boaco. Twelve of these were prioritized municipalities where PROSALUD had been implementing the Fully Functioning Service Delivery Point (FFSDP) strategy since 1999. The second phase was carried out from November 2002 to June 2003 in the remaining 16 municipalities of the same three SILAIS. The third phase began in July 2003 in 34 municipalities of four additional SILAIS (Masaya, Madriz, Estelí and Rivas) plus the management teams of all seven SILAIS, and 56 key managers from central level directorates and departments.8 This third phase was conducted as 8 Beginning in January 2004, the third phase will expand to include a leadership development program for the Managua SILAIS and the Ministry of the Family, but this component will not be included in the in-depth evaluation. Nicaragua Leadership Development Program Evaluation 48 part of the larger multi-component Leadership and Management for Health Project that began in April 2003 and builds on several years of service delivery strengthening interventions in Nicaragua conducted with the MSH PROSALUD project. While the work that MSH and M&L have conducted in Nicaragua has included both management and leadership interventions, this study will focus solely on the leadership development program. Participants and Phases of the Nicaragua Leadership Development Program Number of municipalities per phase SILAIS First Phase 2001-2002 Second Phase 2002-2003 Third Phase 2003-2004 Total Matagalpa 6 9 - 15 Jinotega 5 3 - 8 Boaco 2 4 - 6 Masaya - - 9 9 Madriz - - 9 9 Estelí - - 6 6 Rivas - - 10 10 Total 13 16 34 63 Number of participants per phase Facilitators (municipal and SILAIS) 32 40 129 201 Participants (municipal and SILAIS) 183 284 1245 1712 Central level participants - - 65 65 Total 215 324 1439 1978 The objectives of the leadership development program evolved over the course of the three year the program and in the final year included the following: ¾ Improve organizational climate in the participating municipalities and SILAIS ¾ Develop leadership capacities among the municipal and SILAIS management teams ¾ Develop and publish the MOH Leadership Module consisting of self-instructional units intended for implementation at the municipal level The leadership development program at the municipal level consisted of the following programmatic components: 1. Baseline and follow-up studies of organizational climate in all participating municipalities and SILAIS. 2. Delivery of six self-instructional units for managers and staff in the participating municipalities. 3. Broad replication of the six units to the remaining staff in these municipalities. Nicaragua Leadership Development Program Evaluation 49 4. Implementation of an improvement plan resulting from the analysis of organizational climate data. 5. Technical assistance and mentoring by PROSALUD/M&L staff in the application of concepts learned and the implementation of the improvement plan. The following components made up the program at the central level and SILAIS levels: 1. Leadership Dialogue to define challenges facing the MOH 2. Four 2-day leadership development workshops for central and SILAIS staff 3. Final evaluation workshop 4. Coaching and follow-up An evaluation of the first phase of the leadership development program was conducted in the fall of 2002 to assess the organizational climate outcomes prior to the beginning of the second phase of the program. Following completion of the second phase, a systematization of results from both the first and second phases was conducted in May-June 2003. While this current study is part of the series of in-depth evaluations on the topic of Developing Managers Who Lead, it will also serve as a final evaluation of the third phase of the Nicaragua leadership development program. Statement of Work The purpose of this evaluation is to assess the organizational performance outcomes among teams that have participated in the Nicaragua Leadership Development Program, provide recommendations that will be used to improve subsequent course delivery and follow-up, and contribute to a larger comparative study of M&L leadership interventions. This evaluation will take place immediately following the completion of the third phase of the leadership program in March/April 2004 which is approximately 9 months following the conclusion of the second phase and 19 months after the completion of the first phase. The objectives of the evaluation are to: • Document the content and approach used by the leadership development program in Nicaragua as well as the intended and unintended results achieved by each phase of the program. • Assess the relationship between program inputs, implementation of improvement plans, organizational climate results and the performance of health services (in terms of coverage and use of services). Identify elements of the program content, delivery and follow-up support that are associated with the results achieved. • Assess the extent to which leadership practices and processes, organizational climate levels and health service performance have been sustained in the post-intervention period for the first and second program phases. Nicaragua Leadership Development Program Evaluation 50 • Document the scale-up of the program, its costs, and the degree of institutionalization achieved according to selected indicators. • Analyze the internal reliability of the PAHO organizational climate tool used by the Nicaragua leadership development program. Methods Both quantitative and qualitative methodologies will be used in this evaluation based on information from the following sources: 1. Review of project documents regarding the program design and logical pathways between input-output-outcome, learning modules, training plans, monitoring reports, evaluation and systematization reports. 2. Review of the improvement plans developed by the municipal, SILAIS and Central level participants according to the following quality criteria: o Goals/objectives o Activities logically related to goals o Measurable indicators o Timeline or timeframe for implementation o Resources 3. Review of the results achieved through the implementation of the improvement plans at the municipal, SILAIS and central levels. 4. Analysis of the relationship between organizational climate and performance of services, as follows: o Review of pre- and post-intervention climate data for each program phase as well as methods used to apply the climate tools (sample size selection, application of instrument, tabulation of data) o Analysis of data from the reapplication of the PAHO organizational climate tool within municipalities from the first and second phases to measure the maintenance of organizational climate levels in the post intervention period o Review of selected pre- and post-intervention service results and analysis of the relationship between these results and organizational climate for the first and second program phases o Comparison of pre- and post-intervention service statistics from intervention areas and a suitable comparison area during a similar time period (this analysis will be limited to municipalities from Phase 1 and 2 only due to the availability of service data). This analysis will use selected indicators measuring the use and coverage of family planning and MCH services, as follows: Nicaragua Leadership Development Program Evaluation 51 5. Individual interviews and focus groups with participants and facilitators from each level of the MOH and individual interviews with members of the MSH/Nicaragua team and other key informants, as follows: Municipal level − Focus groups with participating and non-participating staff in selected municipalities − Individual interviews with municipal facilitators (including the municipal director and selected members of the municipal management team) SILAIS level − Individual interviews with selected departmental (SILAIS) facilitators and participating staff Central level − Individual interviews with selected participants and key informants MSH team − Individual interviews with the leadership development program manager, program coordinators and facilitators from MSH/Nicaragua The individual interviews and focus groups will be structured around themes from the key questions for the in-depth evaluations on the topic of Developing Managers Who Lead (below). The individual and group interviews will be conducted by one member of the M&L Monitoring and Evaluation Unit and two local consultants from Nicaragua. All interviews will be taped for transcription. Phase 1: 12 Municipalities in Matagalpa Jinotega Boaco Phase 2 municipalities serve as control area for comparison of service data with Phase 1 municipalities for this period Phase 2: 16 Municipalities in Matagalpa Jinotega Boaco Phase 3 municipalities serve as control area for comparison of service data with Phase 2 municipalities for this period Phase 3: 34 Municipalities in Masaya Rivas Esteli Madriz 6/2001 6/2002 6/2003 6/2004 Nicaragua Leadership Development Program Evaluation 52 The key questions for the evaluation include: 1. What TA approaches and tools were used by M&L? a. What was the content of the leadership intervention? b. How was M&L framework applied? c. Were teams formed to address the challenge and if so, how? d. What process was used for program delivery and follow-up support? 2. To what extent has the MOH achieved its performance objectives and intended results? a. What is the availability/existence of the action plans at all levels of the MOH? b. How and why did participants prioritize the goal/desired performance – what was the selection based on? c. What process was used to prepare the action plan – how were the activities selected to address the challenge and lead to achieving the desired performance? d. Were all activities implemented? Were other activities implemented that were not included in the action plan? e. What means does the MOH have for monitoring their progress? f. Did the implementation of action plans lead to the expected results? g. What other results did the MOH achieve that are unrelated to their action plan? 3. What elements of M&L interventions are associated with organizational performance outcomes (improved management systems and/or improved work climate and/or increased ability to respond to changing environments)? a. Are improvements in health services likely associated with program inputs or an historical trend? How do the service results of the intervention group compare with those of a similar (comparison) group at a similar time? b. What are the logical pathways between the program inputs – outputs – outcomes? c. Did participants work together to address their challenge during the program? What process and skills did the team use? What motivated participants to achieve their results? What prevented them from achieving their goals? d. Did participants continue to work together to address another challenge after the program ended? What processes were used then? Was this similar to or different from how they worked together during the program? How did they motivate the participation or commitment of participants after the program ended? e. How did follow-up support affect the implementation of the action plans? Did participants receive follow-up support or TA related to addressing their challenge from any source other than M&L? 4. Are the measurable changes in organizational performance sustainable? a. Were climate levels and team building processes maintained and/or improved in the period post-program? b. Were service results maintained and/or increased in the period post-program? Nicaragua Leadership Development Program Evaluation 53 c. To what extent was the program replicated after the program ended? d. To what extent and how was the program scaled up to multiple levels of the health system, and at what cost? Composition of the study team The study team consists of one member of the Monitoring and Evaluation Unit and two local consultants from Nicaragua. Team members must have the following skills: • Knowledge of management, leadership and organizational climate • Interview skills • Analysis of qualitative data • Perceptive listening • Sensitivity to client needs • Cultural experience/understanding • Effective writing and communication In addition to the study team, a number of M&L staff will have a role in this evaluation, specifically with regard to reviewing the study design, questionnaires and interview guides and final report. Specific roles and responsibilities of the study team and other M&L staff are found in the calendar of activities below. Deliverables Final evaluation report Database of organizational climate from each program phase Database of service statistics used for evaluation Short video capturing participants’ words and anecdotes (specific content TBD) Nicaragua Leadership Development Program Evaluation 54 Proposed Calendar of Activities Activity Dates Person Responsible Resource persons Draft protocol for in-depth evaluation Dec. 8 – 30, 2003 NL, AE SJ, TA, JG, MSH/Nic team Collection and review of background materials; selection of participants for individual interviews; development of interview guides; identification and contracting of local consultants; preliminary analysis of climate and performance data Feb. 1 – 30, 2004 NL SJ, MSH/Nic team, GR Data collection in Nicaragua; transcription and translation of interview notes March 8 – 26, 2004 NL, KW, local consultants MSH/Nic team Analysis of key interview data with MSH/Nicaragua team March 15 – 18, 2004 NL, KW, local consultants MSH/Nic team Final analysis of all quantitative and qualitative data March 29 – April 9, 2004 NL CP, GR Draft evaluation report (in English) April 12 – May 7, 2004 NL CP Review of draft report May 10 – 21, 2004 AE, SJ, CP, BS, TA, JD, CM, ET AO Revisions to draft report May 24 – 28, 2004 NL AE, SJ, CP Final report distributed May 28, 2004 AE AO Final report translated into Spanish and distributed to MSH Nicaragua team June 4, 2004 AO NL Legend: TA = Tim Allen JD = Joseph Dwyer AE = Alison Ellis JG = Joan Galer SJ = Sarah Johnson NL = Nancy LeMay CM = Claritza Morales AO = Amber Oberc CP = Cary Perry GR = Greg Rodway BS = Barry Smith ET = Eduardo de Trinidad KW = Kate Waldman Nicaragua Leadership Development Program Evaluation 55 LOE Estimates for In-depth Evaluation* Activity NV AE CP SJ TA JD JG GR AO Evaluation Design and SOW 5 .5 .5 1 .5 .25 - - Collection and review of background materials; selection participants for individual interviews; development of interview guides; preliminary analysis of climate and service data 10 1 2 .5 - - - 2 - Interviews with program participants and MSH team 12 - - .5 - - - - - Transcription and translation of interview notes and preliminary analysis of interview data with MSH team 5 - - - - - - - .5 Final analysis of evaluation data 8 1 2 - - - - 2 - Draft report 10 1 1 - - - - - 2 Review of draft report - .5 .5 .5 .5 .5 .5 .5 .5 Revisions to draft report 3 1 1 - - - - - .5 Final report distributed - .25 - - - - - - - Total LOE 53 5.25 7 2.5 1 .5 .5 4.5 3.5 Legend: TA = Tim Allen JD = Joseph Dwyer AE = Alison Ellis JG = Joan Galer SJ = Sarah Johnson NL = Nancy LeMay AO = Amber Oberc CP = Cary Perry GR = Greg Rodway *The Nicaragua Leadership and Management for Health project will support the LOE costs for the following individuals: Barry Smith Kate Waldman Claritza Morales Eduardo de Trinidad Carla Rodriguez Nicaragua Leadership Development Program Evaluation 56 Appendix 2: Interview Guides Evaluación del Programa de Desarrollo de Liderazgo Management Sciences for Health Guía de Entrevista para el Grupo Focal Introducción explicando el objetivo del estudio y el propósito del grupo focal. Se trata de identificar -- del punto de vista del personal de salud -- lo que pasó durante el proceso de desarrollar el programa de liderazgo y reflexionar sobre los avances del programa así como los mejoramientos indicados para el futuro. Las perspectivas y opiniones de los participantes ayudarán a fortalecer el programa en el futuro. Hay que enfatizar que las respuestas y los comentarios de los participantes son absolutamente confidenciales y anónimos. ******** 1. Perfil de los participantes • Introducción de cada persona incluyendo: su cargo actual, el número de años que ha trabajado en este cargo, y el número de años en total que lleva con el MINSA. 2. El proceso de desarrollar el programa • ¿Cómo se desarrolló el programa de liderazgo en este municipio? ¿En sus opiniones, cuál es el propósito de este programa? ¿Ya participaron ustedes en la replica de los módulos de liderazgo? ¿Todos sus colegas han participado en las replicas también? 3. El plan de mejora • ¿Como parte del programa de liderazgo, se desarrolló un plan de mejora? ¿Cómo fue el proceso para desarrollarlo y quien participó en su elaboración? • ¿En sus opiniones, cual es el problema o objetivo que están abordando a través del plan? ¿Cómo se seleccionó este problema entre otros? ¿Quien lo seleccionó? ¿Qué tipo de información y de que fuentes se utilizó para analizar el problema? • ¿Cuándo comenzaron a implementar el plan? ¿Ya se implementaron las actividades del plan de mejora? ¿Cómo figuran ustedes en las actividades del plan -- cuál es su papel en este plan? ¿Se formaron equipos para implementar el plan? • ¿Recibieron seguimiento o apoyo durante el programa? ¿Por parte de quien? ¿Y durante la implementación del plan? ¿Por parte de quien? Nicaragua Leadership Development Program Evaluation 57 • ¿Tienen los recursos necesarios – humanos y otros -- para cumplir el reto? ¿Si sí, como aseguraron los recursos necesarios? ¿Si no, cuales acciones han tomados para enfrentar el reto de toda manera? • ¿Cuentan con el apoyo de los jefes/supervisores para la implementación del plan? ¿Cuál ha sido el nivel de cooperación y colaboración entre sus colegas para su implementación? • ¿Cuales son los resultados hasta la fecha relacionados con el plan? ¿Qué otros resultados no relacionados a su plan logró el municipio? ¿Cómo se esta monitoreando los avances en el plan y por quien? • ¿El plan de mejora ha influido en el ambiente de trabajo en donde trabajan ustedes? ¿De qué manera? ¿El plan ha influido en la producción y calidad de servicios en el municipio? ¿De qué manera? 4. Clima organizacional • ¿Que entienden ustedes por el concepto “clima organizacional?” ¿Y por el concepto de liderazgo? • ¿Cómo se aplicó la encuesta de clima en este municipio? ¿Entendieron todas las preguntas de la encuesta? ¿Conocen los resultados de clima (los datos) de este municipio? ¿Cuáles son? • ¿Perciben ustedes que el clima organizacional ha cambiado en este municipio con relación al año pasado? ¿Si hubo un cambio, cuáles son algunos ejemplos? ¿A que atribuyen el cambio? ¿Si no hubo un cambio, porque no? 5. El desarrollo de liderazgo a todos niveles • ¿Qué cambios han observado en su jefes/supervisores inmediatos a partir del desarrollo del programa de liderazgo? • ¿Qué cambios han percibidos en el desempeño del personal de salud en este municipio a partir del desarrollo del programa de liderazgo? 6. Conclusión • ¿Tienen algunas sugerencias para fortalecer el proceso de mejorar las capacidades de liderazgo al nivel municipal y el proceso de seguimiento? ¡Muchas gracias por su participación y colaboración! Nicaragua Leadership Development Program Evaluation 58 Evaluación del Programa de Desarrollo de Liderazgo Management Sciences for Health Guía de Entrevista Individual para Participantes Introducción explicando el objetivo del estudio y el propósito de la entrevista. Se trata de identificar del punto de vista de los participantes del programa lo que pasó durante el proceso de desarrollar el programa de liderazgo y reflexionar sobre los avances del programa así como los mejoramientos indicados para el futuro. Las perspectivas y opiniones de los participantes ayudarán a fortalecer el programa en el futuro. Hay que enfatizar que las respuestas y los comentarios de los participantes son absolutamente confidenciales y anónimos. ******** 1. Antecedentes • Introducción del entrevistado incluyendo: su cargo actual, el número de años que ha trabajado en este cargo, y el número de años en total que lleva con el MINSA. 2. El proceso de desarrollar el programa • ¿Cómo se desarrolló el programa de liderazgo en este municipio? ¿En su opinión, cuál es el propósito de este programa? ¿Ya participó usted en la replica de los módulos de liderazgo? ¿Todos sus colegas han participado en las replicas también? 3. El plan de mejora • ¿Como parte del programa de liderazgo, se desarrolló un plan de mejora? ¿Cómo fue el proceso para desarrollar el plan y quien participó en su elaboración? • ¿En su opinión, cuál es el problema o objetivo que están abordando a través del plan? ¿Cómo se seleccionó este problema entre otros? ¿Quien lo seleccionó? ¿Qué tipo de información y de que fuentes se utilizó para analizar el problema? • ¿Cuándo comenzaron a implementar el plan? ¿Ya se implementaron todas las actividades? ¿Cómo figuran ustedes en las actividades del plan -- cuál es su papel en este plan? ¿Se formaron equipos para implementar el plan? • ¿Recibieron seguimiento o apoyo durante el programa? ¿Por parte de quien? ¿Y durante la implementación del plan? ¿Por parte de quien? • ¿Tienen los recursos necesarios – humanos y otros -- para cumplir el reto? ¿Si sí, como aseguraron los recursos necesarios? ¿Si no, cuales acciones han tomados para enfrentar el reto de toda manera? Nicaragua Leadership Development Program Evaluation 59 • ¿Cuentan con el apoyo de los jefes/supervisores para la implementación del plan? ¿Cuál ha sido el nivel de cooperación y colaboración entre sus colegas para su implementación? • ¿Cuales son los resultados hasta la fecha relacionados con el plan? ¿Qué otros resultados no relacionados a su plan logró el municipio? ¿Cómo se esta monitoreando los avances en el plan y por quien? • ¿Cree usted que el plan de mejora ha influido en el ambiente de trabajo en donde trabajan ustedes? ¿De qué manera? ¿El plan ha influido en la producción y calidad de servicios en el municipio? ¿De qué manera? 4. Clima organizacional • ¿Que entiende usted por el concepto “clima organizacional?” ¿Y por el concepto de liderazgo? • ¿Cómo se aplicó la encuesta de clima en este municipio? ¿Entendió usted todas las preguntas de la encuesta? ¿Conoce los resultados de clima (los datos) de este municipio? ¿Cuáles son? • ¿Percibe usted que el clima organizacional ha cambiado en este municipio con relación al año pasado? ¿Si hubo un cambio, cuáles son algunos ejemplos? ¿A que atribuye el cambio? ¿Si no hubo un cambio, porque no? 5. El desarrollo de liderazgo a todos niveles • ¿Qué cambios ha observado en su jefe/supervisor inmediato a partir del desarrollo del programa de liderazgo? • ¿Qué cambios ha percibidos en el desempeño del personal de salud en este municipio/SILAIS a partir del desarrollo del programa de liderazgo? 6. Conclusión • ¿Tiene algunas sugerencias para fortalecer el proceso de mejorar las capacidades de liderazgo al nivel municipal y el proceso de seguimiento? ¡Muchas gracias por su participación y colaboración! Nicaragua Leadership Development Program Evaluation 60 Evaluación del Programa de Desarrollo de Liderazgo Management Sciences for Health Guía de Entrevista Individual para el Equipo de Dirección del Municipio Introducción explicando el objetivo del estudio y el propósito de la entrevista. Se trata de identificar del punto de vista del equipo de dirección municipal lo que pasó durante el proceso de desarrollar el programa de liderazgo, y reflexionar sobre los avances del programa así como los mejoramientos indicados para el futuro. Las perspectivas y opiniones de los entrevistados ayudarán a fortalecer el programa en el futuro. Hay que enfatizar que las respuestas y los comentarios de los entrevistados son absolutamente confidenciales y anónimos. ******** 1. Antecedentes • Introducción del entrevistado incluyendo: su cargo actual, el número de años que ha trabajado en este cargo, y el número de años en total que lleva con el MINSA. • ¿Cuántas personas trabajan en este municipio? ¿Cuántas personas tiene a su cargo? 2. El proceso de desarrollar el programa • ¿Cómo se desarrolló el programa de liderazgo en este municipio – como involucraron a los municipios en el proceso? ¿Ya participó todo el personal municipal en las replicas? 3. El plan de mejora • ¿Cómo parte del proceso, se desarrolló un plan de mejora? ¿Qué procesos utilizaron para desarrollar el plan? ¿Cómo se seleccionó este reto entre otros? ¿Quien lo seleccionó? ¿Qué tipo de información y de que fuentes utilizaron para analizar el reto? • ¿Se puede ver una copia del plan? ¿Cuándo comenzaron a implementarlo? ¿Ya se implementaron todas las actividades? ¿Se formaron equipos para implementarlo? ¿Cuál ha sido el nivel de cooperación y colaboración entre el equipo de dirección para su implementación? • ¿Recibieron seguimiento o apoyo durante el programa? ¿Por parte de quien? ¿Y durante la implementación del plan? ¿Por parte de quien? • ¿Tienen los recursos necesarios – humanos y otros -- para cumplir el reto? ¿Si sí, como aseguraron los recursos necesarios? ¿Si no, cuales acciones han tomados para enfrentar el reto de toda manera? • ¿Cuales son los resultados de su implementación hasta la fecha? ¿Qué otros resultados no relacionados a su plan logró el municipio? ¿Cómo se esta monitoreando los avances en el plan y por quien? Nicaragua Leadership Development Program Evaluation 61 • ¿Considera usted que el plan de mejora ha influido en el ambiente de trabajo en donde trabaja usted (centro de salud)? ¿De qué manera? ¿El plan ha influido en el ambiente de trabajo en el resto del municipio? ¿De qué manera? ¿Ha influido en la producción de servicios en el municipio? ¿De qué manera? 4. Clima organizacional • ¿Que entiende usted por el concepto “clima organizacional?” ¿Y liderazgo? • ¿Cómo se aplicó la encuesta de clima en este municipio? ¿Entendió usted todas las preguntas de la encuesta? ¿Cree que el personal las entendieron? • ¿Percibe usted que el clima organizacional ha cambiado en este municipio con relación al año pasado? ¿Si hubo un cambio, cuáles son algunos ejemplos? ¿A que atribuye el cambio? ¿Si no hubo un cambio, porque no? • En su opinión, cómo está el nivel de clima en este municipio con relación a otros municipios en este mismo SILAIS? ¿Si hay una diferencia, porque cree que existe? 5. El desarrollo de liderazgo a todos niveles • ¿Qué cambios ha observado en el desempeño del personal de salud municipal a partir del desarrollo del programa de liderazgo? • ¿Qué cambios ha percibido en su propio desempeño como jefe o supervisor a partir del desarrollo del programa de liderazgo? • ¿Recibió usted durante este programa retroalimentación de su desempeño como líder? ¿De quién? ¿Cómo? 6. Describe por favor un reto importante que usted superó en los últimos 6 meses: • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintió? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? 7. Describe por favor un reto importante que usted no pudo superar (un momento o incidente negativo que le afectó) en los últimos 6 meses: • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintió? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? • ¿Si tuviera otra oportunidad de enfrentar este reto, qué cambiaría? 8. Conclusiones • ¿Tiene algunas sugerencias para fortalecer el proceso de mejoramiento de liderazgo al nivel municipal? ¡Muchas gracias por su participación y colaboración! Nicaragua Leadership Development Program Evaluation 62 Evaluación del Programa de Desarrollo de Liderazgo Management Sciences for Health Guía de Entrevista Individual para el Equipo de Dirección del SILAIS Introducción explicando el objetivo del estudio y el propósito de la entrevista. Se trata de identificar del punto de vista del equipo de dirección del SILAIS lo que pasó durante el proceso de desarrollar el programa de liderazgo, y reflexionar sobre los avances del programa así como los mejoramientos indicados para el futuro. Las perspectivas y opiniones de los entrevistados ayudarán a fortalecer el programa en el futuro. Hay que enfatizar que las respuestas y los comentarios de los entrevistados son absolutamente confidenciales y anónimos. ******** 1. Antecedentes • Introducción del entrevistado incluyendo: su cargo actual, el número de años que ha trabajado en este cargo, y el número de años en total que lleva con el MINSA. • ¿Cuántas personas trabajan en este SILAIS? ¿Cuántas personas tiene a su cargo? 2. El proceso de desarrollar el programa • ¿Cómo se desarrolló el programa de liderazgo en este SILAIS - como involucraron al SILAIS y los municipios en el proceso? ¿Ya participó todo el personal municipal y del SILAIS en la replica de los módulos de liderazgo? 3. El plan de mejora • ¿Cómo parte del proceso, se desarrolló un plan de mejora al nivel del SILAIS? ¿Qué procesos utilizaron para desarrollar el plan? ¿Cómo se seleccionó este reto entre otros? ¿Quien lo seleccionó? ¿Qué tipo de información utilizaron para analizar el reto? • ¿Se puede ver una copia del plan? ¿Cuándo comenzaron a implementarlo? ¿Ya se implementaron todas las actividades? ¿Se formaron equipos para implementarlo? ¿Cuál ha sido el nivel de cooperación y colaboración entre el equipo de dirección para su implementación? • ¿Recibieron seguimiento o apoyo durante el programa? ¿Por parte de quien? ¿Y durante la implementación del plan? ¿Por parte de quien? • ¿Tienen los recursos necesarios – humanos y otros -- para cumplir el reto? ¿Si sí, como aseguraron los recursos necesarios? ¿Si no, cuales acciones han tomados para enfrentar el reto de toda manera? • ¿Cuales son los resultados de su implementación hasta la fecha? ¿Qué otros resultados no relacionados a su plan logró el SILAIS? ¿Cómo se esta monitoreando los avances en el plan y por quien? Nicaragua Leadership Development Program Evaluation 63 • ¿Considera usted que el plan de mejora ha influido en el ambiente de trabajo en el SILAIS? ¿De qué manera? ¿El plan ha influido en el ambiente de trabajo en el resto del departamento? ¿De qué manera? ¿Ha influido en la producción de servicios en el departamento? ¿De qué manera? 4. Clima organizacional • ¿Que entiende usted por el concepto “clima organizacional?” ¿Y liderazgo? • ¿Cómo se aplicó la encuesta de clima en este municipio? ¿Entendió usted todas las preguntas de la encuesta? ¿Cree que el personal las entendieron? • ¿Percibe usted que el clima organizacional ha cambiado en este departamento con relación al año pasado? ¿Si hubo un cambio, cuáles son algunos ejemplos? ¿A que atribuye el cambio? ¿Si no hubo un cambio, porque no? • En su opinión, cómo está el nivel de clima en este municipio con relación a otros municipios en este mismo SILAIS? ¿Si hay una diferencia, porque cree que existe? 5. El desarrollo de liderazgo a todos niveles • ¿Qué cambios ha observado en el desempeño del personal de salud municipal y del SILAIS a partir del desarrollo del programa de liderazgo? • ¿Qué cambios ha percibido en su propio desempeño como jefe o supervisor a partir del desarrollo del programa de liderazgo? • ¿Recibió usted durante este programa retroalimentación de su desempeño como líder? ¿De quién? ¿Cómo? ¿Cómo le afectó? 6. Describe por favor un reto importante que usted superó en los últimos 6 meses: • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintió? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? 7. Describe por favor un reto importante que usted no pudo superar (un momento o incidente negativo que le afectó) en los últimos 6 meses: • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintió? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? • ¿Si tuviera otra oportunidad de enfrentar este reto, qué cambiaría? 8. Conclusiones • ¿Tiene algunas sugerencias para fortalecer el proceso de mejoramiento de liderazgo al nivel municipal y del SILAIS? ¡Muchas gracias por su participación y colaboración! Nicaragua Leadership Development Program Evaluation 64 Evaluación del Programa de Liderazgo Management Sciences for Health Guía de Entrevista Individual para el Nivel Central MINSA Introducción explicando el objetivo del estudio y el propósito de la entrevista. Se trata de identificar del punto de vista de los participantes del nivel central lo que pasó durante el proceso de desarrollar el programa de liderazgo, y reflexionar sobre los avances del programa así como los mejoramientos indicados para el futuro. Las percepciones y opiniones de los entrevistados ayudarán a fortalecer el programa en el futuro. Las respuestas y los comentarios de los entrevistados son absolutamente confidenciales y anónimos. ******** 1. Introducción • Cargo actual; el número de años que ha trabajado en este cargo; el número de años en total que lleva con el MINSA; y cuantas personas tiene a su cargo. 2. El contexto del program a nivel central • ¿Desde su perspectiva, cómo se desarrolló el programa de liderazgo en al nivel central? ¿El programa nació de que necesidad? ¿Cuál es el proposito del programa? ¿Usted participó en todos los talleres? • ¿En su opinión, cual es la relación entre el programa de liderazgo al nivel central y el programa de liderazgo en los SILAIS y al nivel local? 3. El processo de desarrollar el programa • ¿Cómo parte del proceso, se desarrollaró un plan de mejora al nivel central -- qué procesos utilizaron para desarrollar el plan? ¿Cómo se seleccionio el reto entre otros? ¿Quien lo seleccionó? ¿Qué tipo de información utilizaron para analizar el reto? • ¿Ya se implementaron las actividades? ¿Quién participó en la implementación de las actividades? ¿Se formaron equipos para implementarlo? ¿Cuál ha sido su proprio papel con la implementación del plan? ¿Cuál ha sido el nivel de cooperación y colaboración entre los participantes para su implementación? ¿Han encontrado alguna barreras en la implementación del plan? • ¿Tienen los recursos necesarios – humanos y otros -- para cumplir el reto? ¿Si sí, como aseguraron los recursos necesarios? ¿Si no, cuales acciones han tomados para enfrentar el reto de toda manera? Nicaragua Leadership Development Program Evaluation 65 4. Resultados percibidos • ¿Cuales son los logros o resultados relacionados con la implementación del plan hasta la fecha? ¿Algunos ejemplos concretos? ¿Qué otros resultados se lograron que no son relacionados al plan? ¿Cómo se esta monitoreando los avances en el plan y por quien? ¿Como van a sostener estos cambios? • ¿Considera usted que el plan de mejora ha influido en el ambiente de trabajo en su área de trabjo? ¿De qué manera? ¿En el nivel central? ¿De qué manera? • ¿Qué cambios ha observado en el desempeño del personal al nivel central? ¿Y en su propia dirección? ¿A percibido algun cambio en la manera en que ustedes trabajan o colaboran con las otras direcciones? ¿Algunos ejemplos específicos? • ¿Ha percibido un cambio en el desempeño su jefe? ¿Y en su propio desempeño como líder? ¿Ha podido poner en practica las herramientas del curso en su trabajo diario? ¿Algunos ejemplos concretos? • ¿En su opinión, cuáles son las perspectivas para el futuro para este programa? ¿Cree que este programa esta dando pasos hacia la institucionalización? 5. Describe por favor un reto importante que usted superó en los ultimos 6 meses. • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintio? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? 6. Describe por favor un reto importante que usted no pudo superar (un momento o incidento negativo que le afectó) en los ultimos 6 meses. • ¿Que hizo usted para enfrentarlo? ¿Cómo se sintio? ¿Buscó información sobre el reto o sobre como resolverlo? ¿De que fuentes? ¿Colaboró con otra colega(s) o persona(s)? ¿Cómo involucró a esta(s) personas? • ¿Si tuviera otra oportunidad de enfrentar este reto, qué cambiaría? 7. Conclusiones • ¿Tiene algunas sugerencias para fortalecer el proceso de mejoramiento de liderazgo al nivel central? ¡Muchas gracias por su participación y colaboración! Nicaragua Leadership Development Program Evaluation 66 Appendix 3: Key Questions for the Evaluation Key Questions for Evaluating Programs to Develop Managers Who Lead Key questions and line of inquiry: 1. What TA approaches and tools were used by M&L? a. What was the content of the leading and managing intervention? b. How was M&L framework applied? c. Were teams formed to address the challenge and if so, how? d. What process was used for program delivery and follow-up support? 2. To what extent has the client organization achieved its performance objectives and intended results? a. What is the availability/existence of the client’s action plans? b. How and why did the client/team prioritize its objectives/desired performance – what was the selection based on? c. What process was used to prepare the action plan – how were the activities selected to address the challenge and lead to achieving the desired performance? d. Were all activities implemented? Were other activities implemented that were not included in the action plan? e. What means does the client have for monitoring their progress? f. What are the client’s measurable results related to the program’s objectives and their action plans? Did the implementation of action plans lead to the client’s desired results? g. What motivated the teams to own or make a commitment to address the challenges they identified? h. What changes did the team note in their leading and managing behaviors/practices? What are the teams doing differently? i. What other results did the client achieve that are unrelated to their action plan? 3. What elements of M&L interventions are associated with organizational performance outcomes (improved management systems and/or improved work climate and/or increased ability to respond to changing environments) as well as service delivery outcomes (where possible)? a. Are improvements in health services likely associated with program inputs or an historical trend? How do the service results of the intervention group compare with those of a similar (comparison) group at a similar time? (these questions are only appropriate for Egypt and Nicaragua) b. What are the logical pathways between the program inputs – outputs – outcomes? c. Did the team receive TA during the program from any source other than M&L? Nicaragua Leadership Development Program Evaluation 67 Did the team work together to address their challenge during the program? What process and skills did the team use? What motivated the team to achieve their results? What prevented them from achieving their goals? d. Did the team continue to work together to address another challenge after the program ended? What processes were used then? Was this similar to or different from how they worked together as a team during the program? How did they motivate the participation or commitment of team members after the program ended? e. How did follow-up support affect the implementation of the action plans? Did the client receive follow-up support related to addressing their challenge from any source other than M&L? 4. Are the measurable changes in organizational performance sustainable? a. Were climate levels and team building processes maintained and/or improved in the period post-program? b. Were service results maintained and/or increased in the period post-program? c. To what extent was the program replicated after the program ended? (Replicated with same teams that took on new challenge after the program ended or replicated with a new set of teams) What is the quality of the replication? d. To what extent and how was the program scaled up to multiple levels of the health system? Nicaragua Leadership Development Program Evaluation 68 Appendix 4: PAHO Climate Instrument MINISTRY OF HEALTH MSH/PROSALUD QUESTIONNAIRE ON ORGANIZATIONAL CLIMATE 1.- The boss is concerned that we have a proper understanding of our work. 2.- As a rule, we all contribute ideas aimed at improving our work. 3.- Most of the work in our department or area requires reasoning. 4.- In this organization, efforts are made to ensure that each individual makes decisions with regard to how to conduct his or her own work assignments. 5.- The environment in this institution is tense. 6.- People make an effort to fully comply with their obligations. 7.- Our co-workers frequently speak poorly of the institution. 8.- This institution provides good training opportunities. 9.- Here, promotions lack objectivity. 10.- Problems that arise between work groups are optimally resolved to the benefit of the institution. 11.- The objectives of individual units are consistent with the objectives of the institution. 12.- The information required by the various groups flows slowly. 13.- The adoption of new technologies is viewed with distrust. 14.- It frequently happens that when a special problem occurs no one knows who is supposed to solve it. 15.- There is concern here for keeping the staff informed with regard to new work￾related techniques, in order to improve the quality of the work performed. 16.- Here, all problems are discussed constructively. 17.- In order to comply with our work-related goals, we have to use all of our skills. Nicaragua Leadership Development Program Evaluation 69 18.- With this job I feel professionally fulfilled. 19.- In this institution, individuals who work well are rewarded. 20.- In reality, the ideas that we contribute for improving work are never put into practice. 21.- Working conditions are good. 22.- Here, one does not feel any self-motivation for working. 23.- It is a pleasure to observe the order that prevails in our office. 24.- Here, incentives are provided over and above those established in our work contract. 25.- Disciplinary norms are not applied objectively. 26.- When the organization faces a challenge (important work)+B42, all departments participate actively in seeking a solution. 27.- The important thing is to achieve the objectives for the assigned work area; nothing else matters. 28.- As a rule, when something is going to be done, my area is the last to find out. 29.- Initiatives taken by groups do not receive support from higher levels. 30.- If an assignment appears to be difficult, it is delayed as long as possible. 31.- We can only tell our boss what he or she wants to hear. 32.- In this area, recognition is given to the work performed by the staff. 33.- There is no clear definition of the functions to be performed by each individual. 34.- Almost no one holds back in carrying out his or her obligations. 35.- When someone doesn't know how to do something, no one helps him or her. 36.- When we have a problem, no one is interested in solving it. 37.- There is little freedom of action for performing work. 38.- There are groups whose work and behavior do not contribute to the work of the institution. Nicaragua Leadership Development Program Evaluation 70 39.- This organization's development programs prepare staff members to advance along a particular career path. 40.- In this organization, there is concern only for errors. 41.- Here, people are dismissed with considerable ease. 42.- As a rule, work is performed in a mediocre fashion. 43.- Almost everyone performs their work with freedom and autonomy. 44.- We treat the users of our services with respect and diligence. 45.- Efficiency in work does not bring recognition of any kind. 46.- Here, each department or area works on its own. 47.- Here, power is concentrated in just a few departments or work areas. 48.- We periodically experience problems resulting from gossip and rumors. 49.- Here, one can develop his or her ingenuity and creativity. 50.- Our boss is understanding, but demands very little of us. 51.- Work is frequently initiated without anyone knowing the reason why. 52.- The boss is not concerned with contributions of ideas aimed at improving the quality of work. 53.- Training programs are only for a select few. 54.- In this organization, being promoted means being able to deal with greater challenges. 55.- Problems are analyzed by following systematic methods for finding creative solutions. 56.- The dedication of this department or area is deserving of recognition. 57.- Any decision taken must be consulted with one's superiors before it is put into practice. 58.- As a rule, individuals are responsible for monitoring their own work. 59.- Most of the employees of this institution feel satisfied with the physical environment existing in our work area. Nicaragua Leadership Development Program Evaluation 71 60.- We staunchly defend the work and image of our area. 61.- Team spirit in this organization is excellent. 62.- We readily share with other groups within the institution the limited resources of our project. 63.- Those who possess information are reluctant to make that fact known. 64.- In this organization there are groups that are opposed to change of any type. 65.- Each individual has available the elements necessary to perform his or her work. 66.- As a rule, individuals who perform their work well are rewarded with a better position within the organization. 67.- As a rule, we have many things to do and do not know where to start. 68.- When we analyze a problem, the positions adopted by my co-workers are not always sincere. 69.- As a rule, special recognition is given for good on-the-job performance. 70.- My boss is not concerned with the quality of the work performed. 71.- People like to take on important work assignments. 72.- As a rule, everyone treats the organization's assets with care. 73.- Here, the results obtained are the fruit of the work of a handful of individuals. 74.- Employees feel pride in being a part of this institution. 75.- Each individual is considered to be knowledgeable of his or her work and is treated accordingly. 76.- Performance of tasks is properly evaluated. 77.- The various hierarchical levels of the organization are not mutually cooperative. 78.- Here, departments or areas exist in a permanent state of conflict. 79.- Here, information is concentrated in a handful of groups. 80.- Higher levels do not encourage positive change to the benefit of the institution. Nicaragua Leadership Development Program Evaluation 72 Appendix 5: PAHO Climate Definitions Definition of Terms Used Organizational Climate9 Study 1. LEADERSHIP The influence exercised by an individual over the behavior of others, in order to achieve certain results. - Direction: Provides a sense of orientation to the activities of a work unit, by clearly establishing the objectives and goals to be achieved as well as the means by which to achieve them. - Stimulation of Excellence: Promotes and assumes responsibility for the quality and impact of products and of institutional activity. - Stimulation of Teamwork: Promotes teamwork at the internal level and among administrative units, essentially by seeking the achievement of common objectives. - Conflict Resolution: Resolves problems and conflicts inherent in organizational life by promoting constructive change within the organization. 2. MOTIVATION Series of reactions and attitudes in individuals that become manifest in the presence of specific stimuli in the surrounding environment. - Personal Fulfillment: The total fulfillment of the individual can take place only within an occupational context in which that individual is able to apply his or her skills. - Recognition of Contributions: When the organization recognizes and gives credit to the effort put forth by each individual and group in the implementation of the tasks assigned for achieving institutional objectives. - Responsibility: The ability of individuals to face up to their duties and accept the consequences of their actions. 9 Teoría y Técnicas de Desarrollo Organizacional, Vol. III, Lic. José María Marín, Lic. Armando Melgar and Ing. Carlos Castaño, Pan-American Health Organization. Nicaragua Leadership Development Program Evaluation 73 - Adaptation of Working Conditions: Physical and psycho-social conditions present in the environment in which work is carried out, as well as the quantity and quality of resources made available for carrying out the assigned functions. 3. RECIPROCITY The satisfaction of mutual expectations, of both the individual as well as of the organization, that go beyond the formal work contract between the individual and the organization. The individual feels him or herself to be a part of the organization and, consequently, becomes a symbol personifying the organization. - Application to Work: When an individual has fully identified himself with his work and with the institution and adopts behaviors that go beyond the commitments undertaken in the formal employment contract, as manifested in his dedication, ingeniousness and creativity in solving problems and achieving institutional objectives. - Stewardship of Institutional Assets: This is the stewardship that individuals exhibit with regard to the assets of the institution, as well as their concern for strengthening and defending the prestige, values and image of the institution. - Compensation: The optimum use of mechanisms of compensation in terms of the benefits that the organization makes available to its members to contribute to their personal fulfillment and social development and as a response to contributions made in the workplace. - Equity: Access by workers to compensations by means of an equitable system. Being treated with impartiality in processes of selection and promotion based on competition and merit as a function of requirements. 4. PARTICIPATION Involvement of individuals in the activities of the organization, with each contributing that which has been assigned to him or her, in order to achieve institutional objectives. The integration of individuals within the organization is one effect of participation. - Commitment to Productivity: Each individual and unit within the organization, working in harmony with all other components, provides the appropriate service with an optimum level of efficiency and effectiveness. - Synchronization of Interest: To integrate the diversity of the components involved in the organization in a single direction, synchronizing different areas of conflict, such as competition for limited resources, distribution of power, tendencies toward autonomy. Nicaragua Leadership Development Program Evaluation 74 - Exchange of Information: Communicating and exchanging important information among individuals and groups with regard to common objectives and the means available to each to contribute to the achievement of those objectives. - Involvement in Change: The attitude of promotion, acceptance and commitment with regard to decisions of change, participation, contribution of suggestions and learning of new skills. Nicaragua Leadership Development Program Evaluation 75 Appendix 6: Municipal Level Climate Data Climate data for Phase 1 municipalities Municipal climate scores SILAIS Municipality Jun-01 Aug-02 % change La Dalia 3.15 3.24 3.03% Muy Muy 3.44 3.35 -2.46% Paiwas 3.20 3.48 8.67% Rancho Grande 2.56 2.56 0.00% Rio Blanco 3.04 2.19 -27.97% Matagalpa Waslala 3.15 2.64 -16.02% Bocay 1.76 2.16 23.03% Cua 2.49 1.82 -26.92% Jinotega 2.85 3.24 13.74% Pantasma 2.40 2.56 6.65% Jinotega Wiwili 2.70 2.57 -4.88% Camoapa 2.29 2.30 0.50% Boaco San Lorenzo 3.35 4.46 33.39% Phase 1 mean scores and overall percent change 2.797 2.814 0.60% Climate data for Phase 2 municipalities Municipal climate scores SILAIS Municipality Jan-03 Jun-03 % change Matiguas 3.46 3.77 8.96% Esquipulas 3.15 3.70 17.46% Dario 2.55 3.61 41.57% San Isidrio 3.17 3.73 17.67% Sebaco 2.63 3.09 17.49% San Dionisio 3.37 2.62 -22.26% San Ramon 3.35 2.94 -12.24% Terrabona 2.66 2.89 8.65% Matagalpa Matagalpa 2.68 2.80 4.48% San Rafael del Norte 2.22 3.86 73.87% Yali 2.63 3.80 44.49% Jinotega Concordia 2.70 2.96 9.63% Boaco 4.06 4.12 1.48% Santa Lucia 3.34 3.72 11.38% Teustepe 3.38 3.51 3.85% Boaco San Jose de los Remates 3.50 3.41 -2.57% Phase 2 mean scores and overall percent change 3.053 3.408 11.63% Nicaragua Leadership Development Program Evaluation 76 Climate data for Estelí: Phase 3 municipalities Municipal climate scores SILAIS Esteli Jul-03 Jan-04 % change Condega 1.87 2.37 26.77% Estelí 2.66 2.87 7.75% La Trinidad 2.61 2.63 0.66% Pueblo Nuevo 3.65 3.74 2.51% San Juan de Limay 2.59 2.67 3.14% San Nicolás 3.05 3.48 14.21% Hospital La Trinidad 2.38 2.59 8.83% Hospital San Juan de Dios 2.88 2.89 0.39% Mean score for SILAIS and overall percent change 2.71 2.90 7.15% Climate data for Madriz: Phase 3 municipalities Municipal climate scores SILAIS Madriz Aug-03 Jan-04 % increase La Sabana 2.97 4.61 55.26% Palacaguina 3.16 4.01 26.98% San Jose Cusmapa 3.81 3.58 -6.07% San Juan Río Coco 3.25 3.23 -0.76% San Lucas 4.24 4.11 -3.10% Somoto 3.54 3.89 9.81% Telpaneca 3.44 3.46 0.50% Totogalpa 2.98 3.49 17.05% Yalaguina 4.09 4.60 12.62% Hospital Madriz 0.00 4.15 -- Mean score for SILAIS and overall percent change 3.50 3.89 11.10% Climate data for Masaya: Phase 3 municipalities Municipal climate scores SILAIS Masaya Jul-03 Jan-04 % increase Catarina 2.26 2.13 -6.01% La Concepción 1.92 1.79 -6.73% Masatepe 1.90 2.85 49.40% Masaya Norte 2.22 3.12 40.65% Masaya Sur 3.26 3.33 2.40% Nandasmo 2.25 2.10 -6.72% Nindirí 1.69 2.01 19.07% Niquinohomo 2.23 2.95 32.28% San Juan de Oriente 2.46 3.48 41.86% Tisma 2.08 2.25 8.33% Mean score for SILAIS and overall percent change 2.23 2.60 16.83% Nicaragua Leadership Development Program Evaluation 77 Climate data for Rivas: Phase 3 municipalities Mean scores per municipality SILAIS Rivas Jul-03 Jan-04 % change Altagracia 3.75 3.57 -4.67% Belen 2.36 4.01 69.74% Buenos Aires 3.31 3.69 11.25% Cardenas 2.98 3.33 11.66% Moyagalpa 3.19 3.61 13.17% Potosi 2.82 3.41 21.01% San Jorge 2.37 4.01 68.91% San Juan 2.73 2.82 3.39% Tola 2.24 2.91 29.95% Hospital Rivas 2.84 3.02 6.26% Means score for SILAIS and overall percent change 2.86 3.44 20.22% Nicaragua Leadership Development Program Evaluation 78 Appendix 7: Cost Analysis of Leadership Program Cost Analysis of Nicaragua Leadership Development Program Managua, Nicaragua March 8th – 25th, 2004 Kate Waldman Nicaragua Leadership Development Program Evaluation 79 Table of Contents Executive Summary……………………………………………………………..3 Background.……………………………………………………………………...4 Study Objectives…………………………………………………………………6 Methodology……………………………………………………………………...6 Findings and Conclusions……………………………………………………….7 Appendices……………………………………………………………………….8 Nicaragua Leadership Development Program Evaluation 80 Executive Summary The Leadership Development Program (LDP) in Nicaragua is a program designed to strengthen the capacity of health managers and personnel at various levels. Since its inception in July, 2001, nearly 2000 people have been trained in the program by both MSH facilitators and trained municipal and SILAIS level facilitators. This analysis of program costs will demonstrate the cost efficiency of this particular program, using information from M&L’s Monthly Expenditure Reports (MERs) and financial information provided by the MSH/Nicaragua office. The in-kind contributions of program participants from the Ministry of Health, SILAIS, and municipalities, such as their time, are not included in this analysis as data is too difficult to collect. The phases of the program are broken down in two ways: by the timing of the phase, as used in the evaluation of the program, and a more in depth analysis of Phase 3 by the level of the recipients of the training during M&L’s Program Year (PY) 4.10 With this information, we can see both the cost per participant throughout the life of the program as well as a breakdown of what levels of trainings are the most cost efficient. Analysis of the data revealed: • The number of participants increased by 51% from Phase 1 to Phase 2. From Phase 2 to Phase 3, the number of participants increased by 344%. • The total cost of the first phase of the program (2001-2002) was $70,606. This was slightly higher than the cost of the second phase (2002-2003), which came to $54,023. The third phase (2003-2004), where the most expansion occurred, had a total cost of $339,506. • The cost per participant decreased by 51% from Phase 1 to Phase 2, but increased by 41% from Phase 2 to Phase 3. • The average cost per participant over the course of the 3 years of the program was $241. This factors in all of the level of effort (LOE), travel/perdiem, training costs (meeting space rental, food/lodging for participants, and materials), and continuous monitoring and mentoring costs associated with the program. Also included are appropriate health/sick/vacation (HSV), Overhead, and Allocable Cost Factor (ACF)11 rates. 10 The Management and Leadership Program organizes its years into project years on the following schedule: Project Year (PY) 1: September 2000 – June 30, 2001; PY2: July 1, 2001 – June 30, 2002; PY3: July 1, 2002 – June 30, 2003; PY4: July 1, 2003 – June 30, 2004; PY5: July 1, 2004 – September 30, 2005. 11 Allocable Cost Factor (ACF) - This percentage covers certain project costs that benefit the entire project, including such expenses as rent and utilities, recruitment, general project equipment and office expenses, as well dedicated project support staff including finance officers and contracts officers. Individual M&L projects receive support in activities ranging from budgeting to monthly monitoring of expenditures, producing financial and other reports, required USAID reporting, contracting and procuring services. Nicaragua Leadership Development Program Evaluation 81 Background Several years ago the Ministry of Health (MOH) of Nicaragua entered a process of modernization and decentralization. This has required the development of leadership and management capacity at all levels of the organization in order to address the many challenges provoked by external and internal pressures such as economic crisis, political change, growing demands and needs of clients as well as bureaucratic lethargy and increasing apathy among health personnel. As one response to this need, M&L and the PROSALUD project, together with the MOH, launched a leadership development program in 2001 to strengthen the capacity of health managers and personnel in 63 municipalities, 7 SILAIS and the central level over a period of three years. The leadership program was implemented in three phases (see table below). Following a leadership dialogue in February 2001, the first phase of the program took place from July 2001 to June 2002 in 13 municipalities in the SILAIS of Matagalpa, Jinotega and Boaco. Twelve of these were prioritized municipalities where PROSALUD had been implementing the Fully Functioning Service Delivery Program (FFSDP) program since 1999. The second phase was carried out from November 2002 to June 2003 in the remaining 16 municipalities of the same three SILAIS. The third phase began in July 2003 in 34 municipalities of four additional SILAIS (Masaya, Madriz, Estelí and Rivas) plus the management teams of all seven SILAIS, and 65 key managers from central level directorates and departments.12 This third phase was conducted as part of the larger multi-component Leadership and Management for Health Project that began in April 2003 and builds on several years of service delivery strengthening interventions in Nicaragua conducted with the MSH PROSALUD project. While the work that MSH and M&L have conducted in Nicaragua has included both management and leadership interventions, this study focuses solely on the leadership development program. Participants and Phases of the Nicaragua Leadership Development Program Number of municipalities per phase SILAIS First Phase 2001-2002 Second Phase 2002-2003 Third Phase 2003-2004 Total Matagalpa 6 9 - 15 Jinotega 5 3 - 8 Boaco 2 4 - 6 Masaya - - 9 9 Madriz - - 9 9 Estelí - - 6 6 Rivas - - 10 10 Total 13 16 34 63 12 Beginning in January 2004, the third phase will expand to include a leadership development program for the Managua SILAIS and the Ministry of the Family, but this component is not included in this analysis. Nicaragua Leadership Development Program Evaluation 82 Number of participants per phase Facilitators (municipal and SILAIS) 32 40 129 201 Participants (municipal and SILAIS) 183 284 1245 1712 Central level participants - - 65 65 Total 215 324 1439 1978 The objectives of the leadership development program evolved over the course of the three year the program and in the final year included the following: ¾ Improve organizational climate in the participating municipalities and SILAIS ¾ Develop leadership capacities among the municipal and SILAIS management teams ¾ Institutionalize the leadership program within the Nicaraguan MOH The leadership development program at the municipal level consisted of the following programmatic components: 1. Baseline and follow-up studies of organizational climate in all participating municipalities and SILAIS. 2. Delivery of six self-instructional units for managers and staff in the participating municipalities. 3. Broad replication of the six units to the remaining staff in these municipalities. 4. Implementation of an improvement plan resulting from the analysis of organizational climate data. 5. Technical assistance and mentoring by PROSALUD/M&L staff in the application of concepts learned and the implementation of the improvement plan. The following components made up the program at the central level and SILAIS levels: 1. Leadership Dialogue to define challenges facing the MOH 2. Four 2-day leadership development workshops for central and SILAIS staff 3. Final evaluation workshop 4. Coaching and follow-up An evaluation of the first phase of the leadership development program was conducted by M&L in the fall of 2002 to assess the organizational climate outcomes prior to the beginning of the second phase of the program. Following completion of the second phase, a systematization of results from both the first and second phases was conducted in May-June 2003. The current evaluation will serve as a final evaluation of the third phase of the Nicaragua leadership development program. Nicaragua Leadership Development Program Evaluation 83 Study Objectives The purpose of this study is to determine the cost of development and replication of the Leadership Development Program. This will be accomplished by: - Determining costs incurred by MSH/Boston through examination of Monthly Expenditure Reports (MERs); - Distinguishing costs of development of the program versus replication of the program; - Working with MSH/Nicaragua finance team to determine local costs incurred by the MSH/Nicaragua office in each phase of the program. Methodology All of the data for this study was extracted from MERs and MSH/Nicaragua expenditure information. Not included are costs incurred by participants, which is limited only to LOE. The expenditures were broken down in the following ways: • Data for each phase of the program (Phase 1: 2001-2002, Phase 2:2002-2003, and Phase 3: 2003-2004). • While the first two phases of the program (2001-2002 [PY2] and 2002-2003 [PY3]) had their own code assigned, the third phase (2003-2004 [PY4]) is part of the larger Leadership and Management in Health Project13, and therefore specific costs of the Leadership Development Program are slightly more difficult to determine. • The data from Phase 3 (PY4) were broken down further according to the level in the health system of the participants in the leadership development training. There are three levels in this regard: training of participants at the central level of the Ministry of Health, training of facilitators at the SILAIS and the municipal level, and the subsequent training of participants by these facilitators. In addition, monitoring of progress and mentoring of facilitators in previous and current SILAIS and municipalities are significant costs during this period. • In order to determine the costs at each level, interviews were conducted with MSH/Nicaragua program managers and financial managers, who have detailed accounts of expenditures at each level of the program. 13 The Leadership and Management in Health Project started as a 14-month, multi-component project with the Nicaraguan Ministry of Health (MINSA) and to a limited extent with the National Social Security Institute (INSS), in April 2003. The project furthers the objectives of the USAID-funded PROSALUD bilateral Project, which ended in June 2003, and incorporates new components. The overall objective of the Project is to strengthen the capacity of managers and management systems at the central and decentralized levels of the Ministry through a package of management development and institutional reform technical assistance. The project was expanded during PY4 and PY5 to work with the Ministry of the Family, the Ministry of Education, and NicaSalud a network of NGOs, and will continue through March 2005. Nicaragua Leadership Development Program Evaluation 84 Findings and Conclusions As expected, the development of the LDP and its initial delivery were more costly than the subsequent deliveries and replication of the program. As shown in the table above and chart below, the number of participants in each phase increased, with 215 participants in the first phase, 324 participants in the second phase, and 1,439 participants in the third phase. The total cost of the program fluctuates from year to year, but the cost per participant significantly decreases from the first phase to the second and increases from the second phase to the third. This increase in costs can be attributed to the tremendous increase in number of participants during Phase 3, which required hiring additional staff by MSH and conducting monitoring of the replication of the program. It is expected that as the program continues to spread and individual SILAIS or municipalities start to take on some of the program costs, the costs incurred by MSH will decrease. In spite of this, there will be a minimum cost associated with the maintenance, monitoring and evaluation of the program. This minimum cost can more accurately be determined as the program continues, but will likely remain under $250 per participant if the model set forth in PY4 is followed. Finally, the program hopes that the trained SILAIS and municipal level facilitators will continue to deliver the program to participants in their respective areas (referred to hereafter as “participant training”). As the cost per participant for this type of training is minimal ($30/participant), total program costs and average cost per participant will continue to decrease. The chart below titled “Cost per Participant – Phase 3 (PY4)” shows the variance in cost according to each level of training. It is clear that in order to minimize program costs, the replication of the program by SILAIS and municipal facilitators is the best way to keep costs low. Phase 3 (PY4) is a good example to foresee future MSH costs, in that it involved training at all levels and monitoring of the program. The breakdown of the costs of each of these phases is shown on the chart titled “Cost per Component - Phase 3 (PY4).” This chart shows that Central Level training and SILAIS and municipal level training are the most costly, respectively accounting for 35% and 40% of the total expenses. Participant training accounts for a much smaller percentage (11%) of the total due to the fact that MSH facilitators are not used. Preparation and monitoring of the project, as well as mentoring local facilitators, are other expenses, accounting for 14% of the total. While costs will continue to fluctuate based on the number of future participants and the level of training, it an be expected that the model set forth in Phase 3 (PY4) will be an accurate way to anticipate future costs. As this program continues to expand into other areas, like other ministries (it is currently being delivered to the Ministry of the Family – MiFamilia), figures will likely change to adjust to these new areas. Nicaragua Leadership Development Program Evaluation 85 APPENDICES 1. Chart - Number of Participants per Phase 2. Total Cost per Phase 3. Cost per Participant per Phase 4. Cost per Participant – Phase 3 (PY4) 5. Cost per Component - Phase 3 (PY4) Number of Participants per Phase: 32 40 129 201 183 284 1245 1712 65 65 0 200 400 600 800 1000 1200 1400 1600 1800 1st phase 2nd phase 3rd phase TOTAL Facilitators (municipal and SILAIS) Participants (municipal and SILAIS) Central level participants Cost per phase14: First phase: $70,606 Second phase: $54,023 Third phase: $339,506 14 Costs included are those incurred by MSH/Boston and local costs incurred by MSH/Nicaragua, including staff level of effort, materials, and participant training costs (travel, perdiem, facility rental for both facilitators and participants) Nicaragua Leadership Development Program Evaluation 86 Cost per participant per phase of Nicaragua Leadership Development Program: $328 $167 $229 $- $50 $100 $150 $200 $250 $300 $350 1st phase 2nd phase 3rd phase Average: $241 (2001 - 2002) (2002 - 2003) (2003 - 2004) Cost per participant - Phase 3 (PY4) $1,819 $30 $1,051 - 200 400 600 800 1,000 1,200 1,400 1,600 1,800 2,000 Central Level MOH SILAIS/Municipal Level Participant Training Nicaragua Leadership Development Program Evaluation 87 Cost per Component - Phase 3 (PY4) 35% 40% 11% 14% Central level SILAIS/Municipal level Participant Training Monitoring