Evaluation of the Leadership Development Program for the Ministry of Health and Population, Egypt October 2003 Dr. Ersin Topçuoğlu, Senior Technical Officer Monitoring & Evaluation Unit Management and Leadership Program Cooperative Agreement No. HRN-A-00-00-00014-00 Management Sciences for Health 891 Centre Street Boston, MA 02130 ACKNOWLEDGEMENTS This evaluation report would not have been possible without the support of many people. I would like to thank to the following colleagues: Dr. Morsy Mansour, director of the Leadership Development Program in Egypt from the central office of Ministry of Health and Population of Egypt for his dedication to the program and learning and practicing leadership. And for his great friendship in Cairo, Aswan, Boston, and Ankara. Dr. Barakat, Director of Family Planning, Aswan Health Governorate, for making life easier for me during the collection of service statistics. And for arranging the sailboat trips on the Nile. Dr. Tawhida Khalil for her sound advice and support throughout the program, and for conducting the in-depth interviews and focus group discussions with program participants. Also for helping me during the final data collection and picking the best spices and Nubian masks available in the Aswan market. Ms. Omaima Abdel Akher for her continuous administrative support. Ms. Daila El Khodairy for her perfect translations. Ms. Amanda Ip for her patience in solving all adminstrative and travel related issues. Dr. Joan Galer for believing in the importance of and supporting the evaluation work from the very beginning. Ms. Alison Ellis for her thorough input and feedback on the report and also for her patience in editing/correcting my English. 2 TABLE OF CONTENTS EXECUTIVE SUMMARY ..................................................................................... 4 1. BACKGROUND 1.1 Purpose of the LDPE ………………………………………………………. 6 1.2 Objectives of the LDPE ……………………………………………………. 6 1.3 Timeline ……………………………………………………………………. 6 1.4 Program Components of LDPE ……………………………………………. 7 2. EVALUATION FRAMEWORK 2.1 The Leading and Managing Framework ....................................................... 8 2.2 Indicators ....................................................................................................... 9 3. RESULTS 3.1 Teams ability to “Select Challenges” ............................................................13 3.2 Teams ability to “Scan” their environment regarding their challenge ..........14 3.3 Teams ability to “Focus” on their selected challenges ..................................15 3.4 Teams ability to “Align & Mobilize” people and resources ..........................16 3.5 Teams ability to “Achieve Results” ...............................................................17 3.5.1 Daraw District (ANC) ........................................................................17 3.5.2 Daraw Health Center ..........................................................................19 3.5.3 Nafak Health Center ...........................................................................20 3.5.4 Gaafra Health Center ..........................................................................21 3.5.5 Gharb Aswan Hospital .......................................................................22 3.5.6 Kom Ombo District ............................................................................23 3.5.7 Daraw District (FP) …………………………………………………24 3.5.8 Aswan District ………………………………………………………25 3.5.9 Rakkaba Health Center …………………………………………….. 27 3.5.10 Al Aakab Health Center …………………………………………….29 3.6 Improved Climate .......................................................................................... 32 3.7 Teams ability to “Select New Challenge” ..................................................... 34 4. CONCLUSIONS .......................................................................................................37 5. RECOMMENDATIONS .........................................................................................39 ANNEX 1: Qualitative Results from Interviews and Focus Group Discussions.............43 3 EXECUTIVE SUMMARY The Leadership Development Program in Egypt (LDPE) was a one-year pilot program, co-led by the Ministry of Health and Population (MOHP) and the Management and Leadership (M&L) Program of Management Sciences for Health (MSH). The overall purpose of the program was to improve the quality and accessibility of health services in Egypt, specifically in three districts of the Aswan Governorate, by: • increasing the capability of managers to lead others to achieve results; and • managers’ ability to create climates of high performance in their workplaces. The LDPE began in June 2002 and ended in June 2003. It was evaluated by the M&L Monitoring and Evaluation Unit during June 15-23, 2003. The key components of program implementation were bi-monthly leadership workshops followed by monthly district or clinic level meetings. LDPE participants formed ten working teams. Each team selected a performance improvement project and prepared an associated action plan. The LDPE evaluation plan focused on: measuring the managers’ ability to lead others through the development and implementation of an action plan, to create a better workgroup climate, and to achieve service delivery results. The evaluation framework included seven indicators measuring the following leadership elements: Selecting a Challenge, Scanning, Focusing, Aligning and Mobilizing, and Inspiring. Data for this evaluation were collected through the review of meeting and workshop reports, minutes prepared by facilitators, interviews with teams and facilitators, and analysis of selected service statistics. The following pages provide measurable results related to the two main objectives of the program. It is also important to document the very important qualitative results collected through in-depth interviews and focus group discussions conducted with the program participants during the evaluation phase. The following quotes demonstrate that the project was able to achieve its objectives by increasing enthusiasm, creating team spirit, and willingness to solve problems. The teams also committed to being accountable for their performance. ƒ “Before this program, we were distracted, not only myself, but all of my district work group, we were all going in a different direction.” Dr. Fatma Mohamed Shakatawy, female obstetrician/gynecologist, Aswan Health District Manager ƒ LDPE made me focus. Yes, the problem is there, I select it as a challenge, and deal with it. I identify the steps for the plan, which I can follow, and then I start to align and mobilize, monitor and inspire the people. Thank God, we have benefited from this and achieved good results. This project encouraged me to try to overcome the challenge and achieve results. ” Dr. Suheir Tawfik, Family Planning Manager, Aswan Health District 4 ƒ “The most important thing is to share with the people and the people share with me. Second, I must identify my priorities, before starting any work. Third, I have a plan, and I know what to do exactly in a defined timeline. This is one of the important thing I have learned in my life in general. ” Dr. Abdo El Sweissy, Kom Ombo District Other quotes selected from interviews and group discussions are attached in Annex 1. The underlying assumption of the LDPE was that teaching leadership functions and practices to teams and supporting them in the design and implementation specific performance improvement projects would lead to improved results in health services. This evaluation demonstrates that the ten teams produced moderate to significant results at the service delivery level. The majority of the leadership indicators were achieved as well. Results of the leadership indicators measured are: Select Challenge 100% of the teams identified actual challenges. Scan 50% of the teams collected complete valid data. 50% of the teams collected partial valid data. Focus 100% of the teams prepared written action plans with measurable outputs and a time frame. Align & Mobilize 100% of the teams prepared a written action plan defining human and financial resources needed to implement the plan. Achieve Results 70% of the teams achieved 95% or more of their performance objectives. 10% of the teams achieved 33% of their objectives. 20% of the teams did not demonstrate any progress in achieving their objectives. Inspire Workgroup climate improved dramatically in all ten teams. 80% of the teams selected a new challenge, without prompting. In conclusion, the one-year program was very successful in producing results at the clinic and district levels, improving workgroup climate, creating enthusiasm, and inspiring participants in leadership and performance improvement. Before scaling up the program or transferring it to other countries, some design modifications are recommended. 5 1. BACKGROUND The Leadership Development Program Egypt (LDPE) was a one-year pilot program co-led by the Ministry of Health and Population (MOHP) and the Management and Leadership (M&L) Program of Management Sciences for Health (MSH). It was delivered in three districts of the Aswan Governorate. Forty-one district and clinic managers participated in the program. The LDPE’s goal was: to improve the quality and accessibility of health services in Egypt by increasing the capability of managers to lead others to improve performance of clinic services. The participants worked as teams during bi-monthly leadership workshops and returned to their districts and clinics to implement performance improvement projects aimed at involving staff in addressing their selected challenges. 1.1 Purpose of the LDPE Overall purpose: To improve the quality and accessibility of health services in Egypt by increasing the capability of managers to lead others to achieve results, and to create climates of high performance in their workplaces. Specific purpose: To support district level managers to lead their clinics in continuous performance improvement. 1.2 Objectives of the LDPE 1. Support managers to address the critical challenges in their districts. 2. Improve the capability of district level and clinic-level (doctors and nurses) managers to lead performance improvement projects that address these challenges. 3. Build capacity to monitor and track performance results. 4. Support managers to improve the workgroup climate in their workplaces, resulting in an increased commitment of staff to serving clients and continuously improving services. 1.3 Timeline This was a one-year, core-funded, program beginning June 2002 and ending June 2003. 6 1.4 Program Components of the LDPE 1. Core Team Meetings: The meetings were held at the Ministry of Health and Population (MOHP) in Cairo. Their purpose was to orient MOHP managers in leading and sustaining the LDPE. LDPE facilitators, from the MOHP and from MSH, led these meetings. 2. District Leadership Workshops: The workshops were one to two day-long educational sessions held in the Aswan Governorate in June and October 2002, and January, March, April, and June 2003. The workshops were led in Arabic by the director from the MOHP and a local consultant. They were designed and implemented in collaboration with M&L to teach leadership functions and practices and to support participants in designing and implementing their Performance Improvement Projects. 3. Performance Improvement Projects: The projects were designed by the teams and carried out at their clinics or at the district level. Implementation of the projects was supported through Monthly Meetings led by MOHP managers. 4. Monthly Meetings: The meetings were designed to reinforce the learning from the workshops and support participants in monitoring the progress of their projects. They were led by managers from the MOHP. 5. Team Meetings: The meetings were led by LDPE participants and aimed to involve all clinic staff, not just the LDPE participants, in designing and carrying out the Performance Improvement Projects. MOHP Governorate managers supported these meetings. 7 2. EVALUATION FRAMEWORK 2.1 The Leading and Managing Framework The M&L Program’s Leading and Managing Framework below assumes that improved work climate and management systems will lead to improved health services, thus leading to better health outcomes. Organizational Performance Outcomes Improved Management Systems Effect: Impact: Improved Work Climate (IR2) Climate The LDPE evaluation plan focused on: measuring the managers’ ability to lead others through the development and implementation of an action plan, to create a better workgroup climate, and to achieve service delivery results. The LDPE intended to produce results at the district or clinic level by helping managers to design and implement action plans to address their selected challenge. Like any project, the LDPE projects should have the following basic organizational elements: Goal setting Î Input Î Process Î Output Î Outcome The leadership program was designed around the following essential leadership practices defined by M&L: Selecting Challenge Î Scanning Î Focusing Î Aligning & Mobilizing Î Inspiring To evaluate the LDPE, the two sets of elements were matched and indicators were defined accordingly. Organizational Elements Goal setting Input Process Output Outcome Leadership Elements Select Challenge Scan Focus Align & Mobilize Achieve Results Inspire Improved Health Outcomes Improved Services Managers Who Lead Framework Leading & Managing Scan Plan Focus Organize Align & Mobilize Implement Inspire Monitor & Evaluate 8 2.2 Indicators Seven indicators were defined for the LDPE. The first four (1.1, 1.2. 1.3 and 2.0) are implementation-type indicators, and the latter three (3.0, 4.1, 4.2) are result- type indicators. The summary indicator table is given below: Organizational Element Leadership Element Selected Indicator Data Source Select Challenge 1.1 The team is able to formulate a challenge defined by the whole group Written challenges Scan 1.2 The team can cite all “valid data or evidence” regarding its challenge Availability of data for scanning purposes Goal setting Focus 1.3 The team has a written “action plan” with measurable outputs and a timeframe Availability of an action plan Input & Process Align & Mobilize 2.0 The team has a written “action plan” defining human and financial resources needed to implement the plan. Review of the action plan Output Achieve results 3.0 The team achieved all the measurable outputs specified in the action plan Review of the results Outcome Inspire 4.1 4.2 The workgroup climate has significantly improved The team has identified a new challenge(s) Workgroup Climate Assessment Written challenges The measurement unit for all of the indicators was the clinic or workgroup. Each clinic or workgroup was assessed and scored separately. Based on these separate measurements, an overall score was calculated for each indicator. 9 The definition of each indicator, measurement method, and timing are given below: Indicator 1.1 Organizational Element Goal setting Leadership Element Select challenge Indicator The team is able to formulate a challenge defined by the whole group Measure Score based on a scale of 0 to 2 Data Source Written challenges collected from the groups Timing of the measure Beginning of the Scanning Workshop Score Definition 0 There is no “challenge” defined by the team 1 There is a “challenge” defined but it was not defined and selected with the involvement of whole team 2 There is a “challenge” defined by the team Indicator 1.2 Organizational Element Goal setting Leadership Element Scan Indicator The team can cite all “valid data or evidence” regarding its challenge Measure Score based on a scale of 0 to 2 Data Source Availability of data for scanning purposes Timing of the measure During the monthly meeting following the Scanning Workshop Score Definition 0 There are no “valid data or evidence” regarding the challenge cited by the team 1 The team can cite some “valid data or evidence” regarding its challenge 2 The team can cite all “valid data or evidence” regarding its challenge Indicator 1.3 Organizational Element Goal setting Leadership Element Focus Indicator The team has a written “action plan” with measurable outputs and a timeframe Measure Score based on a scale of 0 to 2 Data Source Availability of an “action plan” with measurable outputs and a timeframe Timing of the measure During the monthly meeting following the Focusing Workshop Score Definition 0 The team does not have a written “action plan” 1 The team has a written “action plan” but it has neither measurable outputs or a timeframe 2 The team has a written “action plan” with measurable outputs and a timeframe 10 Indicator 2.0 Organizational Element Input and process Leadership Element Align and Mobilize Indicator The team has a written “action plan” mentioning human and financial resources needed to implement it Measure Score based on a scale of 0 to 2 Data Source Review of the “action plan” Timing of the measure During the monthly meeting following the Focusing Workshop Score Definition 0 The team does not have a written “action plan” 1 The team has a written “action plan” but it does not define the human and financial resources needed to implement it 2 The team has a written “action plan” defining human and financial resources needed to implement it Indicator 3.0 Organizational Element Output Leadership Element Achieve Result Indicator The team achieved all of the “measurable outputs” mentioned in the action plan Measure Score based on a scale of 0 to 2 Data Source Review of the results Timing of the measure At the end of the project Score Definition 0 The team did not achieve any of the “measurable outputs” mentioned in the action plan 1 The team achieved some of the “measurable outputs” mentioned in the action plan 2 The team achieved all of the “measurable outputs” mentioned in the action plan Indicator 4.1 Organizational Element Outcome Leadership Element Inspire Indicator The workgroup climate of the team has significantly improved Measure Score based on a scale of 0 to 2 Data Source Workgroup Climate Assessment Timing of the measure Before and at the end of the project Score Definition 0 The workgroup climate of the team has not improved or worsened 1 The workgroup climate of the team has improved somewhat 2 The workgroup climate of the team has significantly improved 11 Indicator 4.2 Organizational Element Outcome Leadership Element Inspire Indicator The team has identified a “new challenge” Measure Score based on a scale of 0 to 2 Data Source Availability of written challenges Timing of the measure At the end of the project Score Definition 0 There is no “new challenge” defined by the team 1 There is a “new challenge” but it was not defined and selected with the involvement of whole team 2 There is a “new challenge” defined by the team 12 3. RESULTS The methodologies used to collect data included: review of meeting and workshop reports, minutes prepared by facilitators, interviews with teams and facilitators, and analysis of selected service statistics. The evaluation was carried out by the M&L Monitoring and Evaluation Unit in Egypt during June 15-23, 2003. 3.1 Teams’ ability to “Select Challenges” This pilot LDPE involved three districts (Kom Ombo, Daraw and Aswan), five health centers, and one hospital. In total, ten teams were formed from these nine different groups. Daraw district formed two teams, one for family planning, and one for antenatal care. During the first district workshops and monthly meetings, the teams were encouraged and supported to select a challenge with the involvement of all team members. All ten teams identified one challenge area either for their clinics or districts. Table 1 lists the challenge areas identified by each team. Table 1: Challenge areas identified by district and clinic teams Team District Increase the percentage of FP users Increase the average number of antenatal care visits Increase the average number of postpartum care visits Kom Ombo District Kom Ombo Daraw District Daraw Rakkaba Health Center Daraw Aswan District Aswan Al Aakab Health Center Aswan Nafak Health Center Aswan Daraw Health Center Daraw Gaafra Health Center Daraw Gharb Aswan Hospital Aswan As seen in the table above, five teams selected family planning, three teams selected antenatal care, and two teams selected postpartum care as their priority challenges. These three challenges are congruent with Egyptian national health priorities. Discussion with the LDPE facilitators and meeting notes revealed that all ten teams selected a challenge with the active participation of all team members (Table 2). 13 Table 2: “Selecting a Challenge” indicator scores of the teams Name of the Team Score Kom Ombo District 2.0 Daraw District (FP) 2.0 Daraw District (ANC) 2.0 Rakkaba Health Center 2.0 Aswan District 2.0 Al Aakab Health Center 2.0 Nafak Health Center 2.0 Daraw Health Center 2.0 Gaafra Health Center 2.0 Gharb Aswan Hospital 2.0 Average Element Score 2.0 3.2 Teams’ ability to “Scan” their environment regarding their challenge The teams were expected to cite valid data or evidence regarding their challenges. The Daraw district antenatal care team, Nafaq health center, and Daraw health center selected antenatal care as a challenge, and their action plans clearly indicate the low average antenatal care visits. These three teams were able to cite “valid data or evidence” regarding their challenge. Gaafra Health Center and Gharb Aswan Hospital teams selected to improve postpartum care as their challenge. Both teams were also able to cite “valid data” for their challenge. Kom Ombo, Daraw and Aswan district teams, Rakkaba and Al Aakab Health Center teams selected family planning as the challenge area. At the clinic and district levels there were data available to measure the utilization of family planning services, such as new visits and follow-up visits to the clinics. However, Aswan Governorate, and possibly some other governorates, use an unusual indicator to measure the performance of family planning services at the clinic level. The total number of couple years of protection (CYP) produced in a month is first multiplied by 12 and then divided by the total number of eligible couples, with a percentage then calculated. The five FP teams’ challenge was to increase these percentages by the end of the program. This is not the international standard for measuring family planning performance. However, it appears to be a standard requirement of the MOHP. This oddly calculated measure will not indicate the monthly service performance level of a clinic or district. There are several problems with this approach. First, increasing the percentage does not necessarily reflect an increase in the number of family planning users or coverage. By definition, the CYP measure is biased towards more permanent methods. Moreover, at the clinic level, such a measure may lead to big monthly variations. One additional IUD insertion or removal, or one sterilization may cause significant changes in the percentage figures. The teams were aware of low family planning use in their catchment areas but by using this non-standard indicator, they were not able to properly scan and measure their challenge. Thus these five teams scored less than the others (Table 3). 14 Table 3: “Scanning” indicator scores of the teams Name of the Team Scanning Score Select Challenge Kom Ombo District 1.0 2.0 Daraw District (FP) 1.0 2.0 Daraw District (ANC) 2.0 2.0 Rakkaba Health Center 1.0 2.0 Aswan District 1.0 2.0 Al Aakab Health Center 1.0 2.0 Nafak Health Center 2.0 2.0 Daraw Health Center 2.0 2.0 Gaafra Health Center 2.0 2.0 Gharb Aswan Hospital 2.0 2.0 Average Element Score 1.5 2.0 3.3 Teams’ ability to “Focus” on their selected challenges All ten teams were required to prepare an action plan with measurable outputs and a timeframe regarding their challenge. Table 4 below shows each team’s specific challenges and also baseline and target values. Table 4: Challenges, measurable outputs, and timeframes identified by the teams Baseline Target Team District Selected Challenge Date Value Date Value Kom Ombo District Kom Ombo Increase the percentage of FP users Jan 03 37.8% June 03 42.8% Daraw District Daraw Increase the percentage of FP users Jan 03 51.0% June 03 55.0% Rakkaba Health Center Daraw Increase the percentage of FP users June 02 28.5% June 03 33.5% Aswan District Aswan Increase the percentage of FP users Jan 03 29.2% June 03 30.9% Al Aakab Health Center Aswan Increase the percentage of FP users June 02 37.1% June 03 39.1% Daraw District Daraw Increase the average number of antenatal care visits Jan 03 1.0 June 03 2.0 Nafak Health Center Aswan Increase the average number of antenatal care visits Jan 03 0.5 June 03 2.0 Daraw Health Center Daraw Increase the average number of antenatal care visits Jan 03 0.6 June 03 1.0 15 Gaafra Health Center Daraw Increase the average number of postpartum care visits June 02 0.2 June 03 4.0 Gharb Aswan Hospital Aswan Increase the average number of postpartum care visits Jan 03 0.0 June 03 3.0 Table 4 indicates that Rakkaba and Gaafra Health Centers chose June 2002 as their baseline value date while the other eight teams chose January 2003 as their baseline value date. All ten teams decided to complete their projects by June 2003. All teams prepared very detailed action plans. Review of the ten action plans revealed that all teams understood the concepts of setting measurable outputs and defining a timeframe for their projects. One interesting finding is that the targets set for family planning are very conservative, ranging from 1.7 to 5 percentage points. Due to the variance that may be caused by the non-standard indicator used for monitoring family planning service performance in clinics, these targets do not provide appropriate “focus”. Nevertheless, these teams prepared detailed action plans with measurable outputs and a timeframe. On the other hand, baseline and target values for antenatal care and postpartum care were clear, with a meaningful measurement method. Scores for the “Focusing” indicator are given in Table 5 below. Table 5: “Focusing” indicator scores of the teams Name of the Team Focusing Score Scan Select Challenge Kom Ombo District 2.0 1.0 2.0 Daraw District (FP) 2.0 1.0 2.0 Daraw District (ANC) 2.0 2.0 2.0 Rakkaba Health Center 2.0 1.0 2.0 Aswan District 2.0 1.0 2.0 Al Aakab Health Center 2.0 1.0 2.0 Nafak Health Center 2.0 2.0 2.0 Daraw Health Center 2.0 2.0 2.0 Gaafra Health Center 2.0 2.0 2.0 Gharb Aswan Hospital 2.0 2.0 2.0 Average Element Score 2.0 1.5 2.0 3.4 Teams’ ability to “Align & Mobilize” people and resources A review of the ten action plans revealed that all teams mentioned the resources and people needed to accomplish their challenges. The action plans listed activities planned (specific dates were mentioned for most activities) and persons responsible for those activities. One common feature of the action plans was that none of them mentioned a critical external resource needed. Teams preferred to rely on their own local resources. In fact, almost all 16 obstacles and problems mentioned in the action plans were attributed to internal and/or local factors. Indicator scores for “aligning and mobilizing” are given in Table 6 below. Table 6: “Align & Mobilize” indicator scores of the teams Name of the Team Aligning & Mobilizing Score Focus Scan Select Challenge Kom Ombo District 2.0 2.0 1.0 2.0 Daraw District (FP) 2.0 2.0 1.0 2.0 Daraw District (ANC) 2.0 2.0 2.0 2.0 Rakkaba Health Center 2.0 2.0 1.0 2.0 Aswan District 2.0 2.0 1.0 2.0 Al Aakab Health Center 2.0 2.0 1.0 2.0 Nafak Health Center 2.0 2.0 2.0 2.0 Daraw Health Center 2.0 2.0 2.0 2.0 Gaafra Health Center 2.0 2.0 2.0 2.0 Gharb Aswan Hospital 2.0 2.0 2.0 2.0 Average Element Score 2.0 2.0 1.5 2.0 3.5 Teams’ ability to “Achieve Results” The ten teams’ results are summarized in this section. During the evaluation, service statistics from participating clinics and districts were reviewed by the author together with the teams. Due to the close-out of the project in June 2003, the evaluation had to be conducted before project close. Therefore service statistics for June 2003 were not available at the time of the evaluation. They were compiled and sent to the author by Aswan Governorate staff in August 2003. Antenatal Care Results One district team (Daraw) and two health center teams (Daraw and Nafaq) chose to increase the average number of per client antenatal care visits. No measurement problem or confusion was experienced during the evaluation. Since the beginning, these three teams were clear about their challenge, baseline scores as well as their expected achievements. Results of three teams’ work are given below. 3.5.1 Daraw District The Daraw District team selected the following challenge: ƒ Increase the average number of antenatal care visits per client from 1.0 in January 2003 to 2.0 in June 2003. 17 This challenge covers a six months period. Daraw District achieved an average 2.4 antenatal care visits per client as of the end of June 2003. Figure 1 below shows the historical antenatal care visit performance of Daraw district. F igu re 1: A verag e n u m b er of an ten atal ca r e visits (D araw D istrict) 0 1 2 3 Jan F eb Mar Apr May Jun Jul Aug Sep O c t No v Dec Jan Feb Mar Ap r May Jun Target Actual 2002 2003 Average number of visits The increase in the total number of antenatal care visits in Daraw is significant when compared to the non-program Nasr district. Figure 2 below shows the monthly number of total antenatal care visits in the two districts since the beginning of year 2002. Starting with the implementation of action plans, a sharp increase is seen in Daraw district while the total number of visits in Nasr district does not show significant improvement. 18 Figure 2: Monthly num ber of total antenatal care visits in two districts from January 2002 through June 2003 0 200 400 600 800 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Daraw Nasr (non-program district) Number of Visits Start of LD program Implementation of action plans 2002 2003 3.5.2 Daraw Health Center The Daraw Health Center team selected the following challenge: ƒ Increase the average number of antenatal care visits per client from 0.6 in January 2003 to 1.0 in June 2003. This challenge covers a six months period. Daraw Health Center achieved an average 2.6 antenatal care visits per client as of the end of June 2003. Figure 3 below shows the historical antenatal care visit performance of Daraw Health Center. 19 0 1 2 3 Jan Feb Mar Ap r May Jun Jul Aug Sep O c t Nov Dec Jan F eb Mar Ap r May Target Actu al 2002 2003 Figure 3: Av erage num ber of antenatal care v isits (D araw H ealth C enter) Average number of visits It should be noted that the Daraw Health Center team selected a very conservative target. In fact, the average number of per client visits was 0.8 in 2002. In six of 12 months of 2002, the Daraw Health Center had already reached or exceeded an average of 1.0 visits per client. By the end of its project in June 2003, the team had far exceeded its performance goal. 3.5.3 Nafak Health Center The Nafak Health Center team selected the following challenge: ƒ Increase the average number of antenatal care visits per client from 0.5 in January 2003 to 2.0 in June 2003. This challenge covers a six month period. Nafak Health Center achieved an average 1.9 antenatal care visits per client as of the end of June 2003. Although the team fell short of its goal, a significant improvement in the clinic’s performance was seen, as compared to the before the LDPE. 20 Figure 4 below shows the historical antenatal care visit performance of Nafak Health Center. 0 1 2 3 Jan Feb Mar Ap r May Jun Jul Aug Sep O c t Nov Dec Jan Feb Mar Ap r May Jun Target Actu al 2002 2003 Figure 4: A verage num ber of antenatal care visits (N afak H ealth C enter) Average number of visits In conclusion, all three teams that chose to increase antenatal care visits reached their targets. The Daraw Health Center demonstrated the most outstanding result in a very short period of time. Postpartum Care Results Two teams (Gaafra Health Center and Gharb Aswan Hospital) selected to increase the average number of postpartum care visits per client. As with the antenatal care groups, no measurement problem or confusion was experienced during the evaluation. Since the beginning, these two teams were clear about their challenge, baseline scores as well as their desired achievements. Performance results for these two teams are given below. 3.5.4 Gaafra Health Center The Gaafra Health Center team selected the following challenge: ƒ Increase the average number of postpartum care visits per client from 0.2 in June 2002 to 4.0 in June 2003. 21 This challenge covers a one year period. The Gaafra Health Center achieved an average 3.6 postpartum care visits per client as of the end of June 2003. Thus they reached 90% of their target by the end of the action plan. Figure 5 below shows the historical postpartum care visit performance of Gaafra Health Center. Figure 5: Av erage num ber of postpartum care visits (G aafra H ealth C enter) 0 1 2 3 4 5 6 Jan F eb Mar Apr May Jun Jul Aug S ep O ct No v Dec Jan F eb Mar Ap r May Jun Target Actu al 2002 2003 Average number of visits Since the beginning of July 2002, the initiation of the LDPE but prior to implementation of their action plan, the Gaafra Health Center was able to maintain the average number of postpartum care visits per client between 3.0 and 4.0. 3.5.5 Gharb Aswan Hospital The Gharb Aswan Hospital team selected the following challenge: ƒ Increase the average number of postpartum care visits per client from 0 in January 2003 to 3.0 in June 2003. This challenge covers a six month period. Although Gharb Aswan Hospital achieved an average 2.6 postpartum care visits per client as of the end of April 2003, and then 5.0 as of the end of May, in June 2003 this figure dropped to an average of 1.0 visits per client. Figure 6 below shows the historical postpartum care visit performance of Gharb Aswan Hospital. 22 0 1 2 3 4 5 6 Jan F eb Mar Ap r May Jun Jul Aug S ep O c t No v Dec Jan F eb Mar Ap r May Jun Target Actu al 2002 2003 Figure 6: Av erage num ber of postpartum care visits (G harb A sw an H ospital) Average number of visits Although there was a decline in June 2003, Gharb Aswan Hospital demonstrated an outstanding result in terms of postpartum care. In 2002 there was no postpartum care at this hospital. Until March 2003 no visits were recorded. Results show that the hospital was able to exceed even their own ambitious target, although they were not able to sustain the effort. Family Planning Results As mentioned earlier, the biggest problem was measuring the results of the five teams that selected family planning as a challenge area. Although these teams insisted on reporting the indicator they routinely used to measure performance, basic service statistics, such as new family planning users and follow-up visits, were also reviewed and reported. Prior to the evaluation, Nasr District of Aswan Governorate was identified as a comparison site for better assessing the performance of Kom Ombo, Daraw and Aswan Districts. Nasr District has not received any significant technical assistance in family planning services in the last year. The three districts’ family planning service performance was compared with Nasr District. Their performance was also examined by referring to statistics from the previous years’ period, January-April 2002. 3.5.6 Kom Ombo District The Kom Ombo District team selected the following challenge: 23 ƒ Increase the percentage of family planning users from 37.8% in January 2003 to 42.8% in June 2003. This challenge covers a six month period. Based on the CYP/target population indicator, Kom Ombo District reached 36.5% by the end of April 2003. (The district has not yet provided M&L with data for the months of May and June.) Kom Ombo District fell below the baseline value at the end of their project. On the other hand, using service statistics, Figure 7 below shows that there was an average 20% increase in new users and follow-up visits in Kom Ombo compared to the same period last year. However, compared to Nasr District, this increase was not found to be significant. Figure 7: Percentage change in family planning visits in Kom Ombo and Nasr Districts from January-April 2002 and to January-April 2003 0 20 40 60 80 100 New F P Visits Follow-up FP Visits Total Number of FP Visits Kom Ombo Nasr (Comparison) Percent 3.5.7 Daraw District The Daraw District team selected the following challenge: ƒ Increase the percentage of family planning users from 51.0% in January 2003 to 55.0% in June 2003. Similar to Kom Ombo, this challenge covers a six month period. Based on the CYP/target population indicator Daraw District reached 49.7% as of the end of April 2003. According to this calculation, Daraw District team also fell below the baseline value at the end of their project. (Again, data for the months of May and June are not yet available to M&L.) Although the CYP-based indicator shows a failure, an analysis of service statistics summarized in Figure 8 below shows that compared to the same period last year, there was a 24 68 percent increase in the number of new visits in Daraw District. Similarly, follow-up visits increased by 59 percent. Compared to Nasr District, both increases were found to be significant. This is a good example of how misleading the CYP-based indicator is. 0 20 40 60 80 100 New FP Visits Follow-up FP Visits Total Number of FP Visits Daraw Nasr (Comparison) Percent Figure 8: Percentage change in family planning visits in Daraw and Nasr Districts from January-April 2002 to January-April 2003 3.5.8 Aswan District The Aswan District team selected the following challenge: ƒ Increase the percentage of family planning users from 29.2% in January 2003 to 30.9% in June 2003. The Aswan District team’s challenge covers a six month period. Based on the CYP/target population indicator, Aswan District reached 34.0% as of end of April 2003. The Aswan District team exceeded their target, at least as of the end of April 2003. Again, based on an analysis of service statistics, Figure 9 below shows that compared to the same period last year, there was a 36 percentage point increase in the number of new visits in Aswan District. Follow-up visits also increased by 53 percent. Compared to Nasr District both increases were found to be significant. 25 0 20 40 60 80 10 0 New FP Visits Follow-u p FP Visits Total Number of FP Visits Aswan Nasr (Compariso n) Percent Figure 9: Percentage change in fam ily planning visits in Asw an and Nasr Districts from January-April 2002 to January-April 2003 Figure 10 below shows family planning visits (new and follow-up visits) for the three districts that participated in the program: Daraw, Aswan, and Kom Ombo, with Nasr district for comparison, as of the end of April 2003. Nasr district was selected as a comparison because it had had no susbtantial FP interventions during that period. FP visits in Daraw and Aswan Districts did not show a significant change during the year 2002. However, beginning with the implementation of action plans in January 2003, a significant increase in the number of FP visits can be seen in Daraw and Aswan districts while a moderate increase can be seen in Kom Ombo district. In addition, a moderate increase can be seen in the number of FP users in Nasr District, which may be attributed to the effective leadership practices of the Aswan Governorate staff, who participated in the LDPE program. 26 Figure 10: M onthly num ber of total FP v isits in four districts from January 2002 through April 2003 0 500 10 00 15 00 20 00 25 00 Jan Feb Mar Ap r May Jun Jul Aug S ep O c t Nov Dec Jan Feb Mar Apr Aswan Daraw Kom Ombo Nasr Number of Visits (Monthly average) Start of LD program Im p lem entation of action p lans 2002 2003 3.5.9 Rakkaba Health Center The Rakkaba Health Center team selected the following challenge: ƒ Increase the percentage of family planning users from 27.2% (six months average) in June 2002 to 33.5% in June 2003. Although action plans were prepared for a six month period, the Rakkaba Health Center team’s challenge covers a period of twelve months. Table 7 below shows the FP service performance based on CYP/target population indicator. According this indicator Rakkaba Health Center team exceeded its target. Table 7: FP service performance (CYP/Target Population) of Rakkaba Health Center 2002 % Months 2003 % 31.0 January 33.7 23.4 February 43.6 28.0 March 45.4 26.0 April 41.7 34.0 May 42.6 31.0 June 22.6 23.0 July Average for the first 6 months 27.2% 2.4 August Average for the first 6 months 38.3% 27 6.4 September 12.6 October 13.5 November 26.6 December Comparison of the first six months of 2002 and 2003 reveals that there is a difference in the average family planning service performance of Rakkaba Health Center, however it is not statistically significant (p=0.087). Figure 11 below shows the historical service performance of Rakkaba Health Center based on CYP/Target population indicator since the beginning of 2002 and as of the end of May 2003. There is a sharp upward movement starting in January 2003. Figur e 11: Fam ily pla nning se rv ice pe rform anc e of R akkaba H ea lth C ente r 0 20 40 60 80 100 Jan Feb Mar Ap r May Jun Jul Aug S ep O c t No v Dec Jan Feb Mar Ap r May Percentage (CYPx12 months/number of eligible couples) Start of LD program Im p lem entatio n of a ctio n plan 2002 2003 28 3.5.10 Al Aakab Health Center The Al Aakab Health Center team selected the following challenge: ƒ Increase the percentage of family planning users from 37.2% (six months average) in June 2002 to 39.1% in June 2003. Similar to the Rakkaba Health Center, the Al Aakab Health Center team’s challenge covers a period of twelve months. It should be noted that the team’s target is very conservative. Table 8 below shows the FP service performance based on CYP/target population indicator. According this indicator, the Al Aakab Health Center team exceeded its target. Table 8: FP service performance (CYP/Target Population) of Al Aakab Health Center 2002 % Months 2003 % 43.2 January 47.2 21.6 February 52.4 55.8 March 47.2 34.2 April 69.0 30.5 May 61.9 37.8 June 51.3 64.8 July 32.4 August 43.2 September 39.6 October 45.0 November Average for the first 6 months 37.2% 54.0 December Average for the first 6 months 54.8% A comparison of the the first six months of 2002 and 2003 reveals that there is a difference in the average family planning service performance of Al Aakab Health Center but this difference was not found to be statistically significant (p=0.057). Figure 12 below shows the historical service performance of Al Aakab Health Center based on CYP/Target population indicator since the beginning of 2002. There are sharp upward and downward movements throughout the 18- month period. 29 Figure 12: Fam ily planning serv ice perform ance of A l A akab H ealth C enter 0 20 40 60 80 10 0 Jan Feb Mar Ap r May Jun Jul Aug Sep O c t Nov Dec Jan Feb Mar Apr May Percentage (CYPx12 months/number of eligible couples) Start of LD program Im p lem entation of action p lan 2002 2003 In this section, the five teams’ performance regarding family planning services has been examined. Due to the poor quality of the performance indicator selected by these teams, it is hard to conclude whether or not they have succeeded. Surely selecting a challenge and preparing an action plan to meet a target created a momentum and enthusiasm among these teams. Daraw district, for instance, was able to increase the percent of new FP users by 68%. And Rakkaba Health Center made very good improvement compared to same period last year. But because of the performance indicator used by the team, and despite the author’s in￾depth review of family planning service statistics, it was not possible to fully measure and compare the actual performance of these teams against the baseline and/or their intended performance. It can be concluded that these five teams achieved only some of their measurable outputs. Table 9 below summarizes the results for the ten teams. It should be noted that the FP results are based on the CYP/Target population indicator. Five teams (50%) achieved or exceeded 100% of their targets; two teams (20%) achieved 95% of their targets; one team achieved 33% of its target; and two teams (20%) did not demonstrate any achievement. In conclusion, 70% of the teams achieved 95% or more of their targets. Given the short duration of the action plans, this result should be noted as a remarkable outcome of the LDPE. 30 Table 9: Summary of teams’ results Name of the Team Challenge Baseline Target Result Achievement Kom Ombo District Increase the percentage of FP users 37.8% 42.8% 36.5% None (below baseline) Daraw District Increase the percentage of FP users 51.0% 55.0% 49.7% None (below baseline) Daraw District Increase the average number of antenatal care visits 1.0 2.0 2.4 100% Rakkaba Health Center Increase the percentage of FP users 27.2% 33.5% 38.3% 100% Aswan District Increase the percentage of FP users 29.2% 30.9% 34.0% 100% Al Aakab Health Center Increase the percentage of FP users 37.2% 39.1% 54.8% 100% Nafak Health Center Increase the average number of antenatal care visits 0.5 2.0 1.9 95.0% Daraw Health Center Increase the average number of antenatal care visits 0.6 1.0 2.6 100% Gaafra Health Center Increase the average number of postpartum care visits 0.2 4.0 3.8 95.0% Gharb Aswan Hospital Increase the average number of postpartum care visits 0.0 3.0 1.0 33.3% Table 10 below presents the “achieve results” indicator scores. Due to the unconventional indicator used to calculate FP results, it is not possible to fully prove the achievements of the five FP teams. Therefore, the five FP teams’ scores are given as 1.0, meaning that the teams achieved some of their measurable targets. The two postpartum care teams also achieved some of their measurable targets and received a score of 1.0. One of the antenatal care teams reached its target and scored 2.0, while the other antenatal care team reached 95.0% of its target and scored 1.0. Table 10: “Achieve Results” indicator scores of the teams Name of the Team Achieving Results Score Align & Mobilize Focus Scan Select Challenge Kom Ombo District 1.0 2.0 2.0 1.0 2.0 Daraw District (FP) 1.0 2.0 2.0 1.0 2.0 Daraw District (ANC) 2.0 2.0 2.0 2.0 2.0 Rakkaba Health Center 1.0 2.0 2.0 1.0 2.0 Aswan District 1.0 2.0 2.0 1.0 2.0 Al Aakab Health Center 1.0 2.0 2.0 1.0 2.0 Nafak Health Center 1.0 2.0 2.0 2.0 2.0 31 Daraw Health Center 2.0 2.0 2.0 2.0 2.0 Gaafra Health Center 1.0 2.0 2.0 2.0 2.0 Gharb Aswan Hospital 1.0 2.0 2.0 2.0 2.0 Average Element Score 1.2 2.0 2.0 1.5 2.0 3.6 Improved Climate Workgroup climate in the ten teams was measured using the M&L Program’s Workgroup Climate Assessment (WCA) tool. The tool was administered in June 2003, at the end of the program. The 35 members1 of the 10 teams were asked to assess their workgroup climate retroactively for the start of the program and also for the end of the program. Although the evaluation plan intended to measure workgroup climate at the beginning of the LDPE, due to technical and time constraints a real baseline could not be obtained at the beginning of the program. Thus the scores from program start could be skewed negatively. The 14 items measured in the Workgroup Climate Assessment are: ƒ We are recognized for individual contributions ƒ We feel we have a common purpose ƒ We have the resources to do our jobs well ƒ We develop our skills ƒ We have a plan which guides our activities and knowledge ƒ We strive to improve our performance ƒ We understand each other’s capabilities ƒ We are clear what is expected in our work ƒ We seek to understand the needs of our clients ƒ We participate in decisions that affect the workgroup ƒ We take pride in our work ƒ We readily adapt to new circumstances ƒ Our workgroup meets quality standards ƒ Our workgroup is productive Using a Likert scale of 1 to 5 (1 meaning “not at all”, 5 meaning “to a very great degree”), district and clinic managers and other team members were asked to score the importance of each item, to retroactively score the baseline values, and to score the end of project values. Table 11 below summarizes the scores obtained from eight managers. Table 11: Managers’ Workgroup Climate Assessment Scores Importance Baseline Close-out Gap We are recognized for individual contributions 4.1 2.3 4.0 0.1 We feel we have a common purpose 4.9 1.6 4.6 0.3 We have the resources to do our jobs well 4.1 2.1 4.3 -0.1 We develop our skills and knowledge 4.5 1.6 4.4 0.1 We have a plan which guides our activities 4.5 1.5 4.3 0.3 1 Although the total number of program participants was 41, only 35 participants were present during the application of the WCA tool. 32 We strive to improve our performance 4.6 2.1 4.0 0.6 We understand each other’s capabilities 4.3 2.1 4.0 0.3 We are clear what is expected in our work 4.8 1.9 4.8 0.0 We seek to understand the needs of our clients 4.1 2.0 4.1 0.0 We participate in decisions that affect the workgroup 4.8 1.5 4.9 -0.1 We take pride in our work 4.8 2.9 4.9 -0.1 We readily adapt to new circumstances 4.6 2.4 4.6 0.0 Our workgroup meets quality standards 1.9 4.5 Our workgroup is productive 2.4 4.6 Table 12 below summarizes the scores obtained from 27 team members. Table 12: Team members’ Workgroup Climate Assessment Scores Importance Baseline Closeout Gap We are recognized for individual contributions 4.4 2.0 4.2 0.2 We feel we have a common purpose 4.6 2.1 4.7 0.0 We have the resources to do our jobs well 4.4 2.3 4.1 0.3 We develop our skills and knowledge 4.6 2.0 4.3 0.3 We have a plan which guides our activities 4.6 2.1 4.6 0.0 We strive to improve our performance 4.8 2.7 4.7 0.1 We understand each other’s capabilities 4.4 2.0 4.2 0.2 We are clear what is expected in our work 4.5 1.8 4.6 -0.1 We seek to understand the needs of our clients 4.5 2.2 4.3 0.2 We participate in decisions that affect the workgroup 4.5 1.9 4.4 0.0 We take pride in our work 4.6 3.1 4.8 -0.2 We readily adapt to new circumstances 4.5 2.5 4.2 0.3 Our workgroup meets quality standards 2.4 4.3 Our workgroup is productive 2.7 4.7 As may be seen in both tables, all managers and team members scored almost all items quite low at the beginning of the project. The improvement between the baseline and the closeout scores is found to be statistically significant (p=0.000). The gaps between the importance and the actual closeout scores were very narrow for all items. Figure 13 below summarizes the importance, baseline and closeout scores of all groups. 33 Figure 13: A verage change in the w orking clim ate 0 1 2 3 4 5 Im po rta n c e B a s e lin e C los e o u t M a na ge rs' A ve ra ge R a ting Team M emb ers' A ve rage R a ting n=35 Based on these findings, the “improved climate” indicator scores for the ten teams are given in Table 13 below. Table 13: “Improved Climate” indicator scores of the teams Name of the Team Climate Score Achieve Results Align & Mobilize Focus Scan Select Challenge Kom Ombo District 2.0 1.0 2.0 2.0 1.0 2.0 Daraw District (FP) 2.0 1.0 2.0 2.0 1.0 2.0 Daraw District (ANC) 2.0 2.0 2.0 2.0 2.0 2.0 Rakkaba Health Center 2.0 1.0 2.0 2.0 1.0 2.0 Aswan District 2.0 1.0 2.0 2.0 1.0 2.0 Al Aakab Health Center 2.0 1.0 2.0 2.0 1.0 2.0 Nafak Health Center 2.0 1.0 2.0 2.0 2.0 2.0 Daraw Health Center 2.0 2.0 2.0 2.0 2.0 2.0 Gaafra Health Center 2.0 1.0 2.0 2.0 2.0 2.0 Gharb Aswan Hospital 2.0 1.0 2.0 2.0 2.0 2.0 Average Element Score 2.0 1.2 2.0 2.0 1.5 2.0 3.7 Teams’ ability to “Select a New Challenge” The final but one of the most important indicators used to assess the impact of the LDPE program was whether the teams selected a new challenge at the completion of their action plans. During 34 the last team meeting in May 2003, the LDPE facilitators asked each team whether they had selected a new challenge. A form was designed and used for this purpose. Eight out of the ten teams selected a new challenge without prompting. Six teams decided to focus attention on their “old” challenge. This may be due to the limited time spent implementing their existing action plans. Instead of halting the implementation at the end of the program these six teams showed a desire to expand their action plans, indicating a commitment to their challenges. The new challenge areas and the old challenge areas are given in Table 14 below. Table 14: “New and Old Challenge” areas identified by program teams Team Old Challenge Area New Challenge Area Kom Ombo District Increase the percentage of FP users Developing leadership skills in more clinics Increase the percentage of FP users Daraw District (FP team) Increase the percentage of FP users Increase the percentage of FP users Daraw District (ANC team) Increase the average number of antenatal care visits None selected Rakkaba Health Center Increase the percentage of FP users Increase the percentage of FP users Aswan District Increase the percentage of FP users Conduct in-service training courses on FP Al Aakab Health Center Increase the percentage of FP users Increase the percentage of FP users Nafak Health Center Increase the average number of antenatal care visits Increase the average number of antenatal care visits Daraw Health Center Increase the average number of antenatal care visits Increase the average number of antenatal care visits Gaafra Health Center Increase the average number of postpartum care visits None selected Gharb Aswan Hospital Increase the average number of postpartum care visits Increase the percentage of FP users The new challenge indicator scores for the teams are given in Table 15 below. Table 15: “New Challenge” indicator scores of the teams Name of the Team Select New Challenge Score Climate Achieve Results Align & Mobilize Focus Scan Select Challenge Kom Ombo District 2.0 2.0 1.0 2.0 2.0 1.0 2.0 Daraw District (FP) 2.0 2.0 1.0 2.0 2.0 1.0 2.0 Daraw District (ANC) 0.0 2.0 2.0 2.0 2.0 2.0 2.0 35 Rakkaba Health Center 2.0 2.0 1.0 2.0 2.0 1.0 2.0 Aswan District 2.0 2.0 1.0 2.0 2.0 1.0 2.0 Al Aakab Health Center 2.0 2.0 1.0 2.0 2.0 1.0 2.0 Nafak Health Center 2.0 2.0 1.0 2.0 2.0 2.0 2.0 Daraw Health Center 2.0 2.0 2.0 2.0 2.0 2.0 2.0 Gaafra Health Center 0.0 2.0 1.0 2.0 2.0 2.0 2.0 Gharb Aswan Hospital 2.0 2.0 1.0 2.0 2.0 2.0 2.0 Average Element Score 1.6 2.0 1.2 2.0 2.0 1.5 2.0 36 4. CONCLUSIONS The Leadership Development Program in Egypt has provided valuable experience and lessons that will be used in other M&L projects. The program was one of the first to use the Leading and Managing Framework developed by the M&L Program. The underlying assumption of the LDPE was that teaching leadership functions and practices to teams and supporting them in the design and implementation of specific performance improvement projects would lead to improved results in health services. The evaluation of this program intended to serve both formative and summative purposes. While LDPE progress was monitored on an ongoing basis, this final evaluation addressed the following questions: • did the teams’ projects produce their intended results? • did workgroup climate among the teams improve? • did the teams select new challenges? As mentioned in the results section, the teams produced moderate to significant results at the service delivery level, and workgroup climate improved significantly. Most of the leadership elements also scored satisfactorily. Reviewing the LDPE’s performance with respect to its objectives, the following can be concluded: 1. Support managers to address the critical challenges in their districts. This objective was fully achieved. Not only were managers able to address challenges in their districts, but the other program participants were able to address challenges in their districts or clinics. 2. Improve the capability of district level and clinic-level (doctors and nurses) managers to lead performance improvement projects that address these challenges. This objective was fully achieved. The teams were assisted in preparing and leading performance improvement projects. They were able to communicate with and mobilize the other members of the clinics and districts. Despite the short implementation period, the teams demonstrated that they had built capacity to lead their projects. 3. Build capacity to monitor and track performance results. All the teams created a basic monitoring system for their own performance improvement projects and the LDPE was able to engage them to monitor and report performance results. Although some of the teams could not identify the correct measurement for their specific projects, all of them adopted the monitoring concept and demonstrated the importance of measurement in order to improve. This objective was partially achieved. 4. Support managers to improve the workgroup climate in their workplaces, resulting in an increased commitment of staff to serving clients and continuously improving services. This objective was fully achieved. A significant change in the workgroup climate was 37 measured and documented. Observation of the teams during the evaluation period indicates that this change could be sustained for a long period of time. Replicating the LDPE at the central level The scope of this program involved micro-planning, meaning that all ten action plans were designed to improve services at the clinic level. Their focus was operational, not strategic. This design was perfectly appropriate for the district level. On the other hand, organization- wide or sector-wide planning and producing results at the organizational or sector level — macro￾planning — will require a different design. However, the program’s overall approach and purposes — increasing the capability of managers to lead others to achieve results, and managers’ ability to create climates of high performance in their workplaces, remain valid at a higher health system level. Replicating the LDPE across governorates The results clearly indicate that the model used in Aswan Governorate merits replication across the governorates of Egypt. It should be kept in mind that for county-wide implementation, a central training and project management team needs to be developed first. 38 5. RECOMMENDATIONS It is important to mention several features of the LDPE that were observed. These observations may help to better understand the effects created by the program as well as to better design future programs. Working with front-line employees The program involved front-line employees at the district and clinic levels. Since these employees are closest to the client, they were able to successfully identify root causes for their challenges, identify resources needed, and practically design and implement action plans. Historically, they are expected to implement the policies and action plans designed by others at higher levels of the hierarchy. This places such employees in a more passive role. One of the goals of the LDPE was to empower the participants to take on a more active role, to introduce the concept of “leading at all levels”. All team members were able to monitor and act on the same challenge. This allowed the teams to direct their energy toward the same goal. This was an effective way of making their energy most productive. Team members shared the same understanding of the challenges and the action plans to overcome those challenges. They had a better and more accurate perspective on identifying root causes and improving processes. Selecting a limited number of challenges The program avoided using too many indicators for which to hold employees accountable. If people are given too many challenges, they quickly suffer from overload. Primary health care systems in most developing countries are plagued by an overabundance of vertical health programs. Each of these programs has its own objectives, targets, indicators, and measurement and reporting requirements. Employees at the district and clinic levels are required to do it all. The LDPE provided the opportunity to focus on challenges one at a time. All the selected challenges fell into one of the three national program priorities. While focusing on their number one challenge, teams were also able to pursue national program goals. Using appropriate performance measures In this project only two types of performance indicators were used: ƒ Results-oriented indicators: The teams selected challenges that provided measurable service performance indicators. The intention of each team was to deliver results based on its challenge. These service indicators helped to measure whether the teams’ efforts ended in the desired performance. ƒ Implementation-oriented indicators: these indicators, mainly the leadership indicators, helped to monitor the teams’ ability to understand and apply certain procedures and 39 practices. These indicators were expected to provide feedback to the M&L Program on how well the M&L Framework can be utilized. The M&L Program’s Leading and Managing Framework assumes a causal effect between improved work climate and management systems and improved health services and better health outcomes, but this has not yet been fully proven. Scanning, focusing, aligning and mobilizing functions can be performed correctly but may not yield desired results or sustained results, as seen in some of the results reported here. The teams were able to select their own performance indicators. There were pros and cons of this approach. The most important advantage was that it helped to empower the teams that in turn created the energy and willingness to own a challenge. The obvious disadvantage was that it created a “measurement validity” problem. By definition “a measure is valid to the extent that it measures what it is intended to measure”. Five of the LDPE teams chose to use a measure that is not the conventional measure for calculating family planning service performance. Providing support to the teams The LDPE was very successful in terms of creating a safe and productive environment for the teams to perform. Participants were supported to speak up, share ideas, and contribute to their teams. LDPE helped teams to own their challenges and act upon them. MSH staff, local consultants and MOHP counterparts both at the central and governorate levels effectively and continuously provided this help. Bi-monthly workshops and monthly district meetings were very effective and timely tools to regularly support teams. Implementing the action plans Although the LDPE started in June 2002, it was in January 2003 that the teams completed their action plans. There were only six months available for action plan implementation. This short period was mainly due to the long preparation phase. Three or four bi-monthly workshops for a twelve month project left little room for proper and sustained action. The limited absorptive capacity of the teams might have been the reason for this strategy, but a different design is needed for future programs to allow the teams adequate time for focused attention on their challenges. The evaluation framework intended to measure the “aligning and mobilizing” ability of the teams through the availability of written “action plans” defining the human and financial resources needed to implement them. The next measurement was whether the teams had achieved the “measurable outputs” specified in the action plan. What was not measured was to what extent the action plans were implemented. The evaluation framework assumed that the action plans were partially or fully implemented if the teams had achieved some or all of the outputs. Clearly, the evaluation framework needs to be improved to capture this important step. Providing appropriate technical assistance 40 In relation to the observations above, the quality of action plans was another critical issue. Each team chose to increase either family planning or antenatal care or postpartum care visits in their districts and clinics. The activities and resources mentioned in the action plans assumed that those were the key ingredients for improving those services. One of the two purposes of the program was to increase “the ability of managers to create climates of high performance in their workplaces”. The results achieved by the teams demonstrate that project was able to achieve this objective by increasing enthusiasm, creating team spirit and willingness to solve problems. The teams also committed to being accountable for their performance. All of these changes were easily observable at the end of the project. But these achievements might not represent the full potential of the teams. The teams’ lack of good training/knowledge in the selected health programs was an important gap. Better technical knowledge on selected health programs could have led to even better results. The teams would have benefited from access to state-of-the-art knowledge on these health programs. Being located in Aswan Governorate, the teams are distant from the national and international community. Technical components of these services, historical background, success or failure stories from around the world should have been communicated and discussed with the teams. The LDPE was not staffed with local personnel who had knowledge about national and international experience. It is important to fulfill MSH’s mission, which is “closing the gap between what is known about public health programs and what is done to solve them”. An intermediate step should be added to the program design in the future. After the selection of challenges, program staff should be provided with technical information on those health programs. The rationale for selecting those challenges, how to address the challenges, best practices from both national and international experience could be the main topics of such assistance. Results also indicate that the teams needed technical assistance in setting better targets. Several teams chose targets that were too modest. The planned change could be due to monthly or seasonal variation or measurement error. Such a perspective and technical assistance provided to program participants would definitely improve the action plans and might yield better results. Improving scanning skills by access to international knowledge through the internet could also be a solution that can be explored. Application of the Workgroup Climate Assessment Measuring the workgroup climate adds value to the program. The positive change in the climate seemed to be an important ingredient of the program. The author attended one of the first and the final district meetings and witnessed the dramatic change in the participants’ attitudes, their level of participation, and their increased sense of ownership to the program. In particular, the submissive and quiet nature of the nurses displayed at the beginning of the program had totally disappeared, replaced by a vocal and attentive behavior. Measuring, comparing and documenting the change in workgroup climate is important for the M&L Program. Thus all future Leadership Development Programs should apply the workgroup climate assessment tool at baseline and close-out. 41 Regardless of the technical issues mentioned above, it should be noted that LDPE was managed with dedication, great energy and contagious enthusiasm which led to significant changes in the participants’ lives in terms of facing problems, accepting challenges, and committing to producing results. The M&L Program and its current and future clients will greatly benefit from the valuable experience and know-how of the LDPE experience. 42 ANNEX 1 QUALITATIVE RESULTS FROM INTERVIEWS AND FOCUS GROUP DISCUSSIONS Participant responses about the Leadership Development Program of Egypt Aswan Governorate From interviews conducted in June 2003 after the program ended. Dr. Fatma Mohamed Shakatawy, Female ObGyn, Aswan Health District Manager (3 years in the job) Teamwork and focus… “Before this program, we were distracted, we were all going in a different direction. but we are now one team. All of us-- in FP, PHC or immunizations – there is no difference. We all have the same target, which is to improve the level of performance in the district.” Taking responsibility… We used to go inspect on people, telling them only what they are doing wrong, Now we assist people, put them on the right track, to identify their problem and to solve it, not to look at me for solutions, but to find the solutions from within themselves as they are the most aware of their problems and how to solve it.” Continuous improvement… It is not just a program or a project, which has ended in one year, but we have started to implement what we learned in other units. We are showing the doctors how to work and improve their performance. I am supervising by letting them to do their work, not according to my way, but their own way, and I have had good results. The PHC indicators used to be very bad. Now, thank God, they are very good. I have immunizations over 95%. When I took over the district, it was only 60-65%, now it exceeds 95%..” -- Dr. Mohamed Sorour, FP Director Komombo District Facing challenges… “I sat with the work group and explained to them: If we are doing our work because we only care for the money, or we are afraid of facing the problems, or we fear that a supervisor might discover any mistake during an inspection and that will put us in trouble -- what is destined will happen, no matter what. Therefore the will to face challenges has to come from within oneself, and we have to be honest with ourselves. Identifying root causes… “I requested my work group to meet with the women (patients) and ask them about the problems and negative aspects here with regards to antenatal care. What is the problem? Why do they abstain from coming? So they started to befriend the women, talking to them, gradually strengthening their relationship. 43 The problem was usually caused by a specific issue or incident – like that the doctor when he met the woman would ignore what she is telling him, sometimes not even allowing her to talk, and the examination would only take two minutes, before dismissing her. Therefore, we started to put our finger on what drives the women away. Their information, socialization with the people, and their love to improve their performance and to improve the quality of services provided, has helped us to identify the root causes of the problems and how to solve them.” Deciding what can be fixed … “We can only change what is within our hands. When the woman visits the center, she should not be treated like a machine, ordered around. Come! Sit! Go! She should be treated with respect, because she is a human being. When you meet her, as a nurse, you should deal with her slowly and calmly, even if the place is crowded and there are many clients waiting. We should have more patience in order not to lose our clients. The doctor, also, should take care of her, check her pressure, examine her, allow her to speak her mind, and not shut her up. We discovered that if we improved ourselves, treated people right, provided them with an improved service, then we will be excellent, regardless of the trends and rituals prevalent, or the beliefs in the peoples’ minds, which needs a long time to change.” About the L&M Functions… “With regards to the Leading functions, Scanning affected me the most. When I am facing a problem or a challenge, I have to look deeply, not superficially, at the problem to find the rroot causes. If I can’t identify the root causes or the circumstances, I will not be able to find the solution. From the Managing functions, I think planning is the most important. Everyone might be competently doing their work, and they might want to improve. Unfortunately, even if they are killing themselves at work, without planning, the result is zero, because planning is the most important factor of making any work a success. Team spirit… I used to worry about all the work details, I felt that I had to do and check everything myself. Now, this has changed, there is a team spirit. After everyone accepted the new concept-- a month or two in the program, by the fourth month-- there was a new belief. And because this belief was in each individual, they did their utmost, even if their role was small. They came to understand that the no matter what there role is; it is essential to achieve the desired results. There was a team spirit and everyone was cooperative. I now feel, thank God, that I have several arms, I don’t have to do everything myself anymore.” How leaders can shift the work climate… I made each person feel that his role is important. I started to make them feel that it is not important to accomplish my work alone. It is my duty, as a team, after completing my work to help the other team members solve any problem they have, because it is not my work alone that will achieve the desired results, but all our work. I have to have the other team members in my sight andI don’t hesitate to assist.” 44 Enabling others to raise their performance… An individual alone is nothing, but if he is in a work group or there is a team helping him, then he can achieve everything. The problem doesn’t lie in the quality of work the individual performs, even if it exceptional, but in helping others perform their work with the same quality. Supervision or no supervision, if one doesn’t do his work based on a personal conviction or belief, then the work will not succeed.” “If I talked to you for two days, I wouldn’t be able to tell you all that I learned! I learned how to think in an organized manner, to look around me. Also, I learned that I don’t have all the knowledge, and that others, whatever their positions or roles, are capable to come up with great ideas, much better than what I came up with. My work does not end when I do it well. Others should also do their work well. Now, when I go to supervise someone, this does not mean that I pick on his mistakes, but that I go to help and support him. MS. NAGWA IBRAHIM MOHAMED, FP NURSE (YOUNG NURSE) GHARB ASWAN HOSPITAL Helping each other… “The best thing I liked is the cooperation and we learned how to help each other It allows us to love each other and shows us how to work properly.” -- Dr. Wagih Mohamed Farahat ,Unit Doctor EL GAAFRA HEALTH UNİT Facing problems… “The medical team used to ignore any problem without thinking how to solve it. Now the way of thinking has changed. Now we identify the challenge, or the target we want to achieve. The doctor no longer dominates. I give the chance to everyone in the team to express their opinion, in a friendly atmosphere, without ignoring anyone. We take their opinions into consideration. Then we scan the different opinions, and by agreeing together we decide on a challenge. The problem could have been always there, but they were too afraid to face it. No -- face the problem!” RESULTS ACHİEVED… “Regarding the challenge that we chose, the result was very obvious. I used to look at the average no. of postpartum visits, they were in one month 0 visits, another month 1.0 visits, another 1.2 visits. Now, the average no. of postpartum visits is 3.8 visits, in some months 4.0 visits, and it even exceeds 4.2 visits in many months. There are also changes that occurred in each one of us. We are dealing with each other in a very friendly manner, whoever has a problem can speak of it freely, without being scared of any punishment. To solve any problem, the answer has to come from within. If we wait for the solution from outside, then it will not be solved.” Changed work climate… Before this program, the doctor received reports at the end of the month, which he would sign without revising and accountability. Now, any problem that might occur in the unit, any day, will be brought to the 45 doctor’s attention, and we will sit as a team to discuss it. Now this is a very good environment, based on respect, and sharing of opinions, and anyone can present any issue freely, without being afraid. This is a very noticeable change in the work climate. A desire to serve the community… Also, there are the people from the local councils -- people from the local community -- and we want to improve something that will benefit the community that we are working in. These people do not hesitate to help us, but it needs persistence and determination from our side to try to change them gradually. It will not happen from the first time.” Being a leader means not giving up… I have learned from this program to have unlimited ambitions, not to get bored, or to give up on making a change. Being a leader means becoming a role model… “At the beginning of the program one incident occurred: During the polio campaign, a nurse went out to one of the farthest houses and could not find the child, and she told me that she doesn’t want to go there again. We want to change what is inside us. She must care about this child to go to immunize him. Being a good role model for them-- as I learned in the program-- I took the car at my own expense and went to the child’s house to immunize him. They soon felt same way and started copying the same behavior.” Ms. Suheir Sabry Siam, FP Nursing Supervisor, ASWAN HEALTH DISTRICT (SINCE 1996) Supervision means letting the staff find the solutions… “After training with LDPE, we learned how, as supervisors, we shouldn’t go to inspect. When I found a mistake I used to rectify it myself. But after the program, I learned that I should let the nurse heselr find the solutions. I ask, “How do you think you can overcome this point?” So that she herself gives me the solution. I now feel that the way I supervised has changed from inside.” “When we go to the unit we ask them how to increase the number of users of FP methods. They offer different suggestions and solutions for themselves, so that they can achieve the percent that they have selected. How do we achieve this? They hold health awareness meetings, or give the rayedat (women volunteers) the names of the women who visited the unit once only, so that she can follow-up with them and bring them back. We also started to add the client’s telephone number on her form, so where there is a telephone available, the nurse or the rayedat can call the women to check on why she didn’t come back, especially if she has a loop check-up or it is time for her 3-month injection or to receive her oral pills, or any FP methods she is using.” Client Satisfaction “The indicators have thankfully increased. This is something tangible and noticeable in most of the units included in the program. 46 The meeting that you or Dr. Morsy held with the clients has made great impact. It made us feel that the client is important to us. Before, we believed that the clients need us, while we don’t need them. We now feel that our need for them is as strong as their need for us. The client’s satisfaction reflects on us. As a supervisor, I am not in direct contact with the women, but the unit team is. But I am involved with the improvement of staff performance, and asking the client’s opinion about the service provided. It is important that on the 40th day postpartum the woman comes to choose an FP method. Also, if the compulsory immunizations are successful, then we wouldn’t have to suffer during the immunization campaigns, like the polio campaign. Therefore, the clients satisfaction with the services provided is important, important, important.” About the “leading and managing” functions… “I think that the four functions are all important, and they are all connected. You cannot skip one. If I don’t scan the current situation properly, and study to identify the problems, then I will be unable to focus. And if I scan, but when focusing, I didn’t pay attention to collecting the data, looking at the indicators, and the other things I need to measure whether I am on the right track or not, I won’t know if I need to change my working methodology. Also, if I didn’t align and mobilize the people and the work group, and made each person feel that they have an important role, this will also have an affect. And so on.” What changes happened… The girls in El Aakab used to only hold the awareness meetings in the unit, and they didn’t care whether the women attended or not. Since they joined the program, they are now going out to find a youth club to hold the meetings, they write reminders to the women on paper, and go to deliver it themselves, although they could have sent it with a neighbour. They never thought that they would go themselves to the women and regularly.” “As the Aswan team, when we started training in this program, there were times when the team members were very uncooperative. Each team member did as he pleased. I didn’t let anyone tell me that I was doing something wrong, as I felt that it was an order. Now, it’s the opposite. We meet together as a district team, and take each other’s opinions. If we hold a seminar, we include the IE&C specialist, the Rayedat specialist, Dr. Suheir, and I will help them with the counseling. If they are unable to convince the women, then Dr. Suheir asks me to go to talk to them, as each person has her own style of convincing. We really felt that we were one team” “The best thing we gained is that we perform our work as a team not as individuals, and if something affects one person, it affects the whole team. The program showed us how to follow the right steps, and if we are faced with a constraint, we take each other’s opinions to try to overcome it. Dr. Suheir doesn’t ignore our opinions and tell us what to do, but she ask for our input. The program taught us how to really be a team.” Each unit feels that they are the ones capable of identifying their clients’ needs and how to treat these clients. 47 Overcoming challenges… “The people should learn how to prioritize their challenges. Also to take the opinions of the team members in the unit or the hospital. One shouldn’t mock any of their opinions, as you might think that it is not important, whereas it might be the one opinion that can make things better. We have learned to turn people into supporters. For example, the members of the local councils used to always attack us aggressively. Now we tell them that they feel there is a deficiency here, which as a health district, I can’t overcome, how can you help me as the local council or directorate? Therefore, I succeeded to take him on my side. I managed to involve him in the problem.” Dr. Omar Ahmed Youssef Komombo District Manager “Before this program, we were distracted-- not only myself, but all of my district work group, we were all going in a different direction. I am responsible for PHC, FP, immunizations, the district itself, i.e. the doctors’ availability and punctuality at work. Each person worked according to his personal concept, but we are now one team. There is no difference between FP, PHC or immunizations. We all have the same target, which is to improve the level of performance in the district.” “During my supervisory visits, we used to go inspect on people, telling them only what they are doing wrong, but this program told us not to inspect, rather assist people, to put them on the right track, to identify their problem and to solve it, not to look at me for solutions, but to find the solutions from within themselves as they are the most aware of their problems and how to solve it.” Achieving Results… “We learned how to develop an action plan and follow it. We achieved good results. We started to know what is expected from us, how to work, how to develop an action plan, what is the gap and how to identify the gap, what are the methods we use to achieve our target.” On the four functions of leadership… “Focusing is most important. Then comes preparation, you prepare the theatre you will work in. Then you will identify the activities that you will work in, how you are going to progress in them, what are the negative aspects that will be positive. This is the target. Changed Work Climate… “Yes, there is now a spirit where before there was an absence of spirit. The team members care for each other, they now support each other, share the workload. We each have a specific job, yet every person can carry on the other’s work, I now know the other’s work well. “I have learned how to develop a plan, which is real, to achieve a SMART target, to work with a team spirit. These are the most important things we feel in the change, which will help me even on a personal level. --- DR. SUHEIR TAWFIK TAWFIK, FP MANAGER, ASWAN HEALTH DISTRICT 48 (SINCE DECEMBER 2000) Changes in the Work group… LDPE made me focus. Yes, the problem is there, I select it as a challenge, and deal with it. I identify the steps from the plan, which I can follow, and then I start to align and mobilize, monitor and inspire the people. Thank God, we have benefited from this and achieved good results.” “At first we were distracted, if one of us faced a problem, we would ignore the problem and try to forget about it. “We became attached to each other. There is an understanding between us. If one of us is upset about anything, we try to make her feel better. Before, when one of us faced a problem, she would go running to Dr. Barakat with her resignation. As for myself, I did that more than once complaining that I can’t take it any more, even crying and threatening to submit my resignation. Now, anything we face, we try as a team to overcome it. If one of us is upset, the rest of us, as a team, would try to calm her down.” “We take each other’s opinions, together as a team. Not because I am the manager, I enforce my opinions. (Not that I was ever like that, I used to ask for the others’ opinion). But this project has made me do that even more. “ “The work group cares more about the work. It is important for each one to achieve results. own field.” Dr. Abdo El Sweissy Medical Representative Komombo District IDENTIFYING CHALLENGES… “The biggest challenge that we faced in the beginning of the project was -- selecting a challeng! “At first, the situation was not clear, and we started working in more that one direction. Then, we decided to choose the challenge related to our work, the thing that we care about most. Therefore, we chose to increase the percent of FP users serviced/supplied. This is what we decided on.” We always meet to discuss everything together. My role is one of the team. I provide information, and receive from them information. We agreed on the priorities and what we can do. The challenge we chose has a personal, occupational, and even national dimension. “We were able to select a challenge, identify the root causes, develop an action plan, and measure our success. The real accomplishment is the fact that we chose a target, which we can measure. We wanted to increase the percent of FP users serviced/supplied by 5% in 6 months. We succeeded in increasing the percent by 2.8% in 5 months. 49 ABOUT THE “LEADING AND MANAGING” FUNCTIONS… “We do scanning to have a complete picture of the current situation, because if I don’t take all the aspects into my consideration, I could become part of the problem. It is important to look globally at any issue. We cannot do anything else without scanning; we cannot do focusing, if we didn’t do scanning. We really need planning. If we concentrate on these two functions, I believe that they are focal for everything else." WORK CLIMATE AND PERFORMANCE “The work climate has changed 180 degrees. The team spirit that is prevailing now. We were always one of the most organized teams. The organization was there, but now the spirit has changed." We are now more careful not only to do our work, but that our colleagues do their work too. We are not concerned with our work alone, but how my colleagues are doing, and do they need my help." “I will tell you something concerned with the whole group, not only myself. The people came from Aswan, Komombo and Daraw, to attend a preliminary meeting to the district meeting, without there being an official notice of the meeting, or a district manager attending the meeting. We gathered all these because there was love and familiarity between them. IThere was no absentees, no official notice, no incentives, no punishments. That really affected me, to see everyone attend the meeting. It was impossible to happen before the LDPE. People came from all over Aswan, some crossing the Nile, people from all positions, doctors and nurses, all the teams came to the meeting. Without this project, you couldn’t have achieved this. Impossible. FACING THE CHALLENGES The most important things: to share with the people, and the people share with me. To identify my priorities, before starting any work. To have a plan, and know what to do exactly in a defined timeline. This is one of the important things I learned in my life in general. 50