Follow-Up Evaluation of the Leadership Development Program for the Ministry of Health and Population, Egypt Dr. Ersin Topçuoğlu, Senior Technical Officer, Monitoring & Evaluation Unit and Dr. Laila Nawar, Consultant June 2004 Management and Leadership Program Management Sciences for Health Boston, MA 02130 Telephone: (617) 524 7766 www.msh.org/mandl This report was made possible through support provided by the US Agency for International Development, Office of Population and Reproductive Health, under the terms of Cooperative Agreement Number HRN-A-00-00-00014-00. The opinions expressed herein are those of the author(s) and do not necessarily reflect the views of the US Agency for International Development. Follow-Up Evaluation of the Leadership Development Program for the Ministry of Health and Population, Egypt June 2004 Dr. Ersin Topçuoğlu, Senior Technical Officer Monitoring & Evaluation Unit Dr. Laila Nawar, Consultant Management and Leadership Program Cooperative Agreement No. HRN-A-00-00-00014-00 Management Sciences for Health 891 Centre Street Boston, MA 02130 LDPE Egypt Follow-up Evaluation Report 2 ACKNOWLEDGEMENTS In addition to the many Egyptian participants in the Leadership Development Program in Egypt, the authors would like to thank to the following people: Morsy Mansour for his commitment to the program, to this particular evaluation, and for providing guidance before and during the evaluation. He did his best to make us fully understand all aspects of the program. Abdo Alswasy for his hard work before, during, and after the evaluation. He patiently responded to all of our requests for information and also provided logistical support during all stages of the evaluation. Alison Ellis for her guidance on the design of the evaluation,continuous support during the analysis and painful editing and revision stages. Joan Galer for supporting this evaluation as she did during the first evaluation in 2003. Tim Allen for providing feedback on the draft report. LDPE Egypt Follow-up Evaluation Report 3 TABLE OF CONTENTS EXECUTIVE SUMMARY ........................................................................................... 3 1. BACKGROUND ................................................................................................... 7 2. SCOPE OF FOLLOW-UP EVALUATION ............................................................ 9 3. METHODLOGY...................................................................................................11 4. TIMETABLE.........................................................................................................14 5. RESULTS ...........................................................................................................15 6. CONCLUSIONS ..................................................................................................49 7. RECOMMENDATIONS .......................................................................................53 ANNEXES: Annex 1: LDPE 2003 Evaluation Note................ ...................................................57 Annex 2: Interview Guidelines................................................................................58 Annex 3: List of persons interviewed ...................................................................64 Annex 4: M&L Program Principles for Developing Managers Who Lead ..........66 LDPE Egypt Follow-up Evaluation Report 4 EXECUTIVE SUMMARY The Leadership Development Program in Egypt (LDPE) was a one-year pilot program co-led by the Ministry of Health and Population (MOHP) and the Management and Leadership (M&L) Program of Management Sciences for Health (MSH). The program began in June 2002 and ended in June 2003. The M&L Monitoring and Evaluation Unit conducted an evaluation of the LDPE in June 2003. Although M&L’s support for and assistance to the LDPE ended in June 2003, participants decided to replicate and expand the program. M&L conducted a follow-up evaluation in April 2004 to learn more about this self-directed initiative. The follow-up evaluation focused on assessing the following three areas: • performance of the “first” generation teams since the LDPE ended • quality of the replication process • workgroup climate in “first” and “second” generation teams The most obvious result of the LDPE is that it has had a tremendous impact on the behavior of teams at both the personal and professional levels. The evaluation team did not meet any single participant who was not enthusiastic about the program and there was no difference between the first and second generation teams. Staff managing the replication process was able to create the same learning and participatory environment as existed in the original LDPE. The program has greatly contributed to creating a better workgroup climate. The workgroup climate has dramatically improved among the new teams and has been successfully maintained among the old teams. The LDPE is perceived as a powerful tool to improve performance by all participants. The program’s participatory approach has enabled front line service providers to actively participate in discussions and to actively participate in the design and implementation of their own small-scale service delivery improvement projects, as opposed to conducting projects “imposed” by higher levels of the health system. This has contributed substantially to participants’ enthusiasm and ownership of their service delivery challenges. An overall review of the ten first generation teams indicates that during the LDPE all of the teams were able to improve their performance in addressing their selected service delivery challenges and produce differing degrees of improvement in results. While most of the teams were able to further improve their performance following the completion of the LDPE in June 2003, several teams, especially the district teams (and when compared to non-participating districts), did not perform well. Another important finding is the lack of a fully developed LDPE training curriculum that led the three districts to follow different training workshop sequences. This led to temporal conflict between the training, action plan development, and implementation. Several second generation teams had to select challenges, set targets, and prepare action plans before receiving all LDPE Egypt Follow-up Evaluation Report 5 the necessary training in leading and managing concepts and practices. Another important finding related to the replication process is the inadequate commitment and support of senior managers at all levels of the MOHP. An overall look at the LDPE program in Egypt reveals some other important facts and observations: • The LDPE is effective in improving performance, especially at the clinic level. However, the current program design may not address all the issues and needs at the district or higher administrative levels to produce and sustain desired results. In almost all cases, team members at the clinic level were service providers (e.g., doctors, nurses). This makes it easier and operationally feasible to implement service delivery oriented action plans and to achieve results. On the other hand, at the district level, team members are either program managers or supervisors of a number of service delivery facilities. Designing and implementing action plans, aligning and mobilizing service providers, and monitoring the results require a different set of knowledge, skills, and tools not provided in the program. The current design should be tailored to meet those needs to ensure that learning and ownership are successfully transferred to staff working in the facilities. • The LDPE focuses on improving climate, creating and sustaining commitment, and enabling participants to work together in teams to achieve results. However, the M&L Program’s evaluations in 2003 and now in 2004 reveal that since the beginning of the program the teams have not received adequate technical assistance on key public health program management issues. Although most of the teams were able to improve their performance, their current level of knowledge and skills in scanning the most valid, key public health challenges in their communities and measuring and monitoring service delivery performance was found to be very limited. • The lack of human and financial resources has been accepted as a “given” by all teams. This assumption leads to the selection of moderate to limited objectives and humble action plans. The LDPE component that addresses “aligning and mobilizing resources” should not direct or gear participants to rely solely on existing (or in most cases non-existent) resources. The teams should be encouraged and trained to actively mobilize additional resources inside and outside of the health sector, when needed. • The M&L Program’s “Leading and Managing Framework” posits that improved managing and leading practices contribute to improved work climate and management systems, which in turn lead to improved health services and ultimately better health outcomes. The logical flow implies that services cannot be improved without improving workgroup climate and management systems, as well as managing and leading practices. Although not part of the evaluators’ scope of work, it should be mentioned that the LDPE neither validates nor invalidates the logic of this framework. A re-examination of the framework, through reversing its presentation and re-thinking its application may be helpful. Perhaps a better LDPE Egypt Follow-up Evaluation Report 6 approach would be starting from desired outcomes in terms of service delivery and then analyzing the relevance, quality, and gaps in the functioning of current management systems, workgroup climate, and leading and managing practices. In this way, the training or other appropriate interventions can address and emphasize those aspects which require the greatest attention. • During the LDPE replication process the Workgroup Climate Assessment tool was not used. The staff managing the replication process did not feel that this tool was essential. In light of this finding it is recommended that the utility of the tool and the way it is introduced to participants be examined. LDPE Egypt Follow-up Evaluation Report 7 1. BACKGROUND The Leadership Development Program in Egypt (LDPE) was a one-year pilot program co-led by the Ministry of Health and Population (MOHP) and the Management and Leadership (M&L) Program of Management Sciences for Health (MSH). The program began in June 2002 and ended in June 2003. The overall purpose of the program was to improve the quality and accessibility of health services in Egypt, specifically in three districts of the Aswan Governorate, by increasing: • the capability of managers to lead others to achieve results; and • their ability to create climates of high performance in their workplaces. The specific objectives of the LDPE were to: 1. Support managers to address the critical challenges in their districts 2. Improve the capability of district level and clinic-level (doctors and nurses) managers to lead performance improvement projects that address these challenges 3. Build capacity to monitor and track performance results 4. Support managers to improve the workgroup climate in their workplaces, resulting in an increased commitment of staff to serve clients and continuously improve services The key components of the program were bi-monthly leadership workshops followed by monthly district or clinic level meetings. LDPE participants formed ten working teams. Each team selected a performance improvement project and prepared an associated action plan. The underlying assumption of the LDPE was that teaching leadership functions and practices to teams and supporting them in the design and implementation of specific performance improvement projects would lead to improved results in health services. Guided by this assumption and the program’s objectives, the M&L Monitoring and Evaluation Unit conducted an evaluation from June 15-23, 2003. This evaluation was designed to measure the teams’ performance in each of the following program elements separately: LDPE Egypt Follow-up Evaluation Report 8 Organizational Element Leadership Element Select Challenge Goal setting Scan Focus Input & Process Align & Mobilize Output Achieve results Outcome Inspire The evaluation report is available from the M&L Monitoring and Evaluation Unit. The report documents the achievements of the district and health facility teams in applying their new knowledge and skills, implementing their action plans, and achieving results in the three areas of health service delivery they had selected, namely family planning (FP), antenatal care (ANC) and postpartum (PP) care. A brief summary of the major findings from the 2003 evaluation is provided in Annex 1. Although M&L’s support for and assistance to the LDPE program ended in June 2003, graduates in Aswan governorate decided to replicate and expand the program to other clinics and districts within the governorate. The Aswan graduates have been training new teams and supporting these teams to address selected service delivery problems. The Aswan graduates have decided to regularly report progress to M&L even though they were not asked to do so. UNFPA has also started replicating the program in another governorate as well. M&L has not been providing any technical assistance and/or financial support to these self-directed initiatives. M&L decided to conduct a follow-up evaluation to learn more about this self-directed initiative and to document the process, achievements and lessons learned since the termination of M&L funding and technical assistance. The following table illustrates the timeline of LDPE program and the two evaluations. Table 1: LDPE and evaluation timeline LDPE begins (July) LDPE ends (June) 2002 2003 2004 First evaluation (June) Follow-up evaluation (April) LDPE Egypt Follow-up Evaluation Report 9 2. SCOPE OF WORK FOR THE FOLLOW-UP EVALUATION M&L expects that this follow-up evaluation will yield additional information on the results and impact of M&L interventions, as well as substantive learning for the benefit of expanding M&L’s knowledge. It is likewise expected that the follow-up evaluation will help M&L and our counterparts to understand the factors affecting the service delivery achievements in a sample of the “first generation” and “second generation” teams as well as the commitment demonstrated by the Aswan graduates. M&L is anticipating that lessons learned from this program will benefit both the Egyptian health and FP program and the larger public health community. The purpose of the follow-up evaluation is to: • Assess the leadership and service delivery achievements of the original LDPE teams (“first generation”) since June 2003 when M&L support for the implementation of the LDPE ended • Assess and compare the leadership and service delivery achievements of the new teams (“second generation”) in the Aswan governorate who have received the LDPE from their colleagues (the first generation) • Assess the process, quality and content of program replication Specific Objectives Two main areas (program implementation and program replication) were assessed. Area 1: Program Implementation The first area was program implementation by the first and second generation teams. The four specific objectives and required information for each objective are listed below: 1. Assess the selection of challenges • Number of teams (clinics and districts) enrolled in the program • Challenges selected by the teams • Prioritization and selection process for the challenges 2. Assess the availability and quality of action plans • Appropriateness of desired and actual performance (SMART1 criteria) • Availability of action plans • Process used for preparing action plans 1 Specific; Measurable; Appropriate; Realistic; and Time bound. LDPE Egypt Follow-up Evaluation Report 10 • Selection and relevance of activities to address the challenges in leading to the achievement of desired performance 3. Assess the implementation phase and results • Extent of the implementation of activities (were all activities implemented?) • Activities implemented that were not included in the action plan • Mechanisms for monitoring progress in implementation • Measurable results related to the desired performance and action plans • Other results achieved that are unrelated to the action plans 4. Assess changes in attitudes and practices of the teams • Main factors that motivated the teams to achieve results • Main factors that prevented the teams from achieving results • Changes in leading and managing behaviors and practices (what are the teams doing differently?) • Changes in the workgroup climate • Processes and skills used by the teams • Participants’ perceptions on the difference between the LDPE and other program approaches Area 2: Program Replication The second area of assessment was program replication. The two specific objectives and required information for each objective are listed below: 1. Assess the program replication strategy • Main factors that motivated Aswan graduates to replicate the program • Availability of a documented replication plan • Criteria used for the selection of new teams • Future plans for replication in the Aswan governorate 2. Assess the program replication process • Resources needed and used during the replication process • Processes used for program delivery • Processes used for follow-up support • Technical assistance received during the replication from any source other than M&L LDPE Egypt Follow-up Evaluation Report 11 3. METHODOLOGY The follow-up evaluation used four methodologies to collect information: 1. Review of written materials and documents 2. Interviews and focus groups with district managers and facility personnel: 25 teams (first and second generation) and 84 individuals 3. Assessment of workgroup climate using MSH’s Workgroup Climate Assessment Tool 4. Field visits to five facilities (two from the first generation; three from the second generation) to conduct further discussions and collect additional data 3.1 Review of written materials and documents The first task was to assess the achievements of the first generation teams. The challenges selected by the teams, baseline and end of June 2003 service delivery statistics, and end of March 2004 service delivery statistics were collected from these teams. The second task was to collect and assess the second generation teams’ challenges, baseline service delivery statistics, and other information on their progress to date. All 25 teams made presentations to the evaluators, an observer from the USAID-funded Catalyst Project (Pathfinder International), and governorate officials from April 17-19, 2004 in Aswan. The schedule was as follows: April 17th, 2004 Aswan Directorate teams’ presentation April 18th, 2004 Daraw Directorate teams’ presentation April 19th, 2004 Kom Ombo Directorate teams’ presentation Teams were also asked to provide copies of their action plans, timetables, and other relevant documents. The third task was to collect all the documents from the three directorates and the Aswan governorate related to the expansion of the program. These included, but were not limited to, training notes, meeting minutes, invitation letters, and other official documents and reports produced before and during the expansion. The documents were reviewed by the evaluators to assess the challenges selected, relevance and quality of the action plans, extent of implementation, quality of replication plans, and the replication process. The evaluators also reviewed service delivery results achieved by the first generation teams since June 2003. It was not possible to conduct a similar review of the second generation teams’ results because the LDPE replication was still in process at the time of this follow-up evaluation. LDPE Egypt Follow-up Evaluation Report 12 3.2 Focus group discussions and interviews Structured interviews were conducted with people participating from different levels of the program: governorate, district, and clinics. The first group interviewed was Aswan Governorate health managers and staff from the three districts who have been managing and expanding the program. Information collected included: the factors that motivated them to replicate the program; process and criteria for selecting new teams; future plans for replication; and resources used and technical assistance needed for implementation of the LDPE. Individual or group discussions were conducted where appropriate. The second group of people interviewed was selected members of first and second generation teams. During the April 17-19, 2004 meetings with the teams two focus group discussions were conducted every afternoon: one with the first generation participants, and one with second generation participants. The evaluators interviewed the teams regarding: the selection process for challenges; methodology used for preparing action plans; main factors that motivated or inhibited the teams from achieving results; changes in behaviors and practices of the teams; and the teams’ perceptions of the program. An interview guide was prepared in advance to specify the important issues that needed to be explored during the individual and group discussions. This helped conduct interviews in a more systematic and comprehensive way. The guide also helped limit the number of issues discussed and thus ensured focused discussions. All interviews were digitally recorded and stored in separate files. The interview guides are provided in Annex 2. 3.3 Assessment of workgroup climate: In June 2003 at the end of the original LDPE, workgroup climate in the ten teams was measured using MSH’s Workgroup Climate Assessment (WCA) tool. At that time 35 members from the original 10 teams were asked to retroactively assess their workgroup climate at the start of the program and at the end of the program. The same WCA tool was administered to the team members who attended the April 17- 19, 2004 meetings in Aswan. LDPE Egypt Follow-up Evaluation Report 13 3.4 Field visits After the interviews a sampling of teams/clinics was visited to collect additional qualitative and quantitative information. The facility visits were purposefully selected in order to capture the maximum variation within the teams. Challenges that were different from the majority of the teams, varying degrees of achievement and commitment of the teams, and different or innovative approaches in preparing and implementing action plans were the factors used to identify the teams to be visited. Table 2 below lists the teams/clinics visited. Table 2: Clinics/teams visited to collect additional information Name of clinic District Generation Visit Date Abo El Rees Kobly Health Unit Aswan Second April 20th 2004 Gharb Aswan Hospital Aswan First April 20th 2004 Al Rakkaba Health Center Daraw First April 21st 2004 Al Ababda Health Unit Daraw Second April 21st 2004 MCH/FP Center Kom Ombo Second April 22nd 2004 LDPE Egypt Follow-up Evaluation Report 14 4. TIMETABLE The timetable for the follow-up evaluation was as follows: April 1-12, 2004 Preparation April 13th, 2004 Final review of methodology, tools, and interview guides with the local consultant Interview with former MOHP LDPE program coordinator, Dr. Morsy Mansour April 14th, 2004 Travel from Cairo to Aswan April 15th, 2004 Meet with Aswan Governorate staff Discuss and finalize the timetable Interview with Aswan Governorate LDPE program coordinator, Dr. Abdo Aswasy and his team April 16th, 2004 Holiday April 17th ,2004 Morning: Aswan teams’ presentations, WCA application, review of documents Afternoon: 2 focus group discussions with “old” and “new” Aswan teams April 18th, 2004 Morning: Daraw teams’ presentations, WCA, review of documents Afternoon: 2 focus group discussions with first and second generation Daraw teams April 19th, 2004 Morning: Kom Ombo teams’ presentations, WCA, review of documents Afternoon: Two focus group discussions with old and new Kom Ombo teams April 20th, 2004 Visit two teams in Aswan and conduct follow-up interviews Visit Aswan Directorate and conduct follow-up interviews April 21st, 2004 Visit two teams in Daraw and conduct follow-up interviews Visit Daraw Directorate and conduct follow-up interviews April 22nd, 2004 Visit one team in Kom Ombo and conduct follow-up interviews Visit Kom Ombo Directorate and conduct follow-up interviews April 23rd, 2004 Return to Cairo April 24th, 2004 Present preliminary results and overall findings to the MOHP LDPE Egypt Follow-up Evaluation Report 15 5. RESULTS The results are presented in three different sections: • Assessment of achievements at the programmatic level • Assessment of workgroup climate • Findings from focus group discussions and interviews 5.1 Assessment of achievements at the programmatic level 5.1.1 Assessment of the first generation teams’ challenges: 10 teams were involved in the initial implementation of the LDPE from June 2002 to June 2003. In June 2003 the achievements of these 10 teams were analyzed. Results can be found in the M&L evaluation report2 . During the June 2003 evaluation the teams were asked about their future plans. While most of the teams mentioned that they would continue working on the same service delivery challenge, two teams selected new challenges. As of June 2003 the teams’ original and new challenge areas were as follows: Table 3: First generation teams’ challenges Team District Original Challenge New Challenge Kom Ombo District Kom Ombo Family Planning Same Expand the program Daraw District- FP Unit Daraw Family Planning Same Daraw District – ANC Unit Daraw Antenatal Care Same Rakkaba Health Center Daraw Family Planning Same Aswan District Aswan Family Planning Same Al Aakab Health Center Aswan Family Planning Same Nafak Health Center Aswan Antenatal Care Same Daraw Health Center Daraw Antenatal Care Same Gaafra Health Center Daraw Postpartum Care Same Gharb Aswan Hospital Aswan Postpartum Care Add Family Planning With the exception of Gharb Aswan Hospital and Kom Ombo District, the teams did not select a new challenge. The teams may still feel that limited time has been spent addressing their existing action plans and want to continue to expand their coverage and performance. In fact, as shown below, most of the teams were able to further improve their performances after the end of the original LDPE. 2 Dr. Ersin Topçuoğlu, “Evaluation of the Leadership Development program for the Ministry of Health and Population, Egypt,” Monitoring and Evaluation Unit, Management and Leadership Program, Management Sciences for Health, October 2003. LDPE Egypt Follow-up Evaluation Report 16 5.1.2 Assessment of First Generation Teams’ Performance Daraw Health Center The team’s original challenge was to increase the average number of ANC visits per client from 0.6 in January 2003 to 1.0 in June 2003. The team decided to continue to address this challenge, but without changing the target. The team’s original challenge covered the six month period of January through June 2003. The following chart compares the average number of visits per pregnant woman before (January-December 2002), during (January-June 2003), and after (July 2003-March 20043 ) the LDPE. Figure 1: Daraw Health Center: ANC Service Results 0,0 0,5 1,0 1,5 2,0 2,5 Before the program (12 months) During the program (6 months) After the program (9 months) Average ANC visits per pregnant 0 40 80 120 160 Average ANC visits per month ANC visits per pregnant ANC visits per month The team was not only able to increase the average number of visits per pregnant woman (purple bar) but also the total number of ANC visits (blue line). During the 12 month period before the program, on average there were 95 ANC visits per month. During the six month implementation period this figure rose to 146 visits. As of the end of March 2004 there were 148 ANC visits per month during the 9 months since the original LDPE ended. While the total number of ANC visits to the facility increased by 55.8%, the average number of visits per pregnant woman doubled. 3 At the time of the follow-up evaluation, service statistics through the end of March 2004 were available. LDPE Egypt Follow-up Evaluation Report 17 Nafaq Health Center This is one of the first generation teams in Aswan district. The team’s original challenge was to increase the average number of ANC visits per client from 0.5 in January 2003 to 2.0 in June 2003. The team decided to continue to address this challenge, but without changing the target. The following chart compares the average number of visits per pregnant woman before (January-December 2002), during (January-June 2003), and after (July 2003-March 2004) the LDPE. Figure 2: Nafaq Health Center: ANC Service Results 0,0 0,5 1,0 1,5 2,0 2,5 Before the program (12 months) During the program (6 months) After the program (9 months) Average ANC visits per pregnant 0 50 100 150 200 250 300 350 Average ANC visits per month ANC visits per pregnant ANC visits per month The chart indicates that Nafaq Health Center was able to increase the average number of visits per pregnant woman from 0.4 before the LDPE program to 1.4 visits during the implementation of its action plan and to 2.1 visits in the 9 months after the end of the original LDPE. The average number of monthly ANC visits also tripled, from 101 before the program, to 326 in the last nine months after the program. LDPE Egypt Follow-up Evaluation Report 18 Daraw District – ANC Team The team’s original challenge was to increase the average number of ANC visits per client from 1.0 in January 2003 to 2.0 in June 2003. The team decided to continue to address the same challenge without changing the target. The following chart compares the average number of visits per pregnant woman before (January-December 2002), during (January-June 2003), and after (July 2003-March 2004) the LDPE. Figure 3: Daraw District: ANC Results 0,0 0,5 1,0 1,5 2,0 2,5 3,0 Before the program (12 months) During the program (6 months) After the program (9 months) ANC visits per pregnant 0 100 200 300 400 500 600 700 800 ANC visits per month ANC visits per pregnant ANC visits per month There was a 119.3% increase in the average number of ANC visits per month over the last nine months compared to the period before the implementation of the team’s action plan. It should also be noted that in the last nine months each pregnant woman received 2.4 visits. This figure was 1.3 before the LDPE program. It indicates an 84.6% increase in the number of visits. In other words, each pregnant woman receives roughly one additional visit. Another important finding is the average number of newly identified pregnant woman per month. Before the program, in the whole of Daraw district, clinics were able to identify 151 new pregnancies each month. This figure rose to 190 during the implementation of the action plans, and 248 after the LDPE ended and over the past nine months, July 2003 through March 2004. Between October 2003 and January 2004 the Aswan governorate launched an ANC campaign in all five its districts. The purpose of the campaign was to increase both the number of new ANC visits and also the number of ANC visits per pregnant woman. This gave the evaluator an opputunity to compare the LDPE Egypt Follow-up Evaluation Report 19 performance of Daraw district to other non-participating districts. The table below summarizes Daraw’s performance as compared to three other districts in Aswan governorate. (The data from Nasr district were incomplete, therefore it is excluded). Table 4: Comparison of ANC performance of Daraw and other three districts in Aswan governorate before and after the program Daraw Aswan Kom Ombo Edfo % increase in the average number of ANC visits per pregnant woman 84.6 88.1 19.8 23.9 % increase in the average number of ANC visits per month 119.6 65.4 58.9 88.1 % increase in the average number of new ANC visits per month 64.2 17.2 57.5 79.9 Table 4 indicates that the government’s campaign helped improve ANC services in all four districts. While Kom Ombo and Edfo (non-LDPE) districts increased the average number of ANC visits per pregnant woman moderately (19.8% and 23.9% respectively), both Aswan and Daraw districts were able to improve this indicator by 88.1% and 84.6% respectively. On the other hand, Aswan district increased the average number of new ANC visits by only 17.2%. While the other three districts failed to demonstrate similar improvements in all three indicators during the same period, Daraw was able to dramatically improve all three of them. LDPE Egypt Follow-up Evaluation Report 20 Gaafra Health Center Gaafra is also one of the first generation teams in Daraw district. The original target was to increase the average number of PP care visits per client from 0.2 in June 2002 to 4.0 in June 2003. The team decided to continue to address the same challenge, but did not set a new target. The following chart compares the average number of PP visits per client before (January-June 2002), during (July 2002-June 2003), and after (July 2003-March 2004) the LDPE . Figure 4: Gaafra Health Center: PP Care Service Results 0,0 1,0 2,0 3,0 4,0 5,0 Before the program (6 months) During the program (12 months) After the program (9 months) Average number of PP visits per woman The chart clearly indicates that Gaafra Health Unit has more than doubled the average number of PP care visits over the last two years. The team was able to maintain the momentum gained during the implementation of its action plan and continued to improve its performance. There is a 19.4% increase in the average number of PP care visits per client after the completion of the LDPE. Before the LDPE was conducted, each nurse had her own specific area of responsibility. While one was responsible for home visits to PP women, the other was only providing well-baby monitoring services. One of the main activities the team decided to implement in its action plan was to train all 12 nurses and assign them to all services. Now all the nurses conduct PP home visits. This change in the management of service delivery has clearly yielded very positive results. LDPE Egypt Follow-up Evaluation Report 21 Gharb Aswan Hospital The team’s original challenge was to increase the average number of PP care visits per client from 0 in January 2003 to 3.0 in June 2003. Since October 2003 the team has maintained on average more than two PP care visits per woman. It should be noted that in 2002 there was no PP care program at the hospital, and until March 2003, no PP client had been visited. The following chart compares the average number of PP visits per client before (January-December 2002), during (January-June 2003), and after (July 2003- March 2004) the LDPE. Figure 5: Gharb Aswan Hospital: PP Care Service Results 0,0 1,0 2,0 3,0 Before the program (12 months) During the program (6 months) After the program (9 months) Average number of PP visits per pregnant The team was able to provide 0.8 visits per woman during the implementation of its action plan. The most striking improvement in PP services came during the nine months after the original program ended. On average, each woman received 2.5 visits. In June 2003, the Gharb Aswan Hospital team decided to add another challenge. Its new challenge is to increase the number of IUD acceptors. The baseline is approximately 10 IUD insertions per quarter. The team is still in the process of setting targets and preparing its action plan to address this challenge. LDPE Egypt Follow-up Evaluation Report 22 Al Rakkaba Health Center The Al Rakkaba Health Center is one of the first generation teams in Daraw district. Its original target was to increase the percentage of FP users from 27.2% (six months average) in June 2002 to 33.5% in June 2003. The team decided to continue to address the same challenge and set a new goal of 45% by June 2004. This health center is one of the three units serving the Al Rakkaba area with a total population of 7,000. However, the Health Center manager does not know the clinic’s actual catchment area population. The clinic has been sending its service statistics to the Daraw Directorate and all the calculations of service performance and results have been made by the Directorate on the Center’s behalf. In Egypt, a couple year of protection (CYP) indicator is used to monitor FP services. This indicator has some technical problems that are mentioned in the M&L’s October 2003 evaluation report. The technical problems are exacerbated in this case because the actual catchment area population is small and is not precisely known. In these circumstances, it is better to use data on the total number of FP visits, including new and continuing user visits, to measure performance. The following chart compares the average number of FP visits per month before (January-June 2002), during (July 2002-June 2003), and after (July 2003-March 2004) the LDPE . Figure 6: Al Rakkaba Health Center: FP Service Results 0 5 10 15 20 25 30 35 40 New visits Return visits Total visits Average number of FP visits per month Before the program (6 months) During the program (12 months) After the program (9 months) The bar graphs indicate that the clinic was able to recruit fewer and fewer new FP users since the beginning of the program. While the number of continuing users increased by 72.4% during the implementation of the action plan, performance LDPE Egypt Follow-up Evaluation Report 23 declined 1.4% over the last nine months. These results indicate that the momentum gained during the program could not be maintained after the completion of the original LDPE program in June 2003. Al Aakab Health Center This is one of the first generation teams in Aswan district. The team’s original challenge was to increase the percentage of FP users from 37.2% (six months average) in June 2002 to 39.1% in June 2003. The team decided to continue to address the same challenge but without revising the target. Due to the small number of monthly visits, the evaluators decided to use the number of visits per month rather than the CYP based indicator to assess the performance of the team. The following chart compares the average number of FP visits per month before (January-June 2002), during (July 2002-June 2003), and after (July 2003-March 2004) the LDPE. Figure 7: Al Aakab Health Center: FP Service Results 0 10 20 30 40 50 New visits Return visits Total visits Average number of FP visits per month Before the program (6 months) During the program (12 months) After the program (9 months) The analysis indicates that both new and continuing user FP visits increased. Over the last nine months, the total number of visits increased by 48.3% compared to the period before the program. However, it should be noted that the momentum, as indicated by the percentage point increase in the number of new and continuing users, has slowed down since the original LDPE ended. The increase in the total number of visits was only 8.9% after the program. LDPE Egypt Follow-up Evaluation Report 24 FP Teams in Three Program Districts Daraw, Aswan and Kom Ombo District teams selected FP as their challenge. Their original action plans covered the six-month January-June 2003 period. These three team decided to continue focusing on FP after the original LDPE ended in June 2003. During the follow-up ssessment it was possible to gather FP service statistics from the other two districts of Aswan governorate that did not participte to the LDPE. The following chart compares the average number of FP visits per month before (January-June 2002), during (July 2002-June 2003), and after (July 2003-March 2004) the LDPE for three participating (Daraw, Aswan and Kom Ombo) and two non-participating (Nasr and Edfo) districts. Figure 8: FP Service Results of Five Districts– Average number of FP visits per month 0 500 1000 1500 2000 2500 3000 3500 4000 Daraw Aswan Kom Ombo Nasr* Edfo* (*) Non-participating districts Average number of FP visits per month Before the program (12 months) During the program (6 months) After the program (9 months) During the LDPE all three participating districts significantly increased the average number of FP visits per month. This increase was 50.6% in Daraw, 49.7% in Aswan, and 33.5% in Kom Ombo. During the same period, the non￾participating district of Edfo was able increase the average number of visits by 83.6%. The figure was 29.2% for the other non-participating district, Nasr. Over the last nine months since June 2003, while the two participating districts of Daraw and Aswan were barely able to maintain their gains, the other participating district, Kom Ombo, was able to further increase the total number of FP visits by 19.5%. Figure 8 shows that Aswan district only increased the number of FP visits by 1.5% over the last nine months, while the number of visits in Daraw dropped by 4.1% over the same period. LDPE Egypt Follow-up Evaluation Report 25 As compared to the period before the LDPE, the total increase in the average number of FP visits per month was 59.8% for Kom Ombo, 51.9% for Aswan, and 44.4% for Daraw district. During the same period the indicator value increased by 88.1% for Edfo and 40.3% for Nasr. First time visits to the clinics is also an important indicator for assessing the performance of FP programs. The following chart compares the average number of new FP visits per month before (January-June 2002), during (July 2002-June 2003), and after (July 2003-March 2004) the LDPE for the three participating (Daraw, Aswan and Kom Ombo) districts and two non-participating (Nasr and Edfo) districts. Figure 9: FP Service Results of Five Districts: New FP visits per month 0 200 400 600 800 1000 1200 1400 1600 1800 Daraw Aswan Kom Ombo Nasr* Edfo* (*) non-participating districts Number of new FP visits per month Before the program (12 months) During the program (6 months) After the program (9 months) A similar pattern is seen here. During the implementation of their action plans, the three LDPE districts were able to increase the number of new FP visits per month. The increase was 42.2% in Daraw district, 45.9% in Aswan, and 27.9% in Kom Ombo. During the same period Edfo (87.3%) and Nasr (27.0%) were also able to increase the number of new FP visits per month. After the original LDPE ended, a sharp decrease is observed in Daraw (-13.9%) and in Aswan (-20.5%), while there is a slight increase (1.4%) in Kom Ombo. Similar decreases are also observed in Nasr (-9.3%) and Edfo (-20.9%), the two non-participating districts. The following table summarizes the FP service performance of all five districts. LDPE Egypt Follow-up Evaluation Report 26 Table 5: Comparison of FP service performance of participating (Daraw, Aswan and Kom Ombo) and non-participating (Nasr and Edfo) districts in Aswan governorate before and after the program Participating Non-participating Daraw Aswan Kom Ombo Nasr Edfo % increase in the average number of FP visits per month 44.4 51.9 59.8 40.3 88.1 % increase in the new of FP visits per month 22.4 16.0 29.7 15.2 48.2 Although the three LDPE districts improved their FP service performance, it should be noted that the two non-participating districts also did very well. While Nasr’s performance fell behind all four districts, Edfo did remarkably better than all four districts. The scope of this limited follow-up evaluation did not allow us to collect further information on these two non-participating districts. We are therefore not able to explain why Edfo performed so well. An overall review of the ten first generation teams indicates that during the LDPE all teams were able to improve their performance in addressing their selected challenges and produced different degrees of improvement in results. Four of the six clinic teams were able to further improve their performance in the nine months following the completion of the original LDPE. While one clinic team was able to maintain the achievement, one clinic team could not maintain the limited achievement gained during the program. The Daraw district team that selected ANC as their challenge not only further improved their performance following the completion of the LDPE, its performance exceeded the other three comparison districts that did not select ANC. On the other hand, all three district teams that selected FP as their challenge performed only better than the non-participating Nasr district, while their performance fell behind the other non-participating Edfo district. Table 6 below summarizes the overall results for the first generation teams. LDPE Egypt Follow-up Evaluation Report 27 Table 6: Overall assessment of first generation teams Name of the Team Selected challenge Performance during the LDPE Performance after the LDPE ended Daraw Health Center ANC Improved Maintained Nafaq Health Center ANC Improved Further improved Daraw District ANC Improved Further improved Gaafra Health Center PP Improved Further improved Gharb Aswan Hospital PP Improved Further improved Rakkaba Health Center FP Somewhat improved Not maintained Al Aakab Health Center FP Improved Further improved Daraw District FP Improved Not maintained Aswan District FP Improved Maintained Kom Ombo District FP Improved Further improved 5.1.3 Assessment of Second Generation Teams’ Challenges In July 2003, three districts — Aswan, Daraw, and Kom Ombo — selected 6, 4, and 5 new teams, respectively. The following table summarizes the new teams and their challenges. Table 7: Second generation teams and their selected challenges Baseline Target Aswan District Challenge Period Value Value Period Mahmodya Health Unit ANC Dec 03 2.2 3.0 June 04 Abo El Rees Kobly Health Unit ANC June 03 2.7 4.0 June 04 Abo El Rees Bahry Hospital ANC Dec 03 4.7 5.0 June 04 Nagh El Mahata Health Unit Well baby June 03 1.9 4.0 June 04 Gharb Aswan Health Unit ANC June 03 1.3 2.5 June 04 Mukla Health Unit ANC June 03 1.3 3.0 June 04 Daraw District Al Kufteya Health Center ANC Dec 03 1.0 2.0 June 04 Al Ababda Health FP Dec 03 29% 30% June 04 Nagga Wannes Health Center ANC Dec 03 1.5 2.0 June 04 Banban Health Unit FP Dec 03 5.6% 10% June 04 Kom Ombo District Kofor Kom Ombo Hospital FP Dec 03 18.5% 22% June 04 Hagazza Health Unit FP (new) June 03 210 250 June 04 FP (cont.) June 03 125 150 June 04 Al Khor Health Unit FP Dec 03 21.5% 25% June 04 MCH/FP Center All services June 03 7,500 9,000 June 04 MIS Unit of Health Directorate FP MIS Dec 03 Poor Better June 04 LDPE Egypt Follow-up Evaluation Report 28 The table shows that while nine teams decided to take the end of 2003 figure as their baseline and June 2004 as their target date, the other six teams decided to compare their performance with the same period last year. Since all of these teams are still implementing their action plans, it is too early to assess their performance in terms of service delivery or other desired results achieved. Therefore, in this section the challenges selected by the teams are discussed. As seen in the table, all 15 teams have a specific challenge with a measurable baseline and targets. Similar to the first generation teams, the majority of the challenges are either ANC or FP. However, this time there are a few different challenges. One team decided to improve the number of well-baby visits, and one team chose to improve the FP management information system at the district level. The most unusual challenge belongs to the Kom Ombo Maternal Child Health/Family Planning (MCH/FP) Center. This team decided to increase the performance of the clinic’s total output by 20% in all service delivery categories. The categories include FP, ANC, growth monitoring, and vaccination visits, as well as outpatient services such as diarrhea control and management of pneumonia. Outpatient services constitute more than 85% of the total number of clients served in one year. It must be noted that there are no sub-targets for each service to monitor progress. The goal of increasing the caseload of the outpatient clinic is therefore very difficult to interpret. Is it due to better quality of care provided at the clinic? Is it because more people are having health problems? Is it because the health seeking behavior of people has improved? Or is it because patients shifted from other health facilities to Kom Ombo as a result of additional information, education, and communication (IEC) activities by clinic staff? The only figures provided by the Kom Ombo MCH/FP Center in its action plan are 7,500 outpatient visits as the baseline and 9,000 outpatient visits as the target. During the in-depth interview with team members it was observed that the team was not aware that there was a problem with the way in which it had defined its goal. A diarrhea outbreak could help the facility to easily achieve its desired performance results. (The Center serves more than 70,000 people.) The team should have received technical assistance at the time it selected its challenge and determined its baseline and target values. The other teams’ challenges seem reasonable compared to Kom Ombo MCH/FP Center. Seven teams selected ANC as their challenge and their target values range from 2.5 to 5 visits per pregnant woman. On average, the baseline for ANC is 2.1 visits per pregnant woman and the teams are generally planning to increase this figure by 1.1 visit. Four teams selected FP, with three of them planning to measure their performance using the CYP based indicator. The misleading nature of this LDPE Egypt Follow-up Evaluation Report 29 indicator was discussed earlier in this report as well as in M&L’s October 2003 evaluation report. These teams will probably face the same measurement problems when trying to monitor their performance. On the other hand, there is one team (Hagazza Health Unit) that decided to measure its challenge by monitoring the increase in the numbers of new and continuing FP users. This is the best approach for clinics/teams with small catchment area populations. 5.1.4 Assessment of Second Generation Teams’ Action Plans During this follow-up evaluation each team’s planned activities were reviewed. The three tables below show the main activities selected by the 15 new teams. Table 8: Aswan District teams’ challenges and main activities planned Name of the team Challenge Main Activities Mahmodya Health Unit ANC • Assign one nurse full time for home visits and other nurses share her job in the clinic Abo El Rees Kobly Health Unit ANC • Prepare weekly schedule for nurses’ home visits • Education for clients attending the unit • Meetings with community leaders Abo El Rees Bahry Hospital ANC • Improve the waiting area • Make a separate waiting area for ANC clients • Improve lab (blood tests) • Health education for pregnant women Nagh El Mahata Health Unit Well baby • Health education for fathers during child registration • Home visits • Education for mothers during vaccination visits Gharb Aswan Health Unit ANC • Home visits • Training for nurses on pregnancy counseling Mukla Health Unit ANC • No activities planned yet since the Unit joined the program only two months ago LDPE Egypt Follow-up Evaluation Report 30 Table 9: Daraw District teams’ challenges and main activities planned Name of the team Challenge Main Activities Al Kufteya Health Center ANC • Education of mothers on ANC during tetanus/toxoid vaccination Al Ababda Health Center FP • Home visits • Collaboration with local women’s club • Clinic ID cards for pregnant women Nagga Wannes Health Center ANC • Home visits • Collaborate with religious leader (sheik) to give talks to husbands during prayers Banban Health Unit FP • No activities planned yet (since there is renovation at the clinic the team has temporarily moved to a village house and will try to provide services from there) Table 10: Kom Ombo District teams’ challenges and main activities planned Name of the team Challenge Main Activities Kofor Kom Ombo Hospital FP • Public meetings • Meeting with community leaders • Mobile teams (4 per month) to provide services • Health education for clients • Expand FP services to an NGO clinic • Training of nurses on counseling and recording • Improve clinic infrastructure Hagazza Health Unit FP • Monthly clinic meetings for staff • Meetings with community and religious leaders • Seminars with men • Population day activities every month • Counseling training for nurses • Home visits to continuing FP users • Interviews with men visiting clinics Al Khor Health Unit FP • Home visits • Education of women during vaccination of children • Meetings with husbands and community leaders • Training on counseling • Recruit volunteers • Meet with district management to avoid contraceptive stock outs LDPE Egypt Follow-up Evaluation Report 31 • Resupply at home for some users • Mobile teams to visit continuing FP users at home for resupply • Small in-kind gifts for women MCH/FP Center All services • Improve waiting area • Give queue numbers to clients • Buy equipment (chairs, lab kits) MIS Unit of Health Directorate MIS • Recruit full-time statistician • Prepare filing system for data The tables indicate that most of the teams selected similar activities. Home visits, training for service providers, and health education for clients are the main activities. It is also important to note that almost all the activities are low-cost and can be accomplished by the clinics using their own existing resources. The table below reorganizes the new teams’ activities into four main categories. Table 11: Main activity areas in the action plans Main areas Specific activities IEC • Collaborate with community and religious leaders • Education for clients and parents • Population day activities every month • Public meetings Service Delivery • Improve infrastructure (waiting area, laboratory, new equipment, etc.) • Counseling training for nurses • Improve service organization (queue numbers, ID badges for pregnant women, etc.) Outreach • Home visits • Mobile teams • Recruit volunteers • Collaborate with NGOs for service delivery Management • Monthly clinic meetings • Improve MIS • Monitor contraceptive commodity stock outs The evaluators’ review of the specific activities indicates that the teams’ main intentions are to increase access to services and to improve the quality of services. Since the goals set by the teams are relatively modest, these activities may help the teams to achieve their goals during the short time frame of the implementation of their action plans. The specific activities also indicate that almost all of them can be implemented using existing resources available to the clinics. Reliance on their own resources may also ensure that these activities can be accomplished. LDPE Egypt Follow-up Evaluation Report 32 5.2 Assessment of workgroup climate During the 17-19 April, 2004 meetings, a total of 80 participants (48 new and 32 old [first generation] team members) were given the Workgroup Climate Assessment tool. Participants were asked to assess their workgroup climate retroactively, evaluating it at the beginning of the program when they joined, and to date (as of April 2004). The table below shows the overall results obtained from all participants. Table 12: Workgroup Climate Assessment: Overall results Work Group Assessment All Groups I feel that in my work group….. Importance Before Now Gap 1. We are recognized for individual contributions 4.9 2.2 4.0 0.,9 2. We have a common purpose 4.9 2.3 4.5 0.4 3. We have the resources we need to do our jobs well 4.7 2.4 3.5 1.2 4. We develop our skills and knowledge 4.9 2.2 4.1 0.8 5. We have a plan which guides our activities 5.0 2.1 4.4 0.6 6. We strive to improve our performance 4.9 2.7 4.3 0.6 7. We understand each other’s capabilities 4.8 2.4 4.0 0.8 8. We are clear what is expected in our work 4.9 2.2 4.3 0.6 9. We seek to understand the needs of our clients 4.9 2.5 4.1 0.8 10. We participate in the decisions of our work group 4.8 2.3 4.3 0.5 11. We take pride in our work 4.9 3.0 4.6 0.3 12. We readily adapt to new circumstances 4.8 2.3 3.9 0.9 Our work group is known for quality work 2.5 4.2 Our work group is productive 2.7 4.4 The analysis indicates that participants accorded great importance to all 12 components of workgroup climate, with having an action plan considered the most important and least available component before the LDPE started. Participants’ ratings during the final assessment indicate that the lack of resources remains an important issue. Results obtained from the 48 new team members are summarized below. Table 13: Workgroup Climate Assessment: Results for the second generation teams Work Group Assessment Second Generation Teams I feel that in my work group….. Importance Before Now Gap 1. We are recognized for individual contributions 4.9 2.2 3.9 1.0 2. We have a common purpose 4.9 2.2 4.3 0.6 3. We have the resources we need to do our jobs well 4.8 2.3 3.5 1.3 4. We develop our skills and knowledge 4.8 2.2 4.1 0.7 5. We have a plan which guides our activities 5.0 1.8 4.3 0.7 6. We strive to improve our performance 4.9 2.5 4.2 0.7 7. We understand each other’s capabilities 4.8 2.3 3.9 0.9 8. We are clear what is expected in our work 5.0 2.1 4.1 0.9 9. We seek to understand the needs of our clients 4.9 2.4 4.2 0.7 10. We participate in the decisions of our work group 4.8 2.2 4.2 0.6 LDPE Egypt Follow-up Evaluation Report 33 11. We take pride in our work 4.9 2.9 4.5 0.4 12. We readily adapt to new circumstances 4.8 2.5 4.1 0.7 1. Our work group is known for quality work 2.4 4.4 2. Our work group is productive 2.5 4.5 Results obtained from the 32 first generation team members are summarized below: Table 14: Workgroup Climate Assessment: Results for the first generation teams Work Group Assessment First Generation Teams I feel that in my work group….. Importance Before Now Gap 1. We are recognized for individual contributions 4.9 2.2 4.0 0.9 2. We have a common purpose 4.9 2.6 4.7 0.2 3. We have the resources we need to do our jobs well 4.6 2.5 3.6 1.0 4. We develop our skills and knowledge 4.9 2.3 4.0 0.9 5. We have a plan which guides our activities 4.9 2.6 4.4 0.5 6. We strive to improve our performance 4.9 2.9 4.4 0.5 7. We understand each other’s capabilities 4.8 2.5 4.2 0.6 8. We are clear what is expected in our work 4.8 2.3 4.4 0.4 9. We seek to understand the needs of our clients 4.9 2.6 4.0 0.9 10. We participate in the decisions of our work group 4.8 2.6 4.3 0.5 11. We take pride in our work 4.9 3.2 4.7 0.2 12. We readily adapt to new circumstances 4.8 2.1 3.7 1.1 1. Our work group is known for quality work 2.8 3.9 2. Our work group is productive 3.0 4.3 As seen in tables 13 and 14, the differences between old and new teams are very small. The following two tables provide results for the three districts that have both first and second generation teams. Table 15: Workgroup Climate Assessment: Results fore the second generation teams in three districts n 14 13 21 New Teams' Average Scores Aswan Daraw Kom Ombo Importance of 12 items listed above 4.7 5.0 4.9 Actual performance before the program 2.3 2.3 2.3 Actual performance now 4.0 4.2 4.2 Remaining Gap 0.7 0.8 0.7 Table 16: Workgroup Climate Assessment: Results for the first generation teams in three districts n 15 14 3 Old Teams' Average Scores Aswan Daraw Kom Ombo Importance of 12 items listed above 4.9 4.7 4.9 Actual performance before the program 2.8 2.4 1.7 Actual performance now 4.3 4.1 4.4 Remaining Gap 0.6 0.6 0.5 LDPE Egypt Follow-up Evaluation Report 34 There are not significant differences between the old and new teams, or among the districts. Both old and new teams in the three districts indicate that their actual performance is now significantly improved as compared to the beginning of the program. The following graph summarizes the average change in work climate in the old and new teams. It may be concluded that the almost identical scores of both groups at the time the assessment tool was reapplied in April 2004 provide evidence of a successful process for replicating the LDPE. In other words, the new teams recruited, trained, and supported solely by members from the first generation teams benefited similarly from the program. Figure 11: Workgroup Climate Assessment: Results for the first and second generation teams in three districts 0 1 2 3 4 5 Importance Before joining the program Now (during April 2004 assessment) First Generation Teams (n=32) Second Generation Teams (n=48) In conclusion, these findings indicate several important points: • All participants give great importance to working in a better work climate • The LDPE helps to immediately create and maintain a better work climate • The LDPE replication process was successful in terms of creating a similar, improved climate for the second generation teams. LDPE Egypt Follow-up Evaluation Report 35 5.3 Findings from focus group discussions and interviews The first group of interviews was conducted with key Aswan Governorate staff and with the district managers responsible for replication of the program. The expected key outcomes were to: • Understand the main factors leading to the decision to replicate the LDPE • Document the replication strategy, including the selection of districts, teams and future plans • Document the replication process, including resources used and needed, and the current situation • Document behavior change among the managers, if it occurred 5.3.1 Program replication strategy What are the main factors that helped you to decide to replicate the program? What inspired your commitment to continue this program? Aswan graduates reported that they liked the program, had a feeling of commitment and ownership to it, and wanted to expand the experience to other teams. They also wanted to continue to have the opportunity afforded them by the program to express their own choices and options in determining the challenges they wanted to address and developing plans to face them. This feature was something that was absent in other interventions in which they had participated. The decision to replicate the LDPE was a group decision. The respondents felt that they could form a group that could work well together. They discussed the idea with Dr. Morsy Mansour of the MOHP, director of the original LDPE, and with Dr. Joan Galer of MSH, LDPE designer and lead trainer. Dr. Morsy and Dr. Galer were very committed and enthusiastic about the idea. Dr. Galer continued to communicate with the Aswan team and provided overall support and help through e-mails and sometimes phone calls. How different is the LDPE compared to other interventions? What are the strengths of the LDPE? The group cited several features of the LDPE that were different from other experiences. In particular, they learned how to analyze the situation, identify the difficulties, and try to find solutions. The interviewees mentioned some examples. One good example was when they ran out of supplies of the pill some time ago. They analyzed the situation and thought of a number of solutions that they adopted in order to address this major problem. They were successful in resolving the problem and keeping services going. LDPE Egypt Follow-up Evaluation Report 36 The respondents also felt comfortable having the training spaced over a period of time, allowing for practical application of each part of the program before moving on to the next one. As one interviewee said: This particular aspect of the LDPE design was mentioned as very important for the sustainability of the program, as well as for the enthusiasm for its replication. The respondents also liked the opportunity to communicate with each other, and learn from each other’s experiences during the frequent meetings among the teams. Why were the districts and clinics selected for the replication of the LDPE? Daraw and Aswan districts were selected because of their low performance indicators. Nasr and Edfo districts were not included in the replication process because it was difficult to involve them. Transportation to these districts is difficult, especially the rural areas, while Aswan and Daraw districts are closer to the Aswan Governorate. Therefore, supervision and follow-up activities were easier to conduct with the replication sites. These same factors were also considered in the selection of new areas in Kom Ombo district. It was also important to have Dr. Abdo Alswasy, who had participated in the original LDPE, be interested in the program and ready to work as a trainer and coordinator during the replication process. Finally, some of the clinics’ staff heard about the program and asked to be included in the replication. 5.3.2 Program replication process What resources were used during the replication process? Members of the first generation team reported that the financial resources made available for the replication process could not be compared to the amount of resources used to deliver the original program. In the original program, participants received per diem, nice bags, many hand outs, and meetings were held in nice hotels with fancy coffee breaks. For the replication process, the replication team tried to minimize costs as much as possible. Meetings were held at clinics and no per diem was paid to participants. Members from the original teams volunteered to provide simple, inexpensive food and beverages. They also paid out-of–pocket for other costs, such as stationary. It appears that the lack of funds affected the quality of the training environment. One example isthe quantity of hand outs distributed — in some cases the entire team from each clinic shared only one copy of the meeting agenda or other hand outs. There were also comments offered by participants that gave the impression that the physical “We got the training in several bites, not in one shot training” LDPE Egypt Follow-up Evaluation Report 37 environment of the meeting spaces was not comfortable (e.g., insufficient seats for participants, temperature in the meeting space was too hot, etc.) Inexpensive public transportation was used by both the trainers and the trainees, which was also covered out of their own pockets. Trainees were accepting of this, or at least did not complain to the evaluators. The original team members also reported that because trainees were interested and enthusiastic about the program, they were always looking forward to attending the next meeting, even without receiving an official notification about it. There was no indication that the management team at the Aswan Health Directorate provided any resources for the replication activities, either in cash or in kind. What are the similarities and differences between the first and second generation teams? How consistent is this replication program with the original one? What are the problems faced during expansion? The first generation team members mentioned that they had some difficulties in the early phases of the original program. All aspects of the program were not clear to them and the concepts were difficult to grasp. They also had problems with the written materials and hand outs distributed during the original program. Part of the problem was due to the unsatisfactory translation of materials into Arabic — the translations did not completely convey the content of the English versions. The first generation team members tried to make the program documents clearer so that they could be better understood by the new trainees. Specifically, examples and case studies were made clearer and were adapted to local conditions. The first generation team also tried to simplify the process and phases of training. They felt that the scanning workshop was difficult and tried to make it easier. They better categorized materials included in the original program. They also decided to increase the number of trainers to help each other. Modifications to the original program were first made in Kom Ombo district and were then applied in Aswan and Daraw districts. As of April 2004 the status of implementation of the replicated program with new teams in the three districts is: Aswan District The process started in December 2003 by selecting 5 new teams. Key activities completed thus far: December 10th, 2003 Teams selected and presented their challenges January 4th, 2004 Scanning workshop for both old and new teams LDPE Egypt Follow-up Evaluation Report 38 February 7th, 2004 Focusing workshop March 7th, 2004 Gap analysis April 5th, 2004 Review of teams’ gap analysis results April 11th, 2004 Review of progress and preparation for the April 17th Meeting with the evaluators Daraw District The process started in August 2003 by selecting 5 new teams. Key activities completed thus far: October 5th, 2003 Selected teams briefed on the LDPE November 30th, 2003 Scanning workshop for both old and new teams December 29th, 2003 Teams presented their selected challenges February 16th, 2004 Gap analysis and root cause analysis February 29th, 2004 Focusing workshop and action plan preparation April 5th, 2004 Teams presented their action plans to the larger group (both first and second generation teams); also a session on coaching and managing teams was conducted Kom Ombo District The process started in July 2003 by selecting 5 new teams. Key activities completed thus far: July 21st, 2003 Introduction of the program to the new teams Scanning workshop August 14th, 2003 Teams presented their selected challenges September 18th, 2003 Gap analysis and root cause analysis October 15th, 2003 Focusing workshop, definition of SMART objectives, session on different personality types November 20th, 2003 Preparation of action plans and implementation schedules December 25th, 2003 Aligning and mobilizing workshop Teams presented their action plans January 21st, 2004 Coaching workshop, overcoming barriers and problems February 26th, 2004 Inspiring workshop April 1st, 2004 Review of progress and preparation for the M&L follow-up evaluation Some trainers reported that the absence of a training curriculum for the LDPE was a problem. Although they used the available written materials from the original LDPE, they felt that they needed a more complete curriculum. Some of the trainers communicated with Dr. Galer and asked for help. She referred them to some sites on the internet where LDPE Egypt Follow-up Evaluation Report 39 they could retrieve information. These materials were to some extent helpful, but did not meet all of their needs. They did not directly ask Dr. Galer for a formal curriculum as they felt that they should try to implement the replication process by themselves. The first generation trainers also mentioned that while some of the Aswan Governorate and District Health Office managers attended some of the meetings and recognized that the trainees were very enthusiastic about the program, they could have provided more support to program. Also, monitoring activities conducted by the trainers required a lot of their time as they had to visit teams in their clinics or offices several times. At the same time, they were happy to volunteer the time needed and felt it was worthwhile. Concerning the differences in experiences between the old and new teams, some members reported that the training received from Dr. Morsy and Dr. Galer was stronger. They very much liked Dr. Galer’s training skills and methods. In fact, it seems that her character and her very special way of encouraging the whole group created a lot of inspiration and love for the program. Some aspects of the original program were not used in the replication process, such as the WCA. Some of the trainers think that applying WCA is not an essential tool and does not contribute to the program; the LDPE process is much more important. The evaluators’ impression is that WCA is perceived as an external tool and the results do not serve any purpose at the field level. In general, the trainers felt comfortable with the technical assistance they were able to provide to the participating teams as well as the amount of monitoring activities. They also perceived that the replication activities were more organized in terms of sequence and “dose”. What is the ultimate goal? Will you continue expanding? Some members of the original teams reported that they plan to extend the program to Edfo and Nasr districts, in addition to more sites in the current expansion districts. The respondents noted that the program has led to many important behavior changes in both professional and personal terms. The following are some quotes: “We learned how to identify the problem and to think of a plan to solve it…how to identify roles for team members…. who does what to achieve results” “Instead of always receiving instructions to do things, for the first time we can propose changes and our views are seriously considered” “Instead of a top-down system, staff at the base level of the clinic can think of and initiate activities” LDPE Egypt Follow-up Evaluation Report 40 How can this program become a routine system in the organization? The top level management team in the Aswan Governorate reported that the LDPE should be able to lead to measurable achievements. It should lead to improvements in the indicators used by the MOHP (e.g., increase in FP, training of staff to open new facilities, etc.) The LDPE should be regarded as a cross-cutting program. It cannot be sustainable if it “stands by itself.” Even if funds were available for the LDPE, there are other priorities for the MOHP for using any new funds. 5.3.3 Findings from focus group discussions with clinic teams The second group of interviews was conducted with the second generation team members from the three districts. The expected key outcomes were to: • Understand the challenge selection process • Document the quality of action plans • Document the implementation phase, including replication and monitoring and feedback processes • Document any behavior change among the team members In addition to findings from the application of the focus groups discussion (FGD) guide, results presented in this section also rely on findings from field visits to selected clinics, for both first and second generation teams. The clinic visits were conducted by the evaluators during the period April 20 - 22, 2004. How did you select the challenge? Who made the final decision on the challenge selected? The general procedure followed was that the participating clinic or district team trained in the LDPE makes the decision about the challenge selected. The clinic staff picked, in most cases, the lowest performing clinic indicator among MOHP national programs, mainly FP, ANC, and PP care as their challenge. Some participants mentioned that they selected the challenge so as to improve other related indicators. For example, increasing ANC visits could also increase PP care visits later on, as well as FP coverage. A few others mentioned that they selected the service that was most needed by people in their locality. "The most important thing is that each person started to learn how to develop a plan for each activity, even at the personal level, how to deal with their own children at home" LDPE Egypt Follow-up Evaluation Report 41 Some other participants said that they prepared a number of priorities for each proposed challenge. Selection criteria included the performance level of the indicator, the importance of each challenge for the community and for beneficiaries, and the possibility of achieving the desired results within the planned time frame. They prepared scores for each challenge and the one with the highest score was picked. Some participants mentioned that the availability of resources was an important criterion in the selection process: There was no variability among the teams with regard to the decision making process for selecting the team’s challenge. All teams reported that the selection was decided by the clinic team members themselves. During the clinic visits it was also noted that there was a tendency among the team members to indicate that the decision about the selection of the challenge should be made by the clinic team itself and that the trainers should not have a say or influence on the team’s decision. However, this process is not necessarily optimal, especially if the challenge selected is not the best one to choose as regards community and client needs, quality of care, etc. A balance should be struck between a team’s independence in selecting a challenge and the “oversight” role of supervisory authorities in assessing the “quality” of the challenge selected, e.g., whether it is a priority for the local population, for improvements in a facility’s performance, etc. In most cases the performance goal for the challenge selected was very modest. For example, one clinic’s challenge was to increase FP coverage from 29% to 30%. The clinic staff said that they intentionally defined a modest increment to be able to achieve the desired result. In fact, they exceeded their goal soon after they began implementing their action plan. This indicates that some technical assistance from the trainers (i.e. review of and feedback on the performance goal) is needed to make sure that teams are on the right track in the early phases of preparing their action plans. In general, the trainers from the first generation lack some of the skills of the original trainers. This was reported by some of the first generation members who had experienced the training facilitated by Dr. Morsy and Dr. Galer. What are the characteristics of the action plans? Not all of the second generation teams had prepared their action plans at the time of this follow-up evaluation. Some have finalized their action plans and had copies available for the evaluators to review, while others were still in the process of finalizing them. Since some of the teams (especially the Aswan teams) joined the program only three months "Resources are important in determining the challenge" "This was a group decision" "The management team had only an advisory role, no more" LDPE Egypt Follow-up Evaluation Report 42 ago, and the period for learning about the LDPE, preparing the action plans and implementing them (including achieving desired targets) is only six months, this may indicate that some aspects of these plans were not of good quality. All teams relied on information from their clinic registers to calculate actual (baseline) performance. In most cases, the clinic staff did the calculation by themselves (calculated the actual indicators). A few teams reported that they sent their statistics to the district management team to calculate the actual indicators for them. Also, a few teams mentioned that they had problems with file keeping and that the recording of service statistics was not complete. These teams, therefore, had to train all their nurses in file keeping for all services provided by their clinics. It was noted during the field visits to the selected clinics that there were some measurement problems. These are summarized below: • There is some confusion regarding the estimated number of the catchment area population served by the clinic. In most cases the clinic calculated its catchment area as the population of the whole administrative area, however, the whole administrative area may be served by more than one health facility. This was the case with Al Rakkaba Health Center, which did not know exactly what its own catchment area population is. The total population of the area is 7,000 and Al Rakkaba is one of three clinics serving the population. • Another measurement problem concerns the protocol for registration of the number of live births delivered in a health facility. Only the number of births for women who belong administratively to the geographic area served by the facility is considered. The remaining number of births is reported to the health facility in which the mother resides and is recorded there. For example, a woman who lives in area X moves to her parent’s home when her delivery date approaches, which is located in area Y. She delivers at a health facility in area Y. The live birth in this case is recorded in area X, her “home” health facility. The same procedure occurs for immunizations. Immunization services are provided to all people who visit a clinic. But the service is recorded in the health facility located in the administrative area in which the individual usually resides. • In estimating the number of married women of reproductive age (MWRA), which is needed to calculate FP coverage, the total population is divided by six. This assumes that 16.7% of the population is MWRA. This may not be always the optimum estimate for all clinics and districts. In some areas, this figure differs substantially from reality and leads to over- or under- estimation of performance. It may be better to rely on the actual data available. In small rural areas especially, the actual number of MWRA is preferred over using a formula. LDPE Egypt Follow-up Evaluation Report 43 • Some indicators are calculated on a monthly basis, leading to fluctuations from one month to the next, especially if the number of events is small, as for instance the number of births or deliveries in a small catchment area. • In general, only a few clinic staff seemed inexperienced in the procedure for the correct calculation of indicators. Also, a few of them reported that they send their service statistics to the district health office to calculate the indicators for their clinics, as the district health office has the data on the population size in the catchment areas. • Desired performance was in most cases based on a team’s expectation of what it could realistically achieve. In some cases the team tended to propose a low desired performance to make sure it would be able to achieve it by using their own resources. • All teams were active in identifying the activities to be included in their action plans to help them achieve their desired results. In most cases, they choose the same activities that their clinics usually conduct, but increased the frequency of these activities (e.g., more frequent home visits, or more nurses conducting home visits than was the case before participating in the LDPE). However, several teams thought of new activities. Examples include conducting a population day which involved intensive clinic-based and outreach IE&C activities and service provision through mobile teams. Also, one team enumerated the population in the clinic catchment area as it had no accurate number to use for calculating its indicators. The need to drop some or any of the planned activities was rarely mentioned by the teams interviewed. Indeed some teams mentioned that they had added activities that were not originally planned, as it seemed necessary to improve the results. With regard to resources available to the teams to implement their plans, some said that they considered the availability of resources when they began preparing the action plan: “In preparing the action plan, we thought from the very beginning of activities that could be done with the least cost” “We added some activities to increase the number of ANC visits…we encouraged some organizations to partner with us, for example, women’s clubs” “It is possible that after preparing the plan, we find some shortcomings or think of better alternative activities that could increase results” LDPE Egypt Follow-up Evaluation Report 44 “One basic idea of the leadership program is to help others achieve results in difficult conditions” Also human resources were mentioned by many teams to be in short supply as compared to their needs. This was particularly true with regard to female physicians and, to a lesser extent, nurses. The lack of equipment was also reported by some teams. The district management team tried to help sometimes, but not all of what was needed was provided. With regard to time available to implement planned activities, there was near consensus among the teams that the time frame was not sufficient. They said that immunization and ANC campaigns were frequent and keep them busy. What are the main issues with the implementation phase? Participants’ reports differed with regard to activities implemented relative to the time frame. Some teams said they were behind schedule and others said they were on track. Monitoring of activities takes place through clinic staff meetings every two weeks and the monthly meeting with all teams and management staff. Sometimes clinic staff had to postpone the clinic meeting if they had an immunization campaign, but they did not cancel it. During these meetings they discussed and reviewed results achieved, obstacles met and how they could be addressed, as well as roles of team members. They also met every two months with the trainers. Most of the teams reported that the Aswan Governorate and District Health Offices provide support sometimes, if possible. It should be noted, however, that the Aswan Governorate managers provided only normal, routine administrative support to the districts; it did not otherwise support the expansion of the LDPE. For example, if a clinic runs out of medical supplies, if they need to train staff in laboratory procedures, if they want to replace some staff (low performers) and get more competent staff, or if they need to know how to calculate a certain indicator, the management staff do help. Also, some respondents mentioned that the District Health Office staff sometimes provides “Resources are not enough; I don’t have enough human resources. I have to do all outreach activities in addition to activities in the clinics” “I may be planning for some home visits, but then I get busy with immunization campaign activities. Time is not enough [for the action plan]” LDPE Egypt Follow-up Evaluation Report 45 financial or non-financial rewards to recognize outstanding achievements (as in the case of postnatal home visits). There were some difficulties that the management staff could not help with. For example, if there was a shortage in the number of physicians. This was a difficult problem to resolve. Do you think that you have understood the program? Are there some unclear concepts? Participants’ responses differed to some extent between the old and new teams. The first generation teams were more likely to report that they understood the program. Some members of the first generation, though, mentioned that they are still learning. Most of participants from the new teams reported that they are still learning about the program: How do you assess the quality of trainers, meetings, and written materials? All participants were very positive about the quality of the trainers. They liked their method of training and the way they communicated with the participants. They said that the information they were given was not new to them, they knew it, but the trainers “arranged their thoughts” and trained them in how to use the information in the right way. As noted earlier, however, some of the first generation participants who had been exposed to the original trainers felt that the quality of the trainers was not as good. “I feel we are still in the beginning, and there are many things that I still need to know” “We are still leaning by doing, I feel I’ll still learn more than what I’ve already got” “They first listen to us, give us a space to express our views, and then they comment on what we said, and add some more explanation” “Although I’m an old member in the old team and have good understanding about the program, I feel that I always know more and benefit more. Even if I didn’t get something new from the old trainers, I learn more though sharing information and experience with other teams” LDPE Egypt Follow-up Evaluation Report 46 Participants were very enthusiastic and positive about the program in general. One participant said: Another participant said: And: However, some team members were concerned that they were moving slowly as they were supposed to achieve their goal in six months and they were still learning the steps in implementation (and they were now in mid-April). In other words, they were suggesting that less time might be given to learning how to think in organized way and how to determine the challenge, and more time might be needed for the implementation phase. What is different about this program compared to other interventions? Participants described how the LDPE is different from other programs: “This is a new way of education, how to learn by yourself, how to speak out, how to arrange your thoughts and think in the right way…this was very good way to learn new skills” “Dr. Abdo told us you are not going to receive any incentives in this program if you join it, on the other hand, you may bear some costs. But despite that, because we heard about it, we joined and liked it” “When we first came to attend the meetings, we thought that the clinic director will be the leader, but we realized that every one of us is a leader” “When we joined the program and understood it, everybody felt that not only had we made a plan for the clinic, but also inside ourselves and at home. Every thing started to be planned and organized. The program made changes in my character, not only in the workplace” “The last decision was always ours. We discussed with trainer’s many formats, and the one we implemented was the one we chose” ‘In other programs, other persons were making them, but in this program, all steps in the program were done by us, nothing was imposed to us” LDPE Egypt Follow-up Evaluation Report 47 Some participants mentioned that the program is different in terms of improving quality, not only in achieving results. They also mentioned that they now (after the program was introduced) like their work and are not pushed to do it, and this made a big difference to them. Others mentioned that the program helped them to be self-reliant and self￾confident, and better able to solve problems they encountered. They also mentioned that they started to pay attention to the results achieved. Participants also mentioned that they had started to do some things differently after the program. For example, they began to interact in a better way with clients and to encourage them to make return visits to the clinic as needed. They also began to hold meetings at clinics so that clients could come and bring others with them to benefit from the information given. Additionally, the treatment of clients has improved, time management by clinic staff has improved, and they have started to work as a team. Furthermore, the program initiated competition among teams to achieve better results. Some other interviewees mentioned that they had learned they could replace each other if some team members were absent for any reason. In this way the work can continue. In addition, they started to know the role each member played in the clinic and tried to help or replace any person as needed. The respondents also described changes in their professional life as a result of the program: In the personal realm, participants reported some changes in behavior: they were better able to solve problems and to assign priorities and they started to speak out and to not be shy (mentioned by a female communications person). They were also more committed and organized, and they were able to discuss things with family members and involve them in decision making. ‘We used to send our indicators to the management and then they were sent to the health directorate in a routine way; it is only after this program that I have started to review results to see if I’m doing better than last month” “Before the program we had no target to achieve. Now we have a target and we work too hard to achieve our target” “I have younger brothers. Before the program I used to tell them what to do. Now, I discuss with them what they would like to do, allow them to do it as long as it is fine and no problem with it” “I wrote on a sheet of paper: I can do it and I hang it on the wall. God will help me do it” LDPE Egypt Follow-up Evaluation Report 48 If you could make one recommendation to donors, what would it be? Participants’ replies varied greatly in response to this question. Rather than making recommendations for funding the LDPE, the participants decided to address broader, priority issues. Recommendations included: establishing projects for street children, the elderly and handicapped; application of the leadership program in other areas, such as education; scaling-up the program to the national level; giving attention to nurses more than physicians as they work too hard; upgrading the clinic; conducting research to develop a FP method with no or minor side effects; and obtaining more equipment for the clinic. When you review the Principles for Developing Managers Who Lead, can you comment on the role/importance of each of these in the continuation of your program?4 Some participants mentioned that achieving results is important, though they think that it is not always possible to achieve 100% of the target — demonstrating commitment and achieving at least some results are acceptable as well. Participants think that leadership and commitment can be learned and gained at every level. And leadership is not based on hierarchy; skills must be gained to practice it. Finally, participants mentioned that the leader should be able to communicate with others at different levels. 4 The M&L’s Principles are provided in Annex 4 LDPE Egypt Follow-up Evaluation Report 49 6. CONCLUSIONS The most obvious result of the LDPE is that it has had a tremendous impact on the behavior of teams, at both the personal and professional levels. The evaluation team did not meet any single participant who was not enthusiastic about the program. There was no difference between the first and second generation teams. Despite initial difficulties, staff managing the replication process was able to create the same learning and participatory environment as existed in the original LDPE. The program has greatly contributed to creating a better workgroup climate. The workgroup climate has dramatically improved among the new teams and has been successfully maintained among the old teams. The program is perceived as a powerful tool to improve performance by all participants. The program’s participatory approach has enabled front line service providers to actively participate in discussions and to design and implement small￾scale service improvement projects. The majority of first generation teams have been able to demonstrate moderate to significant improvements in service statistics related to their selected challenges. Since the ultimate goal of any such intervention is to improve health services and the health of communities, this is an important finding and achievement of the program. The program has helped to open communication channels between clinic/district managers and service providers. Challenges selected by both old and new teams were limited in scope. Although it is appreciated that clinic managers and staff were able to address challenges at their clinics, there should be a broader perspective and recognition of the most important primary health care challenges that impact health conditions of the population in the teams’ catchment areas. Challenges The desired performance defined by most clinic teams was so modest that it is hard to detect any change using the MOHP’s standard indicators. Several teams selected to increase their FP service performance by one or two percentage points over the next six months. Using the MOHP’s CYP based indicator, which is very sensitive and selective towards long-term methods, the degree of change will be achieved or not achieved rapidly by insertion or removal of several IUDs. This is an especially important issue for the clinics serving a small catchment population. While clinic staff focused on choosing targets that the team would be able to achieve given their limited resources and within the time available, they did not thoroughly assess how the indicators selected would affect the results. According to their statistics, several teams had achieved their targeted results in the very first months of implementation. This negatively impacted the magnitude of results achieved. LDPE Egypt Follow-up Evaluation Report 50 Clinic staff understand the importance of quality from the client’s perspective, have started to pay attention to client satisfaction with clinic services, and have worked hard to meet their expectations. However, staff do not know how to measure and monitor client satisfaction. Clinic staff have been trying to collect information through informal, random chats with clients. They have tried to formally measure client satisfaction but have not succeeded as they did not find a source of technical assistance. Nevertheless, this positive outcome of the program — clinic staff’s attention to quality of care — should receive due encouragement and appreciation and should be strengthened. Clinic teams need to pay attention to the quality of targets achieved, not just the quantity (e.g., not only the number of ANC home visits conducted, but also the content of those visits and the real benefit to the pregnant woman’s health as well as the fetus). Additionally, appropriate follow-up of health problems identified during home visits needs to be emphasized. The quality of counseling in FP by the recently trained nurses should also be assessed. In some cases nurses were trained by their clinic managers or by the nurse usually providing FP services. Formal FP counseling training has not yet been made available. The duration of the replication process — the preparation phase and development of action plans, is very short (six months). The majority of the teams reported that they were behind schedule in implementing their plans. The short duration of the replication process may negatively impact the quality of training received and action plans implemented. As noted on pages 37 and 38, and summarized below on page 52 (review and status of second generation teams’ training to date in the components of the LDPE), the logical flow of the original LDPE is not being followed in the replication process. For example, Aswan District teams selected and presented their challenges before receiving training in the leadership practices of scanning and focusing. Teams from the other two districts have also selected their challenges and/or prepared their action plans before the workshops on focusing, gap analysis, and aligning/mobilizing have been organized and offered. It is of concern that some respondents seem to favor more time being devoted to action plan implementation than to the core of the LDPE’s original intent: to develop an understanding of the essential leadership functions and to allow managers to practice this new knowledge and skill-set while developing and implementing an action plan. The teams and the overall replication process have not received substantive support from the Aswan Governorate. Although senior governorate managers have been verbally supportive and praise the program, the replication process and the teams’ action plans have not received a desired degree of interest and priority. Any other program, initiative, or project always has a greater priority compared to the LDPE. This translates into modest targets which rely on limited resources. As one manager dedicated to the replication of the program mentioned: “This program helps people to solve problems without additional resources” LDPE Egypt Follow-up Evaluation Report 51 This reality limits the scope of the challenges selected and may lead to frustration in the future since improving quality of care and expanding services eventually require funds. When one district manager was asked whether he had ever asked for funds from the governorate, he replied: One key staff member from the first generation mentioned that: 90% of the teams selected FP, ANC or PP care as their challenge areas. While these are critical services, it should be kept in mind that other priority primary health care services should also receive adequate attention. During the field visits clinic managers were asked about the most important health problems in their catchment area. Most of the managers were not able to respond to this question. One clinic manager listed their priority problems in the following order: • Rheumatic heart disease following tonsillitis • Parasitic infections (mainly Bilhariasis) • Pelvic Inflammatory Disease • Water pollution Despite the health problems described above, the clinic team decided to select FP as their challenge. Another clinic’s selection of PP care cannot be justified given the very low number of deliveries in their area: 1 to 6 per month over the last 15 months. One contextual issue that must be made is the interest of USAID, which funded the original LDPE. The LDPE was designed and delivered with USAID population core funding, with the MOHP, and especially the MOHP staff in the Aswan Governorate, as prime M&L stakeholders. USAID and the MOHP’s interests necessarily influence the focus of challenges selected by the teams in essential reproductive health services, as opposed to other, broader primary health care services. Moreover, the continued involvement of MOHP staff at the local level in the replication and expansion of the program tends to influence the focus of district and clinic staff in their selection of challenges. The LDPE does not have a fully developed training curriculum. Districts have been following different training workshop sequences. As of April 2004, the workshop sequences and completed steps were as follows: “We already know the answer, so we did not ask for money” “This program has missed central and governorate support” LDPE Egypt Follow-up Evaluation Report 52 Aswan Daraw Kom Ombo Select challenge Scanning Scanning Scanning Select challenge Select challenge Focusing Gap analysis Gap analysis Gap analysis Focusing Focusing Action plans Action plans Aligning & mobilizing Coaching Action plans Coaching Inspiring This leads to temporal conflict between the training, action plan development, and implementation. Several second generation teams had to select challenges, set targets and prepare action plans before receiving all the necessary training. As noted above, there is consequently concern about the quality of training received to date and of the action plans being implemented. LDPE Egypt Follow-up Evaluation Report 53 7. RECOMMENDATIONS Recommendations for MSH A. Program design The program design should enable district and clinic managers to identify, assess and prioritize health problems and select valid challenges accordingly. The current LDPE practice is to focus on and select a service area (or a national program) rather than a priority health problem. Quantifying the health problem and analyzing current trends should be the logical first step. It is important to keep in mind that individual, family, and environmental factors, resources, and the availability and quality of services all contribute to the current level of health problems. In other words, services provided are not the only determinants of health. Thus, relying only on service statistics for determining the challenge may be misleading. Data on health indicators among the population served by the districts, or catchment area served by individual facilities, are a critical, supplemental source of information to ensure that the challenge selected is a priority of the community and not solely of the concerned ministry or donor. B. Selection of challenges As mentioned earlier, all first and second generation teams selected their challenges from among the MOHP priority programs. Although these are important programs, they may not necessarily reflect the most important primary health care problems in all of the clinics. At the same time, as mentioned in the Conclusions section, the selection of challenges is influenced by the fact that the original LDPE was implemented under the direction of the MOHP and M&L, and the replication was implemented under the direction of reproductive health-related managers. M&L has tools and proven approaches that could be used for this purpose. For example, M&L’s Indonesia program has been making notable progress in working with district teams to first identify and assess the importance of health problems, and then select the most appropriate services to address the problems. Tools and approaches developed for that purpose can be adapted for future program replication. C. Mobilization of additional funds Teams should be encouraged to actively seek additional funds and to learn how to mobilize external resources. It is recommended that M&L address advocacy for additional funding in the content of the “mobilizing/aligning” workshop. D. Technical assistance on program monitoring Most of the teams and the trainers lack a clear understanding and knowledge of how to set targets as well as the basics of health program monitoring. They should be trained on the importance of different data sources, the limitations and advantages of each data source, and how to interpret results. Both first and second generation teams need more training and technical assistance in this regard. LDPE Egypt Follow-up Evaluation Report 54 E. Technical assistance on health program management: What the trainers and the teams have done with the preparation of the action plan also needs some refinement. From the beginning of the LDPE design phase and through its implementation in 2002-2003, a critical issue has been the absence of adequate technical assistance on the health program and service delivery challenges the participants select. The following excerpt from M&L’s October 2003 Evaluation Report is therefore reiterated: The teams’ lack of good training/knowledge in the selected health programs was an important gap. Better technical knowledge on selected health programs could have led to even better results. The teams would have benefited from access to state-of-the-art knowledge on these health programs. Being located in Aswan Governorate, the teams are distant from the national and international community. Technical components of these services, historical background, and success or failure stories from around the world should have been communicated and discussed with the teams. The LDPE was not staffed with local personnel who had knowledge about national and international experience. It is important to fulfill MSH’s mission, which is “closing the gap between what is known about public health programs and what is done to solve them”. An intermediate step should be added to the program design in the future. After the selection of challenges, program staff should be provided with technical information on those health programs. The rationale for selecting those challenges, how to address these challenges, best practices from both national and international experience could be the main topics of such assistance. Results also indicate that the teams needed technical assistance in setting better targets. Several teams chose too modest targets that planned change could be due to monthly or seasonal variation or measurement error. Such a perspective and technical assistance provided to program participants would definitely improve the action plans and might yield better results. This recommendation made last year is still valid. The replication process seems to not have addressed this important issue. The absence of a better understanding of the technical issues concerning the selected challenge, including appropriate measurement, affects the unqualified demonstration of well-deserved success. Skepticism on the part of senior managers and key decision makers can be easily eliminated by demonstrating measurable and comparable results. LDPE Egypt Follow-up Evaluation Report 55 Recommendations for CATALYST and the MOHP A. Marketing the program The program needs to be better marketed to senior managers at all levels. Political support is of ultimate importance for this program to expand further and for the achievements made to date to be sustained. B. Need for a training curriculum The first generation trainers need technical assistance to develop a training curriculum for the LDPE. They also need some training to gain skills as trainers, as some of the current trainers have not received training in adult training techniques. Therefore, if it is desirable to encourage them to serve as trainers, they should be prepared to do a good job, and should be well equipped. C. Need for better indicators Clear, sound indicators for use at the clinic level need to be developed by the MOHP. Some of the indicators currently used are not appropriate at the clinic level, especially for clinics where the catchment area population is too small for the use of coverage estimates. As mentioned earlier, direct measurement counts are more desirable at facilities where the population size is small. The MOHP should consider forming a multidisciplinary steering committee to introduce such measures and related techniques in coordination with its staff. The committee may include among its members social scientists, public health specialists, demographers as well as representation from management and supervisory teams and clinic staff. Clinic staff need to be better equipped with more training on concepts used in calculating indicators, such as catchment area population, FP service coverage, etc. D. Program support from Governorates The relationship between the district and Governorate management teams and clinic staff needs to be strengthened. District managers and clinic staff need to involve senior managers in the early stages of preparatory work and action plan development. In this way, it is more likely that senior managers will provide more support and show more responsiveness to requests raised by clinic staff for specific support needed, including additional human and financial resources. The program has the merit of making tremendous positive behavior change in both the professional and personal lives of clinic staff. The teams recognize the advantage of team work, set goals to achieve, and were able to achieve results. They also worked hard to improve the quality of services provided and gave more attention to clients’ needs. Additionally, they tried to fill gaps in services provided by clinics, tried new activities to maximize results achieved, and explored new ways of doing things. These advantages merit trying to sustain the program. In the meantime, the program should be strengthened based on recommendations provided in this report. LDPE Egypt Follow-up Evaluation Report 56 If further replication or scaling up of the program are to be considered, it is important to maintain those aspects of the program that were most appreciated by the clinic teams, especially those which motivated them to behave differently and achieve results. The participants liked the bottom-up approach having the space to choose their own goals, and the sequential phases of training with its practical components. Furthermore, they appreciated having the opportunity to share information with other teams and exchange experiences. In view of the reports of the senior management staff about the need to consider the LDPE as a cross-cutting program and to incorporate it within another program, it seems that it would be a good idea to incorporate it into the MOHP’s module on on￾site training and facilitative supervision. This module was pilot tested by the Population Council two years ago and was a great success. In this way the LDPE would be sustainable and could continue to help promote quality, strengthen team building, inspire clinic staff to achieve results, and move teams in the right direction to promote health and maximize client satisfaction and other client outcomes. LDPE Egypt Follow-up Evaluation Report 57 Annex 1 LDPE 2003 Evaluation Note The Leadership Development Program in Egypt (LDPE) was a one-year pilot program, co-led by the Ministry of Health and Population (MOHP) and the Management and Leadership Program (M&L) of Management Sciences for Health (MSH). The overall purpose of the program was to improve the quality and accessibility of health services in Egypt, specifically in three districts of the Aswan Governorate, by: • increasing the capability of managers to lead others to achieve results; and • increasing their ability to create climates of high performance in their workplaces. The LDPE began in June 2002 and ended in June 2003. It was evaluated by the M&L Monitoring and Evaluation Unit during June 15-23, 2003. The LDPE evaluation plan focused on: measuring the managers’ ability to lead others through the development and implementation of an action plan, creating a better workgroup climate, and achieving service delivery results. The evaluation framework included seven indicators measuring the following leadership elements: Selecting a Challenge, Scanning, Focusing, Aligning and Mobilizing, and Inspiring. This evaluation demonstrates that the ten participating teams produced moderate to significant results at the service delivery level. The majority of the leadership indicators were achieved as well. Results of the leadership indicators measured are: Select Challenge 100% of the teams identified actual challenges. Scan 50% of the teams collected complete valid data. 50% of the teams collected partial valid data. Focus 100% of the teams prepared written action plans with measurable outputs and a time frame. Align & Mobilize 100% of the teams prepared a written action plan defining human and financial resources needed to implement the plan. Achieve Results 70% of the teams achieved 95% or more of their performance objectives. 10% of the teams achieved 33% of their objectives. 20% of the teams did not demonstrate any progress in achieving their objectives. Inspire Workgroup climate improved dramatically in all ten teams. 80% of the teams selected a new challenge, without prompting. In conclusion, the one-year program was very successful in producing results at the clinic and district levels, improving workgroup climate, creating enthusiasm, and inspiring participants in leadership and performance improvement. Before scaling up the program or transferring it to other countries, some design modifications are recommended. LDPE Egypt Follow-up Evaluation Report 58 ANNEX 2 Interview Guidelines -1- Key questions to be discussed for the assessment of program replication Expected key outcomes: • Understand the main factors leading to replication decision • Document the replication strategy including the selection of districts, teams and future plans • Document replication process including resources used and needed, current situation • Document behavior change among managers if occurred Target persons: In Aswan Health Directorate Dr. Barakat Dr. Abdo Dr. Srouji Dr. Morsy Mansour Others in Aswan, Daraw and Kom Ombo directorates Based on the specific objectives mentioned in the Scope of Work, the following key questions should be discussed with the target persons: 1. Assessment of the program replication strategy Decision • Who has decided to replicate the program to new sites? • Was it a common decision or imposed by the MOHP? • What are the main factors that helped to decide on replication? • What has inspired your commitment to continue this program? • Why did you choose LDP for replication? LDP program characteristics • How different is LDP program compared to other interventions? • What are the strengths of the LDP program? • What are the design components that affect the program outcomes? • What parts of the LDP program were most helpful in sustaining commitment? Bi-monthly workshops, monthly district meetings, clinic team meetings. • What are the weaknesses of the LDP program? • What are the main impacts of this program? What difference has it made to results in the MOHP? What has changed because of this program? Selection of districts • Why did you choose Aswan, Daraw and Kom Ombo for replication? LDPE Egypt Follow-up Evaluation Report 59 • Why didn’t you choose Nasr and Edfo for replication? • What are the differences between those districts? Selection of clinics • How did you select the clinics/teams? • Are there any clinics that you have to drop from the original list? If so, why? • Are there written criteria for selection? 2. Assessment of the program replication process Resources • What are the resources that have been used during the replication process? • What kind of technical assistance have you received since the completion of MSH program? • What are the technical and financial needs for maintaining the program? • Do you feel confident for providing technical assistance to the teams? • Do you think that the teams have received adequate support and feedback during the course of the program? Current situation • What is the current situation for the original program teams? • What is the current situation for the replication teams? • What are the similarities and differences between the first and second generation teams? • Are there differences between the first and second generation teams in terms of receiving support and feedback? Process • What are the details of replication? • How consistent is this replication program with the original one? • Do you have a written timetable for expansion? • Are you using WCA? If yes, why? If no, why? Success/failure factors • What are the factors that may impede successful replication? • What are the factors that may help to better perform? • What are the problems faced during expansion? Future plans • What is the ultimate goal? Will you continue expanding? • How can this program become a routine system of the organization? LDPE Egypt Follow-up Evaluation Report 60 3. Assessment of behavior change • What are you doing differently since the beginning of this program? • What have you learned about leading others to achieve results? • What is the most important change that this program has brought into your professional life? • What is the most important change that this program has brought into your private life? • If you could make one recommendation to fenders, what would it be? • When you review the Principles for Developing Managers who Lead, can you comment on the role/importance of each of these in the continuation of your program? Principles for Developing Managers Who Lead: • Managers who lead enable groups of people to face challenges and achieve results in complex conditions. Results are the true measure of leadership commitment. • Leading and managing are commitments and practices that can be carried out by people at every level of an organization. • You can learn to lead. Leadership commitments and practices improve through a process of facing challenges and receiving feedback and support. • Developing “managers who lead” is a process that takes place over time. This process works best when it is owned by the client organization and addresses critical organizational challenges. • Positive changes in commitments and practices are sustained when they are part of the organization’s routine systems. LDPE Egypt Follow-up Evaluation Report 61 ANNEX 2 Interview Guidelines -2- Key questions to be discussed for the assessment of teams’ service delivery achievements Expected key outcomes: • Understand the challenge selection process • Document the availability and quality of action plans • Document the implementation phase including replication process including the monitoring and feedback mechanisms • Document behavior change among team members if occurred Target persons: 1st Generation 2nd Generation Districts Teams Persons Teams Persons Aswan 4 12 5 15 Daraw 5 15 5 15 Kom Ombo 1 3 5 15 Total 10 30 15 45 Based on the specific objectives mentioned in the Scope of Work, the following key questions should be discussed with the target persons; 5. Assess the selection of challenges Prioritization • Do you think that you have enough evidence for selecting that challenge? • On what basis you prioritized the problems for selection? • What would be the second challenge you would select? Why did not you select that one? Process • How did you select the challenge? Did you discuss as a whole team? • Who made the final decision on the selection? • Did the district management team suggest a challenge? • Did the district management team criticized or disapprove your selection? • Is all the staff in your clinic/unit aware of that challenge? LDPE Egypt Follow-up Evaluation Report 62 6. Assess the availability and quality of action plans Availability • Do you have a written action plan? • If so, can we have a copy of your action plan? Quality • How did you calculate the actual performance? • How did you determine the desired performance? • How long did it take to prepare the action plan? • Do you think that these activities in the action plan are sufficiently addressing the challenge? • Are there any other key activities that you should include? If so, why did not you include those as well? • Do you think you have enough resources to complete all the activities in your workplan? • Do you think that you have enough time to complete all the activities in your workplan? 7. Assess the implementation phase and results Implementation • Are you on schedule according to your timetable? • Are there any activities that you had to drop? • Are there any activities that you had to include to your action plan? Monitoring, feedback and technical assistance • How do you monitor the implementation? • Do you regularly review the results in your clinic/unit? • Do you think you have received adequate support and feedback from district managers? 8. Assess the changes in attitudes and practices of teams Program Image • Do you think that you have understood the program? • Are there some unclear concepts? • How do you assess the quality of trainers, meetings and written materials? • What is different about this program compared to other interventions? Practices • What are you doing differently since the beginning of this program? • What have you learned about leading others to achieve results? LDPE Egypt Follow-up Evaluation Report 63 • What is the most important change that this program has brought into your professional life? • What is the most important change that this program has brought into your private life? • If you could make one recommendation to funders, what would it be? • When you review the Principles for Developing Managers who Lead, can you comment on the role/importance of each of these in the continuation of your program? Principles for Developing Managers Who Lead: • Managers who lead enable groups of people to face challenges and achieve results in complex conditions. Results are the true measure of leadership commitment. • Leading and managing are commitments and practices that can be carried out by people at every level of an organization. • You can learn to lead. Leadership commitments and practices improve through a process of facing challenges and receiving feedback and support. • Developing “managers who lead” is a process that takes place over time. This process works best when it is owned by the client organization and addresses critical organizational challenges. • Positive changes in commitments and practices are sustained when they are part of the organization’s routine systems. LDPE Egypt Follow-up Evaluation Report 64 ANNEX 3: List of persons interviewed Cairo: Yehia El Hadid, MOHP Morsy Mansour, Catalyst Project Ton van der Velden, Catalyst Project Linda Casey, Catalyst Project Mona Khalifa, UNFPA/Egypt Aswan Governorate Ayman Ragab Barakat ElShazely LDPE Managers Fatma Mohamed Shakatawy Abdo El Sweissy Mohamed Sorour Aswan District Soheir Sabri Seiam Heba Gamal Soad Abd ElMoeti Omar Mahdy Ismail Diab Nefissa Ali Nasser Aziza Abd ElFadeel Mohammed Kamal Ahmad Mohammad Helal Fatma Mohammad Hossam Eddin Ahmad Hussein Mahmoud Mohammad Reda Tarek Mostapha Kamal Eman Mohammad Shawki Zeinab Mohammad Ramadan Soheir Abbas Fatma Mohammad Abdo Mounira Nasr Eddin Ashraf Aziz Hussein Sabrin Amin Abbas Tahany Ibrahim Mohammad Mohammad Ahmad Fouad Attiat Sabri Abdo Finy Fine Isaaq Attiat Tohami Mohammad Zeinab Othman AbdEllah Daraw District Hamdy Abd ElKarim Mohammad Mona Naguieb Taha Samia Taher Abd ElAzeem Wagih Mohammad Farahat Nawal Nasr Eldin Mohammad Hoda Younes Ali Mona Gomaa Huessien Sayed Youssef Omar Amr Mahmoud Abd ElKafi Abdou Ahmad Abd ElRahman Mohammad Hareedy Mohammad Wafaa Othman Radwan Azhar Abd ElWahab Othman Sayeda Seliman Eissa Hanaa Hamed Ali Soheir Abbas Nasra Shahat Ali Sameeha Hassan Hamed Omar Neama Ramadan Mostapha Amina Sayed Ahmad Soso Ahmad Hamed Asmaa Saad Ahmad Shadia Abd ElSattar LDPE Egypt Follow-up Evaluation Report 65 Kom Ombo District Omar Yousef Badeha Mostafa Engruid Edward Bolos Envelin Saber Fahmy Elham Mohammad Abd ElRehim Neama ElSayyed Mahmoud Shawki Abd ElMoeti Ahmad Mai Galal Abd ElRehim Yasmine Boshra AbdAlla Medhat Bahig Dawood Badri Khair Hussein Ayman Elnoy Elbadry Mabrouka Edris Omar Manal Mohammad Khalil Ezz Eddin Hamed Yassin Yousria Abd ElHamid Ahmad Youssef Rabei Ahmad Rizk Eskaros Nabeel Salah Eddin Ghazi Ali Mahmoud Ali Maissa Saeed Karam Salma Mohammad Mostapha Hoda Mohammad Ali Galal Mostapha Abd ElKereem LDPE Egypt Follow-up Evaluation Report 66 ANNEX 4: Principles for Developing Managers Who Lead • Managers who lead enable groups of people to face challenges and achieve results in complex conditions. Results are the true measure of leadership commitment. • Leading and managing are commitments and practices that can be carried out by people at every level of an organization. • You can learn to lead. Leadership commitments and practices improve through a process of facing challenges and receiving feedback and support. • Developing “managers who lead” is a process that takes place over time. This process works best when it is owned by the client organization and addresses critical organizational challenges. • Positive changes in commitments and practices are sustained when they are part of the organization’s routine systems.