EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT September 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Donna R. Dinkin and Robert J. Taylor through the Global Health Technical Assistance Project. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 09-001-188) is available in printed and online versions. Online documents can be located in the GH Tech website library at www.ghtechproject.com/resources.aspx. Documents are also available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT i ACRONYMS ACT AIDS Care and Treatment Project (Nigeria) ADRA Adventist Development and Relief Agency CHAN Christian Health Association of Nigeria CSL Commodities Security and Logistics Division (USAID) CSO Civil society organization CTO Cognizant Technical Officer ESAMI Eastern and Southern African Management Institute FBO Faith-based organization FFY Federal fiscal year FP Family planning GEN Global Exchange Network for Reproductive Health GH Bureau for Global Health GH Tech Global Health Technical Assistance Project GHS Ghana Health Service HIV/AIDS Human immunodeficiency virus/acquired immune deficiency syndrome IR Intermediate result LDP Leadership Development Program LMS Leadership, Management, and Sustainability Project LWA Leader with associates M&E Monitoring and evaluation M&L Management and Leadership Project MCH Maternal and child health MOST Management and Organizational Sustainability Tool MSH Management Sciences for Health NGO Nongovernmental organization PEPFAR President’s Emergency Plan for AIDS Relief PRH Office of Population and Reproductive Health (USAID) PRONICASS Programa de Apoyo al Sector Social de Nicaragua (LMS Nicaragua) RH Reproductive health SDI Service Delivery Improvement Division (USAID) ii EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT USAID United States Agency for International Development VBPH Virtual Business Planning for Health VLDP Virtual Leadership Development Program VSPP Virtual Strategic Planning Program WHO World Health Organization EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT iii CONTENTS ACRONYMS ...................................................................................................................................... i EXECUTIVE SUMMARY .................................................................................................................. v PROJECT SUCCESSES ..................................................................................................... v LMS MANAGEMENT SYSTEMS .......................................................................................vi CHALLENGES .................................................................................................................. vii KEY LESSONS LEARNED .............................................................................................. viii FUTURE STRATEGIC DIRECTIONS .............................................................................. viii I. INTRODUCTION ................................................................................................................ 1 II. METHODOLOGY ............................................................................................................... 3 SOURCES OF DATA ......................................................................................................... 3 METHODS OF QUALITATIVE ANALYSIS ........................................................................ 4 ATTRIBUTION .................................................................................................................... 5 STUDY LIMITATIONS ........................................................................................................ 5 III. PROGRAM SUCCESSES .................................................................................................. 7 SUCCESSES IN FAMILY PLANNING AND REPRODUCTIVE HEALTH.......................... 7 SUCCESSES IN OTHER HEALTH SERVICE PROGRAMS ............................................. 9 BROADLY APPLICABLE TRAINING METHODS AND TOOLS ...................................... 10 DEVELOPING LEADERSHIP AND MANAGEMENT AT EVERY LEVEL........................ 11 IV. LMS MANAGEMENT SYSTEMS ..................................................................................... 13 RESPONSIVENESS AND FLEXIBILITY .......................................................................... 13 COMMUNICATIONS ........................................................................................................ 14 FINANCIAL MANAGEMENT ............................................................................................ 15 MONITORING AND EVALUATION .................................................................................. 16 PARTNERSHIPS AND RELATIONSHIPS ....................................................................... 16 HUMAN RESOURCES MANAGEMENT .......................................................................... 16 V. CHALLENGES ................................................................................................................. 19 EXTENSIVE NEED FOR TRAINING AND TECHNICAL ASSISTANCE ......................... 19 CAPACITY LIMITS OF LMS ............................................................................................. 19 DIFFICULTIES IN MEASURING AND EVALUATING CAPACITY BUILDING ................ 20 ASSURING SUSTAINABILITY ......................................................................................... 20 VI. KEY LESSONS LEARNED .............................................................................................. 21 WORK AT THE MOST STABLE LEVELS OF THE SYSTEM ......................................... 21 ASSOCIATE AWARDS OFFER LOCAL BENEFITS ....................................................... 21 VIRTUAL PROGRAMS HAVE PROMISE ........................................................................ 22 iv EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT VII. FUTURE STRATEGIC DIRECTIONS .............................................................................. 23 EMPHASIZE SUSTAINABILITY ....................................................................................... 23 SUPPORT LIFELONG LEARNING .................................................................................. 24 UPDATE AND MAINTAIN THE LIBRARY OF TOOLS AND METHODS ........................ 25 BUILD A CONSTITUENCY FOR LEADERSHIP AND MANAGEMENT .......................... 25 FIND BETTER WAYS TO MEASURE SUCCESS OF LEADERSHIP AND MANAGEMENT ................................................................................................................ 26 LEGITIMIZE BROAD PROGRAM APPLICATION ........................................................... 27 APPENDICES: APPENDIX A. SCOPE OF WORK ................................................................................................ 29 APPENDIX B. EVALUATION WORKPLAN .................................................................................. 37 APPENDIX C. TEAM PLANNING MEETING AGENDA ................................................................ 39 APPENDIX D. MSH MEETING AGENDA ..................................................................................... 41 APPENDIX E. INTERVIEW PROTOCOLS AND INSTRUMENTS ............................................... 45 APPENDIX F. ONLINE USAID MISSION SURVEY ...................................................................... 47 APPENDIX G. PERSONS CONTACTED ..................................................................................... 49 APPENDIX H. REFERENCES ...................................................................................................... 57 TABLES AND FIGURES TABLE 1: NUMBERS OF KEY INFORMANTS AND INTERVIEWS IN EACH STAKEHOLDER CATEGORY ......................................................................................................... 4 FIGURE 1: TREND IN NEW FAMILY PLANNING VISITS.............................................................. 8 FIGURE 2: LMS PROJECT COMMUNICATIONS NETWORK .................................................... 14 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT v EXECUTIVE SUMMARY The Leadership, Management and Sustainability (LMS) Project is designed to improve sustainable service delivery results in the areas of family planning and reproductive health (FP/RH), HIV/AIDS, infectious diseases, and maternal and child health (MCH) through programs in leadership, management, and organizational capacity development. It is implemented by Management Sciences for Health (MSH) and is supported by a five-year leader with associates (LWA) cooperative agreement from the U.S. Agency for International Development (USAID) Office of Population and Reproductive Health (PRH) ending in 2010. LMS builds on proven management and leadership strategies developed by the previous Management and Leadership (M&L) Project, also implemented by MSH. Activities include leadership and management training programs as well as technical assistance activities at all levels of the health system. In 2008, PRH requested the Global Health Technical Assistance Project to conduct an external evaluation of the LMS Project. A two-member evaluation team was asked to address the following four questions: 1. What have been the greatest FP/RH successes resulting from LMS? 2. How successful were LMS management systems in carrying out the various programs? 3. What were the major challenges faced by LMS and the lessons learned? 4. What are the future strategic directions for leadership and management in health and family planning? The evaluation team met in Washington D.C. for a two-day planning meeting and spent three and a half days in Boston for a briefing meeting with the senior staff of MSH. Following several interviews with key LMS stakeholders, the team visited three case-study countries: Nicaragua, Nigeria, and Ghana. These countries were selected by USAID to highlight the variety of funding mechanisms, intervention strategies, and program management structures used by the LMS Project. Three sources of information were used during this evaluation study: documents, key informant interviews, and an online USAID Mission survey. Seventy-eight separate interviews were conducted with 194 individuals. The online survey had a low response rate of 8%. Survey responses, along with the key documents, were used to explain or enhance the data received from the interviews. The evaluation team found LMS, which operates in 23 different countries and regions, to be a very complex and comprehensive program. This document highlights some of the successes, challenges, and lessons learned from its first three and a half years. It concludes with a list of recommendations for future investments in health leadership and management development. PROJECT SUCCESSES The LMS Project has been successful in several ways and has documented successes in FP/RH in many countries. It has also contributed to the success of other health programs such as MCH and HIV/AIDS. In addition, MSH and the LMS Project have developed numerous tools and methods for strengthening leadership, management, and governance that have proven effective in diverse settings and at every level of health systems. vi EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT There have been numerous examples of LMS activities with significant FP/RH impact. For example, teams enrolled in a Leadership Development Program (LDP) conducted in collaboration with EngenderHealth’s ACQUIRE Project, in Kigoma Province, Tanzania, increased the number of new FP clients by 20% to 80%. In 2003, three district teams in Egypt increased the number of new FP visits by 36%, 68%, and 20% respectively. Local health professionals then used their own resources to expand the program to all 185 health facilities in the Aswan Governorate. In Ghana, a six-month LDP was conducted in the Central Region, led by the Adventist Development and Relief Agency (ADRA), an LMS partner. A team of health officials in this program conducted FP talks in mosques and churches, trained 80 volunteers and 60 traditional birth attendants in at least 40 communities, and provided condoms for distribution. From January to May, 2008, FP coverage increased from 13.8% to 18.5%. Given the success of this program, the Ghana Health Service (GHS) has found funds to scale up the LDP to the other nine regions of the country beginning in 2009. LMS has also provided considerable face-to-face technical assistance to FP organizations and programs in Bolivia, Guatemala, Nicaragua, and many other countries. This assistance has included guidance in the areas of governance development, board development, strategic planning, business planning, and the strengthening of financial sustainability, organization of human resources systems, and compensation schemes. In addition to successes in FP/RH, LMS has had many achievements in other areas of health￾service delivery. One example is the work in Nigeria to build management capacity in local civil society organizations (CSOs), nongovernmental organizations (NGOs), and treatment facilities working in the area of HIV/AIDS. Another example is the LMS Project in Nicaragua, PRONICASS. This has provided technical support to three social sector ministries—Health, Education, and Family Welfare—and other institutions in both the public and private sectors. LMS has also had great success in expanding the reach of materials and training programs through virtual networking platforms. The web-based Global Exchange Network (GEN) for Reproductive Health allows FP/RH leaders around the world to maintain contact and exchange technical knowledge. This network currently has a membership of nearly 2,000. LeaderNet, another virtual community of health professionals, has also increased its membership and programming for health managers. LeaderNet has sponsored extended training programs, short￾term seminars, and virtual conferences. Over the course of LMS, hundreds of health professionals from around the world have participated in the Virtual Leadership Development Program (VLDP), the Virtual Strategic Planning Program (VSPP), Virtual Business Planning for Health (VPBH), and various other LMS Internet-based learning programs. The LMS Project provides considerable expertise in method and process, achieving success at every level of health systems. It is the effectiveness and universality of its approach, not its expertise in specialty content (such as FP, MCH, or HIV/AIDS), that make its services unique and valued. Under the earlier USAID-sponsored project, MSH developed, tested, and refined an array of methods and tools for strengthening leadership and management capacity. These methods and tools have now been successfully applied in many countries in a broad array of health sector applications. Since the launching of the LMS Project, 16 new tools and new categories have been added to the Health Managers Toolkit. LMS MANAGEMENT SYSTEMS The LMS Project is large and complex. To back up and manage its programmatic efforts, MSH has invested considerable effort in developing systems for managing communications, finances, monitoring and evaluation, partnerships, and human resources. With only a few exceptions, those who have worked with the LMS team have found these systems to be remarkably responsive and flexible. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT vii The systems needed to maintain communications between and among the complex array of LMS Project stakeholders are remarkably comprehensive and effective, yet imperfect. If there is a failure of MSH’s communication system, it is a product of its own comprehensiveness. Some USAID officers sometimes feel overwhelmed by the sea of information that is available and find it difficult to locate the most salient points. Over this last year USAID Washington and MSH have worked together to greatly improve communications. From all evidence available to the evaluation team, it appears that MSH has well-developed financial management systems—including systems for planning, programmatic activities, budgeting, monitoring performance, and reporting. Despite these systems, financial issues do arise. One issue is that the core burn rate has been lower than planned. Another has been the delays in processing requests through the Office of Acquisitions and Assistance. Challenges with planning the startup of new LMS field offices and the complexities of USAID systems and contracting requirements contribute to the problems. Progress has been made in addressing both of these issues. LMS’ Monitoring, Evaluation, and Communications team is staffed by a cadre of monitoring and evaluation (M&E) experts. In the field, LMS routinely monitors the progress and results of its various program offerings. While useful, these short-term evaluation efforts are internally driven and miss longer-term successes and failures. A timeline which would allow for long-term evaluations is needed. The LMS Project has had mixed success working with partner organizations. At the field level, partnerships appear to be strong. Strategic partners at the international level have not been as successful. In its initial program proposal, MSH anticipated working with three major partners, RF Binder, Eastern and Southern African Management Institute (ESAMI), and ADRA. Lessons to be learned from LMS’ work with partners are the need to be very explicit up front about expectations and to understand the differences in corporate cultures. LMS has had remarkable success in staffing up to meet the rapidly growing demands for its services. LMS has built up its staff to 46 at MSH headquarters in Cambridge and has established offices in 11 developing countries. At present there are 450 people employed under the LMS banner worldwide, plus additional personnel employed by partner organizations such as ADRA. LMS’s largest office is in Afghanistan, with 170 staff on board. CHALLENGES The LMS program successes have occurred in spite of many challenges. These include an inexhaustible need for services, capacity limits of LMS given the rapid expansion of field support, difficulties of measuring and evaluating development, and difficulties in assuring lasting improvements. A major challenge faced by many struggling health systems around the world is the huge need for management and leadership training of health professionals. One reason for this is that talented managers are difficult to recruit and maintain. Another reason is that effective health management is not seen as a strategy for meeting health outcome goals. Given these realities, there has been an increase in the demand for LMS services. In some countries the demand for services is beginning to outpace the capacity of LMS to respond. LMS’ ability to effectively respond to the increasing demand for services needs to be addressed. One of the biggest challenges faced by LMS is the inherent difficulty in measuring and evaluating the impact of leadership and management interventions. It is difficult to isolate the effect of a single intervention from other contributing historical, contextual, or programmatic events. Evaluation is also made difficult by short timelines and the challenge of finding appropriate metrics and indicators of success and sustainability. At this point, the LMS indicators of viii EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT sustainability need to be validated by carrying out evaluations of long-term outcomes using experimental designs. LESSONS LEARNED In its self-assessment and other reports, MSH has listed many lessons learned.1 The in-country visits made by the evaluation team highlighted others, such as the value of working at the most stable level of a system, the value of the associate award partner model, and the limits of the use of virtual programs. Given the instability of some countries and some national governments’ lack of interest in developing a health management workforce, it is not always possible to work at all levels of the health system. While LMS in-country programs attempt to work at all system levels, they have learned to put emphasis on the most stable parts of the system. This allows them to have the greatest chance of success. Currently, the associate award is being used as part of the LMS management model in two countries, Nigeria and Afghanistan. This study found some potential value in exploring this model for future initiatives. The model has a number of perceived benefits. In-country staff feel that it allows for greater emphasis on local needs and context and for more access to USAID decisionmakers, as well as providing a simpler, more streamlined approval process. The use of virtual platforms has expanded during the last four years of LMS. Many have found the offerings very helpful and of high quality. Increased availability of computers and Internet service suggests that virtual programs will allow for even greater connectivity in the future. Despite their potential, the use of virtual programs is currently constrained by the limited availability of computers, low computer literacy, and the uneven access to broadband in many developing countries. LMS has successfully balanced these limitations by using virtual programs and systems as one of many strategies, and by blending them with other complimentary strategies, such as face-to-face team coaching. FUTURE STRATEGIC DIRECTIONS Future investments in leadership and management capacity building will come on line in a new era of U.S. development policy that is likely to emphasize longer-term strategies with greater weight given to building in-country capacities and self-reliance. Programs that strengthen leadership, management, and governance are certainly needed for such a longer-term approach. The PRH office, with its rich experience managing a history of leadership and management programs, can help lead the way. Management Sciences for Health, under the LMS Project and its predecessor projects, has developed, tested, and refined methods and tools for strengthening leadership and management. These efforts are increasingly seen as essential for the successful implementation of a broad array of health service delivery programs. The need to strengthen leadership and management is great, and future efforts should place even greater emphasis on sustainability, lifelong learning, and the further development of the rich library of training tools and methods. In addition, USAID should build a broad constituency that supports efforts to strengthen leadership and management development. To do so, it should find better ways to measure, document, and communicate the impact of leadership and management programs. In addition, it must find ways that legitimize and facilitate the application of core funds for leadership and management development from a wider range of health service programs. 1 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.79. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT ix Emphasize Sustainability USAID is entering a new era in international development that requires a long-term view. Increasingly, USAID will need to pursue long-term health systems strengthening strategies while it continues to address the pressing immediacy and demands of vertical health programs. The overarching aim should be to help countries achieve independence, and to leave a legacy of countries who are able to plan, lead, manage, finance, and deliver basic health services on a sustained basis. Strengthening leadership and management is a central, possibly the only, means to these important ends. Support Lifelong Learning Future initiatives should provide more support to programs designed to encourage lifelong learning. The virtual LeaderNet system is one avenue for continued knowledge exchange. Other efforts such as building the pre-service and continuing education programs of health-related university programs should be expanded. More attention should also be given to developing in￾country consulting talent so training and technical assistance continues to be available to health practitioners beyond LMS. Update and Maintain the Library of Tools and Methods Future programs need to support the updating and maintenance of the MSH library of tools and programs. The library and toolkit are rich resources for training future leaders and managers and for supporting their need for lifelong learning. Maintaining, refining, and updating this collection of materials is a challenge, but it should be a priority for the future. Build a Constituency for Leadership and Management USAID should build support for leadership and management programming based on principles of skills-building and applied learning. Unfortunately, leadership and management training has not had the political support provided to many vertical programs. The long-term success and sustainability of each of the vertical programs, however, is highly dependent on how well they are implemented and managed in the field—and on the ability of the in-country health system to assume the ongoing responsibility for organizing, managing, and financing the effort. USAID should develop a strategy for building a greater political constituency for leadership and management training. Find Better Ways to Measure Success of Leadership and Management Efforts to measure the impact of LMS’ interventions tend to focus on short-term results while true measures of success are more long-term and tenuous. USAID, with MSH’s assistance, needs to find better ways to measure the results of technical assistance in leadership, management, and governance and to demonstrate successes in persuasive terms. USAID should encourage more operational research designed to better measure programmatic results. The findings of such studies should be published and subject to peer review. Legitimize Broad Program Application While core funds and management oversight are centered in PRH, and there are expectations that the LMS Project will demonstrate its impact on FP/RH parameters, its true impact is much broader. Contracting options which legitimize and promote broader support of leadership and management programming at the USAID, Washington D.C. office should be developed. Three options are possible: continue to contract for this type of work under PRH, develop a global contract, or add leadership and management to each service program contract. Under any option, USAID’s important work should continue to strengthen leadership and management capacities to improve health-care services. x EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 1 I. INTRODUCTION The Leadership, Management and Sustainability Project (LMS) is designed to improve sustainable service delivery results in the areas of family planning and reproductive health (FP/RH), HIV/AIDS, infectious disease, and maternal and child health (MCH) through programs in leadership, management, and organizational capacity development. This program was founded on the belief that management and leadership are essential to well-performing health organizations. Even with technically competent staff, adequate equipment and supplies, and evidence-based health programs, quality performance is not assured without skilled health leaders and effective management systems. The LMS Project is implemented by Management Sciences for Health (MSH) and is supported by a five-year leader with associates (LWA) cooperative agreement from USAID’s Office of Population and Reproductive Health (PRH) which ends in 2010. Currently, LMS is being offered in 23 distinct countries or regions around the world. This project was designed to build upon proven management and leadership strategies developed by the previous Management and Leadership (M&L) Project, also implemented by MSH. LMS activities include leadership and management training programs as well as technical assistance activities at all health system levels. Its training programs use the Challenge Model, which focuses on six main techniques to improve leadership and management: scan, plan, focus, align/mobilize, organize, and implement. This is coupled with strong monitoring and evaluation (M&E), enabling participants to face challenges and achieve measurable results. LMS provides virtual leadership development programs held online, face-to-face leadership training, and other innovating training methods such as online forums and virtual conferences. Technical assistance is provided to build organizational and community health systems capacity. Examples include business-planning development, work-climate assessment and financial-systems improvement. The objectives and intermediate results (IRs) of LMS programs are to: 1. Improve management and leadership of priority health programs (IR1) a. LMS will support and equip a critical mass of managers who lead at all levels throughout the health system to advocate for and implement inspired leadership and sound management. 2. Improve management systems in health organizations and priority health programs (IR2) a. LMS will transfer approaches and skills to organizations to ensure that management structures and systems contribute to sustainable organization success. 3. Increase sustainability and ability to manage change (IR3) a. LMS will enable organization and individuals to lead and manage concerted responses to complex health challenges at all levels in NGOs and the public sector, multisectoral bodies, national governments, and international agencies. In 2008, PRH requested the Global Health Technical Assistance Project to conduct an external evaluation of LMS to assess the process and outcomes of its programs in strengthening leadership and management skills among providers in developing countries worldwide (See Appendix A, Evaluation Scope of Work). Specifically, the evaluation study was to address the following four questions: 1. What have been the greatest FP/RH successes as a result of the LMS project? 2 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 2. How successful were LMS management systems in carrying out the various programs? 3. What were major challenges faced by LMS and lessons learned? 4. What are future strategic directions for leadership and management in health and family planning? This document will serve as the final report of the evaluation study conducted by a two-member team during February and March 2009. A summary of the findings and a list of considerations for future investments in areas of leadership and management-capacity building follow a description of the evaluation methodology. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 3 II. METHODOLOGY This study focused on understanding the long-term value of investments made at strengthening multiple, aligned levels of a health system in the developing world. The evaluation team met in Washington D.C. for a two-day planning meeting (Appendix C) and spent three and a half days in Boston for a briefing meeting with the senior staff of MSH. Following several interviews with key LMS stakeholders, the team visited three case-study countries: Nicaragua, Nigeria, and Ghana. These countries were selected by USAID because their LMS programs differed in funding mechanisms, intervention strategies, and program management. SOURCES OF DATA Three sources of information were used during this evaluation study: key documents and reports, key informant interviews, and an online USAID Mission survey. The first source of information was documents and reports that MSH has created to describe and monitor the progress of the LMS Project. These include: Leadership, Management and Sustainability Program: Self Assessment Report (2009), Lessons Learned in Mainstreaming and Scale-Up of Leadership and Management (L&M) Approaches, The Role of Leadership and Management in Strengthening Good Governance, and Leadership Can Be Learned, But How is it Measured? Country-specific program information was also obtained and reviewed including news-note highlights, partner profile information, and specific program reports. The second source of information was obtained through interviews of stakeholders representing six major respondent groups, categorized by the relationship they have to LMS. The first three groups were respondents who work at a global level, including employees of USAID, MSH, and other partner organizations. The remaining groups are stakeholders of the three in-country programs providing the case studies: Nicaragua, Nigeria, and Ghana. In-country stakeholder groups include employees from the USAID Missions in these countries, the MSH/LMS staff members, lead faculty or technical assistance consultants, local partners, and client recipients. A total of 78 interviews were conducted over a two-month period. Table 1 lists the number of key informants and interviews from each of the stakeholder categories. Interviews were led by one or both members of the evaluation team. Interview sessions were typically 1–1.5 hours in length and were conducted in a private setting, most often in a private office or over the telephone. In some instances, MSH staff, USAID employees, or other program partners were present during the interviews. Interviews followed a semi-structured format but allowed for relevant but unplanned discussions. Appendix E contains a copy of the interview tool used to guide these discussions. Appendix G is a list of all persons who took part in the interviews. 4 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT TABLE 1: NUMBERS OF KEY INFORMANTS AND INTERVIEWS IN EACH STAKEHOLDER CATEGORY STAKEHOLDER CATEGORY Number of People Interviewed Total Number of Interviews USAID – Washington DC 14 9 MSH Headquarters – Senior Program Staff 22 10 Strategic Partners and Stakeholders, e.g., ADRA, Family Health International, World Health Organization (WHO) 10 10 Nicaragua 55 19 Nigeria 61 20 Ghana 33 11 TOTAL 195 78 The third source of information is an online survey sent to 64 individuals working for or in partnership with USAID Missions in countries that provide some level of LMS programming and service. The audience for this survey was selected by the Office of Population and Reproductive Health and distribution was via email through the GH Tech office in Washington D.C. The survey tool contained eight questions which were expected to be answered in no more than 15 minutes. Three questions were quantitative in nature, while the rest asked for a brief written response. This survey tool was approved by the USAID Cognizant Technical Officer (CTO) of LMS and was initially distributed on February 19, 2009. Appendix F includes a copy of the survey instrument. Fifteen individuals were removed from the survey distribution: nine who had already been interviewed and six who notified the team that they were not working with the LMS Project. Three reminder notes were sent to those who had not responded to the original request. Four surveys were received by March 27, 2009. The final response rate was 8%. METHODS OF QUALITATIVE ANALYSIS The results of the key informant interviews were analyzed in terms of the four key evaluation questions and with an overarching conceptual model which includes five interconnected system levels and a lifelong learning perspective. A thematic coding scheme was developed for the purpose of analysis. Six separate categories with 28 sub-categories were identified for the sorting of comments made by the key informants. A copy of the scheme can be found in Appendix E. Notes from each interview were completed within 24 hours of the discussion by one or both evaluators. Comments or quotes from each interview were sorted into the relevant thematic categories. A separate summary sheet was created for each interview. Notes taken from multiple discussions with essential individuals, such as in-country MSH staff members, were merged and used to create a single interview summary for each individual informant. All summary sheets were used to identify emergent themes in each category. Information obtained from the document reviews and data obtained from the online survey were used to further understand or to expand on the data collected by the key informant interviews. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 5 ATTRIBUTION This evaluation was not designed to evaluate a particular country program or to clearly identify which particular program elements led to the current outcomes achieved. Instead, this evaluation uses key stakeholder feedback to identify best practices in a developing world context, the ongoing gaps and needs, and the potential directions for future investments by PRH. The evaluators asked interview respondents to reflect and share their perceptions of the value of the LMS Project at individual, team, organization, health system, and health outcome levels. While each person’s perspective is important, it is important to note the inherent difficulty in showing a direct link between a leadership or management initiative and an observable outcome. Skill levels at the time of intervention as well as other contextual factors—such as political environment, other key players in the health system, or the current organizational structure—can work as barriers or facilitators to change. A higher level of scientific evidence and a more structured evaluation design would be needed to demonstrate such direct links. STUDY LIMITATIONS It is important to note a few limitations of the study design which may have an impact on how the results were collected or analyzed. The findings of this evaluation should be viewed with these limitations in mind. Key informant interviews were the primary data for this study. Reported changes in behavior or competencies were reported by LMS staff or by client recipients but, given the limits of the study design, could not be verified. Although triangulation of data sources often allows for a validation of results, the ability to obtain other sources of data was limited due to a short timeline. Many interviews conducted in the field included the presence of at least one or more outside observers. In Nicaragua, observers included the USAID PRH CTO, members of the PRONICASS (LMS office in Managua) staff, and other USAID Mission staff. In Nigeria interviews included the USAID LMS Technical Advisor and at times, one or more members of the MSH/LMS office. In Ghana, all interviews included the Director of ADRA, the lead agency for the LMS LDP program in that country. It is possible that interview respondents alter their answers to questions when either the funder or a key program staff member is present in the room. However, the evaluation team did not feel that the presence of additional observers significantly influenced the quality of the interviews. Another factor that may have affected the quality of the interview data was language barriers. All interviews were conducted in English. In Nicaragua, a Spanish interpreter was hired to translate the conversation. In Nigeria and Ghana, no translation was needed, but slight language differences were apparent. As a result, the interview notes taken and analyzed by the evaluators may have not completely captured the full intent or meaning as offered by the key informants. Every effort was made to assure clarity of meaning during the interviews. It is the intent of this study to identify some of LMS’s best practices and challenges. However, the ability to generalize the findings of this study to all LMS in-country programs should be done with caution. Only three of the 23 LMS in-country programs were visited and reviewed. While specific criteria were used to select these sites, two of the three were in located in Western Africa and the third was in Central America. Since contextual factors play such a significant role in the change process, the results found in these countries may vary greatly from those in other countries. 6 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 7 III. PROGRAM SUCCESSES Among those who are familiar with its work, the Leadership, Management, and Sustainability Project (LMS) has been viewed as successful in several ways. Specifically, it has documented successes in FP/RH in a number of countries. It has also contributed to the success of other health programs such as MCH and HIV/AIDS. In addition, MSH and the LMS Project have developed numerous tools and methods for strengthening leadership and management that have proven effective in diverse settings and at every level of the health system. SUCCESSES IN FAMILY PLANNING AND REPRODUCTIVE HEALTH The LMS Project has applied core USAID funding to address FP/RH issues in three major ways: Through its Mainstreaming Team, by ―transferring field-tested products, approaches, and best practices in leadership and management development to field programs through partnerships with FP/RH service delivery organizations and programs.‖2 Through its Scale-Up Team, by focusing ―primarily on the use of electronic technology to build the capacity of, and transfer knowledge to, health professionals in leadership and management development at a large scale.‖3 Through its Global Leadership Team, by increasing awareness among thought leaders and decisionmakers in the international health community of the link between investments in leadership and management and improved health systems, service delivery and health outcomes, especially in family planning and reproductive health. The MSH paper, ―Linkages between Leadership and Management Strengthening and Service Delivery Improvement and Results,‖ documents a number of case studies of LMS’ success in FP/RH.4 Three examples of programs with significant FP/RH impact, as noted in the LMS self￾assessment—plus a description of the successful Global Exchange Network for Reproductive Health—are summarized below: 1. A six-month Leadership Development Program (LDP) was conducted in collaboration with the EngenderHealth ACQUIRE Project in Kigoma Province, Tanzania, involving teams from six health facilities and three districts, each of which selected ―increasing the number of new family planning (FP) clients‖ as their challenge. All nine facilities increased the number of new FP clients by 20% to 80%. In a follow-on exercise conducted independently by the Kigoma District Council, four out of the five dispensaries involved increased new FP clients by 33% to 53%.5 2. In Egypt, in 2002 under the M&L Project, 10 teams participated in an LDP. By 2003 the districts of Aswan, Daraw, and Kom Ombo had increased the number of new FP visits by 36%, 68%, and 20% respectively. The number of prenatal and postpartum visits also increased. Of special note, after USAID funding ended in 2003, local doctors and nurses, using their own resources, extended the program to all 185 health facilities in the Aswan 2 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.9. 3 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.12. 4 Leadership, Management and Sustainability Program, ―Linkages between Leadership and Management Strengthening and Service Delivery Improvement and Results,‖ Management Sciences for Health, Cambridge, MA, January 2009. 5 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.10. 8 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Governorate. From 2006 to 2007, maternal mortality in Aswan was reduced from 50.0 to 35.5 per 100,000 live births.6 3. At the behest of Ghana Health Services (GHS, an autonomous agency under the Ministry of Health), a six-month LDP was conducted in the Central Region, lead by ADRA/Ghana, with coaching support provided by LMS Cambridge. Seven teams were organized, including six district teams and one team representing the Central Regional Health Directorate. Although FP is a GHS priority, reducing maternal mortality was foremost on the agendas of the district teams, with most achieving success in increasing the numbers of pregnant women seeking prenatal care and those delivering under supervision. The Regional Directorate, however, decided to work on improving FP coverage in an underperforming district. At the end of six months the Directorate had conducted talks on FP in mosques and churches, trained 80 volunteers and 60 traditional birth attendants in at least 40 communities, and provided condoms for distribution. The results were quite notable, as shown in the following illustration. From January to May, 2008, FP coverage increased from 13.8% to 18.5%. The GHS feels the LDP was so successful they plan to scale up the program to the other nine regions of the country beginning in late March, 2009.7 Figure 1: Trend in New Family Planning Visits Source: MSH/ADRA, “Ghana LDP Results from Participating Teams in the Central Region of Ghana,” Accra, Ghana, January-July, 2008. 4. The web-based Global Exchange Network (GEN) for Reproductive Health was developed during the M&L Project to help maintain contact and to encourage the exchange of technical information between USAID and FP/RH programs and organizations in the six countries that had, at that time, graduated from USAID population assistance: Mexico, Brazil, Ecuador, Colombia, Turkey, and Morocco. Since then other countries have graduated and today the network has a membership of nearly 2,000 FP professionals. In April 2008, a worldwide virtual forum was held on the GEN website, ―Effective Programming for Long-Term and Permanent Methods: A Forum for Family Planning Program Managers and Policy Makers.‖ More than 90 people from 34 countries participated. Since June of 2006, eleven GEN- 6 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, pp. 11-12. 7 Ibid, pp. 10-11. Before and After Leadership Development Program Ajumako-Enyan-Essiam District, Central Region, Ghana 0 200 400 600 800 1000 1200 1400 1600 New FP Visits Jan Feb March April May 2007 2008 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 9 sponsored events have been held, involving 1,545 participants. Two additional seminars are being planned for 2009.8 SUCCESSES IN OTHER HEALTH SERVICE PROGRAMS In addition to successes in FP/RH, the MSH paper, ―Linkages between Leadership and Management Strengthening and Service Delivery Improvement and Results,‖ documents case studies of success in other areas of health service delivery in several representative countries: Nepal, Ghana, Nicaragua, Peru, Tanzania, Bolivia, Afghanistan, and Nigeria.9 The Nigerian case studies, which include two components and which the evaluation team was able to observe first hand, are especially illustrative. Also instructive is LMS’ work in Nicaragua where it has provided support to an array of organizations in both the public and private sectors. Finally, the web-based LeaderNet provides an increasingly popular forum for health professionals. 1. In Nigeria, the LMS Project is engaged in two major efforts: the AIDS Care and Treatment (LMS-ACT) Project and the Capacity Building Project. LMS-ACT is an associate award funded by the President’s Emergency Plan for AIDS Relief (PEPFAR) under the overall LMS leader award. The project’s strategic objective is to rapidly scale up and increase utilization of quality comprehensive AIDS prevention, care, and treatment services in Nigeria. Launched in July 2007, LMS-ACT, like all PEPFAR projects, places a strong emphasis on tracking results. Key statistics include 76,252 people tested for HIV/AIDS, with 1,399 newly initiated into antiretroviral therapy, of which 52 were children and 47 were pregnant women. More than 91,000 tests were performed for HIV, tuberculosis, and syphilis. Services were provided to 937 orphans and vulnerable children. Unlike other PEPFAR programs, LMS-ACT also strongly emphasized improving the quality and quantity of services through strengthening the management systems in all the health service facilities in all six of the states involved. Its intent is to build the capacity of these facilities to track and manage their patients more successfully and to continue to provide services after the LMS-ACT program terminates. 2. The Capacity Building Project in Nigeria is working to build the capacity of civil society organizations (CSOs) and NGOs to expand the delivery of HIV/AIDS services. Starting in 2006, when the USAID Mission in Nigeria began offering PEPFAR-funded grants to local CSOs and NGOs, it found that several applicants were qualified to provide HIV/AIDS services but were not qualified to administer a USAID grant. LMS was asked to work with up to 12 new CSO implementing partners in the first year and to continue support to existing and new partners in year two. LMS provided help to build capacity in project management, planning, financial management, organization, and quality assurance. It also provided leadership and management training to CSO managers and officials in selected government agencies. To date, 16 CSOs have successfully secured USAID grants. Some CSOs have parlayed their learning to obtain grants from other funders, such as a recent grant from the Centers for Disease Control and Prevention to PROHEALTH. Notably, as of early March, USAID has received 40 additional grant applications, a number of which will likely require Capacity Building support from LMS. 3. In Nicaragua, PRONICASS, the LMS office in Managua, has provided technical support to three social sector ministries—Health, Education, and Family Welfare—and other institutions in both the public and private sectors. LMS helped strengthen the management systems and organizational structure of PROFAMILIA, which runs several policlinics. It also helped 8 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, pp. 15-16. 9 Leadership, Management and Sustainability Program, ―Linkages between Leadership and Management Strengthening and Service Delivery Improvement and Results,‖ Management Sciences for Health, Cambridge, MA, January 2009. 10 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT NicaSalud, a network of health providers, to become certified to receive USAID funds and will soon launch an LDP involving teams from eight member organizations. PRONICASS worked with the Ministry of Health to help realign its goals and to modernize its management systems. And recently, PRONICASS has been working with the Ministry of Education, which has decided to prescribe 70% of the educational curriculum as common to all communities but to allow 30% of the curriculum to be designed locally. PRONICASS agreed to assist the Ministry of Education in a pilot project, working with local communities in selected regions of the country. Leon is an area where PRONICASS has come the closest to integrating health and education programming. It starts by helping a community identify its local needs and resources and to design its own municipal development plan. Each community is unique and priorities range widely from the quantity and quality of water, citizen security, family violence, and food security. All of these areas have both an educational and health dimension. So far, 43 municipalities have been involved, including 125–150 schools. In Leon, 12 of 16 schools identified adolescent pregnancy as one of their priorities. This is an issue that has both health and education components that must be worked in synergy. 4. LMS core funding also supports LeaderNet, a virtual community of health professionals, managers, facilitators, and technical experts interested in improving leadership and management. With 424 members in 2005, LeaderNet now has over 2,000 members from over 40 countries. LeaderNet provides a platform for continuous learning, ongoing support, and peer exchange for health managers and their teams working in FP/RH, HIV/AIDS, tuberculosis, malaria, and maternal and child health.10 This virtual system is widely valued by strategic LMS stakeholders. BROADLY APPLICABLE TRAINING METHODS AND TOOLS The LMS Project brings to its role considerable expertise in method and process. It is the effectiveness and universality of its approach, not its expertise in specialty health content (such as FP, MCH, or HIV/AIDS), that make its services unique and valued. Under the earlier USAID￾sponsored M&L Project, MSH developed, tested, and refined an array of methods and tools for strengthening leadership and management and building institutional capacity that were based on concepts of personal skills development, team work, and applied learning. These methods differ significantly in both approach and proven success to traditional lecture-based teaching methods. One LDP participant in Ghana, who later became a facilitator, put it this way: Those who are lecture-based have a syllabus to follow. They think that you are wasting time if you are not in lecture. But I think people learn better on their own once they know how the information is useful. I start with myself. As I use the learnings, they become part of me. And then it is easier to continue to use the skills I have learned. Under the LMS Project, these methods and tools have been successfully applied in many countries in a broad array of health sector applications. Increasingly, the MSH approach is seen as essential for the success and sustainability of health service delivery programs of all sorts. The approach would probably have similar success if applied to other non-health programs. There are currently 61 field-tested and well-documented tools on the MSH Toolkit addressing all aspects of leadership and management in health service delivery, including FP/RH. Since the launching of the LMS Project, 16 new tools and new categories have been added. The teaching aids in the MSH’s toolkit and resource library range from simple one-page worksheets, to workbooks that address special needs, to comprehensive programs and manuals. Many tools are 10 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.17. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 11 available in hard copy or on-line, and several are designed to be used in a virtual format where access to broadband is available. In 2008, the MSH ―Toolkit‖ website received 92,876 unique visitors.11 A few tools are worth special mention: 1. The Leadership Development Program (LDP), using the Managers Who Lead handbook and the Challenge Model as guides, has proven to be particularly popular with USAID Missions because of its broad utility and near universal success in face-to-face applications. Typically, a LDP is initiated with a session to orient key facilitators, followed by a sensitization session for senior managers to assure their support. Then six-to-ten work teams are identified, each team composed of four-to-six individuals who have related responsibilities, often from within the same organization. Each team identifies a challenge or project that it wants to tackle. The participants attend four workshops. Each workshop has a different focus and builds upon and reinforces what was learned in the previous workshop with lessons applied directly to the team’s challenge. The sessions are interactive and learner￾centered, with short presentations, discussions, group work, role playing, and problem solving. By the end of the LDP, each team is expected to demonstrate results. 2. The Management and Organizational Sustainability Tool (MOST) is a participatory diagnostic process that enables managers to develop a profile of their organization’s capacity and an action plan for improvement. Developed in 1998 with USAID core funds, and updated and refined under M&L, the tool is designed to be used without MSH involvement. It continues to be successfully used—most recently in Nigeria, Tanzania, and Uganda.12 3. The Virtual Leadership Development Program (VLDP) is gaining in popularity for multiple country applications and shows potential as an economical means of reaching larger audiences. From 2005 through 2008, the VLDP and its sister programs VSPP, VBPH, and Virtual Human Resources Management have been offered twenty times, in five languages, with a total of over 1,500 participants from 35 countries.13 DEVELOPING LEADERSHIP AND MANAGEMENT AT EVERY LEVEL Because the LMS approach to capacity building relies so heavily on process, and draws from the innate energy and resources of program participants, it has had consistent success at every level and strata of the health system where it has been applied. LMS programs have been successfully used with senior level leaders, mid-level managers within the government bureaucracy, medical and nursing professionals working in hospitals and clinics, and staff working in private clinics and NGOs. In Nigeria, for example, LMS is using an approach that strengthens the health system at four levels: individual (Nursing Fellows Program), team (Federal Ministry of Women’s Affairs), organizational (capacity building with NGOs), and public policy (ACT involvement of state/local governments). Strengthening all levels of the health system is an important aspect of the LMS approach in developing countries. 11 Leadership, Management and Sustainability Program, ―Self-Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p. 19. 12 Leadership, Management and Sustainability Program, ―Self-Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p. 80. 13 MSH records 12 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 13 IV. LMS MANAGEMENT SYSTEMS The LMS Project is large and complex. With a five-year time frame, an anticipated budget of nearly $120 million, and a multiplicity of activities in 23 countries, it is a challenge to manage for both USAID and MSH. Given its size and complexity, it is remarkable that LMS has earned a reputation for responsiveness and flexibility. To back up and manage its programmatic efforts, MSH has invested considerable effort to develop systems for managing communications, finances, monitoring and evaluation, partnerships, and human resources. RESPONSIVENESS AND FLEXIBILITY With only a few exceptions, those who have worked with the LMS team have found it to be remarkably responsive and flexible. Those who were interviewed by the evaluation team reported numerous incidents that confirmed LMS’ quick response to requests and inquiries and their flexibility in responding to changing needs and circumstances. One quote was echoed by several, ―LMS is our go-to organization. We know we can rely on them.‖ LMS’ reputation was confirmed in each of the three countries visited by the evaluation team. PRONICASS, the MSH office in Nicaragua, for example, has an in-country presence that dates back to before the launch of the LMS Project and over the years has built a reputation as a trusted, reliable, skilled, and extremely flexible resource. When its promising efforts to help build the capacity of higher-level offices in the Ministry of Health were compromised after a change in government, PRONICASS shifted its emphasis to assisting the more receptive Ministry of Education, piloting an effort to engage local communities in designing municipal development plans, including plans for education and health. In Nigeria, in only two years, LMS has developed a strong reputation for responsiveness, quickly gearing up to address the dual challenges of a USAID leader award designed to build the capacity of local NGOs who have applied to the USAID Mission for PEPFAR funds, and an associate award (the LMS-ACT Project) designed to strengthen the capacity of the public sector to deliver HIV/AIDS and tuberculosis services. In the process, LMS has created a cadre of enthusiastic supporters among the staff of the Nigerian USAID Mission and among government officials at both the national and state levels. One USAID staff member said, It’s been great to work with LMS. The home office team is great, very responsive. The in￾country team is fantastic, extremely productive. And they have a strong capacity to integrate outside consultants – it’s seamless. In Ghana, visiting LMS staff from the MSH Cambridge office have been highly praised, and LMS’ local implementing partner, ADRA, has built a respected working relationship with the USAID Mission, Ghana Health Services (GHS, a department of the Ministry of Health), and the directors of several local district governments. Ironically, there is a dichotomy in LMS’ responsiveness and flexibility. What is seen as a virtue by field officers can become a vice at higher levels. Field officers say ―LMS is so quick to respond,‖ and ―They can do anything.‖ One Mission officer praised LMS because they were quick to change plans to accommodate local needs or unexpected calendar conflicts. But in Washington, officers voice frustration that LMS frequently makes changes in their plans, such as travel schedules, which are difficult to approve or monitor. A few officers from USAID questioned whether LMS might be too flexible and responsive, agreeing to all requests even when outside of the project’s scope. The fact that MSH has multiple initiatives within some countries may account for some of the confusion over project boundaries. 14 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Certainly LMS’ reputation is not flawless. However, among those few who cited exceptions where LMS had not been as quick to respond as expected, they noted that their expectations had been heightened by earlier experiences. They attributed the slower response to LMS’ need to rapidly gear up as their reputation and the demand for their assistance has accelerated, which has necessitated adding staff that were not as experienced as those they had worked with earlier. COMMUNICATIONS The systems needed to maintain communications throughout the complex array of LMS stakeholders are remarkably comprehensive and effective, yet imperfect. The following figure illustrates only a part of the multiple interrelationships, with MSH at the center. Figure 2: LMS Project Communications Network The above figure does not show other direct links between the various stakeholders. It merely illustrates the complexities of communicating on issues of accountability, program scheduling, monitoring, finance, and knowledge sharing. A critical communication link, of course, is between MSH Cambridge and USAID Washington. MSH, by contractual obligation and a desire to be transparent, regularly submits numerous reports, such as this small sampling: Semiannual and annual reports, which review program highlights, milestones, performance monitoring plans, and cost-share information Performance monitoring reports (every six months) Management review reports (periodic) Special reports, such as: – The Role of Leadership and Management in Strengthening Good Governance – Strengthening Management and Oversight of Global Fund Grants In addition, MSH submits a number of financial reports, prepares an internal monthly expenditure report that tracks spending across all LMS programs, and conducts an internal quarterly review to monitor progress in all LMS activities. Over this last year USAID/Washington and MSH have been working together to improve communications—introducing regular conference calls and video-conferences, supplemented by EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 15 frequent emails and telephone conversations as well as periodic meetings in Washington and Boston. FINANCIAL MANAGEMENT From all evidence available to the evaluation team, it appears that MSH has well-developed financial management systems, including systems for planning, programmatic activities, budgeting, performance monitoring, and reporting. In its self-assessment report, LMS states that: LMS has developed an integrated system for work planning that simultaneously pulls together: planning programmatic objectives with a focus on results; budgeting the necessary resources to accomplish program objectives; and defining the monitoring and evaluation (M&E) plan to measure progress and accomplishments. Workplans are developed with a level of detail that defines expected outcomes and outputs and fleshes out activities for all Core and Field Support funded work.14 Such systems are essential given the size and scope of LMS’ financial obligations. As noted in the LMS self-assessment report: The rapid and exponential growth of Field Support and Associate Awards from USAID Missions since 2005 is a strong indicator of the relevance and urgent need for the services provided by LMS. The demand for LMS in Missions to date equals $106,171,388 in FFY 2008. By comparison, the total Field expenditure of the predecessor M&L program, after five years, was $48,156,416.15 The information generated from these systems provides data for numerous financial reports. In addition to internal reports, such as the internal monthly expenditure report, LMS routinely submits a number of reports to USAID, including: SF269 – a standard financial activity report submitted to USAID quarterly, Quarterly baseline reports – summarizing obligations, expenditures, and projected spending for population and other activities, Mission pipeline – financial reports to requesting Missions, Semiannual reports – which include a summary of financial activity, Cost share reports – updating LMS status in achieving the required 10% cost share, Public/private partnership report – a report on new funds leveraged from NGOs, foundations, and other private-sector sources, Semiannual USAID management reviews – including responses to USAID questions on financial issues, and Geographic code waiver tracking – reports on consultant activity under a waiver. Still, financial issues do arise. For example, both USAID and MSH point out that the core burn rate has been lower than planned. MSH attributes the problem to the unexpected absence of key leadership staff due to medical reasons, and competing demands for startup support for two large field support projects.16 Once behind, LMS was challenged to catch up. Whatever the justification, USAID places significant of weight on the burn rate, and the numbers they received 14 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.61. 15 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.2. 16 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.68. 16 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT from LMS did not justify a higher advance. USAID needs to recognize how the complexities and inflexibility of its own systems and requirements, as noted in the LMS self-assessment report, contribute to some of the financial issues. LMS headquarters also needs to continue to strengthen its financial systems to better meet the needs of their funders.17 MONITORING AND EVALUATION LMS’ processes for M&E are based on systems developed under the M&L Project. LMS’ Monitoring, Evaluation, and Communications Team is staffed by a cadre of M&E experts, and an electronic library of M&E reference materials has been assembled. In addition, LMS has updated the MSH Menu of Indicators on Management and Leadership and has incorporated resource materials from other sources such as WHO and PEPFAR.18 In the field, LMS routinely monitors the progress and results of its various program offerings, often including a six-month post-program follow-up. While useful, these short-term evaluation efforts are internally driven and miss longer-term successes and failures. To overcome this shortcoming, a few independent and longer-term studies and evaluations have been undertaken. In Nepal, for example, an external evaluation of the Results-Oriented Leadership Development Program was conducted in 2007 at the request of the government. Other independent studies were conducted in Uganda and Nicaragua. In order to measure longer-term results, a study of a pending LDP is about to be launched in Kenya. The evaluation team recommends that more such studies be undertaken. PARTNERSHIPS AND RELATIONSHIPS The LMS Project has had mixed success working with partner organizations. At the field level, partnerships appear to be strong. Strategic partners at the international level have not been as successful. In its initial program proposal, MSH anticipated working with three major partners, RF Binder, ESAMI, and ADRA. These partnerships have produced some noteworthy successes. For example, RF Binder, a strategic communications organization, helped to produce the acclaimed ―Seeds of Success‖ video on the experiences in Aswan, Egypt. ADRA, a global humanitarian agency, has worked to expand the reach of the LMS flagship LDP program in countries such as Ghana and Nepal. Unfortunately, the full potential of these partnerships has not been fully realized. The following lessons can be learned from LMS’ work with partners: There is a need to be very explicit about expectations from the beginning; and Differences in corporate cultures must be identified and understood. HUMAN RESOURCES MANAGEMENT LMS has had remarkable success in staffing up to meet the rapidly growing demands for its services. LMS has built up its staff to 46 at MSH headquarters in Cambridge and has established offices in 11 developing countries. At present there are 450 people employed under the LMS banner worldwide, plus additional personnel employed by partner organizations such as ADRA. LMS’s largest office is in Afghanistan, with 170 staff on board. At times, the build-up of staff has put substantial pressure on resources. In Nigeria, for example, the LMS office has grown in two years from only four to over 90 people in order to meet the 17 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, pp.68-69. 18 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.71. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 17 demands of the Capacity Building and LMS-ACT programs. It has been in these periods of rapid build-up that LMS has been cited for slower-than-expected responsiveness. The large numbers of new staff pose a significant challenge of orientation and training. MSH and LMS have a distinct approach to their work. They have a large library of unique tools and methods of training that new employees must learn well enough to call their own. They have a comprehensive array of sophisticated management systems, such as programs used to monitor and track financial data that must be mastered before being taught to others. In addition, they have numerous associations and relationships that must be respected and supported. 18 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 19 V. CHALLENGES LMS successes have occurred in spite of many challenges. While a number of these have been highlighted in the MSH/LMS self-assessment report19 and confirmed by those interviewed for this study, there are four current challenges worthy of further description in this report. These include the inexhaustible need for services, the capacity limits of MSH, the difficulty of measuring and evaluating development, and the difficulties in assuring lasting improvements. EXTENSIVE NEED FOR TRAINING AND TECHNICAL ASSISTANCE There is a significant need for management and leadership training of health professionals in the developing world. One reason for this is that talented managers are difficult to recruit and maintain. In many countries there is a shortage of health workers. In others, changes in government lead to frequent changes in key health positions. These factors contribute to the ongoing need for the training and development of new managers. Another reason given for the tremendous need for management and leadership training is that effective health management is not seen as a strategy for meeting health outcome goals. As one international health expert said ―management in these countries is by default not by design.‖ It is commonly believed that a clinical health degree means that you have the skills to not only treat illnesses but also to manage and lead an organization. Unfortunately, this belief has led many countries to move doctors, nurses, and other health professionals into managerial roles without any additional training. Many of the health managers interviewed for this evaluation study confirmed this practice, including a physician in Ghana who is now running a governmental district hospital and a professor from a medical college in Nicaragua. The result of this practice is that the individuals placed in managerial roles are quickly frustrated and overwhelmed. The training of health managers is very much needed but it can only be sustained if the systems in which they work are strong and functional. In the developing world, many health organizations lack the basic managerial systems needed to support a qualified staff and an effective service portfolio. These organizations lack systems for purchasing and monitoring supplies, for obtaining and monitoring finances, for tracking and evaluating service and program outcomes, and for recruiting and maintaining qualified staff. As one expert in the field of reproductive health noted: ―We need to be able to keep a cadre of well trained staff for long term in an environment that helps them to succeed. Health professionals at all levels of the health system need to be developed and supported.‖ CAPACITY LIMITS OF LMS As mentioned previously, LMS teams are well respected both internationally and in the countries in which they provide services. The fact that they are seen as responsive and competent has increased the demand for their services. There has been continued demand for programs such as the LDP and VLDP, as well as capacity-building services such as organizational assessments, pre-service curriculum development, and business planning assistance. Even new countries— such as the Democratic Republic of Congo, Ethiopia, and Pakistan—are now establishing collaborative relationships with LMS. In some countries the demand for services is beginning to outpace the capacity of LMS to respond. The comments of one USAID Mission officer capture the opinions expressed by several people interviewed: ―LMS may be becoming victims of their own success,‖ and ―the success of 19 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.6. 20 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT LMS may be providing the conditions for disaster.‖ In Nigeria, for example, despite a rapid increase in the number of LMS employees, it is becoming a challenge to meet the growing demand for services. Organizations that have received services from LMS in the past—such as PROHEALTH, Centre for Population and Environmental Development, and GECHANN—are expressing a continued desire for support and consultation. In addition, the USAID Mission would like LMS-Nigeria to increase its work with the Global Fund Country Coordinating Mechanism and will soon be referring up to 40 new agencies for the LMS Capacity Building program. LMS’ ability to effectively respond to the increasing demand for services needs to be addressed. DIFFICULTIES IN MEASURING AND EVALUATING CAPACITY BUILDING One of the biggest challenges faced by LMS is the inherent difficulty in measuring and evaluating the impact of leadership and management interventions. Individual development and system improvement are often the result of many factors. Because of this, it is difficult to isolate the effect of a single intervention from other contributing historical, contextual, or programmatic events. Evaluation is also made difficult by short timelines. LMS is in its fourth year, yet outcomes of interest, such as changes in skill levels, enhanced community involvement, strengthening of organizational systems, and improvements in health indicators take time—in some cases years to achieve. To evaluate the success or value of a program, funders and program staff often select and monitor metrics that are easy to count, but are poor markers of success. A significant theme from this study was that the evaluation strategy for LMS has fallen victim to this flawed approach. One individual claimed that the metrics required by USAID reports equated to ―bean counting.‖ Another contrasted the focus on HIV treatment statistics for PEPFAR-funded projects with the absence of indicators that capture the power in the systems that had been strengthened or the community social networks that had been formed. Some quantitative measures may be appropriate for measuring short-term process goals, but in general, a solely quantitative approach to measuring the impact of a complex, multilevel capacity￾building initiative is misguided. The use of other sources of data, such as document reviews that follow changes in public policies and community engagement, as well as qualitative data from key informant interviews, focus groups, case studies, or reflective journals should be used at different stages of the intervention to follow transformative changes in beliefs, practices, and relationships. Understanding the depth and breadth of individual development and health-systems strengthening achieved through the LMS program will require identifying more appropriate quantitative indicators and openness to the use of qualitative data as a way to evaluate intervention success. ASSURING SUSTAINABILITY The concept of sustainability differentiates this program from its predecessor, M&L, yet this concept is difficult to grasp. This creates a unique challenge for evaluators of this type of program. LMS defines the term ―sustainability‖ as the capacity to perform effectively in the future. This definition implies that it is not sufficient just to have individuals maintain their role as a health manager or to have an organization continue to exist. For LMS to meet its goals of sustainability, individuals, organizations, and systems must develop attitudes, skills, and processes that allow them to continuously perform effectively. Sustainability requires an integrated view of the larger health system and multidimensional indicators. The challenge for LMS is to further define sustainability in a way that clarifies success and to validate the indicators of sustainability by carrying out evaluations of long-term outcomes using experimental designs. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 21 VI. LESSONS LEARNED MSH in its self-assessment and other reports has listed many lessons learned.20 A critical lesson learned over the years is the importance of having a clear understanding and appreciation of the impact of environmental and contextual factors in designing and implementing interventions. Another is the need to clarify the roles and expectations between key partners at the beginning of any initiative. A third is the broad applicability of the approaches and tools developed by LMS. Below are three additional lessons learned from the in-country visits made by the evaluation team. WORK AT THE MOST STABLE LEVELS OF THE SYSTEM Health systems development requires approaches that help to strengthen and integrate all levels of the system—individual, team, organizational, community, and national levels. Countries are not always interested in identifying the gaps, challenges, and strengths of their national health system, nor are they always able or interested in finding ways to enhance the effectiveness of their system. Given the lack of stability in some countries and the lack of interest at the national level, it is not always possible to work at all levels. LMS has proven itself very adept at identifying appropriate interventions with in-country partners that take into consideration the politics and other contextual factors that exist. The LMS experience highlights the first key lesson to be shared. In-country programs should work at all levels of the health system, but focus most of their attention on the most stable part of the system. That is, funds used for assisting a country in developing the capacity of its health system should be primarily directed at the part of the system that is most ready for development and has the most chance for long-term consistency and success. All three countries visited in this study are employing intervention strategies that target multiple system levels. Of note, however, is that the Nigerian and Nicaraguan LMS programs have consciously selected to target the parts of the system that are the most stable and ready for change. After the experience of having several efforts not reach fruition, the PRONICASS program in Nicaragua, has decided to work much more intensely at the municipality and community levels. One example is their multi-organizational work in the city of Leon. The LMS team in Nigeria continues to try to involve the national government in health initiatives, but is focusing much of their capacity building efforts on community-based nongovernmental organizations (NGOs). They have been working with 11 NGOs such as PROHEALTH, GECHANN and Christian Health Association of Nigeria (CHAN) to build organizational systems to the level sufficient to meet USG funding standards. These kinds of organizations have been working on health issues in rural parts of Nigeria for years and are expected to maintain their presence in those communities. Building their capacities to run effectively is likely a best practice for LMS. ASSOCIATE AWARDS OFFER LOCAL BENEFITS Currently, the associate award is being used as part of the LMS management model in two countries, Nigeria and Afghanistan. Associate awards are additional cooperative agreements negotiated separately by USAID Missions or Bureaus with the leader (MSH) to support the same objectives as the leader award. Because the leader award was awarded through a full and open 20 Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009, p.79. 22 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT competitive process, Missions and Bureaus can grant associate awards without going through a competitive process. This study found some potential value in exploring this model for the future LMS programs. LMS staff from Nigeria say that this model has a number of benefits. It allows for greater emphasis on local needs and context, a simpler, more streamlined approval process, and more access to USAID decisionmakers. Benefits of associate awards for USAID Missions and Bureaus as listed by Health Systems 20/20, another USAID initiative, include:21 The scope of work is developed by the Mission/Bureau to support the same LMS objectives. No competitive process or need to justify sole source. The Mission/Bureau defines the period of performance, which can extend five years beyond the end of the leader award. Associate awards have ceilings but do not count against the leader award ceiling The CTO and Agreement Officer for an associate award are based in the Mission. The Mission/Bureau receives all program and financial reports directly from the leader recipient. VIRTUAL PROGRAMS HAVE PROMISE The use of virtual platforms has expanded during the last four years of LMS. Many have found the offerings very helpful and of high quality. Of particular note is the blended model of the VLDP, which offers local teams an opportunity to participate in LMS’s flagship leadership development program using a distance-based model but allowing for face-to-face coaching and support at home with a trained coach. Another unique use of technology is the development of an alumni network created by the nursing fellows enrolled in the first Nursing Fellows Program in Nigeria. Through these programs individuals from all over the world have gained access to developmental experiences and networking opportunities. Despite the improved accessibility of virtual programs, several people interviewed for this study said that the current use of these programs is constrained by the limited availability of computers, low computer literacy, and uneven access to broadband in remote parts of many developing countries. Also, for some, the amount of reading that is required in virtual programs makes them less appealing as a learning method. Given the limits of virtual programs, LMS has strategically used them as one of several methodologies for developing individuals and teams. As connectivity expands and technology improves, the uses of virtual programs for professional networking and for learning will expand. LMS programs using virtual platforms are a best practice that can lead the way for future distance-based innovations. 21 http://www healthsystems2020.org/ EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 23 VII. FUTURE STRATEGIC DIRECTIONS The future of U.S. development policy is likely to emphasize longer-term strategies with greater weight given to building in-country capacities and self-reliance. Programs that strengthen leadership and management are certainly one of the building blocks for such a longer-term approach. USAID’s Office of Population and Reproductive Health, with its rich experience managing a history of leadership and management programs, is in a position to lead the way. MSH, under LMS and its predecessors, has developed, tested, and refined methods and tools for strengthening leadership and management that are increasingly seen as essential for the successful implementation of a broad array of health-service delivery programs. However, the need to strengthen leadership and management is great, and health systems strengthening efforts are still needed. To provide recommendations for future investments in this area, the evaluation team considered both the results of this evaluation study and its own knowledge of cutting-edge practices in workforce development and systems strengthening. Several strategies not currently being used by LMS were ruled out as potential suggestions because they were felt to be inappropriate for use in resource-challenged countries. These included labor-intensive strategies that focus more on individual leader development, such as the use of 360 self-assessment instruments and executive coaching programs. In addition, costly strategies such as the development of a new U.S.-based health leaders program like the Institute for Population and Reproductive Health funded by the Gates Foundation at Johns Hopkins University or the Population Leadership Program co￾sponsored by both the Gates Foundation and the Packard Foundation were also ruled out. The evaluation team ultimately concluded that future USAID investments should continue to support those activities which have already demonstrated the potential for success in the context of the developing world. Future efforts need to place even greater emphasis on sustainability: building in-country capacities that support lifelong learning and support the continued maintenance and updating of the rich library of LMS training tools and methods to assure their continued utility well into the future. In addition, for the future, USAID needs to build a broad constituency that understands and supports the vital need to strengthen leadership and management development. To do so, it needs to find better ways to measure, document, and communicate the impact of leadership and management programs. And it needs to find ways to legitimize and promote the use of funds from various vertical programs for leadership and management development to occur throughout the full range of health service programs. EMPHASIZE SUSTAINABILITY USAID is entering a new era in international development that requires a long-term view. Increasingly, USAID will need to pursue long-term strategies while it continues to address the pressing immediacy and demands of vertical health programs. The overarching aim should be to help countries achieve independence: to leave a legacy of countries that are able to plan, lead, manage, finance, and deliver basic health services on a sustained basis. Strengthening leadership and management is a central, possibly the only, means to these important ends. Future systems strengthening projects led by PRH—indeed all USAID health projects—should place a greater emphasis on sustainability. The next LMS Project should include more resources devoted to: 1. Supporting leadership and management training and capacity building for government officials and agencies at national, state, and local levels. Even as the LDPs have focused 24 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT on team development, LMS has recognized the need to gain high-level understanding and support, routinely launching each program with a sensitization session designed to orient senior officials. Too often these senior officials also lack important leadership and management skills. One official in the GHS, who has played a key role in supporting the scale up of the LDP throughout the country, offered a telling story: An earlier program of management training empowered a generation of leaders who have now risen to positions of authority. But there was no follow-up, so there are not leaders to follow us. Now we need to train others, but this time we need to build the capacity to carry on our legacy. 2. Support for building the leadership, management systems, and organizational capacity of local NGOs and faith-based organizations (FBOs). Governments change, team members are reassigned, and it is often difficult to maintain leadership continuity and programmatic momentum. In most developing countries, local NGOs and FBOs have a long history of committed involvement and many have demonstrated their expertise in providing health services. What they often lack are the skills and systems needed to use their resources efficiently and to secure and manage new sources of revenue—such as USAID grants. The Capacity Building project in Nigeria, cited earlier in this report, illustrates what might be done in other countries. A goal of USAID is to bring a country to a point where it has the capacity to perform effectively in the future. If USAID support is terminated too early, program gains can stagnate and erode. Therefore, clear criteria of sustainability are needed. Potential criteria include: Measurable improvements made in health status, A critical mass of local organizations and agencies that are able to organize, manage, and provide clinical health services, A cadre of individuals who have the leadership and management skills needed to continue and extend the gains achieved, Operational essential management systems, Government leaders at every level who are supportive of change and progress, and Government budgets and other revenue sources that are adequate to support and extend the gains achieved. SUPPORT LIFELONG LEARNING Systems for lifelong learning must be in place in order for countries to maintain and continue the gains achieved through the M&L and LMS Projects. Future initiatives to strengthen health systems need to help create, support, and enhance programs designed to encourage lifelong learning. As mentioned previously, virtual programs and networks, such as the LeaderNet, offer great promise for the ongoing development of health professionals in resource-challenged communities. There are also a couple of other strategies that, if further developed, would strengthen in-country lifelong learning systems. These are: 1. Developing in-country pre-service and continuing education programs for health professionals. A promising strategy to promote lifelong learning that is incorporated in the current LMS Project, but has not been as actively pursued as is needed, is the development of in-country pre-service and continuing education programs for health professionals, including curriculum development and teacher training. In most developing countries, those who hold managerial positions are, by training and preference, medical practitioners with little or no EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 25 training in leadership and management. Nicaragua provides an example worth replicating where PRONICASS has been working with the National Autonomous University of Nicaragua to help design teaching modules on leadership and management for those in their fifth year of medical school and students in the Masters Program in Sexual and Reproductive Health. The University expects to also make the module available as a continuing education course for those already in practice. 2. Develop indigenous partners who share a common teaching philosophy. One challenge in developing such in-country capabilities is a critical difference in teaching philosophy between MSH/LMS and many indigenous teaching institutions. Many in-country training centers rely on lecture-based teaching methods backed up by extensive reading and memorization. The approach used by MSH minimizes lectures, promotes greater self-reliance, and stresses applied learning. Recognizing the difference and its importance, and helping institutions and teachers adopt improved methods, cannot be achieved in a short-term intervention. Relationships of trust and understanding must be nurtured, indigenous case studies and teaching materials must be developed, and teachers must be trained in how to teach differently. The integration of the LMS applied-learning model into the curriculum of Makerere, Uganda, School of Public Health and Faculty of Medicine provides an example of the impact of a successful relationship with an indigenous partner. This could also be replicated in other countries. UPDATE AND MAINTAIN THE LIBRARY OF TOOLS AND METHODS Future programs need to support the updating and maintenance of the MSH library of tools and programs. The library and toolkit are rich resources for training future leaders and managers and for supporting their need for lifelong learning. Maintaining, refining, and updating this collection of materials is a challenging task. As noted earlier, there are now 61 tools and programs in the inventory. A few examples will illustrate: The workhorse handbook, Managers Who Lead, was first published in 2005 and is available in English, Spanish, French, Nepali, and soon, in Portuguese. Further translations are likely and like all publications the manual will need to be updated. The Virtual Leadership Development Program (VLDP) is thought by some to be too long, and it has been suggested that a shorter version would be more readily utilized. Others observe the VLDP lacks the dynamics and energy of face-to-face training. The full potential of VLDP and other extended virtual programs will be realized as the constraints of technology in many developing countries are overcome. There is a continuing need to develop and test new materials such as training modules on advocacy, storytelling, and developing strategic networks. MSH should also develop a training course for field officers on LMS tools and their application. The variety of tools is large, each with its own application and approach. Field personnel need to know what is available to draw on to help meet local needs. There is a slide show that briefly reviews various tools, but field officers also need to know how to adapt the tools to best suit local requirements. They also need to become expert in the tools’ use—to internalize their elements—in order to maximize their effectiveness. BUILD A CONSTITUENCY FOR LEADERSHIP AND MANAGEMENT USAID should build a constituency of understanding and support for leadership and management programming based on principles of skills-building and applied learning. Each vertical health program has strong and time honored support at every level: Congress, USAID bureau staff, targeted program specialists, country Mission officers, and local service 26 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT providers. These strong constituencies assure that each vertical program has influential clout and continuing funding that is largely unaffected by changing political climates. Not so for leadership and management training. Yet the long-term success and sustainability of each of the vertical programs is highly dependent on how well they are implemented and managed in the field, and on the ability of the in-country health system to assume the ongoing responsibility for organizing, managing, and financing the effort. USAID should engage MSH to help build greater understanding of its approach and its application. Those who are exposed to the MSH/LMS approach usually become believers. Their tools and methods are qualitatively different from other traditional forms of leadership and management training. In addition, they are effective in building the in-country capacities needed for sustainability and graduation. Not everyone understands their methodologies nor places much weight on their importance. While others could benefit, USAID should target three critical internal audiences for education. These are: 1. Senior USAID officers. MSH should develop better ways to tell its story to USAID’s senior￾level officers, who can be overwhelmed by reports and big-picture statistics but are moved by stories of grass-roots initiative, the kinds of actions that impact on local communities and individual people. The Egypt DVD, produced by RF/Binder, that briefly tells the story of accomplishments in Aswan, is a good example. It was repeatedly cited as a moving story of a successful program. More such stories should be told. 2. Program and contract managers. CTOs are under enormous pressure to assure that contractors fulfill both their contractual and programmatic obligations. Often the demands for monitoring statistics, tracking finances, and managing the ―burn rate‖ co-opt the time they would like to spend discussing with colleagues innovative solutions to nagging problems, or conducting field visits for first-hand exposure to in-country concerns and priorities. 3. Mission staff are already some of MSH’s strongest advocates. As noted by the evaluation team, an experienced and motivated mission officer can be remarkably creative and industrious in cobbling together resources to address a local need. However, over the last several years, increased numbers of experienced field personnel have been transferred or retired and their younger replacements are sometimes sent to the field with minimal orientation. Because many come from technical fields, they have not had much leadership and management training. USAID should develop a strategy for building a greater political constituency. Ultimately, with a larger cadre of supporters, it is possible to envision more health advocates articulating the need for stronger health systems with decisionmakers and politicians. Their recommendations that future programming efforts place a greater emphasis on strengthening leadership and management capacities will support sustainable FP/RH programs as well as other health initiatives. FIND BETTER WAYS TO MEASURE SUCCESS OF LEADERSHIP AND MANAGEMENT As has been recently seen in the worldwide economic downturn, there is a danger of losing sight of broader and longer-term gains. In the U.S. economic and business sectors, stock prices and executive bonuses have been directly linked to short-term quarterly gains, or at best annual profits, which can compromise longer-term growth and stability. Similarly, efforts to measure the impact of LMS’ interventions tend to focus on short-term results while true measures of success are more long-term and tenuous. USAID, with MSH’s assistance, must find better ways to measure the results of leadership and management programs, and to demonstrate successes in persuasive terms. In its EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 27 monograph, ―Leadership Can Be Learned, But How Is It Measured?‖ MSH presents a thoughtful discussion and suggests methodologies for measuring intermediate outcomes (such as changes in work climate or in management systems), and long-term outcomes (such as increased delivery of services). However, the paper makes the point that ―not all leadership results are quantifiable.‖22 Rightfully, most health service delivery programs measure their success in the short term by the volume of services delivered and ultimately, in the longer term, by changes in health status. Because of the nature of LMS’ work and the short timeframe of most of its program activities, many of its successes are based on anecdotal evidence or measures of output rather than longer￾term indicators of health outcome. Admittedly, the line from leadership and management to measurable impact on health status indicators is tenuous, and results are affected by numerous uncontrolled (or unrecognized) variables. It is possible to look at outputs (numbers trained) and proxies of success (access to services), but hard numbers are elusive. Current LMS program efforts are too short to establish much momentum, to demonstrate results, or to build sustainable capacity. The LMS-ACT associate award in Nigeria is an example. LMS￾ACT, like other PEPFAR-supported programs, is driven by clinical and technical numbers, where LMS’ strengths are in leadership and management. LMS believes it needs to provide support for the development of basic management systems before these impact outcomes can be achieved. As a result, LMS-ACT has fallen behind in achieving its target numbers. LMS wants the government to recognize its own responsibilities—and has provided some training for government officials at the national level and in a few selected states. LMS is optimistic that government officials will catch up on their targets and that the program will be more successful in the future because of their efforts. The USAID Mission in Nigeria agrees that more effort needs to go into building understanding and support at all levels of government, and it intends to build this into the associate award extension. What is missing is a follow-on effort to monitor the longer-term impact of the effort. USAID should encourage more operational research designed to better measure programmatic results. The findings of such studies should be published and subject to peer review. Results might be more measurable than might be expected, because most leadership and management efforts are set up as short-term projects with short-term measures. Such programs are rarely set up as research projects, with baseline measures and monitoring over medium and long term. One such study is being undertaken in Kenya, where an evaluation team will attempt to measure the impact of the LDP scale-up effort using a quasi-experimental design. LEGITIMIZE BROAD PROGRAM APPLICATION All the vertical programs need leadership and management support. While core funds and management oversight are centered in PRH, and there are expectations that the LMS Project will demonstrate its impact on FP/RH parameters, its true impact is much broader. Because of the wide applicability of its approach, and its eager responsiveness to every opportunity offered by USAID Missions, the LMS Project has been active in the full array of health and social service programs, such as FP/RH, HIV/AIDS, MCH, and education, as it was designed to be. However, some of the LMS activities are considered by some USAID officials to be out-of-scope excursions that are being subsidized by PRH. In contrast, several other key stakeholders held a positive view of the LMS design to accept funding from areas beyond FP/RH, noting that leadership and management support is an important contribution to all areas of the health sector: 22 USAID, ―Leadership Can Be Learned, But How Is It Measured?‖ Occasional Paper No.8, Management Sciences for Health, Cambridge, MA, 2008. 28 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT I do not agree with the implied criticism of this project that it may not focus enough on ―family planning and reproductive health managers‖—if that was the main intention of the project, the GH bureau should have limited the funding sources available for the project. The development of leaders and managers is a very important topic which should continue to be offered through USAID field support mechanisms, however, as with any health systems intervention, there should not be arbitrary foci placed on the activity as being for ―Family Planning Managers.‖ Most managers are responsible for multiple health interventions and cannot necessarily be isolated as ―family planning and reproductive health.‖ USAID should do more to encourage the development of health-sector managers in general. Expanded support from the various health programs for capacity-building initiatives would raise awareness of the critical importance of leadership and management in the success of all USAID initiatives. Contracting options that legitimize the broader application of leadership and management programming need to be developed. Since efforts supporting health systems’ capacity building and workforce development provide benefit to a variety of health programs, future initiatives sponsored by USAID should assure that barriers to collaborative funding between vertical programs are minimized. USAID Missions are quite adept at managing financial resources from multiple sources. It seems more difficult at the central level in Washington. Three future funding options that might strengthen health systems capacity development in USAID programs are listed below. 1. Contract under the Office of Population and Reproductive Health. This is a continuation of the approach used for LMS and for previous leadership and management projects. It is subject to the same pluses and minuses of the current arrangement. It might be improved with an up-front understanding that legitimizes cross-program applications. 2. Develop a global contract. This approach legitimizes applications in all program areas but suffers from the demands for greater coordination and managerial complexities. Possibly some new collaborative arrangement can be developed that would overcome the limitations of this approach. 3. Add leadership and management to each service program contract. This approach might be pursued independently or in concert with a new PRH-based contract. While not confirmed, it is rumored that PEPFAR II, the follow-on to the very urgent, numbers-driven, PEPFAR program, is expected to place a greater emphasis on long-term sustainability, including providing resources to support human resources development and capacity building. Each of these options will require a full exploration of the benefits and potential challenges. Under any option, USAID’s important work in strengthening leadership and management capacities for improving health care services needs to continue. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 29 APPENDIX A. SCOPE OF WORK FINAL SCOPE of WORK for an EXTERNAL PARTICIPATORY EVALUATION of the LEADERSHIP, MANAGEMENT and SUSTAINABILITY (LMS) PROGRAM I. STRATEGY, PROJECT, OR THEMATIC TOPIC TO BE EVALUATED Project Name: Leadership, Management and Sustainability (LMS) Cooperative Agency: Management Sciences for Health (MSH) Cooperative Agreement Number: GPO-A-00-05-00024-00 Agreement value: $119,309,205.00 Obligation Date: 8 August 2005–7 August 2010 The Leadership, Management and Sustainability Project (LMS) is implemented by Management Sciences for Health (MSH). On August 8th 2005, USAID awarded a competitive five-year Leader with Associates (LWA) cooperative agreement to MSH for the LMS activity ending August 7th 2010, with a ceiling of $119,309,205. An external participatory evaluation of LMS activities will be conducted from February 2009—April 2009. II. BACKGROUND The LMS project was designed to build upon proven management and leadership strategies developed by the previous Management and Leadership (M&L) project. LMS activities integrate leadership and management by using the Challenge Model, which focuses on six main techniques to leadership and management: Scan, Plan, Focus, Align/Mobilize, Organize, and Implement. This coupled with strong monitoring and evaluation enables those that participate in LMS programs to face challenges and achieve measurable results. Programs include virtual leadership development programs held online, face-to-face leadership training, and other innovating training methods such as online forums and virtual conferences. The objectives of LMS programs are to: 1. Improve management and leadership of priority health programs (IR1) a. LMS will support and equip a critical mass of managers who lead at all levels throughout the health system to advocate for and implement inspired leadership and sound management. 2. Improve management systems in health organizations and priority health programs (IR2) a. LMS will transfer approaches and skills to organizations to ensure that management structures and systems contribute to sustainable organization success. 3. Increase sustainability and ability to manage change (IR3) a. LMS will enable organizations and individuals to lead and manage concerted responses to complex health challenges at all levels in NGOs and the public sector, multisectoral bodies, national governments, and international agencies. LMS activities reinforce USAID’s Global Health programming by building capacity in organizations and individuals to manage and lead effective, sustainable health care systems. LMS works with public and private organizations in developing countries that provide primary MCH and RH care, FP, HIV/AIDS-related services, and treatment for infectious diseases. LMS provides technical services to national ministries of health, decentralized health services at various levels of government, and international and local nongovernmental organizations, and it participates in a variety of international programs and policy arenas. 30 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT III. PURPOSE OF THE EVALUATION The purpose of this external participatory evaluation is to assess the process and outcomes of LMS programs on strengthening leadership and management skills among providers in developing countries worldwide. The evaluation should focus on both the technical aspects of the program, including activities to improve RH and FP outcomes, as well as management systems used to carry out the program, including managing different funding sources and communication strategies. This information will help assess the impact of investments in leadership and management projects and will help inform USAID and other stakeholders about new projects targeting leadership and management development. This evaluation will be conducted in 2009, one year prior to the end of project, which will allow for a thorough assessment of activities as well as allow enough time to modify strategies for possible future proposals. Qualitative and quantitative data will be used to illustrate the process and outcomes of leadership and management activities. Data from the evaluation should be specific to leadership and management skills and health systems strengthening for FP/RH programs. IV. STATEMENT OF WORK The evaluation team will be tasked with addressing four overarching questions: Additional questions can be found in Appendix A: 1. What have been the greatest FP/RH successes as a result of the LMS project? 2. How successful were LMS management systems in carrying out the various programs? 3. What were major challenges faced by LMS and lessons learned? 4. What are future strategic directions for leadership and management in health and family planning? We anticipate the evaluation to begin in February 2008 and last approximately 10–12 weeks. Additional time may be required for the team leader to finalize the report. V. METHODS AND PROCEDURES 1. Kick-off meeting: The evaluation team will meet with GH Tech and USAID staff to finalize the scope of work and discuss the data collection and analysis plan. Objectives of the interviews for major stakeholders will be finalized. In addition, a communication strategy and plans for data collection and analyses, including in-depth review of technical documents and interviews, will be discussed and finalized among team members. 2. Self-assessment: MSH will prepare a self-assessment of the LMS program, based largely on the general questions included in Appendix B. This report will be provided to the evaluation team as part of the background materials. 3. Review of background documents/materials: The following documents will also be provided to the evaluation team. Other documents may be added or requested as needed or deemed appropriate. – LMS cooperative agreement, amendments and proposal, – LMS semiannual and annual reports, – LMS annual management reviews, – LMS performance monitoring plan, – Selected LMS publications, – Selected LMS research and technical reports, and – LMS country program evaluations. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 31 4. Interviews: The evaluation team will interview selected USAID staff, including staff from the Commodities Security and Logistics (CSL), Service Delivery Improvement (SDI), and Policy, Evaluation, and Communication (PEC) divisions, and Office of Health, Infectious Diseases and Nutrition (HIDN) and the Office of HIV/AIDS (OHA). The team will also interview LMS staff at MSH headquarters and field-level staff in country. Finally, staff from other cooperating agencies, particularly those that have partnered with LMS, multinational groups such as WHO, donors, selected ministries of health, and participants of LMS programs will also be interviewed by LMS staff. Interviews with U.S.-based USAID or MSH staff will be conducted face-to-face; however, it is expected that some interviews may need to be conducted via conference call. 5. Field Visits: The evaluation team will travel to each country, to visit ongoing LMS subprojects. Likely illustrative countries to be visited include: Peru, Nicaragua, Nigeria, and Ghana. The team will assess program implementation, evidence of collaboration, and the impact of LMS activities. Final selection of countries to be visited will be determined by degree of local Mission support, level of resources invested in the LMS program, and the size and diversity of in-country subprojects. VI. IMPLEMENTATION Team Composition The evaluation team must be qualified and be sufficiently respected so that its recommendations will be authoritative and influential. The evaluation team will be composed of two outside consultants with significant knowledge about leadership and management in developing country settings and strong knowledge about reproductive health and family planning services. Two representatives from USAID with complementary knowledge in family planning and leadership and management may be involved as needed to offer donor perspectives as well as technical input. The team should have expertise in leadership, management, reproductive health, and family planning, with particular focus on: Knowledge of leadership and management in developing country settings, Experience in health systems strengthening, Experience in the management of USAID-sponsored family planning and other reproductive health programs in developing countries, Developing country experience, Excellent writing and communication skills with experience in producing team-based reports, For any team member going to Peru and Nicaragua, fluency in Spanish is required, and Ability to travel extensively in short amount of time. Potential candidates for this team may include: senior-level persons with careers related to health systems strengthening, leadership and management, and human resources in developing countries. The candidates must be able to work in a team to evaluate and synthesize information quickly, make clear and well-founded recommendations, and contribute to the written report and debriefings. Careful judgment should be used to recruit consultants who are knowledgeable and highly respected in this field, but are unbiased about this technical area and its future directions. Approximately six weeks, though not necessarily consecutive, of effort will be required for each of the team members, with an additional two weeks for the team leader. A suggested breakdown of time is included in the following table. Timelines should be adjusted based on the individual requirements of team members as long as the team as a whole can function coherently and complete the task. 32 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Scheduling and Logistics GH Tech will be responsible for providing logistical support to the team, including scheduling meetings and D.C.- based interviews, making copies of documents and drafts, making flight and hotel travel arrangements, and obtaining visas and reimbursements for expenses. Field visit logistics will be organized and managed by the in-country MSH/LMS teams. This includes travel within country and in-country transportation. TABLE 1: ACTIVITY BREAKDOWN Activity Days TL Days Assoc 1* Person Days Calendar Days Preparatory Work Review of background documents/materials 3 3 6 3 Team travels to Washington, D.C. 1 1 2 1 Team planning meeting with USAID 2 2 4 2 Continue review of documents/materials, develop interview instruments, schedule D.C.-based interviews and make travel arrangements 1 1 2 1 Preparation of USAID Mission survey 3 3 6 3 Data Gathering Conduct D.C.-based interviews and other U.S. and/or international conference calls 3 3 6 3 Visit to MSH Boston office (includes travel day) 3 3 6 3 Field visit country 1 (Nicaragua) and writing country report (includes travel) 8 8 16 8 Field visit country 2 (Nigeria) and writing country report (includes travel) 9 9 18 9 Field visit country 3 (Ghana) and writing country report (includes travel) 6 6 12 8 Data Analysis/Drafting Report Data analysis/drafting report (includes required expense reporting) 6 6 12 6 Debrief at USAID (includes travel) 1 1 2 1 Submit draft report - - - - Depart from Washington, D.C. 1 1 2 1 Receive feedback - - - - Revise report based on feedback 5 3 8 5 Submit revised final report - - - - Total Days 52 50 102 52 TL=Team Leader * A second internal USAID team member may be included in on the team with a similar breakdown of time. EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 33 A six-day work week is authorized while conducting country visits. VII. DELIVERABLES 1. USAID Survey of Missions: A survey of LMS activities in country missions will be developed and conducted by the evaluation team consultants. This survey should be completed at the start of the evaluation to ensure results are received and incorporated into the final report. Additional follow-up on results may be conducted during country visits. 2. Evaluation Report: The evaluation report should describe the methodology, provide conclusions on the key evaluation questions and offer recommendations for the future. It is expected to be approximately 25–30 pages, including a five-page executive summary, with attachments as needed to clearly illustrate or highlight key points. A near-final draft should be shared with USAID and MSH for corrections of facts and feedback. Recommendations should be those of the evaluation team as a whole. This report is primarily intended for internal USAID use in assessing the performance of the LMS Program and defining future program needs. All or parts of the report will be shared with MSH. However, any recommendations to USAID regarding future procurement issues may be kept internal to USAID. 3. Debriefings: The evaluation team will provide separate debriefings in Washington D.C. to both USAID and MSH staff. Debriefings will be 30–40 minutes slide presentations with and time allotted for questions. Specific LMS Evaluation Questions What have been the greatest family-planning and reproductive-health successes as a result of the LMS project? 1. What progress has LMS made in improving management and leadership for health, and in demonstrating how improvements in management and leadership contribute to improved access, quality, and sustained service delivery? 2. What are LMS’s primary accomplishments from the investment of population core funds? How have core funds contributed to the overall success of LMS, e.g., have they provided innovation, tool development, scalability/replicability, field performance, leverage of field funding, other? 3. What are LMS’s primary accomplishments from the investment of field support? Are there specific accomplishments that have been achieved in a context of decentralized health services? How successful were LMS management systems in carrying out the various programs? 1. How effective is the LMS organizational and management structure in achieving results? How does the LMS structure maintain the quality of LMS’s work? 2. Is the LMS management team responsive and accountable to its key clients and partners: USAID Missions, USAID/GH, and host country partners (i.e. government and NGOs)? 3. Are the systems developed by LMS for monitoring, evaluation, and knowledge application effective? How have these elements of the program supported the achievement of the overall project objective? 4. Have GH/PRH and relevant USAID Missions been effective in managing the LMS activity? 34 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT What were major challenges faced by LMS and lessons learned? 1. What specific technical approaches or products of LMS have demonstrated the greatest impact in developing strong managers and leaders? 2. What is the value-added of the management and leadership tools developed or refined under the LMS program? Who uses these, why, and how? Specific tools include the Developing Managers Who Lead Handbook, Health Manager’s Toolkit, The Manager, the Business Planning Program, Management and Organizational Sustainability Tool (MOST), the Cost Management Tool, the Financial Management Assessment Tool, and Knowledge Folders. 3. One of the key approaches of LMS has been the introduction and application of electronic learning and exchange platforms. These include the Virtual Leadership Development Program, the Business Planning Program, the Global Exchange – RH, the Technical Cooperation (TC) Network, Leadernet, etc. 4. How relevant and accessible are these e-learning tools to managers in low-resource settings? Has the investment in these electronic platforms contributed substantially to LMS’s ability to replicate and scale up more effectively? Do these e-tools and platforms contribute to good management practices and to overall sustainability of priority health programs? 5. How has LMS replicated and scaled up successful technical approaches and products? What lessons have been learned about the process of replication and scale- up, particularly the transfer (applicability) of approaches and products to different cultural contexts? 6. Compare LMS’s mainstreaming strategies and approaches to transfer, integrate, and institutionalize leadership and management skills with regard to efficiency, effectiveness, and sustainability in the field. What are future strategic directions for leadership and management in health and family planning? 1. What are the priority areas for future population core investments to address USAID’s primary objective to improve global leadership. What gaps and/or future opportunities exist for global technical leadership and field implementation in management and leadership? 2. Is there a justification for GH/PRH to treat management and leadership as a specific technical program to improve the delivery of population and reproductive health services and other priority health services? What are the advantages and disadvantages of maintaining it as a distinct activity versus combining different aspects of the program into other GH activities? 3. What components of the LMS portfolio should be maintained in their current form? What components should be retained, but modified? Are there components or approaches that are no longer needed? 4. What are the prospects and the main challenges for continued utilization of tools developed or refined under LMS after the end of this cooperative agreement? 5. What are the prospects and the main challenges for maintenance and utilization of the different electronic platforms, managed discussion groups, etc., developed or refined under LMS after the end of this cooperative agreement? 6. What are some promising new developments in leadership and management that should be explored in possibly future activities? EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 35 LMS Self Assessment Questions Leadership, Management, and Sustainability Self-Assessment Questions October 14th, 2008 LMS Intermediate Results: Improved management and leadership of priority health programs. Improved management systems in health organizations and priority health programs. Increased sustainability and ability to manage change. Programmatic and Technical Accomplishments 1. What progress has LMS made in improving management and leadership for health, and in demonstrating how improvements in management and leadership contribute to improved access, quality, and sustained service delivery? In responding to the above questions, please focus on family-planning related interventions and: – Describe the key needs/barriers/gaps at the beginning of and throughout the project. – Describe the interventions developed by LMS to address those needs/barriers/gaps. – Provide any available documentation of the scale-up and mainstreaming and results and impact of these interventions. – Describe the greatest challenges and/or constraints faced by LMS. – Describe the most important lessons learned to date. – Describe contributions made to global leadership, to advancing research and innovation, and to transferring new technologies to the field. 2. What are LMS’s primary family planning accomplishments from the investment of population core funds? 3. What are LMS’s primary family planning accomplishments from the investment of field support funds? What specific accomplishments have been achieved in the context of decentralized health services? 4. Describe interventions related to LMS’s work in the Global Leadership Priority (GLP) areas of Maximizing Access and Quality (MAQ), Repositioning Family Planning, and Contraceptive Security. – Describe the specific interventions in each of these GLP areas. – Provide any available documentation of the scale-up and mainstreaming and results and impact of these interventions. – Describe the greatest challenges and/or constraints to working in these GLP areas. – Describe the most important lessons learned in each GLP area. – Describe contributions made to global leadership, to advancing research and innovation, and to transferring new technologies to the field. 5. Please describe interventions related to LMS’s HIV/AIDS work. – Describe the specific HIV/AIDS interventions. – Provide any available documentation of the scale-up, mainstreaming, and results and impact of these interventions. – Describe the greatest challenges and/or constraints to this HIV/AIDS work. 36 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT – Describe the most important lessons learned. – Describe contributions made to global leadership, advancing research and innovation, and transferring new technologies to the field. Management and Implementation 1. What were the most significant structural or management challenges (e.g. with regard to project design, staffing, partnering, or funding) faced by the project? How were they addressed or overcome? 2. How has the project addressed financial reporting, tracking, and documentation? Please give a financial summary of project funds. What are the results and what lessons have been learned? 3. How would you describe your relationship with USAID and specifically the LMS management staff at USAID? How has this helped/hurt the achievement of project results? How would you improve LMS management at USAID? Monitoring and Evaluation 1. How effective are the systems developed by LMS for monitoring, evaluation, and knowledge application? How have these elements of the program supported the achievement of the overall project objective? 2. What are the issues, challenges, and lessons learned in monitoring, reporting, and operations research to support the overall accomplishment of the LMS objectives? What is LMS doing to ensure sustainability of its activities? 3. Where does the project see the best chances for sustainability and what steps are being taken to focus on sustainable activities? Partnerships 1. Describe key partnerships LMS has forged to carry out activities. What have been the challenges? How has LMS maintained communication with partners and how would you change management of partnerships? Lessons Learned 1. What specific technical approaches or products of LMS have demonstrated the greatest impact in developing strong managers and leaders? Use data/indicators to demonstrate impact 2. How has LMS replicated and scaled up successful technical approaches and products? What lessons have been learned about the process of replication and scale-up, particularly the transfer (applicability) of approaches and products to different cultural contexts? 3. What ideas or interventions did LMS pursue that did not achieve anticipated results? What has been learned from this? Future Strategic Directions 1. What are the gaps in technologies, methods, or tools needed to further develop the field of leadership and management in developing country health settings? How is LMS positioned to contribute to filling these gaps? EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 37 APPENDIX B. EVALUATION WORKPLAN USAID LMS Evaluation Project - Evaluation Team Workplan Deliverable Activity Due Date 13-Feb 20-Feb 27-Feb 6-Mar 13-Mar 20-Mar 27-Mar 3-Apr 10-Apr 17-Apr 24-Apr Team Planning Meeting Understand SOW 10-Feb Understand client needs 10-Feb Establish team norms 10-Feb Create team workplan 11-Feb Understand LMS Program Design and History Read critical background reports 19-Feb Create diagram of LMS logic model 19-Feb Meet wi h MSH staff in Boston 2/17-19/09 Identify stakeholders 10-Feb Evaluation Study Design study methodology 10-Feb Identify key references ongoing Identify study limitations & biases 19-Feb Develop data analysis tool 13-Feb Mission Survey Create survey protocol 10-Feb Create survey instrument 12-Feb Distribute"approved" survey tool 12-Feb Analyze survey results 4-Mar Key Informant Interviews Create Interview protocol 10-Feb Create Interview tool 11-Feb Conduct interviews 2/11-3/22 Analyze interviews results - USA 3-Mar Analyze interview results - Nicaragua 28-Feb Analyze interview results - Nigeria 17-Mar Analyze interview results - Ghana 24-Mar Communicate Results - Oral Presentations Debrief field visit with staff - Nicaragua 27-Feb Debrief field visit with staff - Nigeria 16-Mar Debrief field visit with staff - Ghana 21-Mar Debrief USAID Program Directors 26-Mar Create powerpoint presentation 28-Mar Present findings to USAID Senior Sta 30-Mar Present findings to MSH Staff 31-Mar Communicate Results - Report Submit 1st draft final report 1-Apr USAID reviews, comments to team 15-Apr Submit final report 22-Apr 38 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 39 APPENDIX C. TEAM PLANNING MEETING AGENDA Monday, February 9, 2009 9:00 – 9:15 GH Tech welcome 9:15 – 9:45 Team introductions Administrative briefing I 9:45 – 10:30 Initial understanding and discussion of the scope of work 10:30 – 10:45 Break 10:45 – 12:15 Implementing the scope of work I The four areas of inquiry Data collection and analysis strategies Key informants 12:15 – 1:15 Working lunch Teamwork, team roles, team work styles 1:15 – 2:15 Scope of work implementation II Interview protocols 2:15 – 2:45 Preparation for USAID briefing 2:45 – 3:00 Break 3:00 – 5:00 USAID briefing Tuesday, February 10, 2009 9:00 – 9:45 Debrief from USAID meeting Review USAID expectations for the assignment Review background and context Identify additional questions for USAID staff 9:45 – 10:30 Methodology Finalize interview protocols and instruments 10:30 – 10:45 Break 10:45 – 12:15 Calendar Travel schedule Key dates for draft, final products 12:15 – 1:15 Working lunch Administrative briefing II 40 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 1:15 – 2:30 Report outline Individual assignments Team roles and responsibilities 2:30 – 2:45 Break 2:45 – 3:30 Communication plan 3:30 – 4:30 Next steps EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 41 APPENDIX D. MSH MEETING AGENDA Tuesday, February 17, 2009 Session Number Time Topic Presenters And other staff 1 9:00-9:30 Welcome and introductions Overview of the proposed agenda Structure of LMS Joseph Dwyer – Director: LMS Program Tim Allen – Deputy Director: LMS Program Sylvia Vriesendorp – Institutional Development Specialist Cary Perry – Monitoring and Evaluation Specialist Kristin Cooney – Deputy Team Leader – Country Programs Team Alain Joyal – Country Programs Team Leader 2 9:30-10:00 What does LMS do and why? Performance monitoring plan summary Joseph Dwyer Tim Allen Cary Perry Kristin Cooney Alain Joyal Sylvia Vriesendorp 3 10:00-1:00 Country programs Alain Joyal Kristin Cooney Cary Perry Sylvia Vriesendorp Joseph Dwyer Tim Allen 1:00-2:00 Break, review 4 2:00-5:00 Core investments Part 1 Global leadership Mainstreaming Scale-up (non-virtual focus) Joseph Dwyer Tim Allen Kristen Stelljes – Program Officer: Global Leadership Jennifer Leonardo – Senior Program Officer: Mainstreaming Sarah Johnson – Scale-Up Team Leader Karen Sherk – Senior Program Officer￾Virtual Programs Judith Seltzer – Director of Technical Strategy and Quality Assurance Cary Perry - Monitoring and Evaluation Specialist Mary O’Neill – Principle Program Associate: HRH Specialist 42 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Wednesday, February 18, 2009 Session Number Time Topic Presenters And other staff 5 9:00-12:00 Core investments Part 2 Virtual programs and networks Sarah Johnson - Scale-Up Team Leader Karen Sherk - Senior Program Officer￾Virtual Programs Joseph Dwyer – Director: LMS Program Tim Allen – Deputy Director: LMS Program Sylvia Vriesendorp – Institutional Development Specialist Cary Perry – Monitoring and Evaluation Specialist 12:00-2:00 Lunch Time for Review 6 2:00-5:00 Challenges & lessons learned Joan Mansour – Leadership Development Specialist Cary Perry Morsi Mansour – Leadership Development Senior Program Officer Kristin Cooney – Deputy Team Leader – Country Programs Team Alain Joyal – Country Programs Team Leader Sarah Johnson Joseph Dwyer Tim Allen EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 43 Thursday, February 19, 2009 Session Number Time Topic Presenters And other staff 7 9:00-11:00 Management systems Sue Brinkert – Finance and Operations Team Leader Tim Allen - Deputy Director: LMS Program Peter Mahoney – Senior Contract Officer Jessica Sullivan – Contract Officer Kristin Cooney - Deputy Team Leader – Country Programs Team Cary Perry – Monitoring and Evaluation Specialist 11:00-1:00 Review 8 1:00-3:00 Q&A on future directions Joseph Dwyer - Director: LMS Program Joan Mansour - Leadership Development Specialist Morsi Mansour - Leadership Development Senior Program Officer Judith Seltzer - Director of Technical Strategy and Quality Assurance Alain Joyal - Country Programs Team Leader Kristin Cooney Cary Perry Tim Allen Sarah Johnson 9 3:00-4:00 Preparation for field visits Alain Kristin Kathleen O’Sullivan – Principal Program Associate Ana Diaz – Program Officer Diane Carazas – Senior Program Associate – Country Programs Team 4:00-5:00 Remaining questions or materials Joseph Dwyer Tim Allen 44 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 45 APPENDIX E. INTERVIEW PROTOCOLS AND INSTRUMENTS KEY INFORMANT INTERVIEW GUIDE (Note to interviewer: This document is a guide. Specific questions asked may differ depending on the stakeholder’s relationship to LMS) Introduction: ―Thank you for your time. My name is… and this is …. We have been asked by USAID PRH to assess the Leadership, Management and Sustainability (LMS) Program, that is being managed by Management Sciences for Health (MSH). You have been identified as a person with special knowledge of this project. We would like to hear your perspective on the design, management, implementation and impact of the program. The interview should last no more than one hour. We are interviewing a number of people and will be sharing the themes that emerge from these interviews through a final report in April.‖ 1. What is (has been) your involvement with the LMS Program? 2. Overall, what do you think about the LMS Program? Why? Examples? 3. What have been the Program’s greatest successes? Are you familiar with any successes specific to family planning and reproductive health? At what level or levels has the LMS Program had the greatest impact: health system, organization, or individual? 4. How well has the LMS Program been managed? (Note to interviewer: Listen for comments related to financial management, evaluation; trust and follow-through, and communication between funders/field mgrs/service providers/recipients of services.) 5. What are the lessons learned from the LMS Program? (Note to interviewer: Listen for opinions on partnerships, design, delivery, target audience, breadth and depth of programs, and most valuable skill building products/programs.) 6. What would you recommend for future programs in leadership and management? 7. Do you have any other comments or suggestions? Closing: ―We want to thank you for your time and your willingness to share your thoughts and insights with us today. If you have other comments that come to you after we leave, please feel free to contact us.‖ (Note to interviewer: Leave business card or contact information, email or other.) 46 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT THEMATIC CODING SCHEME—LMS EVALUATION PROJECT (2/14/09) THEME/CATEGORIES SUBTHEME * CODE* NEED FOR LEADERSHIP /MANAGEMENT DEVELOPMENT NEED OVERALL FEELINGS FOR MSH or LMS MSH IMPACT IMPACT–HEALTH OUTCOMES IMPACT–INDIVIDUAL IMPACT–TEAM OR ORG. DEVELOPMENT IMPACT–HEALTH SYSTEM IHO II IT/O IHSX PAST OR CURRENT SUCCESSES SUCCESS–DESIGN SUCCESS–EVALUATION /MONITORING SUCCESS–MANAGEMENT OF LMS PARTNERSHIPS COMMUNICATION SUCCESS–FUNDING USAID IN-COUNTRY SD SE SM SM–P SM–C SFUSAID SFIC PAST OR CURRENT CHALLENGES CHALLENGES–DESIGN CHALLENGES–EVALUATION /MONITORING CHALLENGES–MANAGEMENT OF LMS PARTNERSHIPS COMMUNICATION CHALLENGES–FUNDING USAID IN-COUNTRY CD CE CM CM–P CM–C CFUSAID CFIC NEW AND NOVEL IDEAS FOR THE FUTURE FUTURE–DESIGN TARGET AUDIENCE METHODS CONTENT/COMPETENCIES FUTURE–PARTNERSHIPS OR CONTEXT: FUTURE–MANAGEMENT FUTURE–FUNDING/RESOURCES FUTURE–EVALUATION/MONITORING FD FD–TA FD–M FD–CC FP FM FF FE *Comments related to Family Planning/Reproductive Health should be highlighted with an asterisk* EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 47 APPENDIX F. ONLINE USAID MISSION SURVEY ONLINE USAID SURVEY FOR LMS EVALUATION Greetings: The Population and Reproductive Health Office of USAID has requested the assistance of the Global Health Technical Assistance (GH Tech) Project to conduct an evaluation of the LMS Program. Attached please find the scope of work for your information. Your unique perspective on how this program is delivered and the impact it has had would be of great value to the evaluation team. We would appreciate your responses to the following questions by close of business on Wednesday, February 18, 2009. Please include your responses below and reply to all when submitting your responses. Please also use as much space as you need to complete your answers. 1. Using a scale of 1 to 5 (1=not at all, 5 = very), how familiar are you with the LMS programs and services? _________ 2. Using a scale of 1 to 5 (1=not at all; 5=exceeded my expectations), please tell us how well the LMS program met your expectations. _____________ Please give examples of why you rated the program as you did. Use the same scale of 1 to 5 for the following questions on the management of LMS (1=not well, 5=very well). Please feel free to give examples to further explain your ratings. a. _____LMS communication was timely and appropriate. b. _____LMS was responsive to the needs of our office. c. _____Management of LMS between the field office and the LMS headquarters was seamless. d. _____Cooperative agreement /associate award mechanism met our needs. 3. Describe ways in which the management of LMS could be improved. 4. Give examples of how LMS has helped you reach your health sector goals. 5. What barriers or challenges do you face in your country for developing effective health leaders/managers? 6. What suggestions do you have for future efforts to develop family planning and reproductive health managers and leaders? 7. Please include any other comments and suggestions. We thank you in advance for your prompt feedback and look forward to hearing from you soon. Sincerely, LMS Program Evaluation Team 48 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 49 APPENDIX G. PERSONS CONTACTED UNITED STATES OF AMERICA United States Agency for International Development, Washington, D.C. Scott Radloff, Director, Office of Population and Reproductive Health (PRH) Ellen Starbird, Deputy Director, PRH Marguerite Farrell, (CTO LMS Project), PRH Nandita Thatte, (TA LMS Project), Technical Advisor, Global Health Fellows Program, PRH Dana M. Vogel, Division Chief, Services Delivery Improvement Division (GH/PRH/SDI) Carolyn Curtis, Post-Abortion Care, GH/PRH/SDI Rushna Ravji, Services Delivery Technical Advisor, GH/PRH/SDI Lois A. Schaefer, Senior Technical Advisor, Human Capacity Development and Training, Global Health Fellows Program, (CTO Capacity Project), GH/PRH/SDI Alexandra Todd, Repositioning Family Planning, GH/PRH/SDI Kevin Pilz, Senior Technical Advisor, Commodities/Securities, USAID/PRH/CSL (Spain, via telephone) Robert C. Emrey, Chief, Health System Division, Office of Health, Infectious Diseases, and Nutrition (HIDN) Carolyn Indira Mohan, Tuberculosis Advisor, Global Health Fellows Program, GH/HIDN/ID Estelle Quain, Team Leader, Health Systems Strengthening, Office of HIV/AIDS Ishrat Z. Husain, Senior Public Advisor, Africa Bureau Management Sciences for Health, Boston Jonathan D. Quick, President and Chief Executive Officer, MSH Joseph Dwyer, Director, LMS Project Timothy R. Allen, Deputy Director, LMS Project Sylvia Vriesendorp, Institutional Development Specialist Cary Peabody Perry, Monitoring and Evaluation Specialist Sue Brinkert, Finance and Operations Team Leader Diane Carazas, Senior Program Advisor, LMS Latin America Kristin A. Cooney, Deputy Team Leader, Country Program Team Alain Joyal, Team Leader, Country Programs Jennifer Leonardo, Senior Program Officer, Mainstreaming Sarah Johnson, Principal Program Associate, Scale-Up Team Leader Peter Mahoney, Senior Contract Officer Joan Mansour, Leadership Development Specialist Morsi Mansour, Leadership Development Specialist, Senior Program Officer Lawrence S. Michel, Vice President, Center for Leadership and Management Eliana Monteforte, CLM Administrative Coordinator Mary O’Neill, Principal Program Associate HRH Specialist Kathleen O’Sullivan, Principal Program Associate 50 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Judith B. Seltzer, Deputy Director, Technical Strategy and Quality Assurance Karen E. Sherk, Senior Program Officer, Virtual Programs Kristen Stelljes, Program Officer, Global Leadership Jessica Sullivan, Contract Officer Abt Associates Ruth Berg, Project Director, Private Sector Partnerships for Better Health (PSP-One) Barabara O’Hamilton, O’Hamilton Consulting Adventist Development and Relief Agency International (ADRA) Mark Webster, Vice President for Programs Naomi Miller, LMS Project Manager Family Health International Michael Welsh, RTP, NC Global Business School Network Guy Pfeffermann, Chief Executive Officer GH Tech/QED Anne K. Shinn, Program Manager, GH Tech Caroline (Callie) Curtis, TPM Facilitator NICARAGUA United States Agency for International Development, Managua Alexander Dickie, Mission Director Carol Horning, Deputy Mission Director Connie J. Johnson, Supervisory General Development Officer, Chief, Office of Human Investment Dr. Ivan Tercero, Maternal and Child Health Specialist, Office of Human Investment Alicia Slate, Education Specialist, Office of Human Investment Terry Tiffany, Consultant to USAID, GH Tech Ministry of Education, Managua Lic. Guillermo Martinez, Director of Education and Delegation Lic. Luis Ramon Hernandez, Head of Planning Directorate, division of Planning Ministry of Health, Managua Dr. Edmundo Sanchez Cruz, Director General, Health Surveillance National Autonomous University of Nicaragua, Managua Dr. Hugo R. Perez Diaz, Vice-Dean, Clinical Training, School of Medicine Msc. Yadira Medrano Mencada, Coordinator, Masters Program in Sexual and Reproductive Health EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 51 PRONICASS, Managua Barry Smith, Project Director Albahuz Solorzam, Technical Coordinator Violeta Barreto Areas, MINED Coordinator Olga Montalvan, Administrative Coordinator Mon Cacayb, Sub Director, Coordinator MINSA Eduardo de Trinidad, M&E Coordinator Carla Yadua Martinez Martinez, Teritorial Assistant Dra. Maria de Jesus Pastrana, Planning, SILAIS, Nueva Segovia Enf. Maria Auxiliadora Rodriguez, Head of Nursing, Health Unit of Quilali, Nueva Segovia Dr. Victor Manuel Delgado Garcia, Municipal Director, San Jose de los Remates, Boaco Nicasalud, Managua Dra. Josefina Bonilla, Executive Director Dr. Fernando Campos, Sub Director ProFamilia, Managua Dr. Freddy Cardenas Ortega, Executive Director PRONICASS, Leon Lic. Pedro Abarca Dra. Argentina Parajon Lic. Miriam Hermida Lic. Mayela Miranda SILAIS, Leon Dr. Benjamin Barreto, Director Dr. Jose Migual Vilndez, Sub Director Lic. Aida Blanco, Head of Nursing Dra. Mariana Guido Real, Planning Dra. Ana Cristion Melendez Diaz, Epidemiology Mantica Perla Maria Norori Health Unit, Leon Dr. Humberto J. Ramirez, Director Dra. Carmen Maria Delgado, epidemiologist Lic. Flor de MaTonez Palma, Respiratory Therapist Primo de Mayo Health Post, Leon Medical School, Leon Dr. Rodolfo Pena, Dean Dr. J. C. Saravia, Vice Director Dr. Roger Barrios, Clinical Coordinator, Internship Program Evertz A. Delpadillo Moreno, Mayor, Local Government 52 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Glays Baiz, Local Government Filiberto Rodrigez Lopez, Local Government Department of Education, Leon Noelia Gutierrez, Head of Education Lic. Julia Henriquez, Delegate MINED, Malpaisillo Prof. Pedro Joaquin Solis Ruiz, Director, Rural Education, Alfonso Cortes Health Center, Malpaisillo Marvin Rugano Montrago, Director Albeross Balmecda Gutierez, Respiratory Teodolinda Navano Rizo, Epidemiologist Ninett M. Roqez Navela Marlene, Campos Montenegro, Health Education Coordinator Teachers Conference, Malpaisillo Martha Lidia Treminio Salgado, Director Rural Education, Jardin Infantile Pinocho, Malpaisillo Pedro Joaquin Solis Ruiz, Director Rural Education, Real de Tolapa Luis Antonio Espinoza, Director Rural Education, de Elvalle Gerardo Silva, Director Rural Education, las Lomas Julia Henriguez Danila, Delegate, MINED NIGERIA U.S. Agency for International Development Alonzo Wind, Director, Investing in People Office (IIP), USAID, Abuja, Nigeria Christina Chappell, Team Leader, SO-14, HIV/AIDS+TB Siana E. Tackett, Deputy Team Leader, HIV/AIDS+TB, CTO for ACT Linda Crawford, Partner Advisor, SO-14, HIV/AIDS Dr. Jerome Mafeni, Chief of Party, ENHANSE Project Federal Ministry of Women Affairs and Social Development, Abuja Macjohn O. Nwaobiala, Director, Child Development Department Oby Okwuonu, Assistant Director, Child Development Department Ali A. Shafix, Child Development Department Leadership, Management and Sustainability Program, MSH Donald Harbick, Country Director Donna Coulibaly, Deputy Director, Finance/Operations Anddy Omoluabi, Sr. Technical Advisor, Capacity Building Project Ndulue Nwokedi, Director, AIDS Contraceptive Security, AHF Paul Waibale, Project Director, AIDS Care and Treatment Project (ACT) Clare Gibson-Giraud, Grant Management Solutions, Capacity Building Consultant EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 53 Christian Health Association of Nigeria (CHAN) Zipporah Kpamor, Chief of Party, Nigeria Indigenous Capacity Building Project (NiCaB) Country Coordinating Mechanism Dr. Jerome Mafeni, Country Coordinating Mechanism Nigeria, Global Fund Centre for Population and Environmental Development, Benin City, Ebo State Prof. Andrew G. Onokerhoraye, Executive Director Mr. Emmanuel Ideh, Finance/Admin Officer Ms. Lse Akpade, Accounts Officer Mr. Job Eronmuonsele, ICT Officer Gembu Centre for HIV/AIDS Advocacy Nigeria (GECHAAN) Dorothy Helwig Pro-Health International (Faith Based Organization) Dr. Iko Ibanga, Executive Director LMS/MSH, Niger State Dr. Ezekiel James, State Team Leader Dr. Oniyire Adetiloe, Clinical Specialist Ibiang M. Livions Niger State Government, Minna, Niger State Dr. Isah Vatsa, Executive Commissioner of Health Dr. Yahaya Bassa, Commissioner Haj Amina Abdull, D.O./MDGs Ali Shafri Aseesalkar N Bida, DDNS A.C. Tolysaki, CEO/SO Amin Yousfa Nstu Naings Shihu A., ASAPL Ketnuh Usman, M&E Dr. Mohamed B. Usman, DPHC Dr. Adetiloye Oniyire, State Clinical Care Specialist Adam Bana Umar, Director General, Agency for Control of AIDS (SACA) Dr. Chindo I. B., SACA General Hospital Kagara, Niger State Dr. Lamanoro Steve, Acting Head Idrie Dchimu, Consultant Galadima T. Hamzat, House Officer Mohd D. Giwa, House Officer Ezra Inuredia, House Officer 54 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT Ralsi Abdullalhin, CNO Ishaku Musabobi, Pharmacy Ibrahim Taheys, Medical Officer Dr. Okpala Chibuzo Ifeanyi, Medical Officer Ekomanna Sylvia Chimkwesire, Laboratory Scientist Isaiah Jijai, CNO Muh’d Abubel, CNO Yahaya Laiwal, MXE Yusuf D. Moh’d, C.L. Superintendent Rukayyah A. Idrius, Volunteer Amina Suluman, Volunteer Habiba Moh’d, Volunteer Sqadatu Abubakar, Volunteer Mary Johnson, Volunteer Simon D. Anfeto, Volunteer Basic Health Station, Tegina Ward, Niger State Hauma Mohamed, Community Health Education Worker Bimta Florahim, Junior Community Health Education Worker Fati Isah Waziri, Junior Community Health Education Worker Njaica Eberechukwu, Midwife Taleki Sule, Laboratory Technician GHANA U.S. Agency for International Development Susan Wright, Senior Technical Advisor, RH/MCH Ministry of Health Dr. Elias K. Sory, Director General, Ghana Health Services (GHS) Dr. Sylvester Anemana, Director, Human Resources, GHS Adventist Development and Relief Agency (ADRA), Accra Dr. William Y. K. Brown, Country Director, LDP Facilitator Team Leader Victoria Daaku, Programs Director, Training Specialist, Millennium Development Authority (MiDA) Mary Afua Ackummey, Deputy Director, Institute for Educational Development and Extension, University of Education, Winneba Adelaide Maria Angati Ofei, Principal Nursing Officer Brian Sapati, Coordinator, Corporate Governance Program, GIMPA Catholic Hospital, Apam, Gomoa District Francis Yawson, PHSA Dr. T. Amerkan, Acting Medical Superintendent EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 55 Nancy Ekyem, District Director, Agona East Patricia Antwi, DDHS, Awutu-Senja Rev. Sr. Mary Magdalene A. Mearah, PNO, Catholic Hospital Amy Takyi, DHD, Apam Beatrice Kumah, Ngyiresi CHPS Zone Sarah Sarkwah, DPHN, Gomoa West Gifty Ankerk, Ag. DDHS, Gomoa West Gomoa Oguaa Health Center, Gomoa District Alberta Kuofie, Senior Staff Midwife Regional Health Directorate, Central Region Dr. McDamien Dedzo, Regional Director of Health Services Justice Hafobu, Regional Training Coordinator General Hospital, Abura Dunkwa, Abura-Asebu-Kwamankesse (AAK) District Michael Danso, Medical Superintendent Marian Ashu, Principal Nursing Officer (PNO) District Health Station, AAK District Catherine Mokoah Lssier, District Nutrition Officer Mary Amoah, Asuanti Sub-District Rosina Akandi Aselsiya, District Disease Control Officer Margaret Nyarko-Sampson, Senior Midwife Comfort Abayatega, Senior Midwife Sarah Edvarka Mensal, Community Health Nurse Community-based Health Planning and Services (CHPS), Asebu Amantsendo, AAK District Hon. Raphae Cudjoe, Assemblyman District Health Directorate, Mfantsiman Dr. Kwabena Sarpong, Ag. District Director Samuel Sosi, Ag. District Director, Gomoa East, (formerly DD, Mfantsiman) Helena Tibiru, Disease Control Officer Margaret Morrison, Public Health Nurse OTHER Gates Foundation Kathy Cahill, Deputy Director, Global Health Strategies (via telephone) World Health Organization, Geneva, Switzerland (via telephone) Delanyo Dovlo, Health Systems Advisor, Leadership and Management in Health Suzanne Reier, Implementing Best Practices and Scale-Up, MSH/LMS 56 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT 57 APPENDIX H. REFERENCES Leadership, Management and Sustainability Program, ―Self Assessment Report,‖ Management Sciences for Health, Cambridge, MA, January 26, 2009. Leadership, Management and Sustainability Program, ―LMS Programs and Tools: A Technical Introduction,‖ Management Sciences for Health, Cambridge, MA, 2009. Leadership, Management and Sustainability Program, ―Lessons Learned in Mainstreaming and Scale-Up of Leadership and Management Capacity,‖ Management Sciences for Health, Cambridge, MA, January 2009. Leadership, Management and Sustainability Program, ―Linkages between Leadership and Management Strengthening and Service Delivery Improvement and Results,‖ Management Sciences for Health, Cambridge, MA, January 2009. Leadership, Management and Sustainability Program, ―Scaling Up Access to Effective Management and Leadership Practices for Health Organizations through the Use of Virtual Approaches,‖ Management Sciences for Health, Cambridge, MA, January 2009. Leadership, Management and Sustainability Program, ―Semi-Annual Report: July 1 – December 31, 2008,‖ Management Sciences for Health, Cambridge, MA, February 3, 2009. Leadership, Management and Sustainability Program, ―The Results-Oriented Leadership Development Program,‖ Management Sciences for Health, Cambridge, MA, June 2008. Leadership, Management and Sustainability Program, ―The Role of Leadership and Management in Strengthening Good Governance,‖ Management Sciences for Health, Cambridge, MA, January 2009. Mansour, Joan Bragar, Sylvia Vriesendorp, and Alison Ellis, Managers Who Lead, Management Sciences for Health, Cambridge, MA, 2005. USAID and WHO, ―A Guide for Fostering Change to Scale Up Effective Health Services,‖ Implementing Best Practices in Reproductive Health, Management Sciences for Health, 2007. USAID, ―Leadership Can Be Learned, but How Is It Measured?‖ Occasional Paper No. 8, Management Sciences for Health, Cambridge, MA, 2008. USAID, ―Seeds of Success,‖ (DVD), Management Sciences for Health, Cambridge, MA, 2006. 58 EVALUATION OF THE LEADERSHIP, MANAGEMENT AND SUSTAINABILITY (LMS) PROJECT For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com