An Evaluation of the Effects of Re-engineering For Sustainability APROFAM – Guatemala Timothy Farrell, PhD Faro Global, Inc. Guatemala June 2003 MSH/APROFAM Evaluation Timothy Farrell, PhD 3 of 61 June 2003 Executive Summary This evaluation of “Better Health for Rural Women and Children” implemented by Management Sciences for Health under USAID/G-CAP Cooperative Agreement #520-A-00- 98-0045-00 was conducted during May and June 2003. This evaluation was commissioned by Management Sciences for Health (MSH), Boston. The main purpose of the evaluation is to determine the results of MSH work that included a comprehensive review and the initiation of significant structural and institutional changes in APROFAM. This has had a major positive impact on the performance and the financial and institutional sustainability of APROFAM (Asociación Pro-Bienestar de la Familia), Guatemala. The evaluation included reviews of the implementation process and financial results of family planning services and health clinic performance. The data and statistics cover the period 1995–2002. The methods employed include the use of semi-structured interviews; field visits (observations) to clinics, and voluntary promoters as well as detailed review of APROFAM documents and records. The findings strongly support MSH efforts and APROFAM’s willingness and ability to manage the change process in a constructive and productive manner. These joint efforts and the major diversification of clinic and laboratory services resulted in a significant degree of financial and institutional sustainability. The results of MSH activities with APROFAM are institutionalized and sustainable. Already, APROFAM has moved from a traditional NGO highly dependent on donor contributions to a viable enterprise in its own right. The process undertaken with MSH assistance is maintained through a strong new culture in the organization that understands and has internalized those processes so that they are the main tools of management. While under its USAID mandate to urgently work with APROFAM to establish sustainability in its heretofore subsidized clinics, the approach MSH adopted was to develop a holistic approach to management development that consistently and directly linked institutional and management development with improved service delivery, quality, coverage and responsiveness to client needs – in the context of full partnership for ongoing change in APROFAM. Initially, MSH took the lead with a long-term strategy for change. That strategy has now been adopted as its own by APROFAM as is documented in its Strategic Planning documents. This became clear in interviews with APROFAM managers and staff. Positive anecdotes about how “MSH people” provided ample opportunity for APROFAM staff to internalize and act upon proposed major changes clearly indicate the success of the MSH and APROFAM team approach in managing positive institutional change and growth. MSH/APROFAM Evaluation Timothy Farrell, PhD 4 of 61 June 2003 Objective evidence of the successful implementation of the MSH strategies includes: 1. APROFAM has moved from a traditional NGO highly dependent on donor contributions to a viable enterprise in its own right. 2. APROFAM, as an institution, has increased its financial sustainability over this period from about 58% in 1996 to 81% in 2002. 3. Clinic services improved in sustainability from 88% to over 122% in the same period. 4. Four of the five departments of APROFAM have undergone successful re￾engineering and restructuring. The Department of Rural Development is still undergoing this process which began in 1998. 5. The successful re-engineering of the Marketing Department has produced an internal as well as external service quality focus that has become a major part of the “new” APROFAM culture. This “new” Department provides not only forward-looking information for planning and monitoring, but is also a unifying force for institutional identity and culture. 6. This “new” culture is typified by “systems thinking” in which department leaders and staff think about what implications of the changes in their own sphere of activities may have in other areas as well. This is a major key in its current success and bodes well for the future. 7. While APROFAM remains committed to its NGO legal identity and is governed by manifest social goals, mission, and vision, it operates internally as a commercial enterprise. This is clearly a major cultural change in the organization that was spurred by the quest for sustainability. 8. APROFAM is positioned to successfully expand its financial sustainability into the future, even assuming a decrease in major donor funding. To do this, however, there is a need to concentrate on the unprofitable areas within the service delivery operations. These include the Rural Development Department, certain clinics that consistently fail to meet sustainability goals and adjunct programs such as adolescents and violence to women. In addition, given the ever-increasing improvement in the quality of care and services, the sophisticated laboratory equipment and staff, and the improved recognition of APROFAM as a “brand,” expansion into new and more lucrative markets in urban areas is appropriate. MSH/APROFAM Evaluation Timothy Farrell, PhD 5 of 61 June 2003 The process by which MSH assisted APROFAM in realizing these achievements hinged on multiple factors, of which the following stand out: 1. Partnership Approach - MSH proceeded with a high degree of readiness to listen to APROFAM’s needs and end goals rather than arriving with a pre-determined package of changes and activities. This flexibility allowed APROFAM and MSH to work in full and positive partnership on the difficulties faced in the process. At the outset of the process MSH had to take a strong lead and provide the impetus for institutional change. However, after about 1998, APROFAM and MSH have worked consensually and on a full partnership basis. 2. Joint Vision - MSH’s work with APROFAM to develop a specific vision of the final destination of the re-engineering process and the political will (of the Executive and Board of Directors) ensured that activities related to the change process would be carried out and monitored. 3. Quality Technical Assistance - MSH selection of technical consultants provided both the competence and confidence to APROFAM staff that the achievement of goals was possible, making the long hours and effort acceptable. It is important to note that MSH employed a “south-to-south” strategy, using Latin American experts to provide much of the TA. 4. Adequate Time Frame - The longitudinal time frame in the Cooperative Agreement was adequate to the goals of the project. 5. Willingness to Accept and Manage Risk - One serious risk was the probable rate of staff turnover in APROFAM that coincided with each department’s re-engineering process. That turnover reached 60% of the Medical Services Department at its peak and 58% the in Rural Development Program. In addition, it should be noted that 75% of senior staff have been appointed since 1998. 6. USAID/G’s Financial Assistance - This support was key in upgrading the physical infrastructure and equipment needed to engage successfully in the diversification processes. MSH’s work with APROFAM provides an excellent partnership model for positive and future-oriented institutional change leading towards financial and institutional stability. MSH provided fresh and appropriate ideas and concepts and coupled these with high quality technical expertise in generating new approaches. APROFAM generated internal will and competence to take advantage of a unique opportunity and make the most of it. MSH/APROFAM Evaluation Timothy Farrell, PhD 7 of 61 June 2003 Table of Contents Background..............................................................................................................................9 Methods, Activities and Assumptions....................................................................................10 Data Sources………………………………………………………………………………........12 APROFAM’s Target Population.........................................................................................12 Trends in Family Planning in Guatemala ..............................................................................12 MSH Support..........................................................................................................................16 The Present Phase............................................................................................................17 1. The CORE System........................................................................................................17 2. Strategic Planning and Re-engineering ........................................................................18 Results and Effects of Institutional Change..........................................................................21 Human Resource Turnover................................................................................................30 How Did Organizational Change and Sustainability Occur?.................................................32 External Factors – Institutional and Structural...................................................................32 External (MSH) Factors......................................................................................................33 Internal (APROFAM) ..........................................................................................................34 Remaining Sustainability Risks .............................................................................................35 1. Under-performing Clinics ...............................................................................................35 2. The Rural Development Program..................................................................................36 Some Recommendations for the RD Program ..............................................................36 3. Adolescent Reproductive Health Education..................................................................37 4. Protection against Domestic Violence...........................................................................37 APROFAM without Subsidies?..............................................................................................37 Conclusions............................................................................................................................38 APROFAM Change Model Re-Visited ...............................................................................40 MSH/APROFAM Evaluation Timothy Farrell, PhD 8 of 61 June 2003 Appendices Appendix 1 Interventions by MSH 1998 – 2001 Appendix 2 Proposed Scope of Work Appendix 3 Activities Conducted in the Development of the SOW Appendix 4 USAID/G Intermediate Results Appendix 5 APROFAM Staff Interviewed Appendix 6 CORE Index Pages Appendix 7 CORE Documents (Jutiapa) Appendix 8 Market Survey (External Pricing) – Jutiapa Appendix 9 Sustainability of Clinics and Services 1998 – 2003 Appendix 10 Laboratory Brochure and Price Lists (1993) Appendix 11 Organizational Chart 2003 Appendix 12 PowerPoint Photo Presentation Appendix E-1 On CD: Electronic Documents: Full CORE Details Jutiapa Introduction This evaluation was commissioned by Management Sciences for Health (MSH), Boston. The main purpose is to evaluate the effectiveness of the impact of sustained and integrated management development on improved service delivery, self-income generation and overall sustainability of the inputs and changes introduced to APROFAM (Asociación Pro￾Bienestar de la Familia), Guatemala. This report focuses on two main points: 1. Organizational Change - The incorporation of management inputs into APROFAM’s existing management structure; and 2. The empirical results of this instance of organizational change on: a. Management effectiveness in APROFAM b. Increase in APROFAM’s financial sustainability c. Increase in APROFAM’s production and delivery of services (clinical services) and products (family planning methods) The methodology used in the evaluation is a combination of qualitative interviews and observations, review of internal managerial documents, and analysis of quantitative financial and service documentation. The analytic approach used is a simple time-lagged model that expects results about one year after the introduction of an innovation or change in policy. The findings are strongly supportive of MSH’s efforts and APROFAM’s willingness and ability to manage the change process in a constructive and productive manner, which resulted in a significant degree of financial and institutional sustainability. Background APROFAM (Asociación Pro-Bienestar de la Familia) is a non-governmental organization (NGO) that provides a wide variety of Family Planning, Reproductive Health, Maternal and Child Health, Laboratory and other basic health services throughout Guatemala. It also provides a rural outreach program focused on Maternal and Child Health, FP and RH health education, and the distribution (sale) of family planning and other health products. APROFAM historically has accounted for a high percentage of modern contraceptive use in Guatemala and is the second-largest single provider. It only recently fell behind the Ministry of Health (MSPAS) which now provides contraceptives for free. APROFAM has a 35-year history as the principal Family Planning (FP) and Reproductive Health (RH) agency in Guatemala. While IPPF is the principal philosophical, technical, and financial supporter of APROFAM, it has received substantial financial support from USAID and lesser amounts from other donors. USAID requires APROFAM to use USAID funding to pursue certain USAID goals and Intermediate Results (IRs). MSH and APROFAM initiated the establishment of more general Maternal-Child and Family Health and comprehensive clinic services on a fee-for-service basis. This diversification is considered a pillar of MSH’s initial organizational development strategy for APROFAM in 1995 to move towards financial sustainability. Aside from providing a much-needed service in many parts of the country, such comprehensive services also are designed to cross￾subsidize unprofitable services and products. In addition, it strengthened its relationships MSH/APROFAM Evaluation Timothy Farrell, PhD 10 of 61 June 2003 with other NGOs providing educational services to both NGO staff and project participants. Such outreach efforts have established APROFAM as a primary source of FP and RH support for many NGOs. APROFAM has upgraded and diversified its clinical laboratory services to become one of the most modern clinical laboratories in Central America, 1 with computerized analysis capabilities providing accurate results in very short time. The fees for laboratory services are modest and are within the economic reach of the current target population in the urban environments. Laboratory services, along with product sales, represent the two largest sources of net income for the organization. Faced with a serious probable reduction in donor funds in the mid-1990s, as well as on￾going sustainability pressure from IPPF and USAID, APROFAM was forced to re-examine its strategies and practices. This confluence of factors led early on to an initiation of a program by MSH, supported by USAID, to establish a working relationship with APROFAM that would strengthen its management structure and performance-driven goals, and would lead to sustainability as well as support USAID/G Strategic Objectives. APROFAM, in coordination with MSH, has both anticipated and responded to these IRs by developing and implementing a series of initiatives designed to improve APROFAM’s managerial capacity to improve program performance and enhance the probability of achieving a significant level of institutional and financial sustainability. APROFAM today is run internally according to sound business principles and maintains its social service orientation of addressing the populations-in-need through cross-subsidization from its c linical revenues. Methods, Activities and Assumptions The evaluation was carried out from 29 April through 9 June 2003 and included field visits to 4 clinics, interviews with 2 Rural Development Coordinators, 3 Field Managers (Jefes de Campo), 4 Community Educators (Educadoras), and 8 Promoters. The bi-annual application of the CORE cost-management program was observed in one large clinic, Jutiapa. Interviews were also conducted with all senior staff at APROFAM. The initial evaluation model is shown below. The model assumes: 1. That inputs (MSH) are designed based on sound technical grounds and are relevant to and agreed upon by the organization (APROFAM) 2. That the process of implementation is internally managed by APROFAM 3. That outputs (results) in the form of sustainability and improved services and product sales and distribution by APROFAM are directly related to the inputs by MSH 1 Appendix 6. MSH/APROFAM Evaluation Timothy Farrell, PhD 11 of 61 June 2003 Figure 1 Initial Evaluation Model – MSH-APROFAM 1995-2003 The main intervening variables are: 1. Recognition of the need to achieve financial sustainability through change and growth by means of: a. a much more client focused approach based on sound market research and analysis b. improved and standardized quality c. improved and standardized efficiency d. management system’s support of the above 2. Internal managerial political will endorsed and supported by: a. The Board of Directors b. External funding and technical assistance 3. Internal (employee) participation and coordination The main output variables are metrically measured in terms of: 1. Service and clinic statistics 2. Financial results In addition, institutional change can be measured qualitatively in terms of “fit,” or how well the structure of the organization supports its goals and objectives. The MSH MOST instrument (Management and Organizational Sustainability Tool) was also used as a guide for this assessment, but a MOST exercise was not conducted with the staff. Rather, it was used as a guide for the semi-structured interviews conducted with APROFAM managers and helped structure the organization of the evaluation. MSH Interventions Implementation of Activities APROFAM Data And Results 1995 1998 Onwards 1996 1999 Onwards MSH/APROFAM Evaluation Timothy Farrell, PhD 12 of 61 June 2003 Data Sources All data requested was readily provided by APROFAM. The few exceptions were due to lack of a systematized reporting and archiving process prior to 1998. Human Resource files are difficult to recover for analysis since this department does not form part of the administrative system, SCORPIO. Basic information on staff turnover was obtained through the excellent assistance of the HR department. These data are stored in an off-site facility. Other reporting data are not systematic, are stored off-site, and would require a large cost and effort to recover. Financial data do go back, in most instances, to 1995-6 and are complete and readily accessible APROFAM’s Target Population It is important to keep in mind throughout this evaluation that APROFAM’s target population is primarily the lower and lower-middle classes in terms of services and products. In addition, its Rural Development Program, which sells contraceptive methods and related MCH products such as vitamins, is heavily focused on some of the poorest economic groups in the country, including the majority indigenous Mayan population in the highlands (the “North and North-Western” Regions in MOH classification). APROFAM’s FP competition is principally the public sector through the Ministry of Health and social insurance institute (IGSS),2 which accounted for 36.3% of all modern contraceptives provided in Guatemala in 2002. APROFAM alone accounted for 28.6% in the same year. The remaining 35% is provided by the rest of the private sector: private physicians, pharmacies (about 25%), and “other” sources (10.8%).3 Trends in Family Planning in Guatemala While FP is only one of the many components of the “new” APROFAM strategy, it remains a major component of the Rural Development Program (RDP). According to the APROFAM 2002 Strategic Plan the SWOT Analysis indicates that the RDP is still publicly perceived as primarily a FP organization. In addition, the RDP is often the only face of APROFAM in isolated rural areas, hence a review of the context in which APROFAM currently operates is relevant here. Further, one of the AID IRs states: “MSH will assist APROFAM in the design and implementation of culturally acceptable marketing and educational strategies to improve women’s and men’s knowledge of reproductive health and to ensure that increased knowledge translates into increased use of reproductive health services. Assistance will include training, supervision and motivation of health workers to participate in a wide range of outreach activities, with emphasis on regular home visits.” There has been an upward trend in family planning use in Guatemala since 1987 as measured by the same basic DHS-ENSMI (Encuesta Nacional de Salud Materno Infantil) methodology. Table 1 presents this data. 2 The Instituto Guatemalteco de Seguro Social (IGSS) is currently under investigation for a fraud of an estimated 136 million quetzales (roughly US$17.4 million) and is currently having difficulty supplying essential medicines to its clinics and facilities. 3 Encuesta Nacional de Salud Materno Infantil: 2002. Informe Resumido. Ministerio de Salud Pública y Asistencia Social, Instituto Nacional de Estadística, Universidad del Valle, et.al. Marzo 2003. MSH/APROFAM Evaluation Timothy Farrell, PhD 13 of 61 June 2003 Table 1 Trends in Contraceptive Use 1987 - 2002 Percent Users 1987 1995 1998-99 2002 By Regions Any Method Any Method Any Modern Any Modern North West 6.5 11.8 13.9 10.9 27.3 20.2 North 10.6 13.7 20.8 18.8 32.3 22.6 South East 23.3 29.1 32.9 22.8 32.9 27.0 South West 14.8 23.1 30.4 28.1 35.7 27.6 North East 28.3 25.7 31.5 23.3 41.7 33.1 Guatemala - All 23.2 31.4 38.2 30.9 43.3 34.4 Central 21.9 32.8 39.1 31.6 48.0 40.7 Metropolitan 45.0 50.1 57.2 47.0 60.0 48.3 As can be noted both in Table 1 and in both Graphs 1 and 2, the two highly indigenous highland areas lag considerably behind the rest of the country with respect to both traditional and modern contraceptive use. Graph 1 Any Contraceptive Use by Region 1987 – 2002 0 10 20 30 40 50 60 70 Guatemala - All Metropolitan North North West North East Central South West South East 1987 1995 '98-'99 Percent 2002 MSH/APROFAM Evaluation Timothy Farrell, PhD 14 of 61 June 2003 Graph 2 Modern Contraceptive Use % by Region 1998/9 - 20024 0 10 20 30 40 50 60 Guatemala - All Metropolitan North North West North East Central South West South East 1998-9 2002 Table 2 presents data that clearly distinguishes the Mayan and Ladino populations along ethnic and residential lines. Table 2 Differences in Contraceptive Use by Ethnicity & Residence 1987 – 2002 Percent Users 1987 1995 1998-99 2002 By Ethnicity & Urban/Rural Any Method Any Method Any Modern Any Modern Indigenous 5.5 9.6 12.9 8.4 23.8 16.6 Rural 13.8 19.8 27.7 21.5 34.7 26.2 Guatemala - All 23.2 31.4 38.2 30.9 43.3 34.4 Ladino 34.0 43.3 49.9 41.3 52.8 43.2 Urban 43.0 48.9 52.3 43.8 56.7 48.3 The maternal and child health, population, and economic consequences of this slow pace contribute even more to the poverty burden felt by the indigenous population. The total fertility rates for the indigenous and rural populations are considered in Table 3 and Graph 3. 4 Data is not available for previous years although USAID data show a modern contraceptive prevalence of 19% for the entire country for 1987. Percent MSH/APROFAM Evaluation Timothy Farrell, PhD 15 of 61 June 2003 Table 3 Total Fertility Rate by Regions 1995 - 2002 Total Fertility Rates 1995 1998-99 2002 By Regions North 6.7 5.5 6.5 North West 6.8 6.2 5.5 South East 5.7 5.1 5.0 North East 5.1 5.4 4.7 South West 5.5 5.3 4.4 Guatemala - All 5.1 5.0 4.4 Central 5.3 5.0 4.2 Metropolitan 3.9 4.3 3.2 As can be seen in Graph 3, TFR for the indigenous areas of the country have remained higher than they have for all other areas of the country. In fact, in the “North” TFR has actually risen from 5.6 to 6.5 over the last 3 years remaining about the same as the 1995 measure. Graph 3 TFR by Region for 1995 – 2002 0 1 2 3 4 5 6 7 8 North North West South East North East South West Guatemala - All Central Metropolitan 1995 1998-99 2002 Percent MSH/APROFAM Evaluation Timothy Farrell, PhD 16 of 61 June 2003 In short, APROFAM exists in, and significantly contributes to, an environment of positive change in family planning. Indeed, it is a pioneer for modern family planning in Guatemala. Nevertheless, this advance has been slow relative to its Central American neighbors and even more so in Guatemala’s indigenous regions (Bertrand, et. al., 2001). 5 Part of this slow pace is attributed to several factors, including an inconsistent government approach for over 30 years, limited access to services (financial and geographic), education and literacy of women, and certain cultural and religious traditions, especially that of machismo. In 2001 the Ministry of Health began providing contraceptive methods free of charge at its clinics nationwide and included distribution channels through its SIAS program (Integrated System of Health Services). This has had two effects according to APROFAM officials. The first is a greater distribution of contraceptives throughout the country through increased access. The second is a greater degree of competition for market share, which has decreased APROFAM’s earlier estimated total market share of 37% for contraceptive methods by about 10% over the last 3 years. In addition to Family Planning and Reproductive Health, one of the Intermediate Results (IR-1) provides that: MSH will assist APROFAM in the development, promotion and implementation of strategies to increase coverage and improve the quality of its health services (clinics and health promoters), especially in the highlands of Guatemala. The strategy designed by MSH to achieve this included the integration of a comprehensive MSH program with other clinical and laboratory services, not only in the highlands, but also throughout the entire country. Today APROFAM provides direct and referral services throughout the country in 32 clinic settings. MSH Support6 This section describes and analyzes MSH’s major inputs and their contribution to APROFAM’s development and its present sustainability status. Contrary to the model developed (Figure 1), the results of MSH and APROFAM work are rolling and cumulative rather than directly tied to specific outputs. Because of this, while we can track the time of the initial efforts of MSH and APROFAM (see Appendix 1), there is no measurable direct relationship with the timing of the results. Consequently, MSH assistance is best viewed as a whole entity comprised of essential components bound together by APROFAM management. It is also analytically relevant and convenient for this evaluation to distinguish the (1995– 1998) period from “later” (1999–2002) phases of MSH support. This is due to the intensity 5 Bertrand, Jane T., et. al. “Contraceptive Dynamics in Guatemala 1978-1998,” International Family Planning Perspectives, Volume 27, Number 3, September 2001. 6 There are several data gaps in the “early” phase of MSH support. Much of the reporting data from this period are apparently unsystematic and stored off-site; financial data are an exception to this. Unfortunately, direct information on family planning and other service statistics information are not retrievable without a major investment of staff and time and this data would not enhance this evaluation significantly. Additionally, only one senior management member has been with APROFAM in an executive position since 1998, making interviews regarding management, systems, and the implementation processes difficult. MSH/APROFAM Evaluation Timothy Farrell, PhD 17 of 61 June 2003 and the kinds of changes introduced and implemented. It is also analytically important to separate the two periods because beginning in 1998 there were major management changes within APROFAM itself, including the hiring of a new Executive Director in April 1998. This coincided with the final (no-cost extension) period of the “early” MSH Cooperative Agreement 7 from January through March of 1998. The Primary Phase The first or primary phase of MSH assistance to APROFAM laid out the framework, vision, and direction for the evolution of APROFAM from principally a FP service provider to a highly diversified family health organization. MSH has provided APROFAM with managerial assistance since 1995. During this primary period, MSH’s main assistance and support consisted of working towards the diversification and financial sustainability of clinic activities in urban sites throughout the country. Important innovations occurred during this period which included: · The initial development and utilization of the CORE (Cost and Revenue Analysis Tool) System (below and appendices 2 and 3 for details) · MSH development of a USAID investment project to buy/build/remodel clinics, purchase equipment, and do initial marketing in support of diversification of services · An analysis and re-design of client-flow in clinics · Analysis and improved utilization of clinic physical space, including reconstruction and repair of facilities, and purchase of new equipment · Development and implementation of training plans related to diversification, leading to the adoption of a new corporate culture over time · Development and application of protocols for a national marketing system oriented towards developing a competitive niche and sustainability · Development of business plans for local clinic facilities Part of these innovations and changes included the development and implementation of the CORE cost and revenue analysis system which has become a major component of APROFAM’s current managerial system. The Present Phase This diversification with the addition of a variety of services up to and including hospital￾level care in the larger clinics has become the major source of sustainable growth and revenue for cross-subsidization of the rural development program. These fundamental changes provided the initial substance and structure for important changes in other systems into the present. 1. The CORE System8 The Cost and Revenue Analysis Tool was introduced by MSH in coordination with APROFAM’s clinical services expansion and diversification efforts. The basic system is written in Microsoft Excel and provides service unit cost and revenue data by multiple variables at the clinic level. An example of the CORE summary sheets is included in the Appendices. 7 Cooperative Agreement Number 520-0357-A-5123-00) 8 CORE (Cost and Revenue Analysis Tool) was introduced in this “early” phase and its current version dates to 1999. MSH/APROFAM Evaluation Timothy Farrell, PhD 18 of 61 June 2003 This summary (Resumen de Información Clave) is the product of another 66 worksheets and graphics designed by APROFAM on their own initiative to “feed” the main summary. A complete copy of the CORE for the large clinic in Jutiapa is included as Electronic Appendix E-1. These worksheets are initially fed by information from the SCORPIO system which handles all administrative and inventory tasks with the exception of Human Resources. 9 How CORE is applied Each clinic, regardless of size classification, is reviewed twice annually using the CORE as its basic instrument. The information is complied at the central offices by the Finance Department and a summary is sent to the clinic for review. A team from the central office then visits the clinic for a discussion of the findings and to determine what steps, if any, are necessary to address issues raised. Supplementing the information are current findings by the Marketing Department that include findings on service quality and costs of products, medicines, and services in the local market (Appendix 3). This information includes contraceptives, medicines used by APROFAM vs. pharmacy prices, laboratory tests, medical services and so forth. Virtually everything that the local clinic does or provides is regularly tested against the local market. 2. Strategic Planning and Re-engineering10 APROFAM and MSH began the task of Strategic Planning and Re-Engineering in 1997-8. These two interventions are necessarily intertwined in time, activities and effects. In 1998 APROFAM held its first significant Strategic Planning exercise, an event that is now conducted bi-annually by its management group supplemented by members of the Board of Directors. The most recent review was done in February 2002. The re-engineering process focused on decentralization, diversification, and integration of clinics in order to improve efficiency, productivity, and quality of management and services, including both internal and external clients. MSH re-worked or developed new policies, procedures, manuals and instruments that defined the process. The effectiveness of these products and their processes impacted the entire organization. The following paragraphs highlight the main elements of structural and management changes made in APROFAM with the support of MSH. Administration – This department made major changes, most notably for sustainability purposes in its management of logistics (inventory management), purchasing, and information systems organization. It also gained major efficiencies through the out-sourcing of building services and maintenance, the delivery of supplies and medicines through commercial contracts (thus eliminating vehicle and maintenance costs,) the negotiation of annual purchasing of major supplies and materials, and the development and installation of the SCORPIO information system which integrates and automates all administrative activities (except Human Resources). Finance - This department has been re-organized to recognize separate departments under its management - Accounting and Treasury. In addition, it houses the CORE unit which is responsible for monitoring all clinic activities (except medical and technical quality) twice per, year per clinic. 9 See HR section. 10 A year-by-year list of MSH inputs and results by IRs is attached as Appendix 1. MSH/APROFAM Evaluation Timothy Farrell, PhD 19 of 61 June 2003 These separate functions permit the effective management of all cash and assets of the organization as well as tracking clinic performance over time. Such systems provide other departments with invaluable insight and information for operational decision-making in coordination with even the most isolated clinics. Efficient systems have been developed to manage all cash transfers to and from all clinics and units through a series of nation-wide bank accounts. Far from being an isolated and esoteric part of the organization, Finance is integrally involved with all other aspects of APROFAM. Budgeting and budget planning is well integrated with other departments, thus providing both appropriate support and direction in the strategic planning process. Human Resources - This team has professionalized its activities significantly. With the assistance of MSH, it has developed a new salary structure based on a survey of similar agencies in Guatemala, personnel handbooks and manuals, and an annual evaluation system. In addition, MSH participated in the development of a salary incentive system. This was not implemented, but did serve as a basis for the development of a new reward system called “variable compensation” that is now being used and evaluated. Marketing – APROFAM has developed a true social marketing department based on commercial marketing principles and addressing issues of quality of service, competitive pricing of products and services, and new markets. Social marketing is a major part of APROFAM’s long-term strategy through the year 2010. The importance of the APROFAM trademark was stressed, so that today, a majority of individuals sampled recognize APROFAM as a Family Planning and Health organization. A major effort and expense has been made (and is ongoing) to establish a uniform physical appearance in terms of physical recognition in colors schemes, signage, and so forth, stressing image and recognition. This is complemented with regular and serious studies of client service perceptions that include such variables as waiting time, courtesy of employees, etc. According to key interviews with senior staff, there was little, if any social marketing conducted prior to re-engineering. What little IE&C there was, was limited in scope and apparently ineffective. It had been basically dismissed out-of-hand as an integral part of APROFAM by informants. Today, on the other hand, Marketing stands out as a major component of APROFAM’s Vision for achievement by 2010: “By 2010 to be a self-sustaining organization, a leader in the health services oriented to client satisfaction, through a successful [strategy] of social marketing.” (Strategic Plan 2002) Medical Services Department - This flagship of APROFAM services has undergone significant change in all aspects, beginning with the diversification and integration of services in all of its (now) 22 departmental clinics and 13 metropolitan (Guatemala City) clinics. These changes include the classification of clinics into three categories based on the level of services provided and the size of the clinics and population directly served. Both the technical and service quality of each clinic is closely monitored. Technical quality is MSH/APROFAM Evaluation Timothy Farrell, PhD 20 of 61 June 2003 reviewed through an internal procedure of checklists, while service quality (attention to the patient) is monitored through various instruments undertaken by the Marketing Department. In order to increase financial sustainability and to better serve clients, a variety of services including specialists, advanced laboratory technology and hospitalization services (at the largest sites) have been developed. This has required close coordination with all other departments, but especially with Marketing since APROFAM tries to exploit a niche between the public and private (for profit) sectors. As a consequence, their prices need to be especially sensitive to price movement in the private sector; and their service quality needs to far exceed the public sector and remain competitive with the private sector. The re-engineering and structural changes in the department have been so successful that the manager of the department no longer refers to the “sustainability” of the department, but to its “profitability” (rentabilidad in Spanish). The net result of these changes, coupled with the other organizational changes in APROFAM, is that this Department is now essentially financially sustainable and potentially capable of cross-subsidizing other less-sustainable activities in the enterprise. A newly re-engineered unit brought about through MSH guidance is the Planning, Monitoring and Evaluation Department that reports directly to the Executive Director and is responsible for: · Supporting institutional development by coordinating and integrating planning throughout the organization · Providing timely information to users for decision-making · Supporting the Executive Director in monitoring and evaluating strategic and operational plans The Rural Development Department- This department is responsible for family planning and RH services at the community level. It has undergone, and continues to undergo significant structural and organizational changes.11 This is probably the most managerially complicated of all of APROFAM’s activities. At present, there are three central office “coordinators,” 6 field managers (Jefes de Campo), about 90 educators, and over 3,500 volunteer promoters. Random field visits with promoters, educators and managers revealed that these activities are well supervised and supported by APROFAM. Each promoter visited had the recommended supply of contraceptive methods available and displayed in a secure location. Furthermore, each promoter had illustrated manuals on human reproductive systems, as well as other relevant materials, including information on HIV/AIDS. All were knowledgeable about FP methods, possible side effects and when to refer clients to a higher level of medical attention. In addition, each had some sort of main business that earned income and attracted potential FP clients. These included perfume and cosmetics sales, clothing shops, traditional birth attendants (comadronas), trained health workers (auxiliary nurses, child survival workers), shopkeepers, etc. In one case, the municipal treasurer was also a promoter. Nearly all promoters had an APROFAM sign on their home or place of business. 11 This is a work in progress. MSH continues to work with the Rural Development Department and information is not available regarding what changes might be recommended or implemented. MSH/APROFAM Evaluation Timothy Farrell, PhD 21 of 61 June 2003 The RDP faces the greatest competition from other sources of contraceptive methods: the Ministry of Health, other NGOs, pharmacies and other sources. Further, as USAID reduces its allocation of FP methods, APROFAM will need to purchase products from the commercial market, creating an even greater burden for this department. As noted, the entrance by the MOH into the FP market occurred in 2001 and this has reduced APROFAM’s market share for temporary methods such as Depo-Provera, all pills, and condoms. In addition, APROFAM provides some NGOs and the MOH with the following methods: NGOs MOH Depo-Provera Depo-Provera Lofemenol Lofemenol Copper-T Copper T Condoms Condoms Conceptol Ovulo Clearly, the provision of FP methods by APROFAM, especially when they are free through the MOH, will have a serious effect on sales of the same products through APROFAM’s RD promoters. MSH initiated work with the Rural Development Program in 1998 and jointly developed supervisory manuals and checklists for all activities from the central office to the community volunteer promoters. The Rural Development Program probably will not reach full sustainability in its present form. This is recognized by APROFAM and was also pointed out in an evaluation of the extension of the Program by Reynolds (2001). He notes: “APROFAM cannot afford to expand its current rural program without considerable donor support. It should calculate the level of rural coverage it will be able [to] support with and without donor contributions and adjust its expansion targets accordingly.” (p1) MSH began its work with RDP in 1998 and is currently working with the rural development program in an on-going effort towards re-organization and strategy definition. At present, the generally accepted goal is to achieve basic cost-recovery. All senior management interviewed mentioned this goal and all agreed that the distribution of contraceptive methods through the Community Volunteer Promoter Strategy aligns with the organization’s mission of providing services for “...the Guatemalan family especially those with scarce resources.” Results and Effects of Institutional Change The results of MSH activities with APROFAM are institutional and sustainable. Already, APROFAM has moved from a traditional NGO highly dependent on donor contributions to a viable enterprise in its own right. The process undertaken with MSH assistance is maintained through a new culture in the organization that understands and has internalized those processes, so that they are the main tools of management. MSH/APROFAM Evaluation Timothy Farrell, PhD 22 of 61 June 2003 The cumulative effect of the changes and processes has resulted in “systems thinking” at APROFAM. It is now, after several years of changes and operations, very difficult to isolate one change from another since they form a management system, rather than isolated components of single departments. This is readily apparent from the cross-functional committees and groups that are involved in decisions that will have multiple effects throughout the organization. If, for example, marketing discovers price inefficiency in a product at a particular clinic, this is reviewed by finance and the clinic staff before changes are made. Any change in the price of a product is then followed up at three-month intervals to see if it has had any negative effects on sales or other activities at the clinic level. A translation from the Marketing Manual reflects this systems concept well: We are in the business of ‘family well-being’ with an orientation ‘based on the client.’ This means a reorientation of focus, structure and function…that contributes effectively to maximum client satisfaction in terms of value, quality, and time, for a cost that the client can and wants to pay. This systems orientation is shown on the following page. The illustration describes how the functional areas are all linked. While this is from the marketing perspective, it accurately describes the inter-relatedness of all functional and management areas. One singularly notable observation I have made during this evaluation is the degree to which corporate policy is shared at all levels. Normally in NGOs it is usual to find major discrepancies between stated policy in manuals and actual operating procedure. In APROFAM’s case, however, there is very little if any discrepancy between the stated policies and the policies observed on the ground. The systems have been designed and/or modified to address the flexibilities needed at various operational levels. The changes in systems, policies and approach have yielded demonstrable effects in all measures of growth and sustainability with the exception of the rural development, adolescent, and violence against women programs. All these will need some form of external donor support and/or cross-subsidy from the Medical Services Department in the foreseeable future. MSH/APROFAM Evaluation Timothy Farrell, PhD 23 of 61 June 2003 MSH/APROFAM Evaluation Timothy Farrell, PhD 24 of 61 June 2003 Product and Services Growth 1998 – 2002 APROFAM’s growth in revenue over the past four years has been exceptionally good (Graph 4). This is especially true when one considers the recent competition stemming from free methods and services offered by the Ministry of Health. The resulting decline, shown in Graph 5, has been more than compensated for by the increase in other services (Graph 6). Much of the credit for this growth, in spite of the decrease in family planning components, is due to the strong performance of the overall institutional growth. This is due to the diversification and expansion of clinic services provided, the depth and breadth of the marketing efforts developed by MSH in 1996-7, and their ongoing application by APROFAM. Graph 5, contrasted with Graphs 4 and 6, amply demonstrates the effects of these efforts. Graph 4 Growth of All APROFAM Services 1998 - 2002 All Services 0 200000 400000 600000 800000 1000000 1200000 1998 1999 2000 2001 2002 Graph 5 Effects of Competition from MOH’s Entry into the Contraceptive Market in 1991 180000 185000 190000 195000 200000 205000 210000 215000 1998 1999 2000 2001 2002 Total No. Contraceptive Units Units of Service MSH/APROFAM Evaluation Timothy Farrell, PhD 25 of 61 June 2003 Graph 6 Proportion of Family Planning to All APROFAM Services 1998 - 2002 Proportion FP to All Services 0 0.1 0.2 0.3 0.4 1998 1999 2000 2001 2002 One of the major challenges that APROFAM and Guatemala as a whole are facing remains the low volume of contraceptive acceptance and use among the Mayan populations. This challenge is one of the major objectives of MSH support (IR-1, Objective 1). This is recognized in the application for extension and its review by Reynolds (2001) as well as in MSH Report for June 30 – December 2001: “Remaining Challenges: First and foremost, [APROFAM] must face the complete restructuring of its very large and important Rural Development Program which has been almost entirely subsidized by a series of Cooperative Agreements with USAID/G-CAP…That program must confront strong, new competition from both the government and other NGOs also financed by USAID/G-CAP…” The following two graphs illustrate 12-month frequencies for both new and continuing users in various Mayan environments. As can be clearly seen, the distribution of both new and continuing users is significantly lower in the more concentrated Mayan departments of Sololá, Quiché and Huehuetenango. These data reflect national level data for the same regions12 and should not be directly attributed to some shortcoming on the part of APROFAM or MSH. In fact, the largest of APROFAM’s Departmental Clinics is in Quetzaltenango (Xela), in the very heart of the north and northwest areas. In eight visits to these areas, in all cases I found the rural volunteer promoters to be active, very knowledgeable, and to have the appropriate supply of contraceptive and other products available. They also had manuals on reproductive health with pictures appropriate to the cultural area. Promoters said that the manuals were very helpful in explaining the reproductive process and how the various contraceptives worked. 12 This data corresponds to that presented earlier in the latest ENSMI-based Graphs 1 and 2. MSH/APROFAM Evaluation Timothy Farrell, PhD 26 of 61 June 2003 Additionally, of the promoters visited, two are certified midwives, two are auxiliary nurses, and another is a volunteer health worker who was trained by both CARE and the Christian Children’s Fund (CCF). Graph 7 First Time FP Users in Rural – Principally Mayan Departments 12 Months - 2000 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 11 12 Xela Huehue Solola Quiche Chimalt Coban In discussions with APROFAM promoters in the area, there are a number of factors that militate against adoption of FP. As cited by the promoters, these are: religious beliefs (among both evangelicals and Catholics), peer-pressure against utilization (many women will use the services of the FP promoter but will not take advantage of free clinic resources,) “cultural” beliefs (“that which God sends”), and opposition from spouses (machismo). A new rationale has also arisen from the high degree of migration to the US. Migrants are liable to be away for many months, if not years. They do not want their wives using contraceptives during their absence in order to assure their fidelity. Among these are the same general factors that are available in much of the FP literature regarding Guatemala (Bertrand, 2001, op.cit.). These include education, access to information via the media, and working outside the home. Number Of First Time FP Users Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec MSH/APROFAM Evaluation Timothy Farrell, PhD 27 of 61 June 2003 Graph 8 Family Planning Continuation Users in Rural – Principally Mayan Departments 12-Month Period - 2000 Consequently, while the Rural Development Program is neither self-sufficient, nor attracting a significant number of new users, it plays a major role in APROFAM’s integrated strategy. As noted, MSH began working with this department in 1998 and has brought about fundamental changes in the logistical system, manuals, and routing of home visits. This work is continuing into the present, with a focus on a major re-engineering or restructuring of the department. Financial Sustainability APROFAM’s financial sustainability has increased significantly over the period 1998-2002. There has been a marked decrease in the percentage of non-sustainable clinics in both the metropolitan areas of Guatemala City (from 45% to 15%) as well as in the interior or Department Clinics (from 78% to about 22%). Graphs 9 and 10 illustrate these improvements. 0 50 100 150 200 250 300 350 400 450 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC Xela Huehue Sololá Quiche Chimalt Coban Sololá, Huehuetenango, and Quiche have the highest overall Mayan Populations Number Of Continuing FP Users MSH/APROFAM Evaluation Timothy Farrell, PhD 28 of 61 June 2003 Graph 9 Decrease in the Percentage of Non-Sustainable Clinics at the Metropolitan Level 1998 – 2002 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 1998 1999 2000 2001 2002 Graph 10 Decrease in the Percentage of Non-Sustainable Clinics at the Department Level 1998 – 2002 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 1998 1999 2000 2001 2002 The net result in sustainability reflected in financial terms is that all clinics together have become self-sustaining, reaching a level of 122% sustainability at the end of 2002. This means that they produce financial results of 22% above their total costs (Graph 11). Consequently, the degree of sustainability of APROFAM has risen over the same period from about 58% to 81% (Graph 12). This has been a very linear process with only slight Percent Percent MSH/APROFAM Evaluation Timothy Farrell, PhD 29 of 61 June 2003 slippage in the period 1999-2000. This corresponds to a significant increase in payroll￾based13 staff in the same period, as can be seen in Graph 14 when staff increased from about 435 to 515 (about +16%) in 1998-1999. Graph 11 Percent Financial Sustainability at Clinic Levels 1996 - 2002 0.7 0.8 0.9 1 1.1 1.2 1.3 1996 1997 1998 1999 2000 2001 2002 Graph 12 Percent of Institutional Sustainability 1996 – 2002 0 0.2 0.4 0.6 0.8 1 1996 1997 1998 1999 2000 2001 2002 13 Payroll-based staff are regular employees of APROFAM. This excludes a number of professionals who work for the organization on a professional contract basis. Nor does it include its over 3500 promoters. APROFAM has no responsibility for employee benefits to contract personnel, while payroll-based staff have a full range of social and financial benefits, including severance pay, vacations, social security, etc. Percent Percent MSH/APROFAM Evaluation Timothy Farrell, PhD 30 of 61 June 2003 Overall, gross sales of products and services have increased by about 57% during this period with the lion’s share coming from the expansion of clinical services, and in spite of the reduction in the sales of contraceptives. Graph 13 Sales of Services and Products 1998 – 2002 $- $2,000,000 $4,000,000 $6,000,000 $8,000,000 1999 2000 2001 2002 Services Products Totals Human Resource Turnover Change and growth have not been without its costs, however, especially in terms of human resources. While interviews with all levels of staff indicate that they were both aware of the coming changes, and to varying degrees involved in them, there nevertheless has been a significant turnover during this period. As can be seen in Graph 14, the organization had a 15% growth in staff over the two-year period 1998-1999. The cost of new hiring, training and payment of benefits to those departing negatively affects any organization’s ‘bottom￾line.’ This can be seen in terms of the ‘slippage’ in the financial sustainability graph in 2000 (Graph 16). Graph 14 Total Payroll-Based Staff 1998-2002 Total Payroll 1998-2002 380 400 420 440 460 480 500 520 1998 1999 2000 2001 2002 Employees MSH/APROFAM Evaluation Timothy Farrell, PhD 31 of 61 June 2003 The overall turnover rate for the past 5 years is slightly over 30%. Graph 15. Annual Staff Turnover Rate 1998 – 2002 (Percent of All Staff) 0 10 20 30 40 1998 1999 2000 2001 2002 The impact is demonstrated in the Medical Services Department where there was about a 60% turnover rate in 1998. This coincides with the re-engineering process in that department. Subsequently, the turnover rate moved from 20% to 58% in the Rural Development Department during its initial re-engineering efforts. 14 Graph 16 Personnel Turnover Percent and Institutional Sustainability 1998 – 2002 0 0.2 0.4 0.6 0.8 1 1996 1997 1998 1999 2000 2001 2002 % Sust. Instit. Rotacion Des Rural Rotacion Dep. Med. 14 These two departments are the largest in the organization. Percent Percent MSH/APROFAM Evaluation Timothy Farrell, PhD 32 of 61 June 2003 Reflecting the depth and breadth of the staff changes is the fact that 75% of the current senior staff has been appointed to their positions since 1998. Only 3 staff members held senior positions prior to the re-engineering process. Graph 17 Executive Positions “In Place” by Year of Contract or Promotion 75% of Executives have been appointed since 1998 No. of New Executives 0 1 2 3 4 1988 1997 1998 1999 2000 2001 2002 Certainly the turnover in and of itself is not necessarily a negative issue. It must be recalled that when MSH began this work with APROFAM the organization had only had one Executive Director since its founding in 1964. In APROFAM, as in most organizations, this often leads to stagnation and entrenchment of interests. In general, the turnover probably has had more positive than negative effects on APROFAM. Clearly, the new executive staff, coupled with the dynamic and responsive systems, has had a profound positive effect on the sustainability and culture of the organization. How Did Organizational Change and Sustainability Occur? APROFAM existed for over thirty years with only rare changes in management.15 Individuals and departments were entrenched, and it was established as essentially the “only player” in Family Planning in Guatemala. It had no substantial competition in the country. Its main source of financing, USAID, had little choice in selecting alternative organizations to implement its population policies. All of these elements, plus the Guatemalan government’s ambiguous position on population, resulted in an organization bound by its own inertia. External Factors – Institutional and Structural IPPF, APROFAM’s principal intellectual and policy patron, initiated strategic planning changes in the early 1990’s including issues relating to resource diversification and sustainability.16 This, coupled with USAID’s own re-engineering, results-driven framework, and spending reductions for family planning, led to efforts to establish sustainable IPPF affiliates. 15 APROFAM has only had 3 Executive Directors in its history, with one individual serving as Director for over 30 years (1965-1996). 16 “Implementing the Vision 2000 Strategic Plan: A Compendium of Activities”, IPPF, London, 1999. Goal 3, Objectives 1-4, p 57ff. Executive Positions MSH/APROFAM Evaluation Timothy Farrell, PhD 33 of 61 June 2003 One major factor in the design, adoption, implementation and institutionalization of the changes brought about by the MSH intervention was APROFAM’s longitudinal nature. While described as “urgent,” there was no underestimating of the complexity of the task or the organization. As noted in the MSH December 2001 report, “While the initial Cooperative Agreement with MSH [1996-1998] addressed the urgent issue of making the APROFAM network of 32 urban clinics self-financing to allow USAID/G-CAP to remove their long-term subsidization, the current agreement had a much broader focus. Its mandate was to systematically build a modern management structure which would ensure the long-term sustainability of the organization. This was an enormous and urgent task given the outdated and/or non-existent state of many of the key managements systems and procedures necessary for an organization of the size and complexity of APROFAM.” 17 (Emphasis added). External (MSH) Factors 1. Collaborative and “listening” approach by MSH “[MSH] was very helpful in the entire process. They approached us from a perspective of wanting to know what we wanted to change and how we wanted to change it. They always consulted us throughout the process, and made sure we were involved in all decision￾making. They made sure that our goals coincided.”18 From various conversations with the Executive Director, it was clear that at the end of the day APROFAM controlled both the content and the rate of change. 2. Willingness to negotiate scope and pace of change within general limits Several department managers repeated the experience of the Executive Director’s comments above. Because of the complexity involved in the reorganization of the Rural Development Department, the activities relating to this have been postponed to the present Cooperative Agreement. Similarly, development of computer software for Human Resources has been postponed, as has the implementation of a new salary policy (moving from a bonus-based plan to a variable plan). 3. High level of technical expertise Both the representatives of MSH and the local technical people that were involved were said to have good technical skills. One problem was that one of the software providers did not have sufficient personnel to provide the on-going support required for the SCORPIO system. This has been resolved. 4. Adequate level of financing It appears that no activities (except a trip to the Dominican Republic) were cancelled during the period reviewed. Budget amendments were submitted appropriately. No comments were received regarding financial obstacles. 17 MSH Semi-Annual Report June 30 – Dec. 31, 2001, Executive Summary, p. 2. 18 Paraphrased from an interview with Dra. Telma Morales, Executive Director. MSH/APROFAM Evaluation Timothy Farrell, PhD 34 of 61 June 2003 5. Involvement of qualified local and Latin American individuals and firms in the process 6. Building on technical assistance from other international organizations involved in population issues. 19 · Population Council · IPPF · John Snow, Inc. · JOICFP (Japan) · The Futures Group, Inc. · Family Health International Internal (APROFAM) Factors 1. Strong political will at the level of the Executive and the Board of Directors Organizational change carries risk in any organization, and APROFAM is no different. The risks included internal resistance, personnel turnover, financial risk in terms of logistical disruption, client loss, possible legal risks in terms of labor issues, supplier loss, image change, and the need for additional investment (in new systems and physical plant, etc.) This required a comprehensive understanding of: · The extent and causes of the problems and issues to be addressed · A clear idea of what the end results should look like · A will to support and lend credibility to the process The Board and the Executive Director assumed responsibility for the risks involved, supported the changes, and supported the process in light of their view of the potential results. 2. Involvement of Staff at all Levels The rationale for change, its desired outcome, and its internal process were communicated effectively throughout the organization. Working committees were involved in analyzing the innovations and developing the ways and means for implementation. 3. Control of the Internal Process APROFAM was able to control the internal process of change insofar as its pace and direction were concerned. Management was able to negotiate with MSH and USAID as well as with APROFAM staff as to how the process would proceed. Management was also able to assign priority to normal operations while absorbing the changes in process. 19 While these organizations’ technical assistance is listed under MSH, they also are listed under APROFAM, reflecting the organizations’ ability to manage multiple sources of technical support while continuing to operate a rapidly changing and managerially sophisticated core organization. MSH/APROFAM Evaluation Timothy Farrell, PhD 35 of 61 June 2003 4. APROFAM either had or found the technical expertise and management skills to understand and manage the change As noted, 75% of all senior positions were appointed after 1998, and this links back to Internal Factor 1, the political will of the Executive and Board of Directors. In addition, redundant departments and sections (“Development”) were eliminated. Significant Issue for Future Activities Despite the scope and complexity of the re-organization undertaken by APROFAM, there appears to be only one major miscalculation. Regardless of the availability of information on the change process and the involvement of staff, insufficient attention was paid to potential staff turnover. As can be noted from Graphs 15–17, this first struck the Medical Services Department and continued into the Rural Development Department. The high turnover experienced in the Medical Department (nearly 60%) should have been seen by managers as a serious alert to action. The events that occurred in Rural Development may have been minimized had this been addressed. Staff turnover can be viewed both as a boon and as a serious cost. It can be a boon because it casts off staff that either cannot or will not perform under the new systems and standards. It can be costly because it requires significant time to recruit and train new staff. Aligned with this challenge was the failure of MSH to anticipate this in its planning and re￾organization of the Human Resources Department (Administration). HR is still not computerized and data has to be obtained either from “dead files” or payroll. Had this data been immediately available for monitoring, steps conceivably could have been taken earlier to plan for high turnover rates. From the data, it appears that the main response was to hire more people immediately to make up for the resignations or dismissals (Graph 14). Remaining Sustainability Risks Four APROFAM operations remain at risk: underperforming clinics, the Rural Development Program, adolescent programs, and victims of domestic violence. 1. Under-performing Clinics While the sustainability of the Medical Services Department and its service delivery is high (120% overall), there remain pockets of presently unsustainable clinics and activities and programs in APROFAM that require attention. At least two clinics (one in the metropolitan area – Zone 7, the other in Ixcan) are presently unsustainable and likely to remain so. The Zone 7 (minimal services) clinic is poorly located in a high-crime area and has run a deficit since 1998. The Ixcan Clinic has also run deficits since its initiation in 2001, and is currently (as of March 2003) at 21% sustainability. Ixcan is located in the midst of the Guatemalan civil war resettlement area,20 the residents of which are very poor and have little employment opportunity. It is doubtful that this population base can support a clinic to any level of self-sustainability in the foreseeable future. 20 Ixcan is nominally part of the Department of Quiché, however it is treated as a separate unit because of its location and special population. This population is extremely heterogeneous as no single indigenous group predominates. Most residents are former civil war refugees who had sought refuge in Mexico or had fled from the conflict zones, lost land or homes, etc. MSH/APROFAM Evaluation Timothy Farrell, PhD 36 of 61 June 2003 The options for these two clinics are substantially different. The clinic in Zone 7 can be re￾located or even closed without hardship to clients who are within easy bus distance of APROFAM’s central clinic complex in Zone 1. Ixcan, on the other hand, faces the dilemma of service reduction for very low-income clients. This challenges APROFAM’s mission to provide integrated health services “especially to those with scarce resources.” 2. The Rural Development Program Due to its nature and target population (the rural poor), the Rural Development Program is probably not going to be sustainable in the near future. The economic conditions under which these populations live are such that many are barely living at the subsistence level. As we have seen in Graphs 7 and 8, the populations least likely to use family planning are the indigenous groups in the north and northwest. Aside from a variety of reasons for non￾acceptance, the price of FP commodities becomes a barrier to the use of APROFAM￾provided methods. In addition, those who do wish to obtain FP methods have alternative sources available for free such as the MOH Clinics and SIAS-sponsored NGOs. Presently, MSH is renewing its efforts with the RD department to re-engineer its structure and approach. Nevertheless, it is the consensus of all interviewed, including staff from the Rural Development Program, that the best that can be hoped for is basic cost-recovery. Subsidies will most likely be required for the foreseeable future. Even in order to achieve this modest goal, APROFAM will need to spend considerable time, effort and money to create a demand for quality and value-added FP services in these areas. Some Recommendations for the RD Program APROFAM has the basics already in place to improve its level of sustainability in the rural and highland areas of Guatemala. Quality of service, in terms of knowledgeable promoters, good general locations and logistics, is very good. Based on interviews with promoters some of the following ideas may be worth pursuing: · Develop and/or strengthen relationships and alliances with other NGOs in the highland regions. A large number of NGOs now either support or promote Maternal-Child Health activities and have been involved in Child Survival projects sponsored by USAID. · Work with the Ministry of Health and SIAS to provide the educational components of their FP and MCH activities. Promoters have mentioned that the MOH simply provides FP methods on demand, and provides little if any reproductive health education. · Increase the variety of non-FP inventory of promoters to include discounted popular over-the-counter non-prescription items. Maalox® and Pepto-Bismol® generics, mild analgesics, Sal Andrews®, and others are popular items available at local “mom and pop” stores as well as in pharmacies. Additionally, oral rehydration salts have now gained acceptance and popularity for diarhheal episodes. These products could be added on a test basis with a minimal marketing effort targeting local stores and pharmacies for price. · Review signage for placement. Most promoters have standard APROFAM signs, but some are difficult to see, being placed “flat” on the wall rather than MSH/APROFAM Evaluation Timothy Farrell, PhD 37 of 61 June 2003 perpendicular. Others are located so high on walls that they are not obvious unless one is specifically looking for them. 3. Adolescent Reproductive Health Education This is a new program that was formally begun in 2002. It is currently running a substantial deficit and is completely dependent on subsidies. Reynolds (2002) recommended that this program not be expanded, but that services continue to be provided through regular clinic sources. “A substantial investment in an expanded or new youth program would not be a prudent financial decision at this time – unless that investment produced significant CYPs, coverage and/or income…Special youth facilities or programs should be carefully tested before they are implemented, especially if they are costly and unsustainable…However, USAID and APROFAM continue to promote expansion of activities that jeopardize sustainability” (p.8). 4. Protection against Domestic Violence This program is new and its only source of financing appears to be from external grants from FHI ($4,178.01) and FNUAP ($10,297.15) in 2002. While this program definitely reflects a need both in Guatemala City and the interior, it should be subject to thorough testing before becoming part of APROFAM’s core activity budget. APROFAM’s management argues that the reality of the Guatemalan situation of women and youth requires APROFAM intervention in these areas, and that it falls well within the APROFAM and IPPF missions to provide services in support of these populations. No matter how valid these activities are viewed they represent an outlay of (largely) non￾recoverable costs that will ultimately affect APROFAM’s sustainability. An APROFAM without Subsidies? The February 2002 APROFAM Vision statement reads: “Ser en el año 2010, una organización autosustentable, líder en el ámbito nacional en la prestación de servicios de salud orientada a la satisfacción de las necesidades del cliente, a través de un exitoso mercadeo social” To become by the year 2010 a self-sustaining organization, a national leader in the provision of health services oriented to client satisfaction through a successful social marketing (strategy).21 Given the sustainability risks outlined above, APROFAM will have an uphill road to climb over the next 6 or 7 years in order to achieve this vision. Reynolds outlined four principal strategies for increasing income: increasing prices, improving unused capacity (currently nearly 58%22) in clinics, expanding profitable services and identifying other sources of income (p9). 21 Author’s translation. 22 See Appendix 5. MSH/APROFAM Evaluation Timothy Farrell, PhD 38 of 61 June 2003 To Reynolds’ list, it is necessary to add the exploration and development of new markets. While APROFAM has traditionally targeted the lower middle to lower economic classes of Guatemala, the Executive Director recently expressed interest and enthusiasm regarding the marketing of clinical laboratory services to more economically well-off clients. APROFAM boasts one of the most sophisticated clinical laboratory facilities in Guatemala and services are substantially less expensive than the major laboratories that serve the middle and upper classes (in some cases by less than half or more). To tap into this market APROFAM needs to confront at least four major obstacles: 1. APROFAM’s image of being “only a FP” program 2. APROFAM’s focus on economically marginal populations 3. A slightly less significant problem is that APROFAM is still often considered an arm of the Ministry of Health 4. Location of the laboratories. The central clinic complex is located downtown (Zone 1), about a block from one of the major MOH hospitals. It is not an area where higher economic classes frequent by choice. Consequently, if APROFAM wants to tap into this market, cost and quality probably will not be sufficient to attract the more affluent classes. Similarly, since private doctors usually refer patients to clinics associated with hospitals where they are on staff, a significant marketing effort must be made to gain physician confidence and referrals. APROFAM would also need to consider establishing laboratory specimen “collection units” in more affluent zones. Conclusions The MSH–APROFAM partnership has been highly successful, both in terms of concrete results and the processes by which these are achieved. APROFAM has reached about an 80% level of financial sustainability. Clinic services have reached a 120% level of financial sustainability as a whole. Four of the five departments have been fully re-engineered and systems and controls developed that provide both cost-cutting elements as well as growth potential. The process by which these achievements were accomplished generally followed sound concepts relating to institutional cooperation, communication and participation. MSH provided expert technical support and conscientious individuals who sought and acted upon APROFAM’s needs and long-term goals. APROFAM actively managed the process and activities with full support from the Board and the Executive Director. Importantly, APROFAM knew generally what it needed to achieve and was thus in a strong position to guide the technical assistance towards those ends. Therefore, the main tasks of MSH were to seek and provide the very specific technical and organizational means towards APROFAM’s goals. The following table summarizes the factors that contributed to the success of the MSH technical assistance to APROFAM. MSH/APROFAM Evaluation Timothy Farrell, PhD 39 of 61 June 2003 Table 4 Critical Factors and Variables in Institutional Change MSH – APROFAM External Factors Process Variables Internal Factors Propose Concept Negotiation of Type and Duration of Assistance Solicit Assistance – “Know what you want and why you want it” Proposal of Activities Negotiation of Process Political will of Management and Board of Directors “Prepared to accept Sustainability as a Priority” Financial Assistance within Absorptive Capacity of NGO Negotiation of Amounts · Scope of the financial assistance “fits” the size and complexity of the organization · Basic financial systems in place Appropriate Technical Assistance Negotiate Priorities with Donor & Agent · Priority Setting and Decision Making Skills · Control Pace of Progress Solid Technical Assessment and Communication and Listening Skills Full Participation in Assessments at all steps · Basic Skill sets in place · Assertive Willingness and Capacity to Learn · Communication and Teamwork within APROFAM Change Model Re-Visited This review has provided a new model and set of assumptions that are relevant to re-engineering for sustainability based on the empirical findings of this evaluation and some additional thinking regarding the process. MSH Inputs & Support APROFAM MANAGEMENT & BOARD GROUPS Medical Services Department Finance Department Administrative Department Rural Development Department Marketing Department APROFAM-MSH RE-ENGINEERING PROCESS Sustainable Results in Finance, Clinic, Strategic Planning, Marketing and Support System Negotiation & Feedback Propose & Review Alternatives for Implementation Planning & Implementation Groups from All Departments Feeback Implementation and Monitoring of Specific Activities is Conducted By Department Heads and Reported Back to the Planning Group(s) Final Decisions and Control Must Reside Here This diagram with its multiple feedback loops and final control by the Executive Director (supported by the Board) more accurately expresses the process that was undertaken in this case. It also integrates the concepts and goals of re-engineering with the process and activities required for implementation. More importantly, it leaves the “how to” or operational issues to managers and staff who are directly responsible for the activities and who know the individuals and their capabilities for implementation, hence reducing the risk of micro-managing and/or management by committee Finally, Table 5 lists common obstacles and possible resolutions based on observations in this evaluation. Table 5 Obstacles and Possible Resolutions to Institutional Change Obstacles Possible Resolutions Institutional Cultural Resistance 1. Change and doubt about the future reinforces “…how things have always been done.” 2. Traditional “fiefdoms” are challenged · Early and full communication of anticipated outcomes (including the reality that not all outcomes are clearly known, and that uncertainty is normal in change) · Establish “assessment” and “implementation” groups at all levels during the process. Ensure that they have a formal “say-so” · Pro-actively monitor, anticipate and prepare (financially and psychologically) to accept significant staff turn-over as each department implements the process · Be prepared to recruit and train replacements, and have new job descriptions and requirements ready before hiring. Do NOT hire for old or “at￾risk” positions External Risk and Resistance 1. Vendors & Suppliers 2. General Public · Specify what new vendor and supplier criteria will be through both direct and indirect contact · Public Image Campaigns · Reinforce Organizational Quality Values internally and externally MSH/APROFAM Evaluation Timothy Farrell, PhD 43 of 61 June 2003 APPENDIX I MSH ACTIVITIES AND INNOVATIONS MSH/APROFAM Evaluation Timothy Farrell, PhD 44 of 61 June 2003 OBJECTIVE AND ACTIVITIES COMPLETED 1999 “BETTER HEALTH FOR RURAL WOMEN AND CHILDREN” Objective Activity 1. MSH will assist APROFAM in the development, promotion and implementation of strategies to increase coverage and improve the quality of its health services (clinics and health promoters) in underserved areas, especially in the highlands of Guatemala 1. Medical Quality Assurance Program 1.1 Present Medical Quality Assurance Program assessed and selected sites audited 1.2 QA program development plan established 2. Supervision 2.1 Supervision standards and indicators established and implemented 3. Integrated Model for Rural Program 3.1 Prototype integrated Rural Health Promoter Model developed 3.2 Model tested and finalized. 2. MSH will assist APROFAM in the development and implementation of integrated strategies to improve women’s and children’s health, especially in the rural highlands of Guatemala. The integrated strategies will include both the integration of women’s and children’s health care services and linkages between clinic-based and health promoter information and services. 1.1 Four pilot sites for program integration identified 2. Referral/Counter-referral System 2.1 Problem of referrals/counter-referrals between clinics and rural program assessed 2.2 Draft procedures and instruments drafted 2.3 System tested at selected sites 2.4 System finalized 3. MSH will assist APROFAM in the design and implementation of culturally acceptable marketing and educational strategies to improve women’s and men’s knowledge of reproductive health and to ensure that increased knowledge translates into increased use of reproductive health services. Assistance will include training, supervision, and motivation of health workers to participate in a wide range of outreach activities, with emphasis on regular home visits. 1. IE&C/Marketing Program 1.1 Assessment of IE&C/Marketing department, program and materials 4. MSH will assist APROFAM in the development of its strategic planning, health care management/administration (including supervision), monitoring and evaluation capabilities, and decentralization, in both the rural and 1. Strategic, Operational and Local Planning 1.1 Develop monitoring system for strategic plan 1.2 Develop departmental plans consistent with strategic plan MSH/APROFAM Evaluation Timothy Farrell, PhD 45 of 61 June 2003 Objective Activity decentralization, in both the rural and clinic programs. A strong emphasis will be placed on the development of administrative and decision-making capabilities of clinic managers and Jefes de Campo in support of decentralization and improved local supervision, monitoring and evaluation, and bottom-up planning. 5. MSH will assist APROFAM in the development of other management and administrative skills that will directly improve rural program performance 1. Rural Program Component Identification and Definition 1.1 Identify and define project components of the rural program 1.2 Develop strategies, goals, objectives, target populations and program indicators for each component 1.3 Identify management information necessary to monitor and evaluate component performance 2. Commercialization of Urban Product Sales 2.1 Finalize results of assessment of the commercialization of product sales 2.2 Identify pilot project area in Guatemala City 2.3 Develop a Business Plan for pilot project 3. Rural Community Mapping 3.1 Introduce rural community health mapping methodology 3.2 Select initial test sites 3.3 Train staff in use of methodology 3.4 Assist in field testing 3.5 Review initial results and develop plan for expanded use of methodology 4. Rural Program management 4.1 Develop program indicators for the rural program 4.2 Identify needed management information to monitor program indicators 5. Rural Program Supervision 5.1 Finalize supervisory manuals and instruments 5.2 Ensure that staff is trained in the use of supervisory system MSH/APROFAM Evaluation Timothy Farrell, PhD 46 of 61 June 2003 OBJECTIVES AND ACTIVITIES COMPLETED 2000 “BETTER HEALTH FOR RURAL WOMEN AND CHILDREN” Objective Activity 1. MSH will assist APROFAM in the development, promotion and implementation of strategies to increase coverage and improve the quality of its health services (clinics and health promoters) in underserved areas, especially in the highlands of Guatemala 1. Development of a Quality Assurance Program for the Rural Development Program of APROFAM 1.1 Assess current status of efforts to monitor and control quality including, protocols, standards, instruments and information system. 2. MSH will assist APROFAM in the design and implementation of culturally acceptable marketing and educational strategies to improve women’s and men’s knowledge of reproductive health and to ensure that increased knowledge translates into increased use of reproductive health services. Assistance will include training, supervision and motivation of health workers to participate in a wide range of outreach activities, with emphasis on regular home visits 1. Development of a Comprehensive Marketing Program and Department 1.1 Development of a re-engineering plan for APROFAM’s marketing activities and department 1.2 Segmentation of the market by principal units of service offered by APROFAM 1.3 National market research by segment and unit of service 3. MSH will assist APROFAM in the development of its strategic planning, health care management/administration (including supervision), monitoring and evaluation capabilities, and decentralization, in both the rural and clinic programs. A strong emphasis will be placed on the development of administrative and decision making capabilities of clinic managers and Jefes de campo in support decentralization and improved local supervision, monitoring and evaluation, and bottom-up planning 1. Re-engineering of the Department of Planning, Evaluation and Statistics 1.1 Assessment of the current processes, procedures, staffing, reporting formats and functions of the department 1.2 Development of recommendations and an assistance plan for re-engineering of the department 1.3 Development of new, formal procedures to guide the department, new job descriptions and new reporting formats that facilitate program monitoring and evaluation at the local and central levels 2. Development of Management Software Support Manuals and Development and Implementation of the Final Five Additional Modules. 2.1 Define the technical parameters and modules 2.3 Support the software provider in the MSH/APROFAM Evaluation Timothy Farrell, PhD 47 of 61 June 2003 provision of technical assistance to APROFAM 2.4 Review, edit and finalize support manuals for the eight basic management software modules 3. Comprehensive and Continuous Management Training for Rural Development Program Supervisors 3.1 Management training needs assessment 3.2 Design and development of long distance, continuous management training program 4. MSH will assist APROFAM in the development of other management and administrative skills that will directly impact program performance 1. Development and Implementation of an Employee Evaluation System and Accompanying Support Software 1.1 Development of employee evaluation guidelines 1.2 Development of employee evaluation draft policies and procedures 1.3 Development of draft evaluation instruments 1.4 Field testing of draft instruments 1.5 Finalizing of evaluation instruments 1.6 Finalizing of policies and procedures 1.7 Training of staff in use of policies, procedures and use of instruments 1.8 Implementation of evaluation system 1.9 Development of software specifications MSH/APROFAM Evaluation Timothy Farrell, PhD 48 of 61 June 2003 OBJECTIVES AND ACTIVITIES COMPLETED 2001 “BETTER HEALTH FOR RURAL WOMEN AND CHILDREN” Objectives Activities 1. MSH will assist APROFAM in the development, promotion and implementation of strategies to increase coverage and improve the quality of its health services (clinics and health promoters) in underserved areas, especially in the highlands of Guatemala 1. Development of an Incentive Program to Reward Employee Performance 1.1 Develop monetary and non-monetary incentive options 1.2 Review incentive options and analyze implications 1.3 Finalize incentive plan and develop procedures for formally linking it with employee evaluation process 1.4 Apply incentive program to employee evaluation cycle and assess results 3. Develop Systems and Processes, and Procedures Manual, for Staff Training, Client Education and Services Publicity 3.1 Establish mission and goals for each of the three areas/departments 3.2 Review and analyze existing policies, procedures and systems for the three areas/departments 3.3 Redesign systems and procedures to ensure better trained staff, higher quality and more consistent client education and more appropriate and effective publicity 3.4 Develop operations manuals for the three areas/departments 2. MSH will assist APROFAM in the design and implementation of culturally acceptable marketing and educational strategies to improve women’s and men’s knowledge of reproductive health and to ensure that increased knowledge translates into increased use of reproductive health services. Assistance will include training, supervision and motivation of health workers to participate in a wide range of outreach activities, with emphasis on regular home 1. Development of a Comprehensive Marketing Program and Department 1.1 Presentation and review of analysis of national market research 1.2 Finalize long term institutional marketing plan 1.3 Restructure marketing department 2. Development of a Long Term Institutional Business Plan 2.1 Review and define long term institutional financial goals MSH/APROFAM Evaluation Timothy Farrell, PhD 49 of 61 June 2003 visits 2.2 Identify long term income needs including capital investments and depreciation 2.3 Develop income and expense scenarios to reflect varying service and product sales and donor support 2.4 Finalize plan 2.5 Establish short, medium and long term sales and performance goals 3. MSH will assist APROFAM in the development of its strategic planning, health care management/administration (including supervision), monitoring and evaluation capabilities, and decentralization, in both the rural and clinic programs. A strong emphasis will be placed on the development of administrative and decision making capabilities of clinic managers and Jefes de campo in support of decentralization and improved local supervision, monitoring and evaluation, and bottom￾up planning 1. Year Long Management Development Program for Rural Development Program Managers 1.1 Implement program based on assessment and approved design completed last year (First of three modules completed) 1.2 Perform a mid-year and year-end evaluation of program impact on manager knowledge and skills (Evaluation results completed for first module) 2. Provide targeted Training Opportunities for Departmental Managers 2.1 Assess departmental manager training needs and research available domestic and international trainings 2.2 Prioritize and program manager training 4. MSH will assist APROFAM in the development of other management and administrative skills that will directly impact program performance 1. Development of an Operations Manual for the Computer System 1.1 Define goals, objectives and major considerations for the program 1.2 Assess existing policies and procedures to include recommended additions and changes 1.3 Develop draft policies and procedures and field test 1.4 Finalize operations manual 3. Development of a Human Resources Operations Manual 3.1 Review all recently developed policies, procedures and systems 3.2 Develop draft operations manual and field test 3.3 Finalize operations manual 4. Final Management Assessment of APROFAM 4.1 Facilitate MOST process to analyze and discuss the present status and future needs of each major management system 4.2 Discuss, analyze and finalize results and recommendations APPENDIX 2 PROPOSED SCOPE OF WORK EVALUATION OF THE RESULTS OF MANAGEMENT SCIENCES FOR HEALTH INTERVENTIONS WITH APROFAM, GUATEMALA BACKGROUND This proposed Scope of Work (SOW) is based on a request by Management Sciences for Health (MSH/Boston) to provide the basis for a review of the results and lessons learned of the MSH support provided to APROFAM during the period 1995 - 2002. MSH has had a continuous support relationship with APROFAM since 1995. This support focused on strengthening the management structure and the activities of APROFAM (Asociación Pro￾bienestar de la Familia) in order to improve program and overall institutional performance. APROFAM is a non-governmental organization (NGO) that provides a wide variety of Family Planning, Reproductive Health, Maternal and Child Health, Laboratory and other basic health services throughout the country. It also has a rural outreach program focused on FP and RH education and the distribution (sale) of family planning products. It historically has accounted for roughly 40% modern contraceptive use in the country and as such, is the largest single provider. APROFAM has a long history as the principal Family Planning (FP) and Reproductive Health (RH) agency in Guatemala. While IPPF is the principal philosophical, technical, and financial support of APROFAM, it has also received substantial financial support from USAID. Requirements for USAID support include that APROFAM pursue certain USAID goals and Intermediate Results (IRs). MSH and APROFAM initiated the establishment of more general Maternal-Child Health (MCH) services on a fee-for-service basis. This “diversification” of services is considered one pillar of MSH’s initial organizational development strategy for APROFAM in 1995 to move them towards sustainability. Aside from providing a much-needed service in many parts of the country, such services also are designed to cross-subsidize unprofitable services and products. In addition it strengthened its relationships with other NGOs providing educational services to both NGO staff and project participants. Such outreach efforts have established APROFAM as a primary source of FP and RH support for many NGOs.23 In addition, APROFAM has upgraded its clinical laboratory services to become one of the most modern clinical laboratories in Central America, with computerized analysis capabilities providing accurate results in very short time. The fees for laboratory services are modest, and within the economic reach of the current target population in the urban environments24 . 23 In a recent survey of 12 Guatemalan NGOs, several indicated that they used APROFAM clinics as referral points for FP and RH services. 24 The Executive Director is seeking ways to expand its client base beyond the traditional lower, and lower-medium middle class to middle class populations in Guatemala City. MSH/APROFAM Evaluation Timothy Farrell, PhD 52 of 61 June 2003 This confluence of factors early led to an initiation of a program by MSH, supported by USAID, to establish a working relationship with APROFAM that would strengthen their management structure and performance toward meeting goals leading to sustainability as well as supporting USAID/G Strategic Objectives (see Appendix 2 for the USAID IRs and objectives). APROFAM, in coordination with MSH, has both anticipated and responded to these IRs by developing and implementing a series of initiatives designed to improve APROFAM’s managerial capacity to improve program performance and enhance the probability of achieving a significant level of institutional and financial sustainability. PROPOSED SCOPE OF WORK Key Questions There are four key questions to be addressed in the Evaluation. 1. What has been the result (use and relevance to APROFAM) of the MSH interventions? Principal Activities, Innovations and Systems Introduced by MSH25 , 26 · Development and application (by APROFAM) of Strategic and Operational Planning Systems · Review and re-development of Quality of Care System (for both Medical and Service Quality, including a focus on gender and cultural – highland Mayan - issues) o Supervisory system to ensure quality performance · Re-engineering of Administrative, Marketing and Financial Systems (including computerized clinic cost analysis, financial, and inventory systems) · Integration of FP/RH services with Clinic services, including referral systems · Encourage decentralization or “de-concentration” of some decision-making to local clinic management · Development of a “Monitoring and Evaluation Plan and System” · Appropriate training and implementation assistance for all activities 2. How were these interventions incorporated and internalized in APROFAM? 3. How and by ‘how much’ have these interventions contributed to APROFAM’s performance over time as measured by: · Key service statistic indicators over time: · Couple Years Protection (CYP) · Types and numbers (units) of services provided by clinic and by rural promoters · Continuation/Discontinuation Rates · Method mix · Demand for limiting and spacing of children · Quality of Care indicators as measured internally · Financial progress towards self-sustainability – by clinic over time 4. Planning for interruption or decrease in USAID subsidy 25 This “inventory” of activities is valid only from 1998 – 2002. The MSH Action Plans from 1995 – July 1998 are not available at this time. They will be included in the Evaluation itself. 26 From MSH Annual Plans and Reports 1998 – 2002. MSH/APROFAM Evaluation Timothy Farrell, PhD 53 of 61 June 2003 APPROACH AND METHODOLOGY This SOW suggests an approach that focuses on the management functions of APROFAM using more directly observable variables based on the incorporation of MSH activities into APROFAM’S management behavior (implementation of MSH recommendations) and performance results in these areas. This requires a modified and retrospective longitudinal approach beginning with 1994 as Time 1 …. Time n model. The “Time” intervals are defined as one-year after the introduction of an MSH intervention or major activity in order to provide a “fair test” of time for APROFAM to incorporate the intervention and begin to use it. While it is impossible to generate “causal” effects of MSH interventions in the strict application of “cause and effect” we can develop timeline measurements with a one-year lag to “correlate” intervention and APROFAM activities. MSH Interventions APROFAM DATA 1995 1996 1996 1997 Implementation of Activities “How” & “Why” Etc. Evaluation Model MSH - APROFAM 1995 - 2002 The SOW should focus on the following factors related to MSH interventions with APROFAM over time. It is also important to note that there was a change in Executive Directors in 1998 that resulted in a number of changes independent of other external inputs. MSH/APROFAM Evaluation Timothy Farrell, PhD 54 of 61 June 2003 The basic framework for the SOW · Before MSH 1994/5 – End of 2002 o Population coverage (by geographic area and ethnicity) o Financial and Accounting Systems o Decision making sources and processes (how implementation occurred) ß Planning ß Monitoring ß Evaluation · APROFAM’s performance toward the IRs (detailed in Appendix 2) o Objective 1 – Coverage and Quality in the rural highland areas ß By how much has coverage increased in the rural highlands since the introduction of MSH support ß How does the Marketing Department support the investigation of “quality of services” – client satisfaction (vs. technical quality), while at the same time assessing the viability of new markets o Objective-2 – Integrated women’s and children’s clinic-based service strategies in the rural highlands with promoter services ß How are women’s and children’s services integrated? Requires historical documentation and focus group interviews with promoters at a sample of highland clinics regarding what kinds of services they feel comfortable in providing and the guidelines for referral. o Objective 3 – Appropriateness and effectiveness of marketing and educational efforts as applied to increased use of RH services ß What was the process of developing educational content? ß Technical assessment of educational content in the context of the highland population cultural milieu recognizing the diversity in the various areas served; this can be conducted simultaneously with the focus groups noted in Objective 2. o Objective 4 – Planning, supervision (monitoring) and decision-making capacities at the local clinic level ß What systems are in place; how are they used; are they satisfactory to clinic level management? o Objective 5 – Review of the general management practices now in place that affect the overall performance of APROFAM ß Semi-structured interviews with Central Office managers and the correspondence of their stated practices with similar interviews with field clinic managers MSH/APROFAM Evaluation Timothy Farrell, PhD 55 of 61 June 2003 Incorporation of Pro-Active Strategic Planning to external events (How will (has) APROFAM plan for uncertainty) Strategic planning in response to financial environmental change: How has the MSH work in strategic planning prepared APROFAM with the tools to respond pro-actively to major financial reductions by donors? For example: USAID is withdrawing $1m in support this year. Illustrative questions would include: · What degree of cross-subs idy can APROFAM realistically expect from its very modern laboratory and “profitable” clinics to support those still working towards financial sustainability? · Is the contemplated “incentive program” realistic, given that there is general consensus that the most impoverished (highland indigenous population” (target group of AID’s IR)) often cannot afford to pay even minimal amounts for services? · Can APROFAM “profitably” divest itself of some rural service deliveries by partnering with other NGOs? What training and technical support would that require? And how can that be supported? For example, the relationship between APROFAM and SIAS (Sistema Integral de Atención en Salud) of the MSPAS needs to be examined. Proposed Methodology The above “main questions” dictate that a qualitative approach (semi-structured interviews and focus groups) is most appropriate in this SOW. A survey would provide no supporting dimension at this time. The data available at APROFAM Central are certainly adequate and sufficient to draw conclusions over coverage, CYP, and other dimensions of MCH and RH service activities. This can be referred to as “quantitative historical documentation” and should be reviewed from a quantitative perspective over time as suggested in the graphic Evaluation Model above (p.3). MSH/APROFAM Evaluation Timothy Farrell, PhD 56 of 61 June 2003 Proposed Calendar of Activities Activity Dates /time intervals* Number of days Meet with Sr. APROFAM Staff April 22 1 Develop Timeline of MSH Interventions April 23-26 4 Develop instruments April 29-30 3 Begin Field Clinic Interviews May 1-6 5 Full Review of Performance Issues May 7-13 6 Data Analysis May 12-16 4.5 Briefing May 16 .5 Prepare Written Draft Report for Review May 17-19 3 Finalize Report May 26-27 2 Submit Final Report May 28 0 Total April 22 – May 28 29 * proposed dates, tentative for now Level of Effort: 29 Total Working Days 1 Person: NGO Management & Evaluation Specialist Deliverables: Written report: Activities; Timeline of MSH activities corresponding with APROFAM performance and results; Analysis of how and when (where possible “why”) MSH efforts integrated with APROFAM activities and performance. Data Appendices. Documentation Appendices. MSH/APROFAM Evaluation Timothy Farrell, PhD 57 of 61 June 2003 APPENDIX 3 ACTIVITIES CONDUCTED IN THE DEVELOPMENT OF THIS SOW Two Scopes of Work (Appendix 1) governed the basic activities conducted under this activity. All activities described under the Level of Effort (LOE) were conducted except for a visit to program sites. Several delays were encountered due to the end of the Guatemalan Christmas holiday system, resulting in a departure from the originally scheduled period of performance. Visits and interviews were conducted with all key management staff (Appendix 2) and the Guatemala City main clinics, laboratories, and warehouse were visited. All relevant MSH documents were reviewed, as were current and past data on individual clinic performance. The two principal APROFAM documents used to develop the following SOW are: · Resultados: Seminario-Taller, Planeación Estratégica APROFAM 2002 (Seguimiento y Actualización. Febrero, 2002. · Planeación Estratégica: Seguimiento y actualización APROFAM, Febrero del 2000 As noted in the original SOW for this exercise, “Given the donor’s level of satisfaction, the objective of the proposed evaluation is not so much inspired by accountability concerns and requirements as it is by a desire to benefit from project experience through lessons learned that can be applied by MSH to similar interventions.” Consequently, FP & RH service statistics were not reviewed in detail. Nevertheless, a brief document review of service statistics and the Cost and Revenue Analysis Tool (CORE) program in operation suggest that there has been gradual but considerable gain over time with respect to cost management and financial sustainability at some clinics. APROFAM, according to management, has modified CORE for its own needs. It is used regularly (efforts are made to conduct the data collection twice per year per clinic). CORE is considered a basic management tool, and is understood and used by APROFAM management. As a result, one can say that the CORE tool initiated by MSH is a very successful MSH accomplishment. Nevertheless, in APROFAM’s 2002 SWOT/FODA analysis, one weakness identified is “Decisions are not always based on financial information The focus of this exercise was to develop a SOW management issues as they pertain to the various intercepts of APROFAM, MSH and to some degree, USAID/G. Because APROFAM has adopted and operationally incorporated many MSH interventions, it is important for the SOW to gain a portrait of APROFAM’s performance data (coverage, CYP, financial sustainability) “before” and “after” MSH assistance. That is, in this brief exercise, one observes MSH interventions both directly and indirectly influencing operational decisions on a regular basis. As only an example, CORE, a spreadsheet program developed by MSH, that calculates cost by service (or product) and provides a simple cost analysis on a per-clinic basis... During the visits, I observed decisions being debated based on the costs of certain products and services for one clinic area. MSH/APROFAM Evaluation Timothy Farrell, PhD 58 of 61 June 2003 The political and religious environments in Guatemala have not always been supportive to family planning and reproductive health27, and APROFAM has continuously advocated for a more positive national policy. Because of the inconsistent government policies, at times a perception by some segments of the population of Guatemala regarding family planning has been uneven. In some cases, it even has been hostile owing in some degree to the influence of certain evangelical religious sects, especially in indigenous areas, as well as some elements in the Catholic Church. Additionally, at times APROFAM even has been identified as a government entity (Planeación Estratégica APROFAM 2002, “FODA/SWOT” Analysis – Threats “Todavía nos ubican como una entidad del Estado o sólo de PF” – We are still identified as a state entity or only involved in Family Planning). (Author’s translation APROFAM, nevertheless, attempts to turn such misunderstanding or “threat” to its strategic advantage. For example, in APROFAM’s Strategic Planning document (February 2000), it states: “…the national environment (which is not particularly favorable) does not represent a significant threat to the Association, but to the contrary, provides important opportunities to grasp.” “The policies and basic frameworks of the new government…for strengthening programs in sexual and reproductive health do not show (much) political will to establish such programs.” (Author’s translation) APROFAM, as the leader and main provider of FP and RH services in the country, has received substantial financial support from IPPF and USAID over the years. In the early￾mid 1990’s USAID initiated policies to encourage its affiliates to seek, develop, and implement strategies and activities leading to their sustainability.28 27 Young, Anne M. “Preparing for the 21st Century in Health Care: Lessons Learned from APROFAM”, MSH, November 1999, p.3. 28 I personally participated in two sustainability assessments of IPPF projects in the early 1990s, Colombia and the Dominican Republic. MSH/APROFAM Evaluation Timothy Farrell, PhD 59 of 61 June 2003 APPENDIX 4 USAID/G Intermediate Results Intermediate Result 1: More families use Quality Maternal and Child Health Services 1. MSH will assist APROFAM in the development, promotion and implementation of strategies to increase coverage and improve the quality of its health services (clinics and health promoters), especially in the highlands of Guatemala. 2. MSH will assist APROFAM in the development and implementation of integrated strategies to improve women’s and children’s health especially in the rural highlands of Guatemala. The integrated strategies will include both the integration of women’s and children’s health care services and linkages between clinic based and health promoter information and services 3. MSH will assist APROFAM in the design and implementation of culturally acceptable marketing and educational strategies to improve women’s and men’s knowledge of reproductive health and to ensure that increased knowledge translates into increased use of reproductive health services. Assistance will include training, supervision and motivation of health workers to participate in a wide range of outreach activities, with emphasis on regular home visits. Intermediate Result 2: Maternal and Child Health Programs are Better Managed 4. MSH will assist APROFAM in the development of its strategic planning, health care management/administration (including supervision), monitoring and evaluation capabilities, and decentralization in both rural and clinic programs. A strong emphasis will be placed on developing the administrative and decision-making capabilities of clinic managers and Jefes de Campo in support of decentralization and improved local supervision, monitoring, evaluation and bottom-up planning 5. MSH will assist APROFAM in the development of other management and administrative skills that will directly improve program performance MSH/APROFAM Evaluation Timothy Farrell, PhD 61 of 61 June 2003 APPENDIX 5 APROFAM STAFF INTERVIEWED The following is a list of the principal individuals formally interviewed at APROFAM Central Offices: Dr. Edwin Leonel Morales Flores Gerente de Servicios Médicos Licda. Suzette Higueros Gerente de Finanzas Ingeniera Zonia Aguilar Gerente Administrativa Dra. Rebeca Arrivillaga Gerente del Programa de Desarrollo Rural Lic. Edílzar Castro Gerente de Mercadeo Dra. Telma Duarte de Morales Directora Ejecutiva Ing. Sergio Donaldo Cruz Asistente de Dirección Ejecutiva Ing. Selvin Fuentes Jefe de Planeación y Estadística External Resources Dr. Hector Colindres, Consultant to MSH Considerable input from Mr. Michael Hall in response to earlier drafts (email)