FORTE SAÚDE MID-TERM EVALUATION December 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Lia Kropsch and Jennifer Robinson through the Global Health Technical Assistance Project. FORTE SAÚDE MID-TERM EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 08-001-109) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources/. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by the QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. FORTE SAÚDE MID-TERM EVALUATION i ACKNOWLEDGMENTS The evaluation team acknowledges and thanks the many people and organizations that contributed directly and indirectly to this assignment. Among them, very special thanks are offered to The USAID SO 8 Team, especially Vanessa Coelho and Alfredo MacArthur, who were particularly helpful in discussing the assignment, organizing visits and meetings, and participating in the evaluation of the projects Ministry of Health officials, particularly Dr. Martinho Djedje, Dr. Gertrudes Machatine, Dr. Leonardo Chavane, and Dr. Benedita Silva, who found time to meet with the evaluation team in spite of their busy schedules João Aussi and the World Vision COACH Project team, who organized and facilitated the evaluators’ visit to Zambézia Province The FORTE Saúde team, whose time and cooperation in gathering reports, data, and other information made this assignment possible ii FORTE SAÚDE MID-TERM EVALUATION FORTE SAÚDE MID-TERM EVALUATION iii ACRONYMS AI Avian influenza ANC Antenatal clinic CHBC Community home-based care CS/RH Child survival and reproductive health DDS District Health Departments DHS Demographic and Health Surveys DPS Provincial Health Departments EOC/ENC Emergency obstetric/neonatal care EPI Expanded program of immunization FS FORTE Saúde GH Tech Global Health Technical Assistance Project HAI Health Alliance International HIS Health information system HKI Helen Keller International HR Human resources ICT Information communication technology IEC Information, education, and communication IMCI Integrated management of childhood illness IR Intermediate Result IS Information systems JHPIEGO Johns Hopkins Program for International Education in Gynecology and Obstetrics KPC Knowledge, Practice, and Coverage Survey M&E Monitoring and evaluation M&L Management and leadership MCH Maternal and child health MH Maternal health MINAG Ministry of Agriculture MIS Management information system MISAU Ministry of Health MPH Master’s in Public Health NGO Nongovernmental organization NHIS National Health Information System PEN Multisectoral National Strategic Plan PESS Strategic Plan for the Health Sector PMEP Performance Monitoring and Evaluation Plan PMP Performance Monitoring Plan PMR Performance Monitoring Report PMTCT Prevention of mother-to-child transmission PRSP Poverty Reduction Strategy Paper iv FORTE SAÚDE MID-TERM EVALUATION PVO Private voluntary organization QIP Quality improvement process RFP Request for proposal RH Reproductive health SO Strategic Objective SRH Sexual and reproductive health SWAp Sector-wide approach TA Technical assistance UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USA United States of America USAID United States Agency for International Development WHO World Health Organization FORTE SAÚDE MID-TERM EVALUATION v CONTENTS ACKNOWLEDGMENTS .................................................................................................................... i ACRONYMS .................................................................................................................................... iii EXECUTIVE SUMMARY ................................................................................................................ vii Background ................................................................................................................................ vii Major Findings ............................................................................................................................ vii Major Recommendations ............................................................................................................ix Suggestions for Future Directions ................................................................................................ x INTRODUCTION ............................................................................................................................. 1 Evaluation Purpose ..................................................................................................................... 1 Background ................................................................................................................................. 1 Methodology ................................................................................................................................ 2 FINDINGS ........................................................................................................................................ 3 Interviews .................................................................................................................................... 3 Technical ..................................................................................................................................... 9 Program Reporting and Management ....................................................................................... 20 CONCLUSIONS ............................................................................................................................ 21 Lessons Learned ....................................................................................................................... 21 RECOMMENDATIONS ................................................................................................................. 24 Technical ................................................................................................................................... 24 FUTURE DIRECTIONS ................................................................................................................. 26 APPENDIX A: SCOPE OF WORK ............................................................................................ 27 APPENDIX B: PERSONS INTERVIEWED ............................................................................... 35 APPENDIX C: DOCUMENTS REVIEWED ............................................................................... 37 APPENDIX D: INTERVIEW GUIDELINE .................................................................................. 39 APPENDIX E: CONSOLIDATED INTERVIEWS ....................................................................... 41 APPENDIX F: SCREEN SHOTS OF QUALITY IMPROVEMENT DATABASE ........................ 50 APPENDIX G: SAMPLE SUGGESTED RESULTS FRAMEWORK .......................................... 54 APPENDIX H: SAMPLE SUGGESTED INDICATOR REFERENCE SHEET ........................... 57 APPENDIX I: FORTE SAÚDE ORGANIZATION CHART ....................................................... 59 TABLES Table 1: Technical Areas that Have Received M&E Support from FS ..................................... 10 Table 2: Data Collection Forms Developed by MISAU with the Help of FS ............................. 11 Table 3: Documents Produced By Misau, Its Partners, and FS ............................................... 14 Table 4: Documents Drafted by the MISAU with FS Assistance .............................................. 15 Table 5: Other FS Activities and Products ................................................................................ 16 Table 6: Quality Improvement in Pilot Facilities ........................................................................ 18 Table 7: Sample of Results Framework for the FS Project ....................................................... 54 Table 8: Indicator Reference Sheet Template .......................................................................... 57 vi FORTE SAÚDE MID-TERM EVALUATION FORTE SAÚDE MID-TERM EVALUATION vii EXECUTIVE SUMMARY BACKGROUND The health team (SO 8) of the U.S. Agency for International Development (USAID) in Mozambique requested that the Global Health Technical Assistance Project (GH Tech) conduct an external mid-term evaluation of the USAID-funded consortium FORTE Saúde (Fostering Optimization of Resources and Technical Excellence for National Health; FS), led by Chemonics. The objectives of the evaluation were to review the technical and managerial performance of the FS program, including the subcontractors who deliver the four project objectives, and make recommendations for a future program to support the Mozambique Ministry of Health (MISAU). The evaluation employed a consultative methodology that triangulated interview responses, document review, and observations by the evaluators. Those interviewed included FS staff, MISAU staff at both central and provincial levels, subcontractors, representatives of private voluntary organizations (PVOs), partners, and others. (Persons interviewed are listed in Appendix B, documents reviewed in Appendix C.) The primary objectives of the FS project are 1. More effective information and monitoring and evaluation (M&E) systems; 2. Policies and strategies updated and implemented in target provinces; 3. Management, leadership, and quality improvement strengthened; and 4. Strengthened human resources (HR) training process. FS activities are intended to strengthen and expand priority interrelated management systems that will make the MISAU more effective in managing health resources. They are related to USAID/Mozambique’s SO 8 IR 3: More accountable policy and management. MAJOR FINDINGS Technical 1. While the FS project has not always addressed every outcome in the contract agreement, it has done a phenomenal amount of work, significantly building the capacity of the MISAU in M&E, policy and strategy planning and documentation, and quality improvement. The quality of technical materials produced by and with the assistance of FS staff is excellent; the quantity of work accomplished at midpoint in the project is exceptional, particularly given how few staff are working on it. 2. FS has collaborated with MISAU staff and its partners in drafting assessments, results frameworks for 15 target areas, M&E plans for six target areas, strategic plans for 10 target areas, 24 data collection forms, and training materials. It organized and conducted several M&E workshops. Staff participated in updating policies and strategies; provided technical information to complete 15 policy documents (four of which are still being finalized); developed and updated 12 technical documents related to sexual and reproductive health/child health (SRH/CH); and helped the MISAU organize and conduct three national conferences and seminars. 3. FS worked with the MISAU and partners on an action plan to implement the Quality Improvement Process (QIP); defined quality standards for SRH/CH; piloted QIP instruments; developed training materials, an implementation guide, and methodology; and provided training and initiated the pilot phase in 18 health units in six provinces. FS provided technical support for setting QIP baselines and supported the drafting of action plans to address highlighted issues. In the 14 facilities that participated in the second evaluation, QIP produced an average score improvement of 23 percent. A database to monitor health facility QIP scores is being tested and refined. 4. Although there was a management and leadership (M&L) needs assessment in 2006, little was done in this area except that FS included M&L in the QIP. Since January 2008, with a new coordinator, a 10-module course has been created and is under final discussion before the HR/Management Directorate implements it. viii FORTE SAÚDE MID-TERM EVALUATION 5. FS supported 11 scholarships for postgraduate courses for MISAU candidates and financed three maternal and child health (MCH) nurses groups in Nampula and Quelimane, resulting in 92 nurses upgraded from basic to middle level. FS has also assisted the MISAU and the Ministry of Agriculture (MINAG) with training and creating information, education, and communications (IEC) materials for the avian influenza (AI) prevention and control program. 6. FS built credible relations, widely recognized as positive, with lower-level MISAU staff, PVOs, and other partner organizations. The methodology of FS in working with their MISAU counterparts rather than for them has succeeded in building MISAU staff capacity across all departments. 7. The inclusion of local technical staff in the FS program enhances the vision of FS to respond to the real SRH/CH needs of Mozambique and aligns the activities of the program with local cultural norms. 8. High MISAU staff mobility and capacity challenges, in terms of both staff number and technical capacity, has been a significant obstacle for every activity FS has undertaken. Findings Related to Strategic Objectives SO1—More effective information and monitoring and evaluation systems: Uncontrollable factors in MISAU data management processes render its data unsuitable for reporting and use in decision making. Clinical health service indicators do provide a good measure of long-term success, but they should be evaluated at the close of the project as part of a broader impact assessment, not regularly throughout project implementation. The management utility of receiving such information quarterly is out of proportion to the costs both FS and the PVOs incur in collecting it, especially considering the data quality problems. The MISAU National Health Information System (NHIS) is comprised of numerous parallel information systems that are not well integrated. It is based on outdated technology (MS Access and Excel) that is unsuitable for a national data warehouse. There are numerous and well-documented limitations that render NHIS data less than useful for decision making. SO2—Policies and strategies updated and implemented in the target provinces: FS has had extensive involvement with the MISAU in drafting and redrafting policy and strategy documents but has too little political power to ensure that the policies and strategies are approved, much less implemented. While FS can provide technical assistance (TA) to support implementation, decisions about whether to implement lie with the MISAU. SO3—Management, leadership, and quality improvement strengthened: The QIP, even in the pilot phase, has had a conclusively positive impact at the facility level: All participating facilities improved the quality of their SRH/CH services. FS has built significant MISAU support and buy-in to the QIP—the MISAU has asked that this system be rolled out in all provinces. Including M&L in the quality improvement system was a practical way to get results in the absence of an M&L counterpart in the MISAU. SO4—Strengthened human resources training processes: Delays in MISAU’s selection of participants for postgraduate training meant that students had too little time for orientation and learning the requisite language skills. Selection criteria were either not established or insufficiently communicated. FS is accountable for outcomes even though it had little control of the process. AI activities were facilitated through the linkage of the MISAU and the MINAG. Training and communication materials were drafted with the MISAU and the MINAG, piloted in Zambézia, and finalized; however, at the time of the evaluation the minister had not given final approval for the materials. Program and Management Project Planning and Implementation The project design, centering on TA for policy and strategy development and implementation, vests accountability in FS project staff beyond their level of responsibility. The MISAU makes the decisions about policies and strategies to be drafted, their approval, and their implementation. It was clear to the evaluators that FORTE SAÚDE MID-TERM EVALUATION ix the MISAU did not understand that the intention of USAID in funding FS was to provide TA to the MISAU, and that the project scope was broad to give the program flexibility to respond as the MISAU’s needs changed. Considering that FS was the first Chemonics project in Mozambique, the project should have derived greater benefit from the pre-existing positive relationships that subcontractors (JHPIEGO, HAI, HKI, and Austral) had with senior MISAU officials. FS personnel should also have done more traveling to provide training and on-the￾ground support to PVOs and provincial, district, and facility MISAU staff. Project Management The shifting focus of the MISAU has resulted in delays and noncompletion of some FS activities. The impression of the evaluators is that the Chemonics head office may not have sufficiently empowered in-country FS management to manage the project budget and make financial decisions. The interpersonal relationships and management style of FS staff, which emphasized personal and organizational development and building the capacity of the MISAU, PVOs, and partner organizations and is based on working ―with‖ and not ―for,‖ has often meant that activities took longer than expected. Communication with subcontractors, particularly related to financial constraints experienced since February 2008, has been poor. Two of the original five subcontractors are no longer part of the consortium, for different reasons. This has not been adequately communicated to USAID and appears to the evaluators to be a management issue originating with the Chemonics head office. The subcontractors on the FS project have solid relationships with highly placed MISAU staff that FS has not capitalized on. This may be because the project is being promoted as a USAID-funded Chemonics project rather than a USAID-funded consortium. Project Monitoring and Evaluation and Reporting The method of reporting to USAID by activity in the Performance Monitoring Report (PMR) does not highlight tangible project outputs. Because the PMR reports only on activities that have been worked on during a given period, it is very difficult to track progress. Also, the time spent managing the Chronogram (drafting, translating, and retranslating work plans; management; and maintenance) is out of proportion to the benefit derived. The percentages reflected in the report, for instance, are misleading: an activity may be reported as 100 percent complete for two consecutive reporting periods, meaning that the discrete tasks planned for that activity for the reporting period were accomplished, but the reader is led to think that the task has been done twice. Sometimes the progress percentage is reported against the objective, at other times relative to a discrete activity. This makes it even harder to understand just what has been accomplished. Activities listed on the work plan are often not reflected on the reports at all because the PMR has a 10-page limit. FS does not report to USAID against the indicators in the Performance Monitoring and Evaluation Plan (PMEP), and targets for indicators are not specified in the FS PMEP. The majority of FS indicators in PMEP are outcome indicators, which are not useful for management and are not regularly collected. The main FS management documents are the project work plan and the Chronogram. The latter is administration intensive, difficult to understand, and inconsistent from one reporting period to the next. MAJOR RECOMMENDATIONS Build relationships with more highly placed MISAU staff to ensure the visibility of project outputs and enhance MISAU support for their implementation. Concentrate on improving and implementing the QIP. Because it has MISAU support at the highest level, that will enhance implementation of policies and strategies, improve M&L at every level, and enhance MISAU data management processes and capacity to manage data. These advantages are transferable to other areas of concern in the MISAU and will directly contribute to the reduction of maternal and child morbidity and mortality. x FORTE SAÚDE MID-TERM EVALUATION Support the Nutrition Sentinel System, which has MISAU support and has synergy with the QIP that can be taken advantage of where the processes are implemented in the same health facilities. Finalize a list of specific deliverables to the MISAU. – Award and manage consultancies to analyze information technologies supporting the HMIS and deliver a final report to the MISAU. – Update SRH/CH norms with the MISAU to align with the QIP instruments. – As soon as the minister’s approval is secured, produce and distribute training and communication materials for AI. – Continue financial support to MPH students. – Finalize M&L course content and training materials and submit them to the MISAU for implementation. Implement a Gantt chart project management tool, such as MS Project, to replace the Chronogram; give in￾country FS staff more control of the project budget; and update subcontracting documents. Revise the FS PMEP to define outputs that have a definite link to project activities; report on output indicators to USAID, and measure outcome and impact indicators only at the end of the project. Increase the level of effort for technical staff to ensure that TA is available for QIP implementation. Negotiate a budget that ensures sufficient funds. Change the point of contact for FS with the MISAU to the Director of the Health Promotion and Disease Protection Directorate. SUGGESTIONS FOR FUTURE DIRECTIONS 1. SO 1: FS should continue to ensure that the MISAU emphasizes M&E by ensuring that it is included in strategy, policy, and planning documents in target technical areas. While it should also continue to provide TA where possible to PVOs to enhance data quality, it should undertake no new activities in this area. 2. SO 2: FS should continue to work with the MISAU to annually update action plans, strategies, and policies within the National Directorate for Health Promotion and Disease Control, but undertake no new activities in this area. 3. SO 3: FS should concentrate on implementing the QIP—for maximum benefit, in parallel with the Nutrition Sentinel System. 4. SO 4: It should also continue to support current MPH students and finalize AI training and communication materials, but initiate no further activities for this objective. 5. Revise MISAU NHIS: USAID should address revision of the MISAU M&E system as a whole, NHIS, data management, and data quality processes in a new focused program that covers all MISAU activities, not just SRH and CH. 6. Revise the USAID SO 8 M&E Plan: The plan should be redrafted to reflect more accurately the activities and support funded by USAID. 7. Approach for future projects: Funds for infrastructure development, remodeling, equipment, training of MISAU staff, and community involvement should be allocated in future projects. USAID should present new projects to the leaders in the MISAU to get their support before issuing Requests for Proposals (RFPs), in order to support the MISAU as a whole rather than work only on narrow health issues. FORTE SAÚDE MID-TERM EVALUATION 1 INTRODUCTION FORTE Saúde (FS) is a five-year project funded by USAID to help the Mozambique Ministry of Health (MISAU) improve MCH/RH, malaria, and nutrition policies and implementation to enhance the quality and efficiency of services so as to improve the health status of the population. The approximately US$9 million project was awarded to a consortium led by Chemonics. It has four primary outcome objectives: 1. More effective information and monitoring and evaluation (M&E) systems 2. Policies and strategies updated and implemented in target provinces 3. Management, leadership, and quality improvement strengthened 4. A strengthened human resources (HR) training process EVALUATION PURPOSE The evaluation scope of work is attached as Appendix A. Its purpose was to give USAID/Mozambique a mid￾term assessment of how well the FS project is responding to Mission Strategic Objective 8 (SO 8): Increased use of child survival and reproductive health services in target areas. The seven-year SO 8 program (2003–2010) aims to build the capacity of MISAU. Support from Chemonics was intended to strengthen partner private voluntary organizations (PVOs) and nongovernmental organizations (NGOs) and empower targeted communities to increase use of, access to, and demand for quality child survival and RH services in targeted areas. The aims of the evaluation were to evaluate the technical and managerial performance of Chemonics and its subcontractors (Helen Keller International [HKI], IT Shows, Health Alliance International [HAI], JHPIEGO, and Austral) in delivering the FS objectives, and to provide recommendations for a future program to support the MISAU. BACKGROUND Although infant, child, and maternal mortality rates in Mozambique have been decreasing in recent years, they are still among the highest in Africa and the world. Communicable infectious diseases and parasites—malaria, diarrhea, respiratory infections, tuberculosis, and the rapid spread of HIV/AIDS—dominate the country’s epidemiological profile. Inadequate health infrastructure and service provision result in a poor quality of care. The Government of Mozambique is committed to building an equitable, affordable, and sustainable health system, but the health services network has not yet sufficiently evolved to meet the needs of a highly dispersed population. The purpose of SO 8 is to improve the health of Mozambican families so that they become more productive, less vulnerable to disease, and more effective participants in community health and development. The SO statement and its three intermediate results (IR) are as follows: SO 8: Increased use of child survival and reproductive health (CS/RH) services in target areas IR 1: Increased access to quality MCH/RH services in target areas IR 2: Increased demand at the community level for MCH/RH services in target areas IR 3: More accountable policy and management SO 8 covers a combination of national and community-level interventions designed to enhance the policy and management environment, increase access to proven, effective primary health services, and increase community demand for these services by strengthening community participation in managing or influencing the quality of health services and in providing them within the community. Interventions focus on health problems responsible for the largest number of child and maternal deaths: malaria, pregnancy and perinatal complications, vaccine￾preventable diseases, and diarrheal diseases. IR 3 activities are particularly linked with the FS project, which is charged with strengthening and expanding priority interrelated management systems that will improve the MISAU’s effectiveness in managing scarce health 2 FORTE SAÚDE MID-TERM EVALUATION resources. Better planning and management systems would allow the MISAU to more effectively use its financial, human, and other resources. Improved systems and procedures would allow for more comprehensive coordination of internal programs and cost-effective outsourcing of services to NGOs or the private sector. The intent was to help the MISAU better define the roles and responsibilities of its operating units and ensure that staff is adequately trained. This is vital not only to help the MISAU allocate human resources more effectively, but also to ensure that management systems, monitoring, and related interventions are consistent with the increased involvement of stakeholders. TA is being provided to the MISAU through a task order with Chemonics to build policy, program, intervention, communication, and management capabilities at both central and provincial levels. METHODOLOGY The methodology detailed below enabled the team to provide an objective and thorough evaluation of FS performance and ensured sufficient information to make practical recommendations for future activities. Data collected were primarily qualitative. Qualitative methods allowed the team to address deeper questions, allowed respondents to put forward their own opinions; and elicited more detailed responses than would be possible using quantitative methods. The target respondents were personnel of FS, subcontractors, USAID, and MISAU (central and provincial levels); and partners and PVO personnel (see list in Appendix B). Document Review: The evaluation team reviewed a variety of documents (see Appendix C), including performance monitoring plans, work plans, budgets, MISAU strategy and policy documents drafted by or with assistance of FS, reports submitted to USAID, training materials, PVO mid-term evaluation reports, and instruments for data collection and collation. Participative Meetings Team meetings: The evaluation team worked collaboratively, discussing findings and recommendations, preparing the report and presentations, and ensuring adherence to timelines. Stakeholder meetings: Active participation by all stakeholders in meetings ensured that (1) the evaluation addressed all stakeholder needs; (2) stakeholders bought into the process and the report; (3) capacity was built for all participants to engage in continuing informal evaluation; (4) capacity for evaluative thinking was enhanced; and the findings and recommendations are relevant and meaningful to all stakeholders. Debriefing meeting: The debriefing meeting on August 19 informed the USAID/Mozambique health team of the results achieved by the FS project and the resulting recommendations, and solicited feedback on the draft report. Interviews (Individual and Group) Interviews were the primary method of eliciting information about implementation of the FS project. While group interviews were envisioned, most interviews had just one or two respondents. Interviews were semi-structured, with the interviewer using a list of questions to guide the respondent (see Appendix D). Data Quality Assessment The initial methodology proposed a standard data quality assessment. This proved not to be necessary because FS does not report data relating to performance to USAID or the MISAU. Moreover, data included in USAID SO 8 reports from PVOs have widely recognized quality issues that limit their utility for decision making and reporting. Data Analysis Because the data collected during this evaluation were solely qualitative, and the sample is not sufficiently large to warrant a formal thematic analysis, the data analysis is informal. Findings are based on triangulated evidence from interviews, documents, and evaluator observations. The evaluation team compared the progress of FS at the midpoint with the proposed results, major constraints, and best practices in project implementation. The results were used to formulate recommendations for presentation to stakeholders. FORTE SAÚDE MID-TERM EVALUATION 3 FINDINGS In this section of the report the evaluators present findings from interviews with FS, MISAU, and partner organization and subcontractor staff, followed by the technical findings for the FS project, and finally project reporting and management findings. While this is a mid-term evaluation and thus concerns the work of the FS project only from October 2005 through December 2007, the late evaluation date has resulted in respondents giving impressions and results through mid-2008. All those findings are considered. INTERVIEWS This section gives a synopsis of the views of respondents elicited during interviews (for complete interview notes, see Appendix E). These views should not be interpreted as the findings of the evaluators. The Ministry of Health (MISAU) The Health Promotion and Disease Protection Directorate staff who worked directly with FS considers the technical team to be the best they have worked with. Their expertise, availability, and integration with the MISAU team and partners have been remarkable. FS worked with the MISAU in drafting plans, policies, strategies, and technical documents. MISAU staff had a very positive view of the FS contribution to drafting the instruments for quality improvement at services sites; assistance in organizing meetings, workshops, and conferences; and active participation in the Sector-wide Approach (SWAp) SRH and CH Group. There is general recognition that the MISAU staff shortage has seriously constrained the transfer of knowledge and technology and has delayed many FS initiatives. Another factor delaying decisions and activities has been the centralized MISAU management style. HKI has had a longstanding positive partnership with the Nutrition Department. Their most recent initiative, Sentinel Sites, is ready to be piloted. FS supported an initial theoretical training at MISAU headquarters, but it will also be important to support training for health facility providers who will be in Sentinel Sites and for community groups. Some MISAU departments were not able to take advantage of FS TA because FS is limited to four provinces, and the departments operate nationally. ―FS has been always available to provide support but we didn’t have the opportunity to take advantage of [it].‖ The Department of Noncommunicable Diseases received FS TA at a crucial moment, because the AI program had not been planned. FS and MISAU put together training and communications materials and tested them. Now the ministry is reviewing the final revision so that FS can have them produced and disseminated. For the NHIS, FS worked with MISAU staff in revising clinical records, designing record cards for data collection, and health indicators revision. FS started working with the HR and Management Department in 2007 to draw up a curriculum for M&L Training for the Provincial Health Directorate (DPS). The MISAU hopes soon to have FS support to complete and produce trainer and participant manuals and to facilitate the courses planned for the provinces. Some constraints were identified by all those interviewed: First was the difficulty FS had with starting up activities. 4 FORTE SAÚDE MID-TERM EVALUATION Second, the financial constraints faced by FS since February 2008 have not been explained to the MISAU; the feeling is that nothing related to money from USAID is well communicated. In some cases, the MISAU has asked other donors to provide resources to complete planned FS activities, which should not happen. ―If a project doesn’t have resources it has to close.‖ Third, it was mentioned that FS should concentrate on MCH rather than diversifying to information technology, human resources, AI, postgraduate training, etc. This seen as a mistake from all sides: the MISAU, USAID, and FS. Finally, several respondents indicated that the focal point for the project should be within the Directorate of Health Promotion and Disease Control. Provincial Health Department (DPS) DPS staff who were interviewed often raised two technical topics: The QIP and strengthening M&L. FS trained eight nurses in each province in the QIP, making use of the instruments, and how to evaluate whether a health unit had improved. FS also helped set the baseline for the QI process in health units. Nurses and senior DPS staff interviewed are very motivated and considered the pilot health units to already be showing improvement. They found the instrument very useful and easy to use. There have been many requests for additional TA and training on the QIP in order to expand it to other health units. Respondents also indicated that follow-up with in-service training is essential for continuity. Each province should have a group of facilitators consisting of people from both DPS and the Training Institutes. ―FS should organize a provincial meeting for the six provinces that are implementing the Quality Improvement Process, with the DPS director, the medical head, and the training institute director from each of them‖ FS has created interest in M&L leadership issues. Respondents said that many gaps could be addressed by such a course. It is important to emphasize the role of managers in the QIP. ―The majority of managers do not know what it is to be a manager. They only know to fulfill orders.‖ Private Voluntary Organizations (PVOs) The general feeling of the PVOs is that FS started late but is useful in linking PVOs with USAID and organizing the coordination meetings, which are viewed as a very useful space for information exchange and clarification. ―These coordination meetings would not take place without FS.‖ PVOs see the development of the QIP as important work that will have a significant positive impact in Mozambique. FORTE SAÚDE MID-TERM EVALUATION 5 ―The quality standards are very helpful. In applying them, we noticed that they are not flexible and due the lack of resources available, are difficult to achieve. But standards that don’t involve costs are already improving.‖ FS coordinated the process for receiving authorization from the Commission of Bioethics to conduct the Knowledge Practice and Coverage Survey (KPC) and helped review the questionnaire. However, when the minister indicated that the sample size must increase from 300 to 3,000, FS ought to have stepped up and explained why this is not practical or necessary. FS consolidates the PVO reports for MISAU but never sends copies back to the PVOs. All PVOs agreed that FS has very competent staff, and that the relationship with them is excellent. PVOs can call FS at any time to for advice on technical issues. PVOs never felt that FS was imposing anything; they always coordinated as equals. ―Technically, it is worthwhile to work with them. It’s an elite staff.‖ The role of FS and the PVOs could have been more clearly defined. FS financial problems have affected some of the PVOs. FS was not clear about finances, and did not ask for financial participation in advance, which has on occasion affected a local organization negatively. ―They should be more open, to say clearly that they have or they do not have money. They also should be more organized; several times they have postponed meetings and put the PVO in a difficult situation.‖ The PVOs see the future role of FS as centering on the QIP; however, they need greater support in the provinces. To build up M&L at DPS is a key activity, but it has to be conducted almost continuously due to high DPS staff turnover. New people coming in are generally inexperienced. ―FS should support the PVOs in the provinces; their presence is important, and they should go to the provinces at least quarterly.‖ Subcontractors AUSTRAL The FS project took a long time to start. When the consultancy was requested, the two consultants assigned to the project had been committed to other work, but they were able to conduct a needs assessment in MISAU on M&L training. They also conducted focal groups at central level and provincial level (Gaza) on M&L practices. Then they drew up a preliminary proposal for an institutional development strategy. The contract was terminated by mutual consent. Health Alliance International There is a general feeling that the subcontract was poorly managed. There was insufficient staff capacity (50% of one person) allocated to grants management, particularly during the selection and placement phase. Once the students had been placed this would have been sufficient. There was high turnover among HAI staff allocated to the project, which could have resulted in insufficient definitions of the boundaries of grants available to MISAU. 6 FORTE SAÚDE MID-TERM EVALUATION Helen Keller International (HKI) The role of HKI was to provide technical advice on nutrition. FS was not responsible for the initiation of most of the work HKI did. The only exception is the Surveillance Sentinel Sites. The Sentinel Sites are a very important activity for MISAU. The database has been developed and the system is ready for piloting in Gaza and Maputo. Guides have been developed including calculations required. HKI feels that USAID’s expectations for FS were very high given the grant amount, and the project was too broad. However, FS manages the constant changes from MISAU and USAID well. The contract between HKI and Chemonics has never been revised—it is still based on the original six-task contract. HKI hopes to see the following from FS: fewer meetings, fewer reports and more action, more of the work being done in districts and facilities, training of trainers in training skills and clinical training, building capacity in the training centers, a curriculum for nutrition technicians, less focus on policy, and placement of a full-time nutrition specialist from HKI with FS. IT Shows IT Shows was responsible for information systems (IS) and information communication technology (ICT). A person from IT Shows was based at the FS office until November 2007; he worked on the Hospital Information System (HIS), designing forms for presentation to MISAU. While IT Shows is still a subcontractor for the FS project, its assistance is provided only on request. FS currently has a Mozambican acting as ICT specialist because USAID has asked that it make use of local capacity in the area of ICT. There was no person from IT Shows available to be interviewed during the evaluation. JHPIEGO FS is an extremely challenging project, and Chemonics management of it is thought to be extremely poor. Local staff were not empowered to make decisions or negotiate with USAID, MISAU, budget, or program decision makers. The experience of working with FS has not been positive—processes are not transparent. Partners should have more autonomy in program decision making. In the design phase JHPIEGO presented Chemonics with a comprehensive budget to undertake technical activities responsive to the proposal’s required outcomes. It was told to remove direct costs from the budget for training, travel, etc. because Chemonics would take care of these. Only when the project started were they told that there was no budget for these activities. FS has done very little in three years given the amount of funding and the number of people. They are too involved in coordination and administration. JHPIEGO does not have an updated contract that reflects the budget cut and the activities that were accordingly amended. Technical material developed by and with the assistance of FS is excellent. FS has drafted the only material that incorporates prevention of mother-to-child transmission (PMTCT) and malaria in antenatal care. This could be launched and trained on at a large scale to ensure that the project leaves behind a tangible product. Partners In the MCH arena, the agencies that give ―daily‖ support to MISAU are UNFPA, UNICEF, WHO, and FS. The greatest constraint in working with MISAU is a shortage of human resources. There are not enough staff, high staff turnover and limited staff capacity. Partners organize themselves to optimize support to MISAU and to help MISAU be organized. Partners and MISAU staff always support one another. There is always someone from MISAU leading any work. Partners have a very positive impression of FS and the work they have done with MISAU. FS has done a lot in drafting policies and guideline documents, and provides very good and desperately needed technical support to MISAU. The participation of FS in the SWAp SRH & CH Group has had a very positive impact on the capacity of ministry staff. FS shares a lot of information on technical issues, which is seen as a very important learning experience for MISAU. The FS effort is a joint one, is not vertical, and it should make alliances with organizations that can ensure project continuity. FS should not close without ensuring that another organization, FORTE SAÚDE MID-TERM EVALUATION 7 project, or program takes over the work it is doing or there will be a large gap in resources available to and frequently used by MISAU. FS Staff FS staff point out its strengths: qualified human resources; a capable and cohesive team; acknowledgement from MISAU that it can always get technical support from FS; a positive technical reputation; excellent relationships with colleagues and MISAU technical staff; and a good work environment. The team sees beyond its obligation and has a vision of Mozambique’s needs. Feedback about the internal management style was very positive. As weaknesses FS staff mentioned the reduction of technical staff time and consequent lack of availability; financial issues; frustration; budget cuts that had a negative impact on the relationship with MISAU; and the inability of FS to say no to MISAU, which sometimes means a lot of time spent addressing small issues that have been inflated by MISAU. FS consider as challenges joint articulation with MISAU; staff turnover at MISAU; limited capacity of MISAU staff; financial restrictions; insufficient travel to the field; and the limitations of the MISAU NHIS. At the beginning FS had some difficulties with MISAU, whose expectations for the project were not met. Substantial time had to be dedicated to establishing relationships with MISAU. The work plan was approved only after 10 months of negotiation. The result anticipated by FS is to have systems designed and implemented. Things are moving in the right direction and where they need to be. MISAU views FS as a credible and valuable resource. FS has an outstanding presence in the ministry’s MCH area, but the financial restrictions mean that FS has not been able to deliver on some activities MISAU was expecting. Coordination has been an important FS role. It has very positive relationships with PVOs. With regard to AI, FS tested training and communication materials in a workshop in Zambézia. This material should be revised because it is both extensive and complex; it should be less theoretical and more practical. SO 1. More effective information, monitoring, and evaluation systems Staff believes that FS has accomplished a great deal. Data collection and collation forms have been finalized for neonatal health, maternal health, and the Expanded Program of Immunization (EPI). Another 16 forms are being drafted; which when approved they will be integrated them into the NHIS by the Department of Health Statistics. The NHIS is currently fragmented, with information feeding into it from subsystems for each department, based on inappropriate and extremely outdated technology. FS drew up a scope of work for a consultant to evaluate the entire HIS and put forward a proposal for an integrated and consolidated system, and a service provider was selected but due to financial constraints no award has been made. A Quality Information System for RH/CH has been designed, and the database is accessible and includes reports on standards. FS developed the methodology for the KPC survey. The project also worked with PVOs to improve how data are reported to USAID, but it has no management mandate with PVOs, which do not have to take FS advice or report to FS. PVOs pass data to FS in the spirit of collaboration. FS is driven by MISAU priorities to the point where its own priorities are set aside. If FS does not respond as MISAU shifts priorities, it will not be able to work with MISAU at all. The M&E processes are hampered by the weakness of the MISAU M&E system and low culture in data analysis at peripheral level. Indicators do not have management utility for MISAU—there are no feedback loops to the provinces, districts, or health facilities. There is also a lack of understanding in MISAU that M&E is connected to ICT and IS. Each department should have integrated indicators, and FS is working with departments on results frameworks. Because MISAU is nationally connected, it is very difficult to work only in the target areas. There is a need for a results framework for the whole of MISAU. Because data for USAID are connected to data from MISAU, the two issues are codependent and hinge on the same success factors. 8 FORTE SAÚDE MID-TERM EVALUATION SO 2. Policies and strategies updated and implemented in the target provinces This activity is sufficiently advanced. Documents written with the assistance of FS have the support of MISAU, which has recognized the importance of strategy and policy. FS staff have contributed to the drafting of 16 policy and strategy documents. They still need to work with MISAU to update Technical Norms for Sexual and Reproductive Health. The challenge now is to disseminate those documents. SO 3. Management, leadership, and quality improvement strengthened FS started to mobilize quality improvement activities in November 2006. The working group reached consensus for an action plan that the Minister authorized in December 2006. At the beginning of 2007 FS started a series of workshops with MISAU and the partners to arrive at quality standards and create an instrument for performance measurement. The Minister approved the instruments in October 2007. Meanwhile FS prepared training materials for provincial teams. The pilot phase was conducted in six provinces with three health units each. In May 2007 it began training workshops, which were followed by a QI baseline exercise conducted in the health centers, and then by workshops to evaluate the results and design an action plan for each center. Some simplified technical texts and brochures were written and distributed on humanization and quality for health care assistance to women and children during delivery and birth. M&L has been incorporated into the Quality Improvement Modules. FS is also preparing a tailored M&L training course for the DPS. It was anticipated that evaluations would be carried every three months, but actually there has been a longer interval between evaluations. The second evaluation round started early in 2008 and was followed by the second training module. FS conducted some theoretical and practical clinical courses on updated procedures in SRH and newborn care. The third round of evaluation has been completed and the workshops were to be held in September 2008. FS is developing a database for the results. ―The fact of, for the first time, having a strategy applied in the Health Units is excellent. It’s a MISAU central level plan that touches people at the Health Units. We could already see small changes that make a great difference.‖ The main constraint has been the lack of resources for supervision trips to the provinces. SO4 –Strengthened human resources training processes The basic nurse training courses were primarily a financial support for the Training Institutes, but these groups participated in the quality improvement workshops, and the professors are willing to include this topic in the School of Nursing curriculum. FS also supports master’s degree courses in universities in Maputo and South Africa for 11 professionals designated by MISAU. Next steps FS could be a link between the central level and the field. It could conduct a study of the NHIS and a needs analysis. It could also support PVOs in M&E and ICT. FS should complete the work with all departments to draft M&E plans in line with their activity plans. Different departments should be supported to develop results frameworks, training, and workshops and advocate for institutionalization of M&E processes. FORTE SAÚDE MID-TERM EVALUATION 9 The MOH could be helped to develop a culture where information is valued and used to enhance data management quality. Central MISAU staff need support in implement documents already developed. Training of trainers (including the Training Institutes) would help guarantee sustainability of the QIP. There is need for clinical training in child health (newborn resuscitation), and for a clinical course in SRH. FS needs to spend time in the field doing training follow-up, and working with PVOs, DPS, and health units. The QIP needs to be rolled out. The quality improvement system needs to be implemented. There is a need for management training and strengthening at district and provincial levels. AI training and communication materials should be produced and distributed. TECHNICAL Findings are presented by strategic objective, with detailed findings in response to specific questions posed in the evaluation SOW. One section details the findings related to project management and reporting. SO 1: More effective information and monitoring and evaluation systems FS Project Objectives 1. Strengthen capacity of the HIS for generating and sharing quality data/information in RH, Child Health CH, EPI, nutrition, and malaria. Conduct a review of the current situation in all these areas. 2. Develop the capacity of RH, CH, EPI, nutrition, malaria, and epidemics sectors to conduct effective M&E. 3. Monitor the health activities funded by USAID in the areas of RH, CH, EPI, nutrition, malaria, and epidemics, in collaboration with the MISAU. How did the project help PVOs to strengthen information systems, communication technologies, quality performance, and M&E? FS has provided support for the PVOs in data management and quality processes for data that is feeding into the USAID SO 8 M&E system through regular meetings with PVOs to discuss M&E, identify limitations to data quality, and discuss ways to address data quality issues (14 formal meetings between March 2007 and June 2008) and by designing new data collection forms for IR3 indicators as well as a complementary job aid. PVOs are now using the forms. FS drew up a template PVOs can use in consolidating MISAU data for reporting to USAID and the MISAU. It also helped the PVOs complete the form for the first two reporting periods and is now supporting PVOs indirectly by providing TA. FS also helped the PVOs by developing methodology and tools for the KPC survey, having the KPC survey approved by the Commission of Bioethics, and training PVOs on implementing the KPC. Baselines for KPC data have been obtained by using a combination of national data and KPC data collected by the PVOs early in implementation. FS has also done extensive advocacy within the MISAU for community-based data collection. How has the FS project built the capacity of RH, CH, EPI, nutrition, malaria, and epidemic sectors in the MISAU to conduct effective M&E processes and practices? FS conducted a rapid assessment of MISAU M&E processes and helped the MISAU draft an action plan to address the challenges, though the action plan has yet to be carried out in full. One of the outcomes of this assessment was to highlight the need for a comprehensive assessment of the NHIS. In response to an SOW for this work, three tenders were evaluated, but funding constraints have prevented award of the tender. In September and October of 2006 FS conducted a five-session workshop to set indicators for oral health. In November it also organized and conducted workshops to evaluate MISAU M&E for child health, malaria, 10 FORTE SAÚDE MID-TERM EVALUATION tuberculosis, and HIV/AIDS programs. FS worked with the MISAU to draft results frameworks and M&E plans for various technical target areas and to ensure that M&E is incorporated into the strategic plans for specified technical areas (see TABLE 1). TABLE 1: TECHNICAL AREAS THAT HAVE RECEIVED M&E SUPPORT FROM FS Technical Target Area Results Frameworks M&E Plans Strategic Plans National Institute of Health  Avian influenza  Leprosy  School and adolescent health  Sexual and reproductive health   Pharmaceutical Department  Noncommunicable diseases    Child health   National Partnership for Mother and Child Health  Community involvement    National Malaria Prevention and Control   Nutrition   Health Sector Strategic Plan 2007–2012  Expanded program of immunization    Infant and neonatal health   Health education  Mental health  Quality improvement  Malaria   FS also worked with the MISAU to develop updated data collection forms (Table 2). FORTE SAÚDE MID-TERM EVALUATION 11 TABLE 2: DATA COLLECTION FORMS DEVELOPED BY MISAU WITH THE HELP OF FS Name (HIS Code) Level Status EPI register book (A01-A) Health facility On national roll-out Daily register for fixed/mobile outreach (A01) Health facility Mobile brigade On national roll-out VAT daily register for fixed/mobile outreach (A02), pregnant and reproductive age females Health facility Mobile brigade On national roll-out VAT daily register for fixed/mobile outreach (A02- A), workers, and students (non-rep age) Health facility Mobile brigade On national roll-out EPI aggregation forms (A03), BCG, DPT, HepB, polio, measles Health facility On national roll-out EPI aggregation forms (A03-B) for all VAT Health facility On national roll-out EPI aggregation forms (A04 to A06) District, province, national On national roll-out Neonatal case management form (no code) Health facility Approved, roll-out pending Neonatal aggregation form (no code) District, province, national Approved, roll-out pending Antenatal consultation form (no code) Patient Being revised Register book for antenatal consultation (B07) Health facility Being revised Maternity register book (delivery room) Health facility Approval pending Family planning form (no code) Individual Being revised Postpartum and family planning form Health facility Being revised Gynecology register book (no code) Health facility Being revised Maternity monthly summary (B06) District Being revised Antenatal consultation, daily register Being revised Mother and child health monthly summary— antenatal care (B08) Health facility Being revised Emergency obstetric care—monthly and quarterly summary Health facility Being revised Child at risk—consultation form Individual Being revised Child at risk—consultation register book (B17) Health facility Being revised Child at risk—monthly summary (B14) Health facility Being revised MCH/nutritional surveillance—monthly summary (B03) Health facility Being revised MCH/nutritional surveillance—monthly summary (B04) Health facility Being revised 12 FORTE SAÚDE MID-TERM EVALUATION FS provided IT equipment, installed it, and hired a consultant to set it up. It drafted M&E training materials and a training plan for the MISAU and the Ministry of Women and Social Development with the MEASURE project. FS created and facilitated M&E training for the DPS in Tete in May 2008. FS also helped the MISAU to integrate intermittent presumptive treatment in pregnancy into the MCH information subsystem. In November 2007 FS M&E and IT specialists organized and participated in a three-day workshop on reorienting and reorganizing the NHIS, attended by 57 participants from the MISAU and partners. As a result, a bottom-up philosophy was integrated into NHIS planning, and the MISAU has now approved and is beginning to implement a roadmap for NHIS, although very slowly. An FS M&E specialist has been participating in the SWAp technical working group for M&E at the request of USAID, and at the request of the MISAU, FS participated in the working group to formulate the National Health Strategy for 2007–2012. How has the project monitored the health activities funded by USAID in the areas of RH, CH, EPI, nutrition, malaria, and epidemics, in collaboration with the MISAU? In December 2006 FS organized and facilitated an annual coordination meeting of USAID, the MISAU, and all the NGOs funded by USAID. FS has provided extensive input into the USAID SO 8 Performance Monitoring Plan (PMP). The source data PVOs are reporting into the USAID SO 8 M&E system information related to activities at the facility and community level. At the facility level the data include conventional MH, training, Vitamin A, malaria, and vaccination data points, which are collected by nurses through conventional paper processes. Data are collated at the facility level and submitted irregularly to the district level, where it is aggregated. Aggregate district data are then submitted, irregularly, to the provincial level for further aggregation. So far the MIS is a paper-based system, but at the provincial level data are captured electronically. Final aggregation takes place at the center. PVOs have conducted training at facilities on data collection, but high staff mobility makes it very difficult to see the benefits in terms of data quality. Where possible the PVOs obtain data for specific time periods directly from the facilities. If it is not possible, data are obtained from the district level, or district data is obtained from the provincial level. Recognized limitations to NHIS data quality include: High staff mobility and limited staff capacity (both in numbers and qualification) in facilities and throughout the MISAU; There has been no country-wide training for individuals working with data; Definitional issues raise problems of validity and consistency: e.g., at what point is a child considered fully immunized? For how long after giving birth are maternal visits considered postpartum? Incomplete reporting from facilities: One reason is that nurses are too busy, so they start each record but then at the end of the day must try to recall the details to complete them; Fabrication of data at the facility level: Often data are copied from the record above, or whole pages of blood pressures, etc., are identical; Data collection forms that are not user-friendly: They require untrained individuals to interpret what is required and provide duplicate information; Outdated data collection forms: For instance, Vitamin A is not included; Frequent manipulation at district, provincial and central level for political and personal reasons; Lack of data quality control processes; Data that flows from the bottom-up only, with no horizontal or top-down feedback mechanisms, with the result that individuals who not understand the necessity for data collection do not buy into the system and do not collect quality data; Severely limited capacity and no motivation in the MISAU for data analysis; and No demonstrated use by the MISAU of data from the NHIS. FORTE SAÚDE MID-TERM EVALUATION 13 However, the NHIS is the primary source of data reported to USAID for health service indicators. Community-level data are collected for such indicators as numbers of trained community leaders, trained CHBC workers, and home births and deaths. The community information system uses picture-based data collection tools to reduce the literacy requirement. These data are aggregated at the community level. In districts where community data are submitted to the District Health Department (DDS) and aggregated with facility data, the data reach the NHIS. However, most districts do not submit community data into the NHIS, although the quality is known to be better than that of facility data. The PVOs are putting more emphasis on community health information systems. Pathfinder and World Vision in particular have created instruments, trained on data collection and collation, and mentored community leaders in establishing reliable information systems. SO 2: Policies and strategies updated and implemented in the target provinces FS Project Objectives 1. Help the MISAU evaluate the current status of policies, strategic plans, and guidelines in the areas of RH, CH, EPI, nutrition, malaria, and epidemics. 2. Help the MISAU develop, update, and adopt policies and strategies. Actions planned in 2006 included drafting RH, CH, and nutrition action plans, and policies and strategies to fight epidemics and endemics. 3. Help disseminate, implement, and monitor policies and strategies in the target provinces (including community mobilization), in coordination with MISAU and NGOs. How has the project assisted MISAU to evaluate the current status of policies, strategic plans, and guidelines in the areas of RH, CH, EPI, nutrition, malaria and epidemics? FS conducted a needs assessment in 2006 to verify the status of SRH and CH policies, strategies, and guidelines. The main findings were that There is high political commitment to the area of SRH/CH. Delays in document development, approval, and dissemination mean that interventions are implemented before policies are available. Decisions are made based on limited institutional capacity and limited knowledge of the needs. The M&L skills of MISAU senior staff are limited. Although almost all documents cite quality of services as a priority, no systematic action had been taken. A great effort has gone into producing documents on services supervision, but many constraints were found in implementation, particularly a punitive approach. Although vital, the HIS is weak, centralized, shaped by subsystems that do not communicate with each other, and of limited reliability. M&E is sporadic and usually mixed up with supervision and disconnected from the bulk of the HIS. A major problem in the health sector is the scarcity of qualified personnel. The main documents found were the Plan for Absolute Poverty Reduction (PARPA II) 2006–2009 and the Government 5-year Plan 2005–2009, both approved in 2005. These established that investments in health personnel should be increased to reduce maternal and child morbidity and mortality due to endemic and epidemic diseases such as HIV/AIDS, malaria, and tuberculosis. Other documents found were the National Health Policy (1975), Safe Motherhood (1990), First Integrated National Plan for MCH/FP/EPI/SEA (1995), the National Strategy for Maternal and Neonatal Morbidity and Mortality Reduction 2000–2005 (1999) and the National Strategy for Integrated Management of Childhood Illness 2000–2005 (1999). Useful progress was made between 2005 and 2006 in formulating health policies, strategies, and plans: The Community Health Department was upgraded to the National Directorate for Health Promotion and Disease Control. 14 FORTE SAÚDE MID-TERM EVALUATION The National Health Policy Declaration (2005) reaffirms emphasis on quality primary health care by, e.g., humanizing health services, and sets goals for 2010–2015 for reductions in maternal and child morbidity and mortality. Its stated main strategy is the provision and expansion of equitable access to quality health care. The Strategic Plan for the Health Sector (PESS) 2001–2010 and the Economic and Social Plan (PES) are based on the government priorities. There is also the Multisectoral National Strategic Plan for HIV/AIDS 2005-2009 (PEN II). Gender policy and strategies were formulated. Since 2006 FS has been of great help to the SRH/CH group in developing MCH policies, strategies, and guides. How has the project assisted the MISAU to develop, update, and adopt policies and strategies? FS technical staff worked closely with MISAU SRH/CH and partners (UNFPA, UNICEF, WHO, etc.) to produce 15 documents (Table 3). FS support was crucial because of its technical, organizational, and writing expertise. TABLE 3: DOCUMENTS PRODUCED BY MISAU, ITS PARTNERS, AND FS Political Documents Developed Approved 1. National Policy for Sexual and Reproductive Health and Rights 2006 w/Minister 2. National Policy for Child Health 2006 w/Minister 3. Road Map to Accelerate the Reduction of Maternal, Perinatal, and Neonatal Mortality 2006/2007 2008 4. National Strategy for Neonatal and Child Health 2007/2008 w/Minister 5. National Plan for Breast and Cervical Cancer Prevention and Control 2007 Incorporated into 6 6. National Strategic Plan for Prevention and Control of Noncommunicable Diseases for 2008–2014 2007 2008 (not signed) 7. Global Partnership for Maternal, Neonatal and Child Health 2007 2007 8. National Partnership for Maternal, Neonatal and Child Health 2007/2008 w/Minister 9. Strategy for Strengthening Traditional Midwives Interventions 2008 2008 10. Strategy for Pregnant Women Awaiting Housing 2008 2008 11. Communication Strategy for Maternal, Neonatal and Child Health Promotion 2008 Documents to be finalized after Minister comments 12. Certification Norms for EOC/ENC (Emergence Obstetric and Neonatal Care) 2007/2008 13. Guideline for the Auditing National Committee on Maternal, Perinatal and Neonatal Deaths 2007/2008 14. Nutritional Development in Mozambique - An Action Plan for 2007– 2010 2007–2008 w/Minister 15. Strategy for Food Security and Nutrition II 2007–2015 2007 2008 16. National Action Plan for Nutritional Development, 2007–2010 Ongoing 17. National Strategy for Family Planning Ongoing 18. Strategy for Commodity Security on SRH In process 19. Integrated Strategy for the Achievement of MDGs 4 and 5 In process FORTE SAÚDE MID-TERM EVALUATION 15 In order to draft these documents, FS helped MISAU to organize and conduct A national seminar on SRH in August 2006 A national seminar on CH in October 2006 A workshop on Cancer Prevention and Control in May 2007 Two trips to test and apply questionnaires for a rapid needs assessment on SRH and CH for the Global Partnership for maternal and child mortality reduction document in April 2007 A meeting with traditional midwives in Catembe in February 2008 FS provided a framework for the process for document development for three documents in Table 3 and has taken the lead in the process for six others with the MISAU. How has the project assisted the MISAU in disseminating, implementing, and monitoring policies and strategies to the four target provinces? To disseminate policies and strategies, MISAU and its partners, including FS, continue to write and revise technical manuals, training materials, and lists of materials needed to facilitate delivery of services (see Tables 4 and 5). They have also given a series of clinical courses (see below, under SO 3). TABLE 4: DOCUMENTS DRAFTED BY THE MISAU WITH FS ASSISTANCE Technical Documents Developed Approved 1. Updated Map Books for IMCI 2007 2007 2. Breastfeeding Manual 2007 2007 3. Text and folder on Humanization of Childbirth and Neonatal Care 2007 2007 4. Folder on Quality Health Care to Woman and Newborn during Delivery 2007 2007 5. Manual for Nutritional Surveillance Sentinel Sites National Program Implementation 2005–2007 2007 6. List of materials and equipment to be acquired for implementing the Cervical Cancer Prevention and Control Plan 2008 2008 7. Monitoring tool for the Basic Nutrition Pack 2008 2008 8. Survey Questionnaire for Nutrition Activities at the Community Level 2007 Pending 9. Revision of the Nutrition Component of the PMTCT MISAU Manual Revised in 2006 Pending 10. Revision of Health Education Manual 2007 pending 11. Training materials for Cervical and Breast Cancer Prevention and Control In process 12. Revision of Nutrition Technical Course Curriculum In process 16 FORTE SAÚDE MID-TERM EVALUATION TABLE 5: OTHER FS ACTIVITIES AND PRODUCTS Product Comment 1. Uganda Conference on Postpartum Hemorrhage Prevention Poster elaborated and presented: Postpartum Hemorrhage Situation in Mozambique 5 participants from Mozambique April 2006 2. Two training courses in IPT malaria in Gaza and Inhambane provinces April 2006 3. 20 technical update sessions on SRH and CH April 2006– Feb 2008 4. Technical, financial, and logistical support for the African Union Health Ministers Conference 2 posters 2 papers SRH in Mozambique video production Conference Annals Book with CD-ROM September 2006 30 copies 300 copies 5. Participation of 5 professionals in a course in Thailand on prevention of cervical cancer through cervical inspection using acetic acid and treatment with cryotherapy Action plan for Mozambique developed February 2007 6. 2 training courses on Nutrition Sentinel Sites for MISAU staff April 2007 and January 2008 7. Training course on Nutrition Sentinel Sites for provincial medical chiefs, nutrition technicians, those responsible for community health October 2007 8. Support in organizing the launch of ―Presidential Initiative for Maternal and Child Health‖ February 2008 9. Support in organizing the National Meeting on Maternal and Child Health February 2008 SO 3: Management, leadership, and quality improvement strengthened FS Project Objectives 1. Support MISAU development of a strategy to strengthen institutional capacity in management, leadership, supervision, and quality improvement in RH and CH at the central level and in the target provinces. 2. Assist the MISAU strengthen HR capabilities in management, leadership, supervision, and quality improvement in the areas of RH and CH at the central level and in the target provinces. 3. Help the MISAU to institutionalize the improvement of management and quality in the areas of RH and CH at the central level and in the target provinces. How has the project supported the MISAU in developing a strategy to strengthen institutional capacity in management and leadership? During the needs assessment conducted by Austral, no respondents were able to provide information on the actual situation of MISAU M&L. In reviewing documents it is possible to verify the need MISAU staff has for improving their leadership abilities and management processes. FS has done some work in this area, but the high staff turnover in MISAU means this important area needs more support. A preliminary strategy to strengthen MISAU M&L was developed and an annual calendar for thematic meetings set. Five focus groups were carried out (1 at the central level, 2 in Zambézia, and 2 in Gaza); these were part of the strategy of collecting needs assessment information, but the thematic sessions called for in the strategy were not conducted. This strategy takes a building-block approach, using participants’ experience, learning by doing, and including a variety of methodologies (focus groups, meetings on management topics, working groups). The strategy was to be FORTE SAÚDE MID-TERM EVALUATION 17 carried out for the National Directorate of Promotion and Prevention in coordination with the Department of Human Resources. This strategy was never approved nor fully implemented. By the time MISAU was ready for M&L activities, Austral’s consultants had been reassigned and Austral and FS had agreed to terminate the subcontract. This caused a further delay in implementation of M&L activities. In 2007 FS hired two individuals to undertake these activities, but they performed very poorly. In January 2008 FS and the MISAU Human Resources and Management Directorate began work on a 10-module M&L course for Provincial Health Directorates that MISAU is now reviewing. MISAU and other partners also drafted the Quality Improvement Instruments (part 11 of the instrument), but these do not seem to have been implemented yet. To what extent has the project helped the MISAU to strengthen its human resources M&L capabilities? FS has prepared a draft proposal for the 10 M&L modules. A pilot workshop took place in Gaza (August 2007), for PVOs providing support on strengthening management capacity; the 15 participants represented Project HOPE (an FS collaborator), Save the Children, World Relief, ICAP, and Doctors Without Borders. M&L session topics were also introduced in quality improvement training modules and clinical courses. Topics included mobilizing resources, team work, interpersonal communication, improving M&L capability, and managing change. How has the project helped the MISAU to institutionalize the improvement of management and quality in the areas of RH and CH at the central level and in the target provinces? No activities were identified.1 How has the project supported the MISAU in developing a strategy to strengthen institutional capacity in quality improvement and supervision in RH and CH? The needs assessment completed at the beginning of the project showed that in Mozambique, as in countries worldwide with a shortage of resources, there are not enough health providers to meet demand, providers are overburdened, and they have low motivation and insufficient knowledge and skills to perform technical activities and system management. MISAU is addressing the challenges through health sector reform and effective decentralization and institutional capacity improvement. Improving the quality of health services is a priority for the government and is part of all governmental and health policies and plans, but until 2006 no national strategy was identified. In 2006, MISAU and its partners (among which FS has been very active) developed The Action Plan for SRH and CH Services Quality Improvement, which the Minister of Health approved in December 2006. This document specifies how standards will be defined, implemented, and supervised and progress measured and recognized. It also states how the implementation process will start and be expanded, how training will happen and how the methodology will be institutionalized. After a seminar with 38 MISAU staff and partners in February 2007 and a five-day workshop for 58 participants in March 2007, quality standards for SRH and CH were defined. To what extent has the project helped the MISAU to institutionalize improvement of quality in RH and CH and strengthen human resources in those areas? In March 2007 quality improvement instruments were tested in two health units (Centro de Saúde 1º de Junho, Maputo City, and Centro de Saúde de Ndlavela, Maputo Province). The instruments were finalized and approved by the Minister of Health in September 2007. Meanwhile, FS helped develop training materials, including an implementation guide and methodology description and instructions on how to use and reproduce the instrument. A three-day training workshop on Module I was conducted in Maputo for teams from the provinces of Maputo, Gaza, Sofala, Manica, Zambézia, and Nampula in May 2007. The workshop was attended by 69 participants from 1 The day before the evaluators left the country, RH/CH staff, FS, and other partners met with the Vice Minister of Health and it was decided that the QIP will be incorporated into MISAU’s work plan to be implemented in health units in all provinces. 18 FORTE SAÚDE MID-TERM EVALUATION the 18 health facilities (3 in each province) selected to participate in the pilot, and by provincial DPS and NGO staff and MISAU staff from the central level. FS provided technical support for the QI process baseline evaluation and development of the health unit QI action plan, making 14 site visits from July through October 2007. To support the QIP, FS, the MISAU, and their partners, produced simple texts based on the quality standards as a job aid for home health workers to better assist woman and child during delivery. The Minister of Health approved the documents in December 2007, and 2,000 folders were distributed on ―Humanization of Childbirth and Neonatal Care‖ and ―Quality Health Care to Woman and Newborn during Deliveries.‖ A second evaluation of health units was held in January and February 2008. Also in February 2008, Module II training in Maputo for the pilot provinces was attended by 56 persons (36 from health units, 6 from DPS, 5 from PVOs and 9 from MISAU central and other partners). Beyond the methodology training, attendees reported on the improvement achieved by each health unit, so there was considerable experience exchange. All 18 health units are developing their action plans for the next three months. The 14 health units evaluated twice (Table 6) show an average quality improvement of 23 percent; the change ranged from 52 percent for 17 de Setembro Health Center in Quelimane, Zambézia, to 8 percent in Macurrungo Health Center, Sofala. Thus, during the pilot period every facility showed some improvement in its quality scores. TABLE 6: QUALITY IMPROVEMENT IN PILOT FACILITIES Province Facility Result at Baseline Result at Second Assessment— Feb 08 Difference Zambézia Quelimane: 17 de Setembro Health Center 16% 68% 52% Mocuba Rural Hospital 32% 60% 28% Ile Health Center 38% - NA Nampula Nampula: 25 de Setembro Health Center 24% 55% 31% Monapo Rural Hosptal 26% 38% 12% Nacala Porto General Hopital 24% 47% 23% Gaza Patrice Lumumba Heath Center 32% 55% 23% Chicumbane Rural Hospital 22% 41% 19% Manjacaze Rural Hospital 20% 61% 41% Maputo Manhiça Health Center 38% - NA Matola II Health Center 31% - NA Moamba Health Center 14% - NA Manica Catandica Health Center 11% 36% 25% Machaze: Chitobe Health Center 14% 36% 22% Chimoio: 1o de Maio Health Center 6% 43% 37% Sofala Macurrungo Health Center 28% 36% 85 Muxungué Health Center 23% 34% 115 Caia Health Center 17 38% 21% Average for all facilities 23.11% 46.29% 23.18% FORTE SAÚDE MID-TERM EVALUATION 19 After the second evaluation, the MISAU and FS found that further clinical training was needed to improve health providers’ technical skills. They therefore revised and prepared new materials on SRH and neonatal care and acquired anatomic models and instruments for practical training during courses, some of which were given to the Training Institutes in Beira and Nampula and some reserved for MISAU central level to support future training. Two courses on SRH and neonatal care were conducted, one in Nampula April 14–19, 2008, for 25 participants and one in Sofala April 21–26 for 28 participants. The participants came from health facilities, DPS, the Training Institutes, and local NGOs. The curriculum covered theory, in-room practice with anatomic models, and practice at the health centers in order to update the main maternal and neonatal health skills, which included family planning and training on cervical cancer screening using visual inspection with acetic acid. In mid 2008 FS prepared materials for Module III training, thus continuing to build HR capacity for QIP. Training was scheduled for the first week of September. A database to monitor the progress of health facilities in QIP has been set up and was introduced to the participants in Module II (see Appendix F for screen shots). FS is refining this database based on comments provided and expects to train individuals to use the system as part of Module III training. SO 4: Strengthened human resources training FS Project Objectives 1. Identify candidates and make arrangements for postgraduate training in Mozambique and abroad in the fields of public health, epidemiology, RH, and other areas related to the activities described in the contract. 2. Support training in the areas of RH and CH, including aspects of nutrition and prevention, and prevention and treatment of malaria in Mozambique. 3. Support the MISAU in long-term training of nurses in selected provinces. What progress has the project made in terms of making arrangements for postgraduate training? What impediments have there been to implementation of this key component of the project, and what modifications need to be made to improve performance in this component? Six candidates were identified for study abroad. All attended English classes and took the language test twice. Three candidates were admitted to Pretoria University for a four-semester course. Another was sent to the U.S. to improve her English skills but was not able to achieve the necessary minimum scores after three months and came back to Mozambique. The MISAU cancelled the scholarship of one who was accepted at a Brazilian university. The one that took his place did not pass the English language exams and chose not to continue. The language grade of the last was not strong enough for a university in England. Meanwhile she became pregnant; FS is now waiting her confirmation to start studying at Pretoria University in January 2009. FS made available to MISAU 10 scholarships at the national level. The MISAU chose not to include FS and HAI in the selection process. Of the 10 candidates eight began MPH studies at the Eduardo Mondlane University in August 2007. How successful has the project been in providing support to training in the areas of RH and CH, including aspects of nutrition and prevention, and prevention and treatment of malaria in Mozambique? Training in these areas was been reported under SOs 2 and 3 because they are related to implementing those strategies and the QIP. FS was asked to help the Department of Noncommunicable Diseases, the MINAG, and NGOs to design and implement capacity-building activities related to AI. The MISAU and other agencies had not previously included this training in their annual action plans. Three training of trainers’ courses, in which 55 persons from all provinces participated, were conducted in February 2007 in the South, Center and North regions. FS also assisted the MISAU and the MINAG in developing curricula for local technical providers and IEC materials for the public. In November 2007 those 20 FORTE SAÚDE MID-TERM EVALUATION materials were tested in a Rapid Response to Avian Influenza training course conducted by FS, the MISAU, and the MINAG in Quelimane for 25 participants. The curricula and IEC material were redesigned in response to lessons learned through the course results and participant inputs. The materials still need to be approved by both ministries before they can be disseminated. How successful has been the long-term training of nurses in selected provinces? Are modifications needed to improve this component of the program? If it has been successful, please describe. Two coordination meetings were held to define partner roles and responsibilities for financial support to nursing training in the Nampula and Quelimane Health Sciences Institutes and to simplify the support and reporting mechanisms. FS provided additional financial support for three MCH nurses groups, two in Nampula and one in Quelimane; 92 nurses were upgraded from basic to mid-level skills. The project also acquired photocopy machines for the training centers in Nampula and Lichinga and paid for student lodging in Nampula and for student kits. The Nampula and Beira Training Institutes received anatomic models (see SO 3 report). Male Circumcision FS contributed to an assessment of the capacity and needs of health facilities to perform small surgeries, such as male circumcision. Data resulting from this assessment are being analyzed and a report is being prepared. PROGRAM REPORTING AND MANAGEMENT While FS does have a PMEP, it does not report to USAID on those indicators. The main purpose of the FS PMEP is to keep staff focused on objectives and desired outcomes to ensure that project activities lead to the anticipated outcomes. FS is tracking data for indicators where possible. Although there are targets for FS, these are not documented in the PMEP, and progress toward them is not reported to USAID or the MISAU. The focus has been on processes and quality of work rather than on attaining targets. The issue of targets is also tied to the setting of baselines. When the PMP was developed no baselines had yet been established. Initial situational assessment established baselines for some indicators. The MISAU was not inclined to make use of a 0 baseline. (USAID often uses a 0 baseline on the rationale that before a specific project gets underway, no activities have been implemented by that project. However, this is only true if the indicators relate directly to activities of the program—output indicators—not to program outcomes.) The majority of indicators (17 of 27) in the PMEP are outcome indicators. They are important for measuring the impact of the program, although traditionally impact evaluation has been a separate process from monitoring. It takes time after project activities for impacts to be felt, particularly in a situation as fraught with challenges as the MISAU. Moreover, the impacts of the FS project are felt at the provincial and district levels (implementation of policy, procedures, and activity plans, and improved delivery of and demand for quality health services), yet the project is not implementing at those and does not currently have the budget or staff capacity to collect quality data there. The cost-benefit ratio of collecting data is out of balance, particularly considering that FS is not reporting any of it. FS is currently discussing changing its PMEP to emphasize activities over which it has more control. The tool that FS uses more regularly for monitoring and reporting activities is the project work plan. Each year it drafts a work plan in consultation with the MISAU that is submitted first to the MISAU and then to USAID for approval. From the work plan the M&E/IS advisor develops a monitoring tool, the Chronogram, that all staff complete monthly. It lists the operational objective under which each activity falls, a short description of the activity, activity situation at the beginning of the month, planned tasks relating to the activity for the current month, financial information, weeks in the current month for which activity tasks are planned, and activity status at month-end (pending, initiated, midway, advanced, finalized). This information is used to develop a percentage score for each activity that describes activity status for the current period, which then flows into the semiannual reports to USAID. (Details on the method of calculation are presented on Annex B of the FS PMEP.) Due to the 10-page maximum for semiannual reports, only activities that have seen movement during the period are reported. FORTE SAÚDE MID-TERM EVALUATION 21 CONCLUSIONS The evaluators found that while FS has not always addressed every outcome in the contract, it has done a great amount of work to strengthen MISAU capacity in M&E, policy and strategy planning and documentation, and quality improvement. The quality of technical materials produced by and with the assistance of FS staff is excellent. The quantity of work accomplished at mid-point is exceptional, particularly given the small staff. A generally perceived weakness is that FS could have done more traveling and provided greater support to the PVOs and the MISAU in the provinces and districts. The result has been insufficient delivery of intended project outcomes and too little support for actual implementation. LESSONS LEARNED Technical 1. FS built credible, positive relationships with lower-level staff of the MISAU, PVOs, and other partners. These organizations unanimously expressed their opinion that FS has been indispensable in carrying out activities. 2. The methodology of FS—working with their MISAU counterparts rather than for them—has had a very positive capacity-building impact on MISAU staff in all departments. 3. The inclusion of local technical staff in the FS program enhances the vision of FS to incorporate the real needs in Mozambique for MH and CH and aligns program activities to local cultural norms. 4. High staff mobility and capacity challenges in MISAU, in terms of both number and technical capacity of staff, have been a significant problem with every FS activity. 5. Extensive documentation of the processes followed in drafting policies, strategies, roadmaps, results frameworks, activity plans, etc. for the MISAU have enhanced the ministry’s institutional capacity. 6. FS has been very proactive in responding to MISAU needs. 7. FS has been particularly efficient at organizing and providing technical support for meetings and conferences and disseminating conference results. SO 1. More effective information, monitoring, and evaluation systems 1. Uncontrollable factors in MISAU data management render data unsuitable for reporting or decision making. PVOs support facilities with training, nurse salaries, and supervision through visits and logistical support; they do not deliver MH or CH services, yet these are reported on. In normal data quality assessments, an audit note would be issued to the organization. This problem is also apparent in the USAID M&E Plan. 2. Clinical health service indicators are a good measure of the long-term success of the FS and PVO support to the MISAU and facilities. However, these should be evaluated at the close of the project as part of a broad impact assessment, not regularly throughout implementation. The utility of getting this data quarterly is far outweighed by the costs incurred by both FS and the PVOs to collect it, especially given its poor quality. 3. The MISAU NHIS is comprised of numerous parallel information systems that are not well integrated. It uses outdated technology (e.g., MS Access) that is unsuitable for a national data warehouse system. Numerous and well-documented limitations render the data available from the NHIS useless for decision making. 4. FS has insufficient capacity to collect and monitor data for the MISAU and USAID. Data from the NHIS fed by PVOs into the USAID SO 8 M&E system are flawed. The MISAU M&E system requires an absolute overhaul, more effective electronic processes, automated feedback on horizontal and vertical lines, extensive staff training, and continuous support before it can produce good data. M&E support and an integrated approach to data management is needed across the entire MISAU, not just MH and CH (the mandate for FS). 5. FS was tasked to ―assist and monitor USAID-funded activities on a regular basis and collect and monitor data for the MOH annual report and USAID PMP.‖ FS continues to give PVOs extensive TA in M&E; however, the PVOs only have a contractual obligation to report directly to USAID, not to pass data to FS. 22 FORTE SAÚDE MID-TERM EVALUATION SO 2. Policies and strategies updated and implemented in target provinces 1. FS has had extensive involvement in drafting and redrafting policy and strategy documents with the MISAU, but it does not have sufficient political influence to ensure that they are approved, much less implemented. While FS can provide TA to support implementation, it cannot do so until the MISAU decides to implement. SO 3. Management, leadership, and quality improvement strengthened 1. The QIP, even in the pilot phase, has conclusively demonstrated a positive impact, with all 18 participating facilities improving the quality of their MH and CH services. 2. FS has achieved significant MISAU support and buy-in to the QIP; the MISAU has asked that the system be rolled out in all provinces. 3. Including M&L in the quality improvement system was a successful method of achieving these outcomes in the absence of an M&L component in the MISAU. SO 4. Strengthened human resources training processes 1. MISAU was responsible for selecting postgraduate training participants. Selection delays meant that students had too little time for orientation and acquiring language skills. Selection criteria were either not established or insufficiently communicated; FS is accountable for outcomes of a process over which it had little control. 2. Approval of AI activities was facilitated by the link between the MISAU and MINAG. FS worked with both ministries to draft training and communication materials. These were piloted in Zambézia and finalized, but the ministers have not yet approved their distribution. Program and Management Project Planning and Implementation 1. The project design, with a focus on TA for policy and strategy development and implementation, makes project staff accountable beyond their possible authority with the MISAU. FS may lobby for development, approval, and implementation of policies and may assist the MISAU in disseminating and implementing them, but all decisions related to adopting or implementing any policy or strategy are made by the MISAU. 2. The broad scope of the project contract, which was intended to provide flexibility in responding to changing MISAU needs, meant that the MISAU was not clear about what assistance it could anticipate from FS. 3. The project was awarded in October 2005, after a change of government in February 2005, and the contract scope was intentionally flexible, leading to extensive negotiation with MISAU about project activities and unusually long implementation delays. 4. Given that FS was the first Chemonics project in Mozambique, it should have derived greater benefit from the relationships its subcontractors (JHPIEGO, HAI, HKI, and Austral) already had with senior MISAU officials. 5. FS staff should have traveled more to train and support PVO, MISAU provincial, district, and facility staff. Project Finance2 1. Budget constraints since February 2008 were raised in all interviews with FS, MISAU, and subcontractors. The team was unable to get a clear picture of the FS financial situation, with its various modifications, and concludes that in-country staff have too little knowledge of and control over the project budget. 2. Reductions in staff time and activities since February 2008 seem out of proportion to the shortfall in disbursements for 2008 (approximately $300,000), particularly given that (1) FS has not expended the full amount budgeted for nurses training, AI, male circumcision, and long-term training, and (2) FS expenditure 2 A financial audit is beyond the scope of this technical evaluation of progress. FORTE SAÚDE MID-TERM EVALUATION 23 and project duration are well aligned at the midpoint. Any additional funding would carry additional obligations, all of which use the existing resources of the project. 3. The MISAU Health Promotion and Disease Protection Directorate has asked partners to pay for training costs, including FS staff travel expenses. This, combined with complaints about budget limitations since February 2008, has undermined the reputation of both FS and USAID. Project Management 1. Shifts in the focus of the MISAU has resulted in delays and inability to complete some FS activities. 2. The Chemonics head office may not have sufficiently empowered in-country FS management to manage the budget and make financial decisions. There were delays in responding to the team’s budget-related queries, and the team was unable to fully understand the financial limitations beyond those mentioned in interviews. 3. Chemonics provides extensive support to the project through a three-person team at the home office; project staff is satisfied with this support. 4. The interpersonal relationships and management style of FS staff focused on personal and organizational development and capacity building of the MISAU, PVOs, and partner organizations. However, the positive aspects of working ―with‖ and not ―for‖ have often meant that activities took longer than expected. 5. Capitalizing on the previous relationships of consortium members with senior staff in the MISAU could have increased the visibility of project activities and enhanced MISAU’s ownership. Failure to do so may have been because the project is being promoted as a USAID-funded Chemonics project, rather than a consortium. 6. Communication with subcontractors, particularly about the 2008 budget constraints, has been poor. Active subcontractors have not received revised contacts. Two of the original five are no longer part of the consortium (Austral because conflicting consultant schedules resulted in mutual termination, invoking clause FAR 52.243, and HAI because most work with MPS placement was completed and the contract terminated by mutual consent). This has not been adequately communicated to USAID and is perceived as a management issue originating with the Chemonics head office. Although IT Shows is still in the consortium, its assistance is provided only on request, because USAID has asked FS to make more use of local ICT and IT support. Project Monitoring, Evaluation, and Reporting 1. The PMR method of reporting to USAID by activity does not highlight tangible project outputs. While the Chronogram can be a useful monitoring and management tool, it is important that performance information is presented to staff, partners, and donors in an easy-to-read format. Because the PMR reflects only activities that have been worked on in a given period, it is difficult for the donor to track progress. Moreover, the time expended in managing the Chronogram (developing, translating, and retranslating work plans and transferring information to the Chronogram, management and maintenance, etc.) is out of proportion to its benefits. 2. The percentages reflected on the report are misleading: an activity may be reported as 100 percent complete for two consecutive reporting periods because planned discrete tasks for that activity were accomplished in each period, but the reader might rather consider the task to have been done twice. 3. Descriptions of activities are not consistent from one reporting period to the next. This seems to be due to retranslation of work plans and transcription to the Chronogram. Sometimes progress is reported against the objective, at other times relative to a discrete activity. This makes it even harder to understand what has been done. Activities listed on the work plan are often not reflected on the reports due to the 10-page PMR limit. 4. There is no reporting to donors on project progress as measured by the indicators outlined in the FS PMEP, even though the data are being collected where possible. 5. While additional activities usually contribute to attainment of project objectives, they are reported separately. FS attempts to align reporting of activities too strictly to the work plan; even though staff understand that additional activities benefit both FS and the MISAU, it is deemed necessary to explain each deviation from the work plan to USAID. This raises alarms at USAID when in fact the donor ought to be pleased that FS is being responsive to the MISAU and these activities further the goals and outcomes of the FS project. 24 FORTE SAÚDE MID-TERM EVALUATION RECOMMENDATIONS TECHNICAL 1. Relationship with MISAU: Build closer working relationships with more visible and senior members of MISAU to facilitate attainment of project objectives, ensure greater visibility of project assistance in the MISAU and USAID, and secure continuing MISAU support. 2. Quality Improvement Processes: Improve the quality of and access to MCH by implementing the QIP fully in all six target provinces: – Finalize and implement all areas of the QIP, including monitoring the quality of implementation of the Nutrition Sentinel System. – Ensure that the QIP is incorporated into the annual plans at the MISAU central, provincial, and district levels. – Help MISAU define roles and responsibilities at central, provincial, and district levels. – Mobilize managers at all levels to support the QIP. – Ensure that the provincial Training Institutes assimilate QIP into nurses training. – Provide training of trainers for provincial and district directorates, training institutes, and PVO staff. – Provide clinical training to nurses and physicians to address gaps in quality of service identified by the QIP. – Provide on-the-job capacity building for the QIP to supervisors at health facilities, PVOs, and DDS, as well as DPS supervisors and professors from nurse-training institutions at least quarterly. – Assure that there is sufficient budget for the training and travel required in the provinces. – Finalize the quality improvement database. – Ensure that feedback and reporting mechanisms are automated between the database and all levels. – Ensure that sufficient staff in the MISAU at all levels are trained in database maintenance, data entry, and reporting into the database. – Conduct data analysis training so that staff at all levels can use data for decision making. Rationale: Policy, strategy, and technical norms can be implemented and monitored in facilities through the QIP. Continuous on-the-job training and supervision are an aspect of the QIP that continually enhances services. Implementing the QIP will result in improving M&L skills at every level. Institutionalization of good data management processes as a result of the QIP is transferable to all other management outcomes and service provision points, including M&E. The QIP can be scaled up in future to include any other services provided at the facility level, without altering data management processes, by adding new focus areas to the existing instruments and database. Including quality improvement in the annual plans of the MISAU and the nurse-training institutions will assure that the QIP is institutionalized and sustained. Over time this process should have a direct effect on reducing maternal and child morbidity and mortality. 1. Support Implementation of the Nutrition Sentinel System currently being piloted by HKI. Incorporate training on system processes into QI training, which should be funded from the FS budget. If possible the system should be implemented in the same facilities as the QIP. 2. Strengthen M&E in the MISAU: Continue to work with the MISAU to ensure that M&E is part of the strategic plans for nutrition, RH, CH, EPI, health education, MH, quality improvement, and noncommunicable diseases, to lay the groundwork for a future project specifically targeting improvement of M&E and the NHIS in the MISAU. (FS has already done a phenomenal amount of work to strengthen M&E FORTE SAÚDE MID-TERM EVALUATION 25 by helping the MISAU develop results frameworks and M&E plans and to incorporate M&E into strategy documents.) 3. MISAU Deliverables: Finalize specific deliverables: – Award and manage a consultancy to analyze the information technologies supporting the MISAU health information system and produce a final report. – Update the SRH/CH norms with the MISAU to align them with QI instruments. – Produce and distribute training and communication materials for AI (once approved). – Continue financial support to MPH students. – Finalize M&L course content and materials and submit to MISAU for implementation. 1. MISAU NHIS and SO 8 data for USAID: A future project should focus on addressing data quality issues and building a comprehensive health data warehouse. FS does not have time or budget to address the severe NHIS data quality limitations. If USAID will require a project to manage collation of data from other projects, this should be clearly laid out in the contracts. Program and Management 1. Ongoing project management: Implement the use of a Gantt chart tool for internal project management in place of the Chronogram, and make sure it requires significantly less administration. The work plan, which should be translated only once, should form the basis for monitoring activities over a year and easily transfer to the following year. If management information were maintained using an application like MS Project it would be easy to insert additional activities, note slippage of activities, and allocate staff time and other resources. – Give more control and autonomy related to the project budget to the country level. (The current situation may be a result of the management structure employed by Chemonics and subject to the contract.) – Update contracts with subcontractors. 2. M&E of FS: – Revise the PMEP to enhance the quality and relevance of the data collected by the FS program by using output indicators that have a clear link to the activities of the FS project as proposed in the sample results framework (see Appendix G). Although outcome indicators should be included to measure impact, they should be reported on only at the end of the project in the close out report following a final impact study. Data for most indicators suggested can be obtained and reported retroactively. – Use indicator reference sheets to clearly document indicators (see Appendix H). They should specify precise definitions, tools, and methodologies for data collection, collation, and targets. Tools for data collection and adequate provision of audit trails should be specified in the PMEP. 3. Project Staffing: Increase the level of effort of technical staff so that TA to the MISAU in all areas (clinical, nutrition, M&L, M&E) of the QIP is consistently available. 4. Budget: Ensure sufficient technical staff time, training, and travel expenses. 5. Financial Audit: Because there is currently a lack of clarity about FS budget constraints, USAID should conduct a complete financial audit of the FS project. This will highlight both disbursements and perceived challenges to ensure that the financial situation is intelligible. 6. MISAU Focal Point: Change the point of contact for FS in MISAU to the Director of the Health Promotion and Disease Protection Directorate. 26 FORTE SAÚDE MID-TERM EVALUATION FUTURE DIRECTIONS 1. FS activities for the remainder of the project period: Objective 1: More effective information, monitoring, and evaluation systems: To assist the MISAU in developing a comprehensive M&E plan and making NHIS effective across all project areas is beyond the capacity and scope of FS. However, FS can still ensure that M&E remains a priority for the MISAU by ensuring that M&E is incorporated into strategy, policy, and planning documents related to RH/CH. FS should also continue to provide TA where possible to the PVOs to enhance the quality of data being reported to USAID and MISAU. Objective 2: Policies and strategies updated and implemented in the target provinces: FS should continue to work with the MISAU to annually update action plans, strategies, and policies within the National Directorate for Health Promotion and Disease Control in the technical areas where they have been working, but no new activities should be initiated under this objective. Objective 3: Management, leadership, and quality improvement strengthened: Focus on implementing the QIP. The synergy of implementing in parallel with the Nutrition Sentinel System, as recommended, will yield the highest benefit. It will help institutionalize positive data management processes that will contribute to outcomes for Objective 1. Nor should support to the QIP cease when FS ends. USAID should continue to support the MISAU in this area to assure continuity until QI has been assimilated into the MISAU culture. There will be an enormous benefit in terms of reduction in maternal and child morbidity and mortality resulting from better quality services and client satisfaction. Finalize the management and leadership training material and provide technical support to the MISAU in initial implementation. Objective 4: Strengthened human resources training process: FS should continue to support the MPH students and finalize the AI training and communication materials. No further activities should be initiated for this objective. 2. FS budget: USAID should ensure that the project budget is renegotiated to include sufficient funding for extensive travel, training, and technical personnel. 3. Revise the MISAU NHIS: Revision of the entire MISAU M&E system, NHIS, data management, and data quality processes should be the center of a new program that approaches the MISAU as a whole, not just a single health area. This new project would undertake the development of a comprehensive M&E plan for each division, ensure that M&E plans and objectives are integrated into the National Health Strategy, train staff to implement the M&E plan, and build and implement a national health data warehouse. This project should be implemented high up in the MISAU with the agreement of the Minister as a champion of M&E. Data from the system could reliably be used to measure the real impact on the people of Mozambique of the support USAID provides to the MISAU. The evaluators believe that this is beyond what FS can realistically achieve in the remaining project time. 4. Revise the USAID SO 8 M&E Plan: The USAID SO 8 M&E Plan should reflect more accurately the activities and support provided through USAID funding. Clinical outcomes, which are long-term impacts resulting from USAID-funded interventions, should be measured by an external impact study undertaken independently; they may ultimately be drawn from a completely revised NHIS. 5. Approach for future projects: Funds for infrastructure development, remodeling, equipment, MISAU staff training, and community involvement should be allocated in future projects. New projects should be presented to the ministry at a high level to obtain ministry buy-in and support at the RFP stage. Project staff should be introduced to the MISAU very early in the project life cycle. New projects should not focus on single health issues but support the MISAU as a whole. FORTE SAÚDE MID-TERM EVALUATION 27 APPENDIX A: SCOPE OF WORK USAID/MOZAMBIQUE MID-TERM EVALUATION OF SO 8 SUPPORT TO THE CENTRAL/NATIONAL LEVEL OF THE MINISTRY OF HEALTH (USAID REVISED 03-26-08) I. PURPOSE The purpose of this evaluation is to provide the United States Agency for International Development (USAID) Mozambique with a mid-term assessment of its Strategic Objective Eight (SO 8) – Increased Use of Child Survival and Reproductive Health Services in Target Areas. The seven-year program (2003–2010) aims at strengthening the capacities of the Ministry of Health (MOH). Chemonics support was also intended to strengthen PVO/NGO partners, as well as empowering targeted communities to increase utilization of, access to, and demand for quality child survival and reproductive health services in targeted areas. SO8 is now in its fifth year of implementation. Four Cooperative Agreements with partner implementing PVOs, signed in 2005, were evaluated October 7–November 9, 2006 and thereafter modified and extended. Under TASC2, a five-year contract with Chemonics (the subject of this evaluation) was awarded to implement the Forte Saúde program on October 28, 2005, and will expire on August 31, 2010. The principal aim of this five-year project is to assist the central MOH to improve maternal and child health/reproductive health (MCH/RH), malaria, and nutrition policies and implementation to improve quality and efficiency of services to improve health status of the local population. The original contract specified six tasks to be addressed: Task 1: Improve capacity of the MOH (Departments of Community Health, Planning & Cooperation, and Epidemiology) in data collection, analysis, and documentation. Task 2: Assist MOH to improve information and communication technology (ICT) systems. Task 3: Assist MOH to develop policies, strategies, guidelines, and capacity on MCH, RH, nutrition, and malaria. Task 4: Assist MOH to strengthen staff management and leadership skills. Task 5: Improve coordination and monitoring of USAID health funded activities and linkages with the MOH. In this context it is envisioned that the contractor shall have a major role in coordinating and monitoring all USAID￾funded health (SO 8) activities, both at central level and provincial level. Task 6: Provide postgraduate academic training. These tasks were repackaged under Contract Modification No. 2, signed on September 28, 2006, into four objectives: Objective 1: More effective information and M&E systems; Objective 2: Policies and strategies updated and implemented in the target provinces; Objective 3: Management, leadership, and quality improvement strengthened; and Objective 4: Strengthened human resources training process. Purpose of the Evaluation In conducting the mid-term evaluation, the selected team will evaluate the technical and managerial performance of the Chemonics Forte Saúde program, including the Helen Keller International, IT Shows, Health Alliance International, JHPIEGO, and Austral subcontractors, to deliver the above four objectives; and provide recommendations for a future program to support the MOH. The full evaluation report is intended for USAID/Mozambique and its implementing partner, while an Executive Summary and recommendations will be provided to the MOH. Sections of the report may be shared with outside sources at the discretion of USAID. 28 FORTE SAÚDE MID-TERM EVALUATION II. BACKGROUND Although infant, child, and maternal mortality rates in Mozambique have been decreasing in recent years, the rates are still among the highest in Africa and the world at large. Communicable infectious diseases and parasites, namely malaria, diarrhea, respiratory infections, tuberculosis, and the rapid spread of AIDS dominate the country’s epidemiological profile. Health infrastructure and service provision remain weak, resulting in poor quality of care. SO 8 aims to reduce high mortality rates. While the Government of Mozambique is committed to building an equitable health system that is affordable and sustainable, the health services network is not yet sufficiently developed to meet the health needs of a highly dispersed population. The purpose of Health SO 8 is to improve the health of Mozambican families so that they become more productive, less vulnerable to disease, and more effective participants in community health and development. The Strategic Objective statement and its three intermediate results (IR) are as follows: SO 8: Increased use of child survival and reproductive health (CS/RH) services in target areas. IR 1: Increased access to quality MCH/RH services in target areas; IR 2: Increased demand at community level for MCH/RH services in target areas; and IR 3: More accountable policy and management. The health SO includes a combination of national and community-level interventions designed to strengthen the policy and management environment, increase access to proven and effective primary health services, and increase community-level demand for these services by strengthening community3 participation in managing or influencing the quality of health care services, and in providing appropriate services in the community itself. These three key intervention areas will lead to healthier families that are more productive, less vulnerable to disease, and contribute more effectively to improving their economic status. SO 8’s focus interventions are on those health problems responsible for the largest number of child and maternal deaths: malaria, pregnancy and perinatal complications, vaccine-preventable diseases, and diarrheal diseases. IR 3 activities are specifically linked with the FS Project, and are to strengthen and expand priority interrelated management systems to improve the MOH’s effectiveness in managing scarce health resources. Strengthened planning and management systems would allow the MOH to more effectively utilize financial, human, and other resources available. Improved systems and procedures would allow for more comprehensive coordination of internal programs and outsourcing of cost-effective services to the NGO or private sector. The program was to assist the MOH to better define the roles and responsibilities of its operating units and ensure that staff is adequately trained. It is crucial to assist the MOH in not only increasing the effectiveness of allocating human resources but also ensuring that management systems, monitoring and related interventions are consistent with the increased involvement of stakeholders. TA is being provided to the MOH, through a task order with Chemonics, to strengthen policy, program, interventions, communications, and management at central and provincial levels. III. STATEMENT OF WORK A. INTENDED USES OF THE EVALUATION The audience of the evaluation report will be the USAID/Mozambique Mission, specifically the health team, the Africa Bureau, and the implementing partner. An Executive Summary and recommendations will be provided to the MOH. USAID will use the report to make changes to its current strategy of providing support to the central level and to share lessons learned with other stakeholders; Chemonics and its subcontractors will learn about their strengths and weaknesses and adjust their programs accordingly; and the MOH will learn more on how to better 3 For service delivery, ―community‖ will refer to the catchment population of a specific type of service or facility. For local participation, ―community‖ will refer to some appropriate civil society or local governance entity made up of local residents and organized for their common good. FORTE SAÚDE MID-TERM EVALUATION 29 benefit from Chemonics TA. It is expected that the PVO partners and the Provincial Health Directorates (DPSs) will have the opportunity to discuss how the Chemonics Forte Saúde project assisted them and how this type of project could better assist them in the future to meet their goals. The external assessment team will review SO 8’s strategy and the task order with Chemonics to assess how effectively the project is being implemented; identify strengths, weaknesses and lessons learned about the project performance and management; and assess the role Chemonics played in coordinating activities between the PVO grantees and the MOH. Based on the assessment findings, the team will present results achieved to date, document lessons learned, and make recommendations regarding new or modified approaches to achieve the approved SO 8 results in the remaining project period and recommendations to consider in the design of a new Health Strategy. A key focus of the evaluation is an assessment of Chemonics’ role in supporting the achievements of IR 3 goals. This IR focuses on improving the efficient and transparent management of scarce health resources to enable Mozambique’s health sector to derive maximum benefit from all available support. The program’s goal is to strengthen critical systems within the MOH for planning of health services and monitoring program performance. The IR also focuses on improving MOH policies, guidelines, and protocols related to maternal and child health technical areas, malaria, epidemic response, advocacy, and M&E. B. Cross-Cutting issues Monitoring of activities and progress toward achieving results under IR 3 which will be integrated into the Performance Management Plan. Chemonics’ performance will be assessed against anticipated results according to the contract and as expressed through the IR 3 indicators Emergency preparedness for epidemics such as cholera and meningitis. Assess Chemonics’ performance in strengthening MOH and PVO/NGO partners’ capacities to increase use of, access to, demand for, and management of MCH/RH services at provincial, district, and community levels. Capacity building: Programs should strengthen in-country capacity and foster collaboration, as this is the foundation for long-term success. Sustainable health systems and services at the national and local level depend critically on the engagement and commitment of key stakeholders—local people, government, civil society, enterprises, NGOs, and donor institutions. In addition, good cooperation and coordination among USAID implementing partners and other donors is necessary. C. Key evaluation questions: How far has Chemonics progressed at this mid-point in its implementation measured against the proposed results? What are the main constraints found in the project implementation? How have activities like Male Circumcision and Avian Influenza impacted the performance of Chemonics? How well has the consortium been working together? How effective has the overall management of the project been to date? How have MOH counterparts has been helping to reach project results? What are specific recommendations for the future directions of the project? Illustrative questions to assist in the assessment are provided below. The evaluation questions will be refined and finalized with the SO 8 team at the start of the assessment. The contractor will review and document the progress to date of Chemonics in relation to the IR 3 results framework and the performance indicators outlined in the contract. Has the contract contributed to improving access to CS/RH increasing health awareness at community level? How? 30 FORTE SAÚDE MID-TERM EVALUATION Evaluation Objective 1: More effective information, monitoring, and evaluation systems Strengthen the health information system (HIS) capacity for generating and sharing quality data/information in RH, CH, EPI, nutrition and malaria. Conduct an assessment/review of the current situation of the HIS, M&E in the areas of RH, CH, EPI, nutrition, malaria, and epidemics. Develop the capacity of the RH, CH, EPI, nutrition, malaria, and epidemics sectors to conduct effective M&E processes and practices. Monitor the health activities funded by USAID in the areas of RH, CH, EPI, nutrition, malaria, and epidemics, in collaboration with the MOH. How did the project help the PVOS to strengthen information systems, communication technologies, quality performance, management, and leadership? How has the project developed the capacity of the RH, CH, EPI, nutrition, malaria, and epidemic sectors to conduct effective M&E processes and practices? (ADD) How has the project monitored the health activities funded by USAID in the areas of RH, CH, EPI, nutrition, malaria, and epidemics, in collaboration with the MOH? (ADD) Based on results of the assessment recommendations, should the project be extended and in what areas should the project continue to focus? Evaluation Objective 2: Policies and strategies updated and implemented in the target provinces Assist the MOH evaluate the current status of policies, strategic plans, and guidelines in the areas of RH, CH, EPI, nutrition, malaria, and epidemics. Assist the MOH develop/update and adopt policies and strategies. Actions planned in 2006 included drafting RH, CH and nutrition action plans; and drafting policies/strategies to fight epidemics and endemics. Assist in disseminating, implementing, and monitoring policies and strategies in the target provinces (including the community mobilization component), in coordination with the MOH and NGOs with USAID funding4 . Aim: The contractor will review the project’s accomplishments to date against anticipated results as related to a – c above, making recommendations for modifications as necessary. How has the project assisted the MOH to evaluate the current status of policies, strategic plans, and guidelines in the areas of RH, CH, EPI, nutrition, malaria, and epidemics? How has the project assisted the MOH to develop, update, and adopt policies and strategies? How has the project assisted the MOH in disseminating, implementing, and monitoring policies and strategies to the 4 target Provinces? What have been impediments to project implementation in the policy and strategy areas? Evaluation Objective 3: Management, leadership, and quality improvement strengthened Support MOH development of a strategy to strengthen institutional capacity in management, leadership, supervision, and quality improvement in RH and CH at the central level and in target provinces. 4 Text in italics refers to activities implemented at the provincial level with Forte Saúde’s assistance, in coordination with the MOH central level, provincial directorates, and USAID-funded PVOs working in the provinces. While activities at provincial level are primarily implemented by PVOs, Forte Saúde might, subject to availability of funding, need to implement these activities in conjunction with the PVOs. FORTE SAÚDE MID-TERM EVALUATION 31 Assist the MOH strengthen the human resources capabilities in management, leadership, supervision and quality improvement in the areas of RH and CH at the central level and in the target provinces. Assist the MOH institutionalize the improvement of management and quality in the areas of RH and CH at the central level and I the 4 target provinces. Aim: The contractor will review the project’s accomplishments to date against anticipated results in a–c above, making recommendations for modifications as necessary. How has the project supported the MOH to develop a strategy to strengthen institutional capacity in management, leadership, supervision and quality improvement in RH and CH at the central level and in the 4 provinces? To what extent has the project assisted the MOH to strengthen its human resources capabilities in management, leadership, supervision, and quality improvement in the areas of RH and CH at the central level and in the 4 target provinces? How has the project assisted the MOH to institutionalize the improvement of management and quality in the areas of RH and CH at the central level and in the 4 target provinces? What impediments have been encountered in implementing this component of the project and what recommendations do you have for improving the policy and strategy component of the project? Evaluation Objective 4: Strengthened human resources training process. Identify and make arrangements for postgraduate training in Mozambique and abroad in the fields of public health, epidemiology, reproductive health, and other areas related to the activities described in the contract. Support training in the areas of RH and CH, including aspects of nutrition and prevention, and prevention and treatment of malaria in Mozambique. Support the MOH in long-term nurses training in selected provinces. Aim: The contractor will review the project’s progress to date in accomplishing anticipated results and make recommendations for modifications/improvements as necessary to a–c above on a short -term basis. Also the evaluation team will recommend USAID to have a long-scope policy in this regard. What progress to date has the project made in terms of making arrangements for postgraduate training? What impediments have there been to implementation of this key component of the project and what modifications need to be made to improve performance in this component? How successful has the long-term training of nurses been in selected provinces? Are modifications needed to improve this component of the program? If it has been successful, please describe. Evaluation Objective 5: Reporting: Write lessons learned based on assessment findings. Aim: To inform MOH and donor-supported programs in Mozambique. What were the key best practices and challenges in implementation? What did the project contribute that was innovative and that needs to be continued? What project components need to be modified and/or improved? Should any project components be eliminated, and if so, what and why? Is the project design, structure, and implementation approach appropriate for achieving defined results? If not, why not? Is the leadership approach (both technical and management) effective? If not, why not? 32 FORTE SAÚDE MID-TERM EVALUATION Are the resources adequate and well managed, including finance? If not, why not? Does the contractor have effective M&E systems in place, including in the 4 selected provinces? If not, why not? Are activities relevant and moving at a pace that indicates achievement of results within the stipulated time period? What modifications need to be effected in the contract in order to achieve the results stated in the contract? If not, why not? D. SUGGESTED EVALUATION METHODOLOGY The evaluation team will give a detailed description of the methodology for collecting the necessary information and data. The description will include a description of how the methodology responds to the tasks and questions described above; target respondents and how the data will be collected; and a description of how the data will be analyzed. The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information that is required to assess the evaluation objectives. Data collection methodologies will be discussed with, and approved by, the USAID SO 8 team prior to the start of the evaluation. The team approach will be collaborative and participatory and will include key stakeholders as much as possible in planning and conducting the assessment. Document Review: USAID/Mozambique will provide the evaluation team with key documents, such as the monthly performance monitoring reports, PVO Mid-Term Evaluation Report, and the PVO/provincial study undertaken by the MOH in conjunction with other donors. USAID will also collect and annotate additional relevant briefing materials. All team members will review these documents in preparation for the initial team planning meeting. Team Planning Meeting: A two-day team planning meeting will be held in Mozambique before the evaluation begins. This meeting will allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will clarify team members’ roles and responsibilities, establish a team atmosphere, share individual working styles, agree on procedures for resolving differences of opinion, review and develop final evaluation questions; review and finalize the assignment timeline and share with USAID, develop data collection methods, instruments, tools and guidelines, review and clarify any logistical and administrative procedures for the assignment, develop a preliminary draft outline of the team’s report, and assign drafting responsibilities for the final report. Interviews and Site Visits: The evaluation team will meet with key stakeholders to conduct qualitative, in-depth interviews. Semi-structured and open-ended interviews will be administered to primary MOH and DPS officials and PVO staff. The interviews should be loosely structured, but follow a list of key discussion issues and questions as a guide. The interviewer should probe for information and take notes as necessary. The team will review relevant documentation and hold on-site observations, focus group discussions, and interviews with personnel in a minimum of one province, one of which must be either Nampula or Zambezia. IV DELIVERABLES Briefing Meeting: Entrance meeting with SO 8 Team to discuss evaluation parameters and clarify any questions. Debriefing Meetings: The evaluation team will hold two meetings to present the major findings and recommendations of the assessment. The meetings will be held prior to the team’s departure. The first debriefing will be for the USAID/Mozambique health staff and will focus on accomplishments and the assessment team’s recommendations regarding new or modified approaches required to achieve the approved SO 8 results in the remaining three years of the contract. A second briefing will be held for stakeholders and other Mission staff and will focus on findings regarding activity progress and challenges. The team will provide an electronic copy of preliminary evaluation report to USAID/Mozambique not later than 1 week after field visits. The report will be packaged in two parts corresponding with the two purposes of the evaluation. The report shall not exceed 20 pages, not including annexes. FORTE SAÚDE MID-TERM EVALUATION 33 Draft Evaluation Report: Prior to their departure, the assessment team will provide USAID/ Mozambique with a draft report. USAID will provide comments on the draft report to the evaluation team leader within 5 working days of receiving the report. Final Evaluation Report: The contractor will submit two final electronic copies of the report within one week after USAID provides its feedback on the draft report incorporating the comments received from the review of the draft. USAID will share the final but unedited report with the partners (without recommendations – Objective #2) to give them an opportunity to review and make comments. After the final but unedited draft report(s) has been reviewed by USAID and partners, GH Tech will have the documents edited and formatted, and will provide both versions of the final report to USAID/Mozambique for distribution. The evaluation report will include, at minimum, the: scope and methodology used; important findings (empirical facts collected by evaluators); conclusions (evaluators’ interpretations and judgments based on the findings); recommendations (proposed actions for management based on the conclusions); and lessons learned (implications for modifications, future designs, and for others to incorporate into similar programs). Section VIII includes a proposed report outline. V. DURATION, TIMING, AND SCHEDULE It is anticipated that the field work and writing of the draft report will take four weeks and that the final report will be submitted two weeks after receiving comment from USAID/Mozambique. The evaluation team will be authorized to work a six-day work week. The following is an illustrative schedule. Task/Deliverable Projected Timing of Work Schedule Review background documents 2 days Meet with SO 8 team 2 days Information and data collection. Includes interviews with key informants (including partners and USAID staff) 16 days Draft evaluation report in country 6 days Debrief with SO 8 team and stakeholders 1 day USAID provides comments on draft report 5 days Prepare final evaluation report 10 days Total # days 42 days VI. TEAM COMPOSITION The evaluation team will be comprised of three persons, one of whom will be provided by the USAID Mission. The Health and Management Leadership Specialist will be provided by USAID/Mozambique to facilitate setting up meetings. Team Leader: The team leader will provide overall vision and guidance to the team. S/he will be responsible for the overall organization of the work as well as the overarching management and administration aspects of the SOW. S/he will take the lead in developing the tools and methods for data collection. The team leader will facilitate preparation of the executive summary and the full report among the team members; assure that the draft and final products are prepared in accordance with the scope of work; and assure that the required revisions for the final report are incorporated. (GHTech) Information, M&E, Planning Specialist. Must have a background in IT or a master’s degree in epidemiology or public heath with extensive experience in design and implementation of information system, and at least 3 years experience in M&E in developing countries and how to incorporate into a planning cycle. (GHTech) 34 FORTE SAÚDE MID-TERM EVALUATION Health Leadership and Management Specialist, Must have a master’s degree in public health with emphasis in management. Extensive experiences in implementing projects of management, leadership, and donor coordination to support improvement of quality of health services delivery. (USAID) Primary Health Care (MCH, RH, Malaria), Quality Services. Must have a master’s degree in public health with experience in implementing quality improvement services at primary health services levels. Ample experience in primary health care related to maternal and child health services, reproductive health. (GHTech) The Mission prefers that one of the GHTech candidates speak Portuguese, but this is not mandatory. VII. PROPOSED OUTLINE FOR EVALUATION REPORT TABLE OF CONTENTS ACKNOWLEDGEMENTS ACRONYMS AND ABBREVIATIONS EXECUTIVE SUMMARY INTRODUCTION Purpose Background Methodology FINDINGS Technical Program and Management CONCLUSIONS LESSONS LEARNED RECOMMENDATIONS FUTURE DIRECTIONS ANNEXES: Summary of Recommendations Assessment Scope of Work Annotated List of Documents Collected and Reviewed Persons Contacted VIII. RELATIONSHIPS AND RESPONSIBILITIES 1. Overall Guidance: The USAID/Mozambique Office will provide overall guidance to the assessment team within the parameter of the statement of work. 2. Responsibilities: USAID/Mozambique will be responsible for the following: Obtain country clearances for travel. Coordinate and facilitate initial assessment-related field trips, interviews, and meetings. Assist the team with all logistical arrangements. GH Tech will be responsible for the following technical and logistical support: Identify and recruit three team members: the team leader and the public health and PHC specialists. Provide all in-country logistics. Provide administrative and management support to the team while on assignment. Provide support and editing services for the preparation of the final versions of the deliverables. FORTE SAÚDE MID-TERM EVALUATION 35 APPENDIX B: PERSONS INTERVIEWED FS Ellen Eiseman Chief of Party Verônica Reis Senior Technical Officer Humberto Muquimgue Senior M&E/IS Advisor Lidia Cardoso Capacity Building Advisor João Carlos Mavimbe ICT Specialist Natércia Fernandes Child Health Policy Specialist Jorge Anez Ali RH and QI Specialist Isabel Nhatave Health Policy and Strategic Planning Advisor SUBCONTRACTORS Diane Bosch HKI Country Director Maria José Cardoso Austral Project Manager Sally Kamau Health Alliance International Debora Bossemeyer JHPIEGO Country Director—FS Consortium Technical PVOS Ahmed Abajobir Project Hope Country Director Luc Vander Veken Pathfinder Project Director Veronique Kollhoff World Vision Health & Nutrition Director João Aussi World Vision—Coach Project Manager/ Zambézia Luiz Armazia World Vision—Coach Project, Adria Coordinator Jorge Fernandes World Vision—Coach Project, QIP Coordinator MISAU Gertrudes Machatine National Director for Planning & Cooperation Martinho Dgedge Deputy Director for HR & Training—FS Focal Point Hortência Faira Ribeiro Deputy Director for HR & Management Francisco Langa Head, Staff Administration Department Mouzinho Saide National Director for Health Leonardo Chavane Deputy Director for Health Promotion and Disease Protection Benedita Silva Head, Family Health Division & Child Health Section Olga Singauque Head, Reproductive Health and PMCVT Sections Ercília Almeida Head, Health Information Section Ana Charles Head, Environment Hygiene Avone Pedro Head, Nutrition Department Nuno Gaspar Head, Immunization Program Section DPS ZAMBÉZIA Joana Nachaque Provincial Medical Head Suzana Nhamposs Head, Continuous Education 36 FORTE SAÚDE MID-TERM EVALUATION Maria Rosa Head, Quelimane Central Hospital Maternity Linda Moiane Director, Health Institute Fernanda Alfinete Head of MCH, Health Institute DPS GAZA Nubai Calu Provincial Director Joana Tavita Head, Maternal and Child Health Eugenia Fernandes Responsible for PMTCT and QIP, HC of Chicombane DPS MAPUTO Cremilda Anly Provincial Director Stelio Dimande Provincial Medical Head PARTNERS Alicia Carbonell WHO Program Officer, RH & SM Maria da Luz Vaz UNFPA, Head of SRH area OTHERS Abu Saifodine Ex-USAID SOE 8 Team Leader Lilia Jamisse Former Deputy National Director, HPDC FORTE SAÚDE MID-TERM EVALUATION 37 APPENDIX C: DOCUMENTS REVIEWED CONTRACT DOCUMENTS Initial Contract Award, October 27, 2007 Contract Amendment, September 27, 2006 IT Shows, subcontract, February 9, 2006 JHPIEGO Subcontract, February 16, 2006 JHPIEGO Amendment 1, January 16, 2007 JHPIEGO Amendment 2, June 14, 2007 HAI Subcontract, November 25, 2005 HAI Amendment 1, March 20, 2008 HKI Subcontract, June 15, 2006 Austral Subcontract, August 16, 2006 MONITORING, EVALUATION, AND REPORTING DOCUMENTS FS Performance M&E Plan 2006–2010, October 31, 2006 FS Revised Results Framework, January 15 2008 FS Action Plan 2006, June 27, 2006 FS Action Plan 2007, March 20, 2007 FS Action Plan 2008, May 19, 2008 Preliminary Version, FS Action Plan 2008, November 19, 2007 Approved and Revised 2008 Action Plan, April 2008 FS Cronograma mensal de actividades, May 2007–September 2008 FS Performance Monitoring Report, October 28, 2005–June 30, 2006, August 15, 2006 FS Performance Monitoring Report, July 1, 2006–December 31, 2006, March 15, 2007 FS Performance Monitoring Report, January 1, 2007–June 30, 2007, August 17, 2007 FS Performance Monitoring Report, July 1, 2007–December 31, 2007, February 25, 2008 DRAFT FS Performance Monitoring Report, January 1, 2008–June 30, 2008, August 2008 USAID Strategic Objective 8 Performance Monitoring Plan VERSION, May 25, 2007 PVO M&E Plan, 2005/2006 PVO Summary Reports, various USAID Mozambique IR-8.3.2 Indicator Data Collection Form and Job Aid, June 2007 REPORTS Consultancy for the Analysis of Information Technologies Supporting the Ministry of Health’s National Health Information System (NHIS), December 2007 Initial Situation Analysis of MOH of the areas supported by FS, 2006 Report on Pilot of QIP, June 16, 2008 Avian Influenza Training Report, Report on the training of trainers course in rapid response and community mobilization for Avian Influenza, 2007 38 FORTE SAÚDE MID-TERM EVALUATION PRESENTATIONS FS, Presentation to Mid-term Evaluators, July 15, 2008 FS, Technical Update through June 2008, August 6, 2008 Coach, Technical Update, August 6, 2008 FS, Coordination Experience, August 6, 2008 Deliver Project, Coordination Experience, August 6, 2008 World Relief Project, Coordination Experience, August 6, 2008 Project Okumi, Technical Update, August 6, 2008 SWAp RH, CH and Nutrition Group, Presentation to Vice Minister of Health on the QIP OTHER DOCUMENTS FS Strategic Plan 2006–2010, June 28, 2006 FS Organogram, June 2008 List of trips for TA in the provinces from start to April 2008 Report on all primary activities from 2005 – 2008, July 11, 2008 Overview of FS, June 2008 Action Plan for Quality Improvement of RH and CH Services, July 2008 Action Plan for Quality Improvement of RH and CH Services, approved by Minister, November 2006 Quality Improvement Standards for Performance of Services in Sexual and Reproductive Health and Neonatal and Child Health, January 30, 2008 Brief testimony from the MISAU authorities on Sexual and Reproductive Health in Mozambique, CD produced Collated results of Conference of the Health Ministers from the African Union, CD and book produced National Malaria Prevention and Control M&E Plan 2009–2013 FORTE SAÚDE MID-TERM EVALUATION 39 APPENDIX D: INTERVIEW GUIDELINE QUESTIONS TO MISAU How has FS contributed in your area? This contribution has been satisfactory? Why? Is the current approach perceived to be the best or are there other possible approaches that are perceived to be better? How successful is FS in transferring technology and/or developing MISAU capacity? How has the implementation of policies, strategies, procedures progressed in your area? What should be improved or changed? Do you have recommendations for future actions? QUESTIONS TO FS SENIOR STAFF How far has the project progressed at this mid-point in its implementation measured against the proposed results? What are the main constraints found in project implementation? Is there any activity that impacted the project positively or negatively ? (Note: like male circumcision and avian influenza) How well has the consortium been working together? How have MISAU counterparts been helping to reach project results? How do PVOs feel about help received from FS in the different areas? (information systems, communication technologies, quality performance, management and leadership, capacity development, M&E) How effective has the overall management of the project been to date? What activities should continue as they are, should be reinforced, should be changed or eliminated? What are FS’s strengths and weaknesses? What are the specific recommendations for the future directions of the project? NOTE: All areas might be approached: Health information system M&E system Policies and strategy Management and leadership Quality improvement Human resources training QUESTIONS TO FS SUBGRANTEES RESPONSIBLE FOR A SPECIFIC TASK What was the role of FS? How has FS performed it? How is the FS progressing against proposal results? What are the main constraints/challenges? How did the consortium function? How effective was/is USAID/FS/MISAU management? What activities should continue as they are? 40 FORTE SAÚDE MID-TERM EVALUATION What are the specific recommendations for the future directions of the project? NOTE: During/after the interview ask for evidence (reports and other documents). QUESTIONS TO PVOS AND PARTNERS How has FS contributed to your work? What are the best things in their contribution to you? Did they achieve your expectations? Something was missing? How was your relationship with FS? What are your suggestions for the remaining time of the FS project? FORTE SAÚDE MID-TERM EVALUATION 41 APPENDIX E: CONSOLIDATED INTERVIEWS MISAU Technical staff from MISAU who worked directly with FS consider the technical team the best they have worked with. Their expertise, availability, and integration with the MISAU team and partners has been remarkable, beyond what had been planned or anticipated. FS also provided the Family Health Division with equipment such as computers, laptops, and data projectors. FS worked together with MISAU in developing plans, policies, strategies, and technical documents. The FS contribution to the development of the instruments for Quality Improvement at Services Sites; assistance in organizing meetings, workshops, and conferences; and active participation in the SWAp SRH & CH Group is seen in a very positive light. MISAU is dreading the close of FS, and does not know how they will cope without the TA of FS staff. Now MISAU would prefer FS to assist them in implementing the planned interventions. There is general recognition that the staff shortage at MISAU has been an important constraint to the transference of knowledge and technology and has delayed many of the FS processes. An additional significant delay to decision making and activities implementation has been the centralized management style within MISAU. Helen Keller (FS subcontractor) has a longstanding positive partnership with the Nutrition Department. They worked together on the Vitamin A program, database development, developing material for community health workers, and developing a nutrition strategy and action plan. The most recent initiative, Sentinel Sites, is ready to be launched in pilot sites (Maputo Province and Gaza). FS supported an initial theoretical training at the central level and it will be important to support training for health facilities providers who will be working in the Sentinel Sites and for Community Search Groups. FS supported the Expanded Program of Immunization and Heath Education Sector in developing a results framework and an M&E plan. However, since these programs are national ones, they were not able to take advantage of FS TA because FS is limited to working in only 4 provinces. The Department of Noncommunicable Diseases received FS TA at a crucial moment. The avian influenza program had not been planned, and MISAU with MINAG and PVOs working in this area had FS develop a contingency plan. They needed technical and financial support for implementation as well as coordination with different partners. FS assisted MISAU and MINAG to develop training and communications materials, tested them, and now the ministry is finalizing the final revision in order for FS to have them produced and disseminated. FS also had staff in MISAU to assist in partner coordination of these activities. For the NHIS, FS worked with the programs in revising clinical records, record cards for data collection, and health indicators revision. They also developed the quality improvement database, which is still in the pilot phase and will contain real information about maternal and child health service provision, client satisfaction, and health unit situation (infrastructure, medicines, and materials). FS started working with the Human Resources & Management Department in 2007 to develop a curriculum for Management & Leadership Training for the DPS. They have updated and built onto existing materials in MISAU and have had several working meetings, but this activity has not yet launched. MISAU hope to have support to complete and produce trainer and participant manuals and facilitate courses that are planned for the provinces. MISAU also hopes that FS will present some lectures during biannual meetings with senior staff from the provinces. Some constraints were pointed out by all respondents to interviews. First, FS had difficulty with starting up activities. Their project was not aligned with the new minister’s priorities, which emphasize investments in infrastructure, goods and equipment, and training. There was a lack of clarity in MISAU about the focus of the project, mainly around funding issues. The implementation of the project was delayed by the time required to get MISAU approval. 42 FORTE SAÚDE MID-TERM EVALUATION Second, the financial issues arising since February 2008 have never been explained to MISAU. MISAU partners have not understood the financial situation; their feeling is that everything related to money that comes from USAID lacks transparency. This sentiment was expressed by most, if not all, of the interview respondents. In some cases, they have asked other donors to provide resources in order to complete already planned FS activities; this is inappropriate, and both subgrantees/ consortium members and MISAU were unhappy that this was taking place. Third, it was also mentioned that FS have should maintained a technical focus on maternal and child health at the macro level rather than diversifying to information technology, human resources, avian influenza, postgraduate training, etc. This was a mistake from all sides: MISAU, USAID, and FS. Finally, several respondents indicated that the focal point for the project must be within the Directorate of Health Promotion and Prevention. Provincial Health Departments (DPs) The DPS visited by the evaluation team raised two technical topics most frequently: QIP and management and leadership strengthening. FS trained 8 nurses in each province in the QIP, making use of the instruments, and how to evaluate health unit improvement. FS technical staff assisted with conducting the baseline study in 3 health units per province. The nurses and senior DPS staff interviewed are very motivated and indicated that the pilot health units are already showing improvement. They found the instrument very useful and easy to use. There were many requests for additional TA and training on the QIP in order to expand it to other health units. Respondents also indicated that follow-up with in-service training is essential for continuity of the process. Each province must have a group of facilitators, including people from DPS and the Training Institutes. There are problems with lack of materials. They also continue to have problems with data collection and data analysis. Districts should be strengthened in their capacity for data collection, analysis, and the use of information for planning. Services providers do not understand the importance and relevance of data. FS activities have created interest in management and leadership issues, but MISAU is waiting for training materials. They said that many gaps would be addressed by this course. It will be important to highlight the role of managers in the QIP. In Quelimane, financial support for the course in maternal and child health promotion for nurses at the Training Institute was useful despite the limitation in funding and restrictions placed on the use of the money. DPS would like to have stronger support from FS in training, materials, and equipment. Private Voluntary Organizations (PVOs) The general feeling of the PVOs is that FS started when they had already developed their own instruments and methodologies. However, FS is useful in linking PVOs and USAID and in organizing the coordination meetings, which is viewed as a very useful space for information exchange and clarifications. The PVOs feel that they are better informed about the priorities at the central level, particularly on technical issues. The development of the QIP has been very important work that will have an enormous impact in Mozambique. Some of the PVOs had already started quality improvement activity in health centers, but all are now using the instruments and methodology developed by FS. FS engaged in some capacity building of PVOs through management and leadership meetings and workshops that took place in Gaza, Zambézia. FORTE SAÚDE MID-TERM EVALUATION 43 FS coordinated the process of receiving authorization from the Commission of Bioethics to conduct the KPC study, and participated in the questionnaire review. However, when the minister indicated that the sample size must increase from 300 to 3,000, FS ought to have stepped up and explained why this is not practical or necessary. FS consolidates the PVO reports for MISAU but has never sent a copy back to the PVOs. In Zambézia, the FS contribution on avian influenza was well received. PVOs participated in developing an operational plan and organized a training workshop for technical people that responded to Emergency Department needs. All PVOs agreed that FS has very competent staff and that their relationship with them is excellent; always very friendly. PVOs can call FS at any time for advice on technical issues. PVOs never felt that FS was imposing anything; they always coordinated as equals. The role of FS and the PVOs could have been more clearly defined. FS always experiences difficulty in making decisions, and the PVOs realize that the position of FS between USAID and MISAU is not an easy one. However, there is a particular lack of clarity about the role of FS in relation to the PVOs and USAID. The FS financial problems have affected some of the PVOs. FS was not clear and did not ask for financial participation in advance, which has on occasion had a negative impact on the local organization. The PVOs see the role of FS going forward as having a focus on the QIP; however, they need more support in the provinces. FS must accompany the PVOs to follow up on the progress at health units, promote follow-up meetings, and organize and conduct training of trainers. PVOs identified a need for clinical update workshops. The district managers would have to be prepared and mobilized for this process. District managers do not understand the QIP, and therefore did not realize how important their involvement is to achieve better results. It will be beneficial for FS to organize a replication of the Presidential Initiative for the Reduction of Maternal and Child Mortality event in selected districts, or at least in the provinces. To strengthen management and leadership at DPS is a key activity, but it has to be conducted on an almost continuous basis due to high DPS staff turnover; new people coming in are generally inexperienced. An important role for FS would be advocating at the central level for PVO activities in the provinces. With the initiation of the Task 3 malaria project, it will be very important to clarify the roles of TA projects, particularly because malaria activities are critical for projects in the field. The PVOs need more substantial information about how each central project will support them. Subcontractors AUSTRAL The FS project took a long time to start. When the consultancy was requested, the two consultants assigned to the project were committed to other work. They were able to make a needs assessment in MISAU on management and leadership training. They had also conducted focal groups at central level and at provincial level (Gaza) on management and leadership practices. Then they developed a preliminary proposal for an institutional M&L development strategy. They participated in the PES 2005–2010, the PMP for the 2nd semester of 2006, and the preparation of the Action Plan for 2007. They also participated in subcontractor monthly meetings and in the USAID/MISAU/PVO/FS in December 2006. The contract was terminated by mutual consent. 44 FORTE SAÚDE MID-TERM EVALUATION Health Alliance International There is a general feeling that the subcontract was badly managed. There was insufficient staff capacity (50% of one person) allocated to the grants management process, particularly during the selection and placement phase. Once the students had been placed, this would have been sufficient. There was a high turnover of HAI staff allocated to the FS project. This could have resulted in insufficient definitions of the boundaries of the grants available to MISAU, e.g., no limitation in terms of what university they could select. These should have been defined and outlined in communication with the Ministry. The delay in project implementation meant that there was insufficient time to plan and arrange the placement of students from Mozambique, particularly those going to other countries to study. There was not enough time for them to take English courses or orient them correctly. Helen Keller International HKI was involved in the writing of the FS proposal. The role of HKI was to provide technical advice in the area of nutrition. FS was not responsible for the initiation of most of the work done by HKI. It would have been done with or without FS. The only exception is the Surveillance Sentinel Sites. The Sentinel Sites are a very important activity for MISAU. It will monitor the nutrition of children at selected health centers, particularly those in drought-prone areas. A weekly summary will be submitted to the MISAU. The database has been developed and the system is ready for piloting in Gaza and Maputo. Guides have been developed, including calculations required. The database will output graphs of nutritional status of children to provide an overall picture of nutrition in Mozambique. HKI has confidence that MISAU will finish this activity. The situation and coverage for Vitamin A is very low. There has been no specific funding from USAID for Vitamin A. HKI is not working on Vitamin A for FS, but is working on it through other funding sources. This is a policy-based initiative that has achieved a 20 percent increase in Vitamin A coverage. HKI have done some work in Vitamin A with the PVOs but not as part of FS. MISAU staff have not had sufficient training and skills development to take on nutrition seriously. Insufficient commitment and interest from MISAU has been the greatest obstacle. MISAU has only two nutritionists, but only one is working in public health. Nutrition is not an MISAU priority–thus, breastfeeding packs and the nutrition model were never implemented. HKI has insufficient staff time on the FS project: half % of one person’s time has not allowed it to do sufficient advocacy work in MISAU. This person is not based at the FS office, which has led to FS not seeing nutrition as very important, to the point where nutrition is not included in any of the policies that FS has drafted. HKI did comment on one policy, but the comments were not included in the final version. HKI was never formally introduced to the MISAU as the nutrition expert on the FS team. This would have made working with MISAU easier for FS. HKI feels that USAID’s expectations for FS were very high given the grant amount. Also the focus of the project was too broad. The connection of M&E to technical focus areas has been problematic. The inclusion of management and leadership and human resources seems out of place given the project’s technical requirements. The withdrawal of funding has had a very negative impact on the relationship of FS with MISAU. FS manages the constant changes from MISAU and USAID well. The contract of HKI has never been revised–it is still in line with the original six-task contract of FS with USAID. For the future: Fewer meetings, fewer reports, more action: Do more of the work out in the districts and facilities. Train trainers in training skills and clinical training FORTE SAÚDE MID-TERM EVALUATION 45 Build capacity in the training centers. Develop a curriculum for nutrition technicians Focus less on policy Place a full-time person from HKI with FS. IT Shows IT Shows was responsible for IS and ICT. It had a staffer at the FS office until November 2007. He did work on the Hospital Information System, developing forms and presented them to MISAU. JHPIEGO FS is an extremely challenging project. There is a lack of clarity in assigning responsibility in the design of the project by USAID and project management by Chemonics. The project was initially presented to the MISAU as a TA project at the central level, which was not what MISAU wanted. MISAU wants to reap tangible benefits from projects (improved infrastructure, trained staff, etc). The project focal point is no longer in the correct position at MISAU to be of use to FS. The Minister recognizes the need for support in reproductive health. FS has a very good relationship with three people in reproductive health, but these have low visibility in MISAU (with the minister). FS needs to have relationships with individuals in MISAU who have higher visibility. Chemonics management of FS is seen to be extremely poor. There is no empowerment of the local staff in decision making, negotiation with USAID and MISAU, or budget or program decisions. The experience in working with FS has not been positive—the processes are not transparent. Partners should have more autonomy in program decision making. In the design phase JHPIEGO presented Chemonics with a comprehensive budget to undertake technical activities in line with the proposal’s required outcomes. JHPEIGO was told to remove direct costs from their budget for training and travel costs, etc., and told that Chemonics would take care of these. When the project started, they were then told that there was no budget for these activities. FS has done very little in three years given the amount of funding and the number of people. They are too involved in coordination and administration. JHPIEGO has no updated contract that reflects the budget cut and accordingly amended activities. The technical material developed by FS is excellent. FS has developed the only material that incorporates PMTCT and malaria into antenatal care. This could be launched and trained on at a large scale to ensure that the project leaves behind a tangible product. It will be important that MISAU (client) and USAID (donor) meet and decide what activities FS should pursue going into the future. Partners In the maternal and child health arena the agencies that give ―daily‖ support to MISAU are UNFPA, UNICEF, WHO, and FS. The greatest constraint in working with MISAU is the shortage of human resources. There are not enough staff, high turnover, and limited staff capacity. Partners organize themselves to optimize support to MISAU and help MISAU to be organized. Partners and MIASU staff always support one another. There is always someone from MISAU leading any work. The usual method of working with MISAU is for one person to prepare an initial draft and present this to the group, including MISAU. Then one of the partners takes ownership incorporating comments and results of discussion and providing input until MISAU is happy with the product. 46 FORTE SAÚDE MID-TERM EVALUATION Partners have a very positive impression of FS and the work it has done with MISAU. FS has done a great deal of work in the areas of developing policies and guideline documents with the group, and provides very good and desperately needed technical support to MISAU. The participation of FS in the SWAp SRH & CH Group has a very positive impact on the capacity of ministry staff. FS shares a lot of information on technical issues, which is seen as a very important learning experience for MISAU. The partners stated that the group cannot work without FS. They do not have enough staff to do everything that is needed. The coming challenge is to implement the documents already developed. It is important to recognize that ministry priorities change, and if you do not change your priorities, MISAU simply does not work with you. This means that things often get started and then left behind as MISAU focus shifts, which often leads to what seems like a low impact of programs/projects. All organizations working with MISAU find this to be their biggest challenge. The FS effort is a joint one, is not vertical, and should make alliances with organizations that can ensure project continuity. FS should not close without ensuring that another organization, project, or program takes over the work it is doing because it will leave a large gap in the resources available to and frequently used by MISAU. The general feeling is that the handover will be at least a year in duration. FS Staff Strengths Qualified human resources Capable and cohesive team Technical support to MISAU, which acknowledges it can always find support from FS Positive technical reputation Excellent relationship with colleagues Excellent relationship with MISAU technical staff Good work environment The team sees beyond its obligation; it has a vision of Mozambique’s needs. Weaknesses Reduction of technical staff time and consequently lack of availability Financial issues that caused a brusque fall and frustration in the air Budget cuts that had a negative impact on the relationship with MISAU FS does not know how to say ―no‖ to MISAU, and sometimes spends a lot of time addressing small issues that have been inflated by MISAU. Challenges Articulation with MISAU Staff turnover at MISAU Limited capacity of MISAU staff Financial restrictions Not enough travel to the field Very limited NHIS in MISAU. FORTE SAÚDE MID-TERM EVALUATION 47 General FS anticipated result is to have systems developed and implemented. Things are moving in the right direction and are where they need to be. At the beginning FS had some difficulties with MISAU. The project did not meet the expectations of MISAU. Substantial time was dedicated to building relationships in MISAU. After 10 months of negotiation, the work plan was approved. MISAU views FS as a credible and valuable resource. FS has an outstanding presence in the MISAU maternal and child health area. Its integration with MISAU staff is so strong that sometimes FS staff seem like MISAU employees. The financial restrictions have had a negative impact because FS has not been able to deliver on some activities MISAU was expecting. USAID added some activities to the FS project, such as organization of the Conference for Ministers of Health from the African Union. This activity accounted for a substantial portion of FS level of effort in 2006 and early in 2007. An important FS role has been coordination. It has a very positive relationship with PVOs. FS perceives that its staff help strengthen MISAU’s technical capacity. The task of developing public/private partnerships was discarded because MISAU sent a letter saying that ―MISAU does not support the idea at this moment.‖ FS also developed some activities in avian influenza. FS tested training and communication materials in a workshop in Zambézia. This material must be revised because it is both extensive and complex. Material must be less theoretical and more practical. As soon as materials are finalized and approved, FS will print and distribute them. There is very positive feedback about the internal management style and the excellent project environment. SO 1. More effective information, monitoring, and evaluation systems NHIS and IS Staff believe that FS has completed many activities. Neonatal health, maternal health, and EPI data collection and collation forms have been finalized, and 16 more forms are under development, which when approved will be integrated them into the NHIS by the DES. MISAU with the assistance of FS have already developed a plan for rolling out the system and training 1,800 health staff. A workshop with MISAU to define indicators on NHIS was conducted and the indicators developed. The NHIS is currently fragmented, with information feeding into it from subsystems for each department with inappropriate and extremely outdated technology. FS developed a scope of work for a consultant to evaluate the entire NHIS and put forward a proposal for an integrated and consolidated system. A service provider was selected but due to financial constraints has not been awarded. This is seen by MISAU as a necessary step to start the integrating the NHIS system. A quality information system for RH/CH has been developed and the database is accessible and includes reports on standards. It is being piloted in 18 health facilities in six provinces. FS developed the methodology for the KPC survey. M&E FS worked with PVOs to improve reporting presentation of data to USAID. FS has no management mandate with PVOs, which do not have to take FS advice or report to FS. PVOs pass data to FS in the spirit of collaboration. 48 FORTE SAÚDE MID-TERM EVALUATION FS is driven by MISAU priorities to the point where the priorities of FS are set aside. If FS does not respond to shifting MISAU priorities it will not be able to work with MISAU at all. The M&E processes are hampered by the inadequate MISAU M&E system and little cultural commitment to data analysis at the peripheral level. The limited number and capacity of staff in MISAU is a huge obstacle to implementation. Institutional knowledge is not retained, so FS is often in a position where it has to educate new staff. Indicators are not providing management utility to MISAU. There are no feedback loops to the provinces, districts, or health facilities There is a lack of understanding in MISAU that M&E is connected to ICT and IS. Without adequate ICT and IS, it is not possible to get quality data at the central level. The responsibility for M&E rests with the Planning and Cooperation Directorate in the MOH, but all departments must have integrated indicators (this responsibility rests with the individual departments). FS is working with departments to develop results frameworks. The interconnectivity of MISAU makes it very difficult to work only in the target areas. There is a need for a results framework for the whole of MISAU. Data for USAID are connected to data from MISAU; thus the issues are codependent and hinge on the same success factors. SO 2. Policies and strategies updated and implemented in the target provinces This activity is sufficiently advanced. Documents developed had the support of MISAU, which recognized the importance of strategy and policy. FS has contributed to the development of 15 policy and strategy documents. FS took the lead in developing six of the 15 and provided process and framework assistance in developing the others; for one of these FS and the WHO split the writing. FS still needs to work with MISAU to update Technical Norms for Sexual and Reproductive Health. The challenge now is to disseminate the documents. SO 3. Management, leadership, and quality improvement strengthened FS started to mobilize quality improvement activities in November 2006. The working group reached consensus for an action plan to develop the process that was authorized by the Minister in December 2006. At the beginning of 2007 they started a series of workshops with MISAU and the partners to arrive at quality standards and draft the instrument for measuring performance. These instruments were approved by the Minister in October 2007. Meanwhile FS prepared training material for provincial teams. Six provinces with three health unit each were included in the pilot phase. In May 2007 they started the training workshops, followed by baseline workshops conducted in the health centers. After the baseline workshops were conducted, FS evaluated the results with MISAU and provided TA in designing an action plan for each health unit for the next three months. Some simplified technical texts and brochures were developed, reproduced, and distributed on humanization and quality for health care assistance to women and children during delivery and birth. The second evaluation round started early in 2008 and was followed by the second training module. Two theoretical and practical clinical courses on updated procedures in SRH and newborn care were conducted. The third round of evaluation and workshops should occur in August 2008. It was anticipated that evaluations would be carried out every three months. In reality there is a longer interval between evaluations. They are developing a database for the results. The main constraint has been the lack of resources for supervision trips to provinces. FORTE SAÚDE MID-TERM EVALUATION 49 Management and leadership has been incorporated into the quality improvement modules. FS is also preparing a specific management and leadership training course for the DPS. A needs assessment was carried out with the DPS in Gaza and Zambézia in 2007. FS conducted a workshop on Strengthening Management Capacity for the PVOs and DPS in Gaza and will conduct the same workshop in Zambézia in September. SO 4. Strengthened human resources training processes The basic nurse training courses were more a financial support for the Training Institutes. But it was good because these groups participated in the quality improvement workshops, and the professors are willing to include this topic in the School of Nursing curriculum. FS also supports master’s degree programs in Maputo and South Africa universities for 11 professionals designated by MISAU. Going forward: Serve as a link between the central level and the field. Conduct an analysis of information technologies supporting the MISAU NHIS and deliver final report to MOH. Support PVOs in M&E and ICT. Complete work with all departments to develop M&E plans in line with the activity plans of each department. Provide support to different departments to develop results frameworks, training, workshops, and doing advocacy for institutionalization of M&E processes within the MISAU. Assist the MISAU to develop a culture where information is valued and used to enhance quality for data management processes. Support the central level MISAU staff to implement the documents developed. Training of trainers (including the Training Institutes) will be a way to guarantee the sustainability of the QIP. Clinical training in child health (newborn reanimation) is needed. A clinical course in SRH is also needed. Spend more time the field; do training follow-up. Work in the field with PVOs and also with DPS and health units. Expand the QIP. Implement the quality improvement system. Do management training/strengthening at the district and provincial levels. Produce and distribute training materials, bags, and manuals as well as communication materials on avian influenza. 50 FORTE SAÚDE MID-TERM EVALUATION APPENDIX F: SCREEN SHOTS OF QUALITY IMPROVEMENT DATABASE FORTE SAÚDE MID-TERM EVALUATION 51 52 FORTE SAÚDE MID-TERM EVALUATION FORTE SAÚDE MID-TERM EVALUATION 53 54 FORTE SAÚDE MID-TERM EVALUATION APPENDIX G: SAMPLE SUGGESTED RESULTS FRAMEWORK TABLE 7: SAMPLE OF RESULTS FRAMEWORK FOR THE FS PROJECT Impact Improved capacity in the technical areas of RH, CH, malaria, and nutrition in the MISAU. Outcome 1 Outcome Indicators Strengthened information and communication systems, M&E in the technical areas of RH, CH, nutrition, and malaria at the central level and in the 4 target provinces. Percentage of identified challenges in the data management processes of the MISAU in the target technical areas remedied Percentage of MISAU technical areas that can demonstrate the use of data from the NHIS in decision making at the national level Activity Outputs Output Indicators Improve data management capacity in the MISAU. Number of meetings with MISAU staff to build capacity in M&E. (disaggregate by level of MISAU staff and topic) Number of individuals trained in data management processes, data quality processes, data analysis, and M&E. Improve the capacity and implementation of the NHIS. Number of results frameworks developed by the MISAU with the assistance of FS. (disaggregate by results framework status—developed, submitted, approved—and technical area) Number of data collection/collation forms designed by the MISAU with the assistance of FS. (disaggregate by form status—developed, submitted, approved, piloting, finalized, implemented—and technical area) Number of data collection/collation forms incorporated into the NHIS with the assistance of FS. (disaggregate by technical area) Number of monitoring, evaluation, and reporting plans developed by the MISAU with the assistance of FS. (disaggregate by plan status—developed, submitted, approved—and technical area) Number of best practices in information and communication systems and M&E documented Rationale FS implements at the central level of the MISAU, and places a positive emphasis on working with MISAU staff to attain project outcomes. The process for developing M&E systems is consultative: first the existing processes are identified, then critiqued, corrective processes recommended, and then agreed on. Results frameworks are developed and finalized. At this point it is practical to start developing data collection forms incorporating agreed data points. Once the forms have been developed and approved, they are piloted and finalized. Training must take place to ensure that all relevant staff buy into and make use of the system. Then the data points must be incorporated into the NHIS. Up to this point FS has some control of the M&E process and thus may be accountable. Implementation of the new data management processes is dependent on the MISAU. However, at the outcome level this is what FS must advocate for, and what it will have been working toward, and may therefore ultimately be reported to measure the success of the project toward attaining this outcome. Without achieving the outputs in the process of attaining the outcome, the outcome cannot be attained. It is thus important to monitor outputs and provide a measure of progress toward the outcome. FORTE SAÚDE MID-TERM EVALUATION 55 TABLE 7: SAMPLE OF RESULTS FRAMEWORK FOR THE FS PROJECT Outcome 2 Outcome Indicators Strengthened policy and strategy framework of the MISAU in the target technical areas of MH and CH, including components for nutrition, malaria, and epidemics Percentage of identified challenges in policy and strategy framework of the MISAU in the target technical areas remedied Percentage of MISAU technical areas that can demonstrate implementation of updated policies and strategies Activity Outputs Output Indicators Improve policy and strategy capacity for MH and CH, including components for nutrition, malaria, and epidemics in the MISAU. Number of meetings with MISAU staff to build capacity in policy development. (disaggregate by level of MISAU staff and topic) Number of organizations that participate in policy development meetings Number of meetings with MISAU staff to build capacity in strategies/road maps/guidelines/activity plans development. (disaggregate by level of MISAU staff and topic) Number of individuals who participate in workshops/are trained in the development of strategies/road maps/guidelines/activity plans Improve policy and strategy framework for MH and CH, including components for nutrition, malaria, and epidemics in the MISAU. Number of policies developed by MISAU with the assistance of FS. (disaggregate by policy status—developed, submitted, approved—and technical area) Number of strategies/road maps/guidelines/ activity plans developed by the MISAU with the assistance of FS. (disaggregate by status of strategies/road maps/guidelines/activity plans—developed, submitted, approved—and technical area) Improve dissemination and implementation of policies/strategies/roadmaps/ guidelines/activity plans in the MISAU. Number of promotional materials to facilitate dissemination/implementation of policies/strategies/roadmaps/guidelines/activity plans. Number of seminars/workshops to facilitate dissemination/implementation of policies/strategies/roadmaps/guidelines/activity plans. Rationale FS implements at the central level of the MISAU, and places positive emphasis on working with MISAU staff to attain project outcomes. Its actual work is in assisting the MISAU to develop policy and strategy documents. FS can work with the MISAU to disseminate and train on the implementation of strategies; but actual implementation is beyond its control. Thus the implementation can be seen as a longer-term measure of the success of the training/advocacy work done by FS. To ensure that aspects of HIV and AIDS are included in policies, they would be included in the definition of a policy along with other cross-cutting issues, such as gender. In this way quantitative indicators can require specific depth in the intervention. Without achieving the outputs in the process of attaining the outcome, the outcome cannot be attained. It is thus important to monitor the outputs to provide a measure of progress toward the outcome. Outcome 3 Outcome Indicators Strengthened capacity in management and leadership and quality of implementation of MH and CH services in the MISAU Percentage of facilities implementing that quality improvement system that can demonstrate an improvement in the quality of MH and CH services provided to target populations Percentage of identified challenges to management and leadership in the target technical areas remedied 56 FORTE SAÚDE MID-TERM EVALUATION TABLE 7: SAMPLE OF RESULTS FRAMEWORK FOR THE FS PROJECT Activity Outputs Output Indicators Improve capacity for quality of implementation of MH and CH services in the MISAU. Number of meetings with MISAU staff to develop/implement a quality improvement framework for MH and CH services (disaggregate by level of MISAU staff and topic) Number of individuals trained to implement the quality improvement system in MISAU. (disaggregate by gender, level of individual, province) Number of facilities implementing quality improvement system. (disaggregate by province) Number of quality improvement standards developed by the MISAU with the assistance of FS included in the in-service training curriculum Improve capacity for management and leadership in the MISAU. Number of management and leadership curricula adapted/developed and implemented by the MISAU with the assistance of FS. (disaggregate by technical area) Number of individuals in the MISAU trained in management and leadership with the assistance of FS (disaggregate by gender, level of individual, province) Rationale Output indicators provide a measure of the actual activities of the FS project; outcome indicators provide a measure of the success of the implementation of the systems in the MISAU, which is not under the control of FS, but for which it is actively advocating. Outcome 4 Outcome Indicators Strengthened capacity in the MISAU for developing human resources in the areas of MH and CH, including aspects of nutrition and malaria Postgraduate training committee established and active in the MOH Activity Outputs Output Indicators Improved capacity in the MISAU Number of candidates identified by the MISAU with the assistance of FS to undertake postgraduate training supported by FS (disaggregate by gender) Number and value of assistance provided by FS to postgraduate students selected (disaggregate by assistance type) Number of post-graduate candidates selected for support by FS who complete postgraduate training. (disaggregate by gender) Number of nurses supported in long-term training by FS. (disaggregate by gender) Improved capacity for training in the target technical areas in the MISAU Number of curricula developed/adapted and implemented by the MISAU with the assistance of FS. (disaggregate by technical area) Rationale Output indicators with adequate definitions provide a measure of the work being done by FS. FORTE SAÚDE MID-TERM EVALUATION 57 APPENDIX H: SAMPLE SUGGESTED INDICATOR REFERENCE SHEET TABLE 8: INDICATOR REFERENCE SHEET TEMPLATE Indicator Protocol Reference Sheet Number: X Name of Indicator: Result to Which Indicator Responds: Level of Indicator: Description Definition: Unit of Measure: Disaggregated by: Justification and Management Utility: Plan for Data Acquisition and Collation Data Collection Method: Data Collation Method: Primary Data Source: Secondary / Proxy Data Source: Frequency and Timing of Data Acquisition: Estimated Cost of Data Acquisition: Individual Responsible: Location of Data Storage: Data Quality Issues Known Data Limitations and Significance: Actions Taken or Planned to Address Data Limitations: Internal Data Quality Assessments: Margin of Error: Plan for Data Analysis, Review & Reporting Data Analysis: Presentation of Data: Review of Data: Reporting of Data: Baselines: Targets for Indicator Year Target Actual Actual Cumulative Notes 2007 2008 This Sheet Last Updated On: X 58 FORTE SAÚDE MID-TERM EVALUATION FORTE SAÚDE MID-TERM EVALUATION—PRELIMINARY REPORT PAGE 59 APPENDIX I: FORTE SAÚDE ORGANIZATION CHART Chief of Party Ellen Eiseman 100% Senior Technical Officer Veronica Reis 100% Technical Admin Assistant Neusa Xavier 100% Office Manager / Executive Assistant Paula Leão 100% ICT Specialist Joao Carlos Mavimbe 50% Capacity Building Advisor Lidia Cardoso 50% Senior M&E IS Advisor Humberto Muquingue 85% Accountant Towindo Tichaoma 100% Quality / Family Planning Specialist Jorge Anez Ali 50% Child Health Policy Specialist Natercia Fernandes 25% Health Policy / Strategic Planning Isabel Nhatave 75% Driver / Logistics Assistant Jose dos Santos Cotela 100% Housekeeper Dulce Marengule 100% 60 FORTE SAÚDE MID-TERM EVALUATION—PRELIMINARY DRAFT REPORT For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com