This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Leah Ghoston, Hilly Bouwman, Livio Conceicao, and Amelia Barreto. The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/TIMOR-LESTE REINFORCE BASIC HEALTH SERVICE ACTIVITY Final Evaluation April 2020 Task Order No.: 72047219F00003, Contract No. AID-486-I-14-00001 i | REINFORCE FINAL EVALUATION USAID.GOV ACKNOWLEDGEMENTS The evaluation team sincerely thanks all who contributed to the success of this evaluation. Special thanks go to Milca Baptista, Monitoring, Evaluation, and Learning Specialist at USAID/Timor-Leste; Teresa Miller, Director, Office of General Development (OGD) at USAID/Timor-Leste; and Dr. Teodulo Ximenes, Project Management Specialist at USAID/Timor-Leste, for their assistance and valuable insights and feedback throughout the evaluation. The evaluation team is grateful to the staff of John Snow, Inc., Research & Training Institute (JSI) for their unreserved support of this evaluation. They freely shared Reinforce activity documentation and contextual information that proved essential to analyzing results. Further, they provided valuable assistance facilitating field visits and identifying contacts. The evaluation would not have been possible without the cooperation of the Government of Timor-Leste Ministry of Health staff, Municipal Health Services staff, Instituto Nacional de Saúde (INS) staff, and trainers and facility staff, including doctors, nurses, and midwives at Health Posts (HPs) and Community Health Centers (CHCs) who devoted time to provide feedback to the evaluation team. The team thanks the community leaders and members who served as key informants and survey respondents and shared their perspectives. This evaluation would not have been successful without their participation. Additionally, special thanks go to Andrew Carmona, Social Impact (SI) Program Director, for his support in all stages of the evaluation; Sam Mirtaheri, SI Program Manager; and Pechta Sok, SI Program Assistant, for providing operational support and guidance throughout planning and fieldwork. April 2020 The Evaluation Team USAID.GOV REINFORCE FINAL EVALUATION | ii ABSTRACT In 2015, the United States Agency for International Development (USAID) awarded the Reinforce Basic Health Services activity to John Snow, Inc., Research & Training Institute (JSI). The activity supports the Ministry of Health in Timor-Leste by improving health service delivery and promoting healthy behaviors for family planning and maternal and child health in Covalima municipality. USAID/Timor-Leste contracted Social Impact, Inc. (SI), to evaluate activity effectiveness, document promising practices, identify bottlenecks, evaluate the capacity and commitment of stakeholders, including the Ministry of Health (MOH), Municipality Health Service (MHS), and National Health Institute (INS), to institutionalize good practices, and to provide recommendations to guide future investments in public health. The evaluation team used a mixed methods approach, including a document review, key informant interviews, focus group discussions, a mini survey, and review of secondary data, to identify findings to address USAID’s evaluation questions. The evaluation found that Reinforce succeeded in engaging with and building the capacity of MHS and health facilities to adhere to service standards. However, the activity’s progress was limited by longstanding systemic governmental challenges in human resources, equipment, and infrastructure. Initiatives to support INS’s training capacity, improve MOH and MHS Covalima’s data management knowledge and quality, and improve health seeking behaviors and health outcomes for maternal, neonatal, and child health (MNCH) were successful in many ways, but faced strong obstacles to sustainability. Poor knowledge management, weak coordination, and a lack of transparency and consistency in budget allocation between the national and municipal levels prevents institutionalization of the best practices in Covalima. Replicating the model will rely on coordination among multiple government departments and partners seeking to support them. The report makes several recommendations to key stakeholders for future investment in human resources, sustainable and transparent financing, and improved system coordination. iii | REINFORCE FINAL EVALUATION USAID.GOV CONTENTS Acknowledgements ........................................................................................................................................................ i Abstract ................................................................................................................................................................................ ii Contents ............................................................................................................................................................................. iii Acronyms............................................................................................................................................................................. v Executive Summary ............................................................................................................................................................. Background and Methods.............................................................................................................................................. Finding and Conclusions................................................................................................................................................ Recommendations ......................................................................................................................................................... v Introduction ........................................................................................................................................................................ 1 Evaluation Purpose and Audience ............................................................................................................................ 1 Evaluation Questions.................................................................................................................................................... 1 Background .......................................................................................................................................................................... 3 Context ........................................................................................................................................................................... 3 Activity Background...................................................................................................................................................... 4 Development Hypothesis............................................................................................................................................ 5 Design and Methodology ................................................................................................................................................. 6 Evaluation Design .......................................................................................................................................................... 6 Data Analysis.................................................................................................................................................................. 9 Findings and Conclusions...............................................................................................................................................11 EQ 1: Engagement and Capacity Strengthening ...................................................................................................11 EQ 2: Barriers to Institutionalization .....................................................................................................................23 EQ 3: Sustainability .....................................................................................................................................................34 EQ 4: Adapting and Scaling the Reinforce Model................................................................................................40 EQ 5: Improved Capacity at INS .............................................................................................................................45 Recommendations ...........................................................................................................................................................50 Annexes .............................................................................................................................................................................53 Annex I: Rapid Survey Assessment Results ..........................................................................................................53 Annex II: Findings and recommendation workshop agenda and activities....................................................56 Annex III: Field Plan for Respondents and Sites Visited.....................................................................................59 Annex IV: Documents Reviewed.............................................................................................................................61 Annex V: Data Collection Tools.............................................................................................................................63 Annex VI: Evaluation Design Matrix .......................................................................................................................81 Annex VII: Scope of Work........................................................................................................................................83 Annex VIII: Disclosure of Any Conflicts of Interest...........................................................................................94 TABLES Table 1: Evaluation Questions ...................................................................................................................... 1 Table 2. KII Participants by Stakeholder Type (Total 74 KIIs)..................................................................... 7 Table 3. FGD Participants by Stakeholder Type (Total 12 FGDs)............................................................... 8 Table 4. Survey Participants by Stakeholder Type........................................................................................ 8 FIGURES Figure 1: Health worker competency from the beginning of Reinforce in 2016 to December 2019.........17 Figure 2: Capacity Assessment Matrix........................................................................................................................21 Figure 3: Bottleneck Analysis........................................................................................................................................32 USAID.GOV REINFORCE FINAL EVALUATION | iv Figure 4: Assessment of Progress to a Model Municipality ...................................................................................43 Figure 5: Clinical Practice Site External Assessment Scores, 2016 (baseline) to 2020...................................46 v | REINFORCE FINAL EVALUATION USAID.GOV ACRONYMS ADS AE Automated Directive System Administrasaun Estatál (State Administration) ANC Antenatal Care ARH BEmONC BCC Adolescent Reproductive Health Basic Emergency Obstetric and Newborn Care Behavior Change Communication BPHS Basic Package of Health Services CHC Community Health Center CHO Community Health Officer CITL Care International – Timor-Leste CMB Cuban Medical Brigade CYP Couple Years of Protection DFAT Department of Foreign Affairs and Trade DHIS Demographic and Health Information System DHS Demographic Health Surveys DOAG Development Objective Agreement DQA Data Quality Assessments EmOC ENC EQ Emergency Obstetric Care Essential Newborn Care Evaluation Question EQUI Evaluation, Quality, Use, and Impact ET Evaluation Team FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion FP Family Planning FUAT Follow-Up After Training GBV Gender-Based Violence GoTL Government of Timor-Leste HAI Health Alliance International HCDF HMIS Human Capital Development Fund Health Management Information System HP Health Post HR Human Resources HRH Human Resource for Health IMCI Integrated Management of Childhood Illnesses INS Instituto Nacional de Saúde (National Health Institute) IP Implementing Partner IPC/C Interpersonal Communication and Counseling IRB Institutional Review Board IUD Intrauterine Device J2SR Journey to Self-Reliance JDN Juventude ba Dezenvolvimentu Nasional USAID.GOV REINFORCE FINAL EVALUATION | vi JNPK-KR Indonesia National Clinical Training Network for Reproductive Health JSI John Snow Inc. Research & Training Institute KAP Knowledge, Attitudes, and Practices KII Key Informant Interviews KOICA Korean International Cooperation Agency MCH Maternal and Child Health M&E Monitoring and Evaluation MHS Municipality Health Service MNCH Maternal, Neonatal, and Child Health MOE Ministry of Education MOF Ministry of Finance MOH Ministry of Health NGO Non-Governmental Organization NHSSP National Health Sector Strategic Plan PE Performance Evaluation PNDS National Program for Village Development POP Period of Performance QA Quality Assurance RACI Responsible, Accountable, Consulted and Informed RDQA Routine Data Quality Assessment RMNCAH Reproductive, Maternal, Neonatal, Child and Adolescent Health SARA Service Availability and Readiness Assessment BCC Social Behavior Change Communication SI Social Impact, Inc. SISCa Servisu Integrado Saúde Communitária (Integrated Services of Community Health) TNA Training Needs Assessment TL Team Leader ToT Training of Trainers U-FE Utilization-Focused Evaluation UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund UNTL National University of Timor-Leste US United States USAID United States Agency for International Development WHO World Health Organization USAID.GOV REINFORCE FINAL EVALUATION | EXECUTIVE SUMMARY BACKGROUND AND METHODS The United States Agency for International Development (USAID) awarded the Reinforce Basic Health activity (Reinforce) to John Snow, Inc., Research and Training Institute (JSI) to increase the capacity of health service delivery by identifying and investing in one ‘model municipality’ in Timor-Leste.1 The period of performance is December 23, 2015 to December 22, 2020. The activity contains two broad components: (1) strengthening integrated health service delivery and (2) promoting health behaviors in the selected model municipality of Covalima. USAID/Timor-Leste has contracted Social Impact, Inc. (SI) to conduct a performance evaluation (PE) of Reinforce to assess the extent to which the activity has strengthened healthcare service delivery in Covalima, how it could be further improved, and how it could be scaled to other municipalities in Timor-Leste. This performance evaluation, conducted in Reinforce’s final year, will help stakeholders understand how well the Government of Timor-Leste (GoTL) can and is willing to introduce Reinforce’s approach elsewhere in Timor-Leste and continue to strengthen Covalima’s healthcare service system. These stakeholders include USAID, JSI, and the GoTL, including the Ministry of Health (MOH), Covalima Municipality Health Service (MHS), and the National Health Institute (INS). This evaluation uses a mixed methods approach, combining qualitative data collection with a rapid survey and quantitative analysis of secondary data. Working in both the capital of Dili and all seven subdistricts of Covalima, the evaluation team conducted 66 key informant interviews (KIIs), 12 focus group discussions (FGDs), and 31 rapid surveys during three weeks of fieldwork. After collecting data, the team presented preliminary findings to MOH stakeholders to generate key recommendations through a findings and recommendation workshop in Dili. FINDING AND CONCLUSIONS The evaluation identified several findings and conclusions across five evaluation questions (EQs): EQ 1: How did Reinforce Activity engage and strengthen the capacity of stakeholders (MOH, INS, municipality health services, health facilities, and local leaders) to institutionalize best healthcare service practices in Covalima? 1 USAID’s Cooperative Agreement with JSI stated that Reinforce “will support the implementation of the MOH’s Reproductive, Maternal, Newborn, Child and Adolescent Health Strategy continuum of care model and concentrate on proven results, to be showcased through a ‘Model Municipality’,” a select municipality for intervention in Timor-Leste that combines community engagement with skilled health providers, quality service delivery, equipped and accessible facilities, and a municipal health system using data for decision-making. Reinforce engaged stakeholders in varied ways at inception and throughout the activity, yet because Reinforce engaged actors separately instead of collaboratively, stakeholders were unsure who owned responsibility for what. While several skills improved, stakeholders still have systemic capacity gaps that affect service delivery. i | REINFORCE FINAL EVALUATION USAID.GOV Engagement: At the national level, Reinforce helped institutionalize best healthcare practices by supporting implementation of national-level monitoring and evaluation (M&E) guidelines and revising supervision and facility readiness tools for MOH. Reinforce staff in Dili and Covalima engaged national-level stakeholders from MOH and international partners in quarterly, annual, and technical working group meetings, shared reports, and held occasional one-on-one consultations, though partner perceptions of the utility of these meetings varied. At the MHS level, Reinforce used microplanning effectively to institutionalize best healthcare practices among municipality staff, village leaders, and facilities. Reinforce and MHS created a shared workplan to build facility staff skills, support community outreach, and engage Covalima subdistrict administration and village councils through microplanning. Microplanning involved reviewing community health data with facility staff, community members, and MHS officials to create action plans that address gaps in immunization coverage and service delivery. Capacity Strengthening: At the facility level, Reinforce instigated strong and consistent follow￾up after training (FUAT) and provided additional human resources to strengthen capacity of health service workers. Staff respondents at both Health Posts (HPs) and Community Health Centers (CHCs) reported considerable improvements in their clinical skills around safe delivery, newborn care, introduction of family planning (FP) practices and contraceptive methods, and integrated management of childhood illnesses (IMCI), namely in identifying and strictly following standards. The team found evidence that facility staff improved their ability to conduct community outreach, health education, and Behavior Change Communication (BCC) messaging with communities but relied heavily on Reinforce to provide additional staff support. At INS, Reinforce’s technical capacity building led to development of guidelines, standard operating procedures, and training manuals that institutionalized adherence to international health quality standards. Reinforce helped INS address specific gaps highlighted in its 2014 Situational Analysis, including conducting training needs assessments, updating World Health Organization (WHO) guidelines and protocols, and establishing three national clinical practice sites. Reinforce’s technical capacity building assistance reflected current WHO guidance and helped Covalima health facility and INS staff meet international standards and deliver quality services. Key gaps: The evaluation team found evidence of the following gaps in institutionalizing best healthcare service practices in Covalima: • Stakeholders do not consistently recognize their shared responsibility and ownership of the model municipality in part because Reinforce engaged them separately. Reinforce engaged stakeholders in various ways but never brought the MOH, INS, and international partners together, besides at launch. • There is little to no actionable data about how management capacity at higher levels has improved. Reinforce did not conduct capacity assessments for the MOH, MHS, or INS. • Through microplanning, stakeholders made decisions better based on data, though they still lack capacity to consistently produce and analyze usable data. Outside of microplanning meetings, stakeholders did not consistently use data; while many key informants said they were interested in and committed to using data, to date no CHC has conducted the full process of collecting and entering data, holding microplanning meetings, and coordinating and completing action plans based on that data. Further, the MOH does not have a unit whose sole function is to review and analyze data, leaving it with inconsistent capacity to use data to intervene and correct course. • Because stakeholders have limited human resources and finances, they find it challenging to institutionalize best practices in health system services. Because of system-level weaknesses—especially limited human resources—they cannot fully use Reinforce’s assistance to USAID.GOV REINFORCE FINAL EVALUATION | ii strengthen their capacity, either now or in the future. Therefore, health workers and MOH staff will continue to rely on partners like Reinforce for capacity strengthening. EQ 2: What bottlenecks, if any, in the capacity and commitment of stakeholders (MOH, municipality health services, health facilities, and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? This evaluation identifies four bottlenecks preventing the institutionalization of good healthcare service delivery: Budgeting: The MHS has limited budgeting capacity, few controls over budget management, and poor information about municipal-level budget allocation and timelines, stunting progress in community outreach, maintenance, and operations. While the Covalima MHS submits annual budgets with line items for community outreach, maintenance, and operations, national￾level approval of these budgets is severely delayed. Because MHS officials do not know why budget approvals are delayed, they cannot manage CHC and HP expectations. CHC managers in turn say they cannot manage facilities or address HP needs effectively because they do not have enough information about the budgets. MOH does not internally prepare advance budgets by forecasting and anticipating future activities, and it relies on external partners like Reinforce to help respond to immediate needs. Both facility and MHS staff praised microplanning, but because the action plans that facility staff create during microplanning are not tied to budgets, the facility staff cannot plan or implement these activities efficiently. Decision-making: While MHS controls operational expenses like cleaning and small repairs, municipality and sub-district-level authorities say they cannot make autonomous decisions about hiring, procuring equipment, and other large expenses. Although the Constitution of Timor-Leste explicitly endorses decentralization, municipal-level staff in Covalima say this does not match their experience. According to these staff, government at the national level is not transparent about the reasons for its financial and programmatic choices and does not communicate expected timelines for lower-level action or responses. CHC and MHS both expressed frustration about unresponsive central leadership. None of the health facility staff interviewed knew how data fed into national-level decisions; although facilities have shared data that reveals gaps in capacity, their requests for assistance remain unanswered. Additionally, the central government did not pass proposed frameworks that would clarify decision-making authority. Coordination and Communication: The evaluation team found no evidence of a knowledge￾and data-sharing culture across departments. Therefore, well-documented gaps remain unaddressed. National and lower-level authorities do not communicate well. None of the staff at the MOH or MHS interviewed had documented evidence of the successes, lessons learned, or failures associated with their involvement with Reinforce that could be shared to inform current or future programming. While Reinforce operated across several MOH departments, these departments share information poorly or not at all, making activity implementation less efficient and poorly understood. Separate entities that train and place health workers do not collaborate, leaving staff distributed inconsistently by central level MOH management, without shared review or input from INS, MHS, or CHC management. Despite well-documented equipment and human resource needs, higher authorities have not addressed these needs, and there is no shared inventory that would provide “big picture” insight Pervasive bottlenecks in budgeting, decision-making, staff motivation, and financial management limit the capacity and commitment of key actors, including MOH, MHS, INS, and facility staff, to adopt and promote good practices in Covalima. iii | REINFORCE FINAL EVALUATION USAID.GOV about equipment needs. Although the National Logistics Department is responsible for countrywide supply procurement and distribution decisions, it does not receive monitoring and evaluation data that would help inform its decisions, and national and municipal logistics officers do not make field visits to assess needs. Motivation and professional development: Because the MOH does not properly support or incentivize difficult and remote work, remote locales in Covalima lack quality healthcare. MOH guidelines for minimum numbers of staff at facilities are unmet, and Covalima facilities have less than one-third of the staff required at both HPs and CHCs. Isolated staff in remote areas are overworked, cannot leave their posts for training, and are unprepared to take on leadership roles. Continuing to do outreach tasks in the absence of Reinforce seemed daunting to facility staff; while outreach is listed in their job descriptions, transportation and additional staff are needed to efficiently conduct mobile clinics and outreach activities. Remote workers have no career path, either through a recognition system or opportunities for additional pay. Both INS trainers in Dili and trainers in Covalima feel the burden of working as a full-time health worker and as a trainer without adequate pay or financial security (i.e., per diems and travel allowances). Facility staff have improved their clinical skills but lack equipment, infrastructure, and enough patients who request elective procedures (such as IUD insertion). Without regular practice, staff cannot maintain these improved skills. EQ 3: What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? Training: INS and municipal trainers need organizational and financial support for per diem and transportation, and trainees need clear communication about the training resources available to them. INS staff in Covalima will continue to use the training participants’ database and training needs assessment. A decentralized training approach to allow for municipal-level authority over trainers’ activities is more cost effective and can more flexibly address local training needs, but the MHS has not yet mobilized the Covalima training team. Although the clinical practice site model is being scaled up to additional municipalities, this model will need continued investment to ensure accessibility and usefulness. To sustain service delivery improvements in Covalima, INS relies heavily on donor and partner financial support, yet INS and key stakeholders have not yet reached a technical agreement about responsibilities, budget, and accountability. Data collection and management: MHS and facility staff feel confident in collecting and entering data and ensuring good quality data, but capacity for continued, independent comprehensive analysis and reporting is lacking. MHS plans to continue executing facility readiness assessments, supportive supervision, and Routine Data Quality Assessments (RDQAs) with the national M&E Department. However, the M&E Department confirms limited capacity to conduct consistent follow￾up with Covalima staff in the absence of Reinforce and to improve quality control at the national level without additional M&E staff. Respondents praised microplanning at the municipal level, but facility staff are less optimistic about continuing microplanning without Reinforce covering transportation and meals. Community engagement and health promotion: For community outreach to continue, facilities will need funding for transportation, enough human resources, and motivated community leadership. Outreach was included in staff job descriptions prior to Reinforce, so Reinforce’s engagement in community outreach added additional labor for activities in addition to staff transportation. However, staff are not confident that they can continue all of these activities independently. On the other hand, night Training, M&E, and community outreach show promise to be continued after Reinforce but are not without flaws; MOH and MHS lack documented strategies or plans for sustainability. USAID.GOV REINFORCE FINAL EVALUATION | iv events and focus group discussions with expectant mothers and community members show promise of being continued by facility staff. A community-led transport model, TraKom, uses community members with functioning vehicles who voluntarily help when emergencies arise and has shown success in several communities, though it relies on continued engagement and motivation of community leaders. The government mandates Servisu Integrado Saúde Communitária (SISCa, Integrated Services of Community Health) as outreach and health promotion activities under the basic services package, but the number and frequency of these has dwindled in the last four years, owing to construction of new HPs and inconsistent funding. Financial planning: Government reliance on international partners for financial support in Covalima and nationwide has impacted its ability to independently continue to manage budgets. Without financing, government commitment means little. The government has maintained artificially lower budgets because donors and partners have paid for activities that would otherwise require government funding. In particular, Covalima MHS reduced its budget because Reinforce contributed financially to outreach, including fuel, other transportation costs, and per diems. As a result, staff are not used to budgeting for the true cost of these activities. Although MOH/MHS believe that other partners or donors will commit to continue supporting health service delivery in Covalima, no such commitment has been solidified, no partners are currently planning this support, and many international donors are scaling back their maternal, newborn, and child health assistance. EQ 4: What are the capacity and commitment of the GoTL, the MOH, and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? Capacity: Though INS’s draft national Strategic Plan 2020–2024 includes key INS activities bolstered under Reinforce (such as training needs assessments [TNAs], certifications, and guidelines for all training areas), these will depend on partner support and associated budget allocations. Though it will require more investment, many respondents praised the practice site and skill lab model for training new health facility staff, and the MOH has already expressed commitment to replicate it in other municipalities. The national M&E Department will continue to use the Facility Readiness tool (revised with UNICEF and Reinforce assistance), which is the recognized gold standard for assessing and monitoring facility needs nationwide. However, shortages in human resources and funding for transportation will affect the five￾person Department’s ability to train or coach staff outside Covalima and fully scale monitoring and assessment tools to the rest of the country. Commitment: Many government staff do not feel confident in their ability to replicate the activities in Covalima in other municipalities due to an uncertain political climate, budget concerns, and a reliance on international donors and partners. The government acknowledges responsibility to replicate the Covalima model but has not begun planning or identifying responsible parties, activities, and timelines to do so. Replication of the model will rely on integration and coordination of multiple stakeholders including MOH, INS, international partners, and MHS, but there is no evidence of commitment among any of these actors to a shared strategy moving forward. Neglected systemic issues have plagued the health sector for years. Health education in Covalima communities improved some healthcare-seeking behaviors, but HPs and CHCs are understaffed and ill-equipped to consistently address community needs because they often lack transport, equipment, medicine, electricity, and water. Without these basic resources, HPs and CHCs cannot sustain key Many respondents were enthusiastic about adapting the Covalima model in other municipalities, and some activities like development of clinical practice sites have already scaled. However, MOH and INS staff lack confidence in replicating activities without guaranteed funding or human resource capacity. v | REINFORCE FINAL EVALUATION USAID.GOV activities or provide comprehensive essential health services. Several partners and MHS staff felt that the government should focus on securing resources needed to establish a successful model municipality before any thought of scaling up. EQ 5: Beyond Covalima, how has the Reinforce activity strengthened the capacity of INS? Standardization: The key support Reinforce provided was standardizing guidelines, protocols, checklists, and curricula, which in turn improved INS training staff’s ability to adhere to strict competency standards for service delivery. The training and FUAT database designed and implemented under Reinforce created a system for quality improvement guided by data that ensures a comprehensive review of all health staff receiving training at INS. The database allows INS staff to review the full spectrum of courses offered, the number of staff receiving training, and those still in need of both initial training and FUAT, as well as tracking those who have ‘passed’ or ‘failed’ FUAT and need further monitoring. Reinforce supported the United Nations Population Fund (UNFPA), INS, and MOH in drafting a standardized curriculum for MNCH and FP to be used in initial training and as guidance for follow-up competency assessments that can be used in all municipalities. The database and curricula also ensured the improved quality of FUAT provided both in and outside Covalima. For the clinical practice sites to be successful, MHS and MOH are responsible for securing the infrastructure and equipment needs to enable the following of guidelines. However, human resource shortages, including within the INS Department of Standardization and Quality Control, will greatly affect INS’s ability to efficiently address training needs in all sites. Challenges: INS's challenges identified in 2016 are still the main issues impacting sustainability. These challenges include budget uncertainty, poor coordination and communication, limited human resources, and limited capacity of trainers. Reinforce supported INS in establishing a model for junior and advanced trainers, but INS has not hired enough trainers to meet nationwide training demand. Trainers at both the national and municipal levels need more professional development to feel confident in their roles, as well as clear understanding of job expectations and responsibilities. Supervision and scheduling needs are uncertain, and trainers are unmotivated because they are overworked and undercompensated. Continued success for INS will rely on improved coordination with MOH and the many partners that support its work, in an environment where continued funding is uncertain. RECOMMENDATIONS The evaluation team conducted a findings and recommendations workshop in February 2020 with representatives from the MOH, Covalima MHS, and INS to collect feedback on evaluation findings and identify how to best coordinate and address bottlenecks in order to support health facility staff in the short and long term. Recommendations elicited during this workshop are summarized alongside other recommendations from the evaluation team below (a full list of recommendations is included in the last section of this report). Capacity strengthening for health • Future health capacity strengthening activities by international partners must begin with capacity assessments that cover management, administration, operations, and communications across the healthcare system and at all levels. INS capacity has improved in promoting adherence to national and international clinical service delivery standards but will need continued investment in human resources and better coordination with MOH, MHS, and partners to improve overall efficacy as a ‘center of excellence’. USAID.GOV REINFORCE FINAL EVALUATION | vi service delivery • PNDS should create a formal agreement with village-level community volunteers regarding expectations; outreach should be included in their ‘job’ descriptions. Volunteers can provide support and accountability for initiatives like TraKom and SISCas without expecting additional payment. • INS, MHS, and Clinical Practice Sites must formalize a workplan for trainers that includes local and municipal training and FUAT to create a delivery plan compatible with their facility schedules as healthcare providers. • INS should work with MHS, MOH Health Promotion, and partners to develop training curricula on health promotion and BCC so facility staff can confidently engage in outreach, especially with men and adolescents. • To better use clinical practice sites, INS should provide flyers to trainees to take home after leaving INS in Dili that include (1) hours of the clinical practice sites at Comoro, Vera Cruz, and Suai Referral Hospital; (2) the names of trainers at MHS and at the practice site; and (3) the topics each trainer teaches on. • To reduce facility staff reliance on INS in an uncertain funding environment and better use data from monthly supervision, CHC managers should facilitate monthly learning exchanges among staff at their facilities. • CHC management staff can initiate monthly client feedback surveys using community volunteers. These should be collected mid-month, given to CHC management for end-of￾month meetings, and included in microplanning. • The INS, CHCs, and MHS should introduce a mobile skills lab that draws on supplies from practice sites, brings select equipment to facilities for targeted training, and otherwise uses existing equipment within facilities. Using data for decision￾making • Hire a focused, senior-level data interpretation and management analyst for the national M&E Department who will facilitate in-service training to enable M&E staff to better serve the municipalities and train enumerators for large scale data collection efforts, such as the Service Availability and Readiness Assessment (SARA) or knowledge, attitudes, and practices (KAP). • Facility readiness assessments done in the field must be captured by the national M&E Department and disseminated to the Logistics Department and Procurement Department to make a collective decision for needs-based procurement. • Ministry of Finance (MOF) should hold training for MOH and MHS staff on accessing the Transparency Portal and interpreting budget data as well as providing guidance on costing and budget forecasting for activity planning. • CHC Managers and MHS must ensure each facility receiving new equipment from Reinforce inventories it in an Excel file that is updated monthly. This should be maintained by MHS and MOH staff and shared between facilities. • CHC Managers can organize microplanning meetings at a smaller scale without electronic equipment. Management can host monthly sessions internally to review data, discuss gaps, etc. MHS should enforce microplanning as a good practice and hold CHC accountable for completing plans. Human resources and considerations for the healthcare • As part of the MOH’s finalized Human Resource for Health (HRH) policy, the MOH Human Resource Department should include stipulations to increase employee motivation and incentivization for staff who consistently perform outside their job descriptions in the form of certificates of recognition or financial incentives. vii | REINFORCE FINAL EVALUATION USAID.GOV workforce • The MOH Human Resource Department must involve MHS and INS in decision￾making for staff distribution so that skills match population needs, trainer feedback from FUAT is held in high regard, and INS staff motivation improves. Cooperation, collaboration, and knowledge management • The MOH Policy and Cooperation cabinet should be responsible for ensuring that partners and the respective MOH departments operate cohesively. The Cabinet should hold quarterly roundtable meetings with partners and the MOH to document successes, failures, and promising practices that can be replicated. • The MOH, INS, and the Policy and Cooperation cabinet should facilitate a meeting to review alternative and diversified support opportunities, including encouraging partnerships with medical associations. • MOH (supported by USAID) should develop a sustainability strategy that identifies which Reinforce-funded activities can realistically be maintained and how to continue building capacity in the next five years. The strategy should include metrics and targets for human resource and institutional capacity strengthening. 1 | REINFORCE FINAL EVALUATION USAID.GOV INTRODUCTION EVALUATION PURPOSE AND AUDIENCE The United States Agency for International Development (USAID) awarded the Reinforce Basic Health activity to John Snow, Inc., Research and Training Institute (JSI) in 2015 to increase the capacity of health service delivery in the Covalima municipality in Timor-Leste. The activity contains two broad components: (1) strengthening integrated health service delivery and (2) promoting healthy behaviors in the target municipality of Covalima. USAID/Timor-Leste contracted Social Impact, Inc. (SI), to conduct a performance evaluation of the Reinforce Basic Health Services Activity (Reinforce) to assess the extent to which the Reinforce Activity has strengthened healthcare service delivery in Covalima, how it could be further improved, and how it could be scaled to other municipalities in Timor-Leste. The SI evaluation team used a mixed-methods approach to conduct this evaluation, combining qualitative data collection with a quantitative analysis of secondary data. Working in both the capital of Dili and in the targeted Municipality of Covalima, SI gathered qualitative data through key informant interviews (KIIs), focus group discussions (FGDs), and a findings and recommendations workshop. The team collected quantitative data through a rapid mini￾survey and a review of relevant secondary data. SI’s four-person evaluation team balanced evaluation methodology expertise, contextual understanding, and health sector experience. This evaluation, conducted in Reinforce’s final year, provides stakeholders with evidence of the capacity and commitment of the Government of Timor-Leste (GoTL) to introduce Reinforce’s approach elsewhere in Timor-Leste and to further strengthen the healthcare service system in Covalima. Stakeholders include USAID, JSI, and the GoTL, including the Ministry of Health (MOH), Covalima Municipality Health Service (MHS), and the Instituto Nacional de Saúde (INS, or National Health Institute). As USAID supports the GoTL on its “Journey to Self-Reliance” (J2SR), this evaluation is intended to inform USAID about the capacity and commitment of stakeholders to improve quality of health services delivered in facilities throughout Timor-Leste in the future. Further, this evaluation assesses the factors necessary to meet health service delivery standards by identifying responsible parties and gauging intentions and accountability mechanisms. Finally, this evaluation explores how these interconnected factors and actors within the broader system and environment contribute to sustainability, and to what extent Reinforce has influenced the future strength of the system. EVALUATION QUESTIONS The performance evaluation was designed to answer the following Evaluation Questions (EQs), outlined in TABLE 1 below: TABLE 1: EVALUATION QUESTIONS EQ1 How did Reinforce Activity engage and strengthen the capacity of stakeholders (MOH, INS, municipality health services, health facilities and local leaders) to institutionalize best healthcare service practices in Covalima? How effective have they been in engaging and strengthening the capacity of stakeholders? EQ2 What bottlenecks, if any, in the capacity and commitment of stakeholders (MOH, municipality health services, health facilities and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? EQ3 What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? Which part(s) of the model is more promising in terms of sustainability? Why? What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? USAID.GOV REINFORCE FINAL EVALUATION | 2 EQ4 What are the capacity and commitment of the GoTL, the MOH and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? EQ5 Beyond Covalima, how has the Reinforce Activity strengthened the capacity of INS? Where are the gaps, if any, in INS’s capacity and commitment to conduct and to manage in-service training and continued education of health professionals in Timor-Leste? How has Reinforce mitigated these gaps? What, if any, are capacity gaps that remain? 3 | REINFORCE FINAL EVALUATION USAID.GOV BACKGROUND CONTEXT While Timor-Leste has made significant development and democratic progress since winning its independence in 1999, it has faced prolonged structural challenges, including poverty and poor access to basic human services like healthcare. In 2011, Timor-Leste developed a 20-year national health plan, the National Health Sector Strategic Plan (NHSSP) for 2011–2030. The NHSSP identifies four health system priorities: (1) provision of health services, (2) investment in human capital, (3) infrastructure investment, and (4) health management and administration. Since 2002, the health system has progressed in establishing basic services but still struggles to provide vital maternal and child health services, family planning (FP), immunization coverage, and emergency care in the more remote and mountainous areas where most of the population resides. Additionally, almost one quarter of the population is aged 15–24, and while health investments for those in this age group, especially adolescents, set the stage for a lifetime of positive health behaviors, those investments are currently lacking. Between the 2009/10 and the 2016 Demographic Health Surveys, critical FP and Maternal, Newborn, and Child Health (MNCH) indicators improved, such as a greater than 50 percent decrease in maternal mortality and nearly a 30-percentage point increase in deliveries by a skilled birth attendant. At the same time, other indicators experienced very limited improvement, such as the percent of married women 15–49 using modern contraceptive methods and neonatal mortality rates. Some key indicators worsened, including under-five wasting and percent of children age 12–23 months who have received all basic vaccinations. The Public Health System of Timor-Leste: According to the Ministry of Health (MOH), primary healthcare in Timor-Leste is organized through a network of health posts (HPs) and community health centers (CHCs) at subdistrict and district (or municipality) levels, alongside outreach services such as mobile clinics, Servisu Integrado Saúde Communitária (SISCa, or Integrated Services of Community Health), monthly community-based services such as health screenings and distribution of health information, and school health programming. The MOH allocates facilities and services based on population (rural versus urban) and accessibility. At the village (suco) level, the HP is the basic unit for providing primary health services and serves up to 2,000 (rural) or 5,000 (urban) citizens. At the district and subdistrict levels, CHCs provide specific services for up to 12,000 (rural) or 15,000 (urban) citizens. HPs have a staff of up to seven, including a general medical officer, nurse, midwife, pharmacist, and laboratory technician, while CHCs have a staff of up to 30, adding public health technical officers, a family physician, finance staff, and specialists in dentistry, radiology, and ophthalmology. The CHC is responsible for technical support and ensuring the availability of adequate resources, equipment, drugs, and consumables for all administrative areas of the HPs. Both HPs and CHCs provide services for Antenatal Care (ANC), FP, basic emergency obstetrics and newborn complications (BEmONC), Essential Newborn Care (ENC), immunization, nutrition, health promotion, and communicable and non-communicable disease treatment. CHCs provide additional services including surgery, counseling, treatment for the Human Immunodeficiency Virus, premature newborn care, addiction care, chronic disease care, and longer duration emergency management. Beyond the district level, the health system includes referral hospitals and national and private hospitals. Health Workforce: The health system needs a larger workforce, and staff in the health system need improved skills in order to provide comprehensive services. Most Timor-Leste doctors received Cuban Medical Brigade (CMB) training after 2004, and the University of Timor-Leste collaborated with the CMB to train medical students beginning in 2008. Nurses and midwives train at the National University of USAID.GOV REINFORCE FINAL EVALUATION | 4 Timor-Leste (UNTL) and private universities (and previously at the INS); there are twice as many male nurses as female nurses working at the community level, and almost all midwives are women. While medical doctors sign an agreement with the government to work in the public health sector for six years, limited infrastructure and resources constrain public health services, and the country needs more doctors. In addition to needing a larger workforce, the health system needs to provide targeted training and skill development for existing workers. A WHO national health labor market survey conducted in 2014 with doctors, nurses, and midwives in all 13 districts of Timor-Leste showed that lack of knowledge was significantly associated with poor performance. Respondents scored low overall on clinical competence, especially in rural areas, and despite training opportunities afforded by the government and private entities, roughly half (52 percent) of respondents indicated that “there is not enough opportunity to learn current medical knowledge.” The spectrum of knowledge in Reproductive, Maternal, Neonatal, Child, and Adolescent Health (RMNCAH) was also reported to be very limited, with less than one third of respondents having been trained on key topics. Of the survey respondents, only 27 percent received training in Integrated Management of Childhood Illnesses (IMCI), 24 percent on labor and delivery, 19 percent on pre/postnatal care, and only 11 percent on FP. WHO recommendations included regular in-service training and visits from specialists, along with more efficient supervision and monitoring tools to increase health workers’ competence. The report concluded that the two priority areas for improving doctor performance in Timor-Leste are (1) improving the work environment and performance of rural doctors and (2) ensuring compliance with clinical protocols. ACTIVITY BACKGROUND To respond to these systemic challenges, in 2015, the Reinforce activity was initiated to build stronger health service capacity, both nationally and in one Timor-Leste municipality by creating a “model municipality” from which others can learn, in order to improve quality and use of services, leading to better health outcomes in selected areas of RMNCAH. Through its implementing partner (IP), JSI, Reinforce provided targeted technical assistance to MOH and INS stakeholders and public health facility staff. The activity focused on health provider skill development, institutional capacity development, and provision of quality health services in Covalima municipality, focusing on the following key areas: (1) generate baseline evidence of existing gaps in service delivery, (2) improve health service delivery, (3) improve community use of health services, and (4) use health data for decision-making. Reinforce defines a model municipality as: In a model municipality, well-trained and competent health providers deliver quality FP and MNCH services in fully equipped and people-friendly health facilities where providers receive consistent, high￾quality supervision from MOH and INS experts. Health facilities are easily accessed by community members, who have the knowledge and skills to demand quality, client-centered services. Health providers and the facilities in which they work are supported by a municipal health system that uses data to make informed financial and programmatic decisions. In a model municipality, communities are empowered to work with government and civil society structures to generate innovative solutions to their health challenges (“Covalima: Becoming a Model Municipality,” USAID’s Reinforce Basic Health Services Project/JSI). After inception, Reinforce created Vision 2020, a strategic approach to address the above focus areas by the end of the activity in 2020. The activity’s two overarching strategic approaches are: 5 | REINFORCE FINAL EVALUATION USAID.GOV 1. Build capacity in service and systems. Reinforce strengthens the capacity of the INS to improve health provider skill and competency levels and serve as a clinical training and practice center for excellence. Reinforce enables MOH health providers to commit to building service and system capacity through in-service FP training (including counseling and provision and promotion of methods), certification, follow-up after training (FUAT), and supportive supervision where health providers work. This approach promotes adherence to international and national health standards and policies, provision of life￾saving maternal and child health (MCH) care and improved adolescent care, integration of RMNCAH services, improved referrals, greater use of services, and reinforcement of respectful care. Training is targeted to doctors, nurses, midwives, and technical officers. 2. Improve healthy behaviors in the home, community, and among service providers. Reinforce improves the links between community and facility care, increases health promotion and education by providing clinical training for HPs and CHCs, and supports community-based initiatives like SISCas, a monthly community program to increase service accessibility. This approach increases consistent and accessible health information; promotes family planning and awareness of maternal danger signs before, during, and after pregnancy; reduces barriers to transportation; and produces materials to promote FP and improved nutrition, all in an effort to increase demand for quality services. These approaches aim to prioritize adolescents and youth, eliminate inequities, and integrate service delivery across the RMNCAH spectrum in one model municipality. The focus areas are intended to help move Timor-Leste—over the five-year activity—toward lower maternal and neonatal mortality, lower total fertility, more couple years of protection (CYPs), more women receiving at least four antenatal care visits, more deliveries at health facilities, and more postnatal women and newborns receiving postnatal care. Reinforce worked in all seven CHCs in Covalima Municipality and 23 HPs since 2016, as well as the two INS clinical practice sites in Dili and one site in Covalima (Suai). DEVELOPMENT HYPOTHESIS Reinforce’s development hypothesis predicts that enhancing and maintaining the skills of the health workforce to deliver on the continuum of care model for high￾quality FP and RMNCAH services will contribute to improved quality of services. Furthermore, if facilities are adequately supported with resources, medical equipment, and medicines, and communities are supported with transportation, improved health education, and communication, Reinforce’s capacity building interventions may enhance the use of and demand for services and lead to improving maternal and neonatal outcomes. The activity’s approach rests on the assumption that to improve health outcomes, community members must demand high-quality services and health providers must have the knowledge, skills, and infrastructure to provide those high-quality services. By reinforcing connections between health services and people, the activity also aims to improve healthy behaviors and use generated data as solid evidence for informed decision-making within municipality health services (MHS), expanding beyond the model municipality of USAID.GOV REINFORCE FINAL EVALUATION | 6 Covalima. See Annex III for the Statement of Work, which includes the full Theory of Change for Reinforce. DESIGN AND METHODOLOGY EVALUATION DESIGN To answer the evaluation questions, the evaluation team mainly used a mixed-methods approach, including key informant interviews (KIIs) and focus group discussions (FGDs). The team also collected quantitative data through a rapid assessment survey with recipients of services at Covalima public facilities and drew on existing secondary quantitative data. KIIs and FGDs allowed the team to collect in-depth information from those with the richest knowledge of relevant topics. Secondary data analysis included Reinforce monitoring and performance data, baseline surveys, and internal assessments, which complemented the primary descriptive data from interviews and FGDs. The evaluation team collected data over a period of three weeks in January 2020, in Dili and Covalima municipalities. The team comprised six persons: one native English speaker, one non-Tetum speaker, two Tetum speakers, and two Tetum-English interpreters. For interviews, focus group discussions, and surveys, team members posed questions in Tetum or English depending on the context and comfort level of the respondent. A minimum of a two-person team is a data collection best practice, which allows one person to lead the interview and the other to take notes and ask clarifying questions as needed, and one interpreter accompanied each sub-team for data collection. DATA COLLECTION METHODS Desk Review: Team members reviewed documents to understand Reinforce’s design and implementation, national strategies, the historical and political context, and existing research on health systems in Timor-Leste. The team used the preliminary desk review findings to inform data collection protocols and as a data source for triangulation against qualitative data collected during fieldwork. The team received additional documents after fieldwork, warranting further review to validate findings and identify any discrepancies. A complete list of documents reviewed is in Annex IV. Key Informant Interviews (KIIs): The team identified key informants in consultation with USAID/Timor-Leste and JSI, along with information from the desk review. The team requested a comprehensive listing of male and female priority stakeholders at both the national and municipal (Covalima) levels, including health facility contacts and coordination bodies. This process ensured that the team interviewed those most involved in Reinforce’s activities and that interviews were scheduled at times and locations convenient for informants. This sampling approach also helped the team identify female respondents in order to account for gender and social dimensions. Every participant provided informed consent, and the interviews used KII guides developed for each participant category. SI’s Institutional Review Board (IRB) reviewed and approved these guides, and the two professional interpreters on the team translated them into Tetum. The translated tools were also back-translated for clarity by the two Tetum-speaking team members and tested among team members and during the first KII and FGD. After testing, all team members revised the English tools to further streamline the language, then retranslated the final tools for data collection in Dili and Covalima. Some technical terms and phrases such as ‘capacity,’ ‘bottlenecks,’ ‘stakeholders,’ and ‘institutionalization’ are rephrased in Tetum, and during interviewing and group discussions, team members used prompts and examples to illustrate concepts and allay any confusion around terms. 7 | REINFORCE FINAL EVALUATION USAID.GOV The evaluation team’s in-person KIIs included open-ended questions with individuals in both English and Tetum and lasted 1–2 hours. The team conducted 74 KIIs with 51 men and 30 women (some KIIs included more than one respondent) including representatives from JSI, INS, GoTL (specifically Ministry of Health and Finance representatives), MHS Covalima, health staff (including management, doctors, nurses, and midwives), community leaders in Covalima, partners and potential donors, and USAID. TABLE 2 identifies key respondents per stakeholder category: TABLE 2. KII PARTICIPANTS BY STAKEHOLDER TYPE (Total 74 KIIs) STAKEHOLDER CATEGORY # MEN # WOMEN TOTAL JSI 3 7 10 Partners/donors/USAID 11 8 19 GoTL (MOH, MOF) 5 2 7 MHS Covalima 5 1 6 INS 2 1 3 Community leaders 3 0 3 Health Facilities (CHC/HP/Hospital) 22 11 33 Chiefs 8 0 8 Doctors 9 4 13 Midwives 0 5 5 Nurses 4 1 5 Other technical staff (pharmacy, HMIS) 1 1 2 Total 51 30 81 Focus Group Discussions (FGDs): The evaluation team conducted FGDs with INS national and municipal-level trainers and health facility staff (doctors, nurses, and midwives) in Covalima who received training from the INS with support from Reinforce, to obtain in-depth information on their perceptions of the training process, clinical skill building, system strengthening, support mechanisms, and sustainability of training and activities at clinical practice sites. The team used non-probability sampling to select the focus groups and their participants. For the two FGDs with INS trainers (Dili) and INS-trained trainers (Suai), the national- and municipal￾level trainers were selected based on their availability, as trainers are full-time nurses and midwives. In Dili, the team met with male and female in-house trainers. In Suai, the team met with select female midwives who had attended a Training of Trainers (ToT). The evaluation team conducted the other 10 FGDs at five CHCs and five HPs across the seven subdistricts of Covalima. In HPs with two or fewer staff, the team conducted KIIs with nurses or midwives, since FGDs were not feasible. The team selected FGD participants purposively and in consultation with CHC management in each subdistrict. The Local Evaluation/Health Specialist facilitated each FGD using the Tetum language and the FGD protocol in Annex V. Team members obtained informed verbal consent from all participants to take notes and use a recording device. The team conducted 12 FGDs with a total of 47 participants, including 26 men and 30 women (see TABLE 3). More midwives than nurses are trained on MNCH and FP, and midwives are nearly all female in Timor-Leste. FGDs included both male and female nurses, and the evaluation team took the gender breakdown of staff within the INS trainers and within facilities into consideration during sampling, where necessary ensuring that both male and female voices were present and heard during discussions. Doctors, nurses, and midwives were also interviewed separately, and FGDs only included nurses, midwives, and technical staff to prevent dynamics of hierarchy in group discussions. There was an average size of fewer than 5 persons per FGD due to staff presence and availability. USAID.GOV REINFORCE FINAL EVALUATION | 8 TABLE 3. FGD PARTICIPANTS BY STAKEHOLDER TYPE (Total 12 FGDs) STAKEHOLDER CATEGORY # MEN # WOMEN TOTAL INS 3 7 10 Health Facilities (CHC/HP/Hospital) 23 23 46 Nurse 21 - 21 Midwives - 22 22 Other technical staff (pharmacy, HMIS) 2 1 3 Total 26 30 56 Rapid Assessment Survey: The evaluation team administered small-scale rapid assessment surveys with clients at the targeted health facilities in Covalima. Initially, the team anticipated conducting surveys at community outreach activities and during clinic hours, but outreach activities did not occur during data collection (in some communities, they had not occurred in months). Team members instead focused on conducting surveys at facilities. The evaluation team planned to schedule the survey data collection in coordination with a day where clients might converge at the facility, such as for immunization or scheduled ANC days, but facilities did not identify any days allotted for these activities during data collection. Medical staff instead approached clients at health facilities upon arrival. While patients awaited being seen, medical staff asked them if they were willing to participate in a short survey while they waited or after they finished receiving services. Self-selection, rather than team member selection, ensured that clients would not feel coerced into providing information, and staff reassured all potential participants that their participation had no bearing on their ability to receive services on that day or in the future. The rapid assessment survey allowed the evaluation team to garner perceptions of the quality of health services in an accessible way (unlike in a formal KII or FGD, where the time commitment is longer and more difficult to arrange, particularly for mothers with young children). The survey asked a series of closed-ended questions about respondents’ perceptions of changes in the quality and accessibility of health service delivery over the duration of the activity, as well as several open-ended questions to provide broader feedback on experience with health outreach and social determinants of health-seeking behaviors. Two evaluation team members conducted these surveys face-to-face in Tetum, with one team member asking questions and one taking notes on open-ended responses, assisted by an interpreter. All survey respondents provided informed consent, and team members followed procedures for maintaining confidentiality and privacy. Surveys were conducted in private spaces at facilities. At facilities that could not guarantee a private space, no surveys were conducted. As outlined in Table 4 below, the evaluation team conducted 31 surveys with four men and 27 women at six CHCs and five HPs. The sample of survey participants was relatively small because of the lack of scheduled community activities, the limited hours in which facilities were open (8:30–12am) during which the evaluation team also had to interview facility staff, and a lack of spaces to conduct surveys privately. These issues are also discussed under “Limitations” below. The rapid assessment survey results are in Annex I. TABLE 4. SURVEY PARTICIPANTS BY STAKEHOLDER TYPE STAKEHOLDER CATEGORY # MEN # WOMEN TOTAL Health facility clients (CHC/HP) 4 27 31 Total 4 27 31 9 | REINFORCE FINAL EVALUATION USAID.GOV SAMPLING FOR PRIMARY DATA COLLECTION The evaluation team used a non-probability, purposive sampling strategy supplemented by snowball sampling to collect targeted data for each of the five evaluation questions. Purposive sampling involved selecting key informants/participants at each location based on involvement with Reinforce and within the broader public health infrastructure of Timor-Leste. The team consulted with USAID/Timor-Leste and JSI to finalize the complete list of facilities to be visited and the anticipated number of KIIs and FGDs to occur at each. Public facilities across all sub-districts in Covalima include seven CHCs and 21 HPs. The MOH provided the list of facilities, which JSI and MHS verified. Conversations with MHS confirmed those on the list that were not currently staffed or operating at the time of the evaluation, even if they were open at Reinforce’s inception. The sample of facilities included all seven CHCs (100%, n=7) and 17 of the 21 HPs in Covalima (81%, n=21; see Annex III). In subdistricts with more than one HP, the team selected HPs by the operating status of the health facility. The evaluation team did not reach all HPs because two were not staffed and two were not operational. Out of the three INS clinical practice sites, the team prioritized the site in Covalima because of the focus on Covalima as the model municipality. The team reached a total of 88 women and 71 men with 31 surveys, 74 KIIs, and 12 FGDs. Most of the KIIs and FGDs involved health facility staff. GENDER CONSIDERATIONS A limited number of women in senior management positions within the MOH, MHS, and INS were both appropriate respondents for this evaluation and available for interviews. The evaluation team asked CHC management for a list of staff who had undergone training and requested an equal number of male and female staff from the list for an FGD. As discussed above, final staff participation was solely determined by individual availability. FGDs with facility staff asked questions about both IMCI (male nurse) and MCH (female midwife) services so that both respondent sexes could provide input. The team found there was no noted hierarchy in these roles within facilities, except for the relationships between doctors and other staff members (male and female doctors were interviewed separately). Survey participants self-selected, and the number of women at health facilities far exceeded the number of men present and interested in being surveyed. DATA ANALYSIS Primary qualitative data: The team collected data over 23 days in January 2020 in Dili and Covalima. Team members recorded data collection progress and interview and focus group notes in a cloud database system. For qualitative data, team members took KII notes in real time and cleaned and synthesized electronic summaries daily, after each site visit. Digital recordings complemented interview notes to ensure accuracy. The team used content and comparative analyses to identify response categories and patterns and elucidate emergent themes and contextual factors among qualitative data. Qualitative data was disaggregated by stakeholder type, sex, sector/affiliation, and location, to capture any divergences. Primary quantitative data: Team members recorded the rapid survey data in Microsoft Excel, disaggregating results by location (facility or community-level), geographic area, and respondent sex. One person from each field team entered their survey data into the shared spreadsheet, which the Evaluation Specialist reviewed and cleaned, and the Team Leader further reviewed. Secondary Data Collection: The team collated and analyzed secondary quantitative data from Reinforce’s annual workplans, quarterly reports, and INS strategic plans baseline data. The analysis was exploratory and examined the breadth of capacity strengthening activities and results, as well as which activities were the most promising or challenging and have the potential to be sustained or replicated for USAID.GOV REINFORCE FINAL EVALUATION | 10 future scale-up. Data on FUAT and clinical practice site assessments requested and received from INS, MOH, and JSI were also reviewed in Excel to tabulate frequencies and review trends; however, facility readiness data could not be comparatively analyzed over time due to a change in instruments and incomplete data received from sites. Health system analysis: To identify key actors in the system and their relationships, the evaluation team explored causal loops that explain processes that lead to system behavior or functioning and predicted where bottlenecks do and can occur that threaten the longevity of the Covalima model now and could affect future scaling of the model to other municipalities. Health system analysis was an ongoing process during data analysis and was the main subject during the subsequent findings and recommendations workshop with MOH staff, where coordination, decision-making, and knowledge sharing bottlenecks were explored using a fishbone diagramming exercise and stakeholder relationships and responsibilities were identified using a RACI (Responsible, Accountable, Consulted, and Informed) approach. The analyses provided the evaluation team with further insight around key functionalities of structures that uphold best health delivery practices and used this information to complement qualitative findings around how these structures and systems have evolved over time, in addition to how these practices can continue to be supported outside Covalima. INTERPRETATION OF REPORT DATA Throughout this report, respondent feedback from qualitative data sources is specified according to themes and consensus across both a broad range of respondent categories (partners, government, service providers, etc.) and within specific groups (doctors, midwives, etc.). The evaluation team uses terms such as ‘most’ (referring to more than 75 percent of respondents), ‘some’ (more than one-third) and ‘several’ (less than one-third) to indicate the number of respondents repeating a common response. Because of significant concerns with protecting the anonymity of respondents, this report omits specific job categories, job titles, and gender in its analysis. For KIIs, limited numbers of key stakeholders would be immediately identifiable if information in the report were directly tied back to their conversation with the evaluation team; thus, informants signed written consent that they understood that information gleaned from interviews would be compiled across stakeholders to identify key themes and not identify any one respondent group alone. Further, quantification of specific responses would misrepresent population data and may be misinterpreted because of the small sample size (such as in the mini-survey, where n=31). As FGDs included several respondent groups within facilities, it is not possible to disaggregate individual responses from nurses, midwives, or technical officers. Wherever possible and appropriate, the team has noted where there was consensus or discord among any respondent groups. LIMITATIONS AND BIASES Representativeness: Given the limited availability of staff, the team could not meet the initial estimated sample sizes for KIIs, FGDs, and rapid surveys. There were no community activities organized in the period of the fieldwork, CHCs and HPs were understaffed compared to the ET’s expectations (based on MOH estimates), and far fewer clients came to visit the health facilities while the team was there than expected (in some instances this was due to recent rain and road conditions). Where possible, rather than identifying a sample of staff for inclusion in an FGD or KII, the team asked any available staff to participate in hopes of reaching greater numbers. While the team endeavored to examine population￾level outcomes through secondary data, the sample was relatively small and cannot be considered representative of the broader population of Covalima to a statistically significant degree. Further, male representation in surveys was low due to the small number of men presenting at facilities for FP (including counseling and provision of methods) and MCH activities and cannot be considered representative of the male population in Covalima. This sample size limitation is important to keep in consideration when interpreting the results of this evaluation, and sample sizes are noted where applicable. 11 | REINFORCE FINAL EVALUATION USAID.GOV Selection Bias: Given the purposive nature of the sample, there is a risk of activity implementers guiding the evaluation team to key informants who had favorable experiences of the activity. The team minimized this potential bias by ensuring that key informants represented a diverse range of roles with varying levels of engagement with the activity. Additionally, team members represented diverse backgrounds and levels of familiarity with the GoTL and were thus able to identify additional male and female informants outside the recommended list provided by JSI and USAID. Selection bias may also limit the robustness of quantitative data collected through the facility-based rapid survey, given that those who elect to both visit facilities for services and take the survey will have done so of their own volition and are not randomly chosen. Response Bias: Response bias is the tendency of respondents to provide answers that are not completely truthful, including a tendency to over-report positive feedback or under-report negative feedback (social desirability bias). Key informants may provide evaluators positive feedback about the activity because they want the program to continue, or respondents who are dependent on activities may share negative feedback in hopes that it will emphasize a need for further support. In other cases, respondents may be reluctant to disclose failures for fear that their critique will result in problems for their work. The team mitigated this type of bias by administering a clear informed consent script that encouraged honest responses and reassured respondents of their anonymity. The team conducted FGDs and surveys in a private space away from other patients and staff where responses could not be heard and where participation was not noticeable. The consent script included language disclosing that participation invokes no direct benefits and no direct harm, including for their employment or receipt of services. The evaluation team also used data triangulation to validate qualitative data so that diverse perspectives were recorded and represented for analysis. By comparing data provided in interviews against reviewed documents, the team ensured that no single biased data point could skew the analysis. FINDINGS AND CONCLUSIONS EQ 1: ENGAGEMENT AND CAPACITY STRENGTHENING How did Reinforce Activity engage and strengthen the capacity of stakeholders (MOH, INS, MHS, health facilities, and local leaders) to institutionalize best healthcare service practices in Covalima? How effective have they been in engaging and strengthening the capacity of stakeholders? This section addresses the effectiveness of Reinforce through understanding how the activity engaged stakeholders and initiated activities in line with its two approaches: 1) Build capacity in service and systems (for INS, MOH, and MHS) a. INS: improve provider’s skill and competency, serve as center of excellence/certification b. MOH/MHS: improve supervisory capabilities, standards of care, quality (doctors, nurses, midwives, tech officers) 2) Improve healthy behaviors in the home, community, and among providers c. Support community initiatives (SISCa), access to health information (especially for adolescents) and FP tools, and reduce transport barriers USAID.GOV REINFORCE FINAL EVALUATION | 12 JSI/REINFORCE SYSTEMS-LEVEL ENGAGEMENT Reinforce falls under the Development Objective Agreement (DOAG), a document governing USAID's foreign assistance to Timor-Leste signed by the Government of Timor-Leste and USAID in March 2014. Because of the existence of the DOAG, no separate agreement outlining rules of engagement and expectations between JSI and the MOH and MHS was initiated. JSI, as an implementer under this agreement, was advised not to enter into separate agreements with government entities. Reinforce worked with MOH to identify Covalima as the location for the model municipality. Reinforce’s first annual workplan was submitted to USAID prior to the selection of the model municipality, and at that time Reinforce had already begun discussions with MOH national departments (MCH, Monitoring and Evaluation [M&E], Health Promotion, Quality Control) and with INS. To engage stakeholders in the first year of the activity, Reinforce documents and stakeholders noted that Reinforce held several meetings with USAID, partners, and MOH in Dili to discuss the purpose and goals of the activity. In identifying a model municipality, JSI and MOH staff reviewed existing readiness data collected by the M&E department across all municipalities. The MOH selected Covalima based on the limited number of other implementing partners working on health in the municipality and its limited progress toward reaching readiness and health outcome indicator targets. After the Covalima launch in 2016, Reinforce conducted a workshop with members of the MHS and the participation of all health facilities, all community leaders (village, administrative post, and municipality), and all development partners active in the area to introduce Reinforce in Covalima. Reinforce began by improving management’s data-driven decision-making capacity, providing supplies and equipment to MOH, INS, and MHS, and conducting baseline assessments. Throughout the activity, documents and key informants confirmed that Reinforce staff in both Dili and Covalima engaged stakeholders in several ways, including quarterly and annual debrief meetings attended by MOH and partners covering a range of health topics and implementation updates, technical working group meetings on single topics such as MNCH or BCC (mainly attended by partners and sporadically by MOH), sharing reports, and, less frequently, one-on-one consultations. Reinforce’s design did not explicitly include organizational capacity strengthening but instead focused on improving management capacity to use data to make decisions. Reinforce and government representatives confirmed that support provided to the national and municipal levels centered on M&E and supervising staff to review data and make action plans. The first approach identified in 2015 involved capacity strengthening activities targeted to the national MOH, MHS, and INS. Over time, Reinforce also included provision of supplies and equipment such as photocopiers, laptops, printers, generators, and mobile phones to all three entities. Reinforce initiated two assessments that served as a baseline for project activities: a knowledge, attitudes, and practices (KAP) survey with community members to measure existing health information and behaviors and a Service Availability and Readiness Assessment (SARA), which quantitatively measured the presence or absence of equipment, necessary human resources, supplies, amenities, and infrastructure within each health facility. The first of its kind in Timor-Leste, the SARA highlighted gaps and strengths and weaknesses in 26 public and private health facilities in Covalima (23 public facilities included seven CHCs, 15 HPs, and one hospital, while private facilities included three clinics managed by individuals and the Catholic Church). MHS ENGAGEMENT AND STRENGTHENING At the MHS level, Reinforce supported MHS staff attendance at management trainings, introduced staff to the concept of microplanning, and guided staff on data entry and reporting procedures for DHIS-2. A common annual workplan between JSI and MHS was 13 | REINFORCE FINAL EVALUATION USAID.GOV created after Covalima was selected as the model municipality. Reinforce provided transportation and fuel costs for facility staff to conduct outreach activities and assisted facility staff with follow-up to families regarding child immunizations and antenatal care. MHS staff and senior health facility staff at select Covalima CHCs and at Suai Referral Hospital attended a Health Manager Training facilitated by St. John of God, who was contracted by Reinforce to improve management skills for personnel, administration, and operations. To improve data use in decision-making, Reinforce initiated microplanning, a strategy to review and use community health data with facility staff, community members, and leaders and MHS officials to make action plans to address identified gaps in immunization coverage and MNCH, FP, and nutrition service delivery within facilities. With the changing political leadership, MOH and MHS respondents noted that the MOH’s HMIS system underwent changes from a more traditional HMIS to the Demographic and Health Information System (DHIS)-2 system, currently the world’s largest open￾sourced, web-based platform.2 Reinforce conducted a training with MOH on the DHIS-2 platform in 2016, followed by a refresher training one year later. To date, there has not been a formal training for MHS on DHIS-2, but MHS staff received guidance from Reinforce on data entry and reporting using the system. MOH ENGAGEMENT AND STRENGTHENING At the MOH level, Reinforce supported implementation of official M&E guidelines and the Strategic Plan, provided technical assistance in data analysis, and updated facility readiness assessment tools. An M&E technical working group already existed through support from the (Australian) Department of Foreign Affairs and Trade (DFAT) when Reinforce began activities in 2016. Key informants and Reinforce reports confirmed that after Reinforce began, staff attended working group meetings and coordinated with the MOH to support the implementation of the M&E Guidelines (developed in 2013) and Strategic Plan (2015–2020) at the municipal level. Reinforce provided technical assistance to the national M&E Department, improving its ability to compare health outcomes across all municipalities and analyze existing data for presentation at Joint Annual Meetings. While the M&E Department was already using a tool developed in 2013 to gauge facility readiness (a checklist to identify a facility’s ability to provide services through the presence of necessary staff, equipment, and infrastructure), Reinforce provided inputs during the process to update the existing format (supported by UNICEF) to better capture the comprehensive information collected in the SARA. Reinforce revised supportive supervision tools in MCH and partnered with a local NGO to promote the design and development of training guidelines for adolescent reproductive health. Similarly, KIIs and documents confirmed that Reinforce revised the safe motherhood, FP, and IMCI supportive supervision tools used by the MCH department to be piloted in Covalima. For Health Promotion, rather than starting a formal training package for INS on adolescent health as was envisioned in the early stages of the activity, Reinforce instead partnered with a local youth-led NGO, Juventude ba Dezenvolvimentu Nasional (JDN), to develop training guidelines for adolescent reproductive health (ARH) and a package of information on behavioral change and health education for schools. Reinforce also provided financial assistance to the Health Promotion department to send their key staff person to the International Behavior Change Communication (BCC) Summit 2018 in Bali in order to present an MOH and Reinforce-produced fiction film (Benvinda Angela) awarded by the summit’s organizers. Reinforce staff shared quarterly reports with MOH on all outreach, BCC, and M&E capacity strengthening activities. 2 Retrieved from https://www.dhis2.org/about on Feb. 11, 2020. "Some projects focus on their objective and don’t do anything else. But with JSI they planned together… making an action plan with the government and not doing it alone, it is always in line with the MOH." -National M&E Department USAID.GOV REINFORCE FINAL EVALUATION | 14 INS STRENGTHENING Reinforce and INS reports and interviews confirmed that engagement with INS was guided by the results from an INS Situational Analysis completed in 2014, as well as the 2016 “Assessment of official documents used for providing health services and training among RMNCAH stakeholders” and “Clinical Practice Sites Baseline” conducted by the National Network for Clinical Training in Reproductive Health (JNPK-KR) from Indonesia. When Reinforce began in 2016, these prior assessments informed the engagement strategy by identifying key gaps, and JSI and INS created a memorandum of cooperation to address specific gaps over the course of the activity. As one of 12 different partners and donors providing support to INS under the strategic objectives in its 2015–2019 Strategic Plan, Reinforce addressed (1) Strategic priority 6 (“Enhancing quality of training, continuing medical education and research by developing curriculum, training modules and standard operating procedures”) and (2) Strategic priority 7 (“Establishing system for national standards and guideline-based and competency-based training”). Reinforce supported an INS training needs assessment, updated WHO guidelines and protocols for service delivery, and improved the organization and management systems of INS trainings. Reinforce and INS reports confirm that Reinforce financially supported the INS Research Department’s training needs assessment (TNA) at community health centers in Dili and all health facilities in Covalima for 186 service providers. As noted in FGDs and KIIs, facility staff received training from INS with updated WHO guidelines and protocols revised and adapted by Reinforce to follow for family planning, maternal and neonatal care, safe delivery, IMCI, and basic emergency obstetric and newborn care (BEmONC), and Reinforce provided financial support for providers to attend the training in Dili. In 2017, Reinforce reviewed the INS training database, made recommendations to improve its usefulness, and developed an Excel database to track training participants across the country (as already set up for Covalima), trainings attended, FUAT completed, and competency status (passed or failed). To assist in managing this information, Reinforce also worked with INS staff to create a filing system to store certifications, training reports, curriculum, and materials and FUAT results. Reinforce helped INS establish three clinical practice centers and supported certification of trainers. Reinforce worked with INS to establish three national clinical practice centers to enhance hands-on training for facility staff in Dili (Vera Cruz and Comoro) and Covalima (Suai). Financially supported by Reinforce, master trainers from the JNPK-KR in Indonesia traveled to Timor-Leste to update competency standards and facilitated a training of trainers (ToT) to certify new trainers, and Reinforce created a junior training model to promote ToT and reduce the burden on INS training staff. Lastly, Reinforce hired a national consultant to assist INS in developing its next Strategic Plan and an international consultant to externally evaluate the three clinical practice sites. SUB-DISTRICT STRENGTHENING Reinforce brought local leaders to public health events, microplanning meetings, and trainings on best practices in health. KIIs and FGDs with facility staff and community leaders note that Reinforce collaborated with members of the Covalima subdistrict’s administration and village councils to mobilize and attend community health promotion activities such as SISCas and invited them to participate in the microplanning meetings. Both male and female local leaders were trained on how to better understand their role in the microplanning process, as well as basics of interpreting data and using data to drive actions. Additionally, leaders received a training on healthy behaviors, targeted to conflict resolution, gender-based violence, and substance abuse prevention, through JSI and local NGO Ba Futuru. Reinforce provided local leaders a transportation allowance for microplanning meetings outside their villages. 15 | REINFORCE FINAL EVALUATION USAID.GOV Reinforce brought together health facility staff, community leaders, and municipal-level health staff in regular health planning meetings, a collaboration which had to date not been successfully established. The MOH had unsuccessfully attempted to engage Administrasaun Estatál (AE, or State Administration) more formally in the last decade, and to date there is no internal agreement between MOH and the local governance body regarding the role of village councils in health promotion and education. The Village Law states that villages should “identify, plan, and accompany the implementation of activities in the areas of health, education, environment, promotion of employment, food security, and others to be realized for the development of the village,” including a responsibility to “sensitize and mobilize the community members to the importance of the maternal and child health and the participation in vaccination campaigns,”3 which has thus far guided collaboration and expectations between the two entities. KIIs with MHS and Reinforce confirmed that at the subdistrict level, Reinforce promoted collaboration between health facilities and community leaders by involving leadership at all levels, namely the post (subdistrict) administrators and the Municipal Administrator, in quarterly microplanning meetings and twice yearly MHS meetings. PARTNER STRENGTHENING Though outside its mandate, Reinforce coordinated partner meetings with other health￾sector implementing partners in Timor-Leste to ensure collaboration and efficiency of health sector interventions. Reinforce’s activity reports show that at the beginning, Reinforce identified several potential international and local NGO partners who worked in MCH and were providing complementary services. KIIs with multiple stakeholders confirmed that Reinforce worked with partners in different ways depending on their work. For example, Reinforce subcontracted PRADET to train nearly all health facility staff (162 in total) on the “4R” (recognizing, respecting, responding to, and referring victims of domestic violence, sexual abuse, and child assault) and St. John of God for training managers at the referral hospital and selected CHCs. In other instances, Reinforce worked with partners more informally, through verbal commitment between parties such as Health Alliance International (HAI) to introduce Liga Inan (an existing initiative connecting pregnant women to health care providers and midwives through a mobile phone referral network) into Covalima, CARE International Timor-Leste (CITL, already in Covalima at the beginning of Reinforce) to share responsibility for different subdistricts for health promotion activities, UNFPA for training on BEmONC, WHO for technical assistance on immunization, and UNICEF for nutrition-related training. Reinforce coordinated partner meetings intermittently and engaged with partners at technical working group meetings on health promotion and MCH but did not have a formal agreement to guide collaboration. In KIIs, partners saw the collaboration with JSI as very positive and noted that JSI works as the coordinator of multiple organizations, both in and outside technical working groups that meet sporadically. JSI's collaboration with other partners in the MCH space was also reported to be impressive in the absence of a direct coordination mechanism. Several stakeholders within the MOH felt that JSI was more effective in implementation than other IPs because they aligned activities with MOH strategic plans and standards from the activity’s outset. Programming Effectiveness: Institutionalizing Best Practices Without baseline data from Reinforce, the evaluation team could not gauge quantitative changes in capacity for MOH, MHS, and INS. Reinforce did not include a preliminary needs assessment or organizational capacity assessment to gauge existing strengths and weaknesses of the MOH and Covalima MHS capacity prior to the activity. The review of the Reinforce monitoring and evaluation 3 Democratic Republic of Timor-Leste. Law No. 9/2016, July 2016. Law of Sucos. Retrieved from: https://www.ifes.org/sites/default/files/suco_law.2016.english.final_.pdf on Feb. 10, 2020. USAID.GOV REINFORCE FINAL EVALUATION | 16 plan includes capacity strengthening indicators that measure outputs at the facility level, but not for MOH, MHS, or INS. As such, the ET had no clear measurement of institutional capacity improvements for government entities. Though Reinforce could not support strategic MOH workforce development, it did strengthen some MOH departments’ community outreach and health promotion and marketing abilities. Interviews with MOH, MHS, and Reinforce staff noted that Reinforce’s KAP survey resulted in more informed community outreach and enabled the MCH and Health Promotion Departments to develop targeted health promotion messaging and Social and Behavior Change Communication (BCC) materials and harmonize information targeted to men and women about family planning and MCH and nutrition. Reinforce intended to help not only the MCH and Health Promotion Departments within MOH but also the Directorate of Administration and Human Resources for performance monitoring and professional workforce development of MOH staff. MOH was expected to release a human resource for health (HRH) strategy in 2019 to guide this assistance but did not do so. Without a guiding framework from MOH to align with, Reinforce could not provide formal workforce development. National-level respondents could not link the activities of Reinforce to longer term national health objectives and goals. Despite the variety of Reinforce activities, government respondents in KIIs were unclear about the expectations of their specific departments in connection with these activities, how specific activities contributed to personnel development or departmental long-term objectives, and the nature and purpose of a model municipality. There was no main point of contact for Reinforce at the national level, and government respondents were uncertain about the scope of the activity. Facility and INS staff agree that Reinforce successfully institutionalized best practice guidelines and standards around training and supervision within INS. The effects of Reinforce’s capacity strengthening efforts for INS are discussed in more detail under EQ 5. In KIIs and FGDs, the most widely noted achievement for INS was the institutionalization of guidelines and standards, which improved not only trainers’ capacity to follow protocols but also Covalima trainees’ capacity back in their facilities. Reinforce staff accompanied INS and MHS staff to conduct FUAT for health facility staff using a competency-based checklist specific to each training, which provided a clear platform for both INS trainers and Covalima CHC managers to assess trainees and improve mentoring. At the community level, Reinforce strengthened the capacity of facilities to plan for and conduct community outreach. KII and FGD respondents felt that Reinforce helped facility staff conduct more regular community outreach, health education, and BCC messaging. Reinforce installed dedicated Community Health Officers (CHOs) for five subdistricts (decreased to three during the project in order to focus on the most populated subdistricts), who assisted the MHS health promotion officer and the CHCs in developing outreach strategies and reaching targets. Reinforce CHOs conducted follow-up visits to encourage pregnant women or families resisting immunization to visit the health facility and provided financial support for transportation for community outreach and follow-up activities. Remote facilities not receiving Reinforce support nevertheless reported strong confidence in their ability to manage tasks. According to Reinforce activity reports, CHOs did not frequent the more remote Fohorem and Fatumean subdistricts (these two subdistricts were covered by CITL). The evaluation team visited several facilities and spoke with the limited staff in these locations; these remote facilities with limited staff (fewer than 3 persons) and no regular interactions with JSI staff reported “Before we were doing implants, but we weren’t certified. We had the knowledge but weren’t doing it [regularly] because we didn’t want to make a mistake and make the patient or our supervisor angry. But when you are certified it gives you confidence.” - Midwife, Maucatar 17 | REINFORCE FINAL EVALUATION USAID.GOV stronger confidence in managing tasks than their counterparts who received regular FUAT, supportive supervision, and assistance with outreach activities from JSI, namely because they did not have outside entities to rely on, and often managed all aspects of their facility alone. Reinforce helped health providers follow best practice standards, resulting in substantial improvements in clinical skills. All staff respondents at both HP and CHC FGDs reported considerable improvements in their skills around safe delivery, newborn care, family planning, and IMCI, namely in helping them to identify and strictly follow standards (for example, there are 60 steps for safe delivery, and most staff previously were only aware of 6–10 of them). Staff skill improvement and confidence was measured through regular FUAT visits and quality assurance, and there was no difference in training outcomes for male and female trainees. Staff received certificates after successful completion of training and ‘passing’ their follow-up monitoring visit, and those who ‘failed’ received continued monitoring and peer supervision. All staff in KIIs and FGDs at CHCs and HPs agreed that the FUAT practice had implications for improving the quality of services by ensuring they followed strict protocols and best practices with each client and received supervision to ensure they maintained these skills. Reinforce reports for FY 2019 indicated that Reinforce exceeded its targets for a number of providers who were trained and competent on clients and mannequins. Improvements in FP competencies among providers varied based on type of method. Review of FUAT data provided by Reinforce for doctors, midwives, and nurses between 2016 and 2019 showed mixed results, namely disparities between competency on mannequins and actual patients, especially for family planning practices like implants and IUDS (See Figure 1). Due to a low patient demand for IUDs or a lack of equipment to provide implants, most midwives and doctors were unable to improve their competency on real clients. However, most of these health workers managed to practice and gain competence on implant insertions—the most-requested long-acting reversible method—on both mannequins and real clients. FUAT data received did not disaggregate between male and female trainees. Figure 1: Health worker competency from the beginning of Reinforce in 2016 to December 2019 While levels of satisfaction in care received and service quality were generally high, because there was no baseline assessment or ongoing collection of patient feedback, the evaluation team could not understand how satisfaction levels changed over time. In addition to clinical skill improvement, select staff received training in interpersonal skills (IPC) such as respectful care and patient communication. The evaluation team could not follow up directly with clients receiving care from those providers who were trained in IPC but included general questions about respectful care in the survey. In both HPs and CHCs, over 73 percent of survey respondents (n=31) on average stated that they always received attention (24/31), adequate explanation of treatment options (23/31), and were comfortable asking questions (21/31); over 90 percent (28/31) felt that they were always treated with respect. 73% 27% 100% 67% 100% 0% 67% 29% 100% 87% 94% 17% ENBC (mannequin) ENBC (real client) FP- Implant (mannequin) FP- Implant (real client) FP- IUD (mannequin) FP- IUD (real client) Doctors Midwives USAID.GOV REINFORCE FINAL EVALUATION | 18 However, collection and review of client feedback is rare, and this is a missed opportunity to identify improvements to care and service quality over time. The introduction of microplanning meetings by Reinforce increased communication between MHS and local governance structures and improved overall data management skills for MHS and CHCs. All facility staff and MHS KII respondents alike saw microplanning as inherently useful. The introduction of DHIS-2 and guidance from Reinforce helped MHS see trends of other contraceptive use during a stock out of the injectable contraceptive Depo-Provera. Additionally, the microplanning meetings themselves allowed MHS to interact with local governance structures, receive regular data to cross check from facilities, and review action plans to address community health needs; none of this had been systematized previously. Staff at both MHS and CHCs noted that their data management skills had improved through guidance from Reinforce staff, and the activity’s provision of standardized tally sheets, which reduced errors in data entry and allowed staff to more accurately calculate patient data to present coverage data in microplanning meetings. DHIS-2 is not yet widely used in all facilities because of a lack of electricity and network connection, especially at HPs, and both HPs and CHCs visited stated a preference for continuing to use manual forms. Review and use of data at the facility level is highly inconsistent and superficial, and most health providers still lack confidence in analyzing or presenting data. Some HPs do not analyze their data but instead only tally and collect data to share with the CHC, who has a designated staff person for data entry and analysis (see EQ 2). In the HPs where data was only collected to be sent to the CHC, HP staff in FGDs were indifferent to the analysis process. For others, a nurse or doctor on staff took an unprompted, personal interest in analyzing data. Independent planning from data collected—without assistance from Reinforce—is inconsistent across facilities. While several staff in FGDs felt more confident than in the past to collect this data without assistance, few admitted feeling comfortable analyzing or presenting data. Some of the facilities who have done their own analysis have presented their facility data at microplanning meetings, but the majority of staff interviewed stated that up to present day, data at quarterly microplanning meetings is presented by Reinforce staff or sent directly to MHS to present on behalf of all facilities. KIIs and FGDs with facility staff noted that outside the microplanning meetings, staff sometimes review data weekly or monthly, but this is dependent on the facility and happens inconsistently; many facilities only review data and discuss internally in preparation for the microplanning meetings. Analysis of the data to be presented at microplanning meetings was limited to MNCH, FP, and immunization coverage only. All facility staff respondents only cited reviewing and analyzing coverage numbers alone and not performing any meaningful analysis of the figures outside of seeing “where numbers are high or low” and targeting outreach more to those areas. "Before JSI we just had day to day planning, but now we can create and aim for targets” -Nurse, Zumalai Picture: Bulletin board at Covalima MHS with outdated data 19 | REINFORCE FINAL EVALUATION USAID.GOV Most CHCs have started leading parts of the data collection and use process, though no CHC can currently lead all parts of the process, and all CHCs still depend on Reinforce for funding, transport, and printing. Reinforce and CHC KIIs confirm an expressed interest and commitment to using data to organize and plan; however, none of the CHCs are conducting the entire process themselves to date. At the municipal level, Reinforce keeps a quarterly process progression tracker to observe the confidence and ability of facilities to independently manage and adopt key data collection and interpretation processes after Reinforce has concluded. This tracker includes an assessment of each CHC’s ability to record data, update databases, coordinate meetings and logistics, complete meetings, and both record and follow up on action plans, and also looks at the status of equipment provided, such as printers or projectors. The Reinforce tracker currently indicates that all CHCs still rely on Reinforce for funding, transport, and printing, though Tilomar CHC has made the most solid progress in leading all steps in the process without relying on additional financial support or access to printers. Out of the seven CHCs, at least five are independently preparing for and holding meetings to discuss coordination and logistics, and three CHCs independently record health promotion data. New tools for reviewing data exist and are being used by staff at the MOH level; however, a lack of central decision-making authority on data-related issues threatens the future usability of data. At the national level, tools for conducting RDQAs were formalized with the support of UNICEF and Reinforce, and 13 staff from a variety of directorates at MOH were trained on using these tools. MOH KIIs confirm that they now use these tools to review data from various programs within MOH, such as nutrition and immunization, and plan to disseminate the tool widely and enforce use throughout the MOH. However, it was noted in KIIs with the M&E Department that there is no unit within the MOH that solely functions to review and analyze data, and so the capacity to use data for course corrections and interventions when needed is still inconsistent. There are explicit concerns over the validity and reliability of past data, including from the DHS; stakeholders, including MOH, note in KIIs that they frequently reference 2009–10 data rather than more recent 2016 data because of these concerns. INSTITUTIONALIZING BEST PRACTICES IN HEALTH EDUCATION AND BEHAVIOR CHANGE Most respondents agree that Reinforce contributed to an increase in health-seeking behaviors in communities, particularly in immunization and mothers delivering at facilities. While it was not possible to identify a direct causal link between community outreach and improved health behaviors, stakeholders, including MHS, facility staff and clients, in KIIs, FGDs, and surveys believed that improved health education activities affected behavior. Community leaders and the majority of CHC/HP staff agreed that Reinforce’s CHO assistance providing community information on health education and information around FP, safe delivery, facility delivery, and IMCI has brought more people to the facility and improved numbers of mothers delivering in facilities and children receiving immunizations. Of the 31 survey respondents, the most popular reasons for visiting were related to maternal and child health: 29 percent (n=9) came for a newborn or child health checkup, while 23 percent (n=7) were visiting for antenatal care or nutrition assistance, and 16 percent (n=5) were there for FP (counseling or provision of methods). Though Reinforce sought to engage men in family planning, none of the respondents seeking FP were men, and none of the women were accompanied by men. Reinforce’s financial support for conducting community outreach activities and engaging local leaders, along with (in collaboration with DFAT partners) introducing the mobile phone referral system for expectant mothers, Liga Inan, were seen by the majority of both CHC and HP staff as having positive impact on health-seeking behaviors and health education. Indeed, 45 percent of survey respondents (n=14) stated that they were familiar with the services offered “We have the data here, but we don’t analyze it weekly or monthly, only every three months when JSI is arranging the microplanning meeting.” - CHC management USAID.GOV REINFORCE FINAL EVALUATION | 20 at the facility because of information they received through a recent (within the last two years) community outreach activity. There was a lack of consensus from MOH, MHS, and facility staff regarding changes to adolescent health-seeking behaviors, especially for ARH. MHS and facility staff were aware that youth received health education in school but few knew about the school health packages promoted by Reinforce and the Health Promotion department at the MOH. Health staff noted in FGDs that there was a lack of demand for ARH services at health facilities and that ‘shy’ youth prefer not to visit facilities for such services. The evaluation team did not observe any visible health communication materials targeted to adolescents in the facilities visited, and staff reported that adolescents were not specifically targeted during outreach activities in the community. The evaluation team did not directly speak to adolescents. Participation of local leaders contributed to sustaining health outreach and education activities in communities; however, continued participation of leaders in microplanning depends on funding. Document review and KIIs confirm that Chefes de sucos (village leaders) are not expected or responsible for providing community health education and instead only need to mobilize communities to come together for SISCas or outreach activities, but KII respondents at the municipal and national levels note that these leaders played an integral role in providing health education at the SISCas. Additionally, leaders interviewed felt that their participation in microplanning improved their communication with facilities and the MHS and ability to link community health outcomes with behaviors and treatment options. In Reinforce reports, those leaders who attended the Ba Futuru training felt that they improved their ability to resolve conflicts in their community and reflect on unhealthy behaviors within their home and community. While the village law states the expectations around health promotion activities, efforts depend on personal motivation, as there is no accountability mechanism targeting village chiefs. Some leaders are uninterested or reluctant to be fully engaged if they know that the work will be without pay or direct benefit. Without continued funds, CHC and HP staff feel that these leaders will not continue to support the activities or attend the microplanning meetings. An illustrative matrix (Figure 2) identifying and summarizing cross-cutting capacity improvements, limitations, and challenges, according to key stakeholders and areas influencing healthcare service delivery, is below. Red cells are where capacity is weak/not promising, yellow cells indicate where capacity is limited, and green cells indicate generally good or encouraging levels of capacity. Information included in the matrix is generated from the evaluation findings from KIIs, FGDs, observations and surveys, complemented by desk review sources. 21 | REINFORCE FINAL EVALUATION USAID.GOV Figure 2: Capacity Assessment Matrix Key: Weak Capacity Limited Capacity Good Capacity MOH INS MHS CHC/HP Collecting data DQAs No M&E budget Microplanning Microplanning Manage/analyze data No central directorate Relies on partners Inconsistent analysis Inconsistent analysis Uses data for decision￾making Unclear use of data No planning framework; limited autonomy Limited autonomy Limited autonomy Follow up training and support N/A Comprehensive improved FUAT system No formalized municipal training mechanism Few training opportunities Technical skills (mgmt., clinical) Weak overall mgmt. No accreditation; few opportunities Microplanning improved mgmt. Improved outreach; few opportunities Budgeting Delays; no data use Competent; some risks Weak planning Weak planning Facility infras. & equip. Has supplies and data Plays no role Disconnect on local needs HPs largely deficient Placement of staff Uninformed distribution Plays no role Plays no role Plays no role Human resources M&E staff shortage Shortages N/A Not aligned with need Coordination Siloed Coordinates with partners; not MOH Microplanning Microplanning Knowledge mgmt. No sharing; high turnover High turnover; no partner sharing No sharing No sharing USAID.GOV REINFORCE FINAL EVALUATION | 22 CONCLUSIONS • Reinforce worked with stakeholders at all levels of the health system in many different forms with mixed results on institutionalization of best practices. Reinforce worked with partners in various ways, but no activities included MOH, INS, and partners together at the same time outside of the launch. Reinforce’s activities ranged from conducting assessments, supporting staff to attend trainings, and updating best practice guidelines, to helping create clinical practice sites, regularly convening stakeholders at multiple levels, and more. Institutionalization of skills and competencies is mixed. • Institutionalization of best healthcare practices was most strongly seen at the community level and to some degree at INS. Reinforce’s technical capacity building assistance for the development of guidelines, standard operational procedures (SOPs), and training manuals reflect greatly improved health facility and INS staff abilities to adhere to standards and deliver quality services. Competencies of facility-based health providers in critical clinical areas improved over time after Reinforce activities. Microplanning now allows for health facilities and MHS staff to better plan for and carry out community outreach activities. • Definitive capacity improvements at the MOH, MHS, and INS levels are not yet clear. Capacity improved for clinical skills for facility staff, and microplanning benefitted facility and MHS staff and local leaders, but there are no quantitative measurements of improvement for MOH, MHS or INS. The consistent buy-in of stakeholders and the shared responsibility and ownership of the ‘Model Municipality’ strategy is unclear. • Non-use of data in planning at all levels still remains a critical issue for institutionalization of best health services. Review and use of data at the facility level is highly inconsistent and superficial (mostly limited to viewing coverage rates and planning for microplanning meetings), and most facility-level staff still lack confidence in analyzing or presenting data. At the MOH, while many different directorates confirm increased use of data tools, the lack of a central decision-making directorate limits the operationalization of data. • Reinforce played an important role in increasing health-seeking behaviors at the community level, but systemic challenges remain. Most respondents agreed that Reinforce activities contributed to increased uptake of services by communities (though these findings were unclear in the area of ARH), facilitated in part by the involvement of local leaders in Reinforce activities. The uptake of Reinforce’s assistance has been limited by systemic challenges, especially addressing the staffing and supply shortages at facilities. Stakeholders recognize and rely on Reinforce's financial and technical support for continued capacity strengthening activities. 23 | REINFORCE FINAL EVALUATION USAID.GOV EQ 2: BARRIERS TO INSTITUTIONALIZATION What bottlenecks, if any, in the capacity and commitment of stakeholders (MOH, MHS, health facilities, and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? SYSTEM LEVEL Political turmoil and uncertainty around the budget line item for health expenditures may affect health-sector outcomes in the near future. The Reinforce Activity was implemented during a time of political impasse that began in 2017 involving numerous political parties, the parliament, and the office of the presidency. This impasse greatly affected the functioning of the health system; in the span of less than one year, there were two general elections, the second only 10 months after the first. In January 2020, the most recent national budget, which favored infrastructure and significantly reduced expenditures for health, education, and sanitation, was not passed.4, 5 The full significance of these political decisions is outside the scope of this evaluation, but the context in which the activity was operating and the respective considerations for the impact on short- and long-term progress in government bureaucracy, capacity, and commitment at the time of the evaluation is notable. Administrative decentralization is long thought to reduce bottlenecks in bureaucracy and is embedded in the Constitution of Timor-Leste. Section 5 states that in order to make decentralization possible, “the law shall determine . . . the administrative competencies of the respective organs,” and Section 57 (Health) states that “the national health service shall have, as much as possible, a decentralized participatory management.”6 As part of this structure, managers within each district are to be appointed by the government at the central level and have budgetary and direct authority over operations, including staff salaries, within their district.7 However, there are concerns that devolution to municipal authority “may not fit well the needs in equity, service continuity, or staff mobility across the country.”8 Systemic weaknesses in the health system identified in 2016 persist today and threaten the realization of health improvements and progress of Reinforce. An assessment of the National Health Sector Strategic Plan (NHSSP) in 2016 highlighted that pervasive constraints in HRH, procurement, and financing were the major challenges to realizing solid health improvements.9 Feedback from stakeholders, including facility management, international partners, and government entities themselves, during FGDs and KIIs indicated a consensus that there was no systemic change in the last four years. Bottlenecks in the capacity of GoTL to institutionalize best health service practices fall broadly under four categories: Budgeting, Decision-making, Coordination and Communication, and Motivation. 4 Kapoor, K. East Timor votes to break political impasse in parliament. Reuters. May 11, 2018. Retrieved from https://www.reuters.com/article/us-timor-election/east-timor-votes-to-break-political-impasse-in-parliament-idUSKBN1ID03Q. 5 Leach, M. In Timor-Leste,failed budget sparks a political crisis. The Lowy Institute. Jan. 20, 2020. Retrieved from https://www.lowyinstitute.org/the-interpreter/timor-leste-failed-budget-sparks-political-crisis. 6 The Constitution of the Democratic Republic of Timor-Leste. Retrieved from http://timor-leste.gov.tl/wp￾content/uploads/2010/03/Constitution_RDTL_ENG.pdf. 7 Health Financing Strategy 2019–2023. Democratic Republic of Timor-Leste. October 2019. 8 Ibid. 9 Feigl, Andrea, Chris Lovelace, Nikita Ramchandani. Feb. 2016. A Rapid Assessment of Key Areas of the NHSSP for Timor￾Leste: Strengths, Challenges, and Opportunities for Moving Forward. March 2016. Bethesda, MD: Health Finance and Governance Project, Abt Associates Inc. USAID.GOV REINFORCE FINAL EVALUATION | 24 BUDGETING Uninformed national level budget forecasting and a lack of transparency regarding budget amounts and timing for municipalities impedes municipal-level programming. The GoTL’s political impasse resulted in no new national budget. The state budget, which includes allocations for each department, is to be approved by Parliament. Departments like MOH submit reports with programmatic and budgetary needs; however, these are often not aligned. As one senior MOH representative stated: “We go to Parliament and state what we need but it’s not based on real data . . . we don’t know how much money is needed to reduce maternal mortality.” The Ministry of Finance operates a publicly available Transparency Portal through the FreeBalance software and web platform that identifies government expenditure by department and includes an eProcurement portal and aid transparency portal. However, the data and figures included are often not updated, and information by line item is not available. Several stakeholders at the district level did not know how budget allocations were made and expressed frustration at the lack of transparency; they were also unaware of this publicly available database. Extreme delays in approval of budgets submitted at the municipal level leads to an overreliance of municipalities on partners and uncertainty in programming that trickles down to the community level. At the municipal level, the MHS submits budgets annually with line items for activities like community outreach, maintenance, and operations. However, municipal-level respondents report extreme delays in budget approvals; in Covalima, the budget for the final two quarters from the previous year had been approved nearly one year late. When budget approvals are delayed, MHS relies more heavily on partners, reduces planned activities, or combines activities such as follow-up supervision and outreach events to save costs. MHS officials are not informed about what causes delays in budget approvals, and as a result cannot manage the expectations of CHCs and HPs. CHC managers felt limited in managing their facilities and responding to HP needs because of a lack of budget transparency. At both national and district levels, international partners are relied on to budget and respond to immediate needs on behalf of the government. As one Senior MOH representative stated, “we don’t have costing; we only have planning annually, so we have to ask partners to fill in the gaps.” At both national and municipal levels, advance planning, including aligning budgets with future activities and forecasting for maintenance, is absent. A World Bank analysis of Timor￾Leste’s health system in 2012 identified that “variability in budget execution rates across line items and across districts provides strong evidence of the mismatch between activities/needs, on the one hand, and budget allocation, on the other hand.”10 Facility and MHS staff lauded the microplanning process; however, the action plans do not include an associated budget, preventing efficient planning, forecasting, and realization of activities. KIIs with partners noted the importance—but lack of—careful foresight and that “a culture that looks at what will happen three months from now versus what will happen tomorrow is important.” The allocation from Parliament to MOH can vary, and once granted, is not guaranteed to cover actual activity costs; however, the MOH’s inability to accurately forecast district-level needs based on documented evidence does not bode well for advocating to Parliament. 10 World Bank (2014). Timor-Leste Health Resource Tracking Study. Main Findings, Recommendations and Cross-Cutting Issues. “Sometimes when there are challenges to implementing the micro plan, that is because of budget or HR . . . the plans don’t take this into account, which is a mistake—we don’t plan appropriately or anticipate everything, and it is a learning exercise. Sometimes we only reach 10–15% of the plan” - Doctor, Zumalai 25 | REINFORCE FINAL EVALUATION USAID.GOV DECISION-MAKING MHS identifies staffing and equipment needs, but its inability to procure these items, combined with uncertainty and delays at the national level, affects programming. As part of the decentralization process, MHS-level authorities are given decision-making authority over operational costs like cleaning, catering, and small repairs. However, in practice, municipality and subdistrict level authorities still feel constrained by the lack of autonomy they can exercise for decision-making of more substantial issues like staffing and equipment procurement. Covalima MHS and CHC staff identify staffing, vehicle, maintenance, and equipment needs every quarter, but cannot independently procure these items, as these decisions must be made at the national level. Management staff from CHCs and MHS expressed frustration with bureaucracy and extreme delays in responsiveness. There is limited transparency about why financial or programmatic decisions are made at the national level and what the expected timelines are for response or action. As one international partner noted, “people need confidence so that they can make independent decisions that are respected, but this is not there.” There is a disconnect between MOH and lower levels on the status of decentralization, and this contributes to facility-level needs not being met by the MOH. None of the health facility staff interviewed knew how data was used for decision-making at the national level, and all expressed frustrations that despite clear data being collected from their facilities identifying gaps in human resources and equipment and sent to the national level (via the SARA assessment, facility readiness monitoring, supportive supervision visits, and quarterly review meetings), their requests for assistance have gone unanswered. When they inquired, they were only told that there were budget constraints. There is mixed feedback on the decentralization process, with most facility and MHS staff noting that the process is starting but not yet completed, and the national MOH staff confident that the process is finalized. MHS respondents feel beholden to the national level for any and all decision￾making, and facility staff have low expectations, believing MHS is powerless to aid them. A lack of operationalized national-level guidelines leads to inconsistent service delivery at the facility level. Decisions can largely be based around institutionalized policy, but a number of guiding frameworks have not yet been passed at the central level, including the Family Planning Policy and the Basic Package of Health Services (BPHS), which have been drafted but not operationalized, leaving facility decisions open to interpretation. International and local NGO partners remarked that the overarching influence of the Catholic Church in Timor-Leste and health provider’s personal and religious bias against unmarried women and youth seeking services has resulted in an unmet need for FP and an unwillingness of adolescents to seek out reproductive health (ARH) services. While facility staff interviewed stated that they would gladly counsel adolescents about FP, they admitted that they did not have specific guidance on how to do so and would welcome specific training both within their facility and for health education and outreach. COORDINATION, COMMUNICATION, AND KNOWLEDGE MANAGEMENT MOH departments generally do not coordinate or engage in knowledge management, which delays decision-making. Reinforce operated across several MOH departments, including Public Health, Policy Planning and Coordination, Health Promotion, MCH, Logistics, and M&E, as well as with MHS and INS. In addition to the challenge of accessing and utilizing existing data, stakeholders noted that poor knowledge management and information sharing due to the lack of coordination between responsible government parties prevents efficiency in decision-making. “The health system, short of coordination, will not contribute to positive outcomes” - International Partner “The use of the data for policy decisions is almost nonexistent–it is presented in a way that it can be used, but it’s not . . . there is no action” - International Partner USAID.GOV REINFORCE FINAL EVALUATION | 26 Training and placement of facility staff (MOE, MOH, INS, and MHS) Health workers are distributed inconsistently—leading to under and overstaffing of health facilities—due to non-use of data and lack of collective input from relevant institutions such as MHS and INS. The 2016 NHSSP Assessment highlighted the need for integration and collaboration between MOH, Ministry of Education (MOE), and INS to streamline the process of health worker graduation, training, and distribution. Several entities are involved in the training and placement of the health workforce. Future health workers study under the Ministry of Education, new graduates are identified and recruited by Human Resources within the MOH, hired by MOH, placed in facilities by MHS, and then trained by INS after arriving at their field placements. While MOH standards denote the minimum number of specific staff to be present in each HP (seven) and CHC (up to 30), health workers are distributed inconsistently and not based on their competency, patient load, or trends in service utilization, and without the collective review and input from INS and MHS. CHC managers cannot allocate staff based on need even though they have direct facility data to back up requests; for instance, there are no midwives at eight of the 21 HPs, even though microplanning data shows that there is a need for midwives in these locations based on fertility trends. As a result, some facilities are overstaffed, many are significantly understaffed, and skill availability does not match need. Selection of healthcare professionals for training is ambiguous and not aligned with needs at the facility level. Personnel training decisions are in response to national quotas for numbers of health staff to be trained and the outstanding number of staff who have not been trained in a given facility, not based on individual skill development needs. CHC managers only receive a list of staff who are to be sent for training (the list originates with MOH and is sent to MHS for distribution) and cannot independently send staff for training based on identified service, skill gaps, or employee preference. Communication about training selection is ambiguous—none of the health providers interviewed knew how or why they had been selected for training, if they would be selected to attend another training, or if they could request to attend a training outside of their profession. Lastly, after training staff and following up on their competency, INS is not included in decisions on staff distribution, despite requests for inclusion based on their insight gathered from post-training competency results and ToT recommendations. Building Facilities: MOH and the National Program for Village Development (PNDS) Building of HPs is not aligned with the supply of trained staff or available supplies and equipment. PNDS is tasked with building health facilities in line with government standards and based on population density and accessibility: HPs serve an area of fewer than 2,000 persons in rural areas and 5,000 in urban areas and can be one hour or more from the nearest CHC.11 PNDS builds the HPs with targeted donor funding but does not coordinate with MOH to ensure that the new facility will be staffed with skilled personnel or that it will be equipped with needed supplies. CHC managers noted that several HPs were under construction, but they were uncertain how these facilities would be staffed in an already understaffed environment. Equipping Facilities: MOH, Logistics, and M&E Departments Lack of equipment at the facility level is well-documented at CHCs, MHS, and MOH, yet consistently not addressed. The SARA in 2016 was meant to be used by the MHS and the M&E Department as an advocacy tool to request necessary improvements in infrastructure, equipment, and supplies. The M&E Department provided this information to the MOH Equipment Department to be 11 Government of Timor-Leste, Ministry of Health. Comprehensive Services Package for Primary Health Care (2015). 27 | REINFORCE FINAL EVALUATION USAID.GOV included in the next procurement; however, despite this documentation, procurement by the MOH did not take place. Health providers at every health facility visited stated that they receive quarterly facility readiness checks from the M&E department, and the lack of equipment is well documented to the CHC management, MHS, and at the national level, yet none of the gaps identified were addressed by any entity. Reinforce procured equipment for facilities at the end of the project; relying on partners to do so in the future is unsustainable. Reinforce cannot support areas of need such as emergency transport and maintenance and repair of infrastructure, but in 2018, Reinforce opted to procure the required equipment themselves to ensure that the facilities in Covalima would perform as anticipated before the end of the project. WHO, UNICEF, United Nations Population Fund (UNFPA), and the Korean International Cooperation Agency (KOICA) took similar actions in other municipalities. With authorization from MOH and USAID, JSI revisited the MOH equipment requirements and procured the equipment needed based on the facility type under the Package of Health Services; each HP and CHC would receive the same package regardless of the needs specified in the SARA or most recent facility readiness data. At the time of the evaluation, the equipment was received and awaiting branding and distribution. However, only using the minimum standards and not considering facility-specific needs is detrimental. There are only three CHCs in Covalima with beds, and beds are not distributed based on those CHCs in areas of highest fertility, which are most likely to need to perform safe deliveries. Another SARA will be completed later in 2020, and so these results will likely provide the most up-to-date quantification of need and identification of improvements; unfortunately, this will be completed after the evaluation process. Equipment and supply needs are uninformed by current data due to lack of information sharing with the National Logistics Department. The National Logistics Department is responsible for countrywide supply procurement and distribution decisions for non-medical supplies (not only for MOH), and the MOH Equipment Department is responsible for monitoring and procurement of medical supplies and equipment. When the SARA was initiated, the results were presented to MOH representatives, several of which were engaged in the planning and design of the tool. However, current information about specific medical and non-medical equipment fulfillment needs captured by the SARA and by routine facility readiness monitoring by the M&E department are not shared with the National Logistics Department, and its senior management was unaware of the SARA results when asked. The Logistics department uses a manual Excel sheet to capture logistics and equipment procurement and inventory. While the Logistics Department needs information from the field to guide decision-making, physical inspection of equipment does not regularly occur due to a limit of eight days of travel for the Logistics Officer per month in the entire country. The facility readiness checks by M&E team are done quarterly and capture the data on medical and non-medical supplies the Logistics and Equipment departments need, but this data is not shared with Logistics director, preventing evidence-based distribution. There is a disconnect between MHS and CHCs/HPs on the realities of supply and equipment needs at the facility level. At the municipal level in Covalima, there is no regular tracking of equipment distribution or maintenance through an inventory. The Logistics Department does not visit field sites without approval from the MHS Director, creating a disconnect between MHS on the realities of supply needs at CHCs and HPs, where staff believe “they have the equipment, they just don’t know how to use it.” While the MOH guidelines on primary health facilities state that “it is necessary to have solar energy to provide some electricity and to run the freezer,” "The intent among partners and MOH is to have one plan, one budget, one M&E framework, but we are not there yet. " - International Partner “If you create a need through outreach and then the health post isn’t well attended or equipped with medication or FP, then it is damaging” - International Partner/Donor USAID.GOV REINFORCE FINAL EVALUATION | 28 at the time of the evaluation, none of the HPs visited in Covalima had electricity, and those that were provided solar panels noted that they have been in disrepair for over a year. None of the HPs visited had running water. In the client survey, 71 percent (n=22) of respondents felt that supplies and equipment were sometimes or never available, and the highest number of client survey respondents (48 percent, n=13) stated that infrastructure improvements were needed at their facility, including water, space, and appropriate, functional equipment. Transportation-related costs for community outreach and ambulatory services is inconsistently budgeted for, and transport services are oftentimes unavailable. MHS informants and facility staff alike informed the ET that there is one working ambulance in Covalima and three in disrepair in the MHS parking lot. The distribution of vehicles depends on availability, and 14 multi-function vehicles were purchased by MOH in 2019 to be used as ambulances, one for each municipality across the country. General upkeep of ambulances is poor due to bad road conditions and lack of training for drivers on car maintenance and road safety. More than one quarter of client survey respondents (9/31) noted that road conditions often kept them from accessing facilities, especially during the rainy season, and that ambulatory service, while necessary, was absent. Generators were provided by JSI under Reinforce, and CHC managers receive $7,000 per year from MHS for fuel for generators and for government-owned motorbikes at the CHC and HPs; however, most of the generators are in disrepair, and many facility staff report using their own motorbikes owing to the rarity of receiving one from the government. Staff are to complete a logbook of travel so that CHC management can track the number of community outreach activities, including SISCa and immunization campaigns (‘sweeping’ exercises) scheduled and traveled to and their distance from each facility, but the logbooks are filled out inconsistently, so management does not have the information needed to accurately determine the funds required for fuel or hold staff accountable for fuel received. Learning opportunities While several MOH departments engaged with Reinforce, none were able to describe the role and activities of other departments, or even their own role in Reinforce. At the national level, international partners and the MOH attend technical working group meetings, but partners note that both the working groups and the partner meetings are “not for MOH decision-making,” as the meetings occur infrequently and MOH representatives attend infrequently; most knowledge sharing comes from informal meetings between partners after office hours. As noted in EQ 1, there were many opportunities and meetings between stakeholders to review data, share activities, and discuss and address challenges collectively, but none of the respondents stated learning anything new, having collaborative discussions to provide feedback at these meetings, or seeing any changes resulting from learning. In Covalima, microplanning meetings are an opportunity to have community leaders, MHS, and health workers together, but do not appear to generate or document best practices to be shared among attendees; the meeting is used to present coverage data only to inform action planning, but these plans are inconsistently monitored at the CHC level. None of the staff at the MOH or MHS level had collected documented evidence of the successes, lessons learned, or failures associated with their involvement with Reinforce in Picture: One of the ambulances in disrepair at the MHS offices in Suai, Covalima 29 | REINFORCE FINAL EVALUATION USAID.GOV Covalima that could be shared with JSI, with other departmental staff, or with the evaluation team, and one stakeholder in M&E noted that the MOH “should be focusing more on feedback in the future.” Lastly, healthcare research and reports are not widely publicized or shared among MOH staff or to the wider population. At the management level within MOH and INS, frequent changes in political leadership require frequent senior management turnover, which leads to loss of institutional knowledge. At the time of the evaluation, several senior managers were ‘sitting exams’ to determine if they could keep their position. Staff note that turnover affects knowledge management within departments and from one department to another, as many leaders are the sole keepers of critical pieces of information, and documents and policies are not shared with their subordinates. They often “take their knowledge with them” when they leave, resulting in duplication of efforts and inefficiency. Further, government staff do not have a uniform, digitized system for knowledge sharing, and staff use personal email accounts instead of a professional network. DEMOTIVATION OF STAFF At both the facility and government level, staff demotivation and discontent had a negative impact on service delivery. Human resources and expectations Healthcare staff at remote posts are overworked, and staff distribution does not align with need. At the facility level, the number of health staff has greatly increased in the last decade, with a push to train personnel in Cuba. However, there is still a shortage of skilled staff, especially in more rural areas. As noted above, HPs are required to have seven staff, while CHCs should have up to 30 staff, irrespective of population size or medical need. While these requirements are written in the MOH guidelines, they do not accurately represent the staff at each facility in Covalima. Covalima facilities average 2.47 staff at the HPs and 10.14 staff at each CHC. Four of the 7 CHCs operate with fewer than 10 staff, and eight of the 21 HPs operate with only one staff member. In the patient survey, 74 percent (n=23) of respondents felt that the facility sometimes did not have enough staff to manage the patient load, and several noted that the facility staff often served in multiple roles, including cleaner. Every CHC manager interviewed stated that their requests for more staff went unanswered. Staff serving as the lone health workers in remote areas are overworked, unable to leave post for training to better deliver services, and unprepared for leadership roles. As one partner noted, “there are midwife-led facilities, and they haven’t been trained in management.” Facility staff are often forced into positions of management that they are not prepared or compensated for because of a lack of human resources. A policy introduced in 2012 to provide higher pay to remote staff in rural areas has not been operationalized and is thus not reflected in the MHS budget. Most facility-level staff believe it will not be possible to continue community outreach efforts after Reinforce ends because of the loss of extra human resources that Reinforce provided. While Reinforce emphasized community health promotion and education and enabled medical officers/doctors, nurses, and midwives to conduct outreach activities by providing them with transport assistance (and another pair of hands from Reinforce staff), continuing to do outreach tasks in the absence of Reinforce seemed daunting to many staff. While job descriptions for all three positions require staff to “mobilize and organize community to access health education and general community outreach” and “conduct SISCa, community home visits, and mobile clinics,” facility staff expressed disbelief that they could continue to engage in these activities without continued support from Reinforce or without additional staff assigned to their facility. Transportation constraints prevent health workers from doing regular follow up with mothers after delivery, though they are required to do so, and they relied heavily USAID.GOV REINFORCE FINAL EVALUATION | 30 on Reinforce to provide this support; without follow up, they must rely on mothers to come on their own will, which is unlikely. Health facilities lack a performance management system that recognizes and awards staff or identifies career growth opportunities. During Reinforce, staff were encouraged to better track and use data. While some facility staff have improved their capacity to do data analysis, others are constrained by the number of health staff available to do analysis on top of their other work; six of the seven CHCs in Covalima have a designated HMIS staff person, but there is no dedicated data management staff at HPs. As one stakeholder noted, “M&E just gets handed to someone sometimes . . . sometimes it is just staff who have the time.” Thus, tasks like database management are handed off to staff who have not received comprehensive training and are expected to complete this task in addition to their regular work. For those who are taking on extra responsibilities, there is no system of recognition, no additional remuneration, and no human resource system that identifies a career path for growth. INS-trained trainers were unmotivated to continue working as both health staff and trainers in Covalima because of the burden of two jobs without adequate pay and financial security while traveling. Additionally, INS lacks enough trainers to address the wide-ranging training needs, creating a bottleneck in ensuring training and FUAT is consistent. (See EQ 3 and 5 findings for more information about INS trainer capacity and sustainability.) Senior management turnover, noted in the previous section, is also a source of demotivation for staff at MOH and INS. Managers are uncertain how long they will be in their position, and staff must frequently adjust to new leadership that may change the course of their policies, activities, or expectations. Capacity development challenges As one partner noted, “capacity is not relevant unless it’s used.” There are several bottlenecks preventing facility staff capacity improvements. While staff have been trained in a number of areas including FP, Emergency Obstetric Care (EmOC), maternal and newborn care, and safe delivery, they lack the patient load to effectively practice these new skills in real life. Staff at both CHCs and HPs feel they must travel to the hospital or Dili to use new skills with actual patients, and they lack confidence in their skills, especially in FP or safe newborn care. For example, the MOH requirement for competency includes demonstrated capacity to provide Intrauterine Devices (IUDs) to a patient and, given the low rate of use of IUDs, most providers have not had an opportunity to demonstrate that competency. Often staff who are well trained do not have the equipment or infrastructure needed to provide key services. This includes physical space, electricity, water, or supplies; staff at the majority of health posts visited in Covalima noted that while they were competent and trained in FP and safe delivery, they could not provide FP implants or deliver babies due to lack of space, water, or beds, all of which are included as expectations under the MOH primary health care guidelines.12 Staff interviewed stated that 12 Ibid. "I want to go to the practice site and meet other midwives and then we can learn together . . . but I am the only one here, and I can't leave my post" - Midwife, Zumalai 31 | REINFORCE FINAL EVALUATION USAID.GOV they had more training interests and needs, either on additional topics or at a more advanced level; for example, doctors are not selected to receive training in FP but were interested in this. Training selection is not informed by past trainings, resulting in many staff receiving the same trainings over and over. Staff at facilities with three or fewer staff believe they should receive training in multiple areas because they have multiple job responsibilities, but they are not guaranteed to receive any additional training, as staff are only selected for training to fill the quota requested by INS. If staff are selected for follow-up training, it is the same training they already received—trainings provided are not leveled to commend skilled staff with more advanced knowledge or responsibility. Facility staff confidence and ability to improve skills is affected, especially for those working in a setting where they are relied upon as the only health worker. HEALTH SYSTEM ACTORS AND PROCESSES BOTTLENECK ANALYSIS An analysis examining the bottlenecks described above for financing, human resources, information, and communication and procurement of goods is illustrated as a map (Figure 3) showing interconnected actors and responsibilities. A legend for each bottleneck (a-r) is presented on the page following the graphic. USAID.GOV REINFORCE FINAL EVALUATION | 32 Figure 3: Bottleneck Analysis CHC Parliament MOH + PNDS HP Log’ M&E i Partner l MOH TWG Suco MHS MHS INS Community + Micro￾planning + a b d l f k g j m r p q n e h i o c + Bottleneck Information; communication Data Knowledge management Goods; supplies Budget Staff; training Stakeholder Convening Process (major bottleneck) Process (minor bottleneck) Monitoring Legend 33 | REINFORCE FINAL EVALUATION USAID.GOV MOH, MOE, and Logistics do not share data, communicate or collaborate, and knowledge management is weak, hindering accurate allocation and forecasting of resources There is very little transparency to MHS from MOH about why decisions are made, so MHS cannot learn for the future. Budget allocation is extremely delayed from MOH to MHS; MHS budget forecasts are inaccurate owing to lack of their own data from CHCs and HPs MHS lacks autonomy to procure its own goods, which results in extreme delays for much-needed supplies MHS submits data regularly to MOH, but the data will be inaccurate owing to its inability to monitor CHCs and HPs. INS has limited input on who MOH selects to staff CHCs, resulting in inconsistent distribution of staff and mismatch of competencies MHS has limited input on who MOH selects to staff CHCs, resulting in inconsistent distribution of staff and mismatch of competencies PNDS and MOH do not collaborate; HPs are built without consideration for staffing and supply needs MHS does not have the capacity to carry out regular monitoring of HPs, leading to a disconnect between MHS on the realities of supply needs at HPs MHS does not have the capacity to carry out regular monitoring of HPs, leading to a disconnect between MHS on the realities of supply needs at CHCs CHCs are unaware of why some staff are chosen for training and others not; this leads to some staff being double trained and others not receiving training Frequent turnover of upper management at INS threatens institutional knowledge management, hampering staff’s ability to do their job effectively Though partners and the MOH convene a technical working group, learning is not documented, and action items not acted upon Microplanning between MHS, HPs, and community leaders is effective, though best practices are not generated, documented, or shared among attendees MHS does not adequately document lessons learned to improve future programming and forecasting HP staff lack a sufficient patient load to gain critical clinical skills; other HP staff often do not have the time or resources to go out into the communities owing to a high workload HPs have a shortage of critical supplies and infrastructure needed for trained staff to adequately deliver services, lowering their ability to practice clinical skillsets Training of HP staff by INS is variable and uncertain; staff may be double trained, not trained at all, or not have time to attend trainings a b c d e f g h i j k l m n o p q r USAID.GOV REINFORCE FINAL EVALUATION | 34 CONCLUSIONS • There is little evidence that planning and budgeting are complementary or aligned, and a lack of transparency in MOH-level budgetary decisions affects programming at the MHS and community levels. There is limited evidence of data-based strategic decisions currently being made at both the national and municipal levels, which affects accountability and efficiency in institutionalizing best health service delivery standards. Extreme delays in budget approvals mean that MHS and CHC/HP cannot adequately or accurately plan their programming. • Inefficient, uninformed, and non-transparent decision-making authority among CHC, MHS, and MOH influences inaction at both the national and municipal levels. The MOH and MHS are disconnected from the realities on the ground and do not use data to make decisions. Decisions made are often delayed and lack transparency. Official guidelines on some clinical practices have not yet been operationalized, leading to non-standardized delivery of FP and ARH health services. • Despite several mechanisms to collect and share information on facility readiness, shortfalls in infrastructure, HR, and supplies are unaddressed, leaving facilities underequipped and understaffed. Departments across the MOH do not share information, particularly with the national Logistics Department. Departments are unaware of what other departments do or how their activities fit into the bigger picture. There is no culture of knowledge management, exacerbated by frequent turnover of senior level staff. Communities rely heavily on Reinforce for transport and human resources needs. • A lack of opportunities for career advancement, training, and recognition threatens the effectiveness and tenure of health workers. There is no formal human resource or performance management system to recognize staff, and many have double the job responsibilities for no additional pay. Ill-informed staff distribution leaves gaps in skill development and staff motivation and fuels unequal distribution of key staff with relevant skills. EQ 3: SUSTAINABILITY What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? Which part(s) of the model is more promising in terms of sustainability? Why? What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? MUNICIPAL TRAINING, CERTIFICATION, AND FUAT IN FP AND MNCH SERVICES Promising: The decentralized training approach, training participants database and training needs assessment, and clinical practice site model. The decentralized training approach of regional, junior, and advanced trainers is intended to be more cost effective (as it doesn’t require trainers in Dili to travel frequently or per diem for regional health staff to attend trainings in Dili) and can more flexibly address local training needs and maintain the improved skills of trainees in Covalima. The training participants’ database and training needs assessment records trainings, presents results, certifies participants, and prioritizes future training, and FUAT and will continue to be used and updated by INS staff for Covalima to track outstanding training needs and identify trends. The clinical practice site model is currently being scaled up to additional municipalities, and INS accreditation relies on continued assessments of the efficacy of these sites, but this model is not without flaws and will need continued investment, as discussed further below. 35 | REINFORCE FINAL EVALUATION USAID.GOV Less promising: MHS assuming responsibility of trainings from INS. The municipal trainers’ team is slated to take over the responsibility from INS to identify training needs, provide trainings and FUAT, provide refresher trainings, and maintain the quality of the clinical practice site in Covalima. But as of January 2020, the MHS team was not yet organized to take over the trainings from INS. And while INS expects the MHS in Covalima to take over training responsibilities in Covalima, it cannot provide a budget for operational costs. To sustain the capacity building of health staff in Covalima, they rely heavily on donor and partner financial support (see EQ 5). The GoTL budget is only enough to maintain the team and work in Dili. Remaining gaps for sustainability: • Sustainability of the municipal training approach in Covalima is constrained due to lack of ownership and lack of clarity around roles and responsibilities. No technical agreement has been arranged between INS and responsible stakeholders about responsibilities, accountability, and budget allocation. Consistent coordination with all the responsible stakeholders requires communication between the MOH Cabinet of Policies and Cooperation, MOH Department of HR, MHS, regional hospitals, and CHC leaders. • In Covalima, the six municipal trainers are not managed by either INS or MHS. They lack a work plan that identifies their expectations to deliver trainings in their facility or elsewhere (where they should go and how often), they are not well utilized, and they receive requests to travel to subdistricts in Covalima at random, so they are unable to plan appropriately with their day-to-day work. • In order to consistently continue to do FUAT with trained Covalima health staff, trainers from INS and MHS require per diem and transportation allowances that will enable them to travel from Dili and Suai to other subdistricts. This transport is currently being provided by Reinforce, but trainers were not confident that they would receive this support from INS after Reinforce ends, and as one noted, “if I have to pay by myself to go to another district to do a training, then I won’t go there.” • The clinical practice site at Suai Referral Hospital is underutilized after training, mostly because Covalima-based providers cannot travel there. The clinical practice site at Suai Referral Hospital will continue to be used during training, but the communication about the long-term purpose of the clinical practice site at Suai Referral Hospital is not understood by municipal-trained health providers in Covalima. The only time staff report using the site is during the hands-on part of training with INS. None of the Covalima facility staff interviewed had plans to revisit the clinical practice site after training due to their workload and inability to travel to Suai Vila without financial assistance, and only a small proportion of respondents knew that the practice site was an openly accessible resource. • Available resources for follow up training are unknown to many trainers. HP health providers interested in more training believed that they could only receive training from INS in Dili and were unaware of the regional trainers available to them. Further, staff do not utilize peer networks for skill improvement. Many note the need for more training, but few remarked an intent to learn skills from their peers who have been trained. Opportunities to overcome gaps: • Formalize a workplan for trainers that includes monthly local and municipal training and FUAT expectations to create a structured schedule that works with their facility schedule as healthcare providers. Share this schedule with CHCs to better align with HR needs and timelines and ensure staff are aware of the resources available to them, and where and when they can access them (INS, MHS, and Suai hospital). • Include training and refresher training transportation and per diem in the line items of municipal budgets and planning exercises (MHS and CHCs). USAID.GOV REINFORCE FINAL EVALUATION | 36 • Initiate monthly or quarterly peer learning activities for staff in the absence of formal training (or in between trainings) to improve staff camaraderie and motivation and allow skilled staff to share their expertise with less confident staff (CHC Management). • Support INS to initiate a technical agreement with the key stakeholders who are integral to the support of a continued training model including INS, MOH Cabinet of Policies and Cooperation, MOH Department of HR, MHS Covalima, Suai Referral Hospital, and CHC managers. DATA COLLECTION, MANAGEMENT, AND SHARING Promising: Complete, timely, and quality data collection at the municipal level, use of facility readiness tools, supportive supervision, and RDQA by MHS, and microplanning meetings. According to the national M&E Department KIIs, Covalima municipality is providing complete, timely, and quality health indicator data after guidance received by Reinforce, and M&E feels confident that they will continue to produce quality data after Reinforce. Monitoring and data management tools for facility readiness, supportive supervision, and RDQA have been implemented and adopted by Covalima MHS, and all tools are part of the MOH monitoring package of health facilities, which is institutionalized at the national level. Significant coaching and guidance in the implementation of these tools by Reinforce has helped MHS to take on the responsibility to implement them at the municipality level. MHS representatives verbally confirmed their plans to continue executing facility readiness assessment, supportive supervision, and RDQA with the M&E Department, and to take over Reinforce’s coaching of the HMIS officer at the CHC level to provide quality data. MHS submitted a proposal for this fiscal year budget, including activities previously covered by Reinforce (microplanning logistics, monitoring and supervision, refresher training, transport) and is preparing for an exit strategizing meeting with JSI to craft an action plan for 2021. In KIIs, Covalima MHS is enthusiastic about continuing to conduct microplanning meetings to share information between facilities and communities. While microplanning as a concept is well appreciated by a variety of stakeholders, there are some serious concerns at the facility level about the longevity of the practice. Less promising: Data analysis, interpretation, and reporting, and comprehensive assessments like SARA. MHS and facility staff feel confident in collecting and entering data and ensuring good quality data, but capacity for comprehensive analysis and reporting is still lacking. There is a concern among staff that quality may diminish if guidance from higher skilled data management specialists from the national M&E department or from Reinforce (or other partners) is not available; the national-level M&E department confirmed their limited capacity to conduct consistent follow up in the absence of Reinforce. The SARA will be completed by JSI once more in Covalima before the activity completes later in the year. While MHS and MOH staff acknowledge the importance of completing such an assessment, they do not have the number of staff or transportation available to conduct a comprehensive assessment like SARA in the future, in Covalima or elsewhere, without partner support. Remaining gaps for sustainability: • Lack of feedback to facility-level staff on data collection results. Covalima facility staff have not seen results from the submission of information on facility readiness, which demotivates staff to keep up quality data entry and reporting. • Limited capacity of national-level M&E staff to guide the use of monitoring and data collection packages. Data management knowledge has improved for MHS Covalima after Reinforce’s streamlining of M&E tools and provision of coaching. The sustainability of the use of the monitoring and data collection package depends on the leadership of national-level MOH departments, like M&E, HMIS, and Cabinet of Quality Control, but they have limited capacity to properly support the municipalities in monitoring and data collection, analysis, and interpretation in the absence of Reinforce. 37 | REINFORCE FINAL EVALUATION USAID.GOV • Basic logistics-related costs, such as internet and printing, at the MHS level. Internet connection, printing of tally sheets, and transport to CHCs and HPs for collecting data are areas that have been supported by Reinforce in the last four years. The sustainability in these areas depends on continued partner involvement or improved budgeting at the MHS level. • Non-use of data at the MOH level in making procurement decisions. As noted in EQ 2, JSI￾supported procurement will provide Covalima facilities with the minimum package of equipment, but the non-use of SARA data is an example of poor use of data for decision-making; the inaction from the MOH side in executing equipment orders based on clear data does not bode well for future procurements. • Basic financial assistance, such as per diems and transport, for staff to hold and attend microplanning meetings. Microplanning is lauded at the municipal level, but facility staff are less optimistic about continuing the microplanning activities in the absence of Reinforce and suggest that meeting frequency may need to be reduced. Microplanning meetings heavily rely on Reinforce support for laptops, projectors, and financial assistance provided for meals and transportation to these meetings. Community leaders are critical participants at the meetings because they identify trends within community behaviors and health challenges, yet CHCs noted that if they do not receive transport assistance for microplanning, SISCas, and other outreach, they will not continue to attend. AE is not engaged in microplanning to ensure community leaders’ accountability. Additionally, there is a missed opportunity at microplanning meetings to create long￾term planning strategies and match budget to activities, and the action plans created are inconsistently monitored by CHC managers for effectiveness, feasibility, and accountability. Opportunities to overcome the gaps: • Engage with AE to reinforce the role of community leaders in microplanning. Ensure logistical costs for meetings are included in MHS budgets and in CHC level action planning. • Ensure that each facility inventories, all new equipment received from Reinforce is listed, and this inventory is updated and maintained at both municipal and national levels. • Encourage municipal-level dissemination of SARA results after the next round of data collection by Reinforce so that facility staff can see and use data from their subdistrict. COMMUNITY ENGAGEMENT AND HEALTH PROMOTION Promising: Outreach activities such as night events/FGDs with expectant mothers and FP education for men and women, the community-led transport model TraKom, and the ARH package developed by JDN. Men and women seeking services responded positively to activities already established by the national health promotion department such as SISCas and sweeping activities, and facility staff were interested in continuing night events and FGDs with expectant mothers and community members. Reinforce saw an increase in couple years of protection (CYP) due in part to the introduction and education of a variety of family planning options for men and women. To respond to pervasive community concerns around the lack of available transportation (ambulances, multi-function vehicles) for health emergencies, especially for women in labor, Reinforce introduced a community-led transport model, TraKom. The model relies on community members with functioning vehicles who can help provide transportation when emergencies arise. Community members make small financial contributions toward petrol, and either the community leader manages the fund, or the users pay directly. Facility staff and local leaders laud this initiative as a means to work together as a community and take responsibility for each other. TraKom was implemented in four communities in Covalima, and there are now pilot programs at two villages in a different municipality (Tirilolo Baucau and Purugua Bobonaro). The ARH package developed with JDN has already been implemented in four other municipalities, and the MOH has “Microplanning doesn’t include budget, but the financial aspects JSI always pays for, so we never think to include that.” - Doctor, Zumalai USAID.GOV REINFORCE FINAL EVALUATION | 38 advocated piloting the package in additional sites. During partner meetings, MOH has advocated for other development partners to adopt the approach and materials, and partners have since approached JSI to express interest. Additionally, an ARH policy has been introduced by several stakeholders and is awaiting MOH approval. Less promising: Continuation of SISCas or support to attend SISCas. SISCas are mandated by the government as outreach and health promotion activities under the basic services package and included in facility staff’s job descriptions. However, the number and frequency of SISCas has dwindled in the last four years. In some subdistricts, CHCs received support from Reinforce for transport but reduced SISCas in favor of building more accessible HPs and mobile clinics. In Fohorem and Fatumean subdistricts, where Reinforce did not provide outreach assistance, SISCas were stopped because MHS no longer had funds to support the transport assistance required for volunteers ($35) and community chiefs ($25); these entities would not participate if not receiving this assistance, and the health staff are not confident or physically able to conduct the SISCas without support. Remaining gaps for sustainability: • Staff are trained on certain FP methods, but these methods are not always available in the health facilities. If methods are out of stock, staff must counsel patients on switching FP methods, and this is disruptive for those seeking long-term methods and could affect progress in the improvement of Couple Years of Protection (CYP). • Health promotion through community outreach is exceedingly difficult depending on the number of staff at the facility, the patient load, and individual motivation. Staff have not received formal training in conducting outreach and BCC, though it is part of their job expectations. Reinforce’s approach for community outreach created an extra pair of hands to do the necessary outreach activities but did not improve the capacity of the CHC or MHS staff to do so independently. In the absence of Reinforce CHOs (who will cease activities in the coming months), staff will again struggle to take on this second job responsibility. • The supply of volunteers for community health outreach activities is unreliable, and facility-based health staff lack management control over them. Volunteers for community health outreach activities fall under the responsibility of PNDS, who rewards them for their work. They are replaced whenever community leaders decide to do so. The health staff has no official control over the volunteers and has no responsibility in training and rewarding them (as they had before 2015) and thus cannot rely on them to provide continued support for health promotion activities. • Lack of specific budget line items for basic community outreach expenses such as transport and materials. The budget for health promotion is restricted to the activities in the government program such as SISCas and mobile clinics, and, outside of partners or donors, there is no specific budget for transport, activity materials, and printing. • The TraKom system is slow to scale, as it is heavily dependent on community leadership. The TraKom system model is promising, but adoption and consistent implementation and accountability relies on the community and their trust in leadership. While respondents agree the concept is logical, it has not inspired others to implement it, and it has been slow to scale outside of the original locations. It is unclear whether AE or PNDS is involved or informed about TraKom or whether they can integrate the concept in their community outreach programs. Opportunities to overcome the gaps: • Finalize the MOH’s HRH policy to ensure that all facilities are meeting minimum standards and staff are enabled to complete their job requirements, including outreach. • Engage AE and PNDS in adopting and institutionalizing a community volunteer model and community member involvement in the TraKom model. 39 | REINFORCE FINAL EVALUATION USAID.GOV FINANCIAL DEPENDENCY AND PLANNING Budgets at all levels have been reduced to account for partner spending; as a result, MHS staff are not literate in comprehensive budgeting. Capacity and commitment rely in large part on financing; however, the national level budget is reduced to account for donors and partner presence. MHS reduced their budget because Reinforce provided financial support for outreach, including fuel, transportation, and per diem, so staff are not literate in comprehensive budgeting for all activities. MHS and MOH do not show any evidence of short- or long-term planning and have no strategic workplans in place for the future. They assume that other partners will step in to fill the financial void that Reinforce is leaving. Government counterparts at both national and municipal levels state that they own these activities around service delivery, community health promotion, and continued training and are “responsible” for upholding them, but they did not have a strategic workplan in place for any of the activities or promising practices introduced under Reinforce. MHS stated that they have allocated more funds to cover activities for the next budget year, but they do not have an activity or implementation plan for these activities, and KIIs with facility staff and CHC chiefs confirmed that there is no short- or long-term plan from the MHS. None of the other stakeholders were aware of any changes made to future budgeting as a result of learning received under Reinforce. The assumption within the MOH/MHS that other partners/donors will step into Covalima to support or sustain health service delivery is unfounded, as partner commitment has not solidified, and many partners are scaling back their support on MNCH. CONCLUSIONS Reinforce initiatives to support INS’s training capacity, improve Covalima MHS’s data management knowledge and quality, and improve health seeking behaviors and health outcomes for MNCH have been successful in many ways, but all activities are still heavily challenged in their sustainability, owing to a variety of factors. These challenges and opportunities to fill the gaps are: • Lack of organizational, financial, and institutionalized ownership of activities prevents sustainable adoption of many Reinforce activities. This includes activities such as the use of municipal trainers, use of data for decision-making, and conducting community outreach. This is evident in the lack of clarity between MHS and INS in, for example, managing trainers, as well as a centralized decision-making entity on data-related issues at the MOH. • Activities with a better chance of sustainability are those that have clear ownership. Ownership can be by MOH, health staff, or community and are integrated in existing systems. Examples here are outreach programs at the community level and use of training-related organizational tools, facility readiness tools, and supportive supervision at the INS level. • Lack of coordination and communication between responsible stakeholders prevents effective collaboration and the use of all resources available to MOH. This is particularly evident within the MOH, where discussions are not held, and data not shared between departments. Systematic cooperation and knowledge sharing will lead to more efficient work and the potential for longer-lasting effects. • Pervasive financial dependency on donors and partners has prevented municipal- and national-level authorities from taking responsibility to budget and plan independently. At highest risk here is planning and obtaining funding for basic logistics costs (travel, per diems, printing, materials) related to outreach and training activities. A reduction in the revolving door of partner funding may create instability unless financial management is prioritized, and sufficient funding is provided for priority activities. USAID.GOV REINFORCE FINAL EVALUATION | 40 • The sustainability of the use of the monitoring and data collection package depends on the leadership of national-level MOH departments like M&E, HMIS, and Cabinet of Quality Control. but they have limited capacity to properly support the municipalities in monitoring and data collection, analysis, and interpretation in the absence of Reinforce. EQ 4: ADAPTING AND SCALING THE REINFORCE MODEL What are the capacity and commitment of the GoTL, the MOH, and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? According to Reinforce documents, a Model Municipality is demonstrated when: Well-trained and competent health providers deliver quality FP and MNCH services in fully equipped and people-friendly health facilities where providers receive consistent, high-quality supervision from MOH and INS experts. Health facilities are easily accessed by community members, who have the knowledge and skills to demand quality, client-centered services. Health providers and the facilities in which they work are supported by a municipal health system that uses data to make informed financial and programmatic decisions. In a model municipality, communities are empowered to work with government and civil society structures to generate innovative solutions to their health challenges.13 To answer this evaluation question, the evaluation team synthesized the definition of a model municipality into three distinct pillars: health service delivery, health data for decision-making, and community use of services. PILLAR 1: HEALTH SERVICE DELIVERY CAPACITY STRENGTHENING INS is generally behind the idea of replicating the Covalima model outside of Covalima. INS staff and partners suggest that the Covalima model of addressing health worker capacity improvements specifically in MCH can be continued in other municipalities like Viqueque and Manufahi, which continuously show poor performance in meeting targets to improve maternal and child health. Health workers in these areas can particularly benefit from emergency obstetric care (EmOC), safe delivery, and newborn care training and from repeated follow-up and monitoring, and the training database created by Reinforce will help INS to monitor enrollment in these courses for the most in need areas. MOH and INS appear committed to replicating the practice site and skill lab model for training of new health facility staff outside Covalima; however, their capacity to do so is yet undetermined. The practice site and skill lab model for training new health facility staff is widely lauded, and the MOH has already expressed commitment to replicating it in other municipalities. The newest site launched in Oecusse in 2019 under the NGO Health Alliance International (HAI), and INS visited to discuss standard operating procedures and review technical skill development and utilization of the space. There are currently three additional sites in development in Baucau, Ainaro, and Bobonaro funded through several different partners. Key INS activities bolstered under Reinforce (TNA, certifications, guidelines for all training areas) have been integrated into the draft INS Strategic plan 2020–2024 to be rolled out with health professionals across the rest of Timor-Leste but will depend on partner support and associated budget allocation. For more on INS capacity developments, see EQ 5. 13 USAID Reinforce Basic Health Service Project. Covalima: Becoming a Model Municipality. 41 | REINFORCE FINAL EVALUATION USAID.GOV PILLAR 2: HEALTH DATA FOR DECISION-MAKING Data collection, interpretation, and availability has improved at the municipal and national levels; however, its use in programmatic and financial decisions is lacking and constitutes a barrier to replication of the model outside Covalima. In terms of data analysis and data utilization, the National M&E department believes that Covalima served as a “good example” for other municipalities and that the MHS ability to collect and interpret health outcome data had improved substantially, which is promising for other MHSs in the country. The Facility Readiness tool (revised to include comprehensive questions from the SARA) will continue to be used by the M&E Department at the national level and is seen as the gold standard for assessing and monitoring facility needs nationwide. The data collected from microplanning and shared at the national level provides complementary information on community-level challenges that prevent care seeking, especially for pregnant women and families reluctant to get immunizations for children less than one year old. This information is currently used to see that Covalima has made progress since the start of Reinforce and can be used to design outreach strategies in other parts of Timor-Leste. The GoTL is committed to using data to identify health-related financial and programmatic needs outside Covalima; however, it currently lacks the human resource capacity to do so. There are a number of areas of M&E that showed progress under Reinforce, but scaling and replicating these practices will prove difficult for the MOH, and shortages in human resources and funding for transportation will affect the five-person M&E department. While the Covalima MHS staff benefitted from improved capacity in using the DHIS-2 platform and data entry processes due to Reinforce and national MOH staff guidance, there is only one municipal HMIS staff member, and there are insufficient national M&E staff available to independently train or coach additional field staff outside of Covalima in the future. RDQAs are currently done in Covalima and two other municipalities by the National M&E team to review the consistency in completing forms and entering data. However, in order to fully scale this practice to more municipalities, the M&E department will need donor or partner assistance to provide additional human resources, including a dedicated data quality assurance officer and likely another HMIS officer. Likewise, the M&E department feels strongly that assessments like the SARA can be used in other municipalities to assist in prioritizing staffing and equipment needs; however, there is uncertainty about the feasibility of conducting SARA on a large scale due to the same reasons. PILLAR 3: COMMUNITY USE OF SERVICES Systemic, underlying weaknesses within the entire health system, particularly in the areas of inadequately supplied and staffed health facilities, threatens the success of replication of the model in any municipality. Numerous stakeholders expressed disbelief that the GoTL would be successful in scaling the Covalima model, not only due to budget and human resource capacity constraints, but also due to a lack of continuous investment into the systemic issues that have been plaguing the sector for years. Within Covalima, health education in communities has improved health seeking behaviors, but HPs and CHCs are understaffed and ill￾equipped to consistently address community needs due to continuous constraints in resources like transport, equipment, medicine, electricity, and water needed to provide a full-service package. These are critical aspects underlying the ‘Model Municipality’ logic and framework. The MOH’s inaction to procure needed resources over the last three years further erased the trust that these facilities and the MHS have in the GoTL and paint Picture: Empty water storage containers at a HP in Zumalai USAID.GOV REINFORCE FINAL EVALUATION | 42 a more pessimistic picture of what the MOH could accomplish in other municipalities without relying on partner support. Covalima municipal-level stakeholders feel that Covalima’s improvements will be reversed if support is removed from the municipality in favor of others. Some Covalima MHS staff fear that MOH’s scaling-up to other municipalities would mean support would cease to be provided to Covalima; after measurable improvements in facility staff clinical skills and data utilization, progress will decline if they are ‘left behind’ in favor of other municipalities. Partners and MHS staff alike felt that the GoTL should focus on securing the resources necessary to establish a successful model before any thought of scaling up. OVERALL CHALLENGES TO REPLICABILITY GoTL health leadership are not confident in their ability to sustainably replicate the model outside of Covalima due to financial and political uncertainty. KII feedback notes that many government staff do not feel confident in their ability to replicate the activities in Covalima in other municipalities due solely to an uncertain political climate, budget concerns, and a reliance on international donors and partners. Since JSI was providing financial support to uphold Covalima activities, and several other partners have taken responsibilities over other health project financing (including providing human resources and transportation in addition to financial support), the MOH is more capable of managing partners and less capable of budgeting accurately or carrying out these activities independently. A clear example is that of procurement; while Reinforce provided MOH with solid data to take action to equip facilities, JSI ended up procuring equipment themselves. While the MOH says it is committed to replication, its actions say otherwise. There is no evidence to suggest that MOH understands what is required for replication, and MOH has not made any progress toward preparing for replication. In general, MOH interest in and commitment to scaling up to another municipality are unclear and there is no prioritization of activities or plan for replication that would include responsible persons, timeline, or budget. Adaptation and replicability will rely on an understanding of the promising practices and what will need to be continued or be scaled, but there is no evidence of this kind of strategy or analysis by MOH. As noted in EQ 1, none of the staff at the MOH or MHS level had collected documented evidence of the successes or lessons learned associated with their involvement with Reinforce in Covalima that could be shared. Further, stakeholders responsible for ensuring institutionalization of best healthcare practices do not coordinate. Each department has its own strategy guided by donor presence and budget, and overall ownership and a collaborative, strategic direction is questionable. As one partner noted, “The GoTL is struggling with the perception that this is Care’s program, this is JSI’s program, etc., as opposed to this program is supporting the government's priority for our X or Y objective.” Figure 4 below illustrates that capacity and commitment in this context are measured by key guiding indicators that would lead to an enabling environment for progress. While there is promise, these actors are still in the nascent stage of making solid commitments to institutionalizing good healthcare service delivery practices, and suppressed capacity in skills, staffing, and financing thwart any real efforts toward progress. "Look at the government right now . . . do we have a government? Of course, we will need to rely on the partners" - CHC manager, Covalima 43 | REINFORCE FINAL EVALUATION USAID.GOV Figure 4: Assessment of Progress to a Model Municipality Improved health service delivery Communities use services Data is used for decision￾making Well trained and competent health providers Delivery of quality FP and MNCH services Consistent, high quality supervision of providers Fully equipped and people￾friendly health facilities Easily accessible facilities Knowledgeable clients who demand quality MHS uses data to make informed financial and programmatic decisions Communities and local governance structures generate solutions to health challenges Not achieved Achieved USAID.GOV REINFORCE FINAL EVALUATION | 44 CONCLUSIONS • GoTL, MOH, and INS understand where the Covalima model has succeeded and appear committed to replicating those parts of the model. The GoTL is enthusiastic and acknowledges responsibility to replicate the Covalima model. INS and partners acknowledge parts of the model that work well, such as the practice site and skill lab. The national level M&E department feels that data collection and use has improved and is committed to using data for health sector decisions in Covalima and elsewhere. • Commitment is, however, piecemeal and fragmented without a common vision. Replication of the model will rely on integration and coordination of multiple stakeholders, but there is no evidence of a commitment among any of these actors of a shared strategy moving forward. • The MOH and its associated departments are not adequately prepared to replicate the model. MOH has not initiated a plan identifying responsible parties, activities, and timelines to carry out replication. The M&E department does not currently use data to inform the health sector financial and programming in Covalima. The M&E department lacks sufficient human resources needed to perform more wide-reaching data collection and analysis to inform a replication of the model and direct activities to areas most in need. Further, capacity for budgeting and long-term planning within the MOH, INS, and MHS is weak and highly dependent on international donors and partners to take on key activities and scale up across Timor-Leste. • Systemic weaknesses in the current health system seriously threaten the success of replication of the model and the model in general. Obstacles in equipment, coordination, transport and accessibility, and clear budget allocations prevent the institutionalization of best healthcare practices in Covalima, and these issues are not limited to Covalima. Covalima has not yet become a model municipality to replicate in other municipalities due to considerable challenges in creating an enabling environment in which progress can be achieved. Lessons learned from Covalima are not being applied to the larger health system. 45 | REINFORCE FINAL EVALUATION USAID.GOV EQ 5: IMPROVED CAPACITY AT INS Beyond Covalima, how has the Reinforce Activity strengthened the capacity of INS? Where are the gaps, if any, in INS’s capacity and commitment to conduct and to manage in￾service training and continued education of health professionals in Timor-Leste? How has Reinforce mitigated these gaps? What, if any, are capacity gaps that remain? In 2014, a Situational Analysis of INS identified key areas of strengths and weaknesses to inform its future strategic planning for addressing the health workforce needs in Timor-Leste, and their five-year strategic plan for 2015–2019 sought to address these gaps and become a “center of excellence.” In 2016, JSI signed a memorandum of cooperation with INS to help with the implementation of their strategic plan and address 13 program priorities, including regularizing meetings and communication, M&E, systematic national standards, staff classification and motivation, facilities for in-service training, and funding research activities. In 2018, a midterm review was conducted to gauge progress made toward making improvements in the priority areas. The six key areas assessed in the review and also examined by the evaluation team were (1) leadership and governance, (2) training and research, (3) competent human resources, (4) infrastructure, (5) information technology, and (6) finances. CAPACITY FOR LEADERSHIP AND GOVERNANCE Reinforce helped INS clarify its future vision, and INS appears committed to future excellence; however, INS still relies heavily on partners for overall strategic-level guidance and financial support. Externally, INS cooperates with international development partners, including the UN agencies such as the World Health Organization (WHO), the United Nations Children’s Fund (UNICEF), and the United Nations Population Fund (UNFPA), bilateral donors such as USAID, DFAT, the Korean International Cooperation Agency (KOICA), and multiple international non-governmental organizations (NGOs), to implement the strategic plan. It is widely felt that INS coordinated well with partners and that they are “eager and responsible” and “show a commitment to excellence” but have not yet reached a ‘center of excellence’ as declared in their five year strategic plan. Reinforce’s assistance in strategic planning assisted them to better clarify a vision and objectives, as seen in the most recent draft of the 2020–2024 Strategic Plan. However, many stakeholders believe that INS lacks a coordinated framework that outlines what is needed from partners, and instead allows partners to identify the areas of intervention. Several partners agreed that the deference to partners and the need for financial support dictate in large part how INS operates and, as with the MOH, that the dependence on partner support for implementation greatly influences the overarching vision. CAPACITY FOR TRAINING AND RESEARCH Reinforce built the INS capacity to better organize, manage, and track their programming, particularly through the creation of a training database and updated MNCH curricula. At the beginning of the activity, Reinforce’s involvement of INS staff in the KAP survey improved its credibility as a fledgling research institution and enabled it to tie its work to community health outcomes. The training and FUAT database established and implemented by Reinforce created a system for quality improvement guided by data that ensures a comprehensive review of all health staff receiving training at INS, no matter the municipality. The database enables INS staff to review the full spectrum of courses offered, the number of staff receiving training, and those still in need of both initial training and FUAT, as well as tracking those staff who have ‘passed’ or ‘failed’ FUAT and need further monitoring. Reinforce helped to set up a team of INS and MOH technical staff to draft a standardized curriculum for MNCH and contributed updating the FP curriculum. These curricula can be used in initial trainings and as guidance for follow-up competency assessments in all municipalities. The database and curricula also ensured the improved quality of FUAT USAID.GOV REINFORCE FINAL EVALUATION | 46 provided both in and outside Covalima. The database provides rich information on staff competency and trends; however, feedback to facility staff about the results are varied, and staff stated they did not know what INS did with the data from FUAT after they were assessed. INS staff themselves question the impact of the FUAT results at the municipal and national levels. Reinforce strengthened INS most evidently through standardization of guidelines, protocols, checklists, and curricula, which improved INS training staff ability to adhere to strict service delivery standards. Reinforce’s support of standardized guidelines at the three clinical practice sites improved INS’s ability to target practical training to the needs of trainees, especially for those trainees in remote facilities where the patient load is varied and there is less opportunity for hands-on practice. Between 2016 and 2020, five internal (self) and external assessments were completed at each of the three sites. The most recent external assessment, done by JSI, had concluded at the time of the evaluation and measured adherence to standards at the sites (See Figure 5 below). While there was some fluctuation in the scores from external assessments owing to the use of more rigorous measurement tools in 2020, all three sites’ overall scores improved from baseline by at least 20 percent. The assessment noted that all three facilities most commonly referenced the quality checklist tool for safe delivery, which was promoted by Reinforce (60 steps), and normal birth care improved across all three sites on average by 31 percent between 2016 and 2020. Other notable improvements include MCH Program Coordination (following timelines for pre- and postnatal care), from an average of 70 percent to 95 percent and counselling (including FP, healthy behaviors, and ANC), from an average of 64 percent to 88 percent. Though Reinforce conducted Quality Improvement workshops with the MOH and INS, it was noted that all facilities could benefit from improved quality management; outside the aforementioned checklist, there is no evidence of a functional structure to routinely identify problems and try to solve them. Figure 5: Clinical Practice Site External Assessment Scores, 2016 (baseline) to 2020 INS practice sites have the potential to be replicated outside Dili and Covalima, but infrastructure deficiencies and a staff shortage must be addressed first. The practice sites are public health facilities, not training centers, first, which demands that MHS and MOH secure the infrastructure and equipment needed to enable following guidelines. Standards in facility infrastructure have improved since 2016, but the completeness of intervention rooms has remained unchanged in two of the practice sites, only improving marginally in Suai. With the discussion of establishing more practice sites in Baucau, Ainaro and Bobonaro, there is an opportunity to scale up and provided needed assistance to facility staff; however, it is widely noted that human resource shortages, including within the INS “Some [trainers] have challenges analyzing the FUAT results—sometimes they just leave the results here and for those who have time to do the analysis they make recommendations, but there are rarely follow up or actions based on our recommendations.” - INS trainer, Dili 68% 65% 72% 70% 71% 69% 86% 92% 90% 85% 86% 86% Site A Site B Site C 2016 (baseline) 2017 2019 2020 47 | REINFORCE FINAL EVALUATION USAID.GOV Department of Standardization and Quality Control, will greatly affect INS’s ability to efficiently address training needs in all sites. Reinforce’s support for INS to achieve accreditation to bolster its credibility and reputation was short-lived. Initial collaboration with JNPK in Indonesia, part of the 2015–2019 Strategic Plan, failed to be realized once it was determined that JNPK could not guarantee internationally recognized accreditation and no longer provided it. CAPACITY FOR IMPROVING HUMAN RESOURCES Reinforce supported INS to train a cadre of trainers; however, INS still lack enough regular and advanced trainers and currently has no plan to increase their numbers. To better address the unrealistic expectation that the limited number of trainers at the INS in Dili would be responsible for training trainers, training staff, and conducting FUAT throughout the entire country, Reinforce supported INS to initiate a model for advanced and master trainers whose sole responsibilities would be to conduct Training of Trainers (ToT) to expand the number of trainers across the country and to revise curricula in the future. Reinforce hired an external consultant from Indonesia to train advanced trainers based in Dili, who in turn trained (regular) trainers from Dili at Suai hospital, MHS, and two primary health facilities in Covalima. As noted in EQ 3, while this is a promising model, to date there is still a limited number of regular and advanced trainers, and INS frequently relies on external staff from outside Dili to fill gaps. There is no evidence of a plan to increase the number of advanced trainers or ToT targets to increase coverage to multiple municipalities in the new Strategic Plan for 2020–2024. Trainers at both the national and municipal levels require more professional development to feel confident in their roles. While they report improved capacity to address clinical skill competency through the standardized guidelines and tools provided, they have not received any personal training on adult education or professional training delivery on “how to be a good trainer,” which they noted as an area in which they sought improvement. While trainers complete self-assessments at the end of each semester and there are intermittent meetings between evaluators and trainers to discuss feedback, there is no follow up mentorship for their professional development. Trainers are unmotivated because they are essentially working double jobs. Expectations around job responsibilities, supervision and training, and scheduling needs are uncertain. Female trainers specifically noted pressures to return home to take care of their families competing with the long hours required for training. Every trainer is a certified health professional by day but does not receive remuneration commensurate with the additional responsibilities for training and traveling to remote areas, including overtime work and consistent per diem outside of “The top-level people are expecting their indicators to be achievable, but they don’t know the process for those who have to collect this information [FUAT] and how hard and tiring it is.” - INS trainer, Dili Picture: Clinical Practice site at Suai Referral Hospital, Covalima (supervision of IUD insertion practice) USAID.GOV REINFORCE FINAL EVALUATION | 48 Dili or Suai, despite having advocated for it to management. INS reports that receiving “no clear guidance” from MOH on training expectations limits their autonomy. While they expected a technical agreement between MHS and the Suai Referral Hospital on the use of trainers in the region, including a specific coordinator sitting within MHS, this has not occurred. As noted under EQ 2, INS cannot direct where the trainers should go or what they should do, as this is a function of MOH, and this restricts their ability to institutionalize the training model. Municipal trainers should be able to create their own training plan and only receive coaching from INS but are unable to do so without direction from their facility supervisors. Poor coordination between INS and MOH constrains the institute’s ability to build human resource capacity in the future. While information collected in the training database is shared with the MOH, INS is not part of human resource management decision-making at the national level, including selection of staff for training and distribution of facility staff within municipalities. CAPACITY FOR MANAGING FINANCES INS has improved its capacity to budget and manage its finances, although its allotted budget decreases from year to year. INS’s financial resources come from the General State Budget and financial support from partners. At present, there are 16 partners supporting INS in a range of objectives for the next Strategic Plan 2020–2024. Unlike the MOH, INS has worked in the last five years to develop budgets closely aligned with program indicators so that the State Budget is directly and realistically allocated to key activities, despite the fact that funding has consistently decreased. In KIIs, partners report that INS’s capacity in financial management and reporting has improved, with fewer errors in financial reporting and better documentation of expenses; however, there is no internal auditing mechanism at INS at present to provide feedback or insight on financial management. The state budget, used to fund staff salary, goods and services, minor capital, and development capital, has decreased every year since 2017 due to political instability, and between 2018 and 2019, the Human Capital Development Fund (HCDF) reduced the training budget alone by 58 percent. Similarly, funding from development partners has decreased by more than 60 percent.14 Continued follow-up and monitoring of trainees and trainers will be a challenge for INS since Reinforce covered training and monitoring-related costs. There is no government budget for monitoring, and most INS partners do not allocate funds for M&E activities, only for training. While JSI covered these costs in Covalima, it is not guaranteed that the FUAT monitoring will continue with the same consistency from other partners, both in Covalima and in other municipalities. INS has no budget for organizing the trainings in each municipality, so training funds need to be incorporated into MHS-level line-item budgeting. INS will rely on partner human resource support and financial support for transportation for trainers to visit expanded clinical practice sites and conduct FUAT in areas outside Dili. 14 National Institute of Health (INS) Five Years Strategic Plan 2020-2024. Draft. Ministry of Health, Democratic Republic of Timor-Leste. 49 | REINFORCE FINAL EVALUATION USAID.GOV CONCLUSIONS • Reinforce assistance contributed strongly to INS organizational strengthening and adherence to best practice standards for service delivery. Reinforce supported INS to set up a FUAT database as well as helped draft or update standardized curriculum for MNCH and FP service delivery. Further, INS now adheres to best practices in international service delivery standards as a result of Reinforce’s work. • Reinforce’s focus on strategic planning has increased INS’s leadership capacity, but sustained and strong leadership at INS is threatened by an overreliance on partners for guidance and an unclear mandate from the MOH. INS is probably the most committed to future excellence of all the institutions that Reinforce worked with, yet INS is still highly reliant on partners for strategic-level guidance and leadership. Without clear guidance from the MOH on training expectations or involvement in human resource discussions, INS’s ability to lead in the future will be limited. • Though the training of a new cadre of INS trainers has been commendable, there are several challenges to ensuring trainers can remain effective in the future. There are not enough trainers to cover all tasks, and many trainers require more professional development to feel confident in their roles. Further, motivation among trainers to do their job may be threatened by the burden of high workload. • The draft 2020–2024 Strategic plan does include an interest in scaling-up activities but lacks a focus on how to achieve them, and heavily depends on partners to provide funding and uphold the vision. INS continues to rely on Reinforce and other partners to fund training- and monitoring-related costs, and it is unclear how those activities will be funded once Reinforce ends. Further, while the practice site model has potential for replication, systemic issues around staff shortages threaten its success. USAID.GOV REINFORCE FINAL EVALUATION | 50 RECOMMENDATIONS Recommendations were generated through a participatory workshop with MOH, MHS Covalima, and INS, where participants discussed key concerns about health service delivery, bottlenecks to achieving progress, and responsible parties to address issues in the future. These recommendations are complemented by evaluation team data directly pulled from evaluation findings, as they relate to each evaluation question. They are presented in line with key thematic areas explored throughout the evaluation. Capacity strengthening for health service delivery • Future health capacity strengthening activities by international partners must include preliminary assessments of multiple areas of capacity weaknesses, not only in M&E and data for decision-making, but also in management, administration, operations, and communications, in order to elicit sustainable improvements at any scale. • PNDS should create a formal agreement with village-level community volunteers regarding expectations, and like facility staff, community outreach should be included in their ‘job’ descriptions. Volunteers should work in partnership with health facilities to fill outreach gaps to reduce dependency on partners and can further provide sustainable support and accountability for initiatives like TraKom and SISCas, without the expectation of additional payment. • INS, MHS, and Clinical Practice Sites must formalize a workplan for trainers that includes monthly local and municipal training and FUAT expectations to create a structured schedule that works with their facility schedule as healthcare providers. This schedule is to be shared with CHCs to better align with HR needs and timelines and ensure staff are aware of the resources available to them, and where and when they can access them. • INS should work with MHS, MOH Health Promotion, and partners to fund and develop training curricula targeting health promotion and BCC for health professionals to enable facility staff to confidently engage in outreach activities, especially with men and adolescents. • To better use clinical practice sites, INS should provide trainees with flyers to take home after leaving INS in Dili that include (1) hours of the clinical practice sites at Comoro, Vera Cruz, and Suai Referral Hospital, (2) the names of trainers at MHS and at the practice site, and (3) the topics each trainer is certified to teach. One copy of this flyer should remain at each health facility and posted in a location where all staff can see it so they can schedule their own follow-up training or simply receive verbal assistance from their peers if needed. This will ensure staff can continue to practice skills they were trained on. • To reduce facility staff reliance on INS in an uncertain funding environment and better use data from monthly supervision, CHC management should facilitate monthly learning exchanges between staff at their facility to create organized peer learning opportunities focused on specific skill areas to be developed. This would not require financing and would create a peer learning network within the facility, create meaningful management feedback loops, and even use skills from staff at neighboring facilities. 51 | REINFORCE FINAL EVALUATION USAID.GOV • CHC management staff can initiate monthly client feedback surveys using community volunteers, not health staff, to elicit honest feedback from service recipients and effectively gauge improvements in quality and ensure patient satisfaction. This information should be collected mid-month, given to CHC management to discuss at end of month meetings, and included in the microplanning meetings every quarter to inform the action plan. • For staff unable to arrange transport to the practice sites for follow up, MHS and CHC management should coordinate to initiate a facility-based version of TraKom where staff can coordinate intermittent trips to practice sites to receive additional training or practice skills using their own vehicles. CHC management will create a sign-up sheet and work closely with the practice site on monthly training schedules. For staff who are the sole service provider, a mobile skills lab should be introduced with the support of INS, CHCs, and MHS that will use supplies from the practice sites, bring select equipment to the facility for targeted training, and use existing equipment within the facility. Trainers should receive per diem from MHS for participating in the mobile skills lab initiative. Using data for decision making • Hire a focused, senior level data interpretation and management analyst for the national M&E Department who will champion data quality control and facilitate in-service training to other national M&E staff to better serve the municipalities and train enumerators for large-scale data collection efforts such as a future SARA or KAP. • Facility readiness assessments being done in the field must be captured by the national M&E Department and efficiently disseminated to the Logistics Department and Procurement Department to make a collective decision for needs-based procurement. • MOF should hold training for MOH and MHS staff on accessing the Transparency Portal and interpreting budget data, as well as providing guidance on costing and budget forecasting for activity planning. • CHC Managers and MHS must ensure that each facility receiving new equipment from Reinforce inventories it in an Excel file that is updated monthly. This inventory should be maintained by both MHS and MOH staff and should also be shared between facilities so that they can utilize neighboring sites’ equipment in an emergency. • CHC Management can organize microplanning meetings at a smaller scale without the need for projectors or electronic equipment. Management can host monthly sessions with staff internally to review data, discuss gaps, draw out action plans, and circulate action plans to MHS via email for review and feedback. MHS should enforce microplanning as a good practice and monitor CHC management to hold them accountable for completing plans. • Microplans should be drawn by facility staff based on weekly data and always associated with a costing plan that is aligned with activities. CHC management should provide input on the drafting of all plans in their subdistrict and work with staff to ensure action plans are relevant, feasible with limited resources, and cost efficient. USAID.GOV REINFORCE FINAL EVALUATION | 52 Human resources and considerations for the healthcare workforce • As part of the MOH’s finalized HRH policy, MOH should include stipulations to increase employee motivation and incentivization for staff who consistently perform outside their job descriptions in the form of certificates of recognition showing how their additional work led to specific outcomes for the facility, or financial incentives wherever possible. This policy should also reinforce the ignored resolution passed in 2014 to increase pay and professional opportunities to staff working in remote areas for more than one year. • The MOH Human Resource Department must involve INS in decision￾making for staff distribution so that skills match population needs, trainer feedback on competencies from FUAT is held in high regard, and INS staff motivation improves. Cooperation, collaboration and knowledge management • The MOH Planning and Cooperation Cabinet should be responsible for ensuring that not only partners but also respective MOH departments are working cohesively. The Cabinet should hold quarterly roundtable meetings with partners and all MOH departments that provide services with them. National entities responsible for health service delivery should collaborate so that each actor is held accountable by the other departments, successes and failures are shared, and promising practices can be replicated. All meeting minutes should be documented, including responsible parties for action and timelines, and sent to invited stakeholders within one week. • The MOH, INS, and the Planning and Cooperation cabinet should facilitate a meeting to review alternative and diversified support opportunities, including encouraging partnerships with medical associations who can assist in knowledge exchange and training and engaging in private sector partnerships who could provide a diversified funding stream outside an unsteady government. • MOH (supported by USAID) should develop a sustainability strategy that identifies which program elements can realistically be maintained and how they can best build capacity within the health system in the next two years. The strategy will include metrics and targets for human resource and institutional capacity strengthening. 53 | REINFORCE FINAL EVALUATION USAID.GOV ANNEXES Annex I: Rapid Survey Assessment Results USAID.GOV REINFORCE FINAL EVALUATION | 54 55 | REINFORCE FINAL EVALUATION USAID.GOV USAID.GOV REINFORCE FINAL EVALUATION | 56 Annex II: Findings and recommendation workshop agenda and activities Guiding Group Discussion question, Feb 28th Findings and Recommendation workshop Exercise #1- Fishbone Diagram (or Cause and Effect Diagram)  What is the problem statement/specific event? (fish head)  What are the main categories/factors of the causes of/contributors the problem? (create separate Category lines from the middle line)- can use post-it notes for brainstorming  What are all the possible causes for each problem category- why does this happen? (create separate Cause line on each Category line)- can use post-it notes for brainstorming  Keep asking why to determine the causal relationships, and write them down separately (when the group reports back, they should explain these steps and what they discovered in each) 57 | REINFORCE FINAL EVALUATION USAID.GOV Problem Statement Question for discussion, based on Findings and Conclusions: 1. Q 1 Capacity strengthening o EQ 1 found that clinical skills improved for staff but that there are outstanding capacity and skill development needs for the MOH, MHS and INS. o Topic: Institutional Capacity Strengthening needs for MOH, MHS and INS o Discuss: What did you think of the findings? How does your department define capacity strengthening? How do you address outstanding capacity building needs? o Discuss and Diagram: In what areas are there strengths and weaknesses? What are the causes of this? (strengths can be on the top; weaknesses can be on the bottom of the fish). Are these challenges preventable or not?  In what ways does the capacity of you and your department affect health service delivery? Does capacity in other departments affect you? Does your capacity affect other departments? In what ways?  What other skills need to be built, by who, and why?  What stands in the way of capacity strengthening? What is really needed to reach this in a sustainable way? How can you affect this in positive way? What do you need to do things differently? What, if anything, is outside of your control?  By the end of this exercise, participants will identify i) specific types of capacity improvements and why they are needed ii) what influences their ability to improve certain skill areas iii) how and why their department’s capacity affects health service delivery iv) which factors can be addressed or eliminated and how, and which feel out of their control. 2. Q 2 What affects decision-making? o EQ2 found that poor knowledge management, weak coordination and lack of transparency between entities responsible for service delivery challenges the adoption of best health practices. o Topic: Key bottlenecks that affect decision-maker’s ability to take action and affect improvements in health service delivery. o Discuss: What did you think of the findings? What are the key things that affect you and your colleague’s motivation with your work? Are these challenges preventable or not? o Discuss and Diagram: What do you think the main issues are that affect your ability to do your work- what causes them, and who is affected by this?  Do you have the people, resources and information you need to do your job, why or why not? Have you or your department made any pledge for improvement or change, and in what way?  Do you see any evidence of how this affect health service delivery?  If there were greater collaboration to address bottlenecks, what kind of benefits do you think might occur, and why? Who are the most important collaborators to you?  By the end of this exercise, participants will identify i) specific bottlenecks affecting management, administration, operations within their departments and others ii) what happens as a result of those bottlenecks, internally and at the service level iii) where synergy is needed, and why 3. Q 3: Scalability and Sustainability EQ3 found a lack of evidence of documented lessons learned, sustainability or future action planning for potentially scaling-up activities after Reinforce o Topic: Good health practices that are sustained from Reinforce and could be replicated o Discuss: How would you define a good health practice- at the community level, at MHS level, at MOH/INS level? Who is involved to ensure that these practices are upheld? o Discuss and Diagram: Which of the Reinforce related activities are being sustained and how and which could be replicated to national level/other regions? How would this happen, and who would be involved? What would you need to think about? USAID.GOV REINFORCE FINAL EVALUATION | 58  What practices and activities are institutionalized in/by MOH, MHS and INS already? What is not, and why? How do you know? What will be needed to continue?  What makes certain practices successful and others not? How can you address these successful practices in your planning?  In what ways does data currently play a role in informing your decisions? How can data be used for future planning? How is information shared between departments?  By the end of this exercise, participants will identify i) key healthcare practices that are promising to be scaled, why, how and by who ii) what the channels are that prevent learning from being shared iii) what will ensure that institutional capacity improvements will continue 4. Q 4: Communication, coordination and sharing The evaluation found that activities were less effective due to lack of coordination, such as communication and sharing of information, plans and actions of stakeholders. o Topic: What is needed for good coordination? o Discuss: What did you think of the findings in this context? What examples do you have of good coordination in your work? What examples do you have of poor coordination? How does lack of collaboration, coordination and communication affect your department’s activities? o Discuss and diagram:  What are causes of lack of coordination with other stakeholders?  What is needed for successful coordination?  How can/did you address these successful practices in your planning/management/practice?  What is keeping you from implementing good coordination? What do you need for this implementation?  By the end of this exercise, participants will identify i) the role that communication, learning and collaboration plays in institutionalizing best healthcare practices ii) ways in which their department can operate more efficiently through coordination and knowledge sharing Exercise #2 RACI Matrix and Generating Recommendations This matrix will help us to see what the relationships are between stakeholders, where some bottlenecks might show up and determine who is involved in decision-making. We will not have time to develop a full RACI matrix for each department. Rather, we will focus on the concept of identifying RACI persons for each recommendation to ensure that it is well thought out. 1) Please review the evaluation findings, the fishbone diagram you created and the concept of the RACI Matrix. 2) Based on the information you provide in your diagram, think about all the relevant stakeholders who play a part in this cause and effect diagram (who is involved in decision-making about these topics??) and write them down (name community leadership, GOTL departments, etc.) 3) For each Category, identify which of the stakeholders are involved and in what way- are they Responsible, Accountable, Consulted or Informed (RACI)? 4) For the Causes, identify which factors can be addressed or eliminated and how, and which feel out of their control. 5) Using this information, please develop your top 4 recommendations for your topic area. a. “To address this issue in the future, I would propose the following recommendations…” b. Each recommendation should include a list of RACI people linked to it. The goal is to develop actionable recommendations and enable participants to see the interconnectedness of different stakeholders while ensuring there are no bottlenecks in authority. 6) Once you have developed these Recommendations, please share with the group and explain why you made these recommendations and how you identified the RACI parties. 59 | REINFORCE FINAL EVALUATION USAID.GOV Annex III: Field Plan for Respondents and Sites Visited Key Informant Categories GoTL- Ministry of Health (MOH) • Department for Health Promotion (HP) • Department of MNCH • Cabinet of Policy and Cooperation • Department of Logistics • Department of M&E GoTL- Ministry of Finance (MOF) • Development Partners Management Unit Partners/Donors • USAID/Timor-Leste • United Nations Children’s Fund (UNICEF) • United Nations Population Fund (UNFPA) • World Health Organization • St. John of God • Health Alliance International (HAI) • Marie Stopes International (MSI) • Australian Government Department of Foreign Affairs and Trade (DFAT) • CARE • Psychosocial Recovery & Development in East Timor (PRADET) Reinforce Staff (Dili and Covalima) INS Staff (Dili) Covalima Municipality: Municipality Health Services (MHS), Suai • MHS Management Community Health Centers • CHC Management • General Medical Officers (Doctors, midwives, nurses) • Public Health Technical Officers (Health Promotion, HMIS) Health Post • General Medical Officers (Doctors, midwives, nurses) Community Leader (Village Council) USAID.GOV REINFORCE FINAL EVALUATION | 60 LIST OF HEALTH FACILITIES IN COVALIMA15 N o Facility Name Aldeia Village Administrator post Municipality # of Staff Visited by ET Y/N 1 Suai Referral Hospital (CPS) Tabacolot Debos Suai Villa Covalima - Y 2 Fatumean CHC Klauhalek Belulikleten Fatumea Covalima 8 Y 3 Alastehen HP Fatumea Fatumea Fatumea Covalima 2 Y 4 Nanu HP Nanu Nanu Fatumea Covalima - Not operational 5 Fatululik CHC Lia Nain Taroman Fatululik Covalima 8 Y 6 Beidasi HP Bedasi Fatululik Fatululik Covalima 1 N (not accessible) 7 Fohorem CHC Sadahur Fohorem Fohorem Covalima 8 Y 8 Dato Rua HP Fatulidun Dato Rua Fohorem Covalima 1 Y 9 Dato Tolu HP Fatukkabuarlet en Dato Tolu Fohorem Covalima 2 Y 10 Lactos HP Fatuklaran Lactos Fohorem Covalima· 1 Y 11 Maucatar CHC Duut Belecasak Maucatar Covalima 4 Y 12 Has Ain HP Dais Belekasac Maucatar Covalima 2 N 13 Matai HP Cruz mina rai Matai Maucatar Covalima 4 Y 14 Ogues HP Ogues Ogues Maucatar Covalima 6 Y 15 Tilomar CHC Faularan Maudemo Tilomar Covalima 11 Y 16 Foholulik HP Foholulik Biseuk Tilomar Covalima 1 Y 17 LalawaHP Halemea Lalawa Tilomar Covalima 3 Y 18 Suai Vila CHC Ahinarai Debos Suai Villa Covalima 18 Y 19 Beco HP Beco Beco Suai Villa Covalima 6 Y 20 Gala HP Gala Baco Suai Villa Covalima 2 N 21 LabaraiHP Roek Labarai Suai Villa Covalima 3 N 22 Lakonak HP Lakonak Debos Suai Villa Covalima 3 Y 23 Suailoro HP Loro SuaiLoro SuaiVilla Covalima 4 Y 24 Sanfuk HP Sanfuk Camenasa Suai Villa Covalima 3 Y 25 Zumalai CHC Leogol Zulo Zumalai Covalima 14 Y 26 Beilaco HP Beilako Raemea Zumalai Covalima 1 Y 27 Bulu HP Bulu Lour Zumalai Covalima 2 Y 28 KuluOan HP Kuluoan Tasilin Zumalai Covalima 1 N 29 Kakoli HP Webaba Raemea Zumalai Covalima 1 Y 15 The list of facilities is provided by the MOH and verified by JSI and MHS. Conversations with MHS confirmed those on the list that were not currently staffed or operating at the time of the evaluation. The staff number is not inclusive of administration staff, security, and cleaners. 61 | REINFORCE FINAL EVALUATION USAID.GOV Annex IV: Documents Reviewed Activity design • Reinforce Cooperative Agreement with USAID • Reinforce Performance Monitoring and Evaluation Plan 2015–2019 • Reinforce Organagram • USAID’s Country Development Cooperation Strategy: Institutional and human capacity for development strengthened to improve the lives of Timor-Leste’s citizens. Baseline/prior assessments • Knowledge, Attitudes, and Practices (KAP) Baseline Survey Report, 2016 • Service Availability and Readiness Assessment in Covalima Baseline Report, 2016 Activity implementation • USAID’s Reinforce Basic Health Services Project: Annual Workplans, 2016–2019 • Reinforce training materials, tools, and reports of supportive supervision • Reinforce quarterly and annual monitoring data • Manuals or materials developed with assistance from Reinforce • USAID: Covalima Becoming a Model Municipality • Service Availability and Readiness Assessment Report: Covalima Municipality, 2016 • Knowledge, Attitudes, and Practices: Baseline Assessment Report: Covalima Municipality, July￾August 2016 • Mid-term Technical and Programmatic Review of Reinforce (internal), October 2018 National Policy, strategies and resources • MoH guidelines and HMIS data • MoH Timor-Leste National Health Strategic Plan 2011–2030 • MoH Timor-Leste RMNCAH Strategy 2015–2019 • INS Strategic Plan 2015-2019 and draft 2020–2024 • Democratic Republic of Timor-Leste. Law No. 9/2016, July 2016. Law of Sucos • Timor-Leste Demographic and Health Survey 2009–10 and 2016. Dili, Timor-Leste: NSD/Timor￾Leste and ICF Macro • MoH Timor-Leste Comprehensive Services Package for Primary Health Care (2015) Contextual background • USAID: Country Development Cooperation Strategy (CDCS), 2013–2020 • USAID: A Rapid assessment of Key Areas of the NHSSP for Timor-Leste: Strengths, Challenges, and Opportunities for Moving Forward, April 2016 • Matrix Strategic Plan, INS, 2017 • National Statistics Directorate: NSD/Timor-Leste, Ministry of Finance/Timor-Leste, and ICF Macro. 2010 • General Directorate of Statistics: GDS/Timor-Leste, Ministry of Finance/Timor-Leste and ICF. 2018 • Dawson, A, Howes, T, Gray, N, Kennedy, E 2011, Human resources for health in maternal, neonatal and reproductive health at community level: A profile of Republic of Timor-Leste, Human Resources for Health Knowledge Hub and Burnet Institute, Sydney, Australia • World Bank and OPM (2015) “Health Worker Survey in Timor-Leste,” Dili: World Bank and Oxford: Oxford Policy Management (OPM) USAID.GOV REINFORCE FINAL EVALUATION | 62 • Feigl, Andrea, Chris Lovelace, Nikita Ramchandani. February 2016. A Rapid Assessment of Key Areas of the NHSSP for Timor-Leste: Strengths, Challenges, and Opportunities for Moving Forward. March 2016. Bethesda, MD: Health Finance and Governance Project, Abt Associates, Inc. • World Bank (2014). Timor-Leste Health Resource Tracking Study. Main Findings, Recommendations and Cross-Cutting Issues 63 | REINFORCE FINAL EVALUATION USAID.GOV Annex V: Data Collection Tools KII Tools 1. Informed Consent for KIIs 2. Ministry of Health Officials 3. USAID 4. JSI/Reinforce Staff (Dili – Senior Management) 5. JSI/Reinforce Staff (Tech Officers/Covalima Officers) 6. MHS Officers 7. Other Partners/Implementers (UNFPA, UNICEF, WHO, St. John of God, etc) 8. INS Directors 9. Public Health Services Staff (CHC/HP general medical officers, facility chiefs, doctors) FGD Tool 1. Informed Consent for FGD 2. INS Trainers 3. Public Health Services staff (CHC/HP midwives and nurses) Rapid Assessment Survey 1. Informed Consent for survey 2. Family planning, adolescent health, or MCH clients- community 3. Family planning, adolescent health, or MCH clients- facility INFORMED CONSENT STATEMENT- KII (74 KIIs) Introductions: Thank you for joining us today. First, let me introduce the team. [Introduction] Purpose of the Interview: We work for Social Impact, a US-based research firm, and are working with USAID to evaluate the Reinforce Basic Health Services project (referred to as ‘Reinforce’). We are here now to understand the effects this project had on facility service providers and recipients of services in Covalima by speaking with project management staff, government stakeholders, donors, partners, facility staff and clients. The information we collect will be used to create a report that USAID and the Ministry of Health will use to learn about health care service delivery standards in Covalima and how these standards can be applied in other places in Timor-Leste. We are conducting around 80 interviews. You have been selected to participate because of your experience with/knowledge of the Reinforce activity. Voluntary Participation: I would like to invite you participate in an interview to learn more about your involvement with Reinforce and to get your feedback and perceptions about the current and future capacity of the health system in Covalima and Timor-Leste. We anticipate that this interview will take between 1-1.5 hours. With your permission, we would like to audio record our conversation and take notes to ensure we capture the key points of our discussion accurately; this is voluntary, and you may opt out. If you do not wish to be recorded, we will continue the interview and only take our notes on paper. The recordings and the notes will not be shared with anyone outside the evaluation team – we are conducting many interviews, so this is mainly to help us make sure we do not misunderstand anything you tell us or misrepresent anything in our notes. Your participation in this discussion is voluntary and you have no obligation to participate, nor will you be compensated for your participation. If there are any questions you don’t feel comfortable answering, you USAID.GOV REINFORCE FINAL EVALUATION | 64 do not have to do so. You are welcome to excuse yourself at any time during our conversation without any consequences to you. Data Use & Confidentiality: The information you share with us will be combined with other participant’s responses and this data will be shared in a public report and used for future research purposes by USAID or the Government of Timor-Leste. Everything you tell us will be kept confidential by the evaluation team to the fullest extent permitted by the law in Timor-Leste, the United States, and USAID. None of the information you share with us will be shared with your peers and colleagues or attributed to you, and no information you share with us will be attributed to you by name in our report. We will not include your name, and only include sex, respondent type, and location in a list of all respondents in an annex to our report if you consent to that. If we record this interview, the recordings will be destroyed after we have finished data collection. Risks and Benefits: We do not foresee any risks or direct benefits from your participation in this study. We do not anticipate that you will incur any risk from participating in this interview other than losing about an hour of time you could spend doing other things. Neither this research nor your answers to our questions will in any way determine if a project will be implemented here in the future or affect your ability to access or receive any services, now or in the future. I will provide you with a copy of this consent form to take home. Informed Consent Question: (Please respond verbally): • Do you have any questions for us before we begin? [Yes / No] • Are you willing to participate in this interview? [Yes / No] • Is it okay if we record our conversation? [Okay to record / Not okay to record] If you have any questions or concerns, you may contact the Social Impact Team Leader, Leah Ghoston at lghoston@socialimpact.com, or Leslie Greene Hodel, the chair of a research body designed to protect study participants, at phone number +1-703-465-1884; email address: irb@socialimpact.com. Please initial here _________________________Location__________________Date__________ If you have any questions or concerns, you may contact the Social Impact Team Leader, Leah Ghoston at lghoston@socialimpact.com, or Leslie Greene Hodel, the chair of a research body designed to protect study participants, at phone number +1-703-465-1884; email address: irb@socialimpact.com. lhodel@socialimpact.com. EQ 1. How did Reinforce Activity engage and strengthen the capacity of stakeholders (MOH, INS, municipality health services, health facilities and local leaders) to institutionalize best healthcare service practices in Covalima? 1a. How effective have they been in engaging and strengthening the capacity of stakeholders? Question Stakeholder Defining capacity in the context of Timor-Leste’s health workforce and system Please share with us what you understand the term ‘capacity strengthening’ to mean. What is capacity and in what ways can it be developed or strengthened for better health service delivery? (Probe: human level, organization, network, or system level). How do you know when capacity development or strengthening is necessary? MOH, INS, MHS, CHC/HP What are the benchmarks or measures of performance that tell you if skills, structures or systems for health service delivery is/are being strengthened at varying levels? What do these benchmarks look like in the short term? What about long-term? - What criteria should be used for assessing the effectiveness of capacity building? Why do you think this? - In what ways do you currently measure performance? Are there specific systems in place within your institution? MOH, MHS, INS, CHC/HP 65 | REINFORCE FINAL EVALUATION USAID.GOV - How are differential needs of men, women, youth, and adolescents assessed? Thinking of the different types of capacity you mentioned, how would you describe the existing abilities and skills within the municipal health service level for good healthcare practices at public facilities in Covalima before Reinforce began in 2015? What about within INS? (Probe: service delivery, management, workforce, organization, infrastructure, adherence to quality standards, resource availability). - Were there any kinds of health service delivery improvement activities or investments in municipalities like Covalima? If so, what types of activities were they and how were they initiated? By whom (Were you involved and how?)? What were the most successful approaches? The least? Why? Were there any groups that benefited more than others? Less than? - If there were other initiatives before Reinforce, how would you compare these approaches to those under Reinforce? What approaches worked well in Covalima and what did not? - How were capacity development needs assessed for Covalima municipality at that time? Are they assessed in the same way or differently now? MOH, USAID, MHS, Partners, INS, CHC/HP Implementation of the Reinforce Activity What specific support was provided to MOH/MHS/INS through Reinforce when it started in 2015? How were you initially engaged? Please walk us through the activities you participated in under the activity. - Did you have specific expectations of the activity? If so, what were they? Were those expectations met? Why or why not? - What are the priorities for MHS as part of the broader health system? Where did you see Reinforce fitting into the overall needs of Covalima and the broader health system? - What was the effect of their involvement? MOH, MHS, Partners, INS What specific support was provided to MOH/MHS through Reinforce when it started in 2015? How was the MOH/MHS initially engaged? Please walk us through the activities they participated in under the activity. - Did you have specific expectations of the MOH/MHS? If so, what were they? Were those expectations met? - Where did you see Reinforce fitting into the overall needs of Covalima and the broader health system? - What was the effect of their involvement? JSI When was the readiness assessment (SARA) done at your facility? Were the results shared with you? - How would you describe the capacity to manage RMNCAH at your facility now compared to when the assessment was first done? - Are there areas that improved, declined or stayed the same? (Probe equipment, environment, HR, data). How do you know that there were improvements in these areas? CHC/HP For strengthening in-service provider training, what were the key training activities you conducted or participated in? (needs assessment, curriculum design, piloting, ToT, trainer, FUAT/ supervision, refreshing) - How were training curriculums or courses designed, and how were training priorities identified? - How were participants selected for training? (Probe: gender, position) What was their existing skill or knowledge level prior to the training? - In what ways was did INS gauge the effects of training afterwards? Did INS ever receive feedback from participants on the quality of the training? If so, what was done with this feedback? Are there areas of the training that participants felt were more useful than other areas? Where do you think skills have improved the most? Least? INS, CHC/HP USAID.GOV REINFORCE FINAL EVALUATION | 66 Please describe Reinforce’s approach to follow-up after training (FUAT). How frequently did FUAT occur, and by whom? What was done with FUAT data collected? - Was monitoring data used for any course corrections or adjustments to training? If so, how? What was the outcome? - Please describe the key lessons you have learnt in carrying out supportive supervision visits in CHCs and HPs, managing supervision data and ensuring its use. JSI, INS, MHS, CHC/HP Stakeholder relationships How, if at all, did Reinforce involve stakeholders (INS, partners, facilities, MHS) in designing activities and identifying needs at various levels for Covalima facilities and municipal health services? What about identifying the needs within INS? - What needs were identified and prioritized? How? What worked well in their approach, and was there anything that could have been improved? (Probe: feedback mechanisms, communication, planning) MOH, USAID, JSI, MHS, CHC/HP How would you describe the current relationship between the MOH/MHS and other stakeholders in Covalima (including INS, partners, facilities)? Have these relationships affected the ways in which the best healthcare service practices are taken up in Covalima? In what ways? What linkages have been built, and which need to be further strengthened?What are the strengths and weaknesses between these relationships? MOH, MHS, INS, CHC/HP Please describe the process used to identify implementing partners (such as JNPK-KR and Saint John of God). What considerations went into the selection of the agencies and assigning of tasks to them? What were the expectations of partners selected under the Reinforce activity? - Did their scope change over time? Were those expectations met? Why or why not? - What lessons have you learned? JSI Design and Programming The last activity, ‘Improve’ (HIP) set. the stage for ‘Reinforce’. How was the transition from the last activity (HIP) managed while keeping the programmatic focus maintained? - Were there key prior investments in institutional capacity strengthening from USAID and the MOH from HIP that were expected to be expanded under Reinforce? To what extent did this expansion occur? What were some of the challenges in the expansion and how were they overcome? USAID, JSI How would you describe or identify the main changes, if any, in Reinforce programming around capacity development for key stakeholders (MOH, INS, MHS, facilities) from the inception of the activity to its current state? Were there any adjustments to engagement strategies in Covalima? If so, what prompted these changes, and what was the result? USAID, JSI Overall Effects of the Activity Are there specific population health improvement indicators that have seen significant improvement under Reinforce? (probe: ANC visit, facility delivery, FP uptake) What about specific capacity strengthening indicators? (Probe: adherence to standards, in￾service trainings, clinical certification) - Are there indicators that have not improved as expected? Why do you think that is? MOH, USAID, MHS, INS, CHC/HP Looking at Reinforce’s activities over the last 4 years, what was the most effective and least effective regarding building capacity of key stakeholders to deliver quality health services in Covalima? Why? For: - MOH - INS - MHS - Facilities MOH, USAID, JSI, MHS, INS, CHC/HP 67 | REINFORCE FINAL EVALUATION USAID.GOV Were there any actors who the activity could have involved more strongly, at the national level, municipal or community level? Why weren’t they involved? - Are there any specific stakeholders you believe benefited more than others? Why were they able to benefit more? - Have you observed any differences for male and female recipients of capacity building activities? What differences? Has that changed over time? MOH, USAID, JSI, MHS, INS, CHC/HP EQ 2. What bottlenecks, if any, in the capacity and commitment of stakeholders (MOH, municipality health services, health facilities and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? Quality Assurance How would you assess the activity’s efficacy in promoting, aligning and achieving adherence to national standards and guidelines? What are outstanding challenges to adherence to standards at the facility level? - How would you describe the commitment and motivation of facility staff to adhere to standards and to uphold best practices and new skills? How has that changed over time since beginning of Reinforce to now? - What about challenges to quality assurance or control at the INS and MHS level? Why do you think this is? MOH, USAID, JSI, INS, CHC/HP Are there specific policies and standards that are more difficult to be institutionalized within facilities in Covalima? What about within MHS? INS? If so, why do you think that is? What, if anything, is being done by MOH to address this, and by whom? MOH, JSI, MHS, INS, CHC/HP Risk mitigation How frequently did partners in Covalima meet during Reinforce implementation? What was the communication protocol between stakeholders regarding activity decision￾making? How were risks or challenges identified and rectified, and how was authority defined? What challenges, if any, did you encounter when meeting or communicating with Reinforce?” MOH, JSI, Partners Overall, what assumptions from Reinforce’s Theory of Change held up, and what failed? What could have been done better? (Probe: design and implementation, identification/mgmt. of key assumptions, roles and responsibilities of key stakeholders) (Resources and Inputs in TOC) JSI, USAID To what extent did Reinforce identify and address bottlenecks when engaging with stakeholders to improve health care practices in Covalima? Can you give an example? - How would you assess the activity’s ability to mitigate potential challenges? What about real challenges? What was the outcome? If they were unable to address them, why not? USAID, JSI, MHS, Partners Challenges and Lessons Learned What would you consider to be the biggest bottlenecks or threats to the institutionalization of good health care practices (probe adherence to standards, patient care and responsiveness, workforce, service quality and accessibility, equipment availability and functionality) in Covalima? What causes them? How were they dealt with, and by whom? JSI, MHS, INS, CHC/HP To what extent did Reinforce foresee institutional challenges for Covalima facilities and build in mechanisms to address them in the short term? Long term? What about the MOH? MOH, JSI What, if any, exogenous factors over the last 4 years affected the institutionalization of best healthcare practices in Covalima? (Probe: continuity in the workforce, political shifts, financial adjustments, import/export of equip, etc.) In what ways did this affect activities? What was done to mitigate or address these external risks? MOH, JSI, MHS, Partners, CHC/HP Were there any lessons learned from HIP that USAID anticipated would be addressed in Reinforce? If so, what were the expectations and were they addressed? If some weren’t addressed, why not? USAID EQ 3. What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? 3a. Which part(s) of the model is more promising in terms of sustainability? Why? 3b. What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? USAID.GOV REINFORCE FINAL EVALUATION | 68 Changes to Behaviors and Services Reinforce initiated Vision 2020 as a strategic approach to addressing health service delivery focus areas, namely service and systems development and improved healthy behaviors. - In what ways have the activity’s efforts in Covalima contributed to improved service delivery increased uptake of services at facilities there? (probe uptake of services, quality, accessibility) What evidence or data can you point to that links the Reinforce activity’s efforts directly to improved outcomes? (Probe for specific groups- men, women, adolescents) - What areas of service delivery have not seen improvement? How do you know this? MOH, JSI, MHS, Partners, CHC/HP, INS How has the integrated service delivery model worked? Are there any areas in this strategy that improved more or less? (Probe quality, skill, behaviors, communication, RMNCAH services) - To what extent have any lessons learned emerged from the integration of services? How is this knowledge shared? By whom? MHS, JSI, Partners, INS, CHC/HP What are the most notable ways in which the activity has impacted service delivery? To what extent has the Reinforce activity affected What about effects on key health indicators (Probe: for FP, ANC, MNCH, IMCI, IPC, Adolescent health etc.)? What indicators areas have not seen improvement? How do you know this? - What factors have enabled success in these areas? What challenges were overcome and what remain? (Probe for specific groups- men, women, adolescents) - What evidence or data can you point to that links the Reinforce activity’s efforts directly to improved outcomes? MOH, JSI, MHS, Partners, CHC/HP In what ways have the activity’s efforts improved health behaviors among the populations served by Covalima facilities? What about the behaviors among health staff? - What factors have enabled success in these areas? What challenges were overcome and what remain? (Probe for specific groups- men, women, adolescents) - What evidence or data can you point to that links the Reinforce activity’s efforts directly to improved behavioral outcomes? MOH, JSI, MHS, Partners, CHC/HP Which health promotion and behavior change messages have community members been the most receptive to? Why were they successful? Any that have not been as successful? Why do you think so? Have any of these messages changed over the course of the activity? (Probe: from the initial KAP assessment?) Are there any differences for men, women and adolescents? JSI, MHS, CHC/HP From an institutional perspective, what factors enabled progress within facilities related to: - adherence to standards - staff knowledge, attitudes and practices - building of Covalima health facility systems or structures What factors impeded progress? MOH, JSI, Partners, INS, CHC/HP Data Management/M&E Please share what data management practices are currently used in the MOH. How is HMIS data utilized and shared? Has the documentation of data in Covalima changed at all during the Reinforce activity? Is so, in what ways? Who is responsible for data management moving forward? MOH, MHS, CHC/HP Please share what data management practices are currently used to collect information from health facilities in Covalima. How is this data utilized and shared, and by whom? Has the documentation of data changed at all during the Reinforce activity? If so, in what ways? Who is responsible for data management moving forward? JSI, MHS, CHC/HP 69 | REINFORCE FINAL EVALUATION USAID.GOV What types of service delivery and readiness data were generated from the clinical practice sites? From facilities? How was this data used, and how could it be used in the future? Are there any key lessons learned that would benefit future sites? INS, CHC/HP What is your assessment of the current capacity of Covalima Municipal Health Services, including CHC and HPs, for planning, monitoring and performance management? What about for customer service and collection of client feedback? What are the strengths and weaknesses? MOH, JSI, MHS, Partners, CHC/HP The future of improved services What do you see as the areas of capacity that still need to be built or supported in the next 5 years for the GoTL, MOH to best support health service delivery? How would you prioritize these? (Probe institutional, human, resource, financial, communications). MOH, USAID, JSI, Partners, INS What is the role of local actors and community partners in institutionalizing best healthcare service practices and behaviors? How well have those actors been engaged under Reinforce? (Probe: medical associations, civil society organizations, nonprofits) Are there specific opportunities for their future engagement? To what extent could they be engaged in the future? Why or why not? MHS, Partners, JSI, CHC/HP Which parts of the integrated service delivery model (behaviors, skill, quality, communication, RMNCAH services) are more promising in terms of sustainability? Which parts are least promising? Why? - What are the key things that facilities will need to uphold an integrated service delivery model (Probe: enabling environment, political will, individual motivations, funds, equipment, systems)? - Are these resources currently available, and what is the feasibility that they will be available in the future? MOH, USAID, MHS, JSI, Partners, INS, CHC/HP What parts of the Covalima pilot model are most promising and likely to continue in the future? What will be necessary to ensure longevity? What might impede longevity? Why? (Probe: behaviors, funds, political will, geographic considerations, data mgmt., infrastructure) - Are there specific challenges for MHS and facilities? If so, what might they be? What about gaps for CHCs compared to HPs? How might they be overcome? MOH, USAID, JSI, MHS, INS, CHC/HP USAID considerations for sustainability The Journey to Self-Reliance roadmap for Timor-Leste identifies high capacity in civil society but low government and citizen capacity. It also identifies high commitment in governance, social groups and economic policy. - Are there any specific considerations to gauge government capacity for institutionalization of best healthcare service practices in an emerging democracy like Timor-Leste? What might the challenges be? What about gauging commitment? - How would you assess stakeholder readiness to continue best health care practices after the completion of Reinforce? Are there any stakeholders, both in Covalima and centrally in Dili, that have taken and/or may take greater initiative than others? Why those specifically? What may impact their ability to take action? USAID, JSI To what extent have USAID and the GoTL/MOH assessed reviewed and discussed the metrics in the Journey to Self-Reliance? Is there consensus and understanding of the definitions and assessment criteria for capacity? Commitment? Why or why not? - Are there specific metrics that can identify improvements made to capacity in Covalima over time? What might they be? USAID, MOH, JSI, MHS EQ 4. What are the capacity and commitment of the GoTL, the MoH and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? Performance of systems and structures In what ways does the MOH currently emphasize and monitor organizational performance at the municipal level? Nationwide? Is there a system to measure performance after Reinforce has ended? What about INS? Who is responsible? MOH, USAID, JSI, Partners, INS USAID.GOV REINFORCE FINAL EVALUATION | 70 - Does the MOH or INS currently document promising practices or failures emerging from Covalima under Reinforce? If so, what is currently being done with this information? How might this information be used for gauging feasibility to scale up? How is the Covalima model applied onto the existing nationwide health system? In what ways can lessons learned from capacity development within the MOH, including INS and MHS, affect the broader health system of Timor-Leste? - In what areas of the system would you expect changes to occur? What might those changes be? MOH, USAID, MHS, INS The Reinforce activity has incubated several evidence-based models. What process does the INS (and other stakeholders) use to assess the scalability of each? Which are most promising, and why? INS In 2016 USAID issued a rapid SWOT assessment of the health system in Timor-Leste. What key areas came out of this assessment at that time as priorities for capacity building? Today, which of those areas in Covalima have improved the most? The least? Stayed the same? Why? - What do you think the key lessons learned are that should be applied to a scaled-up activity in other municipalities? USAID Commitment and responsibilities of key stakeholders What would the first step be for the GoTL to take the Covalima model to other municipalities? What would the timeline look like? Has any progress been made on behalf of the GoTL and the MOH to facilitate this process? MOH, USAID, JSI, Partners, INS In what ways do you think the MOH adapts to new learning and a changing context? Please provide examples. What might some challenges be in adapting the Covalima model to a new context outside of Covalima? What will the MOH need to be successful? INS? - Have any key actors shown evidence of a drive for continuous improvement? MOH, USAID, INS Who are the main stakeholders responsible for enforcing best healthcare practices throughout Timor-Leste, outside of the MOH? How would you assess stakeholder readiness to continue the institutionalization of best health care practices after the completion of Reinforce in Covalima? What about at the national level? - Are there any stakeholders that may take greater initiative than others? What challenges may impact their ability to act? - What challenges exists, if any, in securing commitment from stakeholders like MOH, MHS and facilities to institutionalize best practices in healthcare service delivery? How can these challenges be mitigated? MOH, JSI, MHS, Partners Considerations for replicability What are the resources necessary for clinical practice sites to continue to serve as training hubs after Reinforce has ended? What about for expansion of sites outside of Covalima? (Probe: HR, equipment, financing) What are the factors that could enable or constrain INS ability to expand? - What is the MOH ability and plan to secure the necessary resources? MOH, JSI, INS, MHS What would you do differently in new municipalities? What key differences, if any, do you find in the operating environments of CHCs and HPs in Covalima vs those in other municipalities? How might those differences need to be taken into consideration if the Covalima model is to be replicated? MOH, MHS, INS, JSI Future planning Looking forward, how do you think access and quality in health services can be improved – is there anything new that could be tried? (Probe for men, women, adolescents, children) MOH, JSI, Partners, INS, CHC/HP What are the major priorities for your facility in addressing and sustaining good health service quality for RMNCAH and FP for the next 5-10 years? How do you envision these will be addressed? CHC/HP 71 | REINFORCE FINAL EVALUATION USAID.GOV What are the development objectives for the next 5-10 years in Covalima? What about other municipalities? Who are the key actors needing to collaborate to meet these objectives? - How would you envision this collaboration takes place? - In what ways can the activity’s current approach help those future efforts? What changes are needed? MOH, USAID What are USAID’s current and future priorities for RMCAH and FP in Timor-Leste? To what extent do the activity’s design and approaches for capacity development reflect or align with these priorities? Were there areas where better alignment was needed, and what changes were made? Are there any remaining areas with less than optimal alignment? USAID EQ 5. Beyond Covalima, how has the Reinforce Activity strengthened the capacity of INS? 5a. Where are the gaps, if any, in INS’ capacity and commitment to conduct and to manage in- service training and continued education of health professionals in Timor-Leste? 5b. How has Reinforce mitigated these gaps 5c. What, if any, are capacity gaps that remain? How would you describe the institutional skills and capacity within INS prior to the start of Reinforce in 2015? How did INS identify priority areas for investment and organizational development at the start of the activity? (Probe management, leadership, resources, human resources, equipment, curriculum development) [EQ1] MOH, USAID, JSI, INS, Partners Effect of Reinforce on INS INS’ Five-Year Strategic Plan 2015-2019 was to prepare the conditions for the INS to become a center of excellence for training, research, and institutional management in Timor-Leste by 2019. How would you describe the scope and functioning of the INS in supporting public health facilities today compared to at the start of the activity? (Probe: training, research, institutional management) - What factors have enabled effective and efficient functioning of the Institute? USAID, MOH, JSI, INS, MHS, Partners, What organizational or institutional gaps in the INS were identified during the Reinforce activity, and how were they addressed? (Probe: skills, relationships, systems, structures) Were corrective measures taken? How effective were they? INS, JSI, USAID, Partners, INS What would you consider as the strongest institutional skills and competencies that INS has built from their involvement with Reinforce? How might these skills be best applied in the agency’s work nationwide? What would you consider to be the least beneficial to INS? (Probe: standard operating procedures, training skill development, accreditation, strategic planning) MOH, USAID, JSI, MHS, Partners, INS, CHC/HP What will INS need in the future? How would you prioritize any areas of capacity (Probe: skills, relationships, systems, structures) that still need to be built or supported in the next 5 years for the INS to best support provider training and improved health service delivery? In Covalima? In areas outside of Covalima? - In what ways can lessons learned from the involvement in Reinforce apply to and affect the INS’s role in the health system in the future? MOH, INS, JSI, MHS, Partners, CHC/HP Regarding skill development and training, what will the agency need to keep up their in-service training skills (Probe: resources, monitoring, personnel, etc.) - How will continued skill development activities be initiated? Monitored? Assessed? - How will continued performance improvements for trainers and trainees be monitored? Assessed? JSI, INS, MHS Moving forward, is there a logical framework for how investing in INS can contribute to long-term improvements in continuing education for health professionals in Timor￾Leste? What does this look like? What are the long-term outcomes envisioned for continuing education? - What are the expectations of INS from the MOH, MHS and health facilities? - Are there key health indicators to be met? MOH, JSI, INS, MHS, Partners USAID.GOV REINFORCE FINAL EVALUATION | 72 What are the key roles of INS within the broader health system of Timor￾Leste? Where do they have the most added value, and why? Is there a potential for them to do more? Why or why not? (Probe: training, research, institutional management, collaborations) USAID, MOH, INS INFORMED CONSENT STATEMENT- FGD (12 FGDs) Introductions: Thank you for joining us today. First, let me introduce the team. [Introduction] Purpose of the FGD: We work for Social Impact, a US-based research firm, and are working with USAID to evaluate the Reinforce Basic Health Services project (referred to as ‘Reinforce’). We are here now to understand the effects this project had on facility service providers and recipients of services in Covalima by speaking with project management staff, government stakeholders, donors, partners, facility staff and clients. We are conducting 16 FGDs with those who have provided or received training or participated in other activities under Reinforce. The information we collect will be used to create a report that USAID and the Ministry of Health will use to learn about health care service delivery standards in Covalima and how these standards can be applied in other places in Timor-Leste. Voluntary Participation: I’d like to invite you participate in a focus group discussion [group interview] to learn more about your involvement with Reinforce and to get your feedback and perceptions about the current and future capacity of the health system in Covalima and Timor-Leste. We anticipate that this discussion will take about an hour of your time. A team member will be translating the responses in this discussion to ensure that we can capture all of the information you are sharing with us and that you are comfortable asking us any questions you have. Additionally, with your permission, we would like to audio record our discussion and take notes to ensure we capture the key points of our conversation accurately; this is voluntary, and you may opt out. If anyone in the group is not comfortable being recorded, we will continue the discussion and only record our notes on paper. The recordings and the notes will not be shared with anyone outside the evaluation team. We are conducting many group discussions, so this is mainly to help us make sure we do not misunderstand anything you tell us or misrepresent anything in our notes. Your participation in this discussion is voluntary and you have no obligation to participate. You will not be compensated in any way for your participation. If there are any questions you do not feel comfortable answering, you do not have to do so. You are welcome to excuse yourself at any time during this discussion without any consequences to you. Data Use & Confidentiality: The information you share with us will be combined with other participant’s responses so that we can identify key points and themes. The combined data will be shared with USAID for future public research purposes and summarized in our report. Everything you tell us will be kept confidential by the evaluation team to the fullest extent permitted by the law in Timor-Leste, the United States, and USAID. We will take every precaution to maintain the confidentiality of this interview, but the nature of group interviews/discussions prevents us from guaranteeing complete confidentiality. We would like to remind participants to respect the privacy of your fellow participants and agree not to share what is discussed in the discussion with others. The team will not share any of the information you share with us with your peers and colleagues. We will assign you a number for attendance so that we know how many men and women attended. We will not confirm your participation in this group discussion with anyone outside of this room. If we record the discussion, the recordings will be stored safely and destroyed after we have completed the report. Risks and Benefits: We do not foresee any risks or direct benefits from your participation in this study. We do not anticipate that you will incur any risk from participating in this interview other than losing an hour of time you could spend doing other things. Neither this research nor your answers to our questions 73 | REINFORCE FINAL EVALUATION USAID.GOV will in any way determine if a project will be implemented here in the future or affect your ability to access or receive any services, now or in the future. Informed Consent Question: • Do you have any questions for us before we begin? [Yes / No] • Are you willing to participate in this interview? [Yes / No] • Is it okay if we record this discussion? [Okay to record / Not okay to record] Signature of Evaluation Team Member/FGD facilitator_______________________________ If you have any questions or concerns, you may contact the Social Impact Team Leader, Leah Ghoston at lghoston@socialimpact.com, or Leslie Greene Hodel, the chair of a research body designed to protect study participants, at phone number +1-703-465-1884; email address: irb@socialimpact.com. 1. FGD/Group Interviews- HP/CHC STAFF (14 FGDs) Number of participants: _______ Sex: ___m/ ____f Health Facility Name:______________ Facility type:__CHC / HP_ Location:______________Date:_______ Questions for staff group interviews (Nurses and Midwives) EQ1.) How did Reinforce Activity engage and strengthen the capacity of stakeholders (MoH, INS, municipality health services, health facilities and local leaders) to institutionalize best healthcare service practices in Covalima? 1a. How effective have they been in engaging and strengthening the capacity of stakeholders? 1. Please tell us how you came to know about the JSI/Reinforce activity and how you were involved. What specific training or skill-building efforts of the activity have you participated in? In what ways have they benefited your work? 2. How were you selected for training? Had you been a part of any other skill development trainings before? If so, how would you compare the trainings? (Probe for: curriculum, environment, trainers, time). 3. After training, what kinds of monitoring or follow up were provided to you? How useful was this follow up? Was there any data collected from you or your facility, and do you know how this data was used? 4. What are the key parts of the training that benefited you most? Are there areas of the training that were more useful than other areas? (Probe adherence to standards, specific practices for family planning, RMNCH, adolescent health). Are there any areas that were not very useful? (What opportunities are provided for trainees to practice new skills? 5. What skills do you think still need improvement? How do you or your facilities currently measure improvements? 6. What would you consider best health care practices to be? Are these practices different for men, women, or adolescents in Covalima? Are they different for CHCs vs HPs? 7. Thinking back to the areas in which you were trained, are there any practices/competencies that will be more difficult for you to implement or maintain within your facilities? What about with your patients? Why do you think this? EQ3.) What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? 3a. Which part(s) of the model is more promising in terms of sustainability? Why? 3b. What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? USAID.GOV REINFORCE FINAL EVALUATION | 74 1. Do you think there is an increase, decrease, or no change in people utilizing services for reproductive, maternal, child and adolescent health and family planning services in the past 2 years? Have you seen any differences in service uptake for men, women, adolescents or children? Why do you think this? 2. Which areas have seen the most increase institutional skill improvements? Why do you think this? (Probe: pre and antenatal care, newborn care, IMCI, immunization, family planning, emergency obstetrics, data mgmt). The least? 3. Are there any areas that have not changed for the better or worse, and have stayed the same? Why do you think this? 4. Which health promotion and behavior change messages have community members been the most receptive to? Are there any differences in men, women, adolescent responsiveness? How do you know that they have been receptive? a. Any messages that have not been as successful? Why do you think so? b. Have any of these messages changed over the course of the activity? 5. In what ways has the training from JSI/Reinforce affected your competencies with addressing adolescent health specifically? To what extent to you think adolescent health needs are being met? Why or why not? What more can be done to improve services to adolescents and to encourage them to use services available? 6. Did you participate in the readiness assessment (SARA) done at your facility? Do you know what the results were? How would you describe any changes to your facility since the assessment was done in 2016? (Probe: equipment, staffing, basic amenities, infrastructure, communications, inpatient needs, transportation and power supply, service availability) 7. What are the key things that facilities will need to continue to provide quality services? (Probe: finances, behavior change, HR, environment, supplies & equipment). Are these things currently available? What is the feasibility that they will be available in the future? (Probe responsible parties, ongoing mentorship/support, additional training) 2. FGD/Group Interview/ INS Trainers (2 FGDs) Number of participants: _______ Sex: ___m/ ____f INS Clinical Practice Facility Name:____________________ Location:________________ Date:________ EQ1) How did Reinforce Activity engage and strengthen the capacity of stakeholders (MoH, INS, municipality health services, health facilities and local leaders) to institutionalize best healthcare service practices in Covalima? 1a. How effective have they been in engaging and strengthening the capacity of stakeholders? 1. As a trainer, how were you selected? What are your specific responsibilities- before, during and after training? 1.How were you selected to be a trainer? Had you been a part of any other trainings like this before? If so, how would you compare the trainings? (Probe for: curriculum, environment, trainees, time) 2. How were RMNCAH/FP training curriculums or courses designed, and how were training priorities identified? Were you involved in these processes? 3. How were facility participants selected for training? What about Trainer of Trainers? Was there any different selection for male and female participants? What was their existing skill or knowledge level prior to the training? In what ways was did INS gauge the effects of training afterwards? 75 | REINFORCE FINAL EVALUATION USAID.GOV 4. Facility staff can practice their skills at clinical training practice sites in both Dili and Covalima. What do you think is working well with this practice site model? Is there anything that needs to be improved? 5. What skills do you think still need improvement for facility staff? Are there any difference in men and women trainees? How do you currently measure improvements? 6. As a trainer, what learnings stuck with you the most from your training? How do you envision utilizing these learnings in your day-to day work? 7. Are there additional skills that you think INS trainers should be competent in to provide support to service providers? Other trainers? What additional competencies do you think would be most important? Where might you get these skills from? 8. What would you consider best health care practices to be? Are these practices different for men, women, or adolescents in Covalima? What, if any, are the differences for CHCs vs HPs? EQ 3) What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? 3a. Which part(s) of the model is more promising in terms of sustainability? Why? 3b. What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? 1. After training, what kinds of monitoring or follow up were conducted? Who was responsible for this? How useful was this follow up? Was there any data collected from trainees or their facilities, and how was this data used by INS? 1. What has worked well for you as a trainer with providing follow up after training? Has anything not worked well or been more difficult? Why? 2. Through this follow-up, have you identified any areas that are more difficult for trainees to uphold within their facilities? Why do you think this? (probe adherence to standards, data mgmt., emergency obstetric and newborn care, adolescent health, etc.) 3. What kinds of quality improvement exercises have you participated in? How effective do you feel these exercises are? Why or why not? 2. Has the training initiated under Reinforce affected your competencies with addressing adolescent health specifically? Why or why not? To what extent do you think adolescent health needs are being met? (Probe male and females) Why do you think this? What more can be done to improve services to adolescents and to encourage them to use services available? 3. What are the key things that facilities will need to continue to provide quality services? (Probe: finances, behavior change, HR, environment, supplies & equipment). Are these things currently available, and what is the feasibility that they will be available in the future? What do you think will be needed to continue to provide services for men, women and adolescents? What role do you think INS has? 4. What are the continued expectations of INS trainers after Reinforce activity ends? Do you plan to continue training? What about the clinical practice sites? What happens to this training model? INFORMED CONSENT STATEMENT- RAPID SURVEY (Facility or Community – 31 Rapid Surveys) Invitation to participate (to be read to service recipient by health facility or community activity staff): We would like to know if you would be interested in participating in a survey today while you wait. There is a research team who wants to speak with some recipients of services here to learn more about people’s experience receiving health services in this area. This is entirely optional and voluntary. They are independent and do not work for this facility or for the government/MOH. It should take about 10 to 15 minutes and will not affect your place in the queue. You can also speak with them after you USAID.GOV REINFORCE FINAL EVALUATION | 76 have been seen. They will speak with you privately and the staff here and other patients will not know your responses to them. If you are interested and have the time, would you like to participate? [Yes/no] Introductions: Thank you for joining us today. First, let me introduce the team. [Introduction] Purpose of the Survey: We work for Social Impact, a US-based research firm, and are working with USAID to evaluate the Reinforce Basic Health Services project (referred to as ‘Reinforce’). We are here now to understand the effects this project had on facility service providers and recipients of services in Covalima Municipality by speaking with project management staff, MoH stakeholders, facility staff and clients like you. We would like to ask you a few questions about your opinions about the availability and quality of services offered to you and your family from this facility/community activity [Adults/guardians can answer on behalf of minors]. We do not represent the facility or the staff here, but we are independent researchers. We are conducting up to 70 surveys with men and women at each CHC, some health posts, and at community health outreach events in Covalima. The information we collect will be used to create a report for USAID and the Ministry of Health to learn about health care service delivery standards in Covalima and how these standards can be applied in other places in Timor-Leste. Voluntary Participation: Your participation in this survey is voluntary and you have no obligation to participate. If there are any questions you do not feel comfortable answering, you do not have to do so. You are welcome to excuse yourself at any time during this survey without any consequences to you. This survey will take about 10-15 minutes of your time. My team member is here translating to ensure you understand the questions we are asking, and that you are comfortable asking us any questions you have. Risks and Benefits: We do not foresee any risks or direct benefits from your participation in this study, and no compensation will be provided. Neither this research nor your answers to our questions will in any way determine if a project will be implemented here in the future. Your participation will not affect your ability to access or receive any services at this facility/community activity or others, now or in the future. Confidentiality: Your responses will be kept confidential by the evaluation team to the fullest extent permitted by the law in Timor-Leste, the United States, and USAID. The evaluation team will assign this survey a number only, and your name will not be recorded or included with this survey. This is an independent survey and will not be shared with anyone at the facility [or associated with this community health activity]. The data captured in this survey will be combined with other respondents’ feedback, shared with USAID and the Government of Timor-Leste for future public research purposes, and summarized in our report. Informed Consent Question: • Do you have any questions for us before we begin? [Yes / No] • Are you willing to participate in this survey? [Yes / No] Signature of Evaluation Team Member/Surveyor_______________________________ If you have any questions or concerns, you may contact the Social Impact Team Leader, Leah Ghoston at lghoston@socialimpact.com, or Leslie Greene Hodel, the chair of a research body designed to protect study participants, at phone number +1-703-465-1884; email address: irb@socialimpact.com. 1. Rapid Survey Assessment- Community, SISCa or non-facility activity Survey Respondent ID:_________ Activity name:___________Location:__________________Date:_______ Sex: M / F Age Range: (18-25) (26-30) (31-40) (41-50) (51+) What services are you expecting to receive/did you receive today? (Check all that apply) o Family Planning o Child immunization o Antenatal care (ANC) o Adolescent health checkup o Newborn or child health checkup 77 | REINFORCE FINAL EVALUATION USAID.GOV o Nutrition assistance o Treatment for illness o Other _____________________________________ How did you hear about the services offered at this community activity? (Check all that apply) o Friend o Family member o Previous community outreach (ex. SISCa, mobile unit) o Health facility o Media (ex. newspaper, radio) o Other______________________________________ Why did you choose to come to this activity? (Check all that apply) o This activity is near to where I live o This activity was recommended to me o I have been to this activity before and found it useful o I don’t know o Other______________________________________ Please state the level that you agree with the following statements: o I receive attention from staff at this activity when I ask to speak with them. Always/sometimes/never/don’t know or not applicable o The activity staff explain the treatment they are providing to me or my family. Always/Sometimes/Never/don’t know or not applicable o When I leave this activity, I feel like all of my questions have been answered. Always/Sometimes/Never/don’t know or not applicable o When I leave this activity, I feel like I have been treated with respect. Always/Sometimes/Never/don’t know or not applicable o When I come to this activity the staff are able to provide me with the services I am expecting. Always/Sometimes/Never/don’t know or not applicable o The staff at this activity have the supplies or medical equipment they need to offer the services I am expecting. Always/Sometimes/Never/don’t know or not applicable o The activity has enough staff to manage all of the patients who come here. Always/Sometimes/Never/don’t know or not applicable o The staff provides me with health education materials or information to take home to help me improve my health. Always/Sometimes/Never/don’t know or not applicable o If I am given health education materials, they are useful for me and my family. Always/Sometimes/Never/don’t know or not applicable USAID.GOV REINFORCE FINAL EVALUATION | 78 o I am able to coordinate transportation to bring me to the activity when I need to come. Always/Sometimes/Never/don’t know or not applicable o Was there a time when you did not come to this activity to get services when you needed them? If so, why not? (select all that apply) o Cost o Transportation o The service I needed was not offered o The medicine I needed was not available o I did not have the time o I did not like how I was treated o Other____________________________________________________ o There was no time that I elected not to come to the activity for any reason o Do you have friends and family who do not come to receive these services, even when they need them? If so, why not? (select all that apply) o Cost o Transportation o The service they needed was not offered o The medicine they needed was not available o They did not have the time o They did not like how they were treated o Other____________________________________________________ o My family and friends have never elected not to come to receive these services o I don’t know 13. Overall, how would you describe the quality of the services provided to you at this activity in the time you have been attending? o I think the activity’s quality of services has improved greatly in the time I have been attending o I think the activity’s quality of services has improved slightly in the time I have been attending o I think the activity’s quality of services have not changed in the time I have been attending o I think the activity’s quality of services has declined slightly in the time I have been attending o I think the activity’s quality of services has declined greatly in the time I have been attending 2. Rapid Survey Assessment Form- Facility Facility Survey Respondent ID:___________ Health Facility name: _______________________ Facility Type: _CHC / HP Location:____________________ Sub-District: _____________ Date:_____________ Sex: M / F Age Range: (18-25) (26-30) (31-40) (41-50) (51+) How long have you been coming to this facility? o Less than 1 year o 1-2 years o 3-4 years o 5 years or more 79 | REINFORCE FINAL EVALUATION USAID.GOV What services are you expecting to/did you receive today? (Check all that apply) o Family Planning o Child immunization o Antenatal care (ANC) o Adolescent health checkup o Newborn or child health checkup o Treatment of an illness o Other _____________________________________ How did you hear about the services offered at this facility? (Check all that apply) o Friend o Family member o Community outreach (ex. SISCa, mobile unit) o Media (ex. newspaper, radio) o Other ____________________________________ Why did you choose to come to this facility? (Check all that apply) o This facility is nearest to where I live o This facility was recommended to me o This facility provides good care and treatment o This facility offers the services that I need today o This facility is low in cost o Other ______________________________________ o I don’t know If you have been coming to this facility for 2 years or more, how would you describe the availability of the services offered and available here? o The number of services that are available have improved greatly in the last 2 years o The number of services that are available have improved slightly o The number of services that are available has not changed in the last 2 years o The number of services that are available has declined slightly in the last 2 years o The number of services that are available has declined greatly in the last 2 years o I don’t know Can you estimate how long it takes for you to be seen by staff, from the moment you arrive to the moment you see the medical professional? o I usually wait less than 30 minutes before I am seen by a care provider o I usually wait between 30 and 60 minutes (one hour) before I am seen by a care provider o I usually wait more than one hour before I am seen by a care provider Please state the level that you agree with the following statements: 1.) I receive attention from staff at this facility when I ask to speak with them. Always/sometimes/never/don’t know or not applicable 2.) The facility staff explain the treatment they are providing to me or my family. Always/Sometimes/Never/don’t know or not applicable 3.) When I leave this facility, I feel like all of my questions have been answered. Always/Sometimes/Never/don’t know or not applicable 4.) When I leave this facility, I feel like I have been treated with respect. Always/Sometimes/Never/don’t know or not applicable USAID.GOV REINFORCE FINAL EVALUATION | 80 5.) When I come to this facility the staff are able to provide me with the services I am expecting. Always/Sometimes/Never/don’t know or not applicable 6.) This facility has the equipment it needs to offer the services I am expecting. Always/Sometimes/Never/don’t know or not applicable 7.) This facility has the medicine and drugs that are required by my doctor. Always/Sometimes/Never/don’t know or not applicable 8.) The facility has enough staff to manage all of the patients who come here. Always/Sometimes/Never/don’t know or not applicable 9.) The staff provides me with health education materials or information to take home help to me improve my health. Always/Sometimes/Never/don’t know or not applicable 10.) If I am given health education materials, they are useful for me and my family. Always/Sometimes/Never/don’t know or not applicable 11.) I am able to coordinate transportation to bring me to the facility when I need to come. Always/Sometimes/Never/don’t know or not applicable 12.) Was there a time when you did not come to this activity to get services when you needed them? If so, why not? (select all that apply) o Cost o Transportation o The service I needed was not offered o The medicine I needed was not available o I did not have the time o I did not like how I was treated o Other____________________________________________________ o There was no time that I elected not to come to the activity for any reason 13.) Do you have friends and family who do not come to receive these services, even when they need them? If so, why not? (select all that apply) o Cost o Transportation o The service they needed was not offered o The medicine they needed was not available o They did not have the time o They did not like how they were treated o Other____________________________________________________ o My family and friends have never elected not to come to receive these services o I don’t know 14.) Overall, how would you describe the quality of the services provided to you at this facility in the time you have been a client here? o I think the facility’s quality of services has improved greatly in my time as a client o I think the facility’s quality of services has improved slightly in my time as a client o I think the facility’s quality of services have not changed in my time as a client o I think the facility’s quality of services has declined slightly in my time as a client o I think the facility’s quality of services has declined greatly in my time as a client 81 | REINFORCE FINAL EVALUATION USAID.GOV Annex IIII: Evaluation Design Matrix Evaluation Question Data Source Data Collection Method Data Analysis Method 1. How did Reinforce Activity engage and strengthen the capacity of stakeholders (MoH, INS, municipality health services, health facilities and local leaders) to institutionalize best healthcare service practices in Covalima? 1a. How effective have they been in engaging and strengthening the capacity of stakeholders? • Review and analysis of USAID-provided documents and secondary data sources • KIIs with MoH, INS, MHS and health facilities staff, other donors and local leaders; • FGDs with health facility staff and INS trainers • Rapid assessment survey with FP and RMCH clients • Desk review • KIIs • FGDs • Rapid assessment survey • Tally sheets/qualitative coding to track KII data and do thematic analyses • Gender and comparative analysis • Analysis of survey data 2. What bottlenecks, if any, in the capacity and commitment of stakeholders (MoH, municipality health services, health facilities and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? • Review and analysis of USAID-provided documents and secondary data sources • KIIs with MoH, INS, MHS and health facilities staff, other donors and local leaders; • FGDs with health facility staff and INS trainers • Rapid assessment survey with FP and RMCH clients • KIIs • FGDs • Rapid assessment survey • Desk review • Tally sheets/qualitative coding to track KII & FGD data and do thematic analyses • Gender and comparative analysis • Analysis of survey data 3. What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? 3a. Which part(s) of the model is more promising in terms of sustainability? Why? 3b. What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? • Review and analysis of USAID-provided documents and secondary data sources • KIIs with MoH, INS, MHS and health facilities staff, other donors and local leaders; • FGDs with health facility staff and INS trainers • Rapid assessment survey with FP and RMCH clients • KIIs • FGDs • Rapid assessment survey • Desk review • Tally sheets/qualitative coding to track KII & FGD data and do thematic analyses • Gender and comparative analysis • Analysis of survey data USAID.GOV REINFORCE FINAL EVALUATION | 82 4. What are the capacity and commitment of the GoTL, the MoH and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? • KIIs with USAID, other partners MoH, and INS • FGDs with INS trainers, Review and analysis of USAID￾provided documents • Review and analysis of USAID-provided documents • KIIs • FGDs • Desk Review • Tally sheets/qualitative coding to track KII data and do thematic analyses 5. Beyond Covalima, how has the Reinforce Activity strengthened the capacity of INS? 5a. Where are the gaps, if any, in INS’ capacity and commitment to conduct and to manage in- service training and continued education of health professionals in Timor-Leste? 5b. How has Reinforce mitigated these gaps? 5c. What, if any, are capacity gaps that remain? • KIIs with USAID, other partners, MoH, and INS • FGDs with health facility staff and INS trainers • Review and analysis of USAID-provided documents • KIIs • FGDs • Desk Review • Tally sheets/qualitative coding to track KII & FGD data and do thematic analyses 83 | REINFORCE FINAL EVALUATION USAID.GOV Annex VII: Scope of Work TITLE OF ACTIVITY Evaluation of Reinforce Basic Health Service Activity. PURPOSE OF THE EVALUATION This evaluation serves as USAID/Timor-Leste’s required Whole of Project Evaluation as per ADS 201. The external evaluation must be completed during the final year of the USAID’s Reinforce Basic Health Service activity’s period of performance, in order to complement all on-going and closing-out Monitoring Evaluation and Learning (MEL) efforts conducted by the implementing partner in accordance to their Activity MEL Plan (AMELP). The findings of this evaluation will be used by USAID, the Government of Timor-Leste (GoTL) including the Ministry of Health (MOH), Covalima Municipal Health Services (MHS), and the Instituto Nacional de Saúde (National Health Institute, INS), and John Snow Inc. Research & Training Institute (the implementing partner) to gain evidence on capacity and commitment of the GoTL to introduce the Reinforce’s approach elsewhere in Timor-Leste. In addition, the evaluation will also inform the GoTL and USAID on how to further strengthen the healthcare service system in Covalima.: • Capture implementation details that were not included in the AMELP, • Learn lessons about the Covalima model implementation, to inform the replication and adaptation of the model in other municipalities in Timor-Leste or in other countries with similar development objectives, • Assess local stakeholders’ capacity and commitment necessary for the sustainability of the model in Covalima • Assess local stakeholders’ capacity and commitment for its replication to other municipalities in Timor-Leste, • Capture lessons learned about USAID’s past and ongoing support to the INS. As the long term goal of USAID’s assistance in Timor-Leste is to support the country on its journey to self-reliance, USAID is most interested in how its assistance is moving the needle in the capacity and commitment of the GoTL, MOH, MHS, the Covalima MHS, and INS to improve the quality of health services delivered in health facilities. SUMMARY OF INFORMATION Activity Title Reinforce Basic Health Service Cooperative Agreement # AID-472-A-16-00001 Life of Activity December 23, 2015 – December 22, 2020 Implementing Partner JSI Research and Training Institute, Inc. (JSI) Total Ceiling of the Evaluated Activity $ 8,480,213 USAID.GOV REINFORCE FINAL EVALUATION | 84 Agreement Officer’s Representative (AOR) Teodulo Ximenes Active Geographic Regions Dili and Covalima Municipality BACKGROUND DESCRIPTION OF THE PROBLEM & CONTEXT Health Sector Context Timor-Leste has created a sound 20-year national health plan, the National Health Sector Strategic Plan (NHSSP) 2011–2030. The NHSSP identifies four health systems priorities: 1) Provision of Health Services, 2) Investment in Human Capital, 3) Infrastructure Investment, and 4) Health Management and Administration. Since 2002, the health system has made progress in establishing basic services but still struggles to provide vital maternal and child health services, family planning, immunization coverage and emergency care in the more remote and mountainous areas where most of the population resides. The table below shows selected indicators from the last two Timor-Leste Demographic and Health Surveys (TLDHS). Indicators 2009–2010 TLDHS 2016 TLDHS Maternal Mortality Ratio 557/100,000 live births 195/100,000 live births Under-5 Mortality Rate 64/1,000 live births 41/1,000 live births Neonatal Mortality Rate 22/1,000 live births 19/1,000 live births Total Fertility Rate 5.7 4.2 Percent of married women age 15–49 using any modern method of family planning 21 24 Percentage of Deliveries by a Skilled Birth Attendant 30 57 Percentage of Under-5 Wasting 19 24 Percentage of Under-5 Stunted 58 46 Percent of children age 12–23 months who have received all basic vaccinations 53 49 Primary health services are provided through the Municipality Health Service (MHS) structure, with Community Health Centers (CHCs), Health Posts (HPs) and outreach activities servicing geographically defined populations within the framework of the Basic Service Package while incorporating an integrated community health service or SISCa. The community based activities consist of Servisu Integradu Saúde 85 | REINFORCE FINAL EVALUATION USAID.GOV Communitaria (SISCa: Integrated Community Health Services) in all villages, mobile services conducted at other sites e.g. schools, markets, community structures, and “mop up” services regularly conducted according to programmatic needs (NHSSP 2011–2030). Part of the primary health care strategy is to staff every health post per village with seven health care providers, including a doctor, nurse, midwife, lab technician, pharmacist, technician, and ancillary workers. Health Posts generally serve a population of 1,500–2,000 in rural areas, and 5,000 in urban areas (Ministry of Health 2014 National Primary Health Care Program). This staffing model is very popular politically, but it is very normative and neither takes into account the health needs of the local population, nor has it been properly costed. In fact, recent analyses and discussions with finance and health officials point to the unaffordability of this model, even in the unlikely event of modest fiscal growth (Feigl at al. February 2016: A Rapid Assessment of Key Areas of the NHSSP for Timor-Leste). At the administrative post level, Community Health Centres provide a higher level of services than the health posts, have a wider range of staff, and they also provide mobile clinic services as well as technical and managerial support to health posts. The types of CHC is not the same across all subdistricts as they have outpatient services and, up until now, the types of services provided are according to the size of the catchment population and distance from higher referral facilities (NHSSP 2011–2030). Municipality CHCs provide inpatient and outpatient services with a staffing of 10-14 personnel. Depending on the vicinity of referral hospitals, inpatients are admitted to an observation unit with two to four beds for pre-referral stabilization of severe cases, or to a ward of 10–20 beds with a set of diagnosis support equipment including a laboratory with capacity for essential tests. Where there is no health post available in remote communities, CHCs provide basic mobile clinic services on a regular basis by motorbike on a twice-per-week basis (NHSSP 2011–2030). The health infrastructure remains a weak link of the Timor-Leste health system. Especially in rural areas, health posts struggle with a lack of electricity, equipment, and sanitation. Newly constructed public health facilities have not improved service delivery as anticipated, in part because they are poorly maintained. In Covalima, the USAID focus municipality, a Service Availability and Readiness Assessment (SARA) conducted in October 2016 by the MOH with Reinforce staff showed poor infrastructure conditions and a lack of basic equipment. On the infrastructure side of the 23 health facilities in Covalima, 10 did not have electricity, 13 did not have piping water into the facility, and 19 did not have flush toilets. On the equipment side, 19 health facilities did not have basic materials for infection control and 22 did not have the necessary equipment to deliver care for newborn babies and obstetrics care for the mothers (Reinforce Service Availability and Readiness Assessment 2016). In April 2016, USAID commissioned a rapid Strengths, Weakness, Opportunities, and Threats (SWOT) assessment for key areas of NHSSP for moving forward. The assessment identified issues related to the management of human resources, particularly its quality and distribution. On the quality, the assessment identified that human capacity in the health sector remains extremely weak in Timor-Leste. This is related not only to the performance of the health sector, but a weak education system, a high fertility rate, and continued poor maternal and neonatal health, including severe malnutrition of mothers and babies. The SWOT found that to address human capacity issues, break the cycle of poverty, malnutrition, and un￾sustained development efforts, there needs to be better integration of plans between the Ministry of Education, INS, and the MOH. The assessment also noted that although over 1,000 Timorese university students were trained as medical doctors in Cuba, the quality and appropriateness of their skills is in question. They were trained in a public health preventive model and received very limited clinical training (Feigl at al. February 2016: A Rapid Assessment of Key Areas of the NHSSP for Timor-Leste). USAID.GOV REINFORCE FINAL EVALUATION | 86 Another significant challenge facing the health sector is the state budget allocated for the health sector is the lowest in the region (in 2017, only comprising 2.0 percent of total government spending as compared to Myanmar 3.6%, Indonesia 5.7%, India 5%, Sri Lanka 11% and Nepal 11%). Eighty percent of the total health budget is allocated to salaries, wages, and public transfer leaving very little fiscal space for medical goods, services, and infrastructure, all in need of substantial investment. Further, Timor-Leste faces the significant challenge in taking the health development agenda into its own hands, against the backdrop of a contracting oil-economy, responsible for up to 80 percent of the GDP (Feigl at al. February 2016: A Rapid Assessment of Key Areas of the NHSSP for Timor-Leste). SAMES (Timor-Leste Central Pharmacy) faces several challenges, primarily drug stock-outs, distribution, management, human capital, and quality control. Many sources show that essential medicine stock-outs are frequent. For example, in 2014, 33 percent of facilities reported a stock-out of essential medicines. The procurement process is painstakingly slow and there are disbursement delays from the government, delays for clearance in the port, and delays in distribution to health facilities. DESCRIPTION OF THE INTERVENTION TO BE EVALUATED AND THEORY OF CHANGE USAID’s Reinforce Basic Health Services Activity The Reinforce activity hypothesis predicts that enhancing and maintaining the skills of the health workforce to deliver on the continuum of care model for high quality family planning (FP) and maternal and newborn health services, particularly in remote settings, will contribute to improved quality of services. USAID/Timor-Leste recognizes capacity building in isolation does not sustain skill enhancement and for interventions to be successful and sustainable, resources should be available in the workplace to carry out a stated objective. Conversely, appropriate actions should transform those resources into performance. Therefore, if adequately supported with resources, medical equipment, medicines, transportation, and communication, Reinforce activity’s capacity building interventions may enhance utilization of services and lead to improved maternal and neonatal outcomes. The graphic below illustrates the Reinforce activity’s Theory of Change. 87 | REINFORCE FINAL EVALUATION USAID.GOV Covalima Municipality The Covalima Municipality is located at the southern coast of the Timor island, on the western border with Indonesia. It has an area of 1,203 square kilometers, and according to the 2015 Population and Housing Census it has a population of 64,550 people living in 13,285 households. The Municipality has seven administrative posts and 30 villages. There is one referral hospital, seven CHCs, and 17 HPs in Covalima Municipality. The Reinforce activity is supporting the MOH in Covalima Municipality to become a model for TimorLeste and an example of strong coordination with the national level in Dili. At the heart of the model, a successful municipality should demonstrate high-quality integrated services and community outreach, quality improvement, teamwork, micro-planning, problem solving, and monitoring. Below are some of the tasks the Reinforce activity conducted in 2019 as part of their effort to establish a model municipality with quality integrated health service delivery (Reinforce January–December 2019 Annual Work Plan): • Support health facilities to implement their facility improvement plans developed during the 2018 planning workshop and support the 2019 planning workshop. • Provide orientation and support for MHS personnel in planning and conducting supportive supervision for facility readiness and maternal, neonatal and child health (MNCH) and FP, and assist them to implement corrective actions (e.g. on-the-job training, orientations, refresher training). • Support MOH HR Department to provide orientation on the roles, responsibilities, and required competencies of all health personnel. USAID.GOV REINFORCE FINAL EVALUATION | 88 • Work with MOH and MHS to ensure adequate stock and good management and distribution of MNCH/FP commodities, supplies, and equipment. • Procure and deliver basic medical equipment for all facilities in Covalima and support MOH in the orientation of equipment use, maintenance, and management. • Support MOH to provide training on the new CQI tool and templates in selected health facilities in Covalima and support ongoing interventions. • Collaborate with PRADET (Psychosocial Recovery and Development in East Timor) to facilitate additional training for primary health care (PHC) workers to recognize, respond to and refer domestic violence cases to the hospital’s Fatin Hakmatek. • Support the MHS and MOH in presenting the improvement measured in the quality and utilization of MNCH and FP services in Covalima with Reinforce activity’s TA. • Coordinate with USAID's Strengthening Inclusive Elections and Political Processes and identify opportunities to support each other's objectives. • Seek further collaboration with St. John of God for expanding the health manager program to CHC managers. The other component of the model municipality is to improve healthy behaviors. To this effect, the Reinforce activity delivered the following in 2019 (Reinforce January–December 2019 Annual Work Plan): • Collate packages of existing Social and Behavior Change Communication materials for different audiences. • Using a variety of media, support PHC workers in holding group discussions in the community in order to increase the knowledge and practice of key Birth preparedness/complication readiness FP and immunization behaviors. • Support PHC workers to facilitate health promotion discussions in the health facilities to prevent drop-out and reduce missed opportunities (e.g., ensuring FP talks take place during immunization). • Support PHC workers to provide health promotion to persons involved in decision-making for MNCH and FP, e.g. males as well as parents and parents-in-law through group discussions. • Support PHC workers to do home visits as per MOH PHC Guidelines to reinforce messages and the practice of new behaviors. • Support INS to conduct Follow Up After Training (FUAT) for PHC workers trained in IPC. • Support MOH to field-test the Sexual and Reproductive Health (SRH) educational materials in one secondary school in Covalima. • Support MOH to provide orientation on the SRH educational materials for the MHS, selected health workers, and relevant municipality stakeholders (education department, etc.). • Support MHS and health facilities in the implementation of gender segmented SRH educational activities for adolescents in school or out-of-school and youth leaders at the village level. The National Institute of Health The National Institute of Health (INS) is an autonomous body within the MOH. Its main mandate, as per Decree Law 09/11, is to conduct and manage in-service training and continued medical education for health professionals in Timor-Leste. INS’s responsibility is to ensure health professionals have the knowledge, skills, and competency to deliver health care through a competency-based training program. INS is managed by one executive director and three directors: (1) the director of administration, finance and logistics, (2) the director of cooperation; and (3) the director of training. The total number of personnel is approximately 50 and the premises (built on 2 hectares) include 4 main buildings, with offices, classrooms, a computer lab, a library, a skills lab, an auditorium, meeting rooms, etc. 89 | REINFORCE FINAL EVALUATION USAID.GOV In 2014, a Situation Analysis study of National Health Institute by the MOH: “Understanding the Past to Better Plan the Future” showed that INS lacked several important elements in order to conduct competency-based training: • curricula and training packages do not exist in various areas, • training rooms are not standardized, • practicum sessions in skill lab or clinical setting do not exist Another report suggested that the CHCs and hospitals utilized by INS and training participants to practice delivery of MNCH care did not fulfill the standard practices (United Nations Populations Fund, 2016). These weaknesses impacted the quality of training resulting in many training participants being unable to deliver services to clients according to international standards. Following the situation analysis, INS’ Five-Year Strategic Plan 2015-2019 was released with the main objective to prepare the conditions for the INS to become a center of excellence for training, research, and institutional management in Timor-Leste by 2019. The Strategic Plan includes 13 strategic priorities addressing INS’ management, leadership and governance; quality of training and research; competency and availability of staff for leadership, management, training and research; infrastructure and facilities; information technology, communication, transportation, and other support services; and financing of institutional development, training and research activities. The Reinforce activity support to INS includes: • Providing ongoing organizational development assistance to the INS in order to improve its performance. This component of Reinforce activity looks at assisting INS to implement its Five-Year Strategic Plan through better coordination with partners. Results to date have included annual plans developed between INS and partners; quarterly working group meetings; conduct of a training needs assessment in Covalima; development of the clean and safe delivery training package; drafting of multiple organizational documents (e.g., checklists, standard operating procedures, internal regulations); a mid-term assessment of the implementation of the Strategic Plan; and daily advisory work through the project advisor based at the INS. • Supporting the establishment of INS as a competency-based clinical training center with standardized clinical practice sites. INS is cooperating with selected health facilities to allow its training participants to practice newly acquired skills on patients. As mentioned above, these facilities – e.g., Vera Cruz and Comoro health centers in Dili and Suai Referral Hospital in Covalima – need technical assistance for reaching adequate standards and serve as a model of care for training participants. Results to date have included training of trainers at INS and clinical practice sites; quality improvement exercises at the practice sites (ongoing assessments-planning-implementation cycles) followed with on-the-job training and technical assistance; support INS in getting accredited as an in-service training center; and set-up of skills lab at the INS and one of the clinical practice sites. • Supporting the delivery of MNCH and FP training and FUAT, in order to increase the number of qualified service providers in Covalima Municipality and the three clinical practice sites and ensure that health worker competency is continuously improved and maintained. Results to date include 7 advanced trainers, 20 trainers, 215 trained health workers on clean and safe delivery, FP, Integrated Management of Neonatal and Childhood Illnesses (IMNCI), and IPC skills; 146 FUAT sessions; one of Reinforce activity’s staff trained as trainers for emergency obstetric and newborn care (EmONC). EVALUATION QUESTIONS USAID.GOV REINFORCE FINAL EVALUATION | 90 The evaluation must address the following questions: 1. How did Reinforce activity engage and strengthen the capacity of stakeholders (MOH, INS, municipality health services, health facilities and local leaders) to institutionalize16 best healthcare service practices in Covalima? a. How effective have they been in engaging and strengthening the capacity of stakeholders? 2. What bottlenecks, if any, in the capacity and commitment17 of stakeholders (MOH, municipality health services, health facilities and local leaders) challenged the institutionalization of best healthcare service practices in Covalima? 3. What evidence is there that the institutionalization of healthcare service best practices in Covalima will likely be sustained? a. Which part(s) of the model is more promising in terms of sustainability? Why? b. What are the remaining gaps that obstruct the sustainability of the activity in Covalima? How might they be overcome? 4. What are the capacity and commitment of the GoTL, the MOH and the INS to adapt and replicate the Covalima model in other municipalities in Timor-Leste? 5. Beyond Covalima, how has the Reinforce activity strengthened the capacity of INS? a. Where are the gaps, if any, in INS’ capacity and commitment to conduct and to manage in-service training and continued education of health professionals in Timor-Leste? b. How has Reinforce mitigated these gaps? c. What, if any, are capacity gaps that remain? EVALUATION DESIGN AND METHODOLOGY The external performance evaluation must make appropriate use of best evaluation practices. Methods will include a mix of quantitative and qualitative data collection and data analysis, as appropriate and to the extent possible. The Evaluation Team should consider the following criteria: • Access to Covalima health facilities • Gender roles • Sensitive context • Type of respondent The evaluation must clearly articulate the link between each evaluation question, the proposed data to address it, and the analysis plan for these data. For example, a qualitative approach may detail each planned analytical step (e.g. coding frame, how it was developed). The evaluation design must use best practice approaches (based on evaluation and research literature) to provide robust answers to each evaluation question and be suitable to the Timor-Leste context and type(s) of respondent. The evaluation team must 16 Institutionalize: establish something (best practices/evidence-based interventions/innovations) as an accepted part of an organization. 17 Commitment: How well a country’s laws, policies, actions, and informal governance mechanisms -- such as cultures and norms -- support progress towards self-reliance. Capacity: How far the country has come in its journey across the dimensions of political, social, and economic development, including the ability to work across these sectors. 91 | REINFORCE FINAL EVALUATION USAID.GOV use standard empirical tools such as statistical software for quantitative analysis (for example SPSS, or STATA) and software for qualitative analysis (for example Atlas.ti or NVIVO), as applicable. The evaluation may design data collection instruments and protocols to gather data as appropriate. Though such instruments and protocol may be based on existing tools, they must be adapted and tested to address the specific evaluation questions and the Timor-Leste context and have a detailed plan for relevant tests of such instruments. USAID expects that Desk Reviews, Key Informant Interviews (KII), and Focus Group Discussions (FGD) will be the primary data collection methods for this evaluation. It is anticipated that the field data collection will be conducted in Tetum and English interchangeably. For reference, USAID considers the following to be main stakeholders of the evaluation activities: • MOH Maternal and Child Health Officers • INS managers and trainers • Covalima Municipality Health Services Officers • Health facility managers supported by USAID in Dili and all public health facilities in Covalima • Health facility midwives, doctors and nurses trained with USAID assistance • Covalima - Municipality Administrator, Administrative Posts administrators, village leaders • Clients of family planning services and maternal and child health services. Additionally, the following organizations worked very closely with USAID’s Reinforce activity: • United Nations Population Fund • World Health Organizations • United Nations Children’s Fund • Health Alliance International • Marie Stopes International • Partnership for Human Development • Saint John of God • CARE International • PRADET The Evaluation Team must properly train all enumerators to appropriately collect data, as needed. Proper data quality systems (manual and automated) and supervision should be put in place. Data quality should be checked frequently, and issues should be reported to COR during weekly updates. Any quantitative data must be collected digitally. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. The evaluation must adhere to relevant quality standards as provided in the USAID Evaluation Policy, as well as international standards for conducting social science research, especially the “Do No Harm” principles such as the principles for treatment of human subjects and collection and safeguard of personally identifiable information. The Contractor must follow USAID’s and the host country government’s protocols for research activity, including any requirements to protect Human Subjects, such as Institutional Review Board (IRB) approval. The Evaluation Team must finalize the evaluation methods with USAID before fieldwork begins. The Evaluation Team must review all documentation provided by USAID and any relevant secondary research USAID.GOV REINFORCE FINAL EVALUATION | 92 or reports they locate in the published and grey literature relevant to the health sector in Timor-Leste prior to conducting field-based work. The Evaluation Team must: • Familiarize themselves with documentation about the Activity and USAID’s assistance in the health sector in the country. USAID will ensure that this documentation is available and shared with the team at the time of award or as soon as possible; • Review and assess the existing performance and effectiveness information or data; • Conduct site visits for field testing of data collection instruments prior to actual data collection; the desk review includes at a minimum the documents provided in the list of references and annex. Gender Integration In compliance with ADS 205 on Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle and ADS 201 on Program Cycle Operational Policy; the evaluation must be gender responsive. Gender-sensitive indicators, sex-disaggregated data, and attention to gender inter-relations are required elements of USAID evaluations; therefore, the Contractor must ensure gender integration in the design and implementation of the evaluation to the extent possible, and document when gender integration is not expected. FINAL REPORT FORMAT The evaluation final report must include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations (if applicable). For more detail, see the USAID Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. An optional Evaluation Report Template is also available in the Evaluation Toolkit. The abstract (of not more than 250 words) must describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions [as well as recommendations and lessons learned, if applicable]. The evaluation methodology must be explained in the report in detail. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report must include: • The Evaluation Scope of Work (SOW); • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, • checklists, and discussion guides; • All sources of information/data, properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of; • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; 93 | REINFORCE FINAL EVALUATION USAID.GOV • Summary information about evaluation team members, including qualifications, experience, and role on the team. USAID.GOV REINFORCE FINAL EVALUATION | 94 Annex VIII: Disclosure of Any Conflicts of Interest U.S. Agency for International Development Rua Travessa Sergio Viera de Mello Municipio Dili No.04 Farol, Dili, Timor-Leste