Evaluation of USAID-funded Advancing Universal Health Coverage Activity September 2021 Evaluation of USAID-funded Advancing Universal Health Coverage Activity September 2021 This publication was produced with the support of the United States Agency for International Development (USAID) under the terms of USAID’s Research for Decision Makers (RDM) Activity, cooperative agreement no. AID-388-A-17-00006. Views expressed herein do not necessarily reflect the views of the US Government or USAID. USAID’s Research for Decision Makers (RDM) Activity icddr,b 68, Shaheed Tajuddin Ahmed Sarani Mohakhali, Dhaka 1212, Bangladesh Phone: +880 (0)2-2222-77001 to 10 PABX: 16340 (direct) Fax: (+88 02) 9827075, 9827077 http://rdm.icddrb.org/ https://www.icddrb.org/ Evaluation of USAID-funded Advancing Universal Health Coverage Activity ii CONTRIBUTORS: Evaluation Team Pial Islam, Consultant Yasmin H Ahmed, Consultant Mahbub Elahi Chowdhury, icddr,b Shusmita Khan, D4I Anadil Alam, icddr,b Farhana Karim, icddr,b Report drafted by Pial Islam, Consultant Yasmin H Ahmed, Consultant Mahbub Elahi Chowdhury, icddr,b Shusmita Khan, D4I Anadil Alam, icddr,b Farhana Karim, icddr,b Report finalised by Mahbub Elahi Chowdhury, icddr,b Yasmin H Ahmed, Consultant Evaluation of USAID-funded Advancing Universal Health Coverage Activity iii ACKNOWLEDGEMENTS This evaluation of United States Agency for International Development (USAID)-funded Advancing Universal Health Coverage (AUHC) Activity was commissioned by USAID’s Research for Decision Makers (RDM) Activity of the International Centre for Diarrhoeal Disease Research, Bangladesh (icddr,b). The Evaluation Team would like to thank the following individuals for their leadership and support during the launch and implementation of this evaluation: Dr Shams El Arifeen, Chief of Party, RDM, Senior Director, Maternal and Child Health Division (MCHD), icddr,b; Dr Quamrun Nahar, Technical Coordinator, RDM, and Head of Research, MCHD, icddr,b; and Dr Md. Shah Alam, Project Coordinator, RDM, icddr,b. The evaluation team would also like to express their gratitude to Ms. Gabriela Maria Escudero, Deputy Director (Acting), Data for Impact, University of North Carolina at Chapel Hill for reviewing the final report. The Evaluation Team would especially like to thank the Office of Population, Health, Nutrition and Education (OPHNE), USAID Bangladesh for its in-depth guidance and engagement with the evaluation. OPHNE’s input greatly aided the Evaluation Team’s understanding of the rather complex project and ensured that the team asked the right questions. This report’s findings were periodically shared with designated members of OPHNE, USAID Bangladesh, with feedback and further input received from the relevant OPHNE staff, including the immediate past Director Mr Xerses Sidhwa, current Director Ms Carrie Rasmussen, Deputy Director Ms Miranda Beckman, immediate past contracting officer’s representative (COR) of AUHC activity Dr Pushpita Samina and the current COR of AUHC activity Dr Fida Mehran. We are especially grateful to Dr Kanta Jamil, Senior Monitoring, Evaluation and Research Advisor, USAID Bangladesh for her deft steering of the exercise, which created an appropriate and enabling environment for the Evaluation Team. The evaluation would not have been possible without the full participation and sharing of information by all partner agencies involved in AUHC activity, including the prime implementer Chemonics, the Thinkwell Group, Population Services International, AUHC country project team, and the Surjher Hashi Network (SHN). This evaluation is built on the discussions and interviews the team had with individuals from these agencies; their frank sharing of stories and concerns and provision of a large number of documents and information as and when requested for was much appreciated. The team is also thankful to the managers and providers of the SHN clinics, who extended their full support to the Evaluation Team members who visited the clinics. The Evaluation Team would also like to thank all of the other external stakeholders who gave their valuable time and who added their insights and suggestions to the project, including senior officials from Directorate General of Health Services and Directorate General of Family Planning, current and ex-private sector partners and donor agency representatives. A detailed list of the individuals the Evaluation Team consulted is given in Appendix 13. Thanks to Mr. Shafayatul Islam Shiblee, Statistical Officer, icddr,b for analyzing data and formatting the final report. Evaluation of USAID-funded Advancing Universal Health Coverage Activity iv Table of Contents Acronyms..............................................................................................................................vi Executive Summary ............................................................................................................ viii A. Context and Background ............................................................................................ 1 Introduction to the AUHC Project....................................................................................... 1 Purpose of the Evaluation.................................................................................................. 2 B. Methodology............................................................................................................... 3 C. Key Findings from the Current State Review .............................................................. 5 1. Result 1 – The SSN Transformed into a Centrally Managed, Sustainable Private Enterprise.......................................................................................................................... 5 1.1 Transformation and optimisation .............................................................................. 5 1.2 Transformation and optimisation challenges ............................................................ 7 1.3 Revision of the discount policy for the poor.............................................................. 8 2. Result 2 – Access to and Uptake of the Expanded Essential Service Package ............ 9 2.1 Enhanced service package ...................................................................................... 9 2.2 Increased informed demand for ESP ..................................................................... 11 2.3 Addressing the needs of the poor........................................................................... 12 3. Result 3 – Financial Sustainability, Private Sector Partnerships and Technology ........ 13 3.1 Financial Sustainability........................................................................................... 13 3.2 Private Sector Partnerships.................................................................................... 16 3.3 Technology ............................................................................................................ 17 4. Result 4 – Improved Quality of Care............................................................................ 18 4.1 Improved customer experience .............................................................................. 18 4.2 Continual quality improvement system................................................................... 19 5. Result 5 – Improve the Effectiveness of Programme Implementation Based Upon Lessons Learned............................................................................................................. 20 6. Organisational Issues .................................................................................................. 23 6.1 Structure and execution of the AUHC project......................................................... 23 6.2 Functioning of the SHN ..................................................................................... 24 6.3 The SHN’s current work environment ................................................................ 25 7 Underlying Factors ................................................................................................... 26 7.1 A highly complex project designed without adequate consideration of the local context ......................................................................................................................... 26 7.2 Poor fund flow management .................................................................................. 26 7.3 USAID’s role .......................................................................................................... 27 Evaluation of USAID-funded Advancing Universal Health Coverage Activity v 7.4 Revolving door leadership...................................................................................... 27 7.5 TA deliverables and limited private sector and commercialisation experience........ 27 D. Go-forward Strategic Options ................................................................................... 28 Guiding Principles ........................................................................................................... 28 Strategic Option 1 – Enhanced status quo (ESQ)............................................................ 29 Strategic Option 2 – Adaptive real transformation (ART) ................................................. 32 2.1 Overview of ART option............................................................................................. 32 2.2 Time frame and phasing ............................................................................................ 32 2.3 Resources for Option ART......................................................................................... 33 CSFs ............................................................................................................................... 34 Appendices ......................................................................................................................... 35 Appendix 1 – Results framework ..................................................................................... 36 Appendix 2 – SoW........................................................................................................... 37 Appendix 3 – Document review categorisation list........................................................... 43 Appendix 4 – Insights from SHN clinic staff: managers, doctors and paramedics ............ 49 Appendix 5 – Evaluation questions based on summary findings...................................... 51 Appendix 6 – Force field analysis .................................................................................... 57 Appendix 7 – Services available in Smiling Sun clinics at close of NHSDP (N = 399)...... 61 Appendix 8 – SHN service offerings by new typology ...................................................... 62 Appendix 9 – Field visit impressions................................................................................ 64 Appendix 10 – Samples of clinic stationery/forms............................................................ 66 Appendix 11 – Key events related to fund flow/budget .................................................... 67 Appendix 12 – Changes in leadership ............................................................................. 68 Appendix 13 – List of meetings and stakeholders............................................................ 69 Evaluation of USAID-funded Advancing Universal Health Coverage Activity vi Acronyms ANC Antenatal care ART Adaptive Real Transformation ATP Able to pay AUHC Advancing Universal Health Coverage BCC Behavior Change Communication CCSDP Clinical Contraceptive Service Delivery Programme CEO Chief Executive Officer COR Contracting Officer's Representative COVID-19 Coronavirus Disease 2019 CR Cost Recovery CRO Client Relationship Officer C-section Caesarean Section Delivery CSF Critical Success Factor CSP Community Service Provider CYP Children and Young People DGFP Directorate General of Family Planning DGHS Directorate General of Health Services DTC District Technical Committee ECG Electrocardiogram EMR Electronic Medical Record EPI Expanded Programme on Immunization ESP Essential Service Package ESQ Enhanced Status Quo FGD Focus Group Discussion FMCG Fast-Moving Consumer Good GBV Gender-based Violence GMP Growth Monitoring and Promotion GOB Government of Bangladesh HMIS Health Management Information System HR Human Resources IMCI Integrated Management of Childhood Illness IR Intermediate Result MCH Maternal–Child Health MECE Mutually Exclusive, Collectively Exhaustive Evaluation of USAID-funded Advancing Universal Health Coverage Activity vii MERL Monitoring, Evaluation, Research and Learning MFS Mobile Financial Service MOHFW Ministry of Health and Family Welfare MNCH Maternal Child Health NCD Non-communicable Disease NGO Non-government Organisation NHSDP NGO Health Service Delivery Project NVD Normal Vaginal Delivery OIG Office of the Inspector General OPD Out Patient department OTC Over-the-Counter PAC Postabortion Care PM Permanent Method PNC Postnatal Care POP Poorest of the Poor PSI Population Services International QA Quality Assurance R&B Registration and Billing RDM Research for Decision Makers RFW Result Framework RH Reproductive Health RTI Respiratory Tract Infection SDS Service Delivery Supervisor SHN Surjer Hashi Network STI Sexually Transmitted Infection SOP Standard Operating Procedure SoW Scope of Work SP Service Provider SSN Smiling Sun Network TA Technical Assistance TB Tuberculosis TBA Traditional Birth Attendance UHC Universal Health Coverage USAID United States Agency for International Development USG Ultrasound Sonography VIA Visual Inspection with Acetic Acid Evaluation of USAID-funded Advancing Universal Health Coverage Activity viii Executive Summary The United States Agency for International Development’s (USAID’s) Advancing Universal Health Coverage (AUHC) project (2017–2022) has been supporting a large-scale country￾wide initiative that has been providing essential health service packages through the Surjer Hashi Network (SHN) since 2017. Its overall objective is to provide affordable and good-quality health services through a sustainable pro-poor network, implemented through a USAID contract with Chemonics International. In June 2021, USAID commissioned a high-level routine performance evaluation of the programme in order to assess the progress of the transition over the previous four years with respect to the (revised) programme priorities and to provide strategic optionson the programme’s future direction. The evaluation was carried out by the Research for Decision Makers (RDM) Activity of icddr,b, which engaged a team of three domain experts (universal health coverage [UHC]), health service provisioning and quality, and financial sustainability), and three research staff from icddr,b and D4I. The evaluation team conducted 48 interviews with 118 participants across 23 organisations, undertook focus group discussions with SHN clinic managers, doctors and paramedics, conducted two clinic visits and carried out extensive reviews and analysis of 700 different documents. The exercise was carried out during June–September 2021. The Evaluation Team assessed the performance of the AUHC project against the five results stated in AUHC’s Results Framework, namely (i)the transformation of the SHN; (ii) expansion of the essential service package; (iii) expansion of sustainable financial systems; (iv) improved quality of care; and (v) the increased effectiveness of programme implementation based on lessons learned. In addition, the team examined various other organisational issues that had also influenced the project’s implementation. During this process, six evaluation questions (on the effect of the optimisation exercise on project implementation; nuances of service delivery provisions; use of technology and innovation; improvement in service quality; overall learning from the project; and organisational issues affecting project management) were also addressed. Evaluation of USAID-funded Advancing Universal Health Coverage Activity ix Key Findings: Result 1: Transformation of Smiling Sun Network (SSN) into a centrally managed and sustainable enterprise Initially, the AUHC project, by engaging the SHN Board, took the first steps to transform the Smiling Sun Network (SSN) from a non-government organisation (NGO) network to a centrally managed private enterprise. The next step was to optimise the size of the network. The five￾year AUHC project began with 399 clinics belonging to a number of NGOs; soon afterwards, however, 30 of these clinics were handed over to another group for management. This was followed by a detailed performance analysis that assessed the potential of the remaining 369 clinics based on their ability to fulfil the needs of local people, offer suitable service provision and generate income. The optimisation process also examined the clinics’ typology (advanced, basic, satellite), including service provision, infrastructure, manpower and equipment required by each category. As a result, the number of clinics was reduced from 369 to 134 (38 advanced and 96 basic), while the satellites were reduced from 10,000 to 4,917 and community service provider (CSPs) from 6000 to 1067. Emphasis was given to the importance of retaining clinics in urban areas (110 of the remaining 134 clinics are in urban locations), which confirms with the SHN’s goal to serve the urban population. However, the transformation process was not smooth and the AUHC project faced a number of barriers that caused delays. One major challenge was how to obtain all the required licences (six to eight per clinic, depending on the scope of the services offered). This difficulty was mainly due to procedural complications involving the licence-issuing authorities. Another barrier was litigation by certain clinic managers, who challenged AUHC’s decision to close down some of the clinics during the optimisation of the network. This, in turn, delayed the necessary clearances from the relevant government authorities that would allow family planning services to be provided by certain clinics. Following the optimisation process, the 2014 poverty-targeting criteria that AUHC had inherited from the NGO service delivery project were revised. Previously, the three SHN client types (poorest of the poor [POP], poor and able to pay [ATP]) enjoyed various levels of benefits according to their economic classification. However, based on the health management information system (HMIS) data review, both the ATP and the poor group were excluded from the provision of discounted services, under the assumption that a substantial proportion of the non-poor had been availing themselves of benefits offered to the poor. This is likely to have Evaluation of USAID-funded Advancing Universal Health Coverage Activity x major implications in terms of limiting the scope of services offered to the poor in the community. Thus, while the establishment of the SHN was an important achievement here, there is a long journey ahead before it is transformed into a sustainable enterprise. Result 2: Expansion of access to and uptake of the essential service package A number of systemic interventions helped to improve the effectiveness of the programme implementation; these include defining the clinic typology, introducing a clinic operations manual, redesigning client flow in clinics, extending clinic hours, re-equipping clinics and filling clinic vacancies. These initiatives, however, have not been completed when the evaluation was undertaken. Additionally, there has been no introduction of new, across-the-board services, and a number of other planned initiatives including antenatal care (ANC) bundling, clinic clustering, specialist placements etc are yet to take off. Demand-creation work has suffered in the absence of a defined marketing plan and longer-term budget allocation, while the phasing out of a large number of CSPs at the community level, the scarcity of education materials and weak planning and oversight have negatively affected community-level health education and demand￾creation work. Client contacts decreased by 67% between 2018 and 2020, with a significant (30%–80%) drop in the uptake of key services (including antenatal care, nutrition services and family planning). Changes in the discount administration system resulted in a decline in the proportion of services delivered to the poor/POP: from 40% in 2019 to 13% in 2020. Therefore, the AUHC project’s overall achievement (in terms of Result 2) has been rather limited. Result 3: Expanded coverage under sustainable financial systems Between July 2020 and June 2021, AUHC achieved a 6.2% year-over-year revenue growth; during this period, average cost recovery at a network and clinic level was 52% and 58%, respectively. Based on current trends, AUHC’s achievements in cost recovery will be close to its projected conservative estimates of 66% and 61%, respectively. The outcome of the private sector partnerships, however, fell far below expectations; in spite of commendable efforts and good partnership designs, they only contributed 0.1% of total revenue. Technology use was limited to the design and deployment of electronic medical records (EMRs). The registration and billing EMR components were rolled out to all clinics during the evaluation period; however, full EMRs were rolled out in only five of the 134 clinics, making Evaluation of USAID-funded Advancing Universal Health Coverage Activity xi any testing of the final output untenable. Various problems need to be addressed in order to make best use of EMRs; these include consistent online connectivity in all locations, the duplication of data input (manual data collection followed by entry into the EMRs), capability of the staff to use the electronic systems, and the use of certain tools required to generate some of the reports. They offer great potential in terms of their ability to track customers, identify cross-selling opportunities and reduce pilferage. Overall, therefore, there has been limited progress towards financial sustainability and the more widespread use of technology. Result 4: Improved quality of care The customer experience has improved due to the lower waiting times resulting from the reorganisation of client flow at the reception desk, as well as to the embedding of counselling within the service provision and the better use of service staff. However, the run-down condition of some of the clinics, obsolete equipment, staff vacancies and non-ideal locations have impeded any further improvements to customer satisfaction. A number of elements of a quality assurance system are in place; however, they have not been tied together to form a complete system. Checklists and supportive supervision protocols are in use, ‘adverse events’ monitoring forms are completed (although they are used more for reporting than for identifying and addressing root cause[s]) and standard operating procedures (SOPs) and other standards have been developed but are not yet fully operational. The clinics are still missing a quality governance system. The caesarean sections being performed in advanced clinics remain high risk due to the dependence on external consultants, over whom the project has little or no control. There is an inadequate number of service delivery supervisors in place, which would provide quality oversight and support various administrative and technical affairs in clinics. As a result, the services offered by clinics are impaired by quality-related issues. Thus, while there has been some progress towards institutionalising quality of care, more needs to be done to put a complete, effective and efficient system in place. Result 5: Increased effectiveness of programme implementation, based on lessons learned A document review has revealed that a number of studies have been conducted by AUHC and its partners, the purposes of which are to generate evidence for strategic planning and improve the effectiveness of SHN programmes. Some of these studies have contributed to Evaluation of USAID-funded Advancing Universal Health Coverage Activity xii the optimisation of the SHN and have developed business plans, service packages, clinic operating systems and discount policies for the poor, etc. However, many of the important study recommendations (such as aligning SHN poverty targeting with Government of Bangladesh [GOB] initiatives, bringing drug-procurement activities under the HMIS and filling vacancies in SHN clinics) are yet to be fully addressed. A monitoring, evaluation, research and learning (MERL) plan has been developed, with its progress against various indicators being regularly reported. A MERL framework has also been developed, serving as a guideline for the sustainable and impactful improvement of SHN service quality. A seven-member MERL team, headed by a director, has been established; this team will work closely with other partners to plan research, participate in the implementation of this research and coordinate knowledge-sharing activities. Despite all these efforts, however, the impact of research on the strategic development of the SHN is as yet limited. Organisational Issues The Evaluation Team identified three sets of issues that have affected AUHC deliverables, including the structure and execution of the AUHC project, the functioning of the SHN and the current SHN work environment. AUHC is a complex project, and coordination and communication among the agencies involved in implementing the project (with Chemonics being the prime contractor and Chemonics, the Thinkwell Group and Population Services International [PSI] serving as the Incubator Team, the AUHC project office and the SHN) has led to delayed decision making and the requirement for a significant amount of management time. Despite the elaborate reporting and information-sharing mechanisms put in place, coordination between Chemonics and its technical assistance (TA) partners sometimes fell through, meaning that certain important TA deliverables could not be produced. The SHN’s position at the bottom of the pyramid and its dependence on AUHC for decisions meant that it was not able to fully transform into a confident, free-thinking organisation. The SHN’s rather traditional and conservative board, whose members are mostly individuals with a long association with the NGO sector, lacks the commercial experience to transform the network as initially hoped; the network’s organogram is weighted more towards management and administrative functions, rather than service delivery, and the capacity of its senior leadership is lacking. The work environment suffers from a culture of adhocism; manual record-keeping systems, poorly structured and planned work in the field and the lapse of Evaluation of USAID-funded Advancing Universal Health Coverage Activity xiii necessary licences have led to inefficiencies, while corruption (although reduced following the installation of a compliance monitoring system) is still present. The Evaluation Team identified a number of underlying factors behind the present situation. AUHC attempted to bring 369 clinics (run and owned by 21 NGOs) under the SHN umbrella. This was an ambitious and complex plan; however, it was designed without adequate consideration of the local context. Although the amalgamation was planned to be achieved in stages, following an ‘optimisation’ exercise the plan was revised and all the clinics were brought under the SHN in one go. This gave rise to litigation, which resulted in family planning supplies being barred by relevant authorities. Another factor that hampered the smooth execution of the project was the weakly managed fund flow; this resulted in delays in fund release and lack of control over expenditure, ultimately leading to fund constraints that affected key activities. The Evaluation Team also noted that USAID became intimately involved in the detailed execution of the project and was behind many important decisions. High turnover in key management positions in all the agencies involved created further gaps and project management issues. Strategic options Against this backdrop, the Evaluation Team is suggesting two options for USAID to manage the direction of the project, going forward. Under the first option, enhanced status quo (ESQ), AUHC would focus on wrapping up the project in a way that would use the remainder of the project duration to position the SHN as a relatively stronger organisation. The SHN would continue its present activities without any major transformation, with a focus on strengthening its existing services and organisational systems, consolidating changes introduced under the project and strengthening its leadership. The second option, adaptive real transformation (ART), would involve reconfiguring the project to address the weaknesses in the AUHC project design (lack of local contextualisation, inadequate commercialisation experience among the current TA providers and inappropriate governance structure). With this option, a financial goal of 100-percent cost recovery would be achievable by the end of 2027. This would, however, require a fundamental strategy rethink, as well as added budget allocation beyond the current AUHC activity. This option would leave a far more positive legacy for USAID and would sustain a health network that hundreds of thousands of low-income people have come to benefit from over the decades. A. Context and Background Introduction to the AUHC Project The five-year Advancing Universal Health Coverage (AUHC, 2017–2022) project is the latest in a series of projects, initiated in 2002, under which a network of non-government organisations (NGOs) are endeavouring to provide essential health services in urban and rural areas of Bangladesh through an effectively distributed clinical network. AUHC’s overall objective is to advance universal health coverage (UHC) through a sustainable pro-poor network. Based on the above objectives, a Results Framework with five major results has been developed. This will drive the project activities and is structured as follows: 1: The Smiling Sun Network (SSN) is transformed into a centrally managed, sustainable private enterprise; 2: Access to and uptake of essential service packages are expanded; 3: Coverage under sustainable financial systems is expanded, ensuring equitable access to health services; 4: Quality of care is improved; and 5: The effectiveness of programme implementation is increased, based on lessons learned. Two to three intermediate results were identified under each of these results (Appendix 1). The NGO Health Service Delivery Project (NHSDP), implemented between 2013 and 2017, was the immediate precursor to AUHC and had objectives closely aligned to it. Much like AUHC, it was implemented through a network of 25 local NGOs and service-provision channels. The NHSDP operated 399 static clinics and 10,754 satellite clinics, serving a population of approximately 26 million people. 1 AUHC inherited this design and infrastructure; it was intended that it would manage the transition of the SSN clinics to a ‘sustainable, pro￾poor social enterprise’, offering new, experimental systems for service delivery to the poor. It was also envisaged that, at activity close, AUHC would leave behind ‘a fully functional social enterprise capable of independently and effectively running a national network of clinics and pro-poor health service delivery outlets’. Through this project, the erstwhile SSN would be transformed from a predominantly donor-supported and decentralised network to a self￾sustaining, centralised network with an expanded essential services package and the ability to offer quality services and to attract development funding (both government and private funds).2 At the design phase, it was decided that the project would use an operator–incubator model to drive the transformation. Accordingly, the prime contractor, Chemonics, put an in-country 1 NHSDP Impact Evaluation, USAID and MEASURE Evaluation, Feb 2019 2 USAID Contract Document AID-388-C-17-00001, Section C Evaluation of USAID-funded Advancing Universal Health Coverage Activity 2 AUHC project team in place, whose responsibility was overall project management and coordination. The incubator included full-time advisors from the Thinkwell Group, Population Services International (PSI) and Chemonics. It would provide revenue generation, business development and operational advisory services to the newly formed social enterprise company, the Surjer Hashi Network (SHN), introduce management information and quality assurance systems, build links to secondary care facilities, enable the professional development of staff and standardise clinic operations. Incubator support would be subsumed within the SHN in Year 3, thereby positioning the network to independently continue operations without AUHC support. Purpose of the Evaluation The five-year AUHC programme is currently entering its last stretch of activities. As such, the United States Agency for International Development (USAID) has decided to undertake a high-level evaluation of the programme to date, focusing on the following two dimensions: ● Assessing the transition progress over the last four years with respect to the revised programme priorities and extracting key lessons; and ● Providing strategic guidance on go-forward options with respect to the programme’s future direction, based on lessons learned and other influencing factors. USAID invited Research for Decisions Makers (RDM) Activity of icddr,b to carry out the evaluation, employing a team with relevant expertise. The evaluation was led by a team leader with expertise in financial sustainability, private sector partnerships and technology. The team also had an evaluation specialist with expertise in strengthening health systems and national health systems, and a maternal–child health family planning (MCH-FP) specialist with expertise in health services provision and quality. The evaluation team was supported by three research staff from icddr,b and Data for Impact (D4I). The evaluation was carried out between June 23 and September 24, 2021. The evaluation’s scope of work (SoW) of the evaluation is given in Appendix 2. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 3 B. Methodology The Evaluation Team used a system thinking–led approach, during which multiple iterations of key lenses were studied simultaneously. The tools used were: a) A document review; b) Stakeholder consultations; and c) Clinic visits. 1.1 Document review: An extensive document review was carried out, covering over 700 documents collected from various stakeholders. The review encompassed project documents; service statistics; partnership agreements; annual and periodic reports; evaluation, assessment and research reports; costings, revenue projections and cost￾recovery trends; strategic exercise documents; and service delivery, training and quality control guidelines. A complete list of documents reviewed (by category) is given in Appendix 3. 1.2 Stakeholder consultations: The Evaluation Team, in consultation with USAID, identified and agreed on the stakeholder segments that were required for consultations during this assignment. Some of the stakeholders suited one-to-one meetings, while others benefited from group discussions. Given the coronavirus disease 2019 (COVID-19) situation prevalent during the evaluation period, all stakeholder discussions were conducted on digital platforms in order to protect the interviewees, the Evaluation Team and any others involved with the logistics required for in-person meetings. The Chatham House rule3 was followed in all stakeholder discussions, and a total of 49 consultations were conducted. Discussions with USAID covered the vetting of scope and methodology and obtaining detailed feedback on work in progress and existing challenges and opportunities. Discussions with AUHC covered financial sustainability, private sector partnerships, technology, service delivery, quality of care, the decision-making process and key operational challenges. Discussions with the SHN covered financial sustainability, private sector partnerships, technology, service delivery, quality, monitoring, the decision-making process and key operational challenges. 1.3 Field visit: Although the intention was to visit five to eight clinics in selected districts around Dhaka, COVID-19 meant that field visits were limited to Dhaka city only. The recommended protective measures against COVID-19 (safe cars, masks, goggles, gloves, 3 The Chatham House Rule reads as follows: When a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed. https://www.chathamhouse.org/about-us/chatham￾house-rule Evaluation of USAID-funded Advancing Universal Health Coverage Activity 4 sanitising of hands, etc.) were adopted for these visits. Two clinics that were selected for a visit covered a range of SHN clinic types: one was a basic and the other an advanced clinic; one was better performing and the other average; and both the clinics had basic electronic medical records (EMRs) used for registration and billing. The use of full EMRs was also being tested in Aftabnagar. The Evaluation Team observed the overall clinic setup, had discussions with clinic managers and providers, observed service provision and assessed the record-keeping system (including the EMR functionality). The team also collected service statistics, organograms, service costs and monthly reports (both those related to services and financial reports). To compensate for a smaller number of clinic visits than originally planned, the Evaluation Team had three separate focus group discussions (FGDs) with clinic managers, doctors, and paramedics from three urban clinics situated outside Dhaka. The findings from these FGDs are detailed in their respective sections, with a comprehensive report given in Appendix 4. The team conducted its analysis across the four key lenses of engagement, which were : ● Transition: tracking progress, partnership effectiveness, financial review, sustainability achievements. ● Service provision: service standardisation, quality, service expansion, digitalisation, cost recovery, staff competency. ● Ecosystem value-add: how the SHN fits in the national health system, cost comparisons, value addition, value for money. ● Future direction: existing challenges (and how to fix them), partnerships, sustainability, critical success factors (CSFs). The evaluation report 's findings are stated against the five intermediate results from AUHC's results framework (RFW); some further organisational issues are added in the findings section. The study used evaluation questions from the SoW as a guide; our responses to those questions are summarised in Appendix 5. To evaluate the results, we used force-field analysis (Appendix 6), which evaluates the net impact of all forces that influence change4 . 4 https://www.mindtools.com/pages/article/newTED_06.htm Evaluation of USAID-funded Advancing Universal Health Coverage Activity 5 C. Key Findings from the Current State Review 1. Result 1 – The SSN Transformed into a Centrally Managed, Sustainable Private Enterprise 1.1 Transformation and optimisation With the support of USAID, the SSN had emerged as one of the largest NGO networks of primary healthcare clinics in Bangladesh. In 2017, AUHC inherited a network of 399 static clinics, 10,000 satellite spots and 6,000 CSPs from the SSN, which amounted to about 24 million people country wide. 5 AUHC, by engaging the SHN Board, transformed the SSN into a centrally managed private enterprise called the Surjer Hashi Network (SHN). Soon after AUHC’s inception, 30 clinics were handed over to another group of NGOs for management. In 2019, AUHC embarked an intensive network optimisation exercise with the remaining 369 clinics, assessing how each individual SHN clinic was meeting the needs of its target population, how well the services were being offered according to the requirements of the new typology and how the revenues were contributing to the SHN’s financial health. In addition, the assessment also took the competitive landscape around the clinic, staffing infrastructure and the distribution of satellite clinics and CSPs into account. As the first step in network optimisation, AUHC and the SHN set the SHN clinic typology and defined three SHN clinic types (advanced, basic and satellite), including service provision and the manpower requirements of each clinic type (Figure 1). During this network optimisation process, satellite service provision was included in the core structure of the SHN service delivery network. Applying this approach in several phases, the number of SHN clinics was reduced from 369 to 134 (38 advanced, 96 basic), satellites were reduced from 10,000 to 4,917 and CSPs from 6,000 to 1,067.6 The optimisation assessment analysed a clinic’s potential to increase service contacts through the provision of an expanded service. It also considered how to mitigate current service gaps to determine if this reform would actually meet population demand and increase cost recovery. Although this transformation was undertaken for the sake of the SHN’s future sustainability, it was a compromise achieved by reducing the SHN’s population coverage at a national level. 5 Year 3 Annual report, AUHC Activity in Bangladesh, Oct 2019–Sep 2020 6 Delivering High-quality and Affordable Health Services to All, SHN presentation to the Directorate General of Family Planning (DGFP), 7 Feb 2021 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 6 Figure 1: SHN Structure and Offering Review7 During the optimisation process, effort was made to retain clinics located in urban areas, in line with the SHN’s goal to deepen its urban footprint. Of the 134 clinics, 110 are in urban areas; thus, the network has the potential to play a lead role in filling the gaps in urban health service delivery. However, if it is to become a champion of urban health, the service provision needs to be expanded in line with the needs of the urban population. 7 AUHC, Network Optimisation Report: Advancing Universal Health Coverage (AUHC) Activity in Bangladesh, 2019, USAID/Bangladesh SHN Structure and Offering Review Delivery (normal, C￾section), PAC, newborn care, OPD consultations (incl. specialised doctors' consultations), wider range of diagnostic services (incl. imaging, USG, ECG, hormonal assays), pharmacy, ambulance Incremental (Core) Services MBBS (duty) doctor, nurse, paramedic/medical assistant, on-call specialists (OBGYN, pediatrician anesthesiologist), labour room attendant, lab technician, pharmacist, clinic manager, admin assistant, receptionist Ambulance Driver Incremental (Core) Staffing Doctors’ consultations (incl. RTI/STI screening, cervical cancer screening using VIA, RH counselling), normal vaginal delivery, essential newborn care, basic lab services (incl. USG, ECG), dispensing corner MBBS doctor, paramedic, nurse/TBA, lab technician, pharmacy salesman, clinic manager, admin assistant, receptionist Paramedics’ Consultations, ANC/PNC, FP ST methods, sick child, growth monitoring, NCD screening (diabetes, hypertension), TB screening (verbal), rapid test (blood grouping, pregnancy), EPI, limited range of OTC drugs and other products Paramedic, clinic aid, service promoter, CSP Advanced Basic Satellite Spot (CSP) Structure Evaluation of USAID-funded Advancing Universal Health Coverage Activity 7 1.2 Transformation and optimisation challenges Following the transformation process, one major challenge AUHC faced was fulfilling the clinics’ licensing requirements (needed for the provision of healthcare services). Six to eight different licenses are required, depending on the scope of services offered by the clinics. As the clinics were previously under NGO jurisdiction, it was previously the NGOs’ responsibility to procure the licenses for the designated services. However, after the transformation, which brought the ownership of the clinics under the SHN, a singular social enterprise company, it became the SHN’s responsibility to fulfil the licensing requirements. Licensing therefore became a challenge, as the SHN had to adhere to USAID and Chemonics policies, which forbid the use of informal payments. The SHN has recently appointed a consultant to deal with these licensing issues, as complete service delivery compliance by the SHN platform is not possible until the licensing requirements have been met. Another requirement was approval by the district technical committee (DTC) of the Directorate General of Family Planning (DGFP). At the time of this assessment, only 97 of the 134 SHN clinics have received DTC approval for the provision of family planning (FP) services, while approval is pending on applications made for the rest of the 37 clinics. Based on our interview with high-level management at the DGFP, it seems there is a gap in the coordination between central management at AUHC and the SHN and central management at the DGFP. There is a need for effective communication by the SHN with high-level management at the DGFP if the long-standing support of and relation with the DGFP is to be maintained and the provision of FP services through the SHN network is to be permitted. The SSN transition, which reduced the number of clinics, caused tension among the staff of the closed-down clinics that had previously operated under the NGO-run platform. The transition was put on hold on 20 June 2020, due to a high-court order following a writ petition that was submitted to the high-court division of the supreme court of Bangladesh on behalf of 48 clinic managers who were challenging the decision to close 158 clinics. The high-court division, by its order dated 16 June 2020, directed the SHN to continue its provision of services. Although AUHC and the SHN are handling this matter through appropriate legal procedures, it still poses a barrier to the completion of the optimisation process. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 8 1.3 Revision of the discount policy for the poor The SHN inherited the poverty-targeting model from NGO Health Service Delivery Program (NHSDP); this was developed following the ‘Guidelines on targeting the poor and poorest the poor’, written in 2014. 8 Previously, SSN clients used to receive one of three types of cards, based on their classification as poorest of the poor (POP), poor, and able to pay (ATP); members of these groups would get different levels of free or discounted services as per the 2014 guidelines. According to a review of health management information system (HMIS) data, the use of SSN services by POP, poor and ATP individuals were 4%, 49% and 47%, respectively. According to that HMIS data, around 53% of the SSN’s services were provided to poor and POP individuals. Surprisingly, however, the SHN’s new discounting policy targets only the POP, excluding the ATP and poor groups. 9 This has major implications regarding the limitation of the scope of the discounted services benefit to the community as a whole. The limit required to qualify for discount benefits for different services (such as imaging and lab services, home delivery and normal vaginal delivery (NVD), medicine and health products) for the POP has also been lowered. According to the new policy, the clinic manager can provide between a 5% to 25% discount off the listed price to the POP, while paramedics at satellite clinics can provide a 25% to 40% discount on all services. Any discount above that set limit is strongly discouraged and needs the prior approval of regional/clinic manager. This new discounting policy is heavily dependent on the discretion of the provider, who decides whether an individual qualifies for a discount and is likely to be influenced by personal bias. In spite of the discretionary authority given to the providers to provide discounts to the POP, they are discouraged to do so due to pressure from AUHC and the SHN administration, which are looking to achieve their cost-recovery targets. This is evidenced by the very low number of discounted services provided to SHN clients (shown by service statistics and as mentioned by clinic managers during this evaluation). Thus, the new discounting policy not only deliberately excludes a large proportion of the poor from the discounting benefit offered by SHN services, it also discourages the SPs from offering said benefit to those poor clients who are eligible. Thus, the current discounting policy requires a critical review and revision. 8 Surjer Hashi Network, Poverty-targeting Functional Review, March 2019 9 Ibid Evaluation of USAID-funded Advancing Universal Health Coverage Activity 9 2. Result 2 – Access to and Uptake of the Expanded Essential Service Package 2.1 Enhanced service package Appendix 7 and Appendix 8 present the essential service package (ESP) services that were available at the SSN clinics on the eve of the launch of the AUHC project, and those that will be introduced in the three types of SHN clinics under the AUHC project. An ambitious plan to add a number of new services was undertaken, and some progress was achieved: tuberculosis (TB) diagnosis through GeneXpert machines was introduced to three clinics, and gender-based violence (GBV)-related guidelines were developed. Overall service provision, however, has mostly been targeted at pregnant women and children. Many of the components of the government’s ESP , such as care for adolescents, males and the elderly and the treatment of non-communicable diseases (NCDs) are not sufficiently covered by the current SHN service package, in part due to unavailability of USAID funding for NCD. Nonetheless, there is particular opportunity for the further expansion of NCD services through collaboration with the government; for example, the SHN could procure free NCD drugs. Attempts for services to be developed in partnership have also fallen through for various reasons; for example, three eyecare collaborations and a pilot for demand generation through pharmacies were abandoned due to funding constraints or a lack of capacity to manage the collaboration. 10 Some systemic improvements have been implemented that have helped to improve service effectiveness: defining the clinic typology has helped clinic management and planning; a clinic operations manual has been introduced in advanced clinics (although not operationalised); the client relationship officer (CRO) role has been introduced; and counselling embedded in service provision is proving helpful at managing client flow and reducing waiting time. Extended clinic hours in six clinics in Sylhet and Rajshahi have been found to draw in 30% new clients; to date, 49 clinics have extended hours.11 Vacancies are in the process of being filled, and some obsolete equipment is being replaced. Targeted renovation work has been started, although more extensive network-wide renovation plans had to be scaled down due to fund constraints. However, a number of important planned activities have not been initiated (e.g. ANC bundling, clinic clustering and placing specialists in advanced clinics [only two are in place currently]).12 In addition, 10 AUHC Year 3 Annual Report, in-depth interviews and FGDs with providers and clinic managers 11 SHN Q4 Report 12 AUHC Year 4 Workplan Evaluation of USAID-funded Advancing Universal Health Coverage Activity 10 much concerted work will be needed to get all licences in place: 414 licences were still pending at the end of Q4 March 2021. 13 The utilisation of services has been going down since the project’s inception: client contact decreased by 67% during 2018–2020. 14 Some services took a greater hit than others: during the above period, antenatal care and nutrition services decreased by 80% and 77%, respectively, while delivery and immunisation services decreased by 43% and 48%, respectively. FP services, which constitute 32% of all services offered, also showed a decline: children and young people (CYP) services almost halved between 2018 and 2020 (1.05 to 0.53 million) and service contacts with youth on FP/reproductive health issues decreased by almost three quarters (from 20 million to 5.14 million). Essential newborn care decreased dramatically, by 87% (0.352 million to 0.045 million), while cases of child diarrhoea treated in SHN clinics decreased by 83% (2.4 million to 0.403 million) during the same period. 15 Similar downward trends in all the above services were observed during the first three quarters of 2021. 16 A large proportion of these declines can be explained by the reduction In network size (by 64%: from 369 to 134 clinics), the re-organisation of the satellite clinics, including a reduction in the number of CSPs, and finally, the COVID-19 pandemic that hit during Years 3 and 4 of the project. Even a year after the optimatisation exercise downward trend in services was observed during the first three quarters of 2021. Total service contacts during the first three quarters of 2021 was 5.4 million. If extrapolated to the end of 2021, the number of contacts would stand at 7.2 million, a decrease of 55 percent compared to 2020. Large decreases were seen in fp/rh counselling of young people (56 percent), anc contacts (33 percent), child diarrhoea treatment (87 percent) between 2020 and extrapolated figures upto 2021. A series of town hall meetings in May 2021 identified a number of other operational factors, including disruptions in FP and vaccine supplies, the redistribution of FP catchment areas by the DGFP, competition from local NGOs and the private sector and vacancies in SP positions. 17 During FGDs with SPs and clinic managers, the negative impact of the optimisation exercise on staff motivation also emerged as a reason. 13 AUHC Year 3 Annual report, p. 62 14 AUHC Report on Analysis of Data for All SHN Clinics 2018–2020 15 AUHC Year 2 and Year 3 Annual Reports. 16 AUHC 2021 Q1, Q2, Q3 Reports 17 Engagement with Clinics for Declining Performance by the SHN, 2–20 May 2021, AUHC–SHN report Evaluation of USAID-funded Advancing Universal Health Coverage Activity 11 2.2 Increased informed demand for ESP The SHN’s Marketing Planning Workshop at the end of Year 2 of the project helped to develop a ‘brand architecture’ for marketing, but the idea of developing a fully fledged marketing plan in Year 3 never materialised. A budget crunch around end-2019 led to a decision to halt marketing activities. An investment plan was developed in October 2020 with specific allocations for marketing budget in Years 4 and 5 of the project; however, the entire allocation could not be made available in Year 4. From June 2021, local-level clinic promotion of around 40 clinics was introduced; this included approaches such as putting up posters, leaflet distribution and miking (Announcement in the community using loud speaker). By the time of the evaluation, it was too early to assess the impact of the intervention, and there was no specific plan in place for such assessment. The clinics are weakly branded. On the outside, the colour and design of the logo and signposts do not stand out; inside the clinics, there is no distinctive feeling that you have entered an SHN clinic (Appendix 9), as the clinic interiors do not make use of the brand colours or logo, except at the reception desk. The stationery is also not branded (in fact, some are from NHSDP days, with the SHN name stamped over the original branding); most forms are poorly photocopied, handwritten prescriptions are hard to decipher and reports appear unprofessional (Appendix 10). At the community level, CSPs have been mostly phased out; service promoters (SPs), who are full-time salaried staff, do field-based promotion and health education. Each team of three is assigned 11,000 households, without any consideration of geographic demarcation. Although an outreach campaign based on CSPs was envisaged in the AUHC’s revised SoW (Section C), the scale and quality of the implementation was impaired by the smaller number of SPs that replaced the CSPs, a scarcity of education materials, 18 the lack of strong oversight from the head office, especially during the pandemic, and the lack of a defined workplan. 19 It was clear through the document review and from several FGDs that there had been inconsistent attention to demand creation and marketing during the project implementation. The workshop to develop a brand architecture at the end of Year 2, the adaptation and distribution to clinics of 23 types of educational materials and job aids, and 18 According to FGDs with paramedics and SPs 19 Ibid Evaluation of USAID-funded Advancing Universal Health Coverage Activity 12 the newly introduced local-level clinic promotion were the only specific activities undertaken to increase demand. 20 As reported during FGDs with clinic managers, while educational materials were reviewed, adapted and distributed to the clinics, there were no clear guidelines as to how they were to be disseminated, thus limiting the potential impact of this activity. It is surprising that the MERL indicators do not cover any demand-creation activity and that no other initiative has been undertaken to date to assess the outcome of whatever efforts were undertaken. 2.3 Addressing the needs of the poor During Q1 of Year 2, a review of current poverty-targeting practices found that the criteria for the identification of poverty in the clinics were vague: NHSDP’s poverty-targeting model used three cards to identify clients for full, discounted or free services (see Result 1, Section 1.3 for details). The definitions of these three categories were not updated based on national poverty data and there were no tools in place to operationalise poverty targeting and/or apply the approach uniformly across clinics. There was also no alignment with the various Government of Bangladesh (GOB) mechanisms used to identify the poor, while the previous system used to administer discounts was found to be too complicated. Based on these findings and the resetting and standardisation of prices, it was decided to abandon the previous discount system and make discounts ‘exceptional’. 21 As outlined under Result 1, a new policy was introduced that required clinic managers and satellite clinic paramedics to approve discounts up to a set level and to procure the approval of the regional manager or clinic manager if they wished to exceed the limits at clinic and satellite￾clinic level, respectively. The above events and the aggressive drive to maximise revenues from Year 3 onwards were a blow to the concept of providing equitable access to the poor and the POP. In 2019, 60% of the clients in the SHN were ATP clients, 38% were poor, and 2% were POP individuals. In 2020, however, these figures were 87%, 12% and 1%, respectively. The absolute number of poor/POP clients decreased by more than four fifths between 2019 and 2020. 22 During the first three quarters of 2021, the combined percentage of poor and POP clients decreased further: to 10%, 7% and 5%, respectively. 20 AUHC Year 2 and Year 3 Annual Reports 21 AUHC Year 2 Annual Report 22 AUHC Year 3 Annual Report Evaluation of USAID-funded Advancing Universal Health Coverage Activity 13 3. Result 3 – Financial Sustainability, Private Sector Partnerships and Technology 3.1 Financial Sustainability When considering Result 3, the Evaluation Team sought to understand to what extent financial sustainability had been enhanced through innovative approaches, private sector partnerships and technology. This report addresses each of these independently. The first line of inquiry, into innovative approaches, is divided into two parts: identification of the innovative approaches used to enhance financial sustainability and the implementation of these innovative approaches as a means of catalysing sustainability. A study was conducted by the AUHC Incubator Team that attempted to identify innovative practices, reviewing 24 projects across nine countries. 23 The study identified three key innovations: community engagement, telemedicine and cross-selling. Community engagement has been deployed on a large scale by organisations like BRAC in Bangladesh for nearly 50 years, 24 telemedicine has existed in Bangladesh for over five years (Grameen/Telenor),25 and cross-selling has been an integral part of Social Marketing Company (SMC) in Bangladesh for several decades. 26 However, we have not seen any evidence that any of these ideas have been implemented at the SHN so far; hence, its ability to catalyse financial sustainability cannot be tested or verified. Outside of innovative practices research, AUHC has developed detailed financial forecasts and conducted strategic planning exercises to improve financial sustainability. Financial forecasts typically rely on assumptions about the future; the strength of the forecast is directly related to the accuracy of the assumptions. Since the forecasts had been made about a year ago, we first attempted to see whether some of the forecasted projections had materialised in the past year. Those projections could not be validated because they relied on ‘if–then’ conditions and the ‘if’ conditions had not been fulfilled. For example, one stated that if specialist doctors were accommodated across the network, then revenue was projected to grow by 5%; however, no growth had been observed because the initiative had not implemented. Although a 14%, seemingly over-optimistic, year-over-year revenue 23 AUHC, 2021, A presentation on financial sustainability dated 2 February 2021 and a technical brief on service delivery innovations dated July 2021 24 Kaosar Afsana, 2012, Empowering the Community: BRAC’s Approach in Bangladesh, CAB International – Maternal and Perinatal Health in Developing Countries 25 Telenor, accessed 5 September 2021 (https://www.telenor.com/tonic-is-the-new-face-of-digital-health-services￾for-telenor/) 26 SMC, accessed 5 September 2021 (https://www.smc-bd.org/success-stories) Evaluation of USAID-funded Advancing Universal Health Coverage Activity 14 growth was projected, over 6% year-over-year revenue growth (estimated from the regression line; Figure 2) was achieved, which is close to AUHC’s conservative revenue growth projection (7%). Relatively lower growth in July 2020–June 2021 (6.15%) compared to that in July 2019–June 2020 (6.59%) may be due to the COVID-19 effect. However, using the actual value at the corresponding time periods, the estimated revenue growths were -0.72% in July 2019–June 2020 and 0.89% in July 2020–June 202127 . Figure 2: AUHC Clinic-level Revenue The strategic planning exercises conducted were reported to be quite elaborate. The best way to ascertain whether such exercises have yielded value is to see whether the strategy that came out of the planning exercises was implemented and whether it led to the intended results (the most prominent of which, for AUHC, is achieving a cost recovery of 65%). 27 Actual revenue (BDT in million) Jul’19=34.67; Jun’20=34.42; Jul’20=36.97; Jun’21=37.30 0 10 20 30 40 50 Apr'19 May'19 Jun'19 Jul'19 Aug'19 Sep'19 Oct'19 Nov'19 Dec'19 Jan'20 Feb'20 Mar'20 Apr'20 May'20 Jun'20 Jul'20 Aug'20 Sep'20 Oct'20 Nov'20 Dec'20 Jan'21 Feb'21 Mar'21 Apr'21 May'21 Jun'21 Jul'21 Aug'21 Million (BDT) Clinic-level Revenue y = β0 + β1x ŷ = 32.085 + 0.197x ŷJul'19 = 32.87 ŷJun'20 = 35.04 ŷJul'20 = 35.24 ŷJun'21 = 37.40 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 15 Figure 3: AUHC Clinic-level Monthly Cost Recovery Figure 4: AUHC Network-level Monthly Cost Recovery Analysing the cost recovery data from Apr 2019 and Aug 2021 demonstrates that the average cost recoveries during this period were 54% (Figure 3) and 48% (Figure 4) at the clinic and network levels, respectively. At clinic level during July 2020 and June 2021, the average cost recovery was 58% (Figure 3), while at network level the corresponding figure was 52% (Figure 4). If the current cost-recovery trend continues, based on our projections for July 2021 and June 2022 the average cost recovery at clinic and network levels will be 65% and 58%, respectively. These figures are close to AUHC’s estimated financial projections of cost recovery (66% for clinic level and 61% for network level) to be achieved by the end of the project life. 0 10 20 30 40 50 60 70 Apr'19 May'19 Jun'19 Jul'19 Aug'19 Sep'19 Oct'19 Nov'19 Dec'19 Jan'20 Feb'20 Mar'20 Apr'20 May'20 Jun'20 Jul'20 Aug'20 Sep'20 Oct'20 Nov'20 Dec'20 Jan'21 Feb'21 Mar'21 Apr'21 May'21 Jun'21 Jul'21 Aug'21 Percentage Clinic-level Cost Recovery y = β0 + β1x ŷ = 45.844+.566x H0 : β1=0; t=3.038, p = 0.005 0 10 20 30 40 50 60 70 Apr'19 May'19 Jun'19 Jul'19 Aug'19 Sep'19 Oct'19 Nov'19 Dec'19 Jan'20 Feb'20 Mar'20 Apr'20 May'20 Jun'20 Jul'20 Aug'20 Sep'20 Oct'20 Nov'20 Dec'20 Jan'21 Feb'21 Mar'21 Apr'21 May'21 Jun'21 Jul'21 Aug'21 Percentage Network-level Cost Recovery y = β0 + β1x ŷ = 40.595 + 0.509x H0 : β1=0; t= 3.129, p = 0.004 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 16 3.2 Private Sector Partnerships The second line of inquiry involved understanding how private sector partnerships were designed, structured and implemented to enhance financial sustainability. AUHC made a commendable effort to explore these partnerships. The net was cast wide; the AUHC looked at medical laboratories, pharmaceuticals, fast-moving consumer goods (FMCGs) companies and mobile financial services (MFSs). Partners were identified and agreements were signed with a few of these. Additionally, financial projections were made to forecast revenue streams from these partnerships in both the short and long term. The design of these private sector partnerships, therefore, was reasonably good. However, challenges emerged in structuring and implementation, leading to a poor contribution to financial sustainability. A partnership with an FMCG called Marico included certain terms that the SHN was expected to fulfil; that is, ensuring that 90% of the SHN customers receiving free Marico products were verifiable, with a penalty if that verifiability threshold was not attained. 28 As it turned out, the SHN failed to implement a sufficiently effective monitoring mechanism to verify this threshold, 29 and as a result, the potential revenue stream from this partnership was placed in jeopardy. On a partnership with a laboratory called Praava, significant deviations from revenue forecasts were observed (BDT31 million30), as well as from what was actually being achieved in the short term31: less than 3% of projected Year 1 revenues were being materialised. In another partnership with a pharmaceutical company called Square Pharma, the agreement that was structured failed to monetise the greater revenue streams achievable through volume discounts if products were procured at the network level rather than at the clinic level. 32 On yet another partnership with an MFS provider named bKash, despite having signed agreements between the two parties, the work never got off the ground. The net impact of private sector partnerships towards financial sustainability was negligible (they contributed 0.1% of total revenue33); this means that, for every $1,000 revenue generated, only $1 has originated from private sector partnerships. We believe that the 28 AUHC, 2020, Merico–AUHC Partnership Agreement (MOU), 19 March 2020 29 Two sources from our KIIs provided verifiable customer estimates at 81% and 45%, respectively 30 AUHC, 2020, Year 3 Annual Report, p. 27. 31 SHN, 2021, SHN Partnership Updates, slide 2 32 The SHN currently receives (approximately) a 10% discount; others with greater volumes can receive over 50% discount 33 AUHC, 2021, SHN Dashboard_Draft_082221 D2.xlsx, tab PV 134+Rg+HQ Evaluation of USAID-funded Advancing Universal Health Coverage Activity 17 primary reason for private sector partnerships’ deficient performance is a significant gap in capability at various levels of the project in terms of the ability to manage private sector expectations and relationships. This was true of the TA providers, who had a strong health￾systems background but were missing recent or relevant experience in commercialisation or the private sector, 34 as well as of the SHN team, which had some people with commercialisation and private sector experience (including a couple of board members and a few senior leadership team members) but did not have the opportunity to play a fuller role in building the partnerships. 3.3 Technology The third line of inquiry involved understanding how technology was deployed as an enabler in the project. There is hardly a domain in the modern world where the pervasive use of technology is not creating significant value, and healthcare is not an exception. However, despite some initial explorations (digital payments with bKash and telemedicine flagged as something to consider), AUHC’s technological endeavours seem to have been limited to the design and deployment of EMRs. It must be noted at the outset that an evaluation is a snapshot in time. In the case of this evaluation, that time is 14 July 2021, which is the date of the last field visit where we reviewed technological deployments on the ground. This evaluation found that only the registration and billing (R&B) components of the EMR system had been deployed in all the clinics; the full EMRs, however, had only just been tested and were still in the process of being installed in five clinics and could not, therefore, be evaluated. It is our observation that the design and implementation of the system seems to have not taken local context fully into consideration. The system tested requires always￾online network connectivity. In the context of Bangladesh (and of many other developing nations) this is not well thought out; this connectivity issue was flagged in a January 2020 USAID learning brief.35 At the time of this evaluation, 18 months later, the same problem persists. The R&B components currently have some functions that are duplicated by existing tools and processes. As has been documented in the field visit impressions,36 in some locations 34 Assessed based on professional backgrounds (extracted mostly from LinkedIn profiles) 35 AUHC, 2020, Learning Brief, SHN HMIS 36 See Appendix 9 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 18 clinic staff manually generate customer record forms and then enter that information into the EMRs during their off-time. Furthermore, we have seen first-hand how the same information is tracked across multiple tools (including Tally, Microsoft Excel and EMRs) and combined to generate certain reports. There is a plethora of opportunities to leverage technology to support financial sustainability. The EMR components that have been reviewed during this evaluation (namely, the R&B components) certainly show promise as a means to support financial sustainability in the long term. Electronic medical records can help track customers through the registration module and potentially identify cross-selling and up-selling opportunities down the line. They can help ensure that all sales are properly billed and potentially reduce pilferage. However, these are only potential value-creation opportunities; during the evaluation process we have not seen any of these opportunities materialise. One reason for this is that R&B had only just been deployed and insufficient time had passed for the kinks (such as the duplications mentioned earlier) to have been ironed out. We recommend that, once the full EMR is deployed, AUHC undertakes a third-party review of the system, to ensure that any issues identified during the deployment of R&B (e.g. duplication of effort, logistical internet connectivity issues) and those that emerged during the full EMR testing phase are addressed. AUHC should also endeavour to introduce the R&B component at the field level, which would make the optimum use of this valuable investment. 4. Result 4 – Improved Quality of Care 4.1 Improved customer experience An important improvement that contributed to the customer experience during this period was the reorganisation of client flow through the introduction of the CRO role, triaging, counselling embedded within the service provision and better use of support staff. These measures have reduced waiting time (to 10 minutes, down from the previous 90 minutes),37 although during clinic visits the Evaluation Team found the waiting time to be around 20– 30 mins. A small client satisfaction survey in Year 3 that reviewed on seven different 37 AUHC, Year 3 Annual Report Evaluation of USAID-funded Advancing Universal Health Coverage Activity 19 aspects of the clinic experience reported a client satisfaction level of 70%–90%.38 An improved customer experience is the ultimate outcome of a number of interplaying factors, including physical infrastructure, supplies, management and personnel with the right knowledge, skills and capacity. 39 The run-down conditions of some SHN clinics, obsolete equipment, the large number of vacancies and non-ideal locations were impediments to greater client satisfaction. 40 Additionally, the interrupted supply of FP supplies resulted in the cessation of some services, while a lack of clinical and/or refresher training compromised the quality of the services offered. 4.2 Continual quality improvement system The design and implementation of a quality improvement system went well in Year 1 under the guidance of an AUHC quality lead; additionally, ten regional quality assurance managers added value, as they were closer to the clinics and could quickly address any quality issues. When the quality function was transitioned to the SHN, however, the position of quality lead was abolished and the regional quality assurance managers were replaced by service delivery supervisors (SDSs), who were responsible for a multitude of other clinic support functions. Finally, when COVID-19 hit, the move to virtual supportive supervision had a further detrimental effect on quality assurance. Presently, some elements of a comprehensive quality assurance (QA) system are in place. Standard operating procedures (SOPs) have been developed for some services, and compliance with these SOPs is being measured through checklists. ‘Adverse events’ monitoring forms are being used, although their use is more for reporting purposes than for root cause analysis or risk identification. The plan-do-study-act (PDSA) model for quality improvement is being introduced. Most of these initiatives were introduced in Year 4 of the project, leaving inadequate time for a complete roll out to all 134 clinics. In addition, at this point in time, these different initiatives are implemented, analysed and acted upon as disparate issues; however, they need to be considered together, thereby contributing to the formation of a system for continuous quality improvement. A ‘quality governance system’, which is the combination of structures and processes at and below board level to deliver organisation-wide quality 38 Ibid 39 Ӧ Tunçalp, WM Were, C MacLennan, OT Oladapo, AM Gülmezoglu, R Bahl, B Daelmans, M Mathai, L Say, F Kristensen, M Temmerman, F Bustreo, 2015, Quality of Care for Pregnant Women and Newborns: The WHO Vision. BJOG 122, pp. 1045–1049 40 Engagement with Clinics for Declining Performance by the SHN, 2–20 May 2021, AUHC–SHN report Evaluation of USAID-funded Advancing Universal Health Coverage Activity 20 services, is missing. Scoring in selected areas of service delivery was introduced in Year 3; scores for four services over 111 monitoring episodes revealed an average improvement of 16 percentage points over two successive visits, although there could be inherent bias in the scores as the monitoring was undertaken by the same SDSs that are responsible for the clinics concerned.41 The management of stores and drug quality have also been considered. A high-risk, high-return service that needs to be reviewed is the caesarean section (C￾section). A study shared in March 202042 found that only 21% of the 9,666 C-sections were for the right indications; the rest were for reasons that were clearly inappropriate (36%), inconclusive (35%) or based on maternal wishes (8%). The way these operations are run is also risky, with a pair of external consultants (an obstetrician and anaesthesiologist) contracted to perform the operations without any validation of their skills or practices nor oversight by the clinic. Neither is there any clear system to assess the indications for the procedure; whether a C-section is required is largely left to the paramedics to decide; these paramedics are led by the patients’ wishes or by the signs of even minimal complications. The clinic doctors are only minimally involved in this whole process. 43 There is also lack of adequate human resources (HR) investment in QA. The six SDSs are not only responsible for quality monitoring; they also support clinics in many other ways. Each SDS has about 22 clinics in their portfolio; considering the high volume of input needed from the SDSs, this arrangement deters them from spending adequate quality time on QA, and the SHN is yet to take on the quality improvement role. 5. Result 5 – Improve the Effectiveness of Programme Implementation Based Upon Lessons Learned The aim of Result 5 is to undertake assessments and research and review lessons learned, so that evidence-based learning is incorporated to improve programme effectiveness. The document review indicates that a good number of studies were conducted by AUHC, some of which have been instrumental in guiding the optimisation of the network and developing business plans, etc. However, study recommendations such as bringing the SHN clinic under the HMIS for the procurement of drugs, incentivising network staff to improve service performance and strengthening system issues to improve the quality of the service provision are yet to be adapted by the project (Table 1). 41 AUHC Year 3 Annual Report 42 Assessing Quality on the AUCH Project: A Caesarean Section Audit, Global health Division, Chemonics International, March 2020 43 Discussed during FGD with paramedics Evaluation of USAID-funded Advancing Universal Health Coverage Activity 21 Table 1: Selected Research with Key Findings and Status of Learning Adaption Research title Key findings and their use in an improved programme strategy First Phase Network Optimisation44 This study guided the optimisation of SSN clinics, satellite clinics and SPs. It also helped redefine the services offered by satellite clinics by including additional services and presenting a plan for incremental services to be offered by the advanced and basic clinics through the deployment of additional manpower. Some of these recommendations are still in the process of adaptation. Market Landscape Analysis45 The landscape study has provided specific input to the SHN business model and resources for the next steps in strategy development. Its input covers four areas: benchmarking SHN core services to the private sector; benchmarking SHN service prices to the private sector; catchment-area analysis to promote network efficiency; and insights into consumer needs, preferences, and behaviours for marketing strategy. Study findings were used to develop a business plan, service packages and a clinic operating system for the SHN. Poverty-targeting Functional Review46 This review was used to design the SHN’s pro-poor mandate by i) Defining core services by clinic type and ensuring affordable and standardised prices; ii) Providing SHN services at a 50% discount to the POP, up to a maximum of BDT 500 per visit; and iii) Aligning with GOB initiatives using poverty targeting mechanisms and ceasing the SHN’s direct identification of the poor at community and clinic level. The review helped develop a discounting policy for the poor. SHN Drug Procurement Analyses47 The study recommended that existing centralised drug procurement policy be upgraded by increasing number of pharmaceutical supplier companies; increasing number of types of drugs on offer; bringing clinics’ drug procurement under management of HMIS. These study recommendations are yet to be implemented Design Considerations for Healthcare Workers’ Incentive Schemes48 The purpose of this review was to draw upon those global practices and incentive schemes that have been proven to improve staff motivation and the performance of healthcare service delivery network staff in order to develop an incentive mechanism to drive clinic-level performance within the SHN. The study’s findings are yet to be applied. Service Delivery Innovations in Pro￾Poor, For-Profit Healthcare Enterprise49 To help inform and strengthen the SHN’s business model and prioritise its strategic initiatives, the AUHC Incubator Team assessed the global landscape of pro-poor, for￾profit healthcare enterprises. Its aim was to learn about service delivery innovations that have been successfully deployed in pursuit of greater impact and financial return. The study identified three service-delivery innovations (community engagement, telemedicine, and cross-selling), which had the potential to deliver impact for Bangladesh. The study findings are yet to be adapted by the SHN. Service Utilisation at SHN Clinics with Representation in the Network50 FGDs with the clinic staff and providers regarding the utilisation and quality of available services elicited suggestions to fill vacant positions with qualified providers; update providers’ clinical skills; upgrade facilities and provide clinics with updated equipment; and strengthen communication between the SHN headquarters and the clinics. The study recommendations are yet to be applied. 44 AUHC, Network Optimization Report: AUHC Activity in Bangladesh, USAID, 2019 45 PSI, AUHC Market Landscaping Study: Recommendations and Next Steps, USAID, 2019 46 SHN Poverty-Targeting Functional Review, 2019 47 SHN, Drug Procurement: Effects of Contract Rollout with Pharmaceutical Companies, USAID, 2020 48 Design Considerations for Effective Healthcare Worker Incentive Schemes: Technical Brief, USAID, 2021 49 Service Delivery Innovations in Pro-poor, For-profit Healthcare Enterprises and their Applicability to SHN, USAID, 2021 50 AUHC, Improving Service Utilization and Service Quality at SHN: Perspectives from FGDs with Clinic Staff, USAID, 2021 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 22 To monitor performance and report on progress towards achieving IR5 of AUHC, a monitoring, evaluation, research and learning (MERL) plan has been developed. 51 Some of the key components of the MERL plan include delineating the SHN’s performance indicators, setting baseline and revised target values for these indicators, planning further research and developing a learning agenda. AUHC developed a set of monitoring indicators by identifying 58 indicators for 2017-2022. Progress made towards achieving these indicators is reported through quarterly reports. To implement the MERL activities, a seven-member MERL team has been formed, with designated responsibilities for research, monitoring and evaluation, learning, communications and database management, etc. The team is led by the MERL director, who is based in AUHC. A MERL specialist is also in the team. The MERL team is intended to work closely with other partners to identify opportunities for further research and participate in the implementation of this research. It is also well-placed to coordinate knowledge-sharing activities for these proposed research studies. The MERL team developed the MERL learning framework, which is used to propose the techniques and tools required to systematise reflective learning practices within the AUHC project. As such, it should contribute to improving the quality of SHN clinic interventions in a sustainable manner, as well as having a broader impact on healthcare. The MERL team has also organised a quarterly ‘pause and reflect’ workshop, which reviews the results of innovative interventions and thereby guides the research team towards further research and the successful adoption of lessons learned. In addition, the MERL team has developed a number of research abstracts, technical briefs, success stories, case studies and learning stories, using them to disseminate their work. However, despite all these efforts, the impact of the research on the strategic development of the SHN intervention is minimal. As yet, neither a complete business model for the financial sustainability of the SHN nor a financial risk protection scheme for the network is available. Although the SHN’s original plan was to connect with the GOB’s social protection scheme (as a possible platform from which to provide large-scale financial protection to SHN clients through the establishment of a strategic purchasing scheme by the government), this plan has not yet been realised. Additional efforts should be made by SHN to review the current policies and to develop a mechanism for the strategic purchase of services from the government. 51 AUHC Activity in Bangladesh: Monitoring, Evaluation, Research and Learning (MERL) Plan, USAID, 2021 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 23 6. Organisational Issues Three sets of interlinked issues (described below) have affected the results of AUHC deliverables. 6.1 Structure and execution of the AUHC project 6.1.1 A complex layered organisational structure. The AUHC project brought together a number of agencies (the donor, the prime contractor/operator, Chemonics International; the Incubator Team (consisting of the Thinkwell Group, PSI and Chemonics itself), the AUHC project office, the SHN board and the SHN, in that hierarchical order). It is evident from our conversations that major decisions have necessitated the appraisal of and buy-in by a number of these agencies; this has resulted in impositions on management time and has made decision-making processes cumbersome. More than 30 meetings have been held every month to facilitate coordination, as mentioned during an in-depth interview. Being at the bottom of the hierarchical structure mentioned above, it is doubtful how much the SHN could get its voice heard or influence decisions (e.g. with regard to procuring TA according to its own needs. When TA partners failed to deliver (for example, the failure to deliver a fully fledged marketing plan based on the marketing architecture, design and delivery of a comprehensive QA system), the SHN did not, according to this structure, have the power to demand said outputs. The reverse power structure between the SHN and AUHC, which resulted from a considered decision that the SHN would be a ‘sub￾contractor’ to Chemonics, was also detrimental to the development of the SHN as an organisation capable of thinking and working independently. 6.1.2 Coordination and alignment between organisations. There was a good sharing of information between all the concerned agencies. In addition to meetings, project annual reports, annual workplans and quarterly reports from the SHN were used to share information and plans. The reports are, however, difficult to piece together, as the information in the work plans and annual reports does not tie together, making it difficult to assess what progress has been made. There were several instances where TA partners and Chemonics were unable to synchronise their input to deliver complete outputs for the project. Some of the reasons for this ineffectiveness in the early years of implementation were: a lack of communication (or the presence of miscommunication) and a lack of coordination among the key actors (Chemonics, TA partners and the SHN); frequent turnover of Evaluation of USAID-funded Advancing Universal Health Coverage Activity 24 key personnel positions within all concerned agencies; a long hiring process for technical experts; and an inability to retain TA partners’ interest and maintain their efforts to deliver complete outputs. 6.1.3 Shifting grounds. During the project’s implementation, there were several changes made to the key premises upon which the project was based. According to the Revised Technical Proposal (3 Aug 2017), the purpose of the project was to create research￾based health services for the poor, while the scope was to transition the erstwhile Surjer Hashi Network (SHN) to a sustainable, pro-poor social enterprise by introducing new, experimental systems and processes designed to expand health-service delivery to the poor. During the project implementation, however, the concept of a ‘double bottom line’ of financial sustainability and health impact evolved, resulting in the de￾emphasis of service delivery to the poor in the interest of maximising cost recovery. At the same time, the IR on this indicator was not changed, meaning that the project implementers had to muddle through the problem without any specific guidance. Other examples of strategically important changes that affected the project implementation were the decision to expedite the amalgamation of the clinics in a single sweep and the introduction of regionalisation (which was soon after replaced by centralisation). 6.2 Functioning of the SHN 6.2.1 The board of directors: The SHN has been invested with a 10-member board of directors. Two of the members come from a private sector/business background, while the remaining members were drawn from retired and in-office senior government officials, academics and individuals with an NGO background. The board is well meaning and has been supportive of the SHN’s endeavours to achieve its mission. It has supported strengthening compliance, large procurements, senior-level recruitments, etc. through various board subcommittees, which have contributed to the SHN’s development. It has also tried to secure funding from the government. However, the board needs more diversified representation from the private, for profit and innovative startup sector. The board also needs to be more agile, take more risks and make quick decisions if the SHN is to function like a private sector organisation and establish good business practices. It needs to support bringing in stronger commercial sector experience to the organisation – the missed opportunity to do so during the recent recruitment is evidence of the board’s reliance on traditional NGO leadership. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 25 6.2.2 The SHN’s organogram: The network’s organogram is unbalanced. Only a quarter of the 49 team members are directly or indirectly involved in service delivery; all others perform administrative and support functions. The SDSs, who are lifelines for clinics, each have 22 clinics under them, making it untenable for them to give in-depth and/or intensive input to the clinics. Thus, each clinic is visited only twice a year. Although the marketing and programme strategy team has commercial experience, they play a sequestered role, and there does not seem to be room for them to take on a stronger role in the design and functioning of clinics from a marketing perspective. 6.2.3 Leadership capacity: The overall capacity of the senior leadership at the SHN is weak. In the course of several FGDs/interviews with the leadership and second-tier team members of the SHN, it emerged that the leadership is more operational than strategic and relies heavily on AUHC for decision-making. This over-dependence could be detrimental to the development and strengthening of the SHN as an independent organisation. The senior management team is fractious; there are internal feuds that result in team members undermining each other. The lack of commercial experience in key team members means that the operations are being run as before, with a few tweaks to accommodate the changed situation. Moreover, the leadership at the field level does not match current needs. Most clinic managers come from a typical NGO background, are not conversant with private sector culture and thinking, and do not have the drive and competitiveness required for clinic turnaround. 6.3 The SHN’s work environment 6.3.1 Ad hocism: In the course of adjusting to the numerous changes within the project, the SHN has become a reactive organisation, and ad hocism has become ingrained. At AUHC and Chemonics as well, responses to challenges have not always been well thought out, partly due to the need to respond swiftly to emerging challenges as a result of time constraints (e.g. the budget forecasting and management during the clinic amalgamation that resulted in over-expenditure during Years 2/3 was not optimal; installing and soon thereafter dismantling regional teams indicated a lack of forethought) or resource constraints (e.g. withholding marketing activities, clinic renovations and instrument procurement until Year 4).52 52 Years 2 and 3 AUHC Annual Reports Evaluation of USAID-funded Advancing Universal Health Coverage Activity 26 6.3.2 Inefficiencies, irregularities, and corruption: These issues continue to haunt the project. Inefficiencies largely stem from manual record-keeping systems at clinics as well as the head office, the lack of a detailed structured approach to the work of SPs in satellite clinics, and the lack of well-structured clinical forms and use of technology. Irregularities relate to, for example, lapse of licences or management of petty cash. Corruption is a long-standing problem inherited from pre-AUHC days and concerns false claims, service revenue embezzlement, procurement-related irregularities and service number inflation. In Year 3 of the project, 30 disclosures had to be made to the Office of the Inspector General (OIG) on reportable misconducts, resulting in 50 terminations and 20 warnings. 53 Installing a compliance monitoring system through a dedicated compliance team/internal auditor reporting to the board of the SHN along with supporting policies and actions has reduced this trend. Only 12 cases required OIG reporting in the first three quarters of 2021. However, planned activities that could have reduced this trend further (e.g. installation of CCTV cameras, digital payment systems and ethics training of clinic staff) could not be undertaken. 7. Underlying Factors Our interpretation of the key factors which led to the situation described above is discussed in the following subsections. 7.1 A highly complex project designed without adequate consideration of the local context The project aim was rather ambitious – to take over and merge a large number of clinics that had been operating under several NGOs for a number of years under one umbrella. This was done in a single sweep rather than in phases and was based upon a viability assessment, as originally planned. The repercussions from this bold step led to the subsequent litigation and moving away of a number of well-performing clinics from the network. This risk should have been anticipated and appropriate risk management strategies planned. 7.2 Poor fund flow management Project funds were delayed at several points, majorly impacting the project implementation, as several key activities were cut to manage the project with the available funds. The amalgamation of all clinics at once led to huge over-expenditure initially, resulting in significant fund constraints in Years 2 and 3. The last reduction of USD 10 53 Year 3 AUHC Annual Report Evaluation of USAID-funded Advancing Universal Health Coverage Activity 27 million – though rational given the reduced number of clinics in the network as opposed to the number in the original plan – left a number of initiatives incomplete. 54 Appendix 11 outlines the events related to fund flow. 7.3 USAID’s role Overall, USAID’s involvement in the project was intense and extended beyond the project coordination and into the detailed execution of the project. This was the result of how the project contract was designed, with Chemonics being a contractor rather than a grantee. This arrangement gave USAID the leverage needed to closely monitor the project implementation and intervene where necessary. However, this led to frequent meetings between USAID and the project stakeholders; almost every decision was first sense￾checked with USAID, and even purely technical tasks, e.g. finalising a client satisfaction survey form or adopting GOB’s service delivery protocols, were signed off by USAID. 55 The decision to encourage more ‘able to pay’ clients’ and adjust the discounting system that led to a diminishing proportion of poor/POP clients also occurred in the course of such interactions and was taken by the SHN and AUHC as USAID’s tacit approval of this strategy to increase cost recovery. While such close involvement aided the project implementation by securing USAID’s buy￾in to various decisions, it also led to an over-dependence on USAID by AUHC and the SHN. Ratifying decisions with all concerned actors also made decision-making time￾consuming. 7.4 Revolving door leadership There was high turnover in key senior management positions in all agencies concerned during the project implementation. Such frequent changes have been detrimental to the project due to the loss of institutional memory, differences in approaches between individuals and lack of decision continuity. Appendix 12 captures these changes across the main agencies. 7.5 TA deliverables and limited private sector and commercialisation experience This has already been mentioned under the private sector partnership (‘Result 3.2’) and ‘Functioning of the SHN’ (organizational issues - 6.2) sections. 54 AUHC Year 4 Workplan 55 AUHC Year 2 and Year 3 Annual Reports Evaluation of USAID-funded Advancing Universal Health Coverage Activity 28 D. Go-forward Strategic Options Guiding Principles This section of the report provides strategic guidance on potential go-forward options. Strategic guidance is distinct from strategy. A strategy development process requires a detailed review of a programme’s current state, future aspirations, customer needs and influencing factors to arrive at fact- and evidence-based analysis that informs a concrete strategy for an organisation or programme. Strategic guidance, however, is limited to only identifying a few high-level options. It is recommended that these options be explored in greater detail at a later point to arrive at a strategy. In arriving at the options, this evaluation considered the following dimensions: 1. We adopted a mutually exclusive, collectively exhaustive (MECE) framework to identify options. This means an attempt was made to ensure that the options identified were both mutually exclusive and comprehensively exhaustive. However, given the limited time/scope of this assignment, the ‘exhaustiveness’ dimension was at a ‘to the extent possible’ depth level. 2. In considering options, we focused on USAID’s perspective (and not the perspectives of other stakeholders explicitly). 3. Often, complex programmes and organisations are not limited to a single goal or objective and, instead, work with several goals and objectives. This is also the case with AUHC (including ensuring financial sustainability, serving the poor and supporting the overall health systems in Bangladesh). Financial sustainability was used as the primary goal to inform this analysis and identification of the options because USAID’s future financial support of SHN is uncertain within the context of (a) nearly two decades of ongoing funding and (b) a rapidly transitioning middle-income Bangladesh. Thus, it is important for SHN to gain financial sustainability for its existence. The two options presented in the following sections reflect the evaluation team’s collective thinking and are based on detailed analysis and much deliberation. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 29 Strategic Option 1 – Enhanced status quo (ESQ) 1.1 Overview of ESQ option The ESQ option focuses on wrapping up the project in a way that would use the remainder of the project duration to position the SHN as a relatively strong organisation. Under this approach, the SHN would continue to conduct activities with a focus on strengthening services and organisational systems, consolidate the changes introduced in the project and strengthen leadership. To facilitate a smoother conclusion, TA deliverables due from partners would need to be made available as quickly as possible to enable rollout. The senior management would need to be supported by the SHN’s board, AUHC and USAID to work diligently through clear action plans and deliverables. The senior management team with board support would need to take on a role in maintaining morale and drive across the organisation. The relationship with the board would have to be managed in such a way that it does not exacerbate the palpable between the board and AUHC/USAID. Timely, effective and consistent communication from senior leadership would be crucial to ensure a smooth conclusion. A communication plan would need to be developed well before information on the next phase is made public, and an opportunity would need to be created for team members to voice concerns and make themselves heard. Once the decision is made public, the SHN would likely lose team members, who would be trying to find other, more stable job opportunities. It would be necessary to identify and retain the team members important for the next phase of the journey, for which continuous and consistent communication would play an important role. 1.2 Recommendations on SHN priorities for the remaining duration As mentioned previously, clarifying the project approach for the remaining period with regard to the confusion arising from the two seemingly conflicting ends of financial sustainability and services for the poor would be of primary importance. It would be most effective to simply pick one priority and let the organisation work effectively towards that end. The following subsections discuss our suggested priority activities for the remaining project duration. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 30 1.2.1 Streamline clinic activities and services Clinic activities need to be streamlined and focused on a select bundle of services that meet one of the following three requirements: services with which the SHN has experience delivering them well (e.g. MCH and family planning); services that have the potential to generate revenue (e.g. NCDs, pharmacies and lab-based services); or services for which there is a demand within the target communities (e.g. NCDs, pharmacies, lab services or deliveries). Based on the chosen ‘core’ bundle, the SHN could reassess its HR needs in clinics and trim/expand it where necessary. Clinic relocation should be halted during this phase, as these are expensive ventures and will invariably lead to loss of client footfall. Extended clinic hours could be expanded to more clinics since they draw in new clients. Introducing specialists in clinics could be helpful but should not be the priority at this stage given the volume of work that would be needed. 1.2.2 Improve the quality of care Improving the quality of care through establishment of a QA system would have a positive impact on client footfall and clinic visit outcomes. A simple system based on the work already done could be developed with the addition of a quality governance system. The system could consist of setting standards, operationalising SOPs/service delivery protocols, instituting regular quality audits and effective adverse event monitoring (not just reporting), and setting up a high-level quality governance body to oversee the system implementation and, more importantly, pre-empt, mitigate and manage risks. Several components of this system are already in place/developed but need to be operationalised. 1.2.3 Procurement, recruitment and licensing Based on the bundle of services mentioned above, the SHN should revisit the procurement and recruitment list and complete the most critical tasks. All licensing issues need to be addressed as early as possible so that clinics can resume services that have been stalled due to problems with licences/registrations. All procurements should be at the network level to take advantage of greater volume discounts. 1.2.4 Identify and implement priority clinical training Clinical/refresher training should be prioritised in service delivery areas wherein there have been changes in protocols/SOPs. Instead of placing trainees into the ongoing training of other projects/organisations, such as the Integrated Safe Motherhood, Newborn Care and Family Evaluation of USAID-funded Advancing Universal Health Coverage Activity 31 Planning Project, which is slow in process, efforts should be made to undertake bulk training through specialised agencies so that the training is effective and timely. 1.2.5 Strengthen the relationship with the government This should be a key focus for this period. There is still a certain level of goodwill for the SHN among both wings of the Ministry of Health and Family Welfare (MOHFW)-DGFP and Directorate General of Health Services (DGHS). Immediate initiatives should be undertaken to rebuild the relationship with the DGFP so that required approval and commodity supplies can be reinstituted. Simultaneously, opportunities for financing some of the family planning work through the operational plans could be explored. With the DGHS, there is room to actively pursue becoming a part of the World Bank–financed urban health project56 and becoming a major player in the prevention, screening and/or treatment of NCDs. The donor situation in Bangladesh is set to change given the country’s plan to graduate to a middle income country by 2026. As many of the current donors exit, the government of Bangladesh could become the major funding avenue for agency like SHN and therefore it is important for SHN to actively strengthen its relationship with the government. 1.2.6 Conduct donor mapping and build relationships The SHN is still attractive to donors by virtue of its size and long experience working in Bangladesh. The present window of about a year could be used to develop and strengthen relationships with bilateral and international private donors. A quick, rough donor mapping exercise, a toolkit to market the SHN to such donors and active networking with potential donors could be initiated during this period. 1.3 Further work in charting the way post 2022 Successful completion of the above recommendations will leave the SHN in a relatively stronger position by the end of 2022 and provide some space for an organisation that has been rocked by changes since its inception to consolidate whatever gains have been made to date. The intervention period should also be used for some deep reflection on the post-2022 era to position the SHN for the future. As part of this positioning SHN should explore opportunities of Public Private Partnership (PPP), collaboration with private sector agencies working on service delivery and innovative financing mechansims including impact investment. 56 This could prove to be a long and convoluted process given what we’ve heard through our key informant interviews – one that might not come to fruition in the remaining time period of the current contract Evaluation of USAID-funded Advancing Universal Health Coverage Activity 32 Strategic Option 2 – Adaptive real transformation (ART) 2.1 Overview of ART option A second option – called ART – embodies a fundamentally different philosophy from the ESQ option. Option ART recognises that many shortcomings of the current contract and its inability to enhance financial sustainability originate from a combination of the lack of locally contextualised strategies, the inadequate capabilities of TA providers in commercialisation, the inappropriate governance structure and a series of inefficient implementations. If these compounding layers of impediments are reconfigured, a financial sustainability goal of 100% is achievable by the end of 2027. This would require a fundamental rethinking of the strategy and added budget allocations and would achieve two very important objectives. First, this would leave behind a far more positive legacy for USAID and its contributions in making the SHN a success story. Second, it would sustain the health network that hundreds of thousands of low-income people have come to benefit from over the decades. 2.2 Time frame and phasing Option ART would require a time commitment up to December 2027, i.e. six years from 2021, including the remaining one year of the current contract. These six years would be divided into three distinct phases, each lasting two years. Phase 1 would involve several reframing activities. First, a locally contextualised strategy would need to be developed. Second, a significant reconfiguration of the senior leadership, core teams and governance structure would be necessary. A key part of this would be replacing the current TA providers with local resources who have extensive experience in commercialisation, the private sector and financial sustainability in addition to strong local networks to get things done within the Bangladesh environment. Furthermore, the SHN board would need to be rethought to include those with the capabilities and networks SHN needs to succeed. The SHN senior leadership would need to undergo a similar reconfiguration, including a rethink of the current chief executive officer (CEO). Lastly, the governance structure would require a near 180-degree overhaul. This is perhaps the only organisation we have seen wherein a client (the SHN in this case) seems to report to external consultants (the AUHC programme with its three TA providers). Such reverse power dynamics gave SHN less space to plan and manage task as the role was taken up by AUHC Evaluation of USAID-funded Advancing Universal Health Coverage Activity 33 as per the contract design. The governance framework must be restructured to have external consultants report and be totally accountable to a strengthened SHN leadership. Phase 2 would entail an arduous transition into a reconstituted structure. This would include, among other things, moving from a largely reactive modus operandi to one that is more strategic and proactive, undertaking a wide number of reasoned locally contextualised initiatives that accelerate revenue growth and rein in costs, implementing significantly more and better structured private sector partnerships that make far more revenue contributions than the current ones and kick-starting other activities based on the new growth strategy developed during Phase 1. Special attention would need to be paid to institutionalising a set of core competencies, competitive advantages and management systems with a long-term perspective. An example of this might be putting in place world-class customer experience that enhances customer loyalty and ensures a greater share of funds for health expenditures. Another example could include implementing a wider variety of technology enablers that help cut costs, expand services and open up revenue monetisation opportunities through data analytics. Throughout Phase 2, the SHN will see cost recovery metrics rise steadily and sustainably towards the established cost recovery (CR) targets. Phase 3 would be designed to primarily focus on eradicating implementation issues within the reconstituted structure while continuing to improve the financial sustainability of the network as it reaches the 100% CR level. This phase would involve expanding the scale of the various initiatives launched during Phase 2 and gradually weaning out external consultant support so that SHN teams can confidently undertake all aspects of SHN operations and growth. An important activity during this phase would be establishing the strategy for the next stage of the SHN’s sustainable growth journey. 2.3 Resources for Option ART A question that might emerge concerns what funding envelop might be required to implement option ART. It is unfortunately not possible to provide that estimate at this stage because the funding envelop would need to be informed by the detailed list of activities required, which would be shaped by the crafted strategy. As indicated earlier, the strategy would be among the first items developed during Phase 1. However, at this stage, it can be indicated that the funding envelop would likely be lower than the budget for the current iteration because option ART proposes a largely local team as opposed to the more costly international team the current project uses. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 34 The two go-forward options outlined in this section differ greatly due to the MECE framework that was used to arrive at them. Their time frames, investments and endpoints are fundamentally different. Critical Success Factors (CSFs) Depending on the path USAID elects to follow, there are several CSFs to consider. These are activities required for ensuring the success of an organisation or programme and include issues vital to an organisation’s current operating activities and future success. 57 For option ESQ, we have identified two CSFs: 1. Big shifts should be avoided. Reshuffling the SHN board, replacing senior management members and significant restructuring of core teams all constitute big shifts. Given the fact that there is limited time between now and the end of the current contract, any such shifts risk demotivating staff across the organisation without the necessary runway to address the organisational issues that would emerge. 2. A dogged focus is required to prioritise activities from the existing activity list and work on only those activities. Time should not be wasted on developing yet another strategic plan. Implementing items from the existing plan should be the goal. For option ART, we have identified three CSFs: 1. Replace the existing TA providers with an experienced local team of TA providers with demonstrated experience in commercialisation, the private sector and transforming social enterprises into financially sustainable entities. 2. Start with the development of a locally contextualised strategy. A new strategy that considers the local context and leverages strong local networks is required. 3. Reconfigure the senior leadership, board and governance structure. Bring in resources who have demonstrated relevant experience in commercialisation, the private sector and social enterprises across the senior leadership and the board. Reframe the governance framework so that external consultants (TA providers) report to SHN senior leadership. 57 The CSFs should not be confused with success criteria Evaluation of USAID-funded Advancing Universal Health Coverage Activity 35 Appendices Evaluation of USAID-funded Advancing Universal Health Coverage Activity 36 Appendix 1 – Results framework Development Objective Improved health and human capital Objective Universal health coverage advanced through a sustainable, pro-poor, social enterprise Result 1 Result 2 Result 3 Result 4 Result 5 The SSN transformed into a centrally managed, sustainable private enterprise Access to and uptake of ESP expanded Coverage under sustainable financial systems expanded to ensure equitable access to health services Improved quality of care Effectiveness of programme implementation increased based upon lessons learned 1.1: The existing SSN consolidated into a centrally managed single entity 1.2: Standardised corporate operating systems designed and implemented 1.3: Systems efficiency and effectiveness of SHN improved 2.1: Enhanced service package offered through SHN 2.2: Increased informed demand for ESP 2.3: Equitable access to SHN services for the poor and the POP ensured 3.1: Business strategies to provide financial protection for the poor and the POP implemented 3.2: Improved financial sustainability of SHN through diverse revenue and funding streams 4.1: Improved customer experience 4.2: Continual quality improvement systems implemented 4.3: SHN staff are skilled and retained 5.1: Learning captured through documentation, research and analysis 5.2: Learning applied to programme activities 5.3: Project learning on UHC disseminated to target audiences Evaluation of USAID-funded Advancing Universal Health Coverage Activity 37 Appendix 2 – SoW Background: Advancing Universal Health Coverage in Bangladesh is a research and development contract of USAID that facilitates the transformation of the SSN to the SHN, as a sustainable, pro-poor social service delivery enterprise. This USAID activity envisions developing and implementing a programme to transform the SSN into a centrally managed and sustainable private social enterprise; adopting proven innovative approaches to create new strategies to expand access to and uptake of essential service packages; developing and implementing sustainable financial systems to facilitate expanded coverage and ensure equitable access to health service; and improving programme strategies drawn from lessons learned. Objective and scope of the assignment: Established in 2017, the five-year AUHC programme is now entering the last stretch of activities, and USAID intends to undertake a high-level mid-term performance evaluation of the programme. This evaluation will be focused on the following main dimensions: ● Assessing the transition progress over the last four years with respect to the revised programme priorities to extract key lessons ● Providing strategic guidance on go-forward options with respect to the programme’s future direction based on the lessons learned and other influencing factors Furthermore, USAID has invited the RDM activity of icddr,b to carry out the evaluation. Thus, the RDM activity will engage a team constituting domain experts relevant to this engagement, including one expert focusing on health services provision and quality, one expert focusing on UHC, and one expert focusing on financial sustainability, private sector partnerships and technology. There will be two more junior-level members to provide overall assistance to the team and one logistics coordinator to provide managerial support for the evaluation. If any other assistance is required by the team, that should be discussed and decided on depending on the need. Evaluation questions: This evaluation will provide a quick analysis of the AUHC activity in terms of the following questions: Evaluation of USAID-funded Advancing Universal Health Coverage Activity 38 1. What has been the overall effect of the optimisation exercise on the project implementation? o To what extent has the planned restructuring taken place? o What are the related barriers to implementation, and what are the plans to overcome these challenges? 2. What are the nuances of service delivery provisions? o To what extent do the present-day services meet the whole range of the ESP as defined by GOB? o What new services have been added to enhance the ESP? o How much are the needs of the poor being addressed? o What is the approach to demand creation, and how is demand creation being addressed? o How well are the demand creation activities/strategies working? 3. What has been the extent of technology use, innovative approach identification and private sector partnership implementation to support financial sustainability? o How is technology deployed as an enabler, particularly EMRs, in the project? o Have any innovative approaches been identified (if yes, then how) and implemented to catalyse financial sustainability? o How well have the project design, structure and implemented private sector partnerships supported financial sustainability? 4. What steps have been taken to ensure/improve the quality of service delivered through the project activities? o What systems are in place to ensure/improve quality service delivery? o What are the strategies for increasing the demand? o What role has customer feedback played in service improvement? o What steps have been taken to improve staff capacity and retention? o What is the effectiveness of the quality monitoring so far? 5. What has been the overall learning experience of the project? o What types of assessment/evaluation/research have been undertaken? o How has the learning from the research been used in strengthening/streamlining the SHN service delivery mechanism, including the restructuring and governance? 6. To what extent have organisational issues affected the overall project management? o What are the project management hierarchies, roles, responsibilities and relationships? Evaluation of USAID-funded Advancing Universal Health Coverage Activity 39 o What are the roles of partners and their deliverables, and what are the team building needs and successes? o What decision-making and coordination mechanisms run the project? o What is the capacity and quality of leadership at different levels of the project implementation? o How does the implementation aspect of decision-making work, and what are the communication dynamics across the project? o What are the capacities and skills/competencies of the various levels of the project team? The evaluation will focus 40% on the transition evaluation (covering the lessons learned based on the challenges faced and ‘what did not work and why’) and 60% on future direction (strategic guidance). The evaluation will examine the transition from a high-level perspective rather than cover the details of integrity and will focus on the learning achieved. The AUHC results framework will be used to assess the AUHC performance so far. In this context, the evaluation will discuss whether the contextual and programmatic factors that were assumed to be in place impacted the results achieved or not achieved. For future directions, considering time constraints, rather than developing a strategy and a detailed understanding of end-user dimensions, a ‘strategy guidance’ document identifying high-level go-forward strategy options will be developed. Methodology: The evaluation will use a qualitative approach to assess the questions listed above. The tools used will comprise the following: a) a document review, b) stakeholder consultations and c) clinic visits. a. Document review: The following documents will be reviewed: ● Project documents ● Service statistics ● Partnership documents ● Annual/periodic reports ● Evaluation/assessment/research reports ● Costing, revenue projection and CR trend documents ● Strategic exercise documents ● Service delivery, training and quality control guidelines Evaluation of USAID-funded Advancing Universal Health Coverage Activity 40 b. Stakeholder consultations: The evaluation team will conduct 20–25 key informant interviews (KIIs) as well as a couple of FGDs with various relevant stakeholder segments. Each of these interviews/discussions will follow a semi-structured questionnaire/discussion guideline. The list of stakeholders is as follows: ● KIIs: i. USAID (Agreement Officer Representative (AOR), alternate AoR and MEL advisor) ii. Chemonics (project lead, clinical service and quality of care lead) iii. AUHC (Chief of Party, finance, business development, clinical service, Quality of care iv. Current partners (PSI, Thinkwell) v. Joint Chief of Planning wing of MOHFW vi. Relevant line directors (CCSDP, MNCH, NCDC, EPI) vii. SHN board members (2/3 interviews) viii. SHN (CEO, head of marketing, business development) ix. Current partners (one pharma, Marico, Praava) x. Previous partners (Ad-Din, Green Delta) xi. Others (SMC, Grameen Kallyan, AccessHealth, etc.) ● FGDs: i. Clinic managers ii. SPs • Doctors • Paramedics iii. AUHC/SHN clinical service delivery team iv. AUHC/SHN clinical service QA team v. AUHC/SHN finance team vi. AUHC/SHN private sector business development team vii. AUHC/SHN technology development team c. Field visit: Due to the current COVID-19 situation, field visits will be limited to Dhaka City. Exceptionally strong COVID-19–compliant measures will be adopted for the visits (safe cars, masks, goggles, gloves, continuous sanitisation, etc.). At least three clinics will be visited based on the following criteria: ● Clinic type (static/satellite) ● Service provision (advanced/basic with NVD/basic without NVD) ● Performance level (good/average/poor) Evaluation of USAID-funded Advancing Universal Health Coverage Activity 41 ● Location (urban/peri-urban/rural) ● Digitisation (presence of EMRs, etc.) Structured checklists and standard guidelines will be followed during these clinic visits, following the points below: ● Service delivery flow (triage system, waiting time, counselling, service provision) ● Quality of service ● Client flow history (for each type of service) ● Performance documents ● Revenues generated ● Branding, marketing and demand creation ● Collaboration with external entities ● Use of EMRs ● COVID-19 safety Suggested lines of enquiry: The suggested lines of enquiry for the evaluation are given below. However, the evaluation team can propose any changes as and when deemed necessary. • IR-1: Explore to what extent the planned restructuring has taken place, the related barriers and the plans to overcome these challenges. • IR-2: Explore how much the present-day services meet the whole range of ESP as defined by GOB, what new services have been added to enhance ESP, how far the needs of the poor are being addressed, what the approach to demand creation is, how demand creation is being addressed and how well demand creation activities/strategies are working. • IR-3: Explore how technology is deployed as an enabler, particularly EMRs. Review if and how innovative approaches have been identified and implemented to catalyse financial sustainability. Review how well the project design, structure and implementation of private sector partnerships have supported the financial sustainability. • IR-4: Determine the systems in place to ensure/improve quality service delivery, strategies for increasing the demand, the role of customer feedback in service improvement, the staff capacity and retention, and the effectiveness of quality monitoring. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 42 • IR-5: Document the types of assessment/evaluation/research undertaken and how the learning from the research had been used in strengthening/streamlining the SHN service delivery mechanism, including the restructuring and governance. • Organisational issues: Determine the project management hierarchies, roles, responsibilities and relationships, roles of partners and their deliverables, team building needs and successes, decision-making and coordination mechanisms, capacity and quality of leadership at different project levels, implementation of decisions, communication across the project, and capacities and skills/competencies at various project levels. Deliverables and timelines Deliverable 1 (by 30 June 2021): An inception report (10- to 12-slide PowerPoint [PPT]) that captures: - Evaluation methodology - Lines of inquiry (broad guidelines to answer each of the evaluation questions) - Stakeholder list - Field visit plan - Evaluation workplan Deliverable 2 (by 31 August 2021): A high-level PPT (12–15 slides) capturing the key highlights from work completed until 24 August 2021: - Field work insights (4–5 slides) - Stakeholder interview insights (4–5) slides - Key findings that will inform strategy options (4–5 slides) Deliverable 3 (on or before 24 September 2021): This deliverable will encompass (after a debrief meeting with USAID and relevant stakeholders) the following: - A Microsoft Word report - An accompanying PPT Team composition: The Evaluation Team will consist of three key staff, including a team leader, an MNCH-FP specialist and an evaluation specialist. It will also have three junior researchers to support the key staff in the evaluation process, including data collection and logistics management. The team members should have extensive knowledge related to health service delivery in Bangladesh, non-profit management, capacity building and financial sustainability, and quality of care, MCH and FP. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 43 Appendix 3 – Document review categorisation list Topic Document Category Project documents • Activity Approval Memorandum – 2021 • NHSDP Milestone_tracker_June2015 • NHSDP Monthly Reports (60) • Broad Agency Agreement (BAA) Bangladesh 2016 • Progress Update February 7, 2021, Meeting of SHN and the DGFP • Contracts and SOW with Chemonics (30) • SHN HQ and Clinic Organogram, 2021 • AUHC Organogram, Updated September 2020 • USAID MERL Results Framework • Technical Proposal-Chemonics and partners | August 3, 2017 • Work plan Y1, Y2, Y3, Y4 • SHN service offerings by new typology • CV of clinic staff (60) • Salary structure of clinic staff • Competitor map (3) • Satellite clinic location map (3) • MOU with Marico • MOU with Praava • MOU with Medipath • New discount policy and pricing for poorest of poor-2019 • Monitoring, Evaluation, Research and Learning (MERL) Plan, October 1, 2017 – September 30, 2022, Contract No. AID-388-C-17-00001 • AUHC PMP Indicators and Interpretation, May 27, 2021 • PO Order-SS IT for Tally Consultancy • Transition plan for Shurjer Hashi Clinics-Thinkwell March 2017 • Due diligence tools 3-15-2018 Thinkwell • HR and administrative policies (15) • Finance and accounting manual 19 August 2018 • Procurement manual 18 September 2018 • SHN service offering by new typology, 2019 • SHN discount policy and pricing for PoP, 2019 • Discount policy for satellite customer, 2019 • Service list and codes (6) • CENTRALIZED PROCUREMENT SYSTEM FOR DRUGS/GOODS/VACCINE PROCUREMENT FOR THE SHN CLINICS-AUG 21, 2019 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 44 Topic Document Category Service delivery protocol • SHN Guidelines for Adverse Event and Complications Management and Pharmacovigilance • Report Format for Obstetric and Perinatal Adverse Events • Adverse event report 2021 • Quality assessment score (4) • Quality Information System (QIS) for SHN • QA System Score Card with Scoring Criteria • Quality Standards in Five Categories at Clinic Level • AUHC Best Practice: Supportive Supervision. Define quality and quality assurance in the context of a clinical setting • Distance QA monitoring report • Quality-related feedback documentation • Quality audit checklist (12) • Clinic Operation Manual-SHN • Price list data of all clinics • SHN price list • Health service delivery checklists (17 checklists) • National guidelines on maternal and child health (10) • Quarterly progress report YEAR Two, QUARTER Four (January–March 2021) • Quarterly Report, Year One Quarter Four January-March 2020 Service statistics • Analysis of data for all the SHN clinics during 2018-2020 • SHN FP services data analysis for year 2020 • Clinic performance data on financial status and service contact 2019, 2020, 2021 (8) • Review of client record (8) • Clinic satellite performance • Referral from CSP and Satellite Centers • Client feedback analysis. • Filled reporting sheet • Service and revenue data for 10 clinics • Pre AUHC-data for 10 clinics-Oct 2015 - Sep 2017 Training • Training need assessment for Y3 (15) • Training Plan Y3, Y4 (5) Partnership documents • SHN Partnership Updates, Feb 02, 2021 • Services Agreement Between PSI AND mPower Social Enterprises Ltd. PO # 11500-0-600 • Partnership Framework for SHN, April 2019 Annual/periodic reports • SHN Year 2-Annual Report Oct 2018-Sep 2019 • SHN Year 3 Annual Report Oct 2019-Sep 2020 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 45 Topic Document Category • NHSDP monthly (45) and annual reports (11) Evaluation/assessment • Bangladesh Smiling Sun Clinic Facility Assessment March 2018 • 2020-03-19 Final 134 Clinic List • BMEL TPM Report - AUHC Jessore 24-26 Feb 2019 • BMEL TPM Report - AUHC Sylhet 19-21 Mar 2019 FINAL • NUPAS Baseline Evaluation of SHN October 2020 • Engagement with Clinics for Declining Performance by SHN from 02-20th May, 2021 • Surveys to inform planning and decision-SHN Clinic Engagement Discussions • HR engagement survey • SHN: Poverty Targeting Functional Review, March 2019 Research reports • Impact of the Bangladesh NHSDP-2019 • NHSDP Impact Evaluation: Preliminary Results-2017 • Literature Review-Designing effective healthcare worker incentive schemes, Social Enterprise Incubator Team, December 2020 • Service delivery innovations in pro-poor, for-profit healthcare enterprises and their potential applicability to the Surjer Hashi Network (SHN) Technical BRIEF December 2020 • Fintech for Health Bangladesh by THE ACCESS Health International • Consultancy Report: Jill Tabbutt-Henry, February 25, 2019 SHN Counseling Assessment--AUHC (USAID) Bangladesh • Market Landscaping Study: April 2019, PSI • Final Report: Functionality and Sustainability Assessment Of CSPs by Capacity Building Service Group (CBSG), April, 2019 • Satellite Spot Assessment of Smiling Sun Network March 2019 by CBSG • SHN: Drug Procurement: Effects of Contract Rollout with Pharmaceutical Companies, December 2020 • Network Optimization Report AUHC, October 2019 • Improving service utilization and quality: perspective from FGD with clinic staff, June 2021 • Design considerations for effective healthcare worker incentive schemes Technical Brief, July 2021 • Service Delivery Innovations in Pro-Poor, For-Profit Healthcare Enterprises, and their Potential Applicability Evaluation of USAID-funded Advancing Universal Health Coverage Activity 46 Topic Document Category to the Shurjer Hashi Network (SHN), Technical Brief, July 2021 • World Bank Urban Health, Nutrition and Population Project (P171144). Concept note • Due Diligence Assessment Process and Tool (DDAPT)- Thinkwell • Desk review finding: CHP and satellite clinics Thinkwell 2018 • EU H&N micro health insurance: concern worldwide • SHN Business Model Development Business Idea: Extended Hour. Prototyping Learning Document, July 2019 • Standardization of SHN’s Clinic Operational Cost: Current Practice and Recommendations. Approach and Findings from Current Practice of Clinic Level Expenses. 4 April 2020 • Drug Procurement: Effects of Contract Rollout with Pharmaceutical Companies, December 2020 • Performance Assessment of SHN’S “Extended Hours” Initiative In 54 Clinics. May 2020 • Service Delivery Innovations in Pro-Poor, For-Profit Healthcare Enterprises, and Their Potential Applicability to the Surjer Hashi Network (SHN) Technical Brief July 2021 • Literature Review: Designing effective healthcare worker incentive schemes, Social Enterprise Incubator Team Dec 2020 • Health Insurance, Bangladesh Context Technical Brief May 2021 Technology • EMR (Reg & Bill) rollout status update • Update on SHN Tally System 2020 • Assessment for Systems – Report, Krithika Mouli and Shubhra Rehman, PSI, May 6th–17th 2018, July 15th– 26th 2018 • HMIS Scale Up Gantt Chart Y1-Y4 • Health Management Information Systems (HMIS) for SHN-AUHC Learning Brief January 2020 • List of clinics operating real time billing • MIS and EMR expenses • Software Requirement Specification (SRS). PROJECT: EMR & HMIS for SHN • SHN HMIS architecture • SHN full EMR system Evaluation of USAID-funded Advancing Universal Health Coverage Activity 47 Topic Document Category Costing, revenue projections and CR trends • Revenue Growth Through Pricing and Discounts • SHN Financial Projection • SHN Financial Projection Summary (based on average of 3-month and 12-month annualised figure) • SHN dashboard cost and revenue • AUCH partner cost breakdown • SHN Financial Sustainability: Financial Projection Estimates, (FY21, FY22 and beyond), Oct 21, 2020 • SHN Revenue Projection: Assumption Validation • Monthly P&L, CAPEX and tech investment • Price data for all clinic V16 • Revenue Growth Through Pricing and Discounts • SHN dashboard service revenue Strategic exercise documents • SHN’s Investment Plan for Y4 and Y5-October 12, 2020 • USAID letter of support of SHN funding request to MOHFW (no date) • Application for grant for SHN under code 2705 of the MOHFW 15 March 2020 • Client focused Information Management System for SHN 2020 • Result 3 Update: Strategic Plan and Financial Protection for the Poor, March 2, 2021 • Financial sustainability, Strategic plan, business initiatives and partnerships, Cluster model, Satellite hub, and CSP, Business analytics-February 2, 2021 • 2 ND Business Strategy Workshop-2019 (6) • Design approach for Cluster Model, Satellite Hub, Satellite team, and CSP, December 22, 2020 • SHN Business Model Development, Cluster Model, Validation Method: Prototyping, June 14, 2021 • Marketing Strategy 2021-2022 • SHN’s Incentive Design • Strategic Plan Update 2021 • PSI Slides for AUHC Business Planning Workshop #1 Katie MacDonald 2018 • A Sustainable Financing Solution for Surjer Hashi Clinic Services August 26, 2018 • Business concept introduction 2018 • Financial Background Analysis Scoping and Clustering 2018 • Review of Relevant Service Delivery Models 2018 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 48 Topic Document Category • Business Model Canvas 2018 (2) • Financial background analysis Cost at different levels 2018 • Prototyping Thinkwell 2018 • Value Proposition Thinkwell 2018 • Consumer Segmentation: Initial information and what we will learn with our primary research 2018 • Market landscaping-Consumer Segments Thinkwell 2018 (3) • Pricing and Revenue-Thinkwell 2018 (2) • Health Care Financing, Costs & Pricing PSI 2018 • Second business strategy workshop report • SHN marketing strategy workshop June 25, 2019 • Functional Review of SHN Poverty Identification and Discount Mechanisms. Jan 21, 2019 • Shurjer Hashi Network Poverty Targeting Functional Review. March 2019 • SHN Value Proposition Development, Business Idea: Extended Hours, Validation Method: Prototyping Target Start: April 2019 • SHN Value Proposition Development, Business Idea: Specialized Services, Validation Method: Prototyping • Strategic plan for SHN 2020-2025 • Financial Sustainability: Financial Projection Estimates (FY21, FY22 and beyond), 21 October 2020 • SHN strategy deck. March 14, 2021 • Impact Investment Roadmap & Action Plan (DRAFT) June 10, 2021 • SHN Impact Investment Action Plan, 18 steps process Evaluation of USAID-funded Advancing Universal Health Coverage Activity 49 Appendix 4 – Insights from SHN clinic staff: managers, doctors and paramedics To compensate for the smaller number of field visits than originally planned, the evaluation team had three separate FGDs with clinic doctors, managers and paramedics of three urban clinics situated outside Dhaka. The findings from these FGDs are summarised below. Transparency ▪ Most of the field-level staff thought the transformation of the SHN had helped improve transparency and led to better governance. ▪ They considered EMRs useful in maintaining transparency, though there were challenges in their use. Training ▪ Within the past 2–3 years, the SHN has not arranged any clinical training for doctors/paramedics. ▪ Some doctors and paramedics have taken different clinical courses on their own. ▪ Decisions on counselling by paramedics/doctors are received positively by clinic staff. ▪ There is no regular orientation on clinic operation manuals and SOPs for doctors. Hard competition in the market ▪ The SHN has faced hard competition with other NGOs and the private sector, which provide similar services at a relatively lower cost. ▪ However, the SHN has goodwill for MCH care, and people still know the SHN as ‘Shobuj Chata’. Less equipped and old-fashioned ▪ Many clinics have relatively older equipment, such as outdated ultrasonogram machines and sucker machines. ▪ Common lab equipment, e.g. doppler ultrasound and analysers, is unavailable. ▪ Clinics have poor infrastructure and furniture, e.g. old plastic chairs and delivery beds. ▪ They also complain about a lack of behavior change communication (BCC) materials for display. ‘During my work at [a] previous NGO, I found [a] lack of transparency and monitoring. Here, everything is online…no question of lack of transparency. The work that is being done now is being done correctly and productively. There is no scope for increasing or decreasing the numbers or entering wrong information’. – Clinic Manager ‘It is my sorrow that I did not get any training from [the] SHN clinic while working here for long periods’. – Doctor Evaluation of USAID-funded Advancing Universal Health Coverage Activity 50 Scarcity of printed materials ▪ The clinics are facing a scarcity of printed materials such as leaflets, different types of cards and even discharge certificates. These are mostly computer printed or photocopied. Partnership for pathological test ▪ District-level clinics are demanding a partnership agreement for pathological tests. Supervision by SDSs ▪ Clinics have been receiving distance monitoring by SDSs throughout the pandemic but do not consider it very effective. Community engagement activities ▪ Community engagement activities have become weaker since the abolishment of the activity of CSPs. Male patients ▪ There is public perception that SHN health services are only for females and children, which may be related to the engagement of female providers in most SHN facilities ▪ However, one SHN clinic that engages a part-time male doctor treats 40% male patients. Discounts for the poor ▪ The providers had a restrictive attitude towards providing discounts to poor patients ▪ Since the restructuring, they have been giving more priority to patients who are able to pay. ‘There [are] some old leaflet[s] inside my bag. I have been doing with those’. – Paramedic ‘In my clinic, 95% [of] cases are female. Shurjer Hashi actually is known for “mother and child care” services. The services we are providing, still majority are MCH related; very rare I get [a] male patient’. – Female Doctor ‘Honestly, distant monitoring is not very effective. If the monitoring was on site, we could have directly been able to communicate properly and see what problems were or were not occurring. This is not possible online’. – Clinic Manager Evaluation of USAID-funded Advancing Universal Health Coverage Activity 51 Appendix 5 – Evaluation questions based on summary findings Eval Q Primary Question Sub-level Questions Responses 1 What has been the overall effect of the optimisation exercise on the project implementation ? To what extent has the planned restructuring taken place? Major transformation of the SSN took place at both the central and clinic levels, and the SHN was established. The SHN board was created, the organogram developed, the recruitment finalised and a new strategic plan developed. The network size was reduced drastically by closing low￾performing static clinics and reorganising the satellite clinics. A plan currently in the process of implementation was developed to strengthen the service provision by upgrading the clinics, recruitment and training of manpower, and expansion of services. A new discounting policy for the poor was developed but has been found to be ineffective. What are the related barriers, and what are the plans to overcome these challenges? The SHN is facing huge challenges during this process of transformation, of which litigation with partner NGOs is the foremost. The matter has been brought to court. Failure to fulfil the licensing requirements of the clinics has ultimately become a barrier in getting support from the government for the family planning services. The SHN has engaged legal experts and consultants to address the above issues. The lack of an effective network with the government and limited engagement with the private sector have been noted as issues in terms of service provision and future sustainability. 2 What are the nuances of the service delivery provisions? How much do the present-day services meet the whole range of ESP as defined by GOB? The NHSDP provides services covering GOB’s five main components (i.e. MNCAH, FP, nutrition, communicable diseases and NCDs) and a sixth component (management of common illnesses). However, not all sub-components under each component are covered. The extent varies according to the clinic type, with advanced clinics providing a wider range than basic ones. The social and behaviour change communication (SBCC) component, which is integrated in all Evaluation of USAID-funded Advancing Universal Health Coverage Activity 52 Eval Q Primary Question Sub-level Questions Responses components, is weak due to a lack of BCC materials, lack of training and lack of plans to operationalise SBCC in satellite clinics. Laboratories, imaging centres and pharmacies that support these services are also not in place in a standardised manner across the board. What new services have been added to enhance ESP? No new service has been introduced across the network. Several new services have been piloted on a miniscule scale (e.g. TB diagnosis), and guidelines have been developed but not operationalised (e.g. GBV identification and referral). How far have the needs of the poor been addressed? The poor have not been addressed as a priority group. The mechanism for identifying poor individuals has been abandoned due to lack of objectivity. The discount system is no longer in place, and discounted services have dramatically reduced. What is the approach to demand creation, and how is demand creation being addressed? How well are demand creation activities/strategies working? There is no defined strategy or plan for demand creation. Local-level demand creation work recently started in phases in around 40 clinics in an ad hoc manner. There are no plans for continuation or follow-up. As of yet, it is too early to assess the impact of this. 3 What has been the extent of the technology use, innovative approach identification and private sector partnership implementation to support Have any innovative approaches been identified (if yes, then how) and implemented to catalyse financial sustainability? Several approaches were identified but could not be classified as innovative. Furthermore, most have not been fully implemented. Cost recovery started off around 50%, and the target is 65%; however, the average CR throughout the four years has been around 48%. There is no evidence that financial sustainability has been achieved. How well has the project design, structure and private Few private sector partnerships have been implemented. The net impact of the private sector partnerships towards financial Evaluation of USAID-funded Advancing Universal Health Coverage Activity 53 Eval Q Primary Question Sub-level Questions Responses financial sustainability? sector partnership implementation supported financial sustainability? sustainability is negligible so far – they contribute 0.1% of the total revenue. This means that every $1,000 in revenue generates only $1 originating from the private sector partnerships. How has technology been deployed as an enabler, particularly EMR, in the project? Technology deployment has taken a rather narrow view focusing only on EMR. This evaluation found that only the R&B components of the EMR system have been deployed at five out of 134 SHN clinics. Other components of the ‘full EMR’ had not been deployed by the time of this study and, therefore, could not be evaluated. Furthermore, the system design and implementation do not seem to have taken the local context fully into consideration. 4 What steps have been taken to ensure/improv e the aspects of the quality of service delivered through the project activities? What systems are in place to ensure/improve quality service delivery? The QA and QI work has happened haphazardly. Along with the QA mechanisms, the quality lead undertook several initiatives initially that could not be continued due to abolishment of these positions. Some elements of a QA system are in place, e.g. SOPs, adverse event monitoring, small-scale client satisfaction surveys and a scoring system for measuring quality. However, other elements are missing (e.g. a quality governance system), not operationalised (e.g. SOPs) or not used to the full extent (e.g. adverse event monitoring). What are the strategies for increasing demand? In the absence of a marketing plan, demand creation work is now taking place through a traditional local-level approach (e.g. miking, leaflet distribution and stickers/banners on public transport) in select clinics. A small marketing budget has been allocated only one year, with no definite commitment of continuity. The effectiveness of the approach is questionable as it is not based on a plan developed with thorough understanding of the target groups and their needs. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 54 Eval Q Primary Question Sub-level Questions Responses What has been then role of customer feedback in service improvement? Customer feedback in Year 3 resulted in changes to the client flow, which reduced the waiting time. Two rounds of scoring in four service areas have demonstrated some improvement, but aside from tracking, there is no other plan for use of the data. What steps have been taken to improve staff capacity and retention? Ambitious training plans have been developed but not fully executed; clinical/refresher training has not yet taken place in the course of the project. It is hard to estimate retention, as most field team members are on short-term contracts. However, from FGDs and interviews, it appears that the majority of SPs are long￾term employees. What is the effectiveness of the quality monitoring so far? Regular quality monitoring happens virtually due to the COVID-19 pandemic, and therefore, its effectiveness is compromised; the frequency is too low (every six months), and scoring is potentially biased due to SDSs scoring their own clinics. 5 What has been the overall learning experience of the project? What types of assessment/evaluati on/research have been undertaken? The SHN and AUHC have conducted several studies on network optimisation, market landscape analysis, poverty targeting reviews, drug procurement optimisation, healthcare worker incentives, service delivery innovations in pro-poor, for￾profit healthcare enterprises and service utilisation at SHN clinics with clinic staff. However, only some of this research has been used for streamlining/strengthening the SHN. How has the learning from the research been used in strengthening/strea mlining the SHN service delivery mechanism, including the The network optimisation study helped plan incremental services from the static clinics and redefine services from the satellite clinics. The market landscape analysis was used to develop a business plan, service packages and a clinic operating system for the SHN. The poverty targeting review helped develop a discounting policy for the poor. However, the application of the other Evaluation of USAID-funded Advancing Universal Health Coverage Activity 55 Eval Q Primary Question Sub-level Questions Responses restructuring and governance? research is either in process or has not been planned yet. Nevertheless, to adapt the research learnings to improve the efficiency of the SHN, a MERL learning framework has been developed. A MERL team has also been engaged for knowledge sharing activation, research communication and learning adaptation. However, this capacity development has been mostly within AUHC, and there is no clear plan for transferring these skills to the SHN. 6 To what extent have the organisational issues affected the overall project management? What are the project management hierarchies, roles, responsibilities and relationships? This is a complex layered project implementation structure, with Chemonics as the prime contractor/implementor working with the incubator team, in-country AUHC project team, SHN board and the SHN – in that order. Moreover, USAID’s close involvement in the project adds another layer. The power dynamics are the reverse of what they should be, with the SHN having a high dependency on AUHC and little room to influence anything. What are the roles of partners and their deliverables, and what are the team building needs and successes? Of the two TA partners, TW has delivered a number of products, but not all products have been helpful. Moreover, PSI has failed to deliver the marketing plan and full￾fledged QA/QI system. It has also delivered only part of the EMR to date; the rest of the EMR is planned to be delivered at the end of Year 4 of a five-year project. What are the decision-making and coordination mechanisms that run the project? Meetings are the instruments for decision￾making and coordination. There have been more than 30 meetings every month, and therefore, information sharing is extensive. Moreover, AUHC and the SHN collaborate and coordinate closely; there are indications of a lack of effective management by Chemonics regarding its relationship with TA partners. For example, PSI could not deliver its final outputs in two cases. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 56 Eval Q Primary Question Sub-level Questions Responses What is the capacity and quality of leadership at the different levels of project implementation? Frequent changes in leadership among all agencies have affected the smooth implementation of project activities. The SHN senior leadership’s capacity is weak, with an overdependence on the AUHC in￾country office and a lack of commercial experience. The field-level leadership is also weak and lacks understanding of the private sector approach. How does the implementation aspect of decision￾making work, and what are the communication dynamics across the project? Most decision-making takes place jointly through meetings; USAID is present in many of these meetings and influences both operational and strategic decisions. Implementation happens through the SHN with the close collaboration and oversight of AUHC. In Years 3 and 4, several vital responsibilities were transferred to the SHN (e.g. compliance and quality of care). What are the capacities and skills/competencies of the various levels of the project team? The capacity at the leadership level has been mentioned above. At the service delivery level, the clinical capacity has been compromised by a lack of clinical/refresher training. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 57 Appendix 6 – Force field analysis Factors affecting Result 01 – The SSN transformed into a centrally managed, sustainable private enterprise Factors affecting Result 02 – Access to and uptake of expanded ESP Evaluation of USAID-funded Advancing Universal Health Coverage Activity 58 Factors affecting Result 03 – Financial sustainability Factors affecting Result 03 – Private sector partnership Evaluation of USAID-funded Advancing Universal Health Coverage Activity 59 Factors affecting Result 03 – Technology Factors affecting Result 04 – Improved quality of care Evaluation of USAID-funded Advancing Universal Health Coverage Activity 60 Factors affecting Result 05 – Effectiveness of programme implementation increased based upon lessons learned Organisational issues Evaluation of USAID-funded Advancing Universal Health Coverage Activity 61 Appendix 7 – Services available in Smiling Sun clinics at close of NHSDP (N = 399)58 Services % of Health Facilities with Available Service(s) Child vaccination services, either at facility or outreach 80.5% (321) Growth monitoring services, either at facility or outreach 100% (399) Curative care services for children under five, either at facility or outreach 100% (399) Family planning services, including modern methods, fertility awareness methods, and male or female surgical sterilisation 98.7% (394) Antenatal care services 100% (399) Normal delivery 23.8% (95) Diagnosis, treatment prescription or treatment follow-up for TB 18.8% (75) Diagnosis or management of NCDs, specifically diabetes, cardiovascular disease and chronic respiratory conditions in adults 89.5% (357) Caesarean delivery (caesarean section) 11.8% (47) Laboratory diagnostic services, including any rapid diagnostic testing 89.2% (356) Blood grouping and typing services 88.2% (352) Blood transfusion services 4.3% (17) Postnatal (newborn) care 99.7% (398) Adolescent health services 100% (399) Nutrition services 99.7% (398) 58 Rachel Jean-Baptiste, PhD, MPH, Oxford Epidemiology Services LLC et al; Bangladesh Smiling Sun Clinics, Facility Readiness Assessment March 2018 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 62 Appendix 8 – SHN service offerings by new typology Services Typology Advanced Clinic Basic Clinic Satellite Clinic Pap-smear Post abortion care New-born care Post-delivery care (eclampsia /hypertension/ diabetes) C-section deliveries Normal vaginal deliveries Essential new-born care First aid and minor injuries Cervical cancer screening by VIA and referral Adolescent RH counselling and treatment TB screening (verbal) and referral TB screening, diagnosis, treatment, DOTS (directly observed therapy) and referral STI/RTI screening (verbal) and referral STI/RTI screening and treatment GBV screening, treatment and referral Nutrition counselling and correction (mild and moderate malnutrition) ANC PNC FP (short-term method) IUD, Implanon, PM, Postpartum FP IMCI GMP LCC (common cold, fever, diarrhoea, gastro-intestinal disorder, scabies) Fistula screening (verbal and referral) Conjunctivitis Refractive error EPI and other vaccinations NCD screening, treatment (diabetes, hypertension, COPD) Dispensary Pharmacy (licensed and certified) Lab diagnosis Lab diagnosis series (+) Lab diagnosis series – Limited lab (pregnancy test, blood Evaluation of USAID-funded Advancing Universal Health Coverage Activity 63 Services Typology Advanced Clinic Basic Clinic Satellite Clinic (hormonal assays) limited lab (+) (CBC, LFT and others**) grouping, HbsAg, RBS, urine for sugar/albumin) USG Imaging – X-rays, ECG * Depends on the service demand, resource availability, staff and equipment, and training. ** Depends on the clinic’s capacity in terms of technical staff and equipment. Note: Referral to be done if any complications arise or require higher management. Evaluation of USAID-funded Advancing Universal Health Coverage Activity 64 Appendix 9 – Field visit impressions The evaluation team visited two clinics in Aftabnagar and Hazaribagh in Dhaka City on 24 and 28 June 2021. Though the original plan was to visit 5–8 clinics, due to COVID-19 situation, we had to restrict the visit to two clinics. Clinics – Setup and typology The clinics we visited were called Aftabnagar and Hazaribagh Clinics. Aftabnagar Clinic was situated in a middle-income private residential area in North Dhaka City Corporation. Hazaribagh Clinic was located in a lower-middle income setting in South Dhaka City Corporation. Both clinics were easily accessible, though we could not find any signs indicating the direction of the clinics. Aftabnagar Clinic was set in a rented private multi-storey building in which three floors were dedicated to clinic services. Hazaribagh Clinic was placed in a rented three-storey building of Dhaka South City Corporation. The clinics signboards were not very visible during broad daylight (Figure 1). However, the inside of the clinics was quite clean and well ventilated, with spacious waiting areas, adequate sitting arrangements and available drinking water. The citizen charters were clearly visible, and a complaints box had been placed in the waiting area (Figure 1). Figure 1: Front view and waiting area of SHN Aftabnagar Clinic, Dhaka Aftabnagar Clinic was an advanced clinic providing 24/7 emergency services for delivery care, and Hazaribagh Clinic was a basic clinic offering outpatient care in the daytime only (Table 1). In Aftabnagar Clinic, evening outdoor services and provision of consultation with specialist doctors had been initiated recently. Among other services, ultrasonography, GeneXpert for TB and common pathological tests were available in both clinics. Arrangements for advanced pathological tests had been made available through collaboration with other private diagnostic Evaluation of USAID-funded Advancing Universal Health Coverage Activity 65 facilities. Regarding the discounted service charges for poor patients, in Aftabnagar Clinic, only 1% of the clients were offered this benefit, while in Hazaribagh Clinic, no discounts had been offered within the month prior to our visit. While assessing the use of technology in EMR, in both clinics, only the billing and registration module of the EMR was used; smooth running of the system was not possible due to the slow internet connection. In Hazaribagh, this problem was further aggravated by a lack of computers and irregular electricity. Moreover, in Hazaribagh, record keeping was a double burden as the staff needed to maintain both paper-based records and enter the data into the EMP after office hours. Table 1: Characteristics of the Aftabnagar and Hazaribagh Clinics regarding set-up and functionality Characteristic Aftabnagar Hazaribagh Clinic set-up and HR Location Urban Urban Type Advanced Basic Full-time staff 36 14 Satellite clinics 3 3 Services Service contact, May 2021 9,180 4,006 Evening outdoor service Yes No MCH services ANC, PNC, NVD, C/S, EPI ANC, PNC, EPI FP services Pill, condom, injection, IUD Pill, condom, injection No trained provider for IUD insertion Other services USG, GeneXpert for TB, lab test, limited general healthcare USG, GeneXpert for TB, lab test, limited general healthcare Collaboration MOU with Praava for lab test MOU with Medipath for lab test Discount for poor in May 2021 1% 0% Technology EMR installed R&B only (on/off) R&B only (on/off) Challenges with using EMR – Staff not fully conversant with the system – Slow internet – Double burden (use paper form and enter data offline) – Slow internet – Electricity problems – Lack of computers Financial sustainability CR in May 2021 72 % 51% Evaluation of USAID-funded Advancing Universal Health Coverage Activity 66 Appendix 10 – Samples of clinic stationery/forms Prescription Postnatal care and newborn care checklist Discharge certificate Evaluation of USAID-funded Advancing Universal Health Coverage Activity 67 Appendix 11 – Key events related to fund flow/budget Month and Year Events October 2017 Project starts April 2018 SHN board is put in place May 2018 In the context of the development and final submission of the final version of the AUHC Year 1 work plan, a decision is made to accelerate the transition of the NGO clinics to SHN January 2019 Rapid transition is completed August 2019 Funding crunch due to over-expenditure in Year 1 occurs: clinic upgrade, branding and procurement put on hold October 2019 Existing funding to stretch for five more months, further scale-down of project expenditure, including training, server procurement, other procurement and recruitment November 2019 Chemonics projects service interruption due to lack of funding Dec 2019 USAID secures USD 1 million to avoid service interruption – obligation received in March 2020 March 2020 COVID-19 hits August 2020 USAID informs of budget optimisation resulting in USD 10 million reduction November 2020 Chemonics submits modified contract to accommodate decreased funding ‘A number of the activities, staffing decisions, and procurements described in this year’s work plan are held over from Year 2 and Year 3 due to incremental funding constraints in the mission that have impacted implementation throughout the past two years. These include vital upgrades to SHN clinic premises, needed clinical training courses for SHN clinic staff, printing of all-new clinic signage and directional signs, procurement of new medical instruments and equipment for SHN clinics, marketing investments and payment of salary arrears to Swanirvar staff that USAID and AUHC agreed to pay in light of Swanirvar’s frozen bank accounts. Funding constraints remain an issue in Year 4, where the budget is almost half of what it was in Year 3. While a funding reduction was expected for Years 4 and 5, AUHC and SHN did not expect this degree of reduction and will need to manage this situation carefully and make some difficult decisions regarding what to prioritize given the competing demands to best position SHN onto the path towards sustainability’. AUHC Workplan Year 4 Evaluation of USAID-funded Advancing Universal Health Coverage Activity 68 Appendix 12 – Changes in leadership Month and Year Organisation Person December 2019 Change in USAID contracting officer's Representative (COR) for this activity Dr Sukumar Sarkar to Dr Pushpita Samina September 2021 Change in USAID COR for this activity Dr Pushpita Samina to Dr Fida Mehran September 2019 Change in USAID Contracting Officer (CO) for this activity Dion Glisan to Kathlyn Bryant December 2019 Change in USAID CO for this activity Kathlyn Bryant to Abdullah Akbar December 2020 Change in USAID CO for this activity Abdullah Akbar to Andrew Holland (acting) February 2021 Change in USAID acting CO for this activity Andrew Holland to Nathan Hilgendorf (acting) June 2021 Change in USAID acting CO for this activity Nathan Hilgendorf to Andrew Holland/in July Howard Weston/now TBD July 2019 Change in USAID health office director Caroll Vasquez to Xerxes Sidhwa August 2021 Change in USAID health office director Xerxes Sidhwa to Carrie Rasmussen October 2019 Change in AUHC PMU director (Chemonics) Anne Boyle to My Di Le (acting) April 2020 Change in AUHC PMU director (Chemonics) My Di Le (acting) to Nathalie Albrow May 2020 Change in leadership (AUHC) – COP Jim Griffin to Parvez Asheque February 2020 Change in leadership (SHN) – CEO Ashfaq Ahmed to Abdul Motin (acting) October 2020 Change in leadership (SHN) – CEO Abdul Motin to Rosetta Haque (acting) December 2020 Change in leadership (SHN) – CEO Rosetta Haque (acting) to Shaila Parvin February 2019 Change in leadership (Thinkwell) Thinkwell Incubator (Parvez) Director left TW to join Chemonics as senior technical advisor November 2019 Change in leadership (Thinkwell) Dr Mursaleena appointed as TW’s new programme director for AUHC, replacing Tapley Jordanwood Evaluation of USAID-funded Advancing Universal Health Coverage Activity 69 Appendix 13 – List of meetings and stakeholders # Org # of meet ings Date Duration (Mins) # Key participants 1. 1 USAID 8 3 June 2021* 90 2 Dr Kanta Jamil, Dr Pushpita Samina 2. 17 June 2021* 120 1 Dr Kanta Jamil 3. 20 June 2021* 120 2 Dr Kanta Jamil, Dr Pushpita Samina 4. 1 July 2021* 90 3 Dr Kanta Jamil, Dr Pushpita Samina, Dr Fida Mehran 5. 2 14 July 2021* 120 3 Dr Kanta Jamil, Dr Pushpita Samina, Dr Fida Mehran 6. 3 27 July 2021* 90 4 Dr Kanta Jamil, Dr Pushpita Samina, Dr Fida Mehran, Mr Xerses Sidhwa 7. 4 24 Aug 2021* 120 5 Dr Kanta Jamil, Dr Pushpita Samina, Dr Fida Mehran, Ms Miranda Backman, Ms Carrie Rasmussen 8. 5 31 Aug 2021* 90 18 Dr Kanta Jamil, Dr Pushpita Samina, Dr Fida Mehran, Nathalie Albrow, Dr Bruno Bouchet, Dr Bazghinawerq Semo, Andrea Bare, Mahfuza Rifat, Eric Seastedt, Shaila Purvin, Roseta Haque, Mohammad Raisul Haque, Fazle Karim, Joseph Urban, Julie Becker, Sharifa Ahmed, Miranda Backman, Parvez Asheque 9. 6 AUHC 10 22 June 2021# 90 8 Parvez Asheque, Fazle Karim, Rizwan Akter, Rakibuzzaman, Tanzina Ahmed, Mahfuza Rifat, Syed Nazibullah, Umme Hany 10. 7 5 July 2021# 90 7 Parvez Asheque, Md Rakibuzzaman, Joseph Urban, Roseta Haq, Shaila Purvin, Syed Najibullah, Mosarraf Hossain Evaluation of USAID-funded Advancing Universal Health Coverage Activity 70 # Org # of meet ings Date Duration (Mins) # Key participants 11. 8 6 July 2021# 90 5 Parvez Asheque, Roseta Huq, Shaila Purvin, Monir Hossain, Syed Najeeb 12. 9 7 July 2021# 60 3 Parvez Asheque, Rezwan Akhter, Shaila Purvin 13. 1 0 11 July 2021# 60 1 Parvez Asheque 14. 1 1 13 July 2021# 90 6 Tanzina Ahmed, Syed Nazibullah, Umme Hany, Shahnaz, Sohel, Nazvi 15. 1 2 13 July 2021# 90 2 Parvez Asheque, Syed Najibullah 16. 1 3 14 July 2021# 60 5 Tanzina Ahmed, Umme Hany, Shahnaz, Sohel, Nazvi. 17. 1 4 1 Aug 2021# 60 1 Parvez Asheque 18. 1 5 9 Aug 2021 60 1 Parvez Asheque 19. 1 6 SHN 12 24 June 2021# 90 7 Shaila Purvin, Dr Md. Raisul Haque, Rosetta Haque, Nd Sohel Rana, Mushfiqul Alam, Ferdous Moin Khan, Parvez Asheque 20. 1 7 1 July 2021# 90 3 Shaila Purvin, Dr Md Raisul Haque, Rosetta Haque 21. 1 8 5 July 2021# 90 1 Rupali Chowdhury 22. 1 9 6 July 2021# 60 1 MM Reza 23. 2 0 8 July 2021# 120 1 Roseta Huq 24. 2 1 11 July 2021# 60 1 Monirul Hassan 25. 2 2 11 July 2021# 60 1 Shaila Purvin 26. 2 3 12 July 2021# 90 7 Dr Tanha, Dr Rozina, Dr Rushan, Dr Mostafa Al Naim, Dr Aotunu Bhattacharia, Dr Ahmed Iftekhar, Dr Mohammad Raisul Haque Evaluation of USAID-funded Advancing Universal Health Coverage Activity 71 # Org # of meet ings Date Duration (Mins) # Key participants 27. 2 4 13 July 2021# 30 1 Mohammad Ali, chairman 28. 2 5 27 July 2021# 90 3 Dulal Roy (Panchagor, Debiganj), Ashif Iqbal (Gazipur), Baby Saha (Chandpur) 29. 2 6 27 July 2021# 90 3 Dr Farzana Begum, Dr Farzana Mazumdar, Dr Swapurno 30. 2 7 28 July 2021# 90 8 Shobita, Dolna, Mahmuda, Parvin (Gajipur), Sharmin (Panchagor, Debiganj), Parvin, Shanjida, Tulshi (Chandpur) 31. 2 8 Chemonic s 2 1 July 2021# 90 2 Nathalie Albrow, Dr Bazghinawerq Semo, 32. 2 9 26 July 2021# 90 2 Dr Bruno Bouchet, Dr Bazghinawerq Semo, 33. 3 0 Thinkwell 1 21 July 2021# 80 3 Mursaleena Islam, Yogesh Rajkotia, Andrea Bare 34. 3 1 PSI 1 5 Aug 2021# 90 2 Rezwan Akhter, Eric Seastedt 35. 3 2 DGHS 2 13 July 2021# 30 1 Dr Md Shamsul Haque 36. 3 3 4 Aug 2021# 45 1 Prof. Dr Robed Amin 37. 3 4 DGFP 1 8 Aug 2021# 30 1 Dr Nurun Nahar 38. 3 5 SMC 1 5 July 2021# 60 1 Md Ali Reza Khan 39. 3 6 Access Health 1 12 July 2021# 45 1 Dr Monica Mittal 40. 3 7 Former COR 1 12 July 2021# 60 1 Dr Sukumar Sarker 41. 3 8 Square Pharma 1 7 July 2021# 60 1 Saiful Murtoza 42. 3 9 Ad-din 1 7 July 2021# 60 1 Dr Muhammod Abdus Sabur 43. 4 0 Green Delta 1 8 July 2021# 90 1 Shubasish Barua Evaluation of USAID-funded Advancing Universal Health Coverage Activity 72 # Org # of meet ings Date Duration (Mins) # Key participants 44. 4 1 Grameen Kalyan 1 8 July 2021# 60 1 Moin Chowdhury 45. 4 2 Praava 1 8 July 2021# 45 1 SM Nazmul Haque 46. 4 3 Concern Worldwide 1 11 July 2021# 60 3 Mary Rashid, Ahsan Ahmed, Mussarat Jahan 47. 4 4 bKash 1 11 July 2021# 60 3 Md Tariqul Islam Khan, Mr Al Amin, Mohammad Nur 48. 4 5 Marico 1 13 July 2021# 60 1 Rohan Ahmed 49. 4 6 World Bank 1 4 Aug 2021# 30 1 Iffat Mahmud 50. 4 7 Daktar Khana 1 5 Aug 2021# 60 2 Dr Rothindra Nath, Dr Nil Ahsan 21 50 144 * Senior management of RDM, icddr,b and the Evaluation Team were present. # The Evaluation Team was present.