Agreement No.: AID-614-A-14-00006 10 April 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Banyan Global for the Saving Maternity Homes in Ghana, Innovate for Health Project. SAVING MATERNITY HOMES IN GHANA, INNOVATE FOR HEALTH INTERNAL END OF PROJECT EVALUATION REPORT September 2017 – January 2018 SAVING MATERNITY HOMES IN GHANA, INNOVATE FOR HEALTH INTERNAL END OF PROJECT EVALUATION REPORT September 2017 – January 2018 Agreement No.: AID-614-A-14-00006 Submitted to: Juliana Pwamang AOR USAID Ghana DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation i PROJECT OVERVIEW Name Saving Maternity Homes in Ghana, Innovate for Health Agreement Number AID-614-A-14-00006 Start Date September 9, 2014 End Date February 28, 2018 Sub-grantee Ghana Registered Midwives Association (GRMA) Geographic Coverage Brong Ahafo, Central, Eastern, Northern, Volta, Western Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation ii CONTENTS Acronyms............................................................................................................................................................................ iii Tables................................................................................................................................................................................... iv Figures................................................................................................................................................................................. iv Executive Summary ...........................................................................................................................................................1 Introduction....................................................................................................................................................................1 Purpose Of The Internal End Of Project Evaluation ............................................................................................1 Project Objectives, Theory Of Change And Evaluation Questions..................................................................2 Methodology ..................................................................................................................................................................2 Findings............................................................................................................................................................................3 Conclusions And Recommendations .......................................................................................................................5 1. Introduction...............................................................................................................................................................7 1.1 Country Context.............................................................................................................................................7 1.2 Overview Of The Smh Project ....................................................................................................................7 1.3 Purpose Of The Internal End Of Project Evaluation...............................................................................8 2. Project Objectives And Theory Of Change.....................................................................................................10 2.1 Project Objectives.........................................................................................................................................10 2.2 Theory Of Change ........................................................................................................................................10 3. Evaluation Questions.............................................................................................................................................13 4. Evaluation Methodlogy ..........................................................................................................................................14 4.1 Tools And Instruments................................................................................................................................15 4.2 Sampling Strategy...........................................................................................................................................16 4.3 Ethical Considerations..................................................................................................................................17 4.4 Analysis............................................................................................................................................................17 4.5 Limitations.......................................................................................................................................................18 5. Findings......................................................................................................................................................................19 5.1 Increased Awareness Of The Benefits Of Business Continuity And Financial Sustainability ......19 5.2 Successful Completion Of Ownership/Mangement Transformation................................................23 5.3 Improved Financial, Sustainability Operational Efficiency, And Quality Of Clinical Services......29 5.4 Expansion Of Transformed Pmhs..............................................................................................................37 6. Conclusions And Recommendations.................................................................................................................41 Annex 1: Evaluation Scope Of Work..........................................................................................................................43 Annex 2: Results Framework .......................................................................................................................................47 Annex 3: Ghana Health Service Ethical Approval....................................................................................................48 Annex 4: Data Collection Tools And Instruments..................................................................................................50 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation iii ACRONYMS BCSP Business Continuity and Succession Planning BFS Business Formalization Score BSS Business Strengthening Support BUSAC Business Sector Advisory Challenge CCB Clinical Capacity Building CDCS Country Development Cooperation Strategy DEA Data Envelopment Analysis EOI Expression of Interest ERC Ethics Review Committee FP Family Planning GEM General Elminiation Methodology GHS Ghana Cedis GoG Government of Ghana GRMA Ghana Registered Midwives Association IDI In-Depth Interview KII Key Informant Interview MNCH Maternal and Neonatal Child Health NHIA National Health Insurance Authority NHIS National Health Insurance Scheme PMH Private Maternity Homes PSV Peer Support Visit QAT Quality Assessment Tool SHOPS Strengthening Health Outcomes in the Private Sector SMH Saving Maternity Homes in Ghana, Innovate for Health SMS Short Message Service TA Technical Assistance USAID United States Agency for International Development USD United States Dollar Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation iv TABLES Table 1. Respondents by Region……………………………………………………………………………3 Table 2. Distribution of Tools and Instruments at Endline ………………………………………………15 Table 3: Improvements in Business and Financial Practices………………………………………………20 Table 4. PMH Owner/Manager Participation in the Project’s Business Trainings…………………….......21 Table 5. PMH Owner/Manager Participation in the Sensitization Events and Activities…………………22 Table 6: SMH Transformations by Type, Disaggregated by Region………………………………………24 Table 7: Pre-existing Desire to Grow or Pre-Existing Demonstration of Growth of a PMH…………....27 FIGURES Figure 1. SMH Theory of Change…………………………………………………………………………...2 Figure 2. Levels of Project Assistance……………………………………………………………………....7 Figure 3. Increased Awareness of the Benefits of Business Continuity and Financial Sustainability……...19 Figure 4. Improved Business and Financial Practices……………………………………………………...19 Figure 5. Increased Awareness of Business Continuity and Financial Sustainability……………………....23 Figure 6. Levels of Motivation to Expand or Grow PMH Prior to the SMH Project……………………..26 Figure 7. Percentage of PMH Owners Over the Age of 55………………………………………………28 Figure 8. Contribution to the Successful Completion of a Transformation……………………………....28 Figure 9. Successful Completion of an Ownership/Management Transformation………………………..29 Figure 10. Average Total Business Formalization Score…………………………………………………..30 Figure 11. Average Total Quality Assessment Tool Scores………………………………………………30 Figure 12. Contribution to the Improvements in Financial Sustainability………………………………....35 Figure 13. Improvements in Financial Sustainability……………………………………………………….36 Figure 14. Contribution to the Improvements in Quality of Clinical Services…………………………....36 Figure 15. Improvements in Quality of Clinical Services………………………………………………….37 Figure 16. Average Number of Services Provided………………………………………………………...38 Figure 17. Average Number of Beds……………………………………………………………………....38 Figure 18. Contribution to the Expansion of Transformed Facilities……………………………………..40 Figure 19. Expansion of Transformed Facilities…………………………………………………………....40 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 1 EXECUTIVE SUMMARY INTRODUCTION This report presents the findings from the Saving Maternity Homes in Ghana, Innovate for Health project’s internal end of project performance evaluation. The USAID-funded $1.78 million Saving Maternity Homes in Ghana Innovate for Health (SMH) project, implemented by Banyan Global in partnership with the Ghana Registered Midwives Association (GRMA), focused on revitalizing private maternity homes (PMHs) in Ghana to expand access to quality maternal, newborn, and child health (MNCH) services. The SMH project, which ran from September of 2014 until February of 2018, targeted six regions within Ghana: Brong Ahafo, Central, Eastern, Northern, Volta and Western. The Saving Maternity Homes in Ghana project was funded through USAID/Ghana’s Innovate for Health Annual Program Statement, which provided funding to “research, introduce and scale-up innovative solutions to key health challenges”, developing “innovative approaches to be tested through operations research”.1 Ghana is experiencing rapid economic growth, yet a large development gap exists between rural and urban areas, higher- and lower-income communities, and in key development indicators, including MNCH, that raise concerns about equity and the long-term sustainability of improvements. Private maternity homes have historically been an important provider of maternal health services in rural and underserved areas. Despite the efforts of the Government of Ghana (GoG) and the donor community, the PMH sector has experienced a steady decline over the past decade, with many PMH’s closing or reducing services as private midwife owner’s neared retirement age or passed away. The SMH project aimed to help address some of the challenges facing the PMH community and ensure continued access to MNCH services in underserved communities by introducing ownership or management transformations, an innovative approach, to revitalizing PMHs. Transformations bring new resources and personnel to a PMH through a sale of the PMH, leasing out the facility, bringing on a new partner, bringing on a new manager, or bringing on an employee with a new skill set. The project raised awareness of the benefits of succession planning and transformations among PMHs and directly facilitated transformations in the target regions. Participating PMHs received various “tiers” of project support for effectively implementing a transformation. Ultimately, the SMH project’s intervention aimed to increase access, improve efficiency and expand quality MNCH and family planning (FP) services in underserved areas. PURPOSE OF THE INTERNAL END OF PROJECT EVALUATION The internal end of project evaluation (evaluation), conducted from FY17 Q4 through FY18 Q1, was designed to generate findings related to the effectiveness and impact of project activities along with lessons learned for the purpose of improving, replicating, or scaling-up this innovative intervention. The most reliable approach to determining and attributing impact is an impact evaluation conducted by an external party per USAID’s Evaluation Policy. Unfortunately, resources were not budgeted for an external impact evaluation, using the methods necessary to create a credible or rigorously defined counterfactual. However, given the innovative nature of the project design and the learning mandate built into the Innovate for Health mechanism, the SMH project has conducted a limited internal evaluation using less rigorous, but nonetheless sound, methods. The primary target audience of the evaluation is USAID/Ghana staff. Additionally, the evaluation will be of interest to other implementing partners, the GoG, public health organizations, academics and research institutions, GRMA, and PMHs. 1 Annual Program Statement, APS-641-13-000001 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 2 PROJECT OBJECTIVES, THEORY OF CHANGE AND EVALUATION QUESTIONS The assistance provided by the SMH project sought to develop the capacity of PMHs in business continuity planning, leverage financing and investment to improve quality and expand service delivery, strengthen the business and clinical capacity of PMH owners, develop the capacity of GRMA, and conduct operations research to understand key issues and the effectiveness of the project’s design. The SMH project theory of change (see Figure 1) focused on the operational and management efficiency, business and financial sustainability and quality improvements expected to result from ownership/management transformations. Figure 1. SMH Theory of Change In response to the theory of change, the evaluation sought to answer the following evaluation questions: 1. To what extent, if any, have SMH project activities increased the awareness of the benefits of business continuity and financial sustainability of PMHs in the target regions? 2. To what extent, if any, have SMH project activities led to the successful completion of ownership/management transformations for PMHs in the target regions? 3. To what extent, if any, have SMH project activities led to improved financial sustainability, operational efficiency and quality of clinical services of transformed PMHs in the target regions? 4. To what extent, if any have SMH project activities led to the expansion of transformed PMHs in the target regions? METHODOLOGY The project employed a modified General Elimination Methodology (GEM),2 a non-experimental methodology that focuses on systematically identifying and eliminating alternative causal factors that could explain an observed result. Drawing on both qualitative and quantitative data collection tools and instruments to inform each of the evaluation questions, the project gained an understanding as to whether the desired results occurred. Additionally, using proportional piling, a participatory research method, the evaluation determined the extent the SMH causal factors contributed to the observed results as compared to non-SMH causal factors. 2 For an overview of GEM, see Howard White and Daniel Phillips, “Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework,” International Initiative for Impact Evaluation, Working Paper 15, 2012. 1. PMH awareness of the beneficial impacts of business continuity and financial sustainability increased 2. Ownership and management transformations completed 3. Operational efficiency, business and financial sustainability, and quality of clinical services in transformed facilities improved 4. Transformed facilities expanded Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 3 The 75 non-transformed facilities and 22 transformed facilities surveyed to inform this evaluation were purposively sampled as the evaluation targeted all operating facilities within the project’s six target regions, as demonstrated in Table 1 below. Table 1: Respondents by Region Total No. of Facilities Surveyed Facilities % of Total Surveyed Non￾Transformed Facilities % of Total No. of Facilities Surveyed Transformed Facilities % of Total No. of Facilities Brong Ahafo 29 28 97% 21 73% 7 24% Eastern 23 18 78% 14 61% 4 17% Volta 14 9 65% 5 36% 4 29% Central 30 22 73% 19 63% 3 10% Northern 6 4 67% 1 17% 3 50% Western 22 16 73% 15 68% 1 5% Total 124 97 78% 75 61% 22 18% The evaluation encountered a few limitations in implementing the evaluation. Due to the internal nature of the evaluation there is a risk of evaluator bias with the evaluation team, having an inherent interest in positive findings. There is also a risk of respondent bias. Respondents may have provided positive answers out of respect and appreciation for the relationship with the SMH project. Additionally, the levels of contribution presented in the evaluation reflect only the respondent’s point of view. Another constraint affecting the evaluation design, was the available resources. The limitation of available resources prevented the SMH project from conducting an impact evaluation and administering the proportional piling activity across all of the assisted facilities to inform all four evaluation questions. Finally, while the project believes that causal factors presented in the evaluation are the most significant factors contributing to the outcomes observed, they are not exhaustive and additional causal factors may have contributed to the observed results. FINDINGS Increased Awareness of Business Continuity and Financial Sustainability. In response to evaluation question one, the evaluation found that 92% of assisted PMHs (non-transformed and transformed) demonstrated an increased awareness of the benefits of business continuity and financial sustainability. Additionally, a large percentage of PMHs were found to have applied this increased awareness and their learning to improve their business and financial practices. As a result, the evaluation determined that SMH activities account for a contribution of 90% of the increased awareness of business continuity and financial sustainability among assisted PMHs. Both SMH and non-SMH causal factors contributed the remaining ten percent. Successful Completion of Ownership/Management Transformation Following the theory of change, the SMH project hypothesized that an increased awareness of business continuity and financial sustainability would lead to an increased desire to pursue and successfully complete an ownership/management transformation. Over half of the project’s assisted PMHs expressed interest in an ownership/management transformation, of which 22 went on to successfully complete a transformation. Of the six causal factors present, the SMH business and clinical trainings and the SMH small grants program were identified by respondents equally as the highest contributing factors (22%), Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 4 followed closely by the SMH sensitization events and activities (18%). The competing explanations including a pre-existing desire to grow the PMH accounted for a contribution of19%; the retirement or death of PMH owner, 14%; and a pre-existing desire to take advantage of capitation, 5%. Thus, the evaluation found that the SMH causal factors contributed to 62% of a facility’s successful completion of a transformation while the remaining 38% was the result of contributions made by non-SMH causals factors. Improved Financial Sustainability, Operational Efficiency, and Quality of Clinical Services In response to evaluation question three, the project discovered that 95% of transformed facilities demonstrated an increase in financial sustainability, fully 100% transformed facilities improved in the quality of their clinical services and 59% demonstrated improvement in technical efficiency.3 Among the six causal factors contributing to the improvement in financial sustainability, transformed facilities indicated the highest levels of contribution among the SMH business trainings (31%) followed closely by the SMH business technical assistance (29%) and the addition of new personnel through transformation (12%).4 The remaining contribution was dispersed among competing explanations including a pre-existing desire to grow the PMH (21%); a pre-existing interest in external financing (3%); and a pre-existing desire to prepare for NHIA capitation (3%). In conclusion, the evaluation found that the SMH activities account for a contribution of 73% of the improvements in financial sustainability. The evaluation respondents (transformed facilities) determined that among the six causal factors contributing to the improvement in clinical capacity, transformed facilities were influenced the most by the addition of new personnel through transformation (29%) followed closely by the SMH post￾transformation clinical technical assistance (TA) (26%) and the SMH clinical trainings at (24%). The remaining contribution was dispersed among competing explanations including participation in continuing professional development programs (13%); TA provided by non-SMH programs (11%); and a pre-existing desire to prepare for NHIA capitation (2%). Overall, the evaluation found that the SMH activities account for a contribution of 79% towards the improvements in the quality of clinical services. Expansion of Transformed PMHs The evaluation found that 65% of transformed facilities expanded, as defined by adding new health services or by increasing the capacity of the PMH to provide services (through additional full-time staff or the number of available beds). Additionally, 46% of transformed facilities are in the process of graduating from a PMH to a health center or clinic. Among the six causal factors influencing the improvement in financial sustainability, transformed facilities indicated the highest levels of contribution among the SMH small grants program (30%) followed closely by the SMH business and clinical trainings and TA (27%) and the addition of new personnel through transformation (13%). The remaining contribution was dispersed among competing explanations including pre-existing desire to grow the PMH (14%); TA provided by non-SMH programs (11%); and a pre-existing desire to prepare for NHIA capitation (4%). As a result of the proportional piling activity, the evaluation found that the SMH intervention accounts for a contribution of 71% towards the expansion demonstrated by transformations. 3 Please note: As an indirect outcome, technical efficiency presented specific methodological challenges for the evaluation. In particular, the complexity of the methodology meant that it would have been difficult to provide respondents with a clear sense of exactly what they were being asked to explain in the proportional piling portion activity. Thus the evaluation was unable to determine contribution for improvements in technical efficiency. 4 One of the responding transformed facilities was unable to complete the proportional piling activity in relation to evaluation question three. In light of this, the total number of available stones was reduced for this evaluation question to reflect the participation of 21 transformed facilities rather than all 22. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 5 CONCLUSIONS AND RECOMMENDATIONS 1. The SMH project made a significant contribution to the outcomes of interest. Among transformed facilities, the SMH project was found to be responsible for 90% of the increased awareness of the benefits of business continuity and financial sustainability, 62% to the successful completion of ownership/management transformations, 73% of the improvements in financial sustainability, 79% of the improvements in the quality of clinical services, and 71% of facility expansion. Overall, the evaluation found that the SMH project accounts for a contribution of 75% of the results observed and the remaining 25% to non-SMH causal factors. Recommendation: It is recommended that USAID consider building upon these gains to extend and/or replicate the SMH project model. The SMH project itself was a pilot effort, and these results suggest that an extension of this pilot would indeed support the expansion of access to quality MNCH services in rural, underserved areas of Ghana. 2. There is an opportunity for substantial gains from strengthening business and financial practices. The evaluation found that 92% of assisted PMHs demonstrated an increased awareness of the benefits of business continuity and financial sustainability. Additionally, 68% of non-transformed and 100% of transformed PMHs were found to have applied this increased awareness and improved upon their business and financial practices. Only ten percent of assisted PMHs indicated that they received business and financial TA outside of the SMH project. These findings suggest that, outside of the SMH project, PMHs receive little assistance to support the business and financial aspects of their facilities. Given the endline finding that 87% of non-transformed facilities and 95% of transformed facilities improved their financial sustainability during the SMH project, it appears that when PMHs do receive targeted TA in business and financial practices, there is an opportunity for substantial gains. Recommendation: It is recommended that USAID consider supporting additional programming focused on the business and finance side of PMHs, and potentially other private providers. 3. New personnel contribute most immediately to quality improvements. Respondents indicated that new personnel were responsible for 29% of the improvements in the quality of clinical services, however new personnel were responsible for only 12% of the improvements in financial sustainability and 13% to the expansion of the PMH representing the fourth most significant causal factor for both outcomes. These findings suggest that facilities were aware of the early-stage contribution that clinical staff made to their facilities, but they placed less value on the benefits of the additional personnel in other areas. The most likely explanation for this finding is that sufficient time had not passed for the benefits of the new personnel, outside of clinical work, to manifest. Sixteen of the 22 transformed facilities (73%) were operating with the new personnel for less than one year at the point of evaluation. While contributions to the core competency of the PMH (clinical service delivery) would have occurred directly – either in the form of applying new clinical skills or freeing up time for the primary midwife – there would have been a lag before contributions manifest in financial sustainability and expansion. Recommendation: It is recommended that GRMA and USAID continue to track the transformed facilities to gain a better sense of the value of the contributions of the new personnel to non-clinical dimensions of the PMH business, in order to gain a clearer understanding of this aspect of the SMH model. 4. The combination of training and technical assistance, small grants, and a pre-existing desire to grow appears to be effective in catalyzing change. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 6 In relation to completing a transformation (evaluation question 2) and expanding the facility (evaluation question 4), the evaluation found that the SMH small grants program, SMH training and TA, and a pre￾existing desire to grow were the top three causal factors. Regarding completion of a transformation, respondents gave equal weight to SMH trainings (22%) and the SMH small grants program (22% each), followed closely by the pre-existing desire for growth (19%). Similarly, regarding expanding the facility, respondents assigned high levels of influence to the SMH small grants program (30%), SMH trainings and technical assistance (27%) and a pre-existing desire to grow (14%). These findings suggest that a combination of structured trainings and technical assistance and a targeted small grants program may be an effective intervention package when working with PMHs that have a desire to grow. The pre-existing desire for growth is an important factor, as it is likely that this motivation is what leads the PMH to apply learning from the training and make optimal use of grant-funded investments. Recommendation: The project recommends that USAID and other donors explore approaches to screening participants in private health sector assistance programs for a pre-existing desire for growth, potentially by building off of the SMH approach focused on PMHs self-selecting by submitting expressions of interest and agreeing to a multi-stage screening process before receiving additional assistance in completing a transformation. 5. While this innovative evaluation methodology provides a useful indication of contribution, a more extensive study would yield more certainty. This evaluation represents a new approach to gaining some understanding of contribution when sufficient resources for a full impact evaluation are not available. Unfortunately, the project was not able to conduct an impact evaluation using the methods necessary to create a credible or rigorously defined counterfactual so as “to control for factors other than the intervention that might account for the observed change.”5 While the SMH innovative research methodology uses a less rigorous approach, it does provide a useful idea of relative contribution. That said, a true impact evaluation would be required to expand and confirm these findings. Recommendation. If USAID decides to replicate or scale-up the innovative SMH intervention, it should conduct an external impact evaluation to validate these conclusions and determine the extent to which this model can support the expansion of MNCH services. 5 USAID, 2011. USAID Evaluation Policy. U.S. Agency for International Development, Washington, DC Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 7 1. INTRODUCTION This report presents the findings from the Saving Maternity Homes in Ghana, Innovate for Health project’s internal end of project performance evaluation. The USAID-funded $1.78 million Saving Maternity Homes in Ghana Innovate for Health (SMH) project, implemented by Banyan Global in partnership with the Ghana Registered Midwives Association (GRMA), focused on revitalizing private maternity homes (PMHs) in Ghana to expand access to quality maternal, newborn, and child health (MNCH) services. The project sought to facilitate ownership and/or management transformations that would result in increased capacity, improved quality, and expanded services. The SMH project, which ran from September of 2014 until February of 2018, targeted six regions within Ghana: Brong Ahafo, Central, Eastern, Northern, Volta and Western. The internal end of project evaluation (evaluation), conducted from FY17 Q4 through FY18 Q1, was designed to generate findings related to the effectiveness and impact of project activities along with lessons learned for the purpose of improving, replicating, or scaling-up aspects of the intervention in line with the project’s learning mandate. 1.1 COUNTRY CONTEXT Ghana is experiencing rapid economic growth, yet a large development gap exists between rural and urban areas, higher- and lower-income communities, and in key development indicators, including MNCH, that raise concerns about equity and the long-term sustainability of improvements. In the years leading up to the launch of the SMH project, the Government of Ghana (GoG) made major efforts to address the development gap and expand equity in access to healthcare services. Despite these efforts, the PMH sector has experienced a steady decline over the past decade as midwives have aged and struggled to adapt to the roll-out of the National Health Insurance Scheme (NHIS). GRMA estimates that in the five years leading up to the launch of the SMH project, 100 PMHs closed, and more were operating at reduced capacity as midwives reached retirement age without any succession plans. 1.2 OVERVIEW OF THE SMH PROJECT The SMH project aimed to help address the challenges noted above through increasing awareness of ownership and management transformations, an innovative approach to revitalizing PMHs. Transformations bring new resources and personnel to a PMH through a sale of the PMH, leasing out the facility, bringing on a new partner, bringing on a new manager, or bringing on an employee with a new skill set. Participating PMHs received various “tiers” of project support, ranging from basic peer support visits (PSVs) led by GRMA, to classroom training in business continuity and clinical services strengthening, to support for effectively implementing a transformation. As noted in the Figure 2 below, only facilities that received Tier 2 support or above were considered “assisted” by the project. Ultimately, the SMH project’s intervention aimed to increase access, improve efficiency and expand quality MNCH and family planning services in rural, underserved areas. Figure 2. Levels of Project Assistance Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 8 The following highlights the four main tiers of SMH assistance to PMHs in the target regions: 1. Tier 1: Peer Support Visits – The lowest level of project assistance was for GRMA to provide PSVs. These visits involved GRMA reviewing specific aspects of PMH service provision and operations, identifying areas for improvement, and providing recommendations. Some facilities at this level participated in further capacity building activities or transformation assistance provided by the project, and some did not. PMHs that received only PSVs and did not attend other SMH project trainings or receive transformation assistance, are regarded as unassisted by the project. 2. Tier 2: Capacity Building Support – At this level of project assistance, PMHs received capacity building support in the form of various SMH project trainings, including the Business Continuity and Succession Planning (BCSP) training (May-June, 2015), the Clinical Capacity Building (CCB) training (May-June, 2015), and the Business Strengthening Support (BSS) training (April-May, 2016). PMHs that received this support are regarded as assisted by the project. 3. Tier 3: Transformation Support – This tier of project support is focused on PMHs that submitted Expressions of Interest (EOIs) in pursuing a transformation and were accepted by the project to receive specialized transformation support. At this level of project assistance PMHs received all of the assistance included in Tiers 1 and 2, plus individual technical assistance (TA) focused on deal structuring, needs analysis, business plan support, and other pre-transformation activities. PMHs that received this level of support are regarded as assisted by the project. 4. Tier 4: Post-Transformation Support – This tier of project support focused on facilities that successfully completed a transformation. The project provided tailored TA to these PMHs to enhance the likelihood of business and financial sustainability in the new arrangement, as well as clinical TA to ensure that transformed facilities provide high-quality health services. PMHs in this tier were also eligible to apply for small grants from the project. PMHs that received this level of support are regarded as assisted by the project. 1.3 PURPOSE OF THE INTERNAL END OF PROJECT EVALUATION The most reliable approach to determining and attributing impact is an impact evaluation. The USAID Evaluation Policy defines an impact evaluation as an evaluation that seeks to measure the change in project outcomes that are attributable to (caused by) the project using methods that construct a “credible and rigorously defined counterfactual” so as “to control for factors other than the intervention that might account for the observed change.”6 Creating a credible and rigorously defined counterfactual requires methods for comparing a treatment group of project participants and a control group of non-participants who are as similar as possible to project participants. Unfortunately, the SMH project did not have the resources for an external impact evaluation using the methods necessary to create a credible or rigorously defined counterfactual. However, given the innovative nature of the project design and the learning mandate built into the Innovate for Health mechanism, the SMH project has conducted a limited, internal evaluation using less rigorous, but nonetheless sound, methods. The purpose of the evaluation is to generate findings related to the 6 USAID, 2011. USAID Evaluation Policy. U.S. Agency for International Development, Washington, DC Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 9 effectiveness and impact of project activities along with lessons learned for the purpose of improving, replicating, or scaling-up aspects of the intervention. The primary target audience of the evaluation is USAID/Ghana staff. Additionally, the evaluation will be of interest to other implementing partners, the GoG, public health organizations, academics and research institutions, GRMA, and PMHs. It is also anticipated that the evaluation will be useful for any other stakeholders interested in supporting the sustainability of PMHs, while enabling local communities to continue to enjoy access to the critical health services PMHs provide. Findings may also have broader implications for working with the private health sector in rural and underserved areas, both within Ghana and in other low resource settings. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 10 2. PROJECT OBJECTIVES AND THEORY OF CHANGE 2.1 PROJECT OBJECTIVES The assistance provided by the SMH project enabled revitalized maternity homes to capitalize on the opportunity presented by the NHIS, while responding to the demand for quality health services in rural areas. This work directly supports the USAID/Ghana Country Development Cooperation Strategy (CDCS) results framework in supporting Development Objective 3: Equitable Improvements in Health Status. Specifically, this work supports IR 3.1: Increased access to integrated health services, and IR 3.3: Strengthened and responsive health systems. Additionally, these improvements support the GoG’s strategic policy objectives, as well as the Ghana Global Health Initiative’s IR 1.1: Increased access to quality reproductive health and HIV care. 2.2 THEORY OF CHANGE The SMH project theory of change (see Figure 1) focused on the operational and management efficiency, business and financial sustainability and quality improvements expected to result from ownership/management transformations. Figure 1. SMH Theory of Change DEFINITIONS To have a clear understanding of the theory of change, several key concepts must be defined: Transformation: Transformation is defined as occurring when; 1) the PMH facility is legally sold to a new owner or investor (sale); 2) the existing owner brings on a partner who has both an ownership stake 1. PMH awareness of the beneficial impacts of business continuity and financial sustainability increased 2. Ownership and management transformations completed 3. Operational efficiency, business and financial sustainability, and quality of clinical services in transformed facilities improved 4. Transformed facilities expanded In alignment with USAID/Ghana CDCS, Development Objective 3: Equitable Improvements in Health Status, the project sought to:  Develop the capacity of PMHs in business continuity planning;  Leverage financing and investment to improve quality and expand service delivery;  Strengthen the business and clinical capacity of PMH owners;  Catalyze transformations;  Develop the capacity of GRMA; and,  Conduct operations research to understand key issues and the effectiveness of the project’s design. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 11 and an active management role (partnership); 3) the existing owner retains ownership, but appoints a manager with a job description that gives this person substantial authority over operational aspects of facility management (management contract); 4) the existing owner retains ownership and leases the facility to a new manager/operator (lease); or 5) the existing owner retains ownership and management, but hires a new individual who brings a specific skill set that enables the facility to offer additional health services (service diversification contract).7 Business and Financial Sustainability: Levels of business and financial sustainability are measured by the “business formalization score” (BFS), a composite indicator of key facility-level performance indicators demonstrating the capacity of a PMH to sustain its business operations. This score includes clearly verifiable considerations, such as whether the business is legally incorporated, whether it is in compliance with regulations and licensing requirements, whether it is currently accredited with National Health Insurance Authority (NHIA), and whether it has a bank account (personal or corporate). Quality of Clinical Services. The quality of clinical services administered by PMHs are measured by the “Quality Assessment Tool (QAT) score,” a composite indicator focused on emergency response, quality assurance, safe motherhood and advocacy and promotion. Facilities received a score for each clinical section of the QAT assessment which were then summed up into a total composite score. The GRMA team then used this outcome and this information to provide feedback to the facility, and this feedback constituted a PSV. Expansion: A transformed facility has expanded when, following the intervention, it has added new health services (e.g., neo-natal services, family planning services, malaria, diarrhea, inpatient, etc.), or increased capacity (e.g., number of beds, increased staff). Such expansion should ultimately increase the number of client visits, which suggests an increase in the PMH’s ability to provide health services to its community. HYPOTHESES This theory of change rests on four hypotheses, as follows: 1. It is hypothesized that project trainings, presentations, direct outreach and other sensitization efforts will result in assisted maternity homes having a better sense of the potential benefits of business continuity and financial sustainability. Anticipated outcome: PMH awareness of the benefits of business continuity and financial sustainability increased. 2. It is further hypothesized that, in a subset of the project’s beneficiaries, an increased awareness of business continuity and financial sustainability will lead to a desire to complete a transformation. The project will assist these facilities to prepare for transformation, and will assist in identifying potential new owners, partners, leasees, and employees (managers and technicians). Anticipated outcome: Ownership and management transformations completed. 3. It is further hypothesized that transformed facilities will engage in targeted efforts to improve the operational efficiency, business and financial management, as well as the quality of care of their newly acquired PMHs. This hypothesis is based on the assumption that these new 7 During the second quarter of Year 2, the SMH project added a new type of transformation, the “service diversification contract,” to the existing transformation types. While not included in the original transformation types, the service diversification contract was derived from an operational research finding that, with the expected arrival of capitation and the general need for PMHs to diversify their service offerings, an additional skill set and the ability to offer additional health services are critical to continuity and sustainability. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 12 owners/managers, or facilities that have diversified their clinical services, will have purchased or taken over management of the facilities in order to make a profit (in addition to the motivation to provide health services), and will therefore work to increase the efficiency of the facility and improve the quality of the health products and services available in order to ensure the financial success of their investments and efforts. Anticipated outcome: Operational efficiency, business and financial sustainability and quality of clinical services in transformed facilities improved. 4. It is further hypothesized that improvements in efficiency, business and financial sustainability, and quality of care will result the expansion of transformed facilities, and the increased capacity of PMHs to administer these services in the target regions. As described in the definition above, this expansion may manifest in adding new health services and/or increasing capacity. Anticipated outcome: Transformed facilities expanded. In order to support the long-term sustainability of these project interventions, the SMH project worked directly with the GRMA, building its capacity to deliver this assistance after the project in the future. This assistance included efforts to link GRMA to non-United States government sources of funding. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 13 3. EVALUATION QUESTIONS The SMH project end of project performance evaluation seeks to answer the following four evaluation questions: 1. Evaluation Question for Step 1 of the Theory of Change. To what extent, if any, have SMH project activities increased the awareness of the benefits of business continuity and financial sustainability of PMHs in the target regions? 2. Evaluation Question for Step 2 of the Theory of Change. To what extent, if any, have SMH project activities led to the successful completion of ownership/management transformations for PMHs in the target regions? 3. Evaluation Question for Step 3 of the Theory of Change. To what extent, if any, have SMH project activities led to the improved financial sustainability, operational efficiency and the quality of clinical services of transformed PMHs in the target regions? 4. Evaluation Question for Step 4 of the Theory of Change. To what extent, if any have SMH project activities led to the expansion of transformed PMHs in the target regions? Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 14 4. EVALUATION METHODLOGY The project employed a modified General Elimination Methodology (GEM),8 a non-experimental methodology that focuses on systematically identifying and eliminating alternative causal factors that could explain an observed result. While a non-experimental evaluation does not provide the level of rigor of an impact evaluation, it will enable the project to gain some understanding of project effectiveness and the extent to which project activities contributed to the outcomes of interest. The evaluation methodology is as follows: 1. Definition of group to be evaluated: The group to be evaluated consists of assisted non-transformed and transformed PMHs in the target regions. 2. Definition of outcomes of interest: In the case of the SMH project, the outcomes of interest are articulated in the theory of change and can be measured through survey questions. 3. List of causal factors for the outcomes of interest: The project relied on the expertise of project staff, project beneficiaries, key informants and local partners to develop a list of SMH intervention as well as non-SMH intervention causal factors that contributed to the outcomes of interest. In an effort to keep the process manageable, and to avoid favoring the SMH intervention, for most evaluation questions the evaluation identified three SMH and three non-SMH causal factors. Please note for evaluation question one, which focuses on raising awareness, the evaluation identified only one SMH causal factor (the project’s efforts to raise awareness) and one non-SMH causal factor (other stakeholders’ efforts to raise awareness). 4. Identification of whether the outcomes of interest have occurred and determination if the causal factors are present. The “outcomes of interest” are the outcomes articulated in the theory of change. For example, for Evaluation Question 1, the outcome of interest is an increased awareness of the benefits of business continuity and financial sustainability. Relying on data from the endline assessment, the evaluation confirmed that the outcomes of interest occurred, and which causal factors were present. The causal factors discussed in this report were all found to be present and thus determined to be potential factors which may have contributed to the outcomes of interest. 5. Determination of the relative contribution to SMH and non-SMH causal factors. The evaluation analyzed the data to gain some idea of the extent to which the project intervention and the non-project interventions could have contributed to each outcome of interest. Using a proportional piling research method (explained below), transformed facilities determined the relative contribution of the project’s interventions as compared to the non-SMH causal factors for the outcomes observed. Proportional piling allowed respondents to rank the causal factors that contributed on some level to the results observed. While this methodology does not provide the same level of statistical rigor and confidence that is achievable in an impact evaluation, it does provide a systematic approach to identifying and weighting factors contributing to various project outcomes, while recognizing budget and time constraints. 8 For an overview of GEM, see Howard White and Daniel Phillips, “Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework,” International Initiative for Impact Evaluation, Working Paper 15, 2012. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 15 4.1 TOOLS AND INSTRUMENTS Data collection for the evaluation was conducted by the SMH personnel during the baseline and endline assessments. Data was collected using five primary methods: the PMH Survey, the QAT, the PMH Extension Survey, an In-Depth Interview (IDI) guide and a Key Informant Interview (KII) guide. The PMH Extension Survey was administered only at endline while the remaining tools were administered at both baseline and endline. The following table highlights the distribution of the surveys, IDIs and KIIs at endline. Table 2: Distribution of Tools and Instruments at Endline Facilities Non-Transformed Facilities Transformed Facilities Key Informants PMH Survey 99 77 22 0 QAT 99 77 22 0 PMH Extension Survey 97 75 22 0 IDI 22 0 22 0 KIIs 0 0 0 7 Survey respondents included the owners and/or managers of the PMH. Of the 99 facilities that participated in the PMH Survey and QAT, 91% of the respondents were the primary owner of the facility while the remaining 9% of respondents were managers. All 22 IDIs included both the PMH owner and new personnel hired through the transformation process. Additional staff participated in the proportional piling activity, discussed below, pending availability. KIIs were conducted with key government stakeholders (four) and implementing partners (three). While the key informant interview (KII) and IDI guides focused on qualitative data, the three survey tools were designed to gather quantitative data. The PMH and PMH Extension Surveys were administered by SMH project staff, while the QAT was administered by GRMA. The PMH Survey and the QAT were also used to inform the business formalization score, measuring the capacity of a PMH to sustain its business operations, and the QAT score, demonstrating the quality of clinical services. Additionally, using the data derived from the PMH Survey and QAT, the project employed data envelopment analysis (DEA) to determine the efficiency of PMHs. The tools and instruments used in the evaluation can be found in Annex 4. PROPORTIONAL PILING RESEARCH METHOD In addition to the tools and instruments noted above, the SMH project integrated proportional piling, a participatory research method, into data collection to determine the contribution of the project interventions as compared to the competing explanations or non-project interventions for the outcomes observed. The proportional piling activity, along with in-depth interviews with transformed facilities, allowed respondents to rank the influential factors that contributed on some level to the results observed.9 The proportional piling activity was administered only to transformed facilities in alignment with the project’s theory of change and evaluation questions. While comparisons were drawn between non￾transformed and transformed facilities under evaluation question one, the last three evaluation questions 9 The levels of influence or contribution determined for evaluation question one do not reflect the findings generated from a proportional piling activity, rather the findings were derived from quantitative data gathered using the PMH Extension Survey. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 16 were designed to focus only on those facilities that successfully completed a transformation. For more discussion on the results observed between non-transformed and transformed facilities, please refer to the project’s endline assessment report. For evaluation questions two - four, the SMH project presented respondents with a series of circles, each representing a different causal factor. The causal factors, comprised of both SMH interventions and non-SMH interventions, were determined based on the expertise of project staff, project beneficiaries, key informants and local partners, (For more discussion of this process, please see step three of the evaluation methodology above.) Participants were provided with 100 stones and were asked to distribute the stones among the six causal factors (circles) to determine the extent to which that factor contributed to the outcome of interest. The number of stones was set at 100 to allow participants and project staff to easily convert the number of stones into percentages of a whole. The respondents were encouraged to review, and adjust the amount of stones in each circle until they were satisfied with the proportions. The evaluation recorded the number of stones per circle and asked a series of probing questions to further understand the respondent’s choices. Through this process, the respondent determined the levels of influence of SMH and non-SMH factors for the outcomes of interest. 4.2 SAMPLING STRATEGY The evaluation relied heavily on data gathered using the PMH Survey and the QAT during the project’s baseline and endline assessments as well as the PMH Extension Survey administered at endline. Given the limited number of PMHs within the six target regions, the project was reasonably able to target the entire population of PMHs for participation in the baseline and endline assessments and subsequently the evaluation. Informed by data from Ghana Health Service and GRMA, at baseline the project developed a list of 170 facilities which was then reduced to 155 at endline to account for the project’s knowledge of the relocation, closure or the recent establishment of facilities within the target regions since baseline. While the project encountered the establishment of new PMHs over the past three years, the rate at which facilities are opening is disproportionate to the rate at which facilities are closing. This largely accounts for the change in number of facilities targeted at endline compared to those at baseline. In each region, the survey team operated out of a local commercial center, chosen based on travel distance between the town and the PMHs to be visited. From there, the team covered all the PMHs in that location and around it. As the team traveled from one major town to the next, it surveyed the PMHs located along the route. This process continued until the team had covered a target region. As a result of these efforts the project gathered data from a pool of 112 assisted facilities (92 assisted non-transformed and 20 Photo Credit: USAID, Saving Maternity Homes in Ghana Project A transformed PMH engaging with the proportional piling activity Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 17 assisted, transformed) and 99 assisted facilities at endline, (77 assisted non-transformed and 22 assisted, transformed).10 4.3 ETHICAL CONSIDERATIONS The SMH project is committed to ensuring the protection of human subjects. The SMH project has performed periodic reviews of the research design and implementation procedures inclusive of ethical considerations and the protection of human subjects. Furthermore, the SMH project submitted the protocol and data collection tools to inform the endline assessment and evaluation to Ghana Health Service Ethics Review Committee (ERC) to obtain ethical clearance and approval. The project received formal approval from the ERC in early September of 2017. (See Annex 3) 4.4 ANALYSIS The report consists primarily of analysis based on descriptive statistics generated from the PMH Extension Survey and IDIs, informed by background information derived from the literature review as well as qualitative data from key informant interviews, where appropriate. Evaluation data analysis focuses on facilities that were deemed assisted, as defined above. For evaluation question one, the analysis is further divided between non-transformed and transformed facilities, in order to explore the differences between these two groups. For the remaining evaluation questions, two through four, the analysis is limited to transformed facilities in alignment with the project’s theory of change. Baseline values represent a dataset of the 92 assisted, non-transformed facilities and 20 assisted, transformed facilities. The endline dataset includes 77 assisted, non-transformed facilities and 22 assisted, transformed facilities. When discussing facilities that demonstrated change from baseline to endline, as is the case with improvements in financial sustainability, quality of clinical services and efficiency, these figures are based on the 66 assisted, non-transformed facilities and the 20 assisted, transformed facilities that participated in both the baseline and endline assessments.11 The findings informed by data gathered for the evaluation represent a pool of 97 assisted facilities (75 assisted, non-transformed and 22 assisted, transformed). While the 77 assisted, non-transformed facilities participated in the PMH Survey and QAT, two facilities were removed from the final dataset due to incomplete PMH Extension Surveys. Please see Table 1 below. Table 1: Respondents by Region Total No. of Facilities Surveyed Facilities % of Total Surveyed Non￾Transformed Facilities % of Total No. of Facilities Surveyed Transformed Facilities % of Total No. of Facilities Brong Ahafo 29 28 97% 21 73% 7 24% Eastern 23 18 78% 14 61% 4 17% Volta 14 9 65% 5 36% 4 29% Central 30 22 73% 19 63% 3 10% 10 While the project sought to collect data from the 170 PMHs believed to be operational, not all of the PMHs could provide the owner or manager to complete the surveys during the project’s visit. Additionally, due the relocation, closure or the recent establishment of facilities within the target regions, this number was further reduced. 11 The project has identified the instances in which findings are based on the results of facilities that participated in both baseline and endline (66 non-transformed and 20 transformed facilities) rather than the full endline dataset representing 77 non-transformed and 22 transformed facilities. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 18 Table 1: Respondents by Region Total No. of Facilities Surveyed Facilities % of Total Surveyed Non￾Transformed Facilities % of Total No. of Facilities Surveyed Transformed Facilities % of Total No. of Facilities Northern 6 4 67% 1 17% 3 50% Western 22 16 73% 15 68% 1 5% Total 124 97 78% 75 61% 22 18% Once the evaluation data was gathered, the project conducted an analysis to identify the extent to which each competing explanation played a causal role, according to respondents, in the outcomes of interest across all 22 transformed facilities. The project developed a system of coding for answers to open ended questions, which incorporates a system of crosschecking against multiple sources and interviews. Regarding the proportional piling, proportional allocations from each individual transformed PMH were aggregated to get the total allocation across all respondents in order to determine the relative weight of each causal factor. Additionally, the analysis then combined the SMH causal factors, and the non-SMH causal factors, to get the extent to which the SMH project contributed to the outcomes of interest. The project then summarized this analysis in drafting the report. 4.5 LIMITATIONS One of the main constraints affecting the evaluation design is bias which could affect the validity of the evaluation data and findings. On the one hand, there is a concern about respondent bias in seeking to provide project personnel with positive answers out of a desire to maintain good relationships, and appreciation for the relationship that has been built. On the other hand, there is potential evaluator bias, which is particularly an issue with an internal evaluation, that the project itself will read the data to reflect positive outcomes in order to validate the project model or activities. In efforts for the SMH project team to maintain independence from the results of the evaluation, the levels of contribution were determined by the project’s beneficiaries. While this approach helps to address potential evaluator bias, the levels of contribution reflect only the respondent’s point of view, which in itself is another limitation. Additionally, in efforts to mitigate potential bias while maintaining the validity of an evaluation, the evaluation research design included the triangulation of findings through multiple sources and employed data analysis practices that ensured the systemic analysis of the qualitative data. Such practices mitigate the risk of identifying positive results in raw qualitative data, and force the analyst to identify true consistencies and valid insights. Another constraint was the available financial resources. Unfortunately, the SMH project did not have the resources for an external impact evaluation. Additionally, the limitations in available resources and constraints on time, also affected the team’s ability to administer the proportional piling activity across all of the assisted facilities and for all questions. As a result, the levels of contribution determined for evaluation question one do not reflect the findings generated from a proportional piling activity, rather the findings were derived from quantitative data gathered using the PMH Extension Survey. These causal factors are not exhaustive, and it can be assumed that there are additional factors which may have contributed to the results observed. In efforts to address this limitation, the project provided respondents with the option to identify additional causal factors. However, out of the 22 proportional piling exercises that were conducted by the project, no respondents suggested additional causal factors. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 19 5. FINDINGS 5.1 INCREASED AWARENESS OF THE BENEFITS OF BUSINESS CONTINUITY AND FINANCIAL SUSTAINABILITY The evaluation sought to determine whether the assisted PMHs demonstrated increased awareness of the benefits of business continuity and financial sustainability. Additionally, the evaluation sought to understand the extent to which the SMH project activities contributed to any change in this awareness in comparison to non-SMH causal factors. DID THE OUTCOME OF INTEREST OCCUR? Prior to exploring the contribution of the SMH project, the evaluation sought to determine whether the outcome of interest, an increased awareness of the benefits of business continuity and financial sustainability of PMHs in the target region, occurred. Evidence As part of the endline assessment, the PMH Extension Survey asked assisted PMHs about the extent to which they felt that they had gained an increased awareness of the benefits of business continuity and financial sustainability since the launch of the SMH project.12 Among both assisted non-transformed and assisted transformed facilities, 92% of respondents selected “strongly agree” or “agree” in response to the survey question. As shown in Figure 3, 90% of non-transformed facilities and 100% of transformed facilities reported increased awareness of the benefits of business continuity and financial sustainability. The evaluation also sought to understand whether PMHs had taken action based on this reported increased awareness of the benefits of business continuity and financial sustainability.13 While 90% of non￾transformed facilities identified an increased awareness, 68% said that they had made changes to their 12 PMH Extension Survey, Question 4.1 “To what extent do you agree with the following statement “I have a better awareness of the benefits of business continuity and financial sustainability.” 13 PMH Extension Survey, Question 4.2 “Have you changed the way in which you have managed or conducted your facility’s business and financial practices in the last three years?” 100% 90% Figure 3: Increased Awareness of the Benefits of Business Continuity and Financial Sustainability, "Strongly Agree" or "Agree" Non-Transformed Transformed Evaluation Question 1: To what extent, if any, have SMH project activities increased the awareness of the benefits of business continuity and financial sustainability of PMHs in the target regions? 92% of assisted PMHs had an increased awareness of the benefits of business continuity and financial sustainability. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 20 business and financial practices, applying learning and knowledge gained. Fully 100% of transformed facilities said they had improved their business and financial practices within the last three years. The evaluation then asked respondents open ended questions as part of the PMH Extension Survey, to describe how they have improved their business and financial practices.14 Please see Table 3 below. Table 3: Improvements in Business and Financial Practices All Assisted Facilities (n=97) Non￾Transformed Facilities (n=75) Transformed Facilities (n=22) Increased Maintenance of Core Financial Documents 64% 60% 77% Improved Budgeting and Increased Budget Monitoring 20% 13% 41% Improved Inventory Management 11% 9% 18% Hired New Staff and Delignated Roles and Responsibilities 7% 4% 18% Commenced Analysis of Financial Data 9% 8% 14% Separated Business and Personal Funds 1% 0% 5% Computerized Submission of NHIA Claims 4% 5% 0% Utilized a Business Bank Account 1% 1% 0% The evaluation identified eight types of improvements among both non-transformed and transformed facilities. The two most often cited improvements were increased maintenance of core financial documents and improved budgeting and budget monitoring. These findings are consistent with the endline assessments finding that 89% of non-transformed facilities and 100% of transformed facilities maintained at least one type of the core financial document at endline, an increase over baseline of 33 percentage points and 19 percentage points respectively. Additionally, among non-transformed facilities, 61% reported that they maintained a budget at endline, compared to only 39% at baseline. All 22 transformed facilities (100%) reported that they maintained a budget, an improvement of 75 percentage points from baseline. Beyond these improvements, the project’s endline assessment informed by the PMH Survey found that both non-transformed and transformed PMHs demonstrated improvements across various business and financial practices. Outcome of Interest Did Occur. In light of these findings, we conclude that the outcome of interest did occur. The majority of assisted PMHs, including both non-transformed and transformed facilities, reported an increased awareness of the benefits of business continuity and financial sustainability since the 14 PMH Extension Survey, Question 4.3 “If so, how?” 100% 68% Figure 4: Improved Business and Financial Practices Non-Transformed Transformed Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 21 launch of the SMH project in 2014. This finding is supported by evidence found at endline that assisted PMH facilities made actual changes to their business administration and financial management practices. With the occurrence of the outcome of interest confirmed, we now explore the causal factors which may have contributed to the outcome of interest. EXPLORING CAUSAL FACTORS The evaluation identified a series of SMH interventions and non-SMH interventions that may have contributed to this increased awareness of the benefits of business continuity and financial sustainability. The SMH interventions which were designed to increase PMHs’ awareness of the benefits of business continuity and financial sustainability prior to transformation include business trainings, as well as sensitization events such as the baseline assessment (through which PMHs were introduced to the SMH project), regional meetings, and Short Message Service (SMS) messages. The SMH project recognizes that there are various other actors working across the six target regions to improve the business and financial practices of health providers, including PMHs, such as Marie Stopes International, Princefield University College, and other entrepreneurship trainings provided by NGOs. Thus, the evaluation takes into consideration the technical assistance (TA) provided by non-SMH interventions as a potential causal factor which may have led to the increased awareness of the benefits of business continuity and financial sustainability. Causal factors are discussed in more detail below. SMH CAUSAL FACTOR SMH Business Trainings and Sensitization Activities. The SMH project delivered a series of two classroom trainings to PMHs in the target regions that were designed, in part, to increase PMH awareness of the benefits of business continuity and financial sustainability. These trainings included the Business Continuity and Succession Planning (BCSP) training, and the Business Strengthening Support (BSS) training, which sought to build the capacity of PMH owners and managers to improve their business practices and promote sustainability. The SMH project provided training to 418 participants representing 149 facilities. All 97 assisted facilities included in this evaluation participated in at least one of the two business trainings. Please see the Table 4 below. Table 4: PMH Owner/Manager Participation in the Project’s Business Trainings All Assisted Facilities (n=97) Non-Transformed Facilities (n=75) Transformed Facilities (n=22) BSS Trainings 87% 84% 95% BCSP Trainings 76% 73% 86% The SMH project also conducted a series of sensitization events and activities that were at least partially designed to strengthen PMHs’ understanding of the benefits of business continuity and financial sustainability. These activities included the baseline assessment, regional meetings, and SMS messaging. The baseline assessment was the first interaction PMHs in the target regions had with the SMH program, providing an introduction to the SMH project and the transformation program to 120 PMHs.15 The SMH project also supported routine regional meetings led by the GRMA, which provided a space for PMHs to share good practices, discuss technical and clinical challenges, and disseminate learning. Additionally, these meetings were a platform for the SMH project to reinforce the key points of the business trainings highlighted above. The project also used SMS messaging to reinforce business and financial best practices Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 22 and promote collaboration and learning among the assisted maternity homes in efforts to raise the awareness of the benefits of business continuity and financial sustainability. As demonstrated in the Table 5 below, of the 97 assisted PMHs included in the evaluation, 89% participated in baseline assessment. Additionally, the evaluation found high levels of engagement among assisted PMHs at regional meetings, with 79% of assisted PMHs in attendance. A total of 27 SMS messages were distributed to PMHs across the project’s six target regions. Among the assisted PMHs included in the evaluation, 72% confirmed that they received the project’s SMS messages. All 97 assisted PMHs participated in at least one of the SMH project’s sensitization events and activities. Table 5: PMH Owner/Manager Participation in the Sensitization Events and Activities All Assisted Facilities (n=97) Non-Transformed Facilities (n=75) Transformed Facilities (n=22) Participation in the Baseline Assessment 89% 88% 91% Attendance at Regional Meetings 79% 75% 95% Received SMS Messages 72% 68% 86% NON-SMH CAUSAL FACTOR Technical Assistance Provided Outside of the SMH Project. There were other initiatives which were, and are currently, operating in the six regions targeted by the SMH project. As such, it can be assumed that a program with similar goals and objectives to the SMH project could have increased PMH awareness of the benefits of business continuity and financial sustainability. The evaluation asked assisted PMHs whether they received TA focused on business continuity and financial sustainability from programs outside of the SMH project.16 Of the 97 assisted facilities, 10% (7% of non-transformed and 23% of transformed PMHs) confirmed that they received TA in this area during the SMH project’s period of performance. Some of the programs mentioned by the assisted facilities include those led by Marie Stopes International, Ipas, Ministry of Health, PharmAccess Foundation, Business Sector Advisory Challenge (BUSAC) Fund, Princefield University College, and other NGOs.17 Interestingly, none of the PMHs in the Western region indicated that they received business continuity and financial sustainability assistance outside of the SMH intervention. 16 PMH Extension Survey, Question 4.8 “In the two years leading up to the SMH project, did you receive training in business continuity and/or financial sustainability outside of the SMH project? If yes, from where?” Question 4.9 “Since participating in the SMH project, have you received training in business continuity and/or financial sustainability outside of the SMH project? If yes, from where?” 17 https://www.mariestopes.org/; http://www.ipas.org/; http://www.moh.gov.gh/; https://www.pharmaccess.org/; https://www.busac.org/; http://www.pu.edu.gh/ The SMH trainings “motivated my facility on the need and importance for business continuity.” Additionally, the trainings “identified areas for upgrade and filled my facility’s knowledge gap.” - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 23 While only a small percentage of assisted PMHs indicated that they received TA from programs other than the SMH project, this causal factor still played a role. DETERMINING CONTRIBUTION While participating in sensitization events or activities does not guarantee an increased awareness of the benefits of business continuity and financial sustainability, it is reasonable to assume that the two do have a relationship to one another. However, as this is not proven, in efforts to be conservative, the SMH project sought to understand the level of influence that training and sensitization had on facilities who identified as having an increased awareness of business continuity and financial sustainability. One hundred percent of assisted facilities participated in at least one SMH sensitization event or activity, with 77% participating in all three. By comparison, while 100% participated in SMH interventions, only 10% of assisted PMHs also reported participating in non-SMH interventions that involved raising awareness of the benefits of business continuity and financial sustainability. Thus, according to this evaluation methodology, the SMH project accounts for a contribution of 90% of the outcome observed. Both the SMH and non￾SMH causal factors contributed to the remaining ten percent. 5.2 SUCCESSFUL COMPLETION OF OWNERSHIP/MANGEMENT TRANSFORMATION The SMH project hypothesized that, in a subset of the project’s beneficiaries, an increased awareness of the importance of business continuity and financial sustainability would lead to the successful completion of transformations. The evaluation sought to explore this hypothesis by investigating whether assisted PMHs successfully completed a transformation. Furthermore, the evaluation sought to uncover the role that the SMH inventions played in producing this outcome. DID THE OUTCOME OF INTEREST OCCUR? Evaluation Question 2: To what extent, if any, have SMH project activities led to the successful completion of ownership/management transformations for PMHs in the target regions? Figure 5. Increased Awareness of Business Continuity and Financial Sustainability 90% 10% SMH SMH and Other contribution of the SMH project 90% Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 24 Prior to exploring the contribution of the SMH project, the evaluation sought to determine whether the outcome of interest, PMHs in the target regions successfully completing transformations, occurred. Evidence The project provided opportunities for five unique types of ownership/management transformation, including sales, leases, partnerships, management contracts and service diversification contracts. As depicted in Table 6 below, the 22 successful transformations spanned across the six target regions, with the most transformations occurring in Brong Ahafo (seven) and the fewest in the Western Region (one).18 More than half of the 22 completed transformations were management contracts (64%). The remaining 36% are comprised of service diversification contracts (32%) and a sale (5%). Of the 22 transformations, there were no leases or partnerships. Outcome of Interest Did Occur. With 22 assisted PMHs successfully completing the transformation process, we conclude that the outcome of interest occurred. With the occurrence of the outcome of interest confirmed, the evaluation explored the causal factors that may have contributed. EXPLORING CAUSAL FACTORS This section introduces the six identified causal factors that could have contributed to the successful completion of a transformation. The causal factors presented below include three SMH interventions (classroom trainings, sensitization activities, and the small grants program) as well as three non-SMH factors (PMH owners’ pre-existing desire for growth, pre-existing awareness of the anticipated arrival of capitation, and the retirement or death of the primary owner/midwife). SMH CAUSAL FACTORS SMH Trainings. In addition to the BCSP and the BSS trainings described above, the project offered Clinical Capacity Building (CCB) training. As noted, all 97 assisted facilities included in this evaluation participated in at least one of the trainings, with 63% of non-transformed facilities and 82% of transformed facilities attending all three trainings. One objective of these trainings was to sensitize PMHs to the benefits of ownership/management transformations. Following the trainings (and the sensitization efforts described below), 49% of assisted PMHs expressed a desire to pursue a transformation through the submission of an expression of interest (EOI) form. 18 For an expanded discussion of geographic considerations and other factors related to PMHs participating in the SMH project, please see the Endline Assessment, section 4.1: Summary of Participating Facilities. Table 6. SMH Transformations by Type, Disaggregated by Region Transformation Type Total Brong Ahafo Central Eastern Northern Volta Western Management 14 3 2 4 1 3 0 Service Diversification 7 4 1 0 2 1 1 Sale 1 0 0 0 0 0 0 Lease 0 0 0 0 0 0 0 Partnership 0 0 0 0 0 0 0 Total 22 7 3 4 4 4 1 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 25 Sensitization events and activities. The SMH project conducted a series of sensitization events and activities that were designed to generate awareness and interest in the project’s transformation program. These activities included the baseline assessment, GRMA regional meetings, and SMS messaging. Through the baseline assessment, 120 PMHs received an introduction to the SMH project and the transformation program. The SMH project also supported routine regional meetings led by the GRMA, which provided a platform for midwives to learn about the transformation opportunities and hear from other PMHs who already successfully completed a transformation. The project also sensitized PMHs on the opportunities of transformation through SMS messaging. As discussed under evaluation question one, 89% of assisted facilities participated in the baseline assessment including 88% of non-transformed facilities and 91% of transformed facilities. Additionally, the evaluation found high levels of engagement among assisted PMHs at regional meetings, with 75% of non￾transformed facilities and 95% of transformed facilities in attendance with majority of attendance at a bi￾monthly or quarterly basis. A total of 53 SMS messages were distributed to PMHs across the project’s six target regions. Among the assisted PMHs included in the evaluation, 68% of non-transformed facilities and 86% of transformed facilities confirmed that they received the project’s SMS messages. The evaluation found higher levels of participation among transformed facilities as compared to non-transformed facilities. As mentioned, one objective of these activities was to sensitize PMHs to the benefits of ownership/management transformations and following these activities 49% of assisted PMHs expressed a desire to pursue a transformation through the submission of an EOI form.19 Of the 49% who expressed interest in transformations, 22 went on to successfully complete a transformation. Small Grants Program. After the design of the SMH project small grants program was completed at the beginning of Year 3, PMHs were made aware that facilities that successfully completed a transformation would be eligible to apply for these funds. It was made clear that each grant would be modest (at end of project, the average grant amount was GHS 18,268, or USD 4,292), and that they were to be used for purchases that strengthened the PMH’s clinical service delivery. The project awarded 22 grants (one to each transformed facility), obligating a total of GHS 401,899 (USD 94,433) in funding, which was used to purchase new equipment to support the provision of prenatal, neonatal and antenatal care, as well as family planning services, deliveries, emergency response, and diagnostic and laboratory services. These grants amounted to an additional incentive that may have motivated PMHs to pursue transformations. 19 The project utilized SMS messages as one platform to sensitize PMHs on the opportunities of transformation. These SMS messages were circulated on an on-going basis throughout the duration of the project’s period of performance. The project also received EOIs on a rolling-bases. Thus, while some of the PMHs may have submitted an EOI in response to the SMS message, other may have expressed interest prior to receiving an SMS based on sensitization through one of the project’s other platforms. The SMH trainings “provided exposure and raised awareness on the importance of continuity and a pathway to ensure sustainability.” The trainings “offered the opportunity and insights on how to pursue a transformation” and “motivated the owner to actualizing that transformation.” - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 26 NON-SMH CAUSAL FACTORS Pre-Existing Desire to Grow the PMH. It is possible that PMHs had a desire that existed before the launch of the SMH project to grow the PMH. If such a desire did exist, then PMHs might have recognized transformation as an opportunity to realize that desire. In such a case, a pre-existing desire to grow the facility would amount to a non-SMH causal factor leading a PMH to pursue a transformation. In order to determine whether a pre-existing desire to grow the PMH existed, the evaluation sought to assess the levels of motivation that existed, and document steps towards growth that occurred, before the launch of the SMH project. Through the PMH Extension Survey administered at endline, the evaluation asked PMH owners about the extent to which they were motivated to expand or grow their PMHs prior to the SMH project.20 Seventy￾eight percentage of assisted PMHs indicated that they were “strongly motivated” or “somewhat motivated” to expand or grow their facility prior to their engagement with the SMH project, including 73% of non-transformed facilities and 95% of transformed facilities. (Please see Figure 6 below.) This finding suggests that the majority of both non-transformed and transformed facilities had a pre-existing desire to grow or expand their facility. Furthermore, the evaluation gathered data on actions PMHs might have taken to grow their facilities in the years leading up to the SMH project. Taking action provides further evidence of a desire to grow the facility and is thus further evidence of pre-existing motivation to complete a transformation. Areas of exploration included applying for a bank loan, use of personal funds to facilitate the expansion or growth of the facility, and the recruitment of additional full-time staff. 20 PMH Extension Survey, 4.13 “In the two years leading up to the SMH project, how motivated were you to expand or grow your facility?” 95% 73% Figure 6: Level of Motivation to Expand or Grow PMH Prior to SMH Project, "Strongly Motivated" or "Somewhat Motivated" Non-Transformed Transformed The SMH grants program “was an additional motivation to spur the [completion of a] transformation” and “offered an opportunity [for PMHs] to own a key piece of equipment.” - Assisted Transformed Facility I “wanted to move my facility to the next level” and “meet demand due to an increase in client visits.” - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 27 As demonstrated in Table 7 below, many non-transformed and transformed facilities took some action to grow or expand their facility prior to the SMH project. For example, over half of non-transformed and three quarters of transformed facilities invested personal funds exceeding GHS 4,000 (USD 877) into their facility.21 Notably, while facilities did report that they took specific actions that would assist in growing the facility, for both non-transformed and transformed facilities the percentage of PMHs that took action is substantially lower than the percentage of PMHs that reported a pre-existing desire to grow. Table 7: Pre-existing Desire to Grow or Pre-existing Demonstration of Growth of a PMH All Assisted Facilities (n=97) Non￾Transformed Facilities (n=75) Transformed Facilities (n=22) Prior to SMH, Invest Personal Funds Exceeding GHS 4,000 into PMH 61% 57% 73% Prior to SMH, Recruit for Additional Full-Time Clinical Staff 39% 35% 55% Prior to SMH, Recruit for Additional Full-Time Non-Clinical Staff 26% 27% 23% Prior to SMH, Complete an Application for a Bank Loan 18% 17% 18% Pre-Existing Desire to take Advantage of Capitation. At the time of writing, the NHIA capitation program has been discontinued. However, in the years leading up to the SMH project and during the majority of the SMH project’s period of performance, capitation was expected to occur, and PMHs were being sensitized to its arrival and the potential impacts it would have on the PMH community. An ownership/management transformation offered PMHs an opportunity to prepare for capitation, and potentially be named as a preferred primary provider, by strengthening their business practices and clinical quality, and potentially expanding their service delivery by adding a new clinician or necessary piece of equipment. In light of this possibility, the evaluation identified the pre-existing desire to take advantage of capitation as a causal factor which may have had an influence on the successful completion of an ownership/management transformation. The evaluation found that 67% of transformed facilities had a pre-existing awareness of when capitation was anticipated to arrive in their region,22 and 60% of transformed facilities believed that capitation would indeed arrive as anticipated.23 However, only 33% of these facilities believed that capitation would arrive “very soon” or “soon.” Given the low percentage of facilities that believed capitation would arrive soon, there is less evidence of the prominence of this causal factor. Nevertheless, the high levels of overall awareness suggest that the arrival of capitation was a factor in PMH thinking and planning. Retirement or Death. Transformation is, in part, a preemptive strategy designed to strengthen the likelihood that a PMH will continue to operate even if the primary owner/midwife were to retire or die. If the primary owner/midwife was entering her advanced years and was concerned about the continuity of her facility in the event of her retirement or death, she would have a motivation – not resulting from SMH activities – to take steps to strengthen the continuity of the facility, such as pursuing a transformation. 21 The USD equivalent was calculated using the exchange rate of 1 GHS = .021915 posted on https://www.oanda.com/currency/converter/ on January 12, 2018 22 In-Depth Interview Guide, Question 5, “Did you have a pre-existing awareness of when capitation was anticipated to arrive in your region prior to the SMH project?” 23 In-Depth Interview Guide, Question 6, “Did you believe capitation would arrive?” Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 28 Even though this motivation is aligned with the design of the SMH project, the evaluation regards these circumstances as independent of the project activities. The project found that 93% of non-transformed PMH owners and 100% of transformed PMH owners were aged 55 or older. One third of non-transformed facilities and nearly one quarter of transformed PMH owners were over the age of 70. (Please see Figure 7 below.) As the vast majority of PMH owners are over the age of 55, the evaluation identified the advanced age of PMH owners as a causal factor which may have contributed to a PMH’s decision to transform. DETERMINING CONTRIBUTION Contribution was determined through the proportional piling activity discussed in the methodology section above. The overall results of the proportional piling activity are presented below in Figure 8. Of the SMH causal factors, respondents allocated the highest number of stones equally between the business and clinical trainings (22%) and the SMH small grants program (22%), followed closely by the SMH sensitization events and activities (18%). The remaining stones were distributed amongst the non-SMH causal factors including a pre-existing desire to grow the PMH (19%); retirement or death (14%); and a pre-existing desire to take advantage of capitation (5%). In total, respondents placed 1,366 stones (62%) on SMH causal factors, and 834 stones on non-SMH causal factors. As a result, the evaluation concludes that the SMH activities account for a contribution of 62% of the outcome of interest – successful completion of ownership/management transformations. 100% 93% Figure 7: Percentage of PMH Owners Over the Age of 55 Non-Transformed Transformed 5% 14% 18% 19% 22% 22% Pre-Existing Desire to take Advantage of Capitation Retirement or Death SMH Sensitization Events and Activities Pre-Existing Desire to Grow SMH Trainings SMH Small Grants Program Figure 8. Contribution to the Successful Completion of a Transformation Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 29 5.3 IMPROVED FINANCIAL, SUSTAINABILITY OPERATIONAL EFFICIENCY, AND QUALITY OF CLINICAL SERVICES It is further hypothesized that transformed facilities will engage in targeted efforts to improve the operational efficiency, business and financial management and quality of care of the PMHs. This hypothesis is based on the assumption that these new owners/managers, or facilities that have transformed their facilities, will have taken this action in order to improve financial performance as well as clinical service delivery, and will therefore work to increase the efficiency of the facility and improve the quality of the health products and services available in order to ensure the financial success of their investments and efforts. DID THE OUTCOME OF INTEREST OCCUR? Prior to exploring the contribution of the SMH project, the evaluation sought to determine whether the outcome of interest – improvements in financial sustainability, operational efficiency and quality of clinical services among transformed facilities – occurred. EVIDENCE Financial Sustainability. A core assumption of the SMH project is that the ability of a PMH to survive financially, and even expand its ability to provide health services to its community, is tied to the effectiveness of its business practices. These business practices require not only the building of the appropriate business and financial skills, but the development of processes and habits that result in the consistent and sustained application of those skills. Evaluation Question 3: To what extent, if any, have SMH project activities led to the improved financial sustainability, operational efficiency and quality of clinical services of transformed PMHs in the target regions? Figure 9. Successful Completion of an Ownership/Management Transformation contribution of the SMH project 62% 38% SMH Non-SMH 62 % Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 30 The evaluation found that, over the course of the SMH intervention, 95% of transformed facilities improved their financial sustainability as measured by the business formalization score. Transformed facilities posted an average increase of seven percentage points from an average baseline BFS of 82% at baseline to 89% at endline (see Figure 10).24 Quality of Clinical Services. The evaluation also found substantial improvements in total quality scores, as measured by the QAT, among transformed facilities from baseline to endline, with all 22 transformed facilities posting improvement. Transformed facilities received an average score of 60% at baseline and 89% at endline, an increase of 29 percentage points. Please see Figure 11 below. Operational Efficiency. The evaluation found that 59%, or 10 out of the 17 transformed facilities, posted improvements in efficiency from baseline to endline, as measured by the DEA. 25 The evaluation found five transformed facilities (29%) were “technically efficient” at endline. This demonstrates a 17 percentage point increase from baseline, at which point only two transformed facilities were found to be technically efficient.26 24 While fewer non-transformed facilities demonstrated improvement in financial sustainability as compared to transformed facilities, 87 percent of non-transformed facilities demonstrated improvements in total business formalization scores. Please see the Saving Maternity Home in Ghana, Endline Assessment Report for a comparison of non-transformed and transformed facilities. 25 Data cleaning requirements to determine operational efficiency using Data Envelopment Analysis (DEA) were more stringent than for other types of analysis. After baseline and endline data cleaning, the project was left with 17 transformed PMHs out of the total 22 to be analyzed using DEA. 26 As an indirect outcome, technical efficiency presented specific methodological challenges for the evaluation. In particular, the complexity of the methodology meant that it would have been difficult to provide respondents with 89% 82% Transformed Figure 10. Average Total Business Formalization Scores Baseline Endline 89% 60% Transformed Figure 11. Average Total Quality Assessment Tool Scores Baseline Endline 95% of transformed PMHs demonstrated an increase in financial sustainability from baseline to endline. 100% of transformed PMHs demonstrated an increase in the quality of clinical services from baseline to endline. 59% of assisted transformed PMHs demonstrated an improvement in efficiency from baseline to endline. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 31 Outcome of Interest Did Occur. In light of these findings, we can conclude that the outcome of interest did occur. Substantial improvements were recorded in each category: 95% of transformed facilities posted improvements in financial sustainability; 100% of transformed facilities posted improvements in the quality of clinical services; and 59% of transformed facilities posted improvements in efficiency. With the occurrence of the outcome of interest confirmed, the project looked to explore the causal factors that may have contributed to this occurrence. EXPLORING THE CAUSAL FACTORS RELATED TO FINANCIAL SUSTAINABILITY The evaluation identified a series of SMH interventions and non-SMH interventions that may have contributed to the improvements noted above in financial sustainability. The SMH interventions which were identified as causal factors include the successful completion of an ownership/management transformation, SMH business trainings, and one-on-one business TA (the SMH pre- and post￾transformation business TA as well as the post-transformation business TA visits). The non-SMH causal factors include a pre-existing desire to grow the PMH, a pre-existing desire to obtain external financing, and pre-existing efforts to prepare for NHIA capitation.27 SMH CAUSAL FACTORS Additional Personnel through Transformation. The successful completion of an ownership/management transformation was identified as a causal factor that may have contributed to the improvements in financial sustainability. Through a transformation, a PMH brings on a new owner, partner, lessee, manager, or clinician with a new set of skills. These new individuals are likely to bring additional knowledge and skills that will benefit the PMH. In the case of new owners, partners, leasees, or managers, it is likely that these skills will involve business administration and financial management, which would have a direct bearing on improvements in business practices and financial sustainability. In all cases, the new individual is likely to take some burden off of the primary owner/midwife, enabling her to pay more attention to all aspects of the business, including the business side. Further, key informants during both the baseline and the endline assessment emphasize that PMH owners tend to view themselves as clinicians first and then business managers second, which suggests that when time is scarce they focus on the clinical side of the PMH. When new personnel create additional time, it is reasonable to expect that it will benefit the business side of the facility. Of the 22 transformations, 21 involved bringing on a new person – either a new manager, or a skilled clinician. The final transformation involved a sale to a new owner. As discussed under evaluation question one, all 22 transformed facilities (100%) indicated they had changed the way that they manage or conduct their business and financial practices since participating in the SMH project. a clear sense of exactly what they were being asked to explain in the proportional piling portion activity. Thus, the evaluation was unable to determine contribution for improvements in technical efficiency. 27 As discussed above, 23% of transformed facilities benefitted from non-SMH interventions from other support organizations. While this support might have influenced the business practices of these facilities, given that this causal factor was only relevant for a small number of facilities, it was excluded from the non-SMH causal factors moving forward. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 32 SMH Business Trainings. As previously discussed under evaluation question one, the SMH project delivered a series of two classroom trainings focused on business strengthening and financial sustainability, the BCSP and BSS trainings. The application of the skills obtained during the BCSP and BSS trainings, such as best practices in record keeping and financial management, may have contributed to the improvements in the financial sustainability demonstrated at endline. Of the 22 transformed facilities, 82% (18) attended both the BCSP and BSS trainings, and the remaining four PMHs attended either the BCSP or the BSS training. SMH Pre- and Post-Transformation Business Technical Assistance. Through the delivery of one￾on-one business TA, transformed facilities received additional training, skills, and tools related to effective business practices. The project delivered pre-transformation business counselling to 37 PMHs, including the 22 who went on to complete a transformation. Post-transformation, the transformed facilities received a set of two business counselling visits which provided tailored TA focused on enhancing the likelihood of business and financial sustainability under the new ownership/management arrangement. The project delivered a total of 44 post-transformation business counselling visits to the 22 transformed facilities. NON-SMH CAUSAL FACTORS Pre-Existing Desire to Grow the PMH. As previously discussed, PMHs may have had an existing desire to grow their facility prior to their engagement with the SMH project. If such a desire did exist, it might have motivated PMHs to take steps towards improving their financial sustainability, independent of any interventions from the SMH project. As discussed in the previous section, the evaluation found that 80% of transformed facilities noted that they experienced a steady increase in client demand in the years leading up to the SMH project, which might inform a desire to expand the PMH. Additionally, 95% of transformed PMHs indicated that they were “strongly motivated” or “somewhat motivated” to expand or grow their facility prior to their engagement with the SMH project. A substantial number of transformed facilities also report having taken actions to expand their facilities prior to the SMH project. (For an expanded discussion of a pre-existing desire to grow the facility, see Section 5.2.) Pre-Existing Interest in External Financing. A PMH may demonstrate improvements in financial sustainability as it made improvements in business practices with the goal of qualifying for a loan from an external financial institution. For example, in order to receive a loan from a bank or microfinance My facility has seen improvements in business and financial practices “due to support from new administrative and accounting staff” hired through the transformation process. - Assisted Transformed Facility The SMH technical assistance visits “increased my knowledge and enhanced my approach” to strengthening my business practices. - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 33 institution, the PMH may have taken steps to secure any outstanding clinical accreditations and certifications, as well as becoming more diligent about separating personal and business funds and keeping financial records. The evaluation found that 19% of transformed facilities had a pre-existing interest in obtaining or pursuing a loan, as measured by the number of facilities that submitted loan applications prior to the SMH project. While this is not a high proportion of facilities, it is substantial enough to confirm the presence of this causal factor. Pre-Existing Desire to Prepare for NHIA Capitation. Facilities may have made improvements to their business practices, specifically with regards to inventory management and designating business and financial management staff, as part of their preparations for the expected arrival of capitation, and potentially to be named as a preferred primary provider. As discussed under evaluation question two, while the capitation program has been discontinued, PMHs were being sensitized to its arrival and the potential impacts it would have on the PMH community. While 60% of transformed PMHs believed capitation would arrive “as expected”, only 33% believed it would arrive “soon” or “very soon”. Despite these doubts about the timing of its arrival, the high levels of awareness suggest that NHIA capitation was a factor in PMH planning. EXPLORING THE CAUSAL FACTORS RELATED TO QUALITY OF CLINICAL SERVICES The following section highlights the six causal factors, comprised of three SMH interventions and three non-SMH interventions that may have contributed to the improvements in the quality of clinical services. The SMH intervention causal factors presented below include the successful completion of a transformation, SMH CCB training, and post-transformation clinical TA. 28 The three non-SMH causal factors include pre-existing preparations for NHIA capitation, TA provided by other non-SMH actors, and participation in continuing professional development programs. SMH CAUSAL FACTORS Additional Personnel through Transformation. The successful completion of an ownership/management transformation was also identified as causal factor that may have contributed to the improvements in the quality of clinical services. The completion of a transformation and the addition of a new owner, partner, lessee, manager, or clinician with a new set of skills, is likely to bring additional knowledge and skills that will benefit the PMH. Of the 22 transformations, 21 involved bringing on a new person – either a new manager, or a skilled clinician. The final transformation involved a sale to a new owner. SMH Clinical Capacity Building Training. As previously noted, the project delivered the CCB training focused on quality assurance, emergency response, safe motherhood and advocacy to 153 participants from 149 assisted PMHs. Of the 22 transformed facilities, 86% (19) attended the CCB trainings. The application of the skills obtained during the CCB trainings may have contributed to the increase in the quality of clinical services (QAT score) from baseline to endline. 28 While the SMH Small Grants Program may have also led to the improvements in the quality of clinical services provided by transformed facilities through the purchase of new equipment to support essential MNCH services; the evaluation did not identify the grants program as a causal factor at this time. As the grants were awarded throughout the final year of the SMH project, the effects of the grants on the quality of clinical service is expected to lag, as it takes time for word to spread to the community that the facility has made these improvements, enabling the facility to fully make use of the new equipment. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 34 SMH Post-Transformation Clinical Technical Assistance. As described above, the project provided one-on-one post-transformation clinical counselling visits to transformed facilities to ensure the provision of high-quality health services. These visits were designed to assist PMHs to improve their skills and tools, improvements that would likely affect the quality of their clinical services as measured by the QAT. The project delivered 44 post-transformation clinical counselling visits to the 22 transformed PMHs. NON-SMH CAUSAL FACTORS Pre-Existing Desire to Prepare for NHIA Capitation. Improvements in transformed facilities quality of clinical services may have been impacted by a facility’s preparations for the arrival of NHIA capitation, as was the case with improvements in financial sustainability. The QAT was designed, in part, to mirror the NHIA accreditation tool to support PMHs in pursuing accreditation. If PMH facilities made specific improvements to the quality of their clinical practices in order to enhance their chances of securing NHIA accreditation, these improvements would be detected by the QAT administered at endline. Technical Assistance Provided by Non-SMH Programs. The project recognizes that there are other initiatives which have and are currently operating in the same six regions targeted by the SMH project with the ultimate goal of improving health outcomes. Eighty-six percent of transformed facilities confirmed that they received clinical TA outside of the SMH project, during the project’s period of performance. Some of the programs mentioned by the assisted facilities include those led by Marie Stopes International, the Ministry of Health, the USAID-funded Strengthening Health Outcomes through the Private Sector (SHOPS) program, Ghana Health Service, District-level Health Directorates and Authorities, and regional hospitals. Participation in Continuing Professional Development Programs. The evaluation found, in addition to the TA provided by non-SMH programs, a large percentage of transformed facilities (93%) indicated that they participated in continuing professional development programs. These programs, including those facilitated by the Ministry of Health, Ghana Health Service and other USAID-funded programs, were identified as a causal factor which may have contributed to the improvements observed in the quality of clinical care. Please note, while the project provided clarification to transformed facilities during the proportional piling activity to differentiate TA provided by non-SMH programs and continuing The SMH technical assistance visits provided an opportunity to “ask all the questions [I desired] one-on-one” with the SMH project. Additionally, “because weaknesses are easily identified and corrected immediately, unlike workshops where you might not know whether you are doing well or not” I found the counselling visits particularly useful. - Assisted Transformed Facility “Ghana Health Service’s interventions and workshops have been useful.” Additionally, “they visit my facility regularly to make interventions and recommendations to improve my clinical practices.” - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 35 professional development programs, the project believes that there may be some overlap. Further research is needed to understand the impacts that these two non-SMH causal factors had on the improvements on the quality of clinical care. DETERMINING CONTRIBUTION Improvements in Financial Sustainability Of the six causal factors present, transformed facilities allocated a total of 1,531 stones of the available 2,100 stones to SMH project intervention causal factors; with the highest number of stones (31%) allocated to the SMH business trainings followed closely by the SMH business TA (29%) and the addition of new personnel through the SMH transformation at (12%).29 The remaining 569 stones were distributed amongst the competing explanations including pre-existing desired to grow (21%); pre-existing interest in external financing (3%); and pre-existing desire to prepare for capitation (3%). Please see Figure 12 below. As a result of the proportional piling activity, the evaluation found that the SMH project accounts for a contribution of 73% of the improvements in financial sustainability and the remaining 27% to non-SMH interventions. 29 One of the responding transformed facilities was unable to complete the proportional piling activity in relation to evaluation question three. In light of this, the total number of available stones was reduce for this evaluation question to reflect the participation of 21 transformed facilities rather than all 22. 3% 3% 12% 21% 29% 31% Pre-existing Interest in External Financing Pre-existing Desire to Prepare for… Additional Personnel through SMH… Pre-existing Desire to Grow SMH Business Technical Assistance SMH Business Trainings Figure 12. Contribution to the Improvements in Financial Sustainability Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 36 Improvement in Quality of Clinical Services The evaluation respondents (transformed facilities) determined that among the six causal factors present, allocated a total of 1,496 stones of the available 1,900 stones to SMH project intervention causal factors. The transformed facilities allocated the highest number of stones (29%) to the addition of a new personnel through the SMH transformation followed closely by the SMH post-transformation clinical TA (26%) and the SMH clinical trainings at (24%). The remaining 404 stones were distributed amongst the competing explanations including participation in continuing profession development programs (13%); non-SMH TA (7%); and a pre-existing desire to prepare for capitation (1%). Please see Figure 14 below. As a result, the evaluation concludes that the SMH project activities account for a contribution of 79% of the outcome of interest – improvements in the quality of clinical services. 1% 7% 13% 24% 26% 29% Pre-Existing Desire to Prepare for NHIA Capitation Non-SMH Programs Technical Assistance Participation in Continuing Professional Development SMH Clinical Capacity Building Training SMH Clinical Technical Assistance Additional Personnel through Transformation Figure 14. Contribution to the Improvements in Quality of Clinical Services contribution of the SMH project 73% 27% SMH Non-SMH 73% Figure 13. Improvements in Financial Sustainability Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 37 5.4 EXPANSION OF TRANSFORMED PMHS The SMH project hypothesized that improvements in operational efficiency, business and financial sustainability and quality of care will result in the expansion of transformed facilities and increased capacity of PMHs to administer essential MNCH services in underserved areas. The evaluation sought to explore this hypothesis by investigating whether assisted transformed facilities expanded. Expansion may manifest itself with the addition of new health services or by increasing the capacity of a facility as measured by number of beds or full-time staff. Such expansion should ultimately increase the number of client visits, which suggests an increase in the PMH’s ability to provide health services to its community. Furthermore, the evaluation sought to uncover the role that the SMH inventions played in producing this outcome. DID THE OUTCOME OF INTEREST OCCUR? Prior to exploring the contribution of the SMH project, the evaluation sought to determine whether the outcome of interest, the expansion of transformed facilities, occurred. Evidence Referencing the endline findings, the evaluation sought to determine whether transformed facilities experienced an increase in the number of services provided, an increase in the number of available beds, or an increase in the number of full-time staff to determine whether a facility had expanded. As part of the QAT, the project asked respondents to identify the range of services provided at the facility using a list of 32 services. Sixty-five percent of transformed facilities demonstrated an increase in services Evaluation Question 4: To what extent, if any, have SMH project activities led to the expansion of transformed PMHs in the target regions? Figure 15. Improvements in Quality of Clinical Services contribution of the SMH project 79% 21% SMH Non-SMH 79% Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 38 provided. Transformed facilities offered an average of 22 services at endline, up from an average of 18 services at baseline. 30 Please see Figure 16 below. In addition to the number of health services, the project also defined expansion by the increased size capacity of a PMH, as measured by the number of full-time staff members and number of beds. While on average, transformed facilities experienced a reduction of three full-time clinical staff members, 45% of transformed facilities did experience an expansion of their staff size, averaging an expansion of three full￾time staff members including both clinical and non-clinical staff. With regards to the number of beds, 65% of transformed facilities demonstrate an increase in available beds, with an average of 11 beds at endline as compared to 9 beds at baseline. Please see Figure 17 below. Outcome of Interest Did Occur. Based on the evidence presented above, we conclude that the outcome of interest occurred, with 85% of assisted transformed PMHs expanding from baseline to endline in number of services provided, number of full-time staff and/or number of beds. Sixty-five percent of transformed facilities demonstrated expansion across two of the three measures, while 25% expanded in all three areas. With the occurrence of the outcome of interest confirmed, the evaluation explored the causal factors to determine which factors contributed to the outcome of interest. EXPLORING THE CAUSAL FACTORS This section introduces the six causal factors. The causal factors presented below represent three SMH causal factors (transformation, business and clinical trainings and TA, and the small grants program) as well as three non-SMH causal factors (PMH owners’ pre-existing desire for growth, pre-existing preparations for NHIA capitation, and assistance from non-SMH programs). SMH CAUSAL FACTORS Additional Personnel through Transformation. The successful completion of an ownership/management transformation was identified as causal factor that may have contributed to the expansion of transformed facilities. As noted above, with introduction of new owners, partner, lessee, 30 Non-transformed facilities also demonstrated improvements in the average number of services provided, averaging a total of 19 service types at endline compared to 15 at baseline. Please see the Saving Maternity Homes in Ghana, Endline Assessment Report for a detailed discussion comparing the findings of non-transformed and transformed facilities. 22 18 Transformed Figure 16. Average Number of Services Provided Baseline Endline 11 9 Transformed Figure 17. Average Number of Beds Baseline Endline Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 39 manager, or clinical staff member, leads to an increase in staff size and possible introduction of new services not previously offered at a facility. SMH Business and Clinical Trainings and Technical Assistance. While the SMH business and clinical trainings and TA were not designed directly to facilitate the expansion of a PMH, these activities were designed to improve business and financial sustainability and quality of clinical care. It is hypothesized that improvements in business and financial practices and quality of clinical services ultimately affect the PMH’s reputation within the community, leading to expanded service delivery. If the PMH is perceived to provide higher quality services and to be financially sustainable, the PMH will expand and experience an increase in clientele. SMH Small Grants Program. After the successful completion of a transformation, PMHs were eligible to apply for a grant under the SMH project’s small grants program. The most common use of funds (46%) were used to support the purchase of examination diagnostic equipment and tools for maternal and newborn care, including ultrasound machines, fetal Doppler and blood pressure monitors. Twenty-five percent of the grants facilitated the procurement of laboratory equipment, including but not limited to hemoglobin probes, auto-hematology analyzers, water baths, microscopes and laboratory ovens. The third most common use of the grant funds was to support the acquisition of beds and other clinical furniture, accounting for 19% of the total obligated funds. These grants supported the expansion of services provided at transformed facilities as well as the number of beds available to treat patients. Non-SMH Causal Factors Pre-Existing Desire to Grow the PMH. As previously discussed, PMHs may have had an existing desire to grow their facility prior to their engagement with the SMH project. If such a desire did exist, it might have motivated PMHs to take steps towards expanding their facility. Pre-Existing Desire to Prepare for NHIA Capitation. Facilities may have supported the expansion of their facility outside of the SMH project in preparation for the arrival of NHIA capitation. As 60% of transformed PMHs believed capitation would arrive “as expected,” PMHs may have taken steps to ensure their survival under capitation by expanding their facility, in hopes that they would be able to serve as a primary preferred provider within their community. Technical Assistance Provided by Non-SMH Programs. As previously discussed in the evaluation questions above, the project recognizes that there are other initiatives operating in the same regions targeted by the SMH project which may have assisted in the expansion of transformed PMHs. Such programs may have delivered financial or technical support that assisted PMHs to expand. (For an expanded discussion of a TA provided by non-SMH programs, see Section 5.1.) DETERMINING CONTRIBUTION Of the six causal factors present, transformed facilities allocated a total of 1,555 stones of the available 2,100 stones to SMH causal factors; with the highest number of stones (30%) allocated to the SMH small The expansion of my facility “is due to the contribution and laboratory set up of the SMH grants program.” In combination with addition of the Physician’s Assistant through the transformation program, the expansion of my facility was “complete with the laboratory and scan services.” - Assisted Transformed Facility Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 40 grants program followed closely by the SMH business and clinical trainings and TA (27%) and the addition of new personnel through the SMH transformation at (13%). The remaining 645 stones were distributed amongst the non-SMH causal factors, including pre-existing desire to grow the PMH (14%); TA provided by Non-SMH programs (11%); and pre-existing desire to prepare for NHIA capitation (4%). Please see Figure 18 below. In total, respondents placed 1,555 stones (71%) on SMH causal factors, and 645 stones (29%) on non￾SMH causal factors. As a result of the proportional piling activity, the evaluation found that the SMH project accounts for a contribution of 71% of the expansion of their facility to the SMH project. 4% 11% 13% 14% 27% 30% Pre-Existing Desire to Prepare for NHIA Capitation Non-SMH Programs Technical Assistance Additional of Personnel through Transformation Pre-Existing Desire to Grow SMH Trainings and Technical Assistance SMH Small Grants Program Figure 18. Contribution to the Expansion of Transformed Facilities contribution to the SMH project 71% 29% SMH Non-SMH 71% Figure 19. Expansion of Transformed Facilities Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 41 6. CONCLUSIONS AND RECOMMENDATIONS 1. The SMH Project made a significant contribution to the outcomes of interest. For each evaluation question, respondents indicated that the SMH project largely contributed to the outcomes observed from 62% for the successful completion of management transformations to 90% for increased awareness of the benefits of business continuity and financial sustainability. Overall, the transformed facilities allocated a total of 7,928 out of 10,600 stones (75%) to the SMH causal factors during the proportional piling activity. Recommendation: It is recommended that USAID consider building upon the achievements of the SMH project, and extend and/or replicate the project model. The SMH project itself was a pilot effort, and these results suggest that an extension of this pilot would indeed support the expansion of access to quality maternal child health services in rural, underserved areas of Ghana. 2. There is an opportunity for substantial gains from strengthening business and financial practices. The evaluation found that 92% of assisted PMHs (non-transformed and transformed) demonstrated an increased awareness of the benefits of business continuity and financial sustainability. Among the causal factors identified, the SMH project accounts for a contribution of 90% to the increased awareness of business continuity and financial sustainability among assisted PMHs. The remaining ten percent spanned across both SMH and non-SMH causal factors. Additionally, a large percentage of PMHs, 68% of non￾transformed and 100% of transformed, were found to have applied this increased awareness to improve their business and financial practices. The endline assessment found that 87% of non-transformed facilities and 95% of transformed facilities improved their financial sustainability. While these findings indicate that the SMH causal factors did have a significant influence on the increased awareness demonstrated by assisted PMHs, they also suggest that, outside of the SMH project, PMHs receive little training or TA in business skills. The SMH experience suggests that when PMHs do receive this assistance, it results in substantial gains. Improved business practices enable a PMH to strengthen its financial sustainability, and potentially to invest increased profits into new equipment, new staff, or facility expansion. Such investments would enable the facility to increase and improve its ability to provide health services to its community. Recommendation: It is recommended that USAID consider supporting additional programming focused on the business and finance side of PMHs, and potentially other private health providers. While SMH focused on PMHs, it is likely that similar support would also benefit other types of private health providers in underserved areas. 3. New personnel contribute most immediately to quality improvements. Respondents indicated that new personnel were the most significant contributor to the improvements in the quality of clinical services. In contrast, the contribution of the additional personnel was identified as only the fourth most significant causal factor for both improvements in financial sustainability and expansion of the PMH. This placement lagged nearly 20% behind the SMH trainings and TA in each case. These findings suggest that facilities were aware of the early-stage contribution that clinical staff made to their facilities, but they placed less value on the benefits of the additional personnel in other areas. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 42 The most likely explanation for this finding is that, at the point of data collection, sufficient time had not passed for the broader benefits of new personnel to manifest. Most transformations were not completed until later in the project, with 16 of the 22 transformed facilities (73%) operating with the new personnel for less than one year at the point of evaluation. While contributions to the core competency of the PMH (clinical service delivery) would have occurred directly, there would have been a lag before contributions manifest in financial sustainability and expansion. Recommendation: It is recommended that GRMA and USAID continue to track transformed facilities to gain a better sense of the value of the contributions of the new personnel to non-clinical dimensions of the PMH business. 4. The combination of training and technical assistance, small grants, and a pre-existing desire to grow appears to be effective in catalyzing change. In relation to completing a transformation and expanding the facility, the evaluation found that the SMH small grants program, SMH trainings and TA, and a pre-existing desire to grow were the top three causal factors. Of the causal factors related to completion of a transformation, respondents gave equal weight to SMH trainings (22%) and the SMH small grants program (22% each), followed closely by the pre-existing desire for growth (19%). Similarly, regarding the causal factors contributing to expansion, respondents were influenced by the SMH small grants program (30%), SMH trainings and technical assistance (27%) and a pre-existing desire to grow (14%). These findings suggest that a combination of structured trainings and technical assistance and a targeted small grants program may be an effective intervention package when working with PMHs that have a desire to grow. The pre-existing desire for growth is an important factor, as it is likely this motivation that leads the PMH to apply learning from the training and make optimal use of grant-funded investments. It may be the case that independently motivated facilities are concurrently capitalizing on a range of opportunities – not only those offered by assistance programs like the SMH project – and so may already be operating at a high level. Nevertheless, the ultimate aim of the SMH project is improved health outcomes, and investing in facilities that are not sufficiently motivated to apply the training or effectively leverage grant funds is often a poor use of development resources. This conclusion suggests that directing resources to facilities that bring an existing motivation is a way to increase the value for money of development assistance. The SMH project’s positive self-selection approach, that involves inviting interested PMHs to submit expressions of interest and then conducting a multi-stage screening process before investing additional time and resources facilitating a transformation, likely contributed to these outcomes. Recommendation: The project recommends that USAID and other donors explore approaches to screening participants in private health sector assistance programs for a pre-existing desire for growth, and using a positive self-selection approach to identify these people. 5. While this innovative evaluation methodology provides a useful indication of contribution, a more extensive study would yield more certainty. Drawing on the GEM methodology described above, this evaluation used a new approach to gaining some understanding of contribution because, resources were not available for a full impact evaluation. A true impact evaluation, including a randomized control trial or other scientific design, would have enabled the project to measure the change in project outcomes that are attributable to the project. Recommendation. It is recommended that, should USAID decide to replicate or scale-up the innovative SMH intervention, it should conduct an external impact evaluation to validate these conclusions and determine the extent to which this model can support the expansion of maternal and child health services. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 43 ANNEX 1: EVALUATION SCOPE OF WORK The Saving Maternity Homes (SMH) Project will carry out an internal end of project performance evaluation to generate findings related to the effectiveness and impact of project activities along with lessons learned for the purpose of improving, replicating, or scaling-up this innovative intervention. The evaluation will employ an innovative methodology to determine the levels of contribution for the outcomes observed.31 PURPOSE OF THE EVALUATION AND TARGET AUDIENCE The most reliable approach to determining and attributing impact is an impact evaluation. The USAID Evaluation Policy defines an impact evaluation as an evaluation that seeks to measure the change in project outcomes that are attributable to (caused by) the project using methods that a construct a “credible and rigorously defined counterfactual” so as “to control for factors other than the intervention that might account for the observed change.”32 Creating a credible and rigorously defined counterfactual requires methods for comparing a treatment group of project participants and a control group of non-participants who are as similar as possible to project participants. Unfortunately, the SMH project does not have the resources to conduct an impact evaluation using the methods necessary to create a credible or rigorously defined counterfactual. However, given the innovative nature of the project design and the learning mandate built into the Innovate for Health mechanism, the SMH project will conduct a limited, internal evaluation using less rigorous, but nonetheless sound, methods. The purpose of the evaluation will be to generate findings related to the effectiveness and impact of project activities along with lessons learned for the purpose of improving, replicating, or scaling-up aspects of the intervention, or to justify a more rigorous evaluation of future programming that takes a similar approach. The primary target audience of the evaluation is USAID/Ghana staff. Additionally, the evaluation will be of interest to other implementing partners, the Government of Ghana (GoG), public health organizations, academics and research institutions, the Ghana Registered Midwives Association (GRMA), and private maternity homes (PMHs). It is also anticipated that the evaluation will be useful for any other stakeholders interested in supporting the sustainability of PMHs, while enabling local communities to continue to enjoy access to the critical health services PMHs provide. Findings may also have broader implications for working with the private health sector in rural and underserved areas, both within Ghana and in other low resource settings. BACKGROUND The USAID-funded, $1.78 million Saving Maternity Homes in Ghana, Innovate for Health project, implemented by Banyan Global in partnership with the Ghana Registered Midwives Association, focuses on revitalizing private maternity homes in Ghana to expand access to quality maternal and child health services. The project aims to facilitate ownership and/or management transformations that will result in 31 The following sections provide a detailed account of the evaluation scope of work. The evaluation methodology was summarized in the SMH project’s Year 4 Monitoring and Evaluation Plan, submitted to USAID on September 1, 2017. 32 USAID, 2011. USAID Evaluation Policy. U.S. Agency for International Development, Washington, DC Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 44 increased capacity, improved quality, and expanded services. The project is targeting six regions in Ghana: Brong Ahafo, Western, Central, Eastern, Volta, and Northern. Ghana is experiencing rapid and sustained economic growth, but large challenges remain, particularly in the area of health. The Government of Ghana has taken steps to address these challenges and improve access to healthcare services, most notably through the launch of the National Health Insurance Scheme (NHIS). However, access to healthcare, particularly in rural areas, remains limited. Private maternity homes are important providers of health services, especially in rural areas; however, in recent years the sector has experienced a steady decline as midwives have aged and facilities have closed. GRMA estimates that in the five years leading up to the start of the project, 100 PMHs closed. The SMH project, which began in September of 2014, aims to help address these challenges through increasing awareness of ownership and management transformations, and facilitating these transformations directly in the target regions. Specifically, the project: develops the capacity of PMHs in business continuity planning; brokers the transfer of declining PMHs to interested investors; leverages financing and investment – both in the form of small funds disbursements directly by the project, and third￾party investment – to improve quality and expand service delivery; provides business strengthening and clinical capacity building to new owners/managers; develops the capacity of GRMA; and conducts operations research to enhance the understanding of important issues and considerations related to the project’s design. Participating PMHs receive various “tiers” of project support, ranging from basic peer support visits (PSVs) led by GRMA, to training in business continuity and clinical services strengthening, to support for effectively implementing an ownership/management transformation. The SMH project theory of change posits that the project can take specific steps to facilitate ownership/management transformations, and that these activities will result in increased sustainability and expanded operations of PMHs in rural areas of Ghana, thereby improving access to maternal, neonatal and child health services. EVALUATION QUESTIONS The SMH project evaluation will seek to answer the following four evaluation questions: 1. Evaluation Question for Step 1 of the Theory of Change. To what extent, if any, have SMH project activities increased the awareness of the benefits of business continuity and financial sustainability of PMHs in the target regions? 2. Evaluation Question for Step 2 of the Theory of Change. To what extent, if any, have SMH project activities led to the successful completion of ownership/management transformations for PMHs in the target regions? 3. Evaluation Question for Step 3 of the Theory of Change. To what extent, if any, have SMH project activities led to the improved financial sustainability, operational efficiency and the quality of clinical services of transformed PMHs in the target regions? 4. Evaluation Question for Step 4 of the Theory of Change. To what extent, if any, have SMH project activities led to the expansion of transformed PMHs in the target regions? Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 45 EVALUATION METHODOLOGY The project will employ a modified General Elimination Methodology (GEM),33 a non-experimental methodology that focuses on systematically identifying and eliminating alternative causal factors that could explain an observed result. While a non-experimental evaluation does not provide the level of rigor of an impact evaluation, it will enable the project to gain some understanding of project effectiveness and the contribution of the SMH project to the outcomes observed. An outline of the planned evaluation methodology is as follows: 1. Definition of group to be evaluated: The group to be evaluated consists of assisted non-transformed and transformed PMHs in the target regions. 2. Definition of outcomes of interest: In the case of the SMH project, the outcomes of interest are articulated in the theory of change and can be measured through survey questions. 3. List of causal factors for the outcomes of interest: The project relied on the expertise of project staff, project beneficiaries, key informants and local partners to develop a list of SMH intervention as well as non-SMH intervention causal factors that contributed to the outcomes of interest. In an effort to keep the process manageable, and to avoid favoring the SMH intervention, for most evaluation questions the evaluation identified three SMH and three non-SMH causal factors. Please note for evaluation question one, which focuses on raising awareness, the evaluation identified only one SMH causal factor (the project’s efforts to raise awareness) and one non-SMH causal factor (other stakeholders’ efforts to raise awareness). 4. Identification of whether the outcomes of interest have occurred and determination if the causal factors are present. The “outcomes of interest” are the outcomes articulated in the theory of change. For example, for Evaluation Question 1, the outcome of interest is an increased awareness of the benefits of business continuity and financial sustainability. Relying on data from the endline assessment, the evaluation confirmed that the outcomes of interest occurred, and which causal factors were present. The causal factors discussed in this report were all found to be present and thus determined to be potential factors which may have contributed to the outcomes of interest. 5. Determination of the relative contribution to SMH and non-SMH causal factors. The evaluation analyzed the data to gain some idea of the extent to which the project intervention and the non-project interventions could have contributed to each outcome of interest. Using a proportional piling research method (explained below), transformed facilities determined the relative contribution of the project’s interventions as compared to the non-SMH causal factors for the outcomes observed. Proportional piling allowed respondents to rank the causal factors that contributed on some level to the results observed. While this methodology does not provide the same level of statistical rigor and confidence that is achievable in an impact evaluation, it does provide a systematic approach to identifying and weighting factors contributing to various project outcomes, while recognizing budget and time constraints. We believe that this approach will enable us to gain some useful understanding of the extent to which project 33 For an overview of GEM, see Howard White and Daniel Phillips, “Addressing attribution of cause and effect in small n impact evaluations: towards an integrated framework,” International Initiative for Impact Evaluation, Working Paper 15, 2012. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 46 activities caused the expected outcomes and desired results, while recognizing budget and time constraints. DATA COLLECTION AND ANALYSIS The evaluation will draw largely on the data collected during the baseline and endline assessment to identify the extent to which the outcomes of interest have occurred. Building on this data, the project will conduct in-depth interviews with transformed facilities to gain a clearer understanding of the events. A proportional piling activity, a participatory research method, will be integrated into the In-Depth Interviews with transformed facilities to determine the relative contribution of the project’s interventions as compared to the competing explanations or non-project interventions for the outcomes observed. The proportional piling activity will allow respondents to rank the influential factors that contributed on some level to the results observed. Once the evaluation data has been gathered, the project will analyze the data to identify the extent to which each outcome of interest has occurred and the extent to which each competing explanation played a causal role in the observed outcomes. The project will then summarize this analysis in an evaluation report, indicating clearly the extent to which project activities achieved the desired impact, and highlighting useful learning that could benefit future interventions. DISSEMINATION The Internal End of Project Performance Evaluation Report will summarize the evaluation results and will be shared with GRMA, GHS, NHIA as well as other stakeholders. SMH will organize an end of project event to present the findings of the evaluation results and lessons learned. Raw data will be provided upon request, although all specific identifying information will be kept strictly confidential and will only be available to project staff and to USAID. In order to ensure that privacy is maintained, dissemination will be limited to information and findings derived from summary analysis. The Internal End of Project Performance Evaluation Report will be submitted to USAID’s Development Experience Clearinghouse (DEC), and the dataset will be submitted to USAID’s Development Data Library (DDL) in a compliant machine-readable, non-proprietary format. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 47 ANNEX 2: RESULTS FRAMEWORK The following summarizes the SMH results framework that captures the overall long-term goal, strategic objective and intermediate results as they relate to the three key project components: pre-transformation activities, transformation activities, and sustainability activities. USAID DO 3: Equitable Improvements in Health Status IR 3.1: Increased access to integrated health services IR 3.3: Strengthened and responsive health system PROGRAM GOAL: Increase private maternity homes' ability to sustain and grow quality of health services in the underserved areas of Ghana STRATEGIC OBJECTIVE: PMHs sustained, MCH services expanded, and efficiency and quality improved IR 3: SUSTAINABILITY Business and programmatic capacity of PMHs and GRMA increased Sub-IR 3.1: Capacity of PMHs to sustain, increase and improve service delivery strengthened Sub-IR 3.2: Increased capacity of PMHs to manage businesses and offer health services Sub-IR 3.3 Management and programmatic capacity of GRMA increased IR 2: TRANSFORMATION PMHs under new ownership / management transformed and improved/expanded health service delivery Sub-IR 2.2: Funding provided and leveraged by the project Sub-IR 2.1: PMH ownership / management transformations brokered IR 1: PRE￾TRANSFORMATION PMH owners and managers ready for ownership / management transformation Sub-IR 1.1: PMHs interested in and ready for transformation identified Sub-IR 1.2: Capacity of PMHs for ownership / management transformation strengthened Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 48 ANNEX 3: GHANA HEALTH SERVICE ETHICAL APPROVAL Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 49 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 50 ANNEX 4: DATA COLLECTION TOOLS AND INSTRUMENTS Annex 3 includes the following five survey tools which were used to inform the evaluation: 1. PMH Survey Tool 2. PMH Extension Survey Tool 3. Quality Assessment Tool 4. Key Informant Interview Guide 5. In-Depth Interview Guide Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 51 SMH PROJECT PMH SURVEY PART 1: PRE-INTERVIEW DATA The interviewer should complete the following questions before beginning the interview. # Category Answers (and Coding) Instructions 1.1 Interviewer Name 1.2 Date 1 7 Day Month Year 1.3 Region 1.4 District 1.5 Town 1.6 GPS Coordinates Please record the GPS coordinates of the facility. 1.7 Photograph of facility sign Please take a photograph of the facility sign or building. PART 2: INTRODUCTION AND INFORMED CONSENT GRMA will provide an introduction. The introduction will include the following:  Description of the SMH project, and the roles of GRMA and Banyan Global  Explanation of the two parts of this visit: 1. The SMH survey questionnaire, for information gathering purposes 2. The GRMA Peer Monitoring Visit (PMV), the purposes of which is to gather information, and to provide feedback to the PMH abut areas in which they might improve  Estimate of the total time required – about 2 hours  An understanding of the sequencing 1. We will ask some general questions about your facility, that are relevant to both GRMA and Banyan Global 2. Banyan Global will ask some questions, while GRMA tours the facility and reviews whether specific equipment and other items are in place 3. GRMA will ask some questions, while Banyan Global works with assistants (if possible) to sight some specific documents related to business operations 4. GRMA will give specific feedback (Banyan Global will not be in the room) 5. We will thank you for your time, and conclude the visit Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 52 Once GRMA has concluded the introduction, and left the room to tour the facility and sight the equipment and other items, the enumerator should read the following to the respondent: Hello. My name is ________________. As you just heard from GRMA, we are conducting a survey of Private Maternity Homes as part of a USAID funded project called Saving Maternity Homes (SMH), which seeks to revitalize declining or closed private maternity homes. The purpose of the survey is to learn more about your facility and the services you offer. Participation in the survey is voluntary and you are free to decline to answer any or all questions. The results will be kept confidential and will only be used to help the project do better work in this area. [Respondents will be limited to persons able to give personal consent. Subjects not able to give personal consent (e.g. for cultural reasons, children or adolescents less than the legal age of consent, subjects with mental illness) should not be subjects of this study, and will not be interviewed. If the respondent is incapable of reading and signing the written consent form, the enumerators will read the form to them in order to acquire informed consent.] This survey usually takes 60 minutes to complete. Will you participate in this survey? Yes………………………………………………………………………….…..1 No……………………………………………………………………………....2 PART 3: FACILITY INFORMATION The interviewer should read aloud: I’d like to start by learning a little bit about your facility. # Question Answers (and Coding) Instructions N/A Could you tell us a bit more about the history of this facility? N/A The purpose of this question is to give the respondent a chance to talk about the facility a bit, and to put her at ease. Do not worry about recording any information at this point (though some of what is said might be relevant for later questions). But do pay attention to the time, and do not let the respondent speak for more than 1-2 minutes. 3.1 What is your name? 3.2 Respondent sex Male……………………………………………..…1 Female………………………………..……..............2 3.3 What is your position or title? If the respondent is not the owner, manager, or Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 53 # Question Answers (and Coding) Instructions another person with significant responsibility, confer with GRMA about the possibility of postponing the interview. 3.4 How long have you been working with this facility? 3.5 What is the name of the facility? 3.6 What is the location address of the facility? Get the address, the district, and the region. 3.7 What is the facility’s landline telephone number (if any)? 3.8 What is the best mobile number for the facility? 3.9 What is the best email address for the facility? 3.10 What is the facility’s website (if any)? 3.11 How many years has the facility been in operation? ___________ years Don’t know………………………………..……..89 Refused……………...………..…………………..99 PART 4: OWNERSHIP AND MANAGEMENT Read aloud: Now I would like to ask some questions about the ownership and management of the facility. # Question Answers (and Coding) Instructions 4.1 How many individuals have an ownership stake in this facility? ___________ individual(s) If 1, go to question 4.2. 4.1.1 If there are multiple owners, please indicate the ownership stake (in percentage terms) for each owner. Collect the relationship of each partner (family, friend, business associate, etc.), and ownership stake of each individual. 4.2 What is the name of the primary owner of the facility? Same as respondent………………………..…........1 ______________ [name]………………………..2 Refused………………………………...………....99 The primary owner is the one who has the largest ownership stake. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 54 # Question Answers (and Coding) Instructions 4.3 What is the sex of the primary owner? Male……………………………..………............….1 Female…………….…………………….…..............2 4.3.1 What is the age of the primary owner? __________ years At this point, GRMA can take their leave to begin the tour of the facility. The Banyan Global enumerators will then continue with the questioning. 4.4 How long has this person been the owner of the facility? ___________ years Don’t know………………………………….........89 Refused………………………………….................99 4.4.1 What is the owner’s professional background? Midwife………………………………..............…….1 Other medical professional……………………..…2 Non-medical health professional……….…..............3 Other (Specify)………………..…………………...4 4.5 How did this person come to be the owner of the facility? Founded the facility………………………………..1 Purchased the facility…………………….………...2 Inherited from parents/family….……..…..………..3 Other (Specify)………………………...…………..4 4.5.1 If this person purchased the facility, what funds were used? Personal savings…………………………….….…..1 Loan from family/friends……………………….….2 Loan from financial institution…………………….3 Third party investment…………………...……….4 Grant from government…………………………..5 Grant from foundation or NGO………………….6 Other (Specify)……………………………..............7 The respondent should check all that apply. 4.6 Is the owner of the facility also the manager of the facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, skip to question 4.10. 4.7 What is the sex of the manager? Male…………………………………………….….1 Female…………….…………………...…………...2 4.8 How long has this person been the manager of the facility? ___________ years Don’t know………………………….……….…..89 Refused……………………………...…….............99 4.9 What is the manager’s professional background? Midwife………………………………...…………..1 Other medical professional……………………….2 Non-medical health professional…………..............3 Other (Specify)…………………………….............4 4.10 To what extent do you agree with the following Strongly agree…………………………..............…..1 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 55 # Question Answers (and Coding) Instructions statement: “The financial future of private maternity homes like this one is promising.” Agree………………………...…………...................2 Neutral……………………..……………….......….3 Disagree……………………………………...……4 Strongly disagree………………………...………...5 4.11 To what extent do you agree with the following statement: that “Expanding or growing my facility would be a wise decision.” Strongly agree…………………………..............…..1 Agree………………………...…………...................2 Neutral……………………..……………….......….3 Disagree……………………………………...……4 Strongly disagree………………………...………...5 4.12 If you were to pursue expansion or growth, where might you see opportunities for expansion or growth? New building or addition………………………….1 Additional employees……………………………...2 Additional equipment or new technologies……….3 Other (specify)…………………………………….4 Respondent should check all that apply. 4.13 What are the constraints to the growth or expansion opportunities mentioned above? Limitations to the range of services you can offer…1 Not willing to invest at this time……………….….2 Insufficient funds…………………………………...3 Delays in payments from NHIA…………………...4 Owner/Manager does not have the time/energy to take the necessary steps………………………...…5 Owner/Manager does not have the business management skills that would be necessary……….6 Owner/Manager does not have the clinical skills that would be necessary………………………....……...7 Competition from other health facilities……..……8 Low client visits……………………………………9 Other (specify)………………………..……....…..10 Respondent should check all that apply. 4.14 What is the average reimbursement delay that you have experienced from NHIA (in months)? _________ months 4.15 To what extent would you be willing to borrow funds to improve or expand this facility? Very willing……..………………...………………...1 Somewhat willing…...……………………………...2 Neutral……………………..…………………...….3 Not willing…..…………………………………..…4 Would never consider this.……………………….5 4.16 How interested are you in the idea of selling your facility someday? Very interested..…………………………………...1 Somewhat interested………………………….…...2 Neutral……………………..……………………....3 Not interested……………….……..……………...4 Emphasize to the respondent that we are only looking for her thoughts on this idea, and that her answers Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 56 # Question Answers (and Coding) Instructions Would never consider this.…………………….….5 will not commit her to any course of action whatsoever. 4.17 How interested are you in the idea of leasing out your facility someday? Very interested..…………………………………...1 Somewhat interested………………………….…...2 Neutral……………………..……………………....3 Not interested……………….……..……………...4 Would never consider this.…………………….….5 Emphasize to the respondent that we are only looking for her thoughts on this idea, and that her answers will not commit her to any course of action whatsoever. 4.18 How interested are you in the idea of bringing on an additional business partner someday? Very interested..…………………………………...1 Somewhat interested………………………….…...2 Neutral……………………..……………………....3 Not interested……………….……..……………...4 Would never consider this.…………………….….5 Emphasize to the respondent that we are only looking for her thoughts on this idea, and that her answers will not commit her to any course of action whatsoever. 4.19 How interested are you in the idea of bringing on a manager, or a new manager if your current manager leaves, for the facility someday? Very interested..…………………………………...1 Somewhat interested………………………….…...2 Neutral……………………..……………………....3 Not interested……………….……..……………...4 Would never consider this.…………………….….5 Emphasize to the respondent that we are only looking for her thoughts on this idea, and that her answers will not commit her to any course of action whatsoever. 4.20 How interested are you in the idea of bringing on a new clinical staff member who brings a new set of skills to the practice? Very interested..…………………………………...1 Somewhat interested………………………….…...2 Neutral……………………..……………………....3 Not interested……………….……..……………...4 Would never consider this.…………………….….5 Emphasize to the respondent that we are only looking for her thoughts on this idea, and that her answers will not commit her to any course of action whatsoever. PART 5: FORMALIZATION AND BUSINESS PRACTICES Read: Now I would like to ask you some questions related to different aspects of formal business activities. These questions are only for our information, and it is okay if the facility is not engaging in some of these practices. Please answer honestly, and remember that your answers will remain confidential. # Question Answers (and Coding) Instructions 5.1 Is the facility’s certification of MIDWIFE with the Nurses and Midwives Council up to date? Not registered………………………………...…...1 Registration is expired…………………………….2 Registration is current…………………………….3 5.2 Is the facility’s certification with the Not registered…………………………………......1 HEFRA was previously known as the Private Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 57 # Question Answers (and Coding) Instructions Health Facilities Regulatory Agency (HEFRA) up to date? Registration is expired…………………………….2 Registration is current…………………………….3 Maternity Hospital Board. 5.3 Is the facility registered as a business with the Registrar General Department? Yes…………………………………………………1 No……………………………………………….....2 5.3.1 What is the incorporation model of the business? Sole Proprietor……………..………….…………..1 Limited Liability Company…………….……............2 Partnership………………………………………...3 5.4 Is the facility registered with the Ghana Revenue Authority (GRA)? Not registered…………………………………......1 Registered but don’t pay tax…….…………………2 Registered and pay tax…………..…………………3 5.5 Is the facility currently accredited with NHIA? Accredited…………………………………………1 Not accredited…………………….………………2 Accreditation withdrawn..………..………………..3 If no, go to question 5.6. 5.5.1 What was the score the facility received on its most recent NHIA evaluation? A+ A B C D Provisional Failed 5.6 Does the facility maintain written contracts or agreements with partners and suppliers? No written contracts…………………………...….1 Some written contracts.………….………………..2 Maintains full written contracts……………….........3 5.7 Does the facility have an up-to-date organogram, or a chart that shows who works at the facility and who everyone reports to? Yes…………………………………………………1 No……………………………………………….....2 Yes, but not up-to-date……………………………3 5.8 Is there a plan in place that clearly indicates who runs the facility if the owner/ manager is no longer able to run the facility? Yes…………………………………………………1 No……………………………………………….....2 5.9 Can you name the person who is going to take over? Yes…………………………………………………1 No……………………………………………….....2 Respondent provides the name, though you don’t need to record it. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 58 # Question Answers (and Coding) Instructions 5.10 To what extent would you agree with the following statement: “If the owner were to die or retire, the facility would likely continue to operate.” Strongly agree…………………………..............…..1 Agree………………………...…………...................2 Neutral……………………..……………….......….3 Disagree……………………………………...……4 Strongly disagree………………………...………...5 5.11 Does the facility maintain an annual business plan? Yes…………………………………………………1 No……………………………………………….....2 Is done informally, not documented…………...…..3 5.12 Does the facility maintain a regular budget? Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 5.14. 5.12.1 How often does the facility create a budget? Annual budget……………………………….……..1 Quarterly budget…………………………………..2 Monthly budget………………………………….....3 5.12.2 How often does the facility review the budget (for example, to manage spending and guide decision-making)? Daily…………………….………………...………..1 Weekly……………….……………………...……..2 Monthly……………...………………………...…...3 Quarterly………………………………………….4 Annually……..……..………………………………5 5.13 Does the facility keep personal funds separate from business funds? No separation exists………………………………1 Some separation exists………………………….....2 Full separation……………………………………..3 5.14 Does the facility keep its funds in a bank account? No bank account…..……………………………1 Personal bank account………………………...2 Business bank account.………………………..3 If no, go to question 5.16. 5.15 Does the facility keep an up-to-date daily cash log? Yes…………………………………………………1 No……………………………………………….....2 5.16 Does the facility keep an up-to-date record of all income generated and expenses incurred (income statement)? Yes…………………………………………………1 No……………………………………………….....2 5.17 Does the facility keep an up-to-date record of your assets and liabilities/debts (balance sheet)? Yes…………………………………………………1 No……………………………………………….....2 5.18 To what extent does your business earn more The business makes much more than it spends…....1 The business makes slightly more than it spends......2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 59 # Question Answers (and Coding) Instructions money (revenues) than it spends (costs)? What the business makes and what it spends are about equal………………………………………...3 The business makes slightly lower than it spends....4 The business makes much less than it spends……..5 Don’t know………………………………………89 Refused……………………………………….......99 5.19 What is the total income that your business earns during a typical calendar month after paying all expenses including salaries of employees? That is, what are the profits of your business during a typical calendar month (in cedis)? Provides an amount……………….………..……...1 Don’t know……………………………………....89 Refused……………………………………….......99 If they provide an amount, go to question 5.19.1. If they don’t know, or refuse, go to Part 6. 5.19.1 [Enter amount provided] __________ cedis 4.1.1 PART 6: SERVICES Read: Now I would like to ask you a few additional questions about your facility. # Question Answers (and Coding) Instructions 6.1 In a typical month in the dry season, what is the total number of outpatient visits your facility receives? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know…..….………………………………..89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.1.1. [Enter amount provided] ___________ visits [Answered based on] Records………………………………………...….1 Best guess………………………………….………2 6.1.1 In a typical month in the rainy season, what is the total number of outpatient visits your facility receives? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know…………………………………….....89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.2. [Enter amount provided] ___________ visits Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 60 # Question Answers (and Coding) Instructions [Answered based on] Records………………………………………….....1 Best guess……………………………………….…2 6.2 In a typical month in the dry season, how many visits does your facility receive for maternal, newborn, and child health services? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.2.1. [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.2.1 In a typical month in the rainy season, how many visits does your facility receive for maternal, newborn, and child health services? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.3. [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.3 In a typical month in the dry season, how many deliveries does your facility perform? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.3.1. [Enter amount provided.] ___________ deliveries [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 61 # Question Answers (and Coding) Instructions 6.3.1 In a typical month in the rainy season, how many deliveries does your facility perform? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.4. [Enter amount provided.] ___________ deliveries [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.4 In a typical month in the dry season, how many counseling visits for family planning or reproductive health does your facility provide? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.4.1. [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.4.1 In a typical month in the rainy season, how many counseling visits for family planning or reproductive health does your facility provide? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.5. [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.5 In a typical month in the dry season, how many Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Please ask the respondent to answer based on records if Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 62 # Question Answers (and Coding) Instructions cases of malaria does your facility treat? Don’t know………………………………………89 Refused…………………………………………...99 those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.5.1. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.5.1 In a typical month in the rainy season, how many cases of malaria does your facility treat? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.6. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.6 In a typical month in the dry season, how many cases of diarrhea does your facility treat? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.6.1. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.6.1 In a typical month in the rainy season, how many cases of diarrhea does your facility treat? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 63 # Question Answers (and Coding) Instructions service, don’t know, or refuse, go to question 6.7. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.7 In a typical month in the dry season, how many cases of minor cuts and injuries does your facility treat? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.7.1. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.7.1 In a typical month in the rainy season, how many cases of minor cuts and injuries does your facility treat? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.8. [Enter amount provided.] ___________ cases [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.8 In a typical month in the dry season, how many visits does your facility receive for the administering of vaccinations? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.8.1. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 64 # Question Answers (and Coding) Instructions [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.8.1 In a typical month in the rainy season, how many visits does your facility receive for the administering of vaccinations? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.9. [Enter amount provided.] ___________ visits [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.9 How many beds does your facility have? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.10. [Enter amount provided.] ___________ beds [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.10 In the dry season, how many inpatient days does your facility provide in a typical month? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.11. [Enter amount provided.] ___________ inpatient days Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 65 # Question Answers (and Coding) Instructions [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.10.1 In the rainy season, how many inpatient days does your facility provide in a typical month? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Please ask the respondent to answer based on records if those are easily available. If not, a best guess will be sufficient. If they don’t provide the service, don’t know, or refuse, go to question 6.11. [Enter amount provided.] ___________ inpatient days [Answered based on] Records……………………………….……...…….1 Best guess………………………………….………2 6.11 How many full-time clinical staff are working at this facility? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Clinical staff refers to: doctors, nurses, pharmacists, medical assistants, physiotherapists, etc. Please stress the importance of getting at least the respondent’s best guess. If they don’t provide the service, don’t know, or refuse, go to question 6.12. [Enter amount provided.] ___________ full-time clinical staff 6.12 How many full-time non￾clinical staff are working at this facility? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Non-clinical staff refers to administrators, orderlies, accountants, nutrition officers, etc. Please stress the importance of getting at least the respondent’s best guess. If they don’t provide the service, don’t know, or refuse, go to question 6.13. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 66 # Question Answers (and Coding) Instructions [Enter amount provided.] ___________ full-time non-clinical staff 6.13 How many part-time clinical staff are working at this facility? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Clinical staff refers to: doctors, nurses, pharmacists, medical assistants, physiotherapists, etc. If they don’t provide the service, don’t know, or refuse, go to question 6.14. [Enter amount provided.] ___________ part-time clinical staff 6.14 How many part-time non-clinical staff are working at this facility? Facility does not provide that service..…………….1 Provides an amount…………….……….………....2 Don’t know………………………………………89 Refused…………………………………………...99 Non-clinical staff refers to administrators, orderlies, accountants, nutrition officers, etc. If they don’t provide the service, don’t know, or refuse, go to Part 7. [Enter amount provided.] ___________ part-time non-clinical staff SIGHTING DOCUMENTS This section focuses on the sighting and confirming of specific documents. If possible, this should be done with the assistance of the facility administration staff, while the owner (respondent) is working with GRMA on finishing the QAT, and discussing the outcomes. Say to the owner: “Thank you very much for all the information that you have provided so far. It has been very helpful, and we are very appreciative. At this point, we would like to review some of the documents discussed above. May we work with your administrative staff to view the relevant documents?” # Question Answers (and Coding) Instructions 7.1 May I view your certification of MIDWIFE with the Nurses and Midwives Council? Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 7.2. 7.1.1 [Certification sighted] Yes…………………………………………………1 No……………………………………………….....2 7.1.2 [Certification current] Yes…………………………………………………1 No……………………………………………….....2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 67 # Question Answers (and Coding) Instructions 7.2 May I view the facility’s certification with the Health Facilities Regulatory Agency (HEFRA)? Yes…………………………………………………1 No……………………………………………….....2 If the certification is not current, ask to view the most recent certification. If no, go to question 7.3. 7.2.1 [Certification sighted] Yes…………………………………………………1 No……………………………………………….....2 7.2.2 [Certification current] Yes…………………………………………………1 No……………………………………………….....2 7.3 May I view the facility’s registration document with the Registrar General Department? Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 7.4. 7.3.1 [Registration sighted] Yes…………………………………………………1 No……………………………………………….....2 For a limited liability company, the document is a Certificate of Incorporation and Commencement; For a sole proprietor, the document is a Certificate of Registration 7.3.2 [Registration current] Yes…………………………………………………1 No……………………………………………….....2 7.4 May I view evidence of your tax registration? Yes…………………………………………………1 No……………………………………………….....2 Sufficient documents include GRA Certificate of Registration, tax receipts, or tax clearance certification If no, go to question 7.5. 7.4.1 [Tax document sighted] Yes…………………………………………………1 No……………………………………………….....2 7.4.2 [Tax document current] Yes…………………………………………………1 No……………………………………………….....2 7.5 May I see your NHIA accreditation certificate? Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 7.6. 7.5.1 [Accreditation certificate sighted] Yes…………………………………………………1 No……………………………………………….....2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 68 # Question Answers (and Coding) Instructions 7.5.2 [Accreditation certificate current] Yes…………………………………………………1 No……………………………………………….....2 7.6 May I view an example of a written contract or agreement with a supplier? Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 7.7. 7.6.1 [Document sighted] Yes…………………………………………………1 No……………………………………………….....2 7.7 May see an example of a bank statement? Yes…………………………………………………1 No……………………………………………….....2 If no, conclude the document sighting exercise. 7.7.1 [Document sighted] Yes…………………………………………………1 No……………………………………………….....2 At this point, you are finished sighting documents. Thank the staff member for his/her time and assistance in reviewing these documents, and ask if there is a place where you can wait for the GRMA staff to finish their meeting with the facility owner. (NOTE: You should NOT re-enter the area in which GRMA is meeting with the owner.) DATA FROM QAT After both the survey questionnaire and the QAT sessions are concluded, there is some information to collect from GRMA. Request that your GRMA colleague provide you with the following information. This can be done in the car as you travel to the next place, or elsewhere. But it is important that you record this information accurately. # Question Answers (and Coding) Instructions 8.1 Emergency services (24 hours) Yes…………………………………………………1 No……………………………………………….....2 8.2 Outpatient services Yes…………………………………………………1 No……………………………………………….....2 8.3 In-patient services Yes…………………………………………………1 No……………………………………………….....2 8.4 Maternity services Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 8.5. 8.4.1 ANC Yes…………………………………………………1 No……………………………………………….....2 8.4.2 Delivery Yes…………………………………………………1 No……………………………………………….....2 8.4.3 PNC Yes…………………………………………………1 No……………………………………………….....2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 69 # Question Answers (and Coding) Instructions 8.5 Family planning counseling Yes…………………………………………………1 No……………………………………………….....2 8.6 Family planning methods Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 8.7. 8.6.1 Condoms Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 8.6.2. 8.6.1.1 Female condoms Yes…………………………………………………1 No……………………………………………….....2 8.6.1.2 Male condoms Yes…………………………………………………1 No……………………………………………….....2 8.6.2 Combined Oral Contraceptives Yes…………………………………………………1 No……………………………………………….....2 8.6.3 Combined Injectable Contraceptives Yes…………………………………………………1 No……………………………………………….....2 8.6.4 Projestin-Only Pills Yes…………………………………………………1 No……………………………………………….....2 8.6.5 Projestin-Only Injectable Yes…………………………………………………1 No……………………………………………….....2 8.6.6 Implants Yes…………………………………………………1 No……………………………………………….....2 8.6.7 IUDs Yes…………………………………………………1 No……………………………………………….....2 8.7 Child welfare clinic Yes…………………………………………………1 No……………………………………………….....2 8.8 Under five immunization Yes…………………………………………………1 No……………………………………………….....2 8.9 Pharmaceutical services Yes…………………………………………………1 No……………………………………………….....2 8.10 Breast cancer screening Yes…………………………………………………1 No……………………………………………….....2 8.11 Cervical cancer screening Yes…………………………………………………1 No……………………………………………….....2 8.12 Diagnostic service Yes…………………………………………………1 No……………………………………………….....2 If no, go to question 8.13. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 70 # Question Answers (and Coding) Instructions 8.12.1 Laboratory Yes…………………………………………………1 No……………………………………………….....2 8.12.2 X-ray Yes…………………………………………………1 No……………………………………………….....2 8.12.3 Ultrasound scan Yes…………………………………………………1 No……………………………………………….....2 8.12.4 ECG Yes…………………………………………………1 No……………………………………………….....2 8.12.5 Other diagnostic services [Please specify] 8.13 Catering services Yes…………………………………………………1 No……………………………………………….....2 8.14 Specialist services [Please specify] Score in Emergency Response __________ points Score in Quality Assurance __________ points Score in Safe Motherhood __________ points Score in Advocacy/Health Promotion __________ points Total score __________ points Grade A+ Excellent………………………...….…………..1 A Very Good………………………...….………….2 B+ Good……………………………...……………3 B Fairly Good……………………………...………4 C Fair………………………………………...…….5 D Pass…...…………………………………………6 E Fail……………………………………………….7 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 71 SMH PROJECT PMH EXTENSION SURVEY PART 1: PRE-INTERVIEW DATA The interviewer should complete the following questions before beginning the interview. # Category Answers (and Coding) Instructions 1.1 Interviewer Name 1.2 Date 1 7 Day Month Year 1.3 Region 1.4 District 1.5 Town 1.6 GPS Coordinates Please record the GPS coordinates of the facility. 1.7 Photograph of facility sign Please take a photograph of the facility sign or building. PART 2: INTRODUCTION AND INFORMED CONSENT Immediately following the delivery of the PMH survey, the team will introduction the PMH extension survey as follows:  Description of the purpose of the endline assessment and evaluation  Estimate of the total time required – 30 minutes Once the respondent has completed the PMH survey, the enumerator should read the following to the respondent: We would like to build up on the information you just provided during the completion of the PMH survey with a few additional questions. These questions will be used to further the project’s understanding of the changes your facility has made over the past three years in relation to business continuity, financial sustainability and quality of clinical services. This information will contribute to the project’s endline assessment and performance evaluation. Participation in the survey is voluntary and you are free to decline to answer any or all questions. The results will be kept confidential and will only be used to help the project do better work in this area. [Respondents will be limited to persons able to give personal consent. Subjects not able to give personal consent (e.g. for cultural reasons, children or adolescents less than the legal age of consent, subjects with mental illness) should not be subjects of this study, and will not be interviewed. If the respondent is incapable of reading and Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 72 signing the written consent form, the enumerators will read the form to them in order to acquire informed consent.] This survey usually takes 30 minutes to complete. Will you participate in this survey? Yes………………………………………………………………………….…..1 No……………………………………………………………………………....2 PART 3: FACILITY INFORMATION The interviewer should read aloud: I’d like to start by learning a little bit about your facility. # Question Answers (and Coding) Instructions 3.1 What is your name? 3.2 Respondent sex Male……………………………………………..…1 Female………………………………..……..............2 3.3 What is your position or title? If the respondent is not the owner, manager, or another person with significant responsibility, confer with GRMA about the possibility of postponing the interview. PART 4: FINANCIAL SUSTAINABILITY, OPERATIONAL EFFICIENCY AND QUALITY OF CLINICAL SERVICES Read aloud: Now I would like to ask some questions about the ownership and management of the facility. # Question Answers (and Coding) Instructions 4.1 To what extent do you agree with the following statement: “I have a better awareness of the benefits of business continuity and financial sustainability than you did three years ago”? Strongly agree…………………………..............…..1 Agree………………………...…………...................2 Neutral……………………..……………….......….3 Disagree……………………………………...……4 Strongly disagree………………………...………...5 4.2 Have you change the way in which you have managed or conducted your facility’s business and financial practices in the last three years? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.2.1. If no, skip to question 4.3. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 73 # Question Answers (and Coding) Instructions 4.2.1 If so, how? 4.3 Did you or a representative of the facility attend the quality circles? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.3.1. If no, skip to question 4.4. 4.3.1 If so, how often? Monthly…………………...…………..............…….1 Bi-Monthly…………………………………....……2 Quarterly……………….…...…………………..…3 Annually………………………...……….…..............4 Less often…….………………..…………………...5 4.4 Did you or a representative of the facility attend the regional meetings? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.4.1. If no, skip to question 4.5. 4.4.1 If so, how often? Monthly…………………...…………..............…….1 Bi-Monthly…………………………………....……2 Quarterly……………….…...…………………..…3 Annually………………………...……….…..............4 Less often…….………………..…………………...5 4.5 Did you receive SMS messages from the SMH project? Yes………………………………………….............1 No…………………………………………..............2 4.6 In the two years leading up to the SMH project, did you seek or hire outside counsel to improve your business and financial practices? Yes………………………………………….............1 No…………………………………………..............2 4.7 Since participating in the SMH project, have you sought or hired outside counsel to improve your business and financial practices? Yes………………………………………….............1 No…………………………………………..............2 4.8 In the two years leading up to the SMH project, did you receive training Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.8.1. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 74 # Question Answers (and Coding) Instructions in business continuity and/or financial sustainability outside of the SMH project? If no, skip to question 4.9. 4.8.1 If yes, from where? 4.9 Since participating in the SMH project, have you received training in business continuity and/or financial sustainability outside of the SMH project? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.19.1. If no, skip to question 4.10. 4.9.1 If yes, from where? 4.10 In the two years leading up to the SMH project, did you apply for a bank loan? If so, when? Yes………………………………………….............1 No…………………………………………..............2 Application Date_________________________ If yes, proceed to question 4.10.1. If no, skip to question 4.11. 4.10.1 If yes, were you successful in obtaining the loan? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.10.2. If no, skip to question 4.11. 4.10.2 If yes, when was the loan approved? 4.10.3 What was the value of the approved loan? 4.10.4 What was the purpose of the loan? 4.11 Have you in the past three years applied for a bank loan? If so, when? Yes………………………………………….............1 No…………………………………………..............2 Application Date_________________________ If yes, proceed to question 4.11.1. If no, skip to question 4.12. 4.11.1 If yes, were you successful in obtaining the loan? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.11.2. If no, skip to question 4.12. 4.11.2 If yes, when was the loan approved? Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 75 # Question Answers (and Coding) Instructions 4.11.3 What was the value of the approved loan? 4.11.4 What was the purpose of the loan? 4.12 In the two years leading up to the SMH project, did you invest personal funds exceeding 4,000 GHS into your facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 4.12.1. If no, skip to question 4.13. 4.12.1 If so, what was the investment used for? 4.13 In the two years leading up to the SMH project, how motivated were you to expand your facility. Strongly motivated….…………………..............…..1 Somewhat motivated………...…………...................2 Neutral……………………..……………….......….3 Not very motivated...………………………...……4 Not motivated…………………………...………...5 4.14 Has the owner/manager of the PMH experienced a decline in their health over the past three years? Yes………………………………………….............1 No…………………………………………..............2 4.15 Is the current owner/manager of the PMH in good health? Yes………………………………………….............1 No…………………………………………..............2 4.16 Has the facility’s owner/manager passed away within the past three years? Yes………………………………………….............1 No…………………………………………..............2 5.1 Have you in the last three years brought on a new manager? Yes………………………………………….............1 No…………………………………………..............2 If yes, continue to 5.1.1 If no, skip to 5.2 5.1.1 If yes, can we see the contract/employment agreement outlining the recently completed transformation? Yes………………………………………….............1 No…………………………………………..............2 5.1.2 [Contract sighted] Yes…………………………………………………1 No……………………………………………….....2 5.1.3 Is it signed by both parties? Yes………………………………………….............1 No…………………………………………..............2 5.1.4 When was it signed? Date__________________________________ Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 76 # Question Answers (and Coding) Instructions 5.2 Have you in the last three years hired an additional clinical staff member to diversify the services offered at your facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, continue to 5.2.1 If no, skip to 5.3 5.2.1 If yes, can we see the contract/employment agreement outlining the recently completed transformation? Yes………………………………………….............1 No…………………………………………..............2 5.2.2 [Contract sighted] Yes…………………………………………………1 No……………………………………………….....2 5.2.3 Is it signed by both parties? Yes………………………………………….............1 No…………………………………………..............2 5.2.4 When was it signed? Date__________________________________ 5.3 Have you in the last three years sold or purchased your facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, continue to 5.3.1 If no, skip to 5.4 5.3.1 If yes, can we see the contract outlining the recently completed transformation? Yes………………………………………….............1 No…………………………………………..............2 5.3.2 [Contract sighted] Yes…………………………………………………1 No……………………………………………….....2 5.3.3 Is it signed by both parties? Yes………………………………………….............1 No…………………………………………..............2 5.3.4 When was it signed? Date__________________________________ 5.4 Have you in the last three years leased out or leased your facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, continue to 5.4.1 If no, skip to 5.5 5.4.1 If yes, can we see the contract outlining the recently completed transformation? Yes………………………………………….............1 No…………………………………………..............2 5.4.2 [Contract sighted] Yes…………………………………………………1 No……………………………………………….....2 5.4.3 Is it signed by both parties? Yes………………………………………….............1 No…………………………………………..............2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 77 # Question Answers (and Coding) Instructions 5.4.4 When was it signed? Date__________________________________ 5.5 Have you in the last three years brought on a new partner to share in the ownership of your facility? Yes………………………………………….............1 No…………………………………………..............2 If yes, continue to 5.5.1 If no, skip to 6.1. 5.5.1 If yes, can we see the contract outlining the recently completed transformation? Yes………………………………………….............1 No…………………………………………..............2 5.5.2 [Contract sighted] Yes…………………………………………………1 No……………………………………………….....2 5.5.3 Is it signed by both parties? Yes………………………………………….............1 No…………………………………………..............2 5.5.4 When was it signed? Date__________________________________ 6.1 Have you contacted any financial institutions to learn about available loan product? Yes………………………………………….............1 No…………………………………………..............2 6.1.1 If so, when? Before engagement with the SMH project…….….1 After engagement with the SMH project…………2 6.2 In the two years leading up to the SMH project, did you receive clinical support or technical assistance? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 6.2.1. If no, skip to question 6.3. 6.2.1 If yes, from where? 6.3 Have you in the past three years received clinical support or technical assistance outside of the SMH project? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 6.3.1. If no, skip to question 6.4. 6.3.1 If yes, from where? 6.4 In the two years leading up to the SMH project, were you recruiting for additional full-time clinical staff? Yes………………………………………….............1 No…………………………………………..............2 If yes, proceed to question 6.4.1. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 78 # Question Answers (and Coding) Instructions If no, skip to question 6.5. 6.4.1 If so, please explain. 6.5 In the two years leading up to the SMH project, were you recruiting for additional full-time non￾clinical staff? Yes………………………………………….............1 No…………………………………………..............2 6.5.1 If so, please explain Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 79 GHANA REGISTERED MIDWIVES ASSOCIATION QUALITY ASSESSMENT TOOL Ghana Saving Private Maternity Homes Project GRMA 8/1/2015 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 80 PREAMBLE PURPOSE This instrument reflects facilities and equipment available and the skills/activities to be performed by service providers at all levels (within the maternity home) as described in the national Safe Motherhood (SM) Protocols Document. The use of this questionnaire helps to (among other reasons) perform quality assessment and support supervision for multiple purposes including:  Determine the baseline for the SMH project  Identify skills and service gaps of the PMH / comprehensive needs assessment  Quality of services provided and continuous performance improvement WHO ARE THE POTENTIAL USERS? The checklist should be used by qualified supervisors and trained enumerators with required technical and clinical skills in SM areas. Managers at various levels should utilized data collected to bring improvement into their facilities. WHAT ARE THE KEY STEPS Trainers/supervisors and enumerators team planning  What are the team objectives for the visit?  Who will visit whom, when and with what?  What are the expected output/outcomes of the visits?  Are tools /checklist ready and are enumerators trained?  Is Regional Chairperson of GRMA and person to be visited aware of date and purpose of visit? VISIT IMPLEMENTATION  Courtesy visit to person in charge of facility (purpose of visit, how the visit will be implemented, who are the persons to be involved, end of visit discussions, etc.)  Proceed with concerned person (explain purpose of visit and that you are not for fault finding, invite to open/honest discussion, discuss the process, proceed, give feedback). VISIT WRAP UP  Discuss findings both positive and those needing improvement from the visit.  Discuss next actions to be taken. TEAM EVALUATION MEETING  What happened?  What are the major outcomes?  What should be the next steps (next visit objectives)  Document findings of the visit.  What are the lessons learnt?  What are the recommendations? Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 81 FACILITY INFORMATION Name of Facility Location Address District Region Postal Address Telephone (Landline, if any) Mobile Phone Number (s) E-mail Website Facility Licensed by License Number Date of Expiry Name of Head of Facility Contact number (s) of Head E-mail of Head/Website Name of Owner of Facility Contact Number of Owner Gender of Owner E-mail of Owner (if different from facility head) Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 82 RANGE OF SERVICES Please tick services provided in the facility SERVICE DETAILS/REMARKS 1. Emergency services (24 hours) YES ( ) NO ( ) 2. Out-patient services YES ( ) NO ( ) 3. In-patient services YES ( ) NO ( ) 4. Maternity services (Tick details) YES ( ) NO ( ) ANC Delivery PNC 5. Family planning counselling YES ( ) NO ( ) 6. Provision of Family planning methods YES ( ) NO ( ) Condoms (tick details) YES ( ) NO ( ) Female Male Combined Oral Contraceptives YES ( ) NO ( ) Combined Injectable Contraceptives YES ( ) NO ( ) Projestin-Only pills YES ( ) NO ( ) Projestin-Only Injectable YES ( ) NO ( ) Implants YES ( ) NO ( ) IUDs YES ( ) NO ( ) 7. Child welfare Clinic YES ( ) NO ( ) 8. Under Five immunization YES ( ) NO ( ) 9. Pharmaceutical services YES ( ) NO ( ) 10. Breast cancer screening YES ( ) NO ( ) 11. Cervical Cancer Screening YES ( ) NO ( ) 12. Diagnostic service YES ( ) NO ( ) Laboratory YES ( ) NO ( ) X-ray YES ( ) NO ( ) Ultrasound scan YES ( ) NO ( ) ECG YES ( ) NO ( ) Other diagnostic services (specify) YES ( ) NO ( ) ………………………………… 13. Catering services YES ( ) NO ( ) 14. Specialist services (specify below) YES ( ) NO ( ) Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 83 GENERAL INFORMATION Year maternity home was opened: …………………………..………………………………………...…… Qualification of Head /Manager: A. Nursing: Certificate No: …………………………………………………………………………..………….. Date of qualification: …………………………….…………………………………….…………… Name /place of training school: ………………………………………………….………………… B. Midwifery: Certificate No: ……………………………………………....…………………..………………….. Date of qualification: ……………………………………………………………..………………… Name/place of training school: ……………………………………………………..………………. Any other Date of qualification: …………………………………………………………………… C. Any other qualification: (professional) ………………………………………….……………………… D. GRMA Registration of midwife who owns/heads the facility………….……… Date: ………………... E. GRMA Registration of all midwives who work in facility i. …………………………… Date: …………………. ii. ……………………………. Date: …………….…… iii. …………………………..... Date: …………………. iv. ……………………………. Date: ………….……… v. …………………………….. Date: …………………. vi. ……………………………. Date: ………...……….. PURPOSE OF VISIT Follow up: …………………………………………………………………………………………………… Needs assessment: ……………………………………………………………..…………………………… Quality care: ………………………..………………………..…………………………..…………………... Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 84 Performance: ……………………………………………………………………………...………………… Team = Specific Objectives: ………………………………………………………………………………… Ask /discuss any success: concerns/problems and address them as appropriate during the visit How is your work progressing? What successes have you seen? What concerns, or problems do you have? Is there anything else you would like to ask or tell me? EMERGENCY RESPONSE (54 MARKS) 24 hrs Services 1. a. Do you have a joint duty roster? (Check for the roster for the past three months) Yes ( ) No ( ) 2 marks b. Is co-ordination of duty roster done by MW/Facility head? (Check for evidence) Yes ( ) No ( ) 3 marks 2. Has emergency team been trained for the past year? (Check for evidence) Yes ( ) No ( ) 5 marks 3. Have you made arrangement with any hospital in case of referral? (Check for evidence) Yes ( ) No ( ) 5 marks 4. Does the PMH have links with a hospital in any of the following? - Clinical meeting Yes ( ) No ( ) 5 marks Check for evidence - Training Yes ( ) No ( ) 5 marks (Check for evidence) Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 85 5. Are the following protocols available and in use /displayed? (Select a midwife to interview and give full marks if displayed and correctly demonstrated by midwife) a. Eclampsia Yes ( ) No ( ) 2 marks b. PPH Yes ( ) No ( ) 2 marks c. APH Yes ( ) No ( ) 2 marks d. Diarrhoea Yes ( ) No ( ) 2 marks e. Convulsion Yes ( ) No ( ) 2 marks f. Anemia Yes ( ) No ( ) 2 marks g. URTI Yes ( ) No ( ) 2 marks h. Malaria Yes ( ) No ( ) 2 marks i. Shock Yes ( ) No ( ) 2 marks 6. Tracer Drugs availability (95 – 100 % availability) 2 marks 7. Do you have Resuscitation sets and functioning? - Sucker Yes ( ) No ( ) 3 marks - Oxygen Yes ( ) No ( ) 3 marks - Ambubag Yes ( ) No ( ) 3 marks (Check for evidence) QUALITY ASSURANCE (83 MARKS) 1. Do you have a functional Quality Assurance Team Yes ( ) No ( ) 3 marks (Check for minutes and copies of latest report) 2. Have you carried out the following surveys? a. Patient Satisfaction Yes ( ) No ( ) 3 marks b. Rational Use of Medicines Yes ( ) No ( ) 3 marks 3. Do you have Action Plans to address QA problems identified (Check for evidence) Yes ( ) No ( ) 3 marks 4. Is there privacy, confidentiality in place for patients Yes ( ) No ( ) 3 marks (Observe consulting room/ward for evidence i.e. screen) INFECTION PREVENTION 5. Infection Prevention Practices (observe the following) a. Neat environment (Landscaping) Yes ( ) No ( ) 2 marks b. Puncture proof container waste disposal (Check)Yes ( ) No ( ) 2 marks c. Waste bin lined with polythene bag(check) Yes ( ) No ( ) 2 marks d. Waste bins boldly labelled (check) Yes ( ) No ( ) 2 marks Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 86 e. Segregation of waste carried out (observe) Yes ( ) No ( ) 2 marks f. Site for Disposing Placenta (Observe) Yes ( ) No ( ) 2 marks g. Veronica Bucket with water/running water Yes ( ) No ( ) 2 marks 6. Is the following done at this level? a. Sterilization of articles (instrument/material) Yes ( ) No ( ) 3 marks b. Correct dilution of chorine (strength) Yes ( ) No ( ) 3 marks 7. Do you use partographs to monitor labour ` Yes ( ) No ( ) 4 marks (Check used partographs) 8. a. Do you have delivery sets? Yes ( ) No ( ) 1 marks (cord scissors, cord clamps) episiotomy scissors, sutures, speculum, kidney dishes) b. If or equals 2 set or more are sited. (check) Yes ( ) No ( ) 2 marks c. Is it sterile and packed Yes ( ) No ( ) 1 mark 9. Do you have a complaint desk or suggestion box Yes ( ) No ( ) 2 marks Check for evidence 10. Display of services available Yes ( ) No ( ) 2 marks a. Directional signs (Observe) Yes ( ) No ( ) 2 marks b. Prenatal / Perinatal NEONATAL SURVIVAL 11. Do you have all your midwives trained in neonatal resuscitation? (Select a midwife to demonstrate) Yes ( ) No ( ) 3 marks 12. Do you have equipment for resuscitation (air way or suction machine)? Check for evidence. Yes ( ) No ( ) 3 marks 13. Do you undertake PNC for mothers/caregivers of children to educate them on the following? 14. (Check register / note book for the following)  Bed net usage Yes ( ) No ( ) 2 marks  Education on hygiene and sanitation Yes ( ) No ( ) 2 marks  Family Planning Yes ( ) No ( ) 2 marks INFANT SURVIVAL 15. Has your facility undertaken any nutrition training program? (Check for evidence) Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 87 Yes ( ) No ( ) 2 marks 16. Are there any functional Mother support groups? (Check for evidence) Yes ( ) No ( ) 2 marks 17. Do you provide education on Infant feeding practices Check for evidence Yes ( ) No ( ) 2 marks 18. Is there a functional ORT corner in this facility? (Check for evidence) Yes ( ) No ( ) 2 marks UNDER FIVE SURVIVALS 19. a. Are health workers trained in malaria case management? Yes ( ) No ( ) 2 marks b. Are prescribers using ACT? Yes ( ) No ( ) 2 marks c. Does your facility confirm suspected malaria cases? (Microscopy/RDT)? (Evidence from OPD Cards or folder) Yes ( ) No ( ) 1 mark 20. a. Do you have the STG Document? Yes ( ) No ( ) 2 marks b. Are prescribers using the Standard Treatment Guideline (STG?) (Check for evidence) Yes ( ) No ( ) 2 marks 21. Are there laminated protocols of top 5 diseases? (Malaria, ART, diarrhoea, anaemia, malnutrition) Yes ( ) No ( ) 5 marks SAFE MOTHERHOOD (25 MARKS) ANTENATAL CARE 1. Is this facility practicing focus antenatal? Yes ( ) No ( ) 2 marks (Observe) 2. Maternal health records properly filled (BP recordings, comment on presence or absence of pedal oedema, temperature recordings, weight, urine test results especially for sugar and protein, hemoglobin measurements, malaria parasites) Yes ( ) No ( ) 2 marks Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 88 MATERNITY SERVICES 3. Diagnostic tools for mothers available at each location i.e. consulting room, labour ward (Thermometers, sphygmomanometers, stethoscope, adult weighing scale, tape measure, spatula) Check for evidence Yes ( ) No ( ) 2 marks 4. Diagnostic tools for infants available (Tape measure, baby weighing scale, foetal stethoscope/sonicaid) Check for evidence Yes ( ) No ( ) 2 marks 5. Basic equipment available (Wheel chair(s), patient trolley(s), drip stands, wall clock, delivery bed,) Check for evidence Yes ( ) No ( ) 2 marks 6. Emergency medicines available (Oxytocin, cold chain for oxytocin, magnesium sulphate, ergometrine) Check for evidence Yes ( ) No ( ) 2 marks 7. Emergency packs for PPH (gloves, syntocinon, IV Fluids, wide bore cannula, elbow length glove, giving set, urinary catheter, urine bags) Yes ( ) No ( ) 2 marks 8. Emergency Pack for eclampsia (magnesium sulphate, IV fluids, gloves, Infusion, Delivery tray, General Infusion Tray. Yes ( ) No ( ) 2 marks OBSTETRIC CARE 9. Are SM protocols laminated and displayed Yes ( ) No ( ) 1 mark 10. Is a referral system in place? (Check for referral book) Yes ( ) No ( ) 1 mark 11. Community transport system pre-arranged? (Check ) Yes ( ) No ( ) 2 marks POST NATAL 12. Postnatal process correctly followed and documented Yes ( ) No ( ) 2 marks 1. Mother: (BP, examination of perineum, weight of mother, Lactation, family planning counselling) 2. Baby: ( temperature of baby, weight of baby, , breast feeding, Observe / check for evidence. FAMILY PLANNING Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 89 1. Is counselling protocol available and in use? Yes ( ) No ( ) 1 mark 2. Availability of stock out of FP logistics for the past 3 months (Check for Family Planning Reports) Yes ( ) No ( ) 2 marks ADVOCACY/HEALTH PROMOTION (8 MARKS) Advocacy/Health Promotion (Check for reports/notebook) 1. Is Health Education plan implemented in the following areas? - HIV/AIDS / STI Yes ( ) No ( ) I mark - Adolescent Health Yes ( ) No ( ) I mark - Regenerative Health & Nutrition Yes ( ) No ( ) I mark - Hypertension Yes ( ) No ( ) I mark - Diabetes Yes ( ) No ( ) I mark - Malaria Yes ( ) No ( ) I mark - Child Health Yes ( ) No ( ) I mark - Reproductive Health Yes ( ) No ( ) I mark Any other comment Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 90 SCORING No. ITEM EXPECTED SCORE FACILITY SCORE 1 Emergency Response 54 marks 2 Quality Assurance 83 marks 3 Safe Motherhood 25 marks 4 Advocacy/Health Promotion 8 marks Total 170marks GRADING No Score Range Grade 1 160 – 170 A+ Excellent 2 145 – 159 A Very good 3 130 – 144 B+ Good 4 115 – 129 B Fairly Good 5 100 – 114 C Fair 6 85 – 99 D Pass 7 Below 85 E Fail DATE: ……………………………………………………………………………………………………….. SUBDISTRICT: ……………………………………………………………………………...….…………… RESPONDENTS:……………………………………………………….……………………..……………... NAME OF ASSESSOR:………………………………………………………...…………...……………..….. TIME STARTED: ……………………….………………. TIME ENDED:……..……….…………………… NAME OF REVIEWER OF ASSESSOR……………...…………………………………….………………… SIGNATURE: ………………………………………………………………………………………………... Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 91 GHANA REGISTERED MIDWIVES ASSOCIATION (GRMA) 1st Circular Road, H/No. 11B, Cantonments, Accra, Tel: + 233-(0) 302-779348/772971, email￾ghanamidwives@yahoo.com PEER SUPPORT FEEDBACK FORM No. ITEM EXPECTED SCORE FACILITY SCORE 1 Emergency Response 54 marks Strengths Gaps 2 Quality Assurance 83 marks Strengths Gaps 3 Safe Motherhood 25 marks Strengths Gaps 4 Advocacy/Health Promotion 8 marks Strengths Gaps Total 170marks GRADING No Score range (No. 1 – 4) Grade 1 160 - 170 A+ Excellent 2 145 - 159 A Very good 3 130 - 144 B+ Good 4 115 - 129 B Fairly Good 5 100 - 114 C Fair 6 85 - 99 D Pass 7 Below 85 E Fail No Score range (No. 5) Grade 1 4 A Excellent 2 3 B Good 3 2 B Fairly Good 4 1 C Fair 5 0 E Fail Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 92 DATE: ……………………………………………………………………………………………………….. SUBDISTRICT: ……………………………………………………………………………………………… RESPONDENTS: ……………………………………………………….…………………………………... NAME OF ASSESSOR: …………………………………………………………………………………..….. TIME STARTED: ……………………….…………. TIME ENDED: ……..……………………………… Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 93 SAVING MATERNITY HOMES IN GHANA KEY INFORMANT INTERVIEW DISCUSSION GUIDE Starting the Session Interviewer provides a brief introduction of the SMH project, and explains the objectives of the project and the endline assessment as well as the interview / discussion points. Interviewer thanks the respondent for his/her time, and stresses that his/her contribution is valuable and appreciated. Participation in this interview is voluntary and you are free to decline to answer any or all questions. [Respondents will be limited to persons able to give personal consent. Subjects not able to give personal consent (e.g. for cultural reasons, children or adolescents less than the legal age of consent, subjects with mental illness) should not be subjects of this study, and will not be interviewed. If the respondent is incapable of reading and signing the written consent form, the enumerators will read the form to them in order to acquire informed consent.] This survey usually takes 60 minutes to complete. Will you participate in this interview? Respondent information Name (first name and surname) Institution Title Date of interview Telephone number Email address Permission to contact with follow-up questions? Background Questions 1 Does your organization interact and/or support private maternity homes in Ghana? If so, how? 2 If your organization interacts and/or supports private maternity homes, has your interactions or support changed in recent years? (i.e. reporting procedures, registration processes, targeted technical assistance, etc.) If so, how? General Questions 3 In your opinion, what is the state of the private maternity homes in Ghana at the moment? Opportunities? Challenges? 4 In your opinion, have you seen a change in the state of private maternity homes in the past three years? If so, in what ways? 5 The SMH project has continued to observe the closure of facilities in rural areas. What are the causes of these closures? Interview Questions 6 The project observed a decline in PMH registration with the relevant regulatory bodies as compared to baseline, specifically with NHIA and HEFRA. In your opinion, what might account for this decline? 7 Over the past three years, the project observed a decline in the average number of full-time employees within maternity homes even when things like profit and patient visits went up. In your opinion, why might have expanded PMHs not increased the number of full-time employees to support the growth of their facility? Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 94 8 The project found that, over the last three years, many PMHs are less optimistic about the future of their facility, and less interested in the idea of expanding. What might account for this change? 9 Over the past three years, the project observed a significant decline in PMH owners’ willingness to borrow funds to support the expansion and growth of a maternity home. What might account for this change? 10 What is the current level of competition among health care facilities and private maternity homes in rural, underserved areas in Ghana? In your opinion, do you feel that the level of competition has increased over the past three years? If the level of competition has increased, what would account for this? 11 What is the role of the NHIS in the financial sustainability of private maternity homes? How, if at all, can facilities apply preparations for NHIA to other aspects of their operations? How has or will the cancellation of capitation affect PMHs? End of Questions Thank for your time and contributions again. Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 95 SMH PROJECT EVALUATION IN-DEPTH INTERVIEW GUIDE Discussion Guide – Key Informants Starting the Session Interviewer provides a brief introduction of the SMH project, and explains the objectives of the project as well as the interview / discussion points. Interviewer thanks the respondent for his/her time, and stresses that his/her contribution is valuable and appreciated. Participation in the survey is voluntary and you are free to decline to answer any or all questions. [Respondents will be limited to persons able to give personal consent. Subjects not able to give personal consent (e.g. for cultural reasons, children or adolescents less than the legal age of consent, subjects with mental illness) should not be subjects of this study, and will not be interviewed. If the respondent is incapable of reading and signing the written consent form, the enumerators will read the form to them in order to acquire informed consent.] This survey usually takes 60 minutes to complete. Will you participate in this interview? Respondent information Name (first name and surname) Facility Title Date of interview Telephone number Permission to contact with follow-up questions? Assessment questions Response Ownership/ Management Transformations 1 Do you feel that you were able to meet the demands and needs of your existing clients during the two years prior to receiving a PSV and/or participating in the SMH trainings, taking into consideration your staffing, facility space and available services? Yes………………………………………….............1 No…………………………………………..............2 1a Please explain. 2 Did you experience an increase in client demand in the two years prior to your engagement with the SMH project? Yes………………………………………….............1 No…………………………………………..............2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 96 2a Please explain. Financial sustainability, operational efficiency and quality of clinical services 3 If you hired a new manager or staff member with a diverse skillset, did the facility qualify for accreditation or certification since the hiring of the new staff member which is was not previously qualified for? Yes………………………………………….............1 No…………………………………………..............2 3a If so, which one? 3b Were you successful in achieving this certification or accreditation? Yes………………………………………….............1 No…………………………………………..............2 4 If you hired a new manager or staff member with a diverse skillset, does the new staff members’ responsibilities include record keeping and/or relate to supporting your facility’s business management and financial practices? Yes………………………………………….............1 No…………………………………………..............2 4a Does the new staff member have experience or qualifications in business or financial management? Yes………………………………………….............1 No…………………………………………..............2 5 Did you have a pre-existing awareness of when capitation was anticipated to arrive in your region prior to the SMH project? Yes………………………………………….............1 No…………………………………………..............2 6 Did you believe capitation would arrive? Yes………………………………………….............1 No…………………………………………..............2 6a If so, when? Very soon……..……………………………............1 Soon….……………………………………..............2 Not soon……………………………………...........3 Not very soon………………………………...........4 Unsure...…………………………………….............5 7 In the two years leading up to the SMH project, did you participate in continuing professional education training and/or Yes………………………………………….............1 No…………………………………………..............2 Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 97 events to improve the quality of care provided at your facility? 7a If so, where? 7b Have you in the last three years participated in continuing professional education training and/or events to improve the quality of care provided at your facility? Yes………………………………………….............1 No…………………………………………..............2 7c If so, where? Expansion of Clinical services 8 If you hired a new manager or staff member with a diverse skillset, does the new staff members’ responsibilities contribute to the expansion of the PMHs available clinical services? Yes………………………………………….............1 No…………………………………………..............2 8a Please explain. 9 If you hired a new manager or staff member with a diverse skillset, does the new staff member have experience or qualifications which were not previously covered by the existing staff at your facility? Yes………………………………………….............1 No…………………………………………..............2 9a Please explain. Proportional Piling Activity Saving Maternity Homes in Ghana, Innovate for Health, Internal End of Project Evaluation 98 At this point in the interview, the interviewer will introduce the proportional piling activity, a participatory research method, used to determine the relative attribution of the project’s interventions as compared to the competing explanations for the outcomes observed. Proportional piling allowed respondents from transformed facilities to rank the causal factors that were present, determining the levels of attribution across the SMH project interventions and the competing explanations. The interviewer will present each competing explanation for the observed outcomes of interest, and provided a set of 100 stones to the interviewee. Interviewees will then be prompted to allocate the relative number of stones per causal factor to identify the proportional significance of that causal factor in contributing to the observed outcome of interest. Following the allocation of the stones, the interviewer will record the number of stones allocated to each causal factor and ask some probing questions to further understand why the facility owner ranked the causal factors in the order that they chose. EQ No. 2 Please attribute the relative number of stones, demonstrating significance, across the various causal factors present that have contributed to the successful completion of your facility’s ownership/management transformations. Record answer on score card. EQ No. 3a Please attribute the relative number of stones, demonstrating significance, across the various causal factors present that have contributed to the improvements in the financial sustainability of your facility. Record answer on score card. EQ No. 3b Please attribute the relative number of stones, demonstrating significance, across the various causal factors present that have contributed to the improvements in the quality of clinical services of your facility. Record answer on score card. EQ No. 4 Please attribute the relative number of stones, demonstrating significance, across the various causal factors present that have supported the expansion of your facility. Record answer on score card. US Agency for International Development | Ghana Saving Maternity Homes in Ghana Banyan Global, Implementing Partner P.O Box GP 147 Accra, Ghana Tel: 0302-779348/772971 www.banyanglobal.com