2 [Title] TAKING KNOWLEDGE FOR HEALTH THE EXTRA MILE AN EVALUATION OF THE MALAWI KNOWLEDGE FOR HEALTH PROJECT (K4HEALTH) BY THOKOZANI BEMA NATALIE CAMPBELL BRIAN JUMBE CARY PERRY NOVEMBER 30, 2011 3 ACKNOWLEDGEMENTS Many people in MSH and in Malawi contributed their time to this study. Nancy Vollmer LeMay, as the Monitoring and Evaluation Advisor for the K4Health Project, helped to design the evaluation and data collection tools; Liz McLean, K4Health Director at MSH for providing guidance and reviews on the report; and Mariah Boyd-Boffa, Communications Senior Program Associate for editing and formatting. K4Health Malawi intern Meria Million traveled to the farthest reaches of Malawi to assist the evaluation team in interviewing respondents for the LQAS survey and gathering and transcribing notes for the Focus Group Discussions. Many thanks also to the Salima and Nkhotakota District Health Officers, and several data collectors: Dimitra Kaitana, Chifundo Mollen, Tinkhani Galazi, Gregory Chilongo, Felton Ziyaya, Emmanuel Kamanga, Ester Nyangulu, and Moffat Madzulo. In addition we would like to thank our partners at JHU/CCP/K4Health, Piers Bocock and Tara Sullivan, for reviewing drafts and providing input on the report. Also we thank everyone from the Malawi Ministry of Health and international donor community who so generously shared their experiences and ideas with us during the evaluation, we extend our thanks: Madeleine Short, Peggy D’Adamo, and Beth Deutsch. And finally, thanks to those in the Knowledge Management Task Force at national and district levels who participated in the evaluation. 4 TABLE OF CONTENTS ACKNOWLEDGEMENTS 2 ABBREVIATIONS 4 1. EXECUTIVE SUMMARY 5 2. INTRODUCTION 11 2.1 The Knowledge for Health Project (K4Health) 11 3. EVALUATION QUESTIONS 12 4. METHODOLOGY 13 4.1 Information Sources 13 4.2 Net-Mapping at National and District Levels 14 4.3 LQAS Survey of Community Health Workers 16 4.4 Review of mobile statistics from the SMS server (Hub) for SMS users 22 4.5 District Learning Center Registries 22 4.6 Malawi Toolkit Web Statistics 22 5. FINDINGS AT THE NATIONAL LEVEL 23 5.1 Program Implementation 23 5.2 Results of the Leadership Development Program 24 5.3 Toolkits 25 5.4 Sustainability of the Toolkits 27 5.5 Findings from Net-Maps at the National Level 29 5.6 Impact of K4Health Interventions at the National Level and Suggestions for Future Improvements 32 6. FINDINGS AT THE DISTRICT LEVEL 35 6.1 Net mapping at the district level 35 6.2 Program Implementation of the District Learning Centers 36 6.3 District Learning Center Results 37 6.4 Sustainability of District Learning Centers 42 6.5 Impact of K4Health Interventions at the District Level and Suggestions for Future Improvements 43 7. FINDINGS AT THE COMMUNITY LEVEL 44 7.1 Program Implementation 44 7.2 Results of K4Health Interventions at the Community Level 47 7.3 Results for SMS Users by Supervision Area 51 7.4 Impact of K4Health Interventions at the Community Level and Suggestions for Future Improvements 59 8. CONCLUSIONS 60 9. RECOMMENDATIONS 64 APPENDICES 67 Appendix A. List of Tables and Figures 67 Appendix B. Net-Mapping Guide 68 Appendix C. Focus Group Discussion Guide for DLCs 75 Appendix D. Focus Group Discussion Guide for HSAs and CBDAs 77 Appendix E. Key Informant Interview Guide for District Coordinators 80 Appendix F. LQAS Questionnaire for SMS Users 83 Appendix G. LQAS Questionnaire for Control District 88 5 ABBREVIATIONS AIDS Acquired Immune Deficiency Syndrome CBDA Community Based Distribution Agent CBO Community Based Organization CHAM Christian Health Association of Malawi CHW Community Health Worker DHMT District Health Management Team DHO District Health Office DLC District Learning Center DMPA Depo-Provera FBO Faith Based Organization FGD Focus Group Discussion FHI360 Family Health International FP Family Planning FLSMS Frontline SMS HBC Home Based Care HIMS Health Information Management System HIV Human Immunodeficiency Virus HSA Health Surveillance Assistant HTC HIV Testing and Counseling Hub SMS Server IFPRI International Food Policy Research IM Information Management INGO International Non-Government Organization JHU∙CCP Johns Hopkins University Center for Communication Programs K Malawian kwacha K4Health Knowledge for Health KII Key Informant Interview KM Knowledge Management LDP Leadership Development Program LiST Lives Saved Tool LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MOH Ministry of Health MSH Management Sciences for Health NAC National Aids Commission NGO Non-governmental Organization PH Private Hospital RH Reproductive Health RHU Reproductive Health Unit RI Research Institutions SNA Social Network Analysis SPSS Statistical Package for the Social Sciences STI Sexually Transmitted Infection TWG Technical Working Group 6 1. EXECUTIVE SUMMARY 1.1 Introduction New approaches to information management have great potential to change the landscape of health service delivery in less developed countries — by expanding the reach of health information to frontline health workers. The Malawi Knowledge for Health Project (K4Health) was designed to address gaps in how information on family planning and reproductive health (FP/RH) and HIV/AIDS is generated, shared and used at all levels of the health system in Malawi. Implemented by Management Sciences for Health over an 18 month period from January 2010 to June 2011, the project was part of the larger global Knowledge for Health Project, funded by the U.S. Agency for International Development’s (USAID’s) Office of Population and Reproductive Health, within its Global Health Bureau. K4Health is implemented by the Johns Hopkins Bloomberg School of Public Health's Center for Communication Programs (JHU∙CCP), FHI 360, and Management Sciences for Health (MSH). Project interventions focused on three key areas: • Formation of a national Knowledge Management Task Force to manage and disseminate technical information on FP/RH and HIV/AIDS, including centralized “toolkits”. • Establishment of two District Learning Centers (DLCs) in Nkhotakota and Salima to support information flows from and to the national, district, and community levels. • Creation of an SMS-based mobile phone network to improve communication and information sharing among community health workers. This report presents the results of an evaluation carried out in May-June 2011 to analyze the planning, implementation, and outcomes of the K4Health Malawi Project; and makes recommendations for similar knowledge for sharing programs in the future. Study questions were: 1. How successful were the approaches at the national and district levels to mobilize stakeholders around a common vision for improving health information access and use in Malawi? 2. How successful were the Malawi toolkits on RH/FP and HIV/AIDS and other resources available to health care providers through the District Learning Centers in increasing access to and use of health information to improve services? 3. To what extent did the SMS network in Salima and Nkhotakota reduce the communication gap between health workers and their district teams and thereby increase access to technical information among these health workers? 4. To what extent did the SMS network in Salima and Nkhotakota improve the ability of health workers to provide quality services and care? 5. How has the project built ownership within the Ministry of Health (MOH) to master key knowledge management processes promoted by K4Health, and to what extent can the MOH carry on project interventions? 7 1.2 Methodology The evaluation used quantitative and qualitative methodologies and collected information from the following data sources: review of key project documents, including the Needs Assessment; Net￾Mapping results of RH/FP and HIV/AIDS information flows at national and district levels; a survey of Community Health Workers (CHWs) using lot Quality Assurance Sampling (LQAS); focus group discussions (FGDs) and key informant interviews (KIIs) with SMS network users and non￾users, District Health Coordinators, and DLC users; review of mobile statistics from the SMS server (Hub) for SMS users; Review of DLC registries; and Malawi Toolkit Web statistics. K4Health conducted a pre-test of the SMS intervention from June – October 2010. A mid-term assessment of the SMS and DLC interventions was carried out in November 2010 prior to expanding the SMS network and included use of qualitative and quantitative measures and comparison groups. At mid-term community health workers in the two intervention districts and in one control district, without access to SMS technology were surveyed using LQAS. At endline both intervention districts and the control district had almost complete cell phone coverage, so that a new control district was selected. The use of these comparison sites was intended to correct for historical changes that might have affected results. An endline assessment was carried out in May/June of 2011. 1.3 Key Findings 1.3.1 Leadership Development Program (LDP) The Knowledge for Health project in Malawi offered an adapted Leadership Development Program (LDP) to bring together critical stakeholders from national and district levels and mobilize them around a common vision. The team selected three desired measurable results. The first was partially implemented but in the end not measurable, and the second and third surpassed their original targets. They were: 1. 60% of HIV/AIDS, family planning and reproductive health program managers at the national level are accessing comprehensive, accurate, up-to-date information through K4Health tools. Result: The team had difficulty developing a workable denominator for this result with all of the players at the national level in FP/RH and HIV/AIDS. 2. 40% of HIV/AIDS, family planning and reproductive health program managers at the district levels in Salima and Nkhotakota are accessing comprehensive, accurate, up-to-date information through K4Health tools. Result: By June 2011, 65% of the targeted workers were accessing information through K4Health tools including DLCs, Frontline SMS (FLSMS), and Toolkits. 3. A 60% increase in the number of health workers at the community level in Salima and Nkhotakota who are accessing comprehensive, accurate, up-to-date information through K4Health tools. Result: Over 77% of CHWs were successfully sending and receiving messages through their cell phones by the end of the project. 8 The Knowledge for Health project through the LDP initiated and supported the creation of the KM Task Force. This Task Force was led by the Health Education Unit, and reorganized during the period of the K4Health Project to include all relevant stakeholders, including Ministry of Health departments, Donors, INGOs, and the National AIDS Commission. By the end of this evaluation eighteen months later, the committee was in the process of becoming a permanent sub-committee of the Ministry of Health on Knowledge Management. 1.3.2 Results of Net-Mapping Information Flows at the National and District Levels A baseline Net-Mapping was undertaken in March 2010 before project implementation to assess bottlenecks or challenge points, and strengths and opportunities in information flows, to determine any actors that should be engaged, and to identify weak communication links that must be strengthened for the initiative to be successful in improving information flows to health workers at all levels. An endline Net-Mapping took place in June 2011. National level maps at baseline and endline identified a significant disconnect between information flowing from the national level down to the district level. In this aspect, national level maps did not change much from baseline to endline. National level actors have a high outdegree centrality, meaning they give the same or more information than they receive. The main changes from the 2011 maps are found at district level. What the project hoped would happen in districts, in terms of overcoming challenges and cut points (points in the network where all information passes through) in knowledge flows occurred. According to the participants in the Net-Map, the introduction of the DLCs and mobile phones has enabled front line health workers to share knowledge in a way that improves their service delivery, their supervisee relationships, and connections with their colleagues. 1.3.3 Malawi Health Toolkits The Malawi Health Toolkits were developed by the KM Task Force to serve as collections of information resources on a particular topic, created specifically for health policy makers, program managers, and service providers. They are downloadable from the Malawi page of the global K4Health website. Four National Level toolkits were developed. When asked during the Net-Map exercise and Focus Group Discussions what influence the toolkits might have on the existing flow of information at national or district level, the participants stated that it was too soon to tell. The toolkits were only recently published — 10 months before the end of the project for the RH/FP toolkits and 5 months before the end of the project for the HIV/AIDS toolkit. Findings on toolkit use indicate that resources are not yet in as much demand as expected within Malawi. Over a 10 month period, there were 79 Malawi specific visits to the Young People and RH toolkit, 117 visits to the Maternal and Neonatal Health toolkit, 152 visits to the Family Planning Toolkit, and over 5 months there were 172 visits to the HIV/AIDS toolkit. Monthly unique page views decreased for all toolkits between launch and endline in June 2011 from 455 to 247 for HIV/AIDS; 104 to 25 for Young People and Reproductive Health; 564 to 298 for Maternal and Neonatal Health; and 428 to 201 for Family Planning. The most useful aspect of toolkit development was the process itself of consensus building within the KM Task Force. 1.3.4 District Learning Centers Coordinated efforts between the District Health Officers (DHOs) in Nkhotakota and Salima Districts and K4Health resulted in a furnished DLC with at least four computers and Internet service in each of the districts. The DHOs covered the monthly Internet cost to the service 9 provider. Two DLC coordinators were recruited by K4Health to roll out the DLC intervention and also to build the capacity of health officers responsible for managing the learning centers. Users of the DLCs were trained in how to access both online and off line K4Health Malawi toolkits and other relevant websites. The Frontline SMS Hub was also housed in the learning center. Between June 2010 and June 2011, a total of 6874 DLC visits were registered in Salima and Nkhotakota. Of the total, 3,165 were from Nkhotakota while 3,709 were registered in Salima. The overall average number of visits to the DLCs per month was 529, translating to a daily average of 24 visits per work week. A gender gap in use of the DLCs was documented. Most of the visits to the DLC were from male health care providers and managers. When asked in FGDs what the reasons were for this gender disparity in DLC visits, participants responded that women work mostly as nurses, are always busy in wards, and have little time free to find answers to technical questions or use the Internet. They also have duties at home after work hours that their male counterparts did not have. A number of benefits of the DLCs were identified, including: their ability to address multiple needs, increased access to health information in general, more reliable clinical information about treating patients, improved reporting, better communication with community health workers through the SMS Hub, and an increase in information sharing. Challenges for the DLCs included: insufficient computers to meet demand; inconvenient hours, especially for women; lack of user knowledge about the toolkits; and erratic Internet availability. Access to the DLCs was dependent on the availability of Internet in the Learning Centers; DLC visits dropped in any month where the DLC experienced Internet interruptions. 1.3.5. SMS Network in Nkhotakota and Salima Between June – October, 2010, the project trained and provided mobile phones and solar chargers to 633 Community Health Workers (CHWs). This represented 77% of all health workers in Salima and Nkhotakota combined. The full 100% were not given phones as the project focuses on Family Planning and HIV/AIDS, the target were CHWs providing village clinics and DNPA. The project used technology created by K4Health partner FrontlineSMS, which offers a software platform that enables organizations in remote and underserved areas to turn a laptop computer and a mobile phone into a mass communications Hub, enabling two-way SMS (short message service or text messaging) communications without the use of the Internet. The average number of SMS messages sent to the Hub per CHW per month rose from 5.0 per month at mid-term to 6.2 at endline, as measured by the LQAS survey of cell phone users. As measured by Hub statistics, the total number of messages sent by CHWs rose from 44 messages per month between June and October 2010 to 1,091 messages per month between November 10, 2010 and June 11, 2011.The percent of health workers reporting on the LQAS survey that they sent an average of at least 5 messages per month increased from 44% at mid-term to 62.5% at endline, as training and mobilization of CHWs in the use of the SMS network intensified. Emergencies as a percentage of overall reasons for using the phone dropped from 64% at mid-term to 22% at endline. One of the most important outcomes of the SMS network was the reduction in time for CHWs to report important events and receive technical support from their supervisors. At endline, the average time required for CHWs to report important events (stock outs, transportation breakdowns) was 3 minutes, compared to 523 minutes for the control district of Nkhatabay for the same time period. 10 When asked how much time on average it took to contact and receive feedback from a supervisor or person providing technical support, CHWs participating in the SMS network reported it took 9 minutes, compared with 1,681 minutes (1.2 days) for CHWs in Kasungu District at mid-term and 1,498 minutes (1 day) for CHWs in Nkhatabay at endline. The detection and prevention of stock￾outs was an important result of the intervention self-reports from CHWs, and evidence from Hub statistics document their beliefs that the SMS network was also able to increase CHW self￾confidence, increase trust between CHWs and the communities they serve, decrease the cost of communication, widen service coverage, and increase the efficiency of referrals. 1.4. Conclusions • The adapted LDP and other approaches for mobilizing stakeholders around a common vision for improving health information access and use in Malawi were somewhat successful at the national level and highly successful at the district level. • DLCs improved access to and use of health information to improve services among district level health workers. By June 2011, 65% of the targeted workers in both districts were accessing information through K4Health tools, with more male than female workers benefitting. • The Malawi K4Health toolkits did not demonstrate high demand, but they had only been in existence for 10 months at the time of the evaluation (5 months in the case of the HIV/AIDS toolkit). • The SMS network in Nkhotakota and Salima succeeded in greatly reducing the communication gap between health workers and their district teams, and thereby increased access to technical information among these health workers. Over 77% of CHWs were successfully sending and receiving messages through their cell phones by the end of the project. • Self-reports from CHWs document that the SMS network was able to increase CHW self￾confidence, increase trust between CHWs and the communities they serve, decrease stock￾outs, decrease the cost of communication, widen service coverage, and increase the efficiency of referrals. • One of the most important lessons to be drawn from this multilevel intervention is that its effectiveness depended on the fact that it was delivered as a package to national, district and community levels, and each level made a particular contribution to improving health information flows. • The K4Health Project built in sustainability measures from the beginning of the project, and some were more successful than others. The Ministry of Health mandated its Health Education Unit to take over responsibility for of all the Knowledge Management interventions and dedicated an officer to continue updating the toolkits. The two intervention districts are committed to continuing the DLCs and the SMS network. The long term prospects for sustainability are not assured, however, in a country suffering from high levels of poverty, and shortages of fuel, medicines and health care workers. Six months post project, the early measures of sustainability will be evaluated by an external consultant to find out which of the interventions proved to be sustainable. 11 1.5. Recommendations 1.5.1 Weigh the effectiveness of each of the elements in the package of KM interventions and select those that yield immediate impact, are the most flexible, and offer the greatest return on investment. This pilot project has demonstrated the potential for real impact in health service delivery at the community level with a very small financial investment. To expand and scale up the package of interventions piloted by the K4Health Malawi Project, the evaluation team recommends continuing to grow the use of toolkits by the MOH and others through additional promotion, but without investing further in new platform development until the toolkits have proven their worth. Preference should be given to investing in and expanding the DLCs and SMS Networks. 1.5.2. Revise and expand the monitoring and evaluation design. Several elements yielded important information at a reasonable cost. These were the Net-Mapping workshops, the LQAS surveys, the focus group discussions and key interviews, and the collection of Hub statistics. Other methodologies proved difficult to implement in the field and probably should be dropped in future projects. These are lengthy DLC user surveys that included questions on toolkit use and the inclusion of control districts in the design of the LQAS survey. In the future design, an operations research study to measure associations between the use of cell phones and impacts on health services would be valuable. 1.5.3. During the design phase of future projects, engage districts in using data to identify their most pressing health problems and prioritize them for intervention. The Needs Assessment provided valuable information about multiple levels of the health system. There is added value in also engaging districts to use their own data to identify which interventions will have the most impact on health outcomes for their particular populations. Prioritize interventions that address those issues first and foremost. 1.5.4 Consider all potential costs up front and take steps to provide for them in the budget. Budget sufficiently for gas and vehicle use needs to get phones and training materials out to community health workers. Internet lines may need to be installed in intervention districts in order for the DLCs and the Hub to function. Future projects should establish a payment schedule that pays for Internet a month in advance to avoid interruptions in service. Future projects should work with mobile phone companies from the beginning to establish in-network groups that allow all messages passing through the Hub to be paid for. 1.5.5 Use the Leadership Development Program (LDP) and Net Mapping to energize and mobilize stakeholders. The adapted LDP enabled the KM Task Force to create a vision around which a team that had never worked together mobilized to achieve two out of its three desired measurable results. The Net-Map workshops also enhanced the sense of shared purpose and concretely demonstrated the constraints to information flows between national, district, and community levels. The Net-Maps also visually demonstrated the improvements in these flows following the introduction of the SMS network. Both interventions should be first steps in future KM projects. 12 2. INTRODUCTION One of the biggest challenges any health care system faces is how to turn overworked and stressed staff into a proactive, motivated team focused on delivering quality health services.1 A missing component for many health care workers in developing countries – from community level providers up to national level policy makers – is access to practical, up to date, locally relevant, and evidence￾based information they can use to improve the health of the populations they serve. In Malawi, where 80% of the population lives in rural and hard to reach areas, the lack of health information for front line workers can mean the difference between life and death. New approaches to information management, however, have great potential to change the landscape of health service delivery in less developed countries — by expanding the reach of health information to frontline health workers in remote areas. The Knowledge for Health Project (K4Health) was designed to address the gaps in how information on family planning and reproductive health (FP/RH) and HIV/AIDS is generated, shared, and used at all levels of the health system in Malawi. 2.1 The Knowledge for Health Project (K4Health) K4Health is a knowledge management project designed to increase the dissemination and use of the latest research and best practices to improve health systems, health service delivery, and health outcomes worldwide. K4Health is funded by the U.S. Agency for International Development’s (USAID’s) Office Population and Reproductive Health (PRH), within its Global Health Bureau. K4Health is implemented by the Johns Hopkins Bloomberg School of Public Health's Center for Communication Programs (JHU∙CCP), FHI360, and Management Sciences for Health (MSH). The K4Health Malawi Project, which is the subject of this evaluation, was a demonstration project carried out by MSH over an eighteen month period (January 2010-June 2011) as part of the larger global K4Health project. MSH conducted a Health Information Needs Assessment in Malawi from July – September 2009 to identify gaps in information flows on family planning and reproductive health (FP/RH) and HIV/AIDS across the different levels of the health system in Malawi. 2 Findings showed that there was a substantial information gap among managers and health providers working at all levels in HIV/AIDS and FP/RH. Most respondents reported that they lack up-to-date and relevant information that would help them improve the work they do. Information is scattered and difficult to access. The lack of available knowledge and information was even more pronounced among Community Health Workers (CHWs), who are generally trained once when they are recruited, and then rarely receive additional training to update their skills. Most CHWs reported having problems accessing new information and resources necessary for their jobs. The study also revealed a number of challenges that managers and service providers encountered when seeking information. For instance, Internet access was available among most health professionals at the national level, but was generally limited at the district level. Other district staff had to leave their posts and walk for a considerable distance to seek access through Internet cafés. Access to the Internet for CHWs was 1 Management Sciences for Health. Health Systems in Action: An eHandbook for Leaders and Managers, Cambridge 2010 http://www.msh.org/resource-center/health-systems-in-action.cfm 2 Sikwese, S. Malawi: Assessment of Information Needs Among Health Managers and Health Service Providers: Knowledge for Health Summary Report, November 2009. 13 non-existent. In terms of information management (IM), most organizations at the central level, including the government, had websites; however, most websites were not updated and did not offer a good platform for accessing up-to-date information. This assessment found that widespread access to mobile phones offers a promising new opportunity for information sharing, particularly at the district and community levels, where access to FP/RH and HIV/AIDS information is limited. In response to this and other findings from the assessment, K4Health designed an eighteen month demonstration project (January 2010 to June 2011) intended to test different ways to improve the exchange and use of FP/RH and HIV/AIDS knowledge within the health system in Malawi. Project interventions focused on three key areas: • Formation of a national Knowledge Management Task Force to manage and disseminate technical information on FP/RH and HIV/AIDS, including centralized “toolkits”, or electronic libraries of essential resources. • Establishment of two District Learning Centers (DLCs) at hospitals in Nkhotakota and Salima to support information flows between national, district, and community levels. The DLCs were designed to provide a range of resources, including print and electronic materials, and to house computers with free access to online and CD-ROM-based learning, including the Malawi Toolkits. • Creation of an SMS-based mobile phone network to improve communication and information sharing among community health workers. 3. EVALUATION QUESTIONS The goal of this evaluation is to analyze the planning, implementation, and outcomes of the K4Health Malawi Project, and to make recommendations for similar knowledge for health programs in the future. The evaluation looks at program assumptions and implementation, identifies challenges and success factors, and analyzes how they contributed to desired outcomes. The study aims to answer five key questions: 1. How successful were the approaches at the national and district levels at mobilizing stakeholders around a common vision for improving health information access and use in Malawi? 2. How successful were the Malawi toolkits on RH/FP and HIV/AIDS and the other resources available to health care providers through the District Learning Centers at increasing access to and use of health information to improve services? 3. To what extent did the SMS network in Salima and Nkhotakota reduce the communication gap between health workers and their district teams, and thereby increase access to technical information among these health workers? 4. To what extent did the SMS network in Salima and Nkhotakota improve the ability of health workers to provide quality services and care? 5. How has the project built ownership within the Ministry of Health (MOH) to master key knowledge management processes promoted by K4Health, and to what extent can the MOH carry on project interventions on its own? 14 4. METHODOLOGY 4.1 Information Sources and Design of the Evaluation This study used quantitative and qualitative methodologies, and collected information from the following data sources: • Review of key project documents including the Needs Assessment, Monitoring and Evaluation Plan, monthly monitoring reports, and Leadership Development Program (LDP) reports • Net-Mapping of RH/FP and HIV/AIDS information flows at national and district levels to assess bottlenecks, analyze strengths and opportunities, identify actors that need to be engaged, and identify weak communication links that must be strengthened for the initiative to be successful in improving information flows to health workers at all levels • A survey of Health Surveillance Assistants (HSAs) and Community Based Distribution Agents (CBDAs), for both SMS mobile phone users and non-users in intervention and control districts using lot Quality Assurance Sampling (LQAS) • Focus group discussions (FGDs) and key informant interviews (KIIs) with SMS network users and non-users, District Health Coordinators, and District Learning Center users • Review of mobile statistics from the SMS server (Hub) for SMS users • Review of DLC registries • Malawi Toolkit Web statistics K4Health conducted a pre-test of the SMS intervention from June – October 2010. A mid-term assessment was carried out in November 2010 prior to expanding the SMS network and included use of qualitative and quantitative measures and comparison groups. At mid-term community health workers in the two intervention districts and in one control district who did not have access to SMS technology were surveyed using LQAS. At endline both intervention districts and the control district had complete cell phone coverage so that a new control district needed to be selected. The CHWs in the control district of Kasungu had been given phones and instruction on FLSMS through a different Family Planning project underway in Malawi during the 18 months of the K4Health project, thus contaminating the use of Kasungu as a control district. A new district of Nkhatabay was chosen for control that had similar characteristics as the original control district but without phones. The use of these comparison sites was intended to correct for historical changes that might have affected results. An endline assessment was carried out in May/June of 2011. Figure 1 highlights the major project interventions together with evaluation activities. 15 Figure 1. K4Health Malawi Evaluation Timeline 4.2 Net-Mapping at National and District Levels In May 2010, Management Sciences for Health (MSH) and the International Food Policy Research Institute (IFPRI) jointly conducted group Net-Mapping in Malawi to provide a rapid appraisal of the network within which MSH was undertaking its intervention, as an addition to the previously completed extensive needs assessment activities. This ‘snapshot’ of the network was also designed to serve as one component of the monitoring and evaluation activities, to be repeated at the end of the intervention to track the changes in the network over the life of the intervention. Net-Map highlights formal and informal interactions among key actors in a network, their degree of influence, and the patterns of communication and information exchange. The Net-Map tool draws on social network analysis approaches as described in Schiffer and Waale.3 However, the tool goes further by including principles of power mapping4 and stakeholder analysis approaches. 5 This evaluation describes the specific process of applying Net-Map in this context and the questions that were asked in both May 2010 and July 2011. Then, it displays selected maps and provides a brief explanation of their characteristics. Finally, it highlights and discusses key lessons learned from the Net-Map group interviews. For a full description of the Net-Map component of the K4Health Malawi project, please see “Net-Mapping Report, K4Health Malawi” by Natalie Campbell and Thokozani Bema, Management Sciences for Health, July 7, 2011. 3 Schiffer, E., and D. Waale. Tracing power and influence in networks: Net-Map as a tool for research and strategic network planning. IFPRI Discussion Paper 772. Washington, D.C.: International Food Policy Research Institute, 2008. 4 Schiffer, E. The power mapping tool: A method for the empirical research of power relations. IFPRI Discussion Paper 703. Washington, D.C.: International Food Policy Research Institute, 2007. 5 Holland, J. Tools for institutional, political, and social analysis of policy reform: A sourcebook for development practitioners. Washington, D.C.: World Bank, 2007. 16 Three Net-Map group interviews were conducted at baseline and endline with selected key stakeholders at the national and district levels. In total 30 respondents participated at baseline and 26 at endline. The same organizations were included in each assessment but not necessarily the same individuals. There was, however, some overlap between individuals. District interviews were undertaken in both demonstrations sites, Salima and Nkhotakota. Each group interview took place during a half-day workshop. Participants included representatives from government at various levels of the Ministry of Health, donor organizations, national and international non-profit organizations, and multilateral organizations. The purpose of the group interview, as explained to participants, was to understand if there have been any changes with key actors and challenge points in the network of information flow of technical health information to health workers at the national, district, and community levels. The scope of technical information covered by the K4Health Malawi Demonstration Project is quite broad. In order to facilitate clear and concrete responses from participants, the facilitator narrowed the scope of the information discussed in the context of the group mapping session. Table 1 presents the questions used during the Net-Map sessions with questions in bold being those that were added for the follow up Net-Map. Table 1. Net-Map Interview Structure In developing the above interview structure, the first group of stakeholders (those convened at the national level) in May 2010 were asked to choose some specific aspects of HIV/AIDS and FP/RH knowledge that represent particular challenges in terms of information flows. The group thought there were many challenging areas, but that two tangible ones that would be useful to map were HIV/AIDS Testing and Counseling (HTC) and family planning (FP). The question put to participants was: “How much influence does this actor have in this specific field of HIV/AIDs Testing and Counseling or Family Planning?” The selection of these two sub-areas set the structure for the three mapping exercises at baseline in May 2010 and endline in June 2011 6 “Who” refers to all stakeholders in Malawi. It encompasses individuals, offices, organizations, people etc. Overall Question Who6 plays a role in improving flows of technical information on HIV/AIDS and Reproductive Health to Healthcare Service Providers? Step 1: Name generator: List the actors involved7 Step 2: Link generator: What are the critical information flows for Malawi in improving Healthcare for HIV/AIDS and Reproductive Health? Specific Links: - Who provides technical information on HIV testing - Who provides technical information on family planning Step 3: How influential is each actor over improving these information flows? Step 4: Discussion a. Who are the critical actors? b. What are the critical linkages? c. How can we ensure improved information flows? d. Of the links, which have been strengthened due to the formation of Toolkits? e. Of the links, which have been strengthened due to the mobile phones? f. Of the links, which have been strengthened due to the DLCs? Step 5: Comparison of map: What are some of the significant changes you see? What do you think are the reasons for these changes? 17 In step one, participants in the Net-Map session identified all actors involved in the network of interest. In this case, the network was defined as the flow of HIV/AIDS and family planning/reproductive health (FP/RH) information in Malawi, which includes the production, exchange, and storage of information. The names of all actors were then written on sticky notes and placed on a large sheet of paper. In step two, the facilitator drew arrows to depict the links between the actors, using a different colored arrow for each kind of link. In step three, participants used checker pieces to create “influence towers” next to each actor card; the higher the influence, the higher the tower. Sources of influence could range from legitimate decision-making capacity, to giving advice or incentives, to bending or breaking the rules. Respondents were then asked to rank the actors they listed with an influence value from 0 to 10 (0 is the least influence and 10 is the highest). Those with more influence had a taller "influence tower". Measurement of influence is subjective; participants in the room determined how influential a particular actor is. That said, because it is a group exercise, participants collectively agree on how influential a particular actor is. This process resulted in a three dimensional sketch of the actors, their links, and their direction of influence. This sketch was used as a basis for the group’s discussion about challenges and opportunities regarding information flow (see photo below). Using Net-Map led to a combination of visual results (the network maps), quantitative results (network data), and qualitative results (network narratives). The facilitator also asked about the “how” and “why” of this network as it evolved in front of participants’ eyes. Finally, data from the maps were entered into VisuaLizer software to create computer generated maps and to calculate centrality values. Degree centrality is a measure of the position of individual actors within the network; it shows how many incoming and outgoing links each actor has. We differentiate between in-degree (incoming links) and out-degree (outgoing links) to determine who is receiving information on FP and HIV (in-degree) and who is sending information (out-degree). 4.3 LQAS Survey of Community Health Workers 4.3.1 Sampling In June, 2010 the K4Health project established the SMS-based mobile telephone network using FLSMS in Salima and Nkhotakota districts, a catchment area of 652,326 people. The intervention 18 began with the provision of training and mobile phones for an initial group of health workers in these two districts. The first 5 months – June to October, 2010 – served as the pretesting phase of the intervention. Because of the complete lack of cell phone penetration among CHWs in the two intervention districts, the project decided not to carry out a baseline during this pretesting phase but to wait until mid-term of the project in November 2010, to assess how implementation was proceeding prior to expanding the network to additional health workers. K4Health carried out the mid-term assessment in November 2010 using LQAS. One of the primary goals of the mid-term LQAS survey was to establish average values for project indicators that could inform coverage benchmarks or decision rules for the endline survey. Little was known about how many SMS messages could be expected from CHWs per month, or how much the cell phones would reduce the time needed to get answers to critical service delivery questions. Because the mid-term was mounted after the pretesting phase, it used a staged design to assess process data from the pretest and establish mid-term values for the next group of users to be included in the network. The mid-term study compared three groups: (1) SMS users in the pretest phase in Salima and Nkhotakota (2) Non-users in Salima and Nkhotakota who were expected to enroll in the SMS system in the next wave of the project (3) Non-users in a non-random control district (Kasungu) Kasungu was selected as the control because it had a similar socio-economic profile as Salima and Nkhotakota, but health workers did not have access to an SMS network. During the 18 month period of the project, however, health workers in Kasungu were also given cell phones and access to an SMS network through another Family Planning project, so for the endline survey, the district of Nkhatabay was selected as the control as it had a similar socio-economic profile. The map below shows intervention districts Nkhotakota and Salima (circled in blue) and control districts Kasungu and Nkhatabay (circled in orange). 19 Figure 2. Map of Malawi showing intervention and control districts Sampling in all four districts included both Health Surveillance Assistants (HSAs) and Community￾Based Distribution Agents (CBDAs). HSAs can be either male or female and are the lowest level of civil servant in the public health system. They are paid employees of the MOH and are based in communities. They work in mobile or outreach clinics, village clinics, or health posts and assist with implementing Malawi’s Essential Health Package. CBDAs are male and female volunteers selected by and based in their communities to provide FP counseling, oral contraceptives, and condoms. The agents can refer women desiring other FP methods or who need more counseling on side effects to health centers or outreach clinics. CBDAs are recruited, trained, and managed by nongovernmental organizations (NGOs), such as the Christian Health Association of Malawi.8 HSAs and CBDAs are both referred to as CHWs or health workers throughout this evaluation report. For the mid-term LQAS, each of the three districts was divided into five supervision areas or lots. However, because true administrative supervision areas do not exist at the district level in Malawi, a decision was made at the time to cluster CHWs according to the health centers at which they gathered for meetings and roll these into supervision areas. By endline, the project had considerable experience with how the districts worked through the SMS activities with CHWs, and a clearer 8 Richardson, F., M. Chirwa, M. Fahnestock, M. Bishop, P. Emmart, and B. McHenry. Community-based Distribution of Injectable Contraceptives in Malawi. Washington, DC: Futures Group International, Health Policy Initiative, Task Order 1, 2009. 20 picture emerged of a better way to divide up the supervision areas operationally. For the endline, therefore, the two intervention districts and the control district were each divided into 4 supervision areas. Although the decision to use a different sampling frame at endline makes results more meaningful to the districts in terms of identifying weaker supervision areas needing more support, it means that we cannot compare individual supervision areas at mid-term and endline. We can only compare indicators at mid-term and endline using aggregate data at district level across all of the supervision areas. We can use endline data, however, to determine the degree to which individual supervision areas in the two intervention districts met or did not meet coverage benchmarks for key project indicators. The LQAS is a powerful performance monitoring tool. General sampling may not give us a picture of specific location performance measures similar to LQAS. The districts are encouraged to use this information to mobilize and empower health workers in low performing locations to improve their performance. For mid-term and endline surveys, respondents were sampled from each supervision area using a random number table and Ministry lists of all HSAs and CBDAs. At mid-term, this yielded a total sample size of 285, of which 65% were HSAs and 35% were CBDAs. At endline, it yielded a total sample size of 228, of which 63% were HSAs and 37% were CBDAs. Tables 2 and 3 below outline the distribution of respondents at mid-term and endline. 21 Table 2. Distribution of LQAS respondents at mid-term Catchment area Target group No. of supervision areas No. of respondents/ supervision area Total respondents Salima & Nkhotakota FLSMS users 5 Nkhotakota Central = 18 Nkhotakota North = 14 Nkhotakota South = 18 Salima Chipoka = 16 Salima Khombedza = 29 95 Salima & Nkhotakota Non users 5 (same 5 as above) 19 95 Kasungu (control) Non users 5 19 95 Total 285 Table 3. Distribution of LQAS respondents at endline At endline, the project conducted focus group discussions with HSAs and CBDAs in the intervention districts in order to explore how they communicated with each other and with supervisors at the district level, how often, for how long, for what purpose, and at what cost compared to how they communicated before having cell phones. SMS users described barriers to using the SMS network and ways in which the network enabled them to provide quality care to clients. In addition, the project conducted FGDs and individual interviews with FP/RH and HIV/AIDS Coordinators, and District Learning Center users in both Salima and Nkhotakota. Respondents for the focus groups at the community level were selected to Catchment area Target group No. of supervision areas No. of respondents/ supervision area Total respondents Salima SMS users 4 19 76 Nkhotakota SMS users 4 19 76 Nkhatabay (control) Control sample 4 19 76 Total 228 22 have a mix of HSAs and CBDAs and different supervision areas represented. Respondents for the focus groups at the district level were selected to have a mix of professional profiles using the DLCs, including physicians, nurses, dentists, District Health Management team members, and members of the K4Health District Task Force who participated in the Leadership Development Program (LDP). Respondents for Key Informant Interviews at the District level were selected based on their positions as District Coordinators who would be most involved in the SMS network. Focus group discussions included between 5 and 10 participants in each discussion. Table 4 below summarizes the number of Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs) at endline by target group and type of respondent. Table 4. Summary of Focus Groups Discussions and Key Informant Interviews at endline District Target Group/KII No. of FGDs per respondent type Total HSAs &CBDAs District FGDs/KII Salima SMS users 1 FGD 3KIIs 4 FGD/KII DLC Users 0 2 FGDs 2 FGDs Nkhotakota SMS users 1 FGD 3KIIs 4 FGD/KII Non users of SMS 0 2 FGDs 2 FGD Total 2 10 12 FGD/KIIs 4.3.2 Data Collection, Entry, and Analysis for LQAS survey and FGDs Data collection was carried out by seven interviewers hired by the project. The interviewers were trained for three days on best practices in data collection and use of the mid-term study tools. The third day of training also involved pre-testing the tools to be used in this study. LQAS survey data were entered, cleaned and processed using SPSS software. Data from the SMS Hub were also exported to SPSS for analysis. For the qualitative data, each FGD was recorded and an enumerator was trained to take notes of the dialogue during each focus group and individual interview. Interviews were not individually transcribed but summarized in tables using Microsoft Word. This approach saved resources and allowed the study team to analyze the interview notes using structured data summary tables highlighting the major focus areas of the study. When individual quotes were needed, these were transcribed verbatim. No names were linked to quotes. Verbal consent to record the discussions was obtained from participants before the discussions began. 4.3.3 Quality Control Procedures The project used the following quality control measures: • Interviewers were trained on data collection for two days, including pre-testing of the tools to provide practical experience and feedback. • Interviewers checked for completion of each questionnaire. Once the questionnaire was submitted to the supervisor, the supervisor checked the completeness of the questionnaire and returned it to the interviewer if gaps were identified or if it was not filled out properly. • The study team held daily review meetings to discuss common errors and find ways to ensure these would be avoided in subsequent days. 23 4.4 Review of mobile statistics from the SMS server (Hub) for SMS users Data on the number of messages, their origin, and the purpose of the message were extracted from the Hub database. The FLSMS application has fields with sender’s name and number, date, and the content of the message sent. The FLSMS data allows export to data analysis software such as SPSS. SPSS allows monitoring of all incoming and outgoing text messages (including the name and location of the sender and purpose), and allows manipulation of the data to give an overall picture of the number of messages sent and received per month. Further analysis gives the number of messages by category, or by purpose, or by sender, depending on the reporting requirements. The FLSMS system set up a key word mechanism that allows sending group messages to a specified target group depending on the key word used. 4.5 District Learning Center Registries A DLC User’s Survey had been in use since the launch of the DLCs in September 2010 and had encountered a number of difficulties, including the limited time visitors to the DLC had to complete the survey. Very few visitors to the DLCs completed the survey, and therefore it had little validity in terms of representing the experiences of all users. In April 2011, a more streamlined version of the survey was developed, but even this survey took too much time for visitors to complete. Two indicators in the project M&E plan relied on the User’s Survey as a source of data and will not be possible to measure. These are: • Total number of DLC users who state they have applied the knowledge they gained in their daily work • % of DLC users who state they have applied the knowledge they gained in their daily work Users did provide qualitative feedback through the FGDs on how they applied their knowledge to improve patient care, and these comments are reported in the evaluation. Data from DLC registers that have been entered into an electronic database were used to assess overall use of the DLCs. The registers collected data on name, sex, and the general purpose of the visit. 4.6 Malawi Toolkit Web Statistics During the K4Health project, four Malawi specific toolkits were produced and posted on the Malawi page within the global K4Health website. These were: Family Planning; Young People and Reproductive Health; Maternal and Neonatal Health; and HIV/AIDS. Examples of materials include national policies and strategies, Ministry of Health Guidelines, facilitator guides, reports, and educational posters. Most of the materials are in English with a few in Chichewa. The following toolkit statistics were gathered using Google analytics: page views, unique page views, top content, top downloaded resources, average time on top content pages, top country visits to top content, visits from Malawi, number of countries visiting toolkit, and top country visits to toolkit. Questions about use of the HIV/AIDS and FP/RH toolkits could not be collected from the DLC electronic database. Users did not have individual user names and so never logged in or out of the computer when their sessions ended; new users simply picked up where another user left off. Data on number of hits to the toolkit and specific resources downloaded were available from Google analytics as described above, but even those underestimate the number of hits, since only a single IP address is counted, and not multiple users from the same IP address. 24 5. FINDINGS AT THE NATIONAL LEVEL 5.1 Program Implementation The Malawi Health Information Needs Assessment of September 2009 reported that Malawi had a number of network organizations, associations, and government-sponsored TWGs that could potentially play an important role in information sharing. However, these networks needed considerable capacity-building support. Recognizing that creating a knowledge sharing culture requires a commitment from many individuals, MSH began by establishing a Knowledge Management (KM) Task Force that included representatives from both the national level and the district level. In February 2010, K4Health Malawi launched a Leadership Development Program (LDP) to mobilize project stakeholders around creating a common vision they themselves chose: to see “happier, healthier, and more productive individuals, families, and communities as a result of better access to and use of family planning, reproductive health and HIV/AIDS information.” 5.1.1 Overview of the Adapted Knowledge for Health LDP The LDP is a four to six-month process that develops the leading and managing capacity of people working at all levels of the health system. Over a series of workshops, teams apply action learning and problem-solving techniques to address real challenges, and design an action plan to achieve a measurable result. At the heart of the LDP is the challenge model. The challenge model is a simple tool which enables teams to take a systematic look at how to produce desired measurable results. Teams work together to produce a vision of what success would look like, scan their environment, and focus on measurable results they can achieve within a short time period. They conduct a systematic analysis to understand their current situation, the roles of their stakeholders, the obstacles they are facing, and the root causes that are preventing them from achieving their results. They then formulate their challenge and develop an action plans to meet the challenge and come closer to their vision. They design a monitoring and evaluation (M&E) plan that identifies indicators to monitor their measurable result and decide on the data sources they will use to measure their indicators. Evaluations carried out by MSH of LDPs across over 30 countries since 2003 have shown that it achieves public health results and in the process builds effective committed teams. This was especially important for the Malawi KM Task Force which had never worked together on a regular basis. The Knowledge for Health project in Malawi offered an adapted Leadership Development Program (LDP) over four 2-day workshops from February to November 2010 to bring together and mobilize 25 critical stakeholders around a common vision. The adaptations consisted of condensed content in order to fit into the short time frame of the K4Health Project. For its mission, the KM Task Force chose to make family planning and HIV/AIDS information easy to access and use to improve health impact in Salima and Nkhotakota, a catchment area of 652,326 people. These two districts were chosen to enable collaboration and integration with other MSH projects working in these districts at the hospital level and with the CHWs. The common vision they chose was to see “happier, healthier, and more productive individuals, families, and communities as a result of better access to and use of FP/RH and HIV/AIDS information.” They took into account workers at all levels, and identified the following measurable results they hoped to achieve by December 31, 2010: • Desired Measurable Result 1. 60% of HIV/AIDS, family planning and reproductive health program managers at the national level are accessing comprehensive, accurate, up-to￾date information through K4Health tools. 25 • Desired Measurable Result 2. 40% of HIV/AIDS, family planning and reproductive health program managers at the district levels in Salima and Nkhotakota are accessing comprehensive, accurate, up-to-date information through K4Health tools. • Desired Measurable Result 3. A 60% increase in the number of health workers at the community level in Salima and Nkhotakota who are accessing comprehensive, accurate, up￾to-date information through K4Health tools. The K4Health project supported the LDP teams to achieve these three results through the following project activities: 1. Development and promotion of four Malawi-specific electronic toolkits containing RH/FP and HIV/AIDS information and resources 2. Identifying sites for the District Learning Centers, procuring necessary equipment, and recruiting personnel to staff them 3. Launching an SMS system to connect CHWs to supervisors and the DLCs in Salima and Nkhotakota Districts 4. Training district and community health workers on technology skills, including computers and SMS messaging 5. Capacity building for MOH staff to manage information toolkits, District Learning Centers, and mHealth interventions 5.2 Results of the Leadership Development Program Prior to the LDP, the two districts did not have a formal mechanism for knowledge exchange. Bringing district representatives and national representatives together through the LDP to create the task force greatly facilitated communication and knowledge exchange and created a sense of shared ownership. In December 2010, the final LDP results workshop was held, and the three taskforces made presentations on progress made towards their results. A cross section of stakeholders including Donor agencies, International and National Non-Governmental Organizations, Government Ministries, and District Health Offices came for the presentations. In the short time frame of 6 months after the end of the LDP, the following measures that were put in place to foster ongoing knowledge exchange, and are indications of sustainability: • The Salima District Hospital, of its own accord, finished a large reconstruction to expand the room housing the District Learning Center because of its popularity among health workers. • Both Salima and Nkhotakota seconded staff to work in the learning centers so they could assume maintenance of the SMS Hub and centers once the K4Health Project ended in June 2011. • National Technical Working Groups had assumed responsibility for maintaining the electronic toolkits. • The 2 Districts had incorporated monthly internet subscriptions as regular components of their budgets In addition to improving the ability of the KM Task Force to move forward as a team, one of the key measures of an LDP’s success is the ability of the team to achieve its desired measurable results. 26 Of the desired measurable results listed above, by June 2011, two out of the three were achieved, as shown in Table 5. Table 5. Achievement of LDP Task Force Desired Measurable Results The Knowledge for Health project through the LDP process initiated and supported the creation of the KM Task Force. This Task Force, which initially was formed to lead Toolkit development, was, with the leadership of the Health Education Unit, reorganized during the period of the K4Health Project to include all relevant stakeholders, including Ministry of Health departments, Donors, INGOs, and the National AIDS Commission (NAC). By the time of this evaluation eighteen months later, the committee was in the process of becoming a permanent sub-committee of the Ministry of Health on Knowledge Management. The committee planned to take charge of the Malawi page and toolkits on the global K4Health Website to ensure that it is updated with information and resources relevant to Malawi. The committee has also met to discuss plans for a complete takeover of the Malawi K4Health knowledge management interventions, including looking for additional donor funding to support these activities. The plan clearly stipulates detailed activities to be carried out, responsibility lines, resources needed, and how the resources will be acquired. 5.3 Toolkits The Malawi Health Toolkits were developed by the KM Task Force to serve as collections of carefully selected information resources on a particular topic, created specifically for health policy makers, program managers, and service providers. Four National Level toolkits were developed on Family Planning, Maternal and Neonatal Health, Young People and Reproductive Health, and HIV/AIDS. Users are encouraged to suggest resources or post comments to the discussion board in each toolkit. Toolkit promotion activities were carried out by the K4Health Project with a wide variety of stakeholders, including Health Training Institutions and National stakeholders implementing Family Planning and HIV/AIDS interventions. Figure 3 displays a screen shot of the Malawi K4Health Web page. Desired Measurable Result Status 1. 60% of HIV/AIDS, family planning and reproductive health program managers at the national level are accessing comprehensive accurate, up-to￾date information through K4Health tools. Not measurable: The team had difficulty developing a workable denominator for this result with all of the players at the national level in FP/RH and HIV AIDS. There was also no way to measure only national managers’ use of the toolkits through Google analytics. 2. 40% of HIV/AIDS, family planning and reproductive health program managers at the district levels in Salima and Nkhotakota are accessing comprehensive accurate, up-to-date information through K4Health tools. Achieved and surpassed: By June 2011, 65% of the targeted workers were accessing information through K4H tools including DLCs, FLSMS, and Toolkits. 3. A 60% increase in the number of health workers at the community level in Salima and Nkhotakota who are accessing comprehensive accurate, up-to￾date information through K4Health tools. Achieved and surpassed: Over 77% of CHWs were successfully sending and receiving messages through their cell phones by the end of the project. 27 Figure 3. Screen Shot of Malawi K4Health Web Page with Toolkits The K4Health Malawi Project and KM Task Force members undertook promotional activities including PowerPoint presentations, oral presentations, fact sheet distribution, and K4Health website demonstrations. Specific target audiences for toolkit promotion included: • Over 500 Undergraduate Nursing and Medicine students studying Nursing and Clinical Medicine • 24 graduate students studying for a Masters Degree in Reproductive Health and their Lecturers at the University of Malawi – Kamuzu College of Nursing • College Librarian who trains students to conduct electronic information research • 30 National Stakeholder organizations involved in HIV Prevention activities at a National Condom rebranding conference in Blantyre • Malawi BRIDGE project and their 22 sub grantee organizations Students expressed their feedback on the usefulness of the toolkits: “We are very grateful to this wonderful work which comes at the right time when all of us were struggling to find Malawi specific documents on RH. This will go a long way in easing our work as we do our final dissertations. We are eager to work with the project for more interaction so as to make sure the site contains relevant material for our information needs.” Team leader for the Master in Reproductive Health group The majority of the trainings listed above occurred in February. There was a direct correlation to toolkit activity from within Malawi in the month of February due to these promotional trainings, which can be seen in Figure 4 below. 28 Figure 4: Visits to Malawi Toolkits from within Malawi 5.4 Sustainability of the Toolkits The toolkit development process was led by the KM Task Force made up of already existing national technical working groups, i.e., the Sexual & Reproductive Health and HIV Prevention Technical Working Groups (TWGs). This deliberate effort made the Toolkit Development Task Forces accountable to the permanent structures, and also ensured that the Toolkit development capacity remained within the TWGs even after K4Health project phase-out. The K4Health project also played an advocacy role with the Ministry of Health to identify a specific department to take over the implementation and coordination roles. The Ministry of Health has since mandated its Health Education Unit to takeover and lead all the Knowledge Management Interventions, and dedicated an officer to continue updating the toolkits. The National Task Force is now turning into a permanent Knowledge Management sub-committee with the leadership of the Health Education Unit. When asked during the Net-Map exercise what influence the toolkits might have on the existing flow of information at national or district level, the participants unanimously stated, ‘It’s too soon!” It was felt that the toolkits were only recently published — 10 months before the end of the project for the RH toolkits and 5 months before the end of the project for the HIV/AIDS toolkit — and needed more time to become established resources used by health professionals as national or district level. The most useful aspect of toolkit development was the process itself of consensus building within the KM Task Force. As displayed in Table 6, statistics collected through Google analytics paint a picture of resources that are in demand internationally, but that are not yet in as much demand as expected within Malawi. While the initial launch of the toolkits correlated to high numbers of page views, those quickly lowered over the months into a steady small stream of visitors, with a increases in visits correlated to promotional trainings in Malawi. A typical trend in page views of any new web site or online platform is to start of very high. There is the initial excitement of a new release with significant activity reflected in the analytics, followed by a period of lower visits, with an eventual leveling off. Without MOH ownership and requirements to use or drive traffic, it is not surprising that the numbers dipped quite low. Monthly global unique page views decreased for all toolkits between 0 20 40 60 80 100 120 140 160 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Number of visits Total visits per month Toolkit visits from Malawi Young People and Reproductive Health Toolkit Maternal Newborn Health Toolkit Family Planning Toolkit HIV/AIDS Toolkit 29 launch and endline in June 2011 from 455 to 247 for HIV/AIDS; 104 to 25 for Young People and Reproductive Health; 564 to 298 for Maternal and Neonatal Health; and 428 to 201 for Family Planning. Table 6. Use of Malawi specific toolkits between 10/10 and 6/11 Name of Toolkit # of months Total Unique Page Views Top Five Downloaded Resources in June 2011 Number of Malawi Visits Number of Other Countries Visiting Young People and RH 10 1,400 • Reaching Out to Teen Mothers in Malawi • Report-- Profiling Early Marriages in Malawi • Abstinence Strategy (PowerPoint) • Malawi Youth Friendly Health Services Facilitators Guide 79 57 Maternal and Neonatal Health 10 3,677 • Maternal and Neonatal Health (PowerPoint) • Basic Emergency Obstetric and Newborn Care (BEmONC ) or Comprehensive Emergency Obstetric and Newborn Care (CEmONC) Site Supervision Checklist • Mukuchedweranji (poster) • Newborn Health Program: Knowledge, Practice and Coverage Survey for Mothers of Children 0-23 Months in Mzimba District, Malawi • AKMC Register 4 Book [Kangaroo Mother Care Patient Tracking Register] 117 91 Family Planning 10 3,265 • Monthly Injectable (Counseling Card graphic) • Monthly Injectable (brochure graphic) • Community Based Family Planning and HIV/AIDS Services Project Presentation (PowerPoint) • Male Condom (brochure graphic) • Male Condom (Counseling Card graphic) 152 97 HIV/AIDS 5 1,785 • National pediatric HIV flipchart • Module 9 of National PMTCT Training Package • Situation Analysis of Male Circumcision in Malawi • HIV-AIDS Research Strategy 2005-2006 172 14* *Only 2 months of data on other country visits represented Other issues surrounding the toolkits were the types of information included. It was felt that toolkits can be very relevant to specific groups of people (i.e. students and researchers) and less relevant to other players who would only require FP and HTC reference material once in a while, such as when they are either developing a proposal or planning a training event. Other comments on the toolkits were that, while they contain valuable information, participants said they would prefer toolkits to be relevant to the daily challenges they meet in their clinical and nursing duties when they need such 30 references. The use of toolkits by frontline health workers was not mentioned at all due to their lack of Internet access. One idea for toolkit use came from someone who works with youth outreach programs. He suggested that the MOH could accelerate toolkit use by young people, who use the Internet extensively, by linking toolkits to social networks like Facebook. Another suggestion for how to drive traffic to toolkits was to have the Ministry of Health mandate the use of the K4Health/Malawi website by using it as a central place to hold organizing documents such as a calendar of events, meeting minutes, and other such documents. 5.5 Findings from Net-Maps at the National Level Figure 5 below presents the national level stakeholder Net-Map at baseline for HIV/AIDS and Family Planning. Each actor is depicted with a yellow circle, with the circle size varying according to influence level (the larger the circle, the higher the relative influence according to the influence towers set up by participants). Linkages between actors are depicted by lines, and the arrowhead shows the direction of the information flow. For instance, the international NGOs provide technical information on family planning to the national NGOs: INGO  NGO. Each type of linkage is shown in a different color and pattern — dotted blue lines for HIV Testing and Counseling (HTC) and solid green lines for Family Planning (FP). 31 Figure 5. National Level Net-Map in 2010 Figure 6. National Level Net-Map in 2011 HIV Testing Information Family Planning Information Service provision *Acronym list on page 4 *Acronym list on page 4 HIV Testing Information Family Planning Information 32 In the baseline map influence among the national-level actors is relatively uniform, spread out among relevant sections of the Ministry of Health (MOH), donors, and international NGOs. The National AIDS Commission is seen as having the highest influence on the map and the community￾level actors have the least. The Health Education Unit of the MOH was not seen as playing a large role, but as an actor who could potentially be influential if engaged. Though all actors on the map seem fairly well connected, upon closer inspection, one can see that the community-level actors (service providers linked via black dashed line) tend to be recipients of information, but less frequently provide information, whether to other community-level actors or passing information back up the line. In short, information flow in the Malawi health system was largely vertical and top￾down. The Reproductive Health Unit (RHU) and the National AIDS Commission (NAC) were seen as influential actors; the most arrows converge at these points on the map. This was said to be due to their mandate to ensure that dissemination of all information is conducted through the two units. There is a big dependence on these actors to disseminate information. Comparison of Baseline and Endline Net-Maps at the National Level When looking at the 2010 map compared to the 2011 map, we see that despite some increase in detail at endline, the national level mapping produced very few changes related to K4Health interventions in actors and influence. 9 There is still a significant disconnect between information flowing from the national level down to the district level. A few changes were highlighted. Media was one actor identified as having more influence now, and reporting more on Family Planning/Reproductive Health issues in the news. The most significant change was that there was more recognition of CHWs (circled in the 2011 map above in Figure 5) as having greater influence on information flows with increased numbers of inflowing and outflowing arrows. Figure 7 below presents the centrality values for each actor from the mapping in 2011. Degree centrality is a measure of the position of individual actors within the network; it shows how many incoming and outgoing links each actor has. The larger the centralization score, the more likely it is that a single actor is quite central with the remaining actors considerably less central, and probably at the periphery of the network. Essentially, it expresses how unequal, variable, or heterogeneous the actor centralities are in the network. Actors are listed according to their level: national level actors on the left, district level actors in the middle, and community level actors on the right. The vertical axis in the diagram displays the centrality values which are expressed as percentages (rather than other units, such as kg or years). For the purposes of this project, we would want to see that district and community level actors have an equally high centrality score as those at national level. This would indicate that health managers and workers have access to and can request and receive the health information they need in order to improve service provision. 9 For further details and analysis on the Net-Map findings, please see full report. 33 Figure 7. In Degree and Out Degree Centrality Values from National Stakeholder Net-Map 2011 We can see that the national level actors give the same amount or more information than they receive, or that they have a high outdegree centrality. This means that overall they have more out￾flowing links than in-flowing links and are more influential than other actors. This did not change much from the first map 15 months earlier. Conversely, the community level actors’ centrality values have changed a great deal. In 2010, community health workers and clients had high indegree levels of centrality, and were receiving far more information than they gave. In the case of clients, they had no outdegree centrality whatsoever. Yet, in 2011, we see CHWs and clients have outdegree centrality almost equal to the indegree centrality at the district level. This finding is even more striking in the net maps produced at the district level to which we will turn next. When asked why this might be the case in the districts, participants expressed that these changes are due to the cell phones they now have that give them instant access to their supervisors and district hospital staff for not only asking questions but responding to questions and requests from district hospital staff, essentially creating a two way flow of information that did not exist previously. In network mapping, we gather in depth qualitative information during the group interview based on the “perception” of the participants as they answer the question about how the information flows between the actors of this network. In Salima and Nkhotakota, the new perception of supervisors and CHWs at endline is that they now have the ability to both send and receive health information. This was reflected in the maps they drew through the links and the direction of the links flowing both ways. The data analysis within the SNA software interprets this as indegree and outdegree to reflect this two way flow of information that did not previously exist. 5.6 Impact of K4Health Interventions at the National Level and Suggestions for Future Improvements Results from the work of the K4Health Project at the national level, including the creation of the KM Task Force, the adapted KM LDP, and the Net-Map exercise, are mixed. On the one hand, the LDP enabled the KM Task Force to create a mobilizing vision around which a team that had never worked together coalesced to achieve two out of its three desired measurable results. The Net-Map 0% 10% 20% 30% 40% 50% 60% 70% 80% HIV&Aids Unit TWGs RHU NAC MOH Academic_Inst Donors OPC Prof_Assoc INGOs HEU CHAM NGO media JUDICIARY POLICE FBO DAC CBO Health Facilities Dist_Health_Office Zone_Office Counselors VHC VAC CHWs beneficiaries Out degree In degree 34 workshops also enhanced the sense of shared purpose and concretely demonstrated the constraints to information flows from national through district to community levels, and the improvements in these flows after the introduction of the SMS network. The LDP is known to be highly successful in building leadership skills among intact teams, or teams that are already working together to achieve organizational and health goals. These “intact” teams work together post LDP as well. This adapted LDP lacked one of the important prerequisites for success – an intact team. Given that constraint, the results are even more impressive. It is probably too early to draw any conclusions about the success of the toolkits. The launch of the toolkits took time, not only to agree on toolkit content, but also to train managers from the MOH to maintain and update them. This left little time to publicize the toolkits and advocate for their use. It is questionable, however, if busy KM Task Force members can play the role of generating interest in the use of the toolkits. The designation of the Health Education Unit within the MOH to take over responsibility for updating and promulgating the toolkits is a promising sign. If the toolkits become an important vehicle for centralizing locally produced knowledge on FP/RH and HIV/AIDS, then they may very well serve an important purpose moving forward. The main intervention at national level during this project was the creation and publication of four online toolkits. The toolkits were not identified as an intervention that strengthened actors or links. The perception of the Net-Map participants was that the project was too short in duration to be able to properly market and publicize the toolkits to change information flows. The majority of the 18 months of the project was spent locating and organizing the information to build the toolkits, and then getting approval and launching them. Nevertheless, the participants felt that the toolkits have the potential to strengthen linkages between INGOs, NGOs, TWGs, and the Districts. Participants expressed the opinion that this would only be successful with a specific focused advertising effort via the media, for example, radio and newspapers. Participants were able to identify key stakeholders and links that could be strengthened if the project had been longer. The map below in Figure 8 highlights, using darkened links and red circles, the actors and links that might be strengthened given additional time in this project to publicize and therefore increase demand for the toolkits. Participants identified actors most likely to benefit from use of toolkits, including the National Aids Commission, CHAM and NGOS, as they often need the resources included in the toolkits to conduct their work and have no other shared central repository of information on HIV/AIDS. Likewise, the Reproductive Health Unit, CBOs and FBOs would benefit from joint access to resources in the three Family Planning toolkits. 35 Figure 8. Actors and links that could be strengthened with a longer project 36 6. FINDINGS AT THE DISTRICT LEVEL 6.1 Net mapping at the district level When we compare Net-Map findings to the LQAS findings, we find supporting evidence for claims in both research approaches. At district level, both district maps in 2010 were very sparse, with few actors and few linkages between actors, indicating a rather weak network of information flows around Family Planning and HIV Testing and Counseling. The baseline map from Salima in Figure 8 clearly shows the weakness of the network of information flows. Figure 9. 2010 Baseline Net-Map of Information Flows with District level Stakeholders in Salima Figure 10. 2011 Endline Net-Map of Information Flows with District level Stakeholders in Salima HIV Testing Information Family Planning Information 2010 Baseline Map- Salima 37 However, in 2011, the maps indicate a complex network of information flows, which the participants claim is a direct result of the K4Health DLC and SMS project activities. Figure 9 below from the endline Map in Salima demonstrates greatly strengthened information flows involving CHWs. When asked which specific links were strengthened by the mobile phones, the participants labeled the following links indicated by dark black lines in Figure 10: District Health Facility, Community Based Distribution Agents, Clinics, and Health Surveillance Assistants. Also, there was improved communication with the Reproductive Health Unit at National Level. Some of the discussions and findings from the Net-Map sessions are listed below, and the full details can be found in the Net-Map report. Net-map participants at district level claimed that communication was more direct, as the system was able to link Community Health Workers with each other, and with Health Facility based experts. This new line of communication increased information flow and was seen as having a direct impact on service provision. For instance, Community Health Workers took less time to restock their supplies. Therefore, it was reported that service provision had improved. Another critical aspect of the SMS impact was its use in reporting reproductive health and HIV/AIDS commodity stock outs (contraceptives, HTC kits, condoms) and requesting resupply. When participants were asked what they think may have been the reason for the differences between the maps, the consensus in Salima was that in the past year, the district had experienced a significant increase in communication and information exchange between DHO, CBDAs, HSAs, and Health Centers. This was mostly attributed to mobile phones which were seen as breaking geographic barriers. The Net-Map participants felt that through the use of mobile phones, the quality of health in Salima District has been improved. They claimed this was achieved in many ways. First, CHWs are now able to get immediate help for their clients by using mobile phones to send an SMS to the Hub and getting a rapid response. Second, they are able to send and receive important technical information with District Hospital staff through SMS. Before, there was no communication among District Health Facility staff and CHWs on information about FP and HIV/AIDs. Now, they are able to communicate about any and all information using the phones, which they reported has helped CHWs increase expertise in their field of work. With regard to the role of the District Learning Centers, participants in the Net-Map workshops agreed that the District Learning Center provides clear benefits in terms of access to information, timely reporting, and knowledge sharing. These themes were echoed and expanded upon in the focus group discussions at the district level. 6.2 Program Implementation of the District Learning Centers The K4Health project engaged the District Health Management Teams (DHMT) to advocate for the establishment of DLCs in Salima and Nkhotakota. The DHMT and K4Health teams facilitated the establishment of District Task Forces mandated to oversee the smooth implementation of the knowledge management project in the district, but also provided space for the establishment of the DLCs. Coordinated efforts between the District Health Officers (DHOs) and K4Health resulted in a furnished DLC with at least four computers and Internet service in each of the DLCs. The DHOs covered the monthly Internet cost to the service provider. Two DLC coordinators were recruited by K4Health to roll out the DLC intervention, and also to build the capacity of health officers responsible for managing the learning centers. Users of the DLCs were trained in how to access both online and offline K4health Malawi toolkits and other relevant websites. The Frontline SMS 38 Hub was also housed in the learning center. A register was developed and oriented to users of the DLC aimed at monitoring usage of the district learning center. 6.3 District Learning Center Results By June 2011, 6874 DLC visits were registered in Salima and Nkhotakota. Of the total, 3,165 were from Nkhotakota, while 3,709 were registered in Salima. The overall average number of visits to the DLCs per month was 529, translating to a daily average of 24 visits (See Figure 10 below for DLC visit trends). Access to the DLCs very much depended on the availability of Internet in the Learning Centers, such that DLC visits dropped in any month where the DLC experienced Internet interruptions. For example, in May and June of 2011, visits dropped because payment by the government to the Internet service providers was delayed, resulting in Internet interruptions. Figure 11. Trends in District Learning Center visits between June 2010 and 2011 Most of the visits to the DLC were from male health care providers and managers. Over the life of the project, 37% of visitors to the DLC were women. When asked in FGDs for the reasons for this gender disparity in DLC visits, the response was that women work mostly as nurses and are always busy in wards, and have little time free to find answers to technical questions or to use the Internet. It will be important moving forward to investigate further with women about their use of the DLCs and specify steps that would facilitate that use. Another reason, that women do not visit the DLC to the extent as men, is that their responsibilities in the home require them to depart as soon as their shift ends at the hospital. Figure 11 below shows the access to the DLC between September 2010 and June 2011 by gender. 39 Figure 12. Breakdown of DLC access by sex between September 2010 and June 2011 One other unintended but positive result of the DLC intervention was the provision of basic computer training to health workers. Most of the health worker at the two DHOs did not know how to use computers prior to this project. The project responded to this need by rolling out basic computer training to interested health workers. Over 100 health workers in Salima and Nkhotakota acquired basic computer skills through the Learning Centers. Benefits of DLCs From Focus Group Discussions with district managers, clinicians, and members of the District KM Task Force, a number of benefits of the DLCs were identified, including: their ability to address multiple needs, increased access to health information in general, more reliable clinical information to treat patients, improved reporting, better communication with community health workers through the SMS Hub, and an increase in the culture of information sharing. District health personnel reflect varied backgrounds. Some are dentists, some pharmacists, and some are responsible for areas other than HIV/AIDs and family planning such as Malaria. This variety of technical specialties influenced answers to questions about toolkit use, for example. The DLC addressed multiple needs District health personnel mentioned the many ways in which they used the DLCs. This ability to address district health personnel’s multiple needs for different types of information was one of the hallmarks of the DLC. People use the DLC to: • Consult Family Planning books • Research findings on family planning over the Internet • Write group SMS to HSAs and CBDAs about meetings • Receive SMS from HSAs and CBDAs who ask questions whenever they have problems • Send reports every quarter to the HIV Unit • Check mail 40 • Browse new information for presentations at meetings • Log onto Facebook to make professional connections • Download district maps for planning purposes • Take out IEC materials such as manuals • Consult the Toolkits • Seek out information on family planning policies and practices are not yet in the toolkits • Receive computer training • Locate hard copies of policies on reproductive health • Find information for a presentation • Download socio-demographic information useful for planning purposes Increased Access to health information overall Before the advent of the DLC, the District Health Officer had an agreement with an Internet café that the DHMT members could access Internet, and at the end of the month the DHO would pay the bill. However, in practice this system was not functional. One district health worker describes how difficult access was prior the DLC. “The DHO had an agreement with an internet café that the DHMT members could access internet and at the end of the month the DHO would pay the bill. We had to take a letter whenever we were going there from the District Health Officer, and most of the times she was not there. So we were ending up paying our own money. As a result we stopped and we were just using the information that we acquired from school without reading up to date information and it was very challenging with our job as we are always supposed to acquire up to date information, but with the DLC we are now able to look for up to date information on internet and help our clients accordingly’’. District Environmental Health Officer, Nkhotakota DHO More reliable clinical information available to treat patients Several District Task Force members mentioned that one of the advantages of the DLC was easy access to the toolkits. Perhaps because of their increased awareness of the toolkits as a resource, they considered them a more trustworthy and better source of policies and training manuals. A number of respondents, however, did not know about the toolkits and tended to use the Internet to locate clinical information to treat patients. They gave several examples of ways in which the DLC enabled them to make correct diagnoses, to use the right drug to treat a particular illness, or to find information on how to counsel a family planning patient. “A week ago I was discussing with my fellow clinicians about a certain patient, and we were not sure of the disease that he was suffering from. We argued for a while and then we decided to go to the DLC to look for information on the type of disease that the patient was suffering by looking at the signs and symptoms. We found the type of the disease and I treated the patient accordingly.” Clinician, Salima DHO 41 “I had a patient, a nine year old boy who had a heart failure, and we had a debate on what drug was safe on a heart failure case, after checking information on the internet at the DLC for internet, one clinician found that Captopril was not safe for a child, so I gave him another type of medicine which helped the child, and he is now recovering.” Clinician, Nkhotakota DHO Improved Reporting Although improved reporting was not one of the aims of the K4Health Project, it was nonetheless cited by a number of focus group participants as one of the advantages of having easy access to the DLC. Two district coordinators explained the difference: “We send reports to Kamuzu Central Hospital every month, and they have to be typed. Before the DLC we all had to give our reports to the DHO secretary to type for us, and there are many coordinators here and she was supposed to type all our reports apart from her day to day duties. Because of that, we were not able to send our reports to the central hospital on time since it was also taking a lot of time for the secretary to type the reports for us. But with the coming in of DLC we are able to type our reports because we all went for a Basic Computer Training and we are able to send our reports on time.” Family Planning Coordinator, Salima DHO “The DLC has enriched the HIMS, in the sense that, am able to get the reports from all the community health workers on time as they use FLSMS to send their reports. Before the DLC I had a lot of problems to get all the reports from different health facilities, it was either through a two-way radio or if someone from that area was coming to the DHO. All these ways were unreliable, and then I was not able to meet deadlines.” HMIS Coordinator, Salima DHO Better communication between CHWs, health facilities and district health officers District Health Coordinators and other personnel all commented on the difference the DLCs had made in improving communication with HSAs and CBDAs, as the SMS Hub is located in the DLC. “As a coordinator, the DLC has helped me a lot in terms of sending group messages to community health workers. Now I just go to the DLC and send a message to all the community [health workers] when we have review meetings and the turnout is always [better] than the time before we had the DLC.’’ Family Planning Coordinator, Salima DHO Promotes a culture of information sharing Several focus group participants commented on the creation of a culture of information sharing with the coming of the DLCs. Every Wednesday, for example, district managers have Continuing Professional Meetings, and whenever one gets new information on the Internet they share it with the other health workers at the meeting. Because everyone is able to access up to date information through the use of the DLC, one outcome is that at the District Health Office they now have a weekly meetings where everyone is asked to present on an interesting topic. 42 “Even in weekly meetings people are nowadays giving very good presentation with rich information that they get on [the] Internet at the DLC.’’ Clinician, Salima DHO Even other people from different organizations who deal with health issues are able to access the Internet using the DLC. This has helped the relationship between those organizations and the DHO. This is both a positive and a negative; since there aren’t enough computers with Internet connection, these other organizations can increase wait time to use the computers. “People from other organizations like, Banja La Mtsongolo, World Vision, Word Medical Fund and many other organizations come to the DLC to look for information on [the] Internet and this has improved our relationship with them since we interact with them when they come here”. Information and Education Communication Coordinator, Nkhotakota DHO “However, this has affected us negatively since when these other people from other organizations come to the DLC they occupy the space that could have been used by us (DHO health workers) so you may find that you want to look for a certain information and you find the other computers have been occupied by them.” Nurse, Nkhotakota DHO Challenges for DLCs As noted above, there were challenges for the DLCs including: insufficient equipment to meet demand; inconvenient hours, especially for women; lack of knowledge of users about the toolkits; and erratic Internet availability. Insufficient equipment to meet the demand In general, Internet and equipment maintenance issues are large challenges that will require sufficient resources to address in any future project. The challenge most frequently cited was the small number of computers linked to the Internet. Usually, there were four computers operating, but viruses and broken equipment could reduce this. Maintenance tended to be slow, and computers, printers, and photocopiers could be out of use for some time. Several health workers commented that it was only in the DLC that people are able to access wireless Internet, and that if the radius of the wireless Internet could be increased, it would reduce the need for equipment. Hours of the DLC are not convenient, especially for women The gender gap in use of the DLC was an issue that focus groups participants acknowledged. For example, in the maternity ward the nurses put on uniforms. When they want to leave the ward to visit the DLC, they have to change first. Women who work in the maternity ward expressed their discomfort with wearing the same maternity ward clothing to the DLC. They prefer to change. This additional time is a deterrent to visiting the DLC. The DLCs were also closed during lunch time when most workers can come, and were not open on Saturdays. Increased hours were cited as one way to increase use of the DLCs, especially by women. 43 Lack of knowledge about the toolkits inhibits their use Some district level providers mentioned the toolkits in a positive way: “To me toolkits are more trustworthy and better sources of policies and training manuals. This is because the information that I find in toolkits is proven and appropriate for Malawi’’. District Nursing Officer, Nkhotakota DHO However, most of the participants in the focus groups did not know about the toolkits. Several reasons were given for their low use: • People are working in areas other than HIV/AIDS and FP/RH do not access the toolkits because their work is not reflected in them. • The toolkits aren’t promoted. Several people suggest the District Learning Center Coordinator should attend the morning report meetings and explain to health workers about K4Health website especially to new staff members who don’t know about them. • TB information which is critical for HIV/AIDs patients isn’t in the toolkits nor is information on opportunistic infections which are what many clinicians are facing. Internet availability can be erratic There were a number of complaints about the fact that Internet connectivity could be erratic. Problems with delays in payment from the MOH, which in turn caused delays in payments by the Districts to the Internet provider, created additional delays in restoring connectivity. The lack of security resulted in the modem being stolen on one occasion. 6.4 Sustainability of District Learning Centers Sustainability for the district learning centers in Salima and Nkhotakota was incorporated into the project design and implementation. The startup coordinators worked with assigned DHO staff in the implementation of the DLC intervention. The DHOs identified MOH staff meeting certain criteria to coordinate activities in the DLCs. The new MOH DLC coordinators in both Salima and Nkhotakota have been mentored for at least four months by K4Health DLC coordinators in the management of the DLCs. Areas of emphasis in the mentorship period were the operation and management of the SMS program; Internet and toolkit access; analyzing SMS and DLC usage data; and, specifically for Salima, access to print resources in the DLC. The DHO have laid out plans to train the new DLC coordinators in basic maintenance and running of antivirus software for the computers to curb the challenge of computer problems due to viruses. The DHOs will also develop the capacity of the DLC coordinators to analyze and report data from the DLC. As part of the sustainability plan, the two DHOs committed to continue to meet the Internet subscription fees to the service providers. To sustain the SMS network, both Salima and Nkhotakota will buy enough airtime monthly for running the SMS Hub. In addition, Nkhotakota will continue to top up HSAs in the district with $2 worth of airtime to active members of the SMS network. This equates to 20 SMS messages per month or five minutes of air time. However, DHO Salima will not be giving any airtime to HSAs. Their argument is that if health workers use an average of at least K450.00 (US$2.50) to report and receive feedback from supervisors using public transport, it is clear that they will top up their own phones to communicate with supervisors at as low as K10.25 (US$ .06). This difference between the sustainability plans of the two districts will be important to follow in order to determine the minimum level of support necessary to run the program in any given 44 district in the future. The K4Health Project will contract a consultant in November 2011 to monitor this and other short term sustainability results. 6.5 Impact of K4Health Interventions at the District Level and Suggestions for Future Improvements In contrast to the mixed results of interventions at the national level, and despite the constant difficulties of maintaining equipment and Internet service in difficult to reach areas, the DLCs were clearly a success. Although the original desired measurable result chosen by the KM Task Force during the LDP was that 40% of the targeted workers in both districts were accessing information through K4H tools including DLCs, FLSMS, and Toolkits, by June 2011, that figure rose to 65%. Both the endline Net-Map and DLC focus group findings paint a picture of the strengthening of several key information flows between CHWs and their district coordinators, among district health personnel, and between the district and national levels in terms of strengthened reporting. District health workers were uniformly enthusiastic about the multiple uses of the DLC. Their only requests were that hours of operation be expanded in order to allow for more people – especially women – to participate in what the DLCs offered, and that the number of computers be increased to meet demand. 45 7. FINDINGS AT THE COMMUNITY LEVEL 7.1 Program Implementation Between June – October, 2010, the project trained and provided mobile phones and solar chargers to 253 health workers in both Salima and Nkhotakota districts. This represented 30% of all health workers in Salima and Nkhotakota combined. An additional 385 CHWs received phones during a second distribution in November 2010, bringing total SMS coverage to 77% of health workers in both districts. The project targeted the most remote health workers for the initial distribution of cell phones. At mid-term CHWs participating in the program in the two intervention districts lived on average 11.5 km from the nearest health center while non-users in the control district of Kasungu lived on average 13.9 km from the nearest health center. At endline, the CHWs participating in the intervention districts lived 12.25 km on average from the nearest health center, while the CHWs in the control district of Nkhatabay lived 23.5 km on average from the nearest health center. Because of the greater distances in Nkhatabay, CHWs had fewer clients and made fewer visits than in the intervention districts. The project used technology created by K4Health partner Frontline SMS, an initiative of the Kiwanja Foundation, which offered a software platform that enables organizations in remote and underserved areas to turn a laptop computer and a mobile phone into a mass communications Hub, enabling two-way SMS (short message service or text messaging) communications with large populations without the use of the Internet. It offers advantages over traditional voice services including reduced cost and the ability to send messages to large numbers of people in a short amount of time. The cost to send one SMS is approximately K10.25 versus traditional methods of public transport which on average cost K525.00. Figure 12 below shows how the SMS Network was structured. 46 Figure 13. SMS Network in Nkhotakota and Salima In a remote area, a Community Health Worker (CHW) or client has a question. The CHW sends a SMS to the Hub at district hospital, using a direct line or key word messaging, or to other CHWs in district. Hub The CHW receives an answer to their question via SMS from Hub, District Coordinator, or another CHW. The CHW can now make a more informed decision and provide case specific guidance to clients. SMS System Average time required to contact and receive feedback from the person providing technical support: 9 minutes Step 1: Client/CHW has a question Step 2: CHW sends Step 4: CHW receives question via SMS answer via SMS Step 5: CHW takes action District Coordinators/Supervisors receive the SMS message and respond at the Hub, from their personal phones if key word messaging was used, or another CHW responds directly. Step 3: District responds 47 FrontlineSMS is a software that allows the sending and receiving of text messages between organizations and their stakeholders via laptops or personal computers and an attached mobile modem/phone. In the case of the K4Health Malawi Project, key stakeholders included CHWs who were linked to the District Hospitals and their supervisors. Mobile phones were distributed to 630 CHWs in Salima and Nkhotakota districts. Subsequently these CHWs received training in sending and retrieving text messages, use of key words and auto reply functions of the SMS system. CHWs were placed in groups – either geographically or by theme – for single messages to be broadcast to entire groups from anywhere with a mobile phone signal. An auto-reply feature also allowed set messages to be sent out automatically to all mobile phones, triggered by keywords. This functionality works on any type of mobile phone, from simple to smart phone. For instance a Community Health Worker can send a particular message to a Family Planning Coordinator using the SMS Hub programmed with a particular key word to recognize the coordinator. Once the Hub received this message it recognized the keyword and auto-forwarded it to the mobile phone of the coordinator. Although CHWs were provided with units for air time, district coordinators had to use their own units or they were free to go to the Hub, which the project was topping up, to respond. mLearning pilot After successfully implementing the SMS network, the Knowledge for Health Project identified other possible avenues for using the phones to maximize benefits in terms of technical knowledge exchange. This came from a finding early on in the project that only a limited number of Community Health Workers were using knowledge pull factors, i.e., demanding technical information. In view of this, the project used some push factor mechanisms to promote knowledge exchange. One such methodology was to initiate structured learning through mobile phones or “mLearning”. This involved a sizeable target group with a reasonable amount of content material. The process involved knowledge gap identification through a Question and Answer approach, content development using approved training manuals and customization for SMS, weekly questions and information tips to Community Health Workers on various subjects and evaluation questions. The project posed Frequently Asked Questions and later used follow-up questions to narrow down the gaps into specific areas. Some examples of question are listed below: HSAs: Depo-Provera (DMPA) Mention two advantages of using DMPA? • Long acting (3 months) • Completely reversible (an average of 9 months delay in return to fertility after discontinuing DMPA) How long does it take for one to return to fertility after discontinuing DMPA? • 9-12 months What are the common side effects that most women experience after using DMPA? • Continuous bleeding for some women • May cause amenorrhea in some women • Increase appetite causing weight gain for some women • DMPA does not provide protection against STIs/HIV • Since DMPA is long acting, it cannot easily be discontinued or removed from the body if pregnancy is desired 48 • Spotting in some women Does DMPA provide protection to women against STIs/HIV? • No Is it true that DMPA cause amenorrhea in some women? • Yes it’s true CBDAs (Pills) Mention two advantages of using pills? • Do not cause a breastfeeding woman’s milk to dry up • Reduce menstrual pain • Reduce vaginal bleeding and thus help women to have enough blood • Does not interfere with sexual intercourse Must the pills be taken every day or only the day a woman wants to have sexual intercourse? • Yes, they must be taken every day whether or not a woman has sex that day 7.2 Results of K4Health Interventions at the Community Level Table 7 provides a summary of eleven key project indicators from the pooled data (all of the supervision areas together). 49 Table 7. Key Project Indicators at Mid-Term and Endline Indicator Salima and Nkhotakota Control district Mid-term (Kasungu) n=95 Endline (Nkhatabay) n=76 SMS Users – Mid-Term (n=95) SMS Users Endline (n=76) 1. Average number of SMS messages sent per CHW per month (from LQAS data) Average of 5 messages per CHW per month. Range (1 - 15) Average of 6.17 messages per CHW per month. Range (3-25) NA 2. % of participating CHWs who sent at least 5 SMS messages per month (from LQAS data) 44% 62.5% NA 3. Average frequency of SMS use per month for all FLSMS users (from Hub statistics) Hub data: 444 messages/month (June-October 2010) Hub data: 1091 messages/month (November 2010-May 2011) NA 4. Reported use of SMS disaggregated by purpose Stock outs (n=84) 88% Emergencies (n=61) 64% Information (n=61) 64% Meetings (n=27) 28% Tech support (n=19) 20% Stock outs (n=128) 84% Information (n=58) 38% Meetings (n=27) 28% Referrals (n=41) 27% Tech support (n=39) 26% Emergencies (n=34) 22% Dispelling myths (n=5) 3% NA 5. Average time required for providers to report important events (stock outs, transportation breakdowns) 2 Minutes 3 minutes Not asked at Mid-term Nkhatabay=523 minutes (8.7 hours) 6. Average time required to contact and receive feedback from the person providing technical support 9 minutes 9 minutes Kasungu=1681 minutes (28.0 hours) Nkhatabay=1498 minutes (25.0 hours) 7. Average time required for providers to receive feedback on important questions (e.g. specific medical conditions, effects of contraceptive uses, dosage amounts) 12 Minutes 35 minutes Not asked at Mid-term Nkhatabay=975 minutes (16.3 hours) 8. Average number of stock outs reported per month, per health worker: 1 stock out /month/CHW 0.3 stock outs (less than one) /month/CHW NA 9. Average number of client referrals reported per month, per health worker: 16 referrals/month/CHW 6 referrals/month/CHW Kasungu=31 Nkhatabay=8 10. Average no of clients visited per CHW per month 89 clients 74 clients Kasungu= 142 Nkhatabay=30 11. Average amount of time saved by using SMS over walking or taking transportation to nearest health center to report and receive support 1436 minutes (24 hours) (1445 for non-users in the control district vs. 9 for SMS users) 1741 minutes (29.0 hours) (1750 minutes for non-users in the control district vs. 9 minutes for SMS users) NA 50 As can be seen in Table 7, the average number of SMS messages sent to the Hub per CHW per month rose from 5.0 per month at mid-term to 6.2 at endline, as measured by the LQAS survey of cell phone users. As measured by Hub statistics, the total number of messages sent by CHWs rose from 44 messages per month between June and October 2010 to 1,091 messages per month between November 10, 2010 and June 11, 2011.The percent of health workers reporting on the LQAS survey that they sent an average of at least 5 messages per month increased from 44% at mid-term to 62.5% at endline, as training and mobilization of CHWs in the use of the SMS network intensified. Emergencies as a percentage of overall reasons for using the phone dropped from 64% at mid-term to 22% at endline. This could be due to several reasons. The period of the mid-term assessment in October was the rainy season when the disease burden in the communities is higher. Another possible reason was that the SMS network had resulted in improved distribution of supplies and drugs to communities. 7.2.1 Time Savings One of the most important outcomes of the SMS network was the reduction in time for CHWs to report important events and receive technical support from their supervisors. At endline, the average time required for CHWs to report important events (stock outs, transportation breakdowns) was 3 minutes, compared to 523 minutes for the control district of Nkhatabay during the same time period. When asked how much time on average it took to contact and receive feedback from a supervisor or person providing technical support, CHWs participating in the SMS network reported it took 9 minutes, compared with 1,681 minutes (1.2 days) for CHWs in Kasungu District at mid-term and 1,498 minutes (1 day) for CHWs in Nkhatabay at endline. The average time required for providers to receive feedback on important questions (e.g. specific medical conditions, effects of contraceptive uses, dosage amounts) was not in the mid-term survey of non-users, but in the control district at endline this figure was 975 minutes (16.3 hours). For SMS users in the two intervention districts, feedback on important questions took 3 minutes at mid-term and 35 minutes at endline. The increase in time at endline was due to a doubling of the number of CHWs with cell phones, and also the fact that the district coordinators were not given phone units to be able to respond to the greater volume of requests. Initially, the coordinators were responding immediately from their personal phones, but with the increase of additional 50% of mobile phones in the hands of CHWs at mid-term, the volume of incoming messages started to affect their personal airtime costs substantially. Thus, they preferred to visit the DLC and use the Hub to respond to the questions. 7.2.2 Stock-outs The number of reported stock outs declined between mid-term and endline from 1 per month per CHW on average, to 0.3 per month per CHW, which is most likely due to the successful resolution of earlier stock-out issues. However, stock-outs still remained an important issue in December 2010, as seen in Figure 13. 51 Figure 14. Trends in Stock-out Messages 7.2.3 Referrals On average the number of referrals/month/CHW declined between mid-term and endline from 16 referrals per month per CHW to 6. There could be several underlying reasons for the decrease. CHWs reported that they felt more confident to handle cases on their own with cell phone back up so that they didn’t need to make as many referrals as they had without the cell phones. 7.2.4 Cost Savings On average it takes about 9 minutes for health workers on SMS to send and receive feedback from supervisors at an average cost of K86.50 (US$0.48). For non SMS users in the control district, it takes an average of 1,750 minutes (1.2 days) to report and receive feedback to queries made to their supervisors at an average cost of K551.52 (US$3.06). This implies that health workers using the SMS network make an average saving of 1,741 minutes (1.2 days) and K465.02 (US$2.58) Kwacha per contact with their supervisors. By end of June, 2011 the Hub had received and sent over 10,319 messages to and from Community Health Workers in Salima and Nkhotakota. The average number of messages sent from a health worker to and from the Hub at baseline was five. At the time the endline evaluation was conducted, the average numbers of messages sent to and from the Hub per health worker increased to 6. The most common reasons for health workers exchange of messages listed in order of frequency are technical notifications, technical reports, mobile learning questions, stock outs, technical inquiries and e-learning answers. Figure 14 provides a breakdown of messages sent and received by category. 52 Figure 15. Breakdown of SMS messages by category between June 2010 and June 2011 The frequency of messages sent and received fell between April and June, as seen in Figure 15 below. This was due to interruptions in service in Salima during the process of transitioning to a reverse billing system towards the end of the project. This was done to allow usage of air time to be charged to a single Hub number, rather than to each individual user’s number. The idea, which was aimed at cost effectiveness and minimal interruptions due to lack of airtime, required signing a contract with the mobile service provider to finalize the transaction. K4Health was not in a position to sign the contract, given the scale down of support and departure planned for June 30, 2011. Negotiations have begun to have the MOH sign the contract. However, this created a delay, and CHWs in Salima did not get a monthly top up of airtime for sending messages. 7.3 Results for SMS Users by Supervision Area Table 8 summarizes the four key project indicators by supervision area (SA) and provides information that is helpful for project implementation. The table displays the target for each indicator and whether or not (yes/no) this target was reached in each of the eight SAs in the intervention zone. This information allows us to compare the SAs for each indicator and identify areas with weaker levels of implementation where the project should likely direct time and resources. The targets and decision rules in the table were determined by the average amounts calculated for each indicator from the pooled LQAS data at baseline. In some cases the desired coverage rates for the decision rule were increased in order to differentiate better between the SAs in terms of performance. All of our targets were determined as we designed the M&E for this project. Unlike other established public health sectors, mHealth interventions are relatively new and proven indicators do not exist. These indicators and targets were thus determined based on averages we were seeing in SMS traffic on the Hub on a monthly and daily basis. For example, in the first indicator, SMS Message Sent Rate, we would hope to see that 75% of all CHWs send at least 5 messages in any given month. Therefore, our target was 75% for this indicator. The “Decision rule” correlates directly to predetermined LQAS methodologies around decision rules, where sample size is correlated to benchmarks. So, with a target of 75%, the decision rule for a sample of 19 within a supervision area is 12. In other words, 12 people of 53 the 19 who are interviewed have to state that “YES” they have sent at least 5 messages in the last month in order to pass. If not all 12 say yes, then that supervision area fails. In the case of the first indicator, and NK South, we see that only 10 people said yes, they sent at least 5 messages, so that particular supervision area fails. LQAS is useful in this sense, as it tells us there may be a problem in this supervision area that needs further investigation to find out why CHWs are not sending at least 5 messages per month. The overall rate is an average of all supervision area respondents for this indicator. The overall rate is reached by calculating the following: Numerator: The sum of all people who said “YES” to the question about 5 or more messages sent per month = 95 Denominator: Number of supervision areas (8) times the number of people surveyed in each supervision area (19) = 152 Dividing 95 by 152 gives us 63% Therefore 63% is the average percentage of all respondents across all 8 supervision areas that said yes they have sent at least 5 messages in the last month. Given our target of 75%, it indicates that while we didn’t reach the target of 75%, we were close, and if the district used this information, and looked into the specific supervision areas that failed, they may be able to improve and reach the target during the next survey. The indicator that is most problematic for all but two of the SAs is the Timely Feedback Rate, defined as the amount of time on average it takes a CHW to receive feedback on an important issue. Here, the overall rate for achieving the target value was 36%. There may be several reasons for this. Between the mid-term and the endline, the average time for feedback on important events rose from 12 minutes to 35 minutes. Although this is a relatively small increase when compared to the average time for feedback in the control district of 975 minutes, it will need to be followed to make sure that an undue burden isn’t being placed on district coordinators to respond quickly to questions from CHWs, especially if District Coordinators are not receiving top-ups for the air time it takes to respond to queries. Perhaps the limit of 12 minutes to respond is too short for supervisors. Or perhaps when looking into the districts specifically, there is something wrong in Salima Khomeza which had only 1 person out of 19 say that they received timely feedback. It would require further investigation to find out what exactly is happening in this supervision area that is causing a delay in feedback. 54 Table 8. Summary Results of K4Health Key Indicators from LQAS Endline Survey by Supervision Area Indicating if Target Value is Reached (Yes) or not Reached (No) Indicators Supervision Area Overall Rate NK South NK Central NK North Central NK North Salima Kaphatenga Salima Katelera Salima Khomeza Salima Mafco 1. SMS Message Sent Rate (% of CHWs who sent at least 5 messages in the last month Target = 75% Decision rule= 12/19 10 No 12 Yes 11 No 14 Yes 9 No 13 Yes 13 Yes 13 Yes 63% 2. SMS Message Receive Rate (% of CHWs who received at least 5 messages in the past month) Target = 80% Decision rule= 13/19 15 Yes 15 Yes 16 Yes 16 Yes 13 Yes 13 Yes 14 Yes 12 No 75% 3. CHW Timely Report Rate (% of CHWs who reported important events within 2 minutes) Target = 75% Decision rule= 12/19 15 Yes 10 No 11 No 11 No 13 Yes 11 No 9 No 10 No 59% 4. CHW Timely Feedback Rate (% of CHWs who received feedback within 12 minutes on important events) Target = 60% Decision rule= 9/19 8 No 7 No 11 Yes 3 No 9 Yes 6 No 1 No 9 Yes 36% 55 Benefits of the SMS Network The focus groups and key informant interviews with SMS users and District Coordinators identified several benefits of the SMS network for care. Increased Self-Confidence When HSAs and CBDAs have a complicated case at hand and they have asked for support at the Hub, the response that they get from the district coordinators gives them courage and confidence to manage the case, because it has come from a specialist, and they are sure they are doing the right thing. “Before the knowledge for Health Project I was not confident enough to help my clients because I was not sure when answering some of the technical questions that some of my clients were asking me. But with the phones that we received from K4Health, I am able to help my clients with the firsthand information from the coordinator, and the phone has also helped me to know a lot of things that were giving me problems when asked by my clients.’’ HSA, Nkhotakota, Chididi Health Facility Cost savings HIV Positive people were getting ARVs at the district hospital and clients were spending their own money on transport and food when going to pick up ARVs. With the phones, CBDAs and HSAs started lobbying with the District Hospital to decentralize the system such that currently ART clinics are being conducted closer by at the health centers rather than at the District Hospital. In Salima District the distance from the health center to the CBDA located at the farthest point is about 20 km. To report important events or seek technical support CHWs used to hire a bicycle costing k200, but with the phones, travel is not always necessary. They just send an SMS, which costs only k10, which is significantly lower than transport costs. Again, they are not traveling to the district hospital as frequently as they used to, because most of the information is done through either messaging or calling. As one CBDA reported, “I used the phone that I received from MSH to send an SMS to the Hub when I had stock outs of DMPA and the Family Planning Coordinator answered me through an SMS as well that she will send the DMPA with the driver who was coming to our health center the same day. I only used K10.25 (US$.06) to get the DMPA using the phone, but without it, I could have used K1,000.00 (US$ 5.56) for transport money.” CBDA, Salima, Changunda Heatlh Facility An HSA also confirmed the financial savings for clients in relation to reduced referrals: “The phones are also helping the clients to save their money. This is because in the past before this project, I was making a lot of referrals since I was not able to communicate with the Family Planning Coordinator when asked some technical questions by my clients, and most of my clients were spending a lot of money on transport since from my catchment area to the DHO is K800.00. But with the coming in of this project I am able to answer some of the questions that my clients ask me since I just send an SMS to the Coordinator and she always give(s) me the answers immediately.” HSA, Salima, Changunda Heatlh Facility 56 Decreased stock-outs When HSAs and CBDAs have run out of supplies like condoms, they know that when they send a message to the Hub they will be supplied, sometimes even on the same day. In the past, it could take up to a week. They can also use the phones to get supplies from each other. “At first it was taking us weeks without having the supplies whenever we had stock outs, but with the coming in of this project we are able to get all the supplies that we want in time, because now it’s just a matter of sending an SMS to the supervisor and if he does not have the supplies we send it to the Hub and we get the response immediately from the Family Planning Coordinator that we can go and collect some. Sometimes the Coordinator also send the supplies to us when the Ambulance is coming to our Health Facility or when one of us (HSA or CBDA) went to the DHO (District Health Office).’’ CBDA, Nkhotakota, Chididi Health Facility One HSA related how the decreased stock-out time improved continuity in family planning commodities. “In the past before we received the phones, there was a big problem for us to have supplies whenever we had stock outs. This was due to the lack of communication that was there between us and the DHO health workers, it even came to the point that my clients knew that most of the times I was not having supplies and they even didn’t bother to come to me to ask for supplies. That made a lot of women who were practicing different family planning methods to stop since it is very expensive for them to travel to the DHO to collect condoms and pills. But with the phones that we received our lives are now simple, because our clients know that we now have supplies whenever we want them since we just send an SMS to the coordinators at the DHO.’’ HSA Salima, Changunda Health Facility Increased ability to respond to emergencies In addition to having a significant impact on avoiding and resolving stockouts, the Frontline SMS application helped the District Hospital to act on a Meningitis Outbreak. On October 9, 2010 a message was sent by a Health Surveillance Assistant based at Ngala Health Center announcing a suspected outbreak of Meningitis in his catchment area which had claimed the life of two men and one woman in just two days. Although it was past working hours when the message was received by the computer Hub, it was processed and immediately forwarded to the District Environmental Health Officer, who immediately mobilized a taskforce on Meningitis. The taskforce left for the site of the outbreak first thing the following day to start responding. On April 6, 2011, an HSA went to a Group Village Headman on a monthly supervision visit. During the visit the HSA visited a Faith‐ Based Organization (FBO) which provides home‐based care (HBC) to HIV/AIDS affected patients and households. During the visit, the HSA discovered that the FBO had no supplies and patients were struggling to get drugs for opportunistic infections. He sent a message to the HBC Coordinator through the Hub: “St. Cipriano, an FBO in my area has no HBC drugs.” In reply, the community HBC responded, “I will visit the FBO on April 9th.” The coordinator visited the FBO and brought with her basic drugs and supplies from the DHO. 57 Increased Trust The phones have also increased trust between clients and CBDAs because errors have been reduced. Now, CBDAs always get support from district coordinators on the spot; they no longer wait for the monthly review meetings, which seldom provided time for individual questions about cases. “My clients trust me very much nowadays than before I received the phone because they know that everything that I advise them comes from the District Family Planning Coordinator and not from me as a CBDA since they know that I always communicate with the Coordinator using the phone that I received.’’ HSA, Salima, Changunda Health Facility Another HSA described how the phones allow for more targeted technical support: “These phones have also helped us to communicate with other coordinators of different departments who are helping us to help the people who are suffering from other diseases rather than family planning issues. For example, this other time, I asked the Malaria Coordinator on how I could have treated a person who was taking Artemether/Lumefantrine (an artemisinin based antimalarial), and there was no change after taking the whole dosage.’’ HSA, Nkhotakota, Chididi Health Facility Widening service coverage Reduced travel time is giving the HSAs and CBDAs more time to visit the communities they serve. Although the phones are only a contributing factor, one Family Planning Coordinator reported that the phones have contributed to an increase in Contraceptive Prevalence Rate, which went from 21% in 2007 to 34% in 2010. Participants reported that the phones have motivated HSAs and CBDAs to work even harder and reach a bigger community. “The phones have also helped us to have a lot of clients because with the trust that the people in our communities have on us, they are able to encourage one another not to go to the DHO whenever they have problems but rather to see us first and then go to the DHO only when they have been referred by us.’’ CBD, Nkhotakota, Chididi Health Facility Increased Efficiency of referrals In the past, HSAs and CBDAs just referred patients to the hospital without knowing whether the specialist was there or not. With the phones, referrals are only made upon contacting the specialist, who now knows that there is a client coming. This means that clients get treatment more quickly. “With the coming in of this system, a lot of people go to the hospital when they have been referred to and they get the help that they are supposed to receive because we are able to make follow ups with the DHO health workers using the phones that we received. And this has made a lot of people in our catchment areas to come to us first before they go to the DHO because they know that when we refer them they get the help they require since we make follow ups.’’ HSA, Salima, Changunda Health Facility 58 mLearning as a spur to increasing Community Health Worker knowledge When they received a question on their phones during the mLearning pilot, HSAs and CBDAs were required to read manuals in order to give the right answer, thereby reviewing and increasing their knowledge. This process also improved relationships between CBDAs and HSAs, as they were able to ask each other for answers. mLearning has encouraged supervisors to conduct supervision more often, since they know which HSAs and CBDAs have information needs. “There was this other time when we were receiving different questions from the Hub about myths and side effects. Those questions have helped us a lot because they made us to go back to our manuals and read what we were trained on such issues, and we were able to send the answers to the Hub as well. That really helped us to remember some of the things that we forgot since some of us were trained some years ago.’’ CBDA, Salima, Changunda Health Facility One CBDA reported that he himself had been giving out incorrect information: “The questions also helped us to know other things about different myths that we were not sure of, because some of us we did not know the right answers and we were sending wrong answers to the Hub. But after some days we received the right answers from the Hub for the questions that we received.’’ CBDA Nkhotakota, Chididi Health Facility Challenges in Implementing the SMS Network District Program Coordinators identified a number of additional benefits of the Frontline SMS network: • CHWs are able to report even when fuel isn’t available, which is important, as fuel shortages are becoming more and more frequent. • Use of key words routes messages to the appropriate coordinators • Improved FP coverage, as problems women have are solved immediately They also identified challenges: • Malfunctioning of the Hub and network led to delayed responses. • The phones sometimes failed due to dead batteries. This could be solved by providing solar chargers, but not all CHWs had been given solar chargers by June 30th due to delays in procurement. • Sometimes feedback from supervisors is delayed, because they have to use their own airtime to communicate with CHWs. • The SMS network has reduced physical contact between colleagues thereby weakening social interaction. • CBDAs reported not being able to send messages in the months of May and June. SMS usage levels in Salima faced a setback in May and June due to the initiative to change the system into a closed user group and reverse billing system. 59 7.4 Impact of K4Health Interventions at the Community Level and Suggestions for Future Improvements Findings from the LQAS survey and focus groups are corroborated by results of the district level Net-Map in demonstrating the success of the K4Health SMS Network in improving information flows to a long neglected but extremely important segment of the Malawi national health system – the Community Health Worker. This is the extra mile for information flows; CHWs experience an information gap that is difficult to close, but is perhaps the most important intervention for directly impacting service delivery. Although the third desired measurable result of the KM Task Force had been a 60% increase in the number of health workers at the community level in Salima and Nkotakota who are accessing comprehensive accurate up-to-date information through K4Health tools, by June 2011 that figure stood at 77%. Self-reports from CHWs and evidence from Hub statistics document their beliefs that the SMS network was able to increase CHW self-confidence; increase trust between CHWs and the communities they serve; decrease stock-outs; decrease the cost of communication; widen service coverage; and increase the efficiency of referrals. This is a remarkable achievement in a project of such short duration. What is clear from the results, however, is that the success of the SMS intervention at the community level would not have been possible without the Hub and the management system that had been set up at the district level, as well as without the training and support that went into launching the project. It is not enough merely to hand out cell phones to Community Health Workers without setting scaffolding in place at the district level. The management system that was established enabled the project to approach the use of cell phones in a more systematic and strategic manner. Without any structures or processes, such as the Hub software for data collection, the training of how to send SMS, and the coaching of supervisors on the importance of responding in a timely fashion, the phones alone probably would not have been used to the same extent. 60 8. CONCLUSIONS The goal of this evaluation is to analyze the planning, implementation, and outcomes of the K4Health Malawi Project, and make recommendations for similar knowledge exchange health programs in the future. Based on the results achieved by the eighteen month demonstration project detailed in this report, we can respond to the five research questions posed at the beginning of the study: 1. How successful were the approaches at the national and district levels to mobilize stakeholders around a common vision for improving health information access and use in Malawi? The implementation of K4Health Project interventions required mobilizing stakeholders from various parts of the MOH in Malawi, most of which had not previously worked together as a team to achieve a common goal. The main strategy used at the national and district levels to create an effective Knowledge Management Task Force was to adapt MSH’s flagship Leadership Development Program to mobilize and inspire these stakeholders around a common vision. The common vision they chose was to see “happier, healthier, and more productive individuals, families, and communities as a result of better access to and use of FP/RH and HIV/AIDS information.” This vision served as the core of the K4Health project and as the core to the prioritized activities carried out through the LDP action plans. The success of the LDP in mobilizing the KM Task Force is clearer from the success of the team in achieving its desired measurable results at the district and community levels than it is at the national level. The first desired result for use of the toolkits by national level managers was unfortunately not measurable, but the statistics produced by Google Analytics do not paint a picture of widespread use of the toolkits in Malawi as yet. The Net-Map workshops also enhanced the sense of shared purpose and concretely demonstrated the constraints to information flows from national through district to community levels and the improvement in these flows after the introduction of the SMS network. The designation of the Health Education Unit within the MOH to take over responsibility for updating and promulgating the toolkits is a promising sign that the results for the toolkits may improve with more time. The lesson from the K4Health Malawi Project about the strategy of using the LDP is that it was a necessary and important step in mobilizing key stakeholders around a common vision for improving access to health information but probably not a sufficient strategy to engage national level managers in actually promoting the use of the toolkits. The district level managers and community health workers surveyed for this evaluation, on the other hand, were nearly unanimous in their enthusiasm for the DLCs and the SMS network. If the proof of successful mobilization of stakeholders is in the results achieved, then the biggest return on investment occurred from the LDP at the district and community levels. 2. How successful were the Malawi toolkits on RH/FP and HIV/AIDS and the other resources available to health care providers through the District Learning Centers in increasing access to and use of health information to improve services? Results from the Focus groups and District level Net-Maps demonstrated qualitatively the achievement of the DLCs in improving access to and use of health information to improve services. Although the original desired measurable result chosen by the KM Task Force during the LDP was that 40% of the targeted workers in both districts were accessing information through K4Health 61 tools, by June 2011, that figure had risen to 65%. However, it is not clear that the toolkits played a significant role in resolving the kinds of issues that district level managers and clinicians face in their day to day work. Although the toolkits were praised as being more trustworthy as resources and Malawi specific, they are set up to provide static sources of information such as manuals, guides and policies that are more appropriate for researchers or students just learning about health care in Malawi. What front line health managers and workers described in this evaluation as the benefit of the DLC was access to information on a need to know basis to resolve specific problems they were encountering with patients or in managing services. The DLCs also contributed to a culture of knowledge sharing, once again around problems that users could research on the Internet and present at district meetings. One difficulty with assessing the benefit of the toolkits was the barrier the project encountered in monitoring toolkit use. Because they had limited time to visit the DLC, users were not easily persuaded to fill out a survey on how they used the DLC and which toolkits they might have accessed. Users also did not have individual user names and so never logged in or out of the computer when their sessions ended, so that a new user would simply pick up where another user left off. Data on number of hits to the toolkit, therefore, are underestimates, since only a single IP address is counted, not multiple users from the same IP address. It is unlikely that future KM projects can easily resolve the monitoring issue. Issues such as erratic Internet connectivity, computer viruses, and insufficient hardware to meet the demand limited the effectiveness of the DLCs in providing uninterrupted access to health information. However, these kinds of issues should lessen over time with improvements in the availability of information technology in general in Malawi. Of more concern is the gender gap in the greater use of DLCs by male as opposed to female health workers. The gender gap does not seem to be caused by lesser demand for information by female workers, but rather by issues such as the hours the centers are open and women’s responsibilities for childcare and family. A concerted effort to make DLC access more ‘women friendly’ could probably lessen the gap. 3. To what extent did the SMS network in Salima and Nkhotakota reduce the communication gap between health workers and their district teams and thereby increased access to technical information among these health workers? The answer to this research question is clearly “to a great extent.” The striking differences between the baseline and endline district Net-Maps in terms of improvements in information flows between CHWs, district coordinators and health centers bears witness to the positive changes achieved in a very short time by the SMS network. The LQAS surveys at mid-term and endline also attest to the reduction in response time to important queries from HSAs and CBDAs to their technical supervisors and the cost savings achieved. The ability of the K4Health Malawi Project to provide 77% of the CHWs in both intervention districts with cell phones and to successfully train them in their use is a remarkable outcome in such a short time frame. Even the very small mLearning pilot successfully engaged CHWs in looking for answers to technical questions in their learning guides and from each other. District coordinators who were skeptical at first about what the SMS network could achieve and concerned about the burden it might place on them to respond, were won over in the end by the resulting improvement in communication and the increased attendance at meetings and other learning events. 62 The most important shift that occurred due to the SMS system was that instead of being only occasional recipients of information from the district level, CHWs became initiators of information flows through their stock-out messages and clinical questions. This is perhaps the greatest legacy of this project in a predominantly rural country where CHWs serve as a crucial link with clients. This is not to say that the SMS system didn’t encounter problems. Start-up issues with getting everyone solar chargers and setting up the keyword system to rout all messages through the Hub to the appropriate coordinators, were learning experiences in an area where there was no previous experience to guide the K4Health Malawi Project. In its role as a demonstration project with a mandate to test out new approaches, this was to be expected. A more daunting task may be to persuade the districts that it is their best interest to financially support the SMS network going forward. Because some of the cost of communication was born by CHWs prior to the coming of the SMS network, the districts may not feel the cost savings as acutely as if they had previously paid for all use of motor bikes or public transportation. CHW time wasted in making long trips to the district hospital was also never factored into district budgets prior to K4Health. Salima has elected not to pay for cell phone top ups to CHWs after K4Health project support ended in June 2011 but Nkhotakota has. This natural variation between the two districts should provide useful information about how much districts need to commit financially for the SMS network to be a success. 4. To what extent did the SMS network in Salima and Nkhotakota improve the ability of health workers to provide quality services and care? Self-reports from CHWs document that the SMS network was able to increase CHW self￾confidence; increase trust between CHWs and the communities they serve; decrease stock-outs; decrease the cost of communication; widen service coverage; and increase the efficiency of referrals. Anecdotal evidence was collected about the effects of avoiding stock-outs and possessing accurate information on service delivery by HSAs and CBDAs. These stories are compelling, and in a demonstration project such as K4Health they are more than sufficient evidence of the success of the approaches used. However, future KM projects will need to provide evidence of the effects of access to an SMS system on services and health outcomes. This will not be easy to achieve, but creative ways that do not unduly burden CHWs need to be found to make that link. The K4Health Project was not designed to rigorously test the link between increasing information flows to and from CHWs and increasing services and quality of care. The link between increasing information flows and provider knowledge and behavior was also not possible to make. To test both of these links would have required greatly expanded M&E tools, including provider observation or knowledge testing through surveys. Because literacy tends to be low among CBDAs in particular, traditional provider assessment tools may not be appropriate. Districts have also expressed an interest in knowing which CHWs are using the SMS system effectively in order to increase investment in their preparation. Using the system effectively can be defined by the number of messages sent and client questions resolved or it can be measured through direct indicators of services delivered in a particular catchment area. Under the recommendations section we will explore possible solutions to this M&E challenge. 63 5. How has the project built ownership within the Ministry of Health (MOH) to master key knowledge management processes promoted by K4Health and to what extent can the MOH carry on project interventions? The K4Health Project built in sustainability measures from the beginning of the project in the form of obtaining stakeholder buy-in through the LDP and Net-Mapping: providing manuals and guides to all processes used in the project; and assuring that MOH personnel were trained to take over implementation of the K4Health interventions after the project ended. However, not all of these sustainability measures were successful. For instance a full year before the end of the project, the MOH identified personnel who could shadow the District DLC Coordinators who had been paid staff of the K4Health Project so that there would be overlap. Although staff were identified and trained, the MOH later decided that those particular individuals should be placed elsewhere. This meant that there was a very brief period at the end of the project to bring on board and train new MOH staff. It is not unexpected that K4Health priorities would have to compete with other MOH priorities in mainstreaming knowledge management into MOH structure and processes, but this competition does mean that an 18 month project such as this is less likely to be able to both demonstrate results and assure their sustainability in such a short time. The K4Health Project has built in a 5 months extended period after its official end in June 2011 for the national and district level KM Task Forces to determine how they want to proceed to maintain or not K4Health interventions. In July the two districts and the National Task Force completed challenge models based on those they learned to create in the LDP to define their desired measurable results for 6 months after the end of the project. The K4Health Project team will not provide any additional technical assistance to district or national KM Task Forces unless expressly requested by the teams themselves as part of their action plans. In November 2011 K4Health Malawi will contract a consultant to follow up on the action plans and document their progress as well as sustained leadership or management effects of the LDP. Although even 6 months may not be long enough to guarantee sustainability, this assessment should provide an early indication of longer term success. 6. Other Lessons Learned One of the most important lessons to be drawn from this multilevel intervention is that its effectiveness depended on the fact that it was delivered as a package to national, district and community levels, and each level made a particular contribution to improving health information flows. Without national level leadership and buy-in, especially of the RH/FP and HIV/AIDS Task Forces and the Health Education Unit of the MOH, the project could not have obtained country ownership and its best chance at long term sustainability. The District level served as an important node in information flow between the national and community levels, as could be seen from the Net-Maps. Without interventions at the district level, including district participation in the LDP, the DLCs and the role of the DLCs in serving as the Hub for the SMS network, the project would also not have been successful. Finally, without the cell phones and SMS system for community health workers in remote, underserved areas to begin to have ownership over their own information needs, the national and district level interventions would have been far more limited. 64 9. RECOMMENDATIONS Based on the qualitative and quantitative findings documented in this report, the Malawi K4Health Evaluation Team offers the following five recommendations for future Knowledge for Health Programs: 1. Weigh the effectiveness of each of the elements in the package of KM interventions and select those that yield immediate impact, are the most flexible and offer the greatest return on investment. It may be tempting for future KM projects to merely select from the three basic interventions piloted in this demonstration project for scale up or replication, especially if it meets a particular donor’s interest. However, not all of the interventions reviewed here demonstrated that they provide the greatest return on investment. For example, demand for the toolkits is currently low. Although a concerted campaign to disseminate information about the toolkits is possible, by design toolkits are dependent on users having access to the Internet and possessing a high level of literacy. The pattern of information seeking frontline health workers described for solving problems required having access to tailored information when they needed it from colleagues and from a variety of resources on the Internet. Community Health Workers cannot be reached through toolkits, which are more appropriate for highly educated users who are comfortable with reading to gather information. The major advantage of the toolkits was their contribution to building consensus among the members of the Malawi KM Task Force during the development phase. This pilot project has demonstrated the potential of real impact in health service delivery at community level with a very small financial investment. In terms of how to expand and scale up the package of interventions piloted by the K4Health Malawi Project, the evaluation team recommends continuing to grow the use of toolkits through additional promotion as well as use by the MOH, but without investing further in new platform development until the toolkits have proven their worth. The hallmark of the DLC and SMS Network is that they are decentralized, flexible, and content independent. They do not require constant updating and maintenance. For this reason, donors and the MOH need not make large capital outlays for a centralized system of content generation. By prioritizing expansion of the DLCs and the SMS Network, investments can be made gradually over time with technical content generated by the health workers themselves. 2. Revise and expand the monitoring and evaluation design. The M&E design for this project was laudable in its thoroughness and rigor. Several elements yielded important information at reasonable cost. These were the Net-Mapping workshops, the LQAS surveys, the focus group discussions and key interviews, and the collection of Hub statistics. Other methodologies proved difficult to implement in the field and probably should be dropped in future projects. These include lengthy DLC user surveys (not the registries), the collection of data on toolkit use, and data collection from control districts. The baseline Net-Map exercise built consensus around information flows and needs and provided a visual before and after snapshot of information flows. One recommendation is that it be carried out earlier in the project so that it can inform the design of interventions. In the case of K4Health, the design of the project had already been determined before the Net-Mapping took place. 65 The DLC user surveys are not practical as routine monitoring tools given the tightness of district health workers’ schedules. A more useful approach may be to carry out periodic phone surveys of district health workers using random sampling in order to assess use of the DLCs and suggestions for improvement. The difficulty in getting accurate statistics on toolkit use using Google Analytics has been mentioned throughout this report. Future evaluation of toolkit usefulness and use could be better carried out as part of a media campaign or in conjunction with concerted efforts by the Health Education Unit of the MOH to promulgate their use. The use of control districts for the LQAS survey of CHWs did not add information that was not obtained in the mid-term survey from non-cell phone users within the intervention districts. Finding an adequate match in terms of district health worker density and size proved difficult. For example, whereas Nkhotakota has 1.75-2.00 health workers per 1,000 population, Kasungu (the control district at baseline) has only 1.00 health worker per 1,000 population, and far greater distances for CHWs to travel. When another project provided cell phones to CHWs in Kasungu, it was also no longer possible to use it as a control district. More useful variation can be detected by carrying out follow up LQAS surveys and comparing supervision areas to each other, as was done in the endline survey. The next frontier in M&E for KM projects in Malawi is to demonstrate that improved information flows lead to improved services. This can be achieved by designing an operations research study for any future phases of the program in order to measure associations between the use of cell phones and impact on health services. Districts can disaggregate data they collect on, for example, family planning visits by supervision area using LQAS, so that they can draw links with CHWs’ use of cell phones. There was sufficient variation in the number of messages sent by CHWs during this pilot project to test the hypothesis that more messages are associated with improved services. 3. During the design phase of future projects engage districts in using data to identify their most pressing health problems and prioritize them for intervention. The Malawi K4Health Project selected family planning and HIV/AIDS as the subject of intervention, and there is little doubt that these are key areas of concern in Malawi. However, one of the most intriguing findings from recent work in Health System Strengthening has been the success of using data to identify the most pressing health issues in a particular catchment area and prioritizing interventions that address those issues first and foremost. Districts could be engaged to use their own data to identify which interventions will have the most impact on health outcomes for their particular populations. Estimations of morbidity and mortality averted by increased service utilization can be made using the Lives Saved Tool (LiST) from the Spectrum suite of tools. 10 4. Consider all potential costs up front and take steps to provide for them in the budget. A project that focuses on using cell phones and Internet to increase information flows might not seem to require as much investment in gas and internet lines as other types of projects, but this is 10 The Spectrum suite of tools, developed by Futures Institute, is accessible online at http://www.futuresinstitute.org/Pages/Spectrum.aspx 66 not the case. The budget for gas and vehicle use needs to be sufficient to get the phones and training materials out to community health workers. Internet lines also needed to be installed by the project in the intervention districts in order for the DLCs and the Hub to function. If there is a risk that payment to Internet providers will be delayed, future projects should establish a payment schedule that pays for a month in advance. This would avoid services being cut off. Also, future projects should work with mobile phone companies from the beginning to establish in-network groups that allow all messages passing through the Hub to be paid for. When donor funding isn’t available, consider charging a small fee to other users of the DLCs such as NGOs. 5. Use the Leadership Development Program (LDP) and Net Mapping to energize and mobilize stakeholders. The adapted LDP enabled the KM Task Force to create a vision around which a team that had never worked together mobilized to achieve two out of its three desired measurable results. The Net-Map workshops also enhanced the sense of shared purpose and concretely demonstrated the constraints to information flows from national through district to community levels, and the improvement in these flows after the introduction of the SMS network. The LDP is known to be highly successful in building leadership skills, even when the team members do not work with each other on a regular basis. The LDP should be a first step in any future KM project. Without it, it is doubtful that national or district teams can find the energy and sense of purpose needed to carry out such challenging action plans. 67 APPENDICES A. List of Tables and Figures TITLE PAGE Figure 1. K4Health Malawi Evaluation Timeline 14 Table 1. Net-Map Interview Structure 15 Figure 2. Map of Malawi showing intervention and control districts 18 Table 2. Distribution of LQAS respondents at endline 20 Table 3. Distribution of LQAS respondents at endline 20 Table 4. Summary of Focus Groups Discussions and Key Informant Interviews 21 Table 5. Achievement of LDP Task Force Desired Measurable Results 25 Figure 3. Screen Shot of Malawi K4Health Web Page with Toolkits 26 Table 6. Use of Malawi specific toolkits between 10/10 and 6/11 28 Figure 4. Visits to Malawi Toolkits from within Malawi 27 Figure 5. National Level Net-Map in 2010 30 Figure 6. National Level Net-Map in 2011 30 Figure 7: In Degree and Out Degree Centrality Values from National Stakeholder Net-Map 2011 32 Figure 8: Actors and links that could be strengthened with a longer project 34 Figure 9: 2010 Baseline Net-Map of Information Flows with District level Stakeholders in Salima 36 Figure 10: 2011 Endline Net-Map of Information Flows with District level Stakeholders in Salima 36 Figure 11. Trends in District Learning Center visits between June 2010 and 2011 38 Figure 12. Breakdown of DLC access by gender between September 2010 and June 2011 39 Figure 13. SMS Network in Nkhotakota and Salima 46 Table 7: Key Project Indicators at Mid-Term and Endline 49 Figure 14. Trends in Stock-out Messages 51 Figure 15. Breakdown of SMS messages by category between June 2010 and June 2011 52 Table 8. Summary Results of K4Health Key Indicators from LQAS Endline Survey 54 68 Appendix B. Net-Mapping Guide Guide to Net Mapping in Malawi Before the interview - On a flipchart sheet write the date and the question at the top. - Write the names of the links on the corner of the flipchart sheet, using a marker with the color that corresponds to each link. Research Overview: - Give a brief introduction of the team that is to conduct the interviews and of the K4Health Project. - Explain the purpose of the interview and give a short summary of the network map (Net-map). - Provide a short definition of the information on family planning and reproductive health and HIV/AIDS and HIV/AIDS and of the information flow. - Enter the basic demographic data about the respondent. Hello. I am. I am part of a team that is conducting an assessment for the Knowledge for Health Project (K4Health) of the need people have for information on family planning and reproductive health and HIV/AIDS and HIV/AIDS in Malawi. K4Health promotes the exchange of knowledge and encourages the use of the latest research for decision making and program implementation. The purpose of this assessment of needs in Malawi is to analyze the demand, access and use of information on family planning and reproductive health and HIV/AIDS and HIV AIDS among health professionals and managers at different levels of the health system, and to identify opportunities to increase the exchange of information where it is needed. To achieve this purpose, we are using the Network Map (Net-Map) technique to interview participants at different levels of the national and district health system in Malawi. Net-Map is an interview technique that studies the prospects and influence of different stakeholders involved in reproductive health in Malawi, and examines how these individuals or entities interact with each other. Since you are an expert in this field, we would like to interview you to help us understand who is involved in the management and exchange of information and knowledge on reproductive health and HIV/AIDS in Malawi. We will begin by listing all the people (or players) involved in the exchange of information and then we shall determine how they are linked. We will examine how influential each person is in the process and finally, we will discuss ways to improve the flow of information to health providers at the lowest levels of the health system. When using Net-Map, it is crucial to take into account how things actually happen and not how they should happen or what is printed on official documents. Therefore, we need the expertise of people like you who are part of the process and know it first-hand. We would like to know what kind of information on family planning and reproductive health and HIV/AIDS providers of the Malawian health system need, in order to better serve the needs for information on health and to improve the exchange of information in Malawi. The general question we seek to answer in this interview is: Important terms: • When we talk about the information on family planning and reproductive health and HIV/AIDS and HIV/AIDS and HIV/AIDS, we mean information and knowledge such as best practices, clinical protocols, program guides, Who plays a role in improving flows of technical information on HIV/AIDS and Reproductive Health to Healthcare Service Providers? 69 research articles, auxiliary equipment, information materials, education and communication (ECI). We are not referring to the statistics of services or other data provided by the Health Information System. • When we talk about the exchange of information on family planning and reproductive health and HIV/AIDS and HIV/AIDS, we are referring to the collection, production and dissemination of technical information among the stakeholders of the national health system. • When we talk about improving the exchange of information on family planning and reproductive health and HIV/AIDS and HIV/AIDS, we mean that health personnel should handle adequate technical information in a timely and high quality fashion in order to improve the quality of these services. The interview will last approximately two hours and you can stop the interview at any time if you so desire. Please share your views, both positive and negative. Everything you say today will remain confidential. I will take notes during the interview and I would like to record our conversation to back up my notes. The team members who are assessing needs are the only ones who will be reading my notes and listening to the recording. Is it OK that I record this conversation? Let me pause for a moment to answer any questions you have. Do you have any questions? [PAUSE: ANSWER QUESTIONS]. Step 1: Enter demographic data (note: do not put names in final report) Step 2: Identifying the players/Actors 1. Explain that the first step in developing the map is to identify the key players who work in family planning and reproductive health and HIV/AIDS and HIV/AIDS. By players we mean the organizations that work in FP and RH, including government institutions, NGOs, professional associations, private clinics, and donors. Present the global list of players so that the person or group can select the key players. Then ask the following questions: 1. Institution/organization the person interviewed belongs to: 2. Position of person interviewed: 3. Gender: Male / Female 4. Age: <25 years old / 25-40 years old/40 years old or more 5. What is your level of education: 6. Location: District, province, department 7. Telephone: 8. Email: 9. Date of interview: 10. Interviewer: 11. Time the interview begins: Time the interview ends: 12. Interview duration: 70 2. Explain that the different-colored sticky notes represent the various categories of players: o Government (Pink) o NGOs (Yellow) o Donor (Blue) o Networks or professional associations (Green) o Community groups (Purple) o Other (White) 3. Place the players on the flip chart sheet, in no particular order. Step 3: Draw links between the players Explain that in this step the links between the players are identified and analyzed. Explain the meaning of link: "A link is a connection, relationship or exchange between players". In this study, we are looking specifically at the information exchange between players. The purpose of the link in the development of the map is to understand how each player relates to other players and to how many other players he is connected - this way we are trying to visualize the flow of information between players. Links should be related to the main question (information on family planning and reproductive health and HIV/AIDS). Then ask the following question, explaining every time what the meaning of each link is: a) Which players are involved in exchanging information on family planning and reproductive health with first-level health workers? Please list all organizations that work on family planning and reproductive health at all levels of the national health system, including social security, police and armed forces health service, private clinics, NGOs and other services you know. Do not forget to place yourselves on the map. • Who are the players that produce, distribute and collect technical data on family planning and reproductive health? • Who are the players who can influence people who do this, or develop policies that determine how to do this? • Who are the players who are involved directly or indirectly in exchanging information? 71 1. Draw arrows between stakeholders using a different color for each link. Draw only one link at a time (for example, complete all links related to capacity-building before starting with the links related to supervision), but allow them to add links afterwards if they remember anything else. - If it seems the interviewee is not clear on the differences between the links, or is talking about exchanges which are off the subject, repeat the definitions and ask the question about the arrow once again. - If the interviewee is unsure about the arrow—does not know if it happens or not, does not know how often it happens, or does not know if it really happens or should happen—draw it if the arrow refers to something that: o the interviewee is sure that happens and o occurs frequently or is important (for example, a meeting with the Ministry, which only happens once a year, is very important though not frequently occurring). Step 4: Analyze the map (with people after they have drawn it) 1. Explain that this step consists of analyzing the map and the exchange of technical information between the stakeholders. Ask the following questions: For the following links, who provides ____________ to whom? a. Capacity-building Meaning: sessions organized with the aim of improving reproductive health knowledge and practices of the people receiving capacity-building • MAKE A NOTE OF WHAT KIND OF TECHNICAL INFORMATION IS REFERRED TO IN THE LINK b. Supervision Meaning: visits aimed at reviewing local practices, services or data • MAKE A NOTE OF WHAT KIND OF TECHNICAL INFORMATION IS REFERRED TO IN THE LINK c. Technical Assistance Meaning: providing technical information to develop, enhance, and distribute policies, procedures, protocols, guidelines / manuals, curricula, information / education / communication materials (IEC), tools to conduct surveys. Includes evaluation meetings or national informative meetings. • MAKE A NOTE OF WHAT KIND OF TECHNICAL INFORMATION IS REFERRED TO IN THE LINK d. Consultancy / counsel Meaning: expert counsel given on family planning and reproductive health; this relation is more informal than that of technical assistance • MAKE A NOTE OF WHAT KIND OF TECHNICAL INFORMATION IS REFERRED TO IN THE LINK e. Financing Meaning: the transfer of resources, including funds and supplies • MAKE A NOTE OF WHAT KIND OF TECHNICAL INFORMATION IS REFERRED TO IN THE LINK 72 Step 5: Attribute decision-making power 1. Explain that in this step the decision-making power or influence of each stakeholder is identified. Explain the meaning of decision-making power or influence: “When we talk about decision-making power or influence, we mean…….” The capacity or ability to implement, change or improve the exchange of information. Then, ask the following questions: 2. Explain that in this section the power of decision / influence of each stakeholder is attributed. Explains the levels used for the attribution and the meaning of each level: “……” Then, ask the following questions: 3. Then, ask the interviewee to explain the level of influence of each player: a) How does the exchange of information between different stakeholders occur? b) What are the occurring challenges in the information exchange? c) What are the barriers to the access and use of information on family planning and reproductive health? d) Who are the stakeholders that need technical information on FP / RH but that are not receiving it? e) What kind of specific information do they need? a. What are the different ways in which someone can influence the exchange of information on family planning and reproductive health in Malawi? Facilitator: If you forget anything, you can mention some examples: formal supervision, financing / funding, technical information, counsel, advocacy and pressure, or professional level / respect. b. Who has more power to change or modify the exchange of information on family planning and reproductive health toward first level workers? a. Where does the influence of each stakeholder come from and how is it used? b. What is the origin of the influence of all stakeholders whose influence is very high, very low, or appears to vary, or is not very clear? c. Ask for explanations, for example: "Stakeholders X and Y have the greatest degree of influence. Where does that influence come from?" "Why is Stakeholder W more influential than Stakeholder Z if Stakeholder W does not have many links?" DO NOT ASK INTERVIEWEE TO CHANGE THE LEVEL OF INFLUENCE. SIMPLY, COTINUE ASKING QUESTIONS UNTIL YOU UNDERSTAND THE INTERVIEWEE'S ANSWER OR UNTIL IT IS CHANGED BY THE INTERVIEWEE. a. What is the level of influence of each stakeholder, beginning with the most influential stakeholders? 73 a. Are there conflicting objectives or priorities among stakeholders? b. Are there synergistic objectives or priorities among stakeholders? c. What stands obstacle to cooperation or knowledge sharing between organizations? d. What contributes to or enhances cooperation or knowledge sharing between organizations? 4. Finally, review the entire board, starting by stating the level of influence of the stakeholder with the highest level, down to the stakeholder with the lowest level. The goal of this exercise in three stages is to allow the interviewee to reflect upon his/her answers and possibly make changes whenever contradictions are noted. Step 6: Determine the focus of each stakeholder 1. Explain that in this step each stakeholder's primary activity in the issue of family planning and reproductive health and HIV/AIDS is identified–the organizational focus or programmatic priority. If interviewee wishes to attribute more than one, ask which is paramount and underline, marking the first letter on the map. Ask the following question: 2. After determining the priorities of each stakeholder, reflect over the map and ask the following probing questions: Step 7: Probing questions Explain that, at this time, the interviewee may add, to the map, other stakeholders that are currently not on the map but should be, as well as signaling those important links that are weak, nonexistent or that have the potential of being fortified. This is an open discussion. a. What is the organizational focus of each stakeholder and his/her programmatic emphasis on reproductive health and family planning? - (S) Services (meaning: organization with people for the delivery of contraceptive supplieas and sexual and reproductive health care - (A) Advocacy / communication (meaning: actions aimed at influencing people, perceptions, policies and decisions - (I) Research (meaning: a process aimed at the pursuit of knowledge through methodologies - (P) Technical assistance - (F) Funding Based on the map that we have created and our discussion today, we talk about possible solutions: a. How could family planning and reproductive health information which arrives at health care providers of each level be improved? b. What are the mechanisms that could be created or improved in order to achieve those changes? c. Who are the stakeholders that are key to achieving those changes? Why them? 74 1. Explain that this final section focuses on the interviewee's organization specifically. } a. What specific information about family planning and reproductive health does the health staff at your level need in order to better perform their job? b. Where do people at your level turn to for the latest information on family planning? Please give a recent example. [Ask about individuals, organizations, etc.] c. What are the communication channels staff members at your level prefer to use to receive information? [Inquire: Radio? Television? Internet? Mobile phone? Printed Documents? Interpersonal communication?] Please rate them. If Internet, what browser? Do they resort to any Web page from a specific institution? Which one? d. How could your organization better reach other professionals from remote areas in order to provide them with important information about health? 75 Appendix C. Focus Group Discussion Guide for DLCs DLC USER FOCUS GROUP DISCUSSION GUIDE Name of District ___________________Venue:____________________ Number of Interviewees: Male________ Female________ Total________ Recorded by: _________________________ Facilitator__________________________ Date: __________________________________ No Question Responses Section 1 : Background Information 101 Let’s start today’s discussion by describing the profile of this group... Please describe your role in the health system. How long have you worked in this role? (Please make sure to ask each participant to respond) Section 2 : Knowledge for health information 201 Now let’s discuss how you get and use information for your job... What kind of information do you usually need to do your job? How do you usually seek this information? Please give specific examples? 202 Are you able to seek any of this information from the DLC? Please give a specific example. 203 Before the District Learning Center was launched, how did you usually get information to do your job? Is the DLC an improvement over the past? Please explain? 204 Why do you usually visit the DLC? Which DLC services (toolkits, general internet, email, hard copy resources, computer training, FLSMS) are most relevant or useful to your work? Are there other resources that would be useful but are not currently available in the DLC? Please explain? 76 No Question Responses 205 In the past 3 months, have you faced a particular challenge in your work that led you to seek information from the DLC? How did you use this information in your job? And what was the result? Please provide details. 206 Have you experienced any barriers to using the DLC? What do you think can be done to eliminate these barriers? Please explain? 207 Is there anything you would to see improved in the DLC? Please provide details. 208 Now let’s talk specifically about the toolkits. Are you familiar with the toolkits on HIV/AIDS and Reproductive Health that are available on the K4Health website? Which of the toolkits have you used? Which are most relevant or useful to your work? Why? 209 Is the information contained in the toolkit helpful to your work? Why or why not? Have you applied any of the toolkit knowledge in your job? Please give examples. 210 Do you prefer having technical information presented in a toolkit or would you rather access other general websites (such as Google) to search for information? 211 If you haven’t accessed any toolkit, can you please explain why? Have you experienced any barriers to using them? How could these barriers be eliminated? 212 In what way can we improve the toolkits? Is there anything else you’d like to say specifically about the toolkits or about the DLC in general? The End, Please thank the group. 77 Appendix D. Focus Group Discussion Guide for HSAs and CBDAs HSA and CBDA FOCUS GROUP DISCUSSION GUIDE Name of District ___________________Venue:____________________ Number of Interviewees: Male________ Female________ Total________ Recorded by: _________________________ Facilitator__________________________ Date: __________________________________ 1. Let’s start today’s discussion by describing the profile of this group... Please describe your role in the health system. How long have you worked in this role? (Facilitator: please make sure to ask each participant to respond) 2. Now let’s discuss how you get and use information for your job... When we talk about information today, we mean specific information about a case or general information about drug dosages, stock outs, meeting times, IEC materials, etc. What kind of information do you usually need to do your job? How do you usually seek this information? Please give specific examples? 3. Are you able to seek any of this information through the FLSMS system? Please give a specific example. 78 4. Now think about a particular challenge that you face in the past 3 months that led you to seek information using the SMS system... Can you describe the challenge? How did you use the SMS information to address it? And what was the result? Please provide details. 5. When you ask a question or seek support through the SMS system, do you feel confident in the response you receive? Please explain? How do you feel about receiving information and support through a text message? Please elaborate? 6. How does having a phone affect your ability to serve your clients? How does it affect the number of client visits you make? How about the number of referrals? Please give specific examples. 7. Are there any drawbacks to having a phone? Benefits? Please explain. 8. How do you normally communicate with your supervisor? How has having a cell phone changed the way you communicate with him or her? Please provide examples. 9. Do you prefer communicating with your supervisor by SMS, phone or face-to-face? Why? 79 10. What do you think should be done to improve communications between CBDAs/HSAs and the district level? 11. Have you experienced any barriers to using the SMS system? What do you think can be done to eliminate these barriers? Please explain? 12. Is there anything you would to see improved in the SMS system? Please provide details. 13. Did you participate in the e-learning by phone that the K4Health project is promoting? How has the e-learning by phone benefited your work? How could it be improved? The End, Please thank the group. 80 Appendix E. KII guide for District Coordinators Malawi K4Health Key Informant Interview Guide District Coordinators Name of District __________________________________Venue____________________ Name of interviewee _________________________Position ________________________ Gender_________________________ Recorded by _________________________ Facilitator_________________________ Date: __________________________________ Introduction Hello, Good Morning/Afternoon. My name is . I am part of a team conducting a study for the USAID-funded Knowledge for Health Project. This project is working in Malawi to promote information sharing among health workers, especially in the areas of HIV/AIDS and Family Planning/ Reproductive Health. For our study, we are talking to health workers and district coordinators to learn more about your work The interview will take less than one hour to complete and you can stop the interview at any point if you’d like. Please try to respond as honestly as possible -- it will help in future activities. Everything that you say today will be kept confidential. I will be taking notes during the interview and recording the conversation to back up my note-taking. The assessment team will be the only ones to read my notes and listen to the audio-tape. Let me stop for a moment to respond to any questions you may have. [PAUSE - ANSWER ANY QUESTIONS]. May I turn on the tape recorder? 81 Section 1 : FLSMS District Coordinators 101 Please describe your role in the district. How long have you served in this function? 102 Are you familiar with the front line SMS system that is being used to link community level providers (CBDAs and HSAs) to district level health workers? Please describe your role with the FLSMS. 103 How many CBDAs and HSAs are currently under your direct supervision? 104 Which zones or catchment areas does this include? 105 How often do you normally communicate on average with each CBDA and HSA under your supervision? Please provide details. 106 How (using what means) do you normally communicate with these CBDAs and HSAs? Please provide details. 107 How do you normally provide information to CBDAs and HSAs under your supervision? Please provide details. 108 For what purposes do you usually use the FLSMS? Please explain. (PROBE beyond using the SMS for health worker questions and concerns.) 111 What are the common types of questions or issues that you receive through the FLSMS? Please be specific. 112 Can you describe how one of these issues was resolved using the FLSMS? Please explain. 113 How does the FLSMS affect your ability to supervise the CBDAs and HSAs assigned to you? Please explain. 114 How do you normally make community support visits to community health workers? What role does the FLSMS play with the community support visits? Please explain. 82 Section 1 : FLSMS District Coordinators 115 In your opinion, what are some of the advantages of having the FLSMS system in this district What are the drawbacks? 116 Are there any other ways in which the the FLSMS system has affected the delivery of services in this district? Please give specific examples 117 How has the FLSMS affected communication and information sharing among health workers in this district? Please give specific examples. 118 Overall, do you think this system is worth continuing or replicating in other areas? 119 Overall, what do you think should be done to improve communication and information sharing between CBDAs or HSAs and the district level? Please give suggestions. 120 What do you think should be done to improve the FLSMS system? Please give suggestions. 83 Appendix F. LQAS questionnaire for SMS Users K4HEALTH/MALAWI FLSMS PROGRAM ENDLINE STUDY – JUNE 2011 LQAS QUESTIONNAIRE for SMS USERS MODULE 0: RESPONDENT IDENTIFICATION AND INTERVIEW SUMMARY District Name: ______________________________________ CODE: |____| (See code list) Name of Supervision Area: ___________________________ CODE : |____|____| (See code list) Name of Health Center: ______________________________ CODE : |____|____| (See code list) Position of Respondent: _____________________________________ Duration on Position: _______________________________________ Sex of Respondent: ___M ___ F Age of Respondent: _____ Years Date of interview: |____|____||____|____||___|___| D D M M Y Y Enumerator (Name) __________________________________ CODE: |____|____| (See code list) TO BE COMPLETED AFTER INTERVIEW HAS BEEN DONE Name of supervisor__________________ Date checked |____|____||____|____||___|___| D D M M Y Y Is questionnaire complete? Yes |____| No |____| Date of data entry |____|____||____|____||_1_|_1_| D D M M Y Y 84 INTRODUCTION Hello, Good Morning/Afternoon. My name is……I am working for Management Sciences for Health/Knowledge for Health Project as a research assistant and you have been randomly chosen to participate in a survey. We are trying to learn more about how we are doing with the K4Health project. I would like to talk to you about your work, your involvement with the K4Health project, how you access health related information and technical support especially in the areas of HIV/AIDS and Family Planning/Reproductive Health. If you can answer our questions as honestly as possible it will help in future related developments. You should not hesitate to say you do not understand a question, or do not know the answer. Your answers will be kept completely confidential. This interview takes about 30-40 minutes. Would you be willing to talk to me? Thank you. DID THE RESPONDENT CONSENT TO PARTICIPATE? Yes |____| MODULE 1: Health Communication through SMS technology # QUESTION RESPONSE CODES 01 Do you have a cell phone provided by the K4Health Project for the FLSMS system? (If NO, END the interview) Yes……………………………1 No……………………………..2 02 If so, when did you receive the FLSMS phone? June 2010…………………….1 January 2011…………………2 03 How often do you send an SMS to the FLSMS system per month? …………………………………. 04 What are the main reasons or purpose for sending the SMS? Emergencies……………………1 Stock outs……………………….2 Ask general information…….…4 Technical support………………5 Patient Referrals………………..6 Guidance on dispelling myths…7 Others(specify)………………….8 05 In the past month, how many messages did you send to others related to your work (this includes, co-workers, supervisors and the Hub)? …………………………………. 06 In the past month, how many messages did you receive from others related your work (this includes, co-workers, supervisors and the Hub)? …………………………………. 07 Please explain any barriers you may have encountered related to sending an SMS message. Connectivity……………………...1 Technical difficulties using the phone …………………………….2 Difficulty using the keypad to write messages………………………..3 Difficulty using Key Words……..4 Others (specify)………………...5 …………………………………... 85 08 Please explain any barriers you may have encountered related to receiving an SMS message. Connectivity…………………….1 Technical difficulties using the phone……………………………2 Limited airtime………………….3 Others (specify)………………..4 …………………………………… 09 In addition to using your phone for sending text messages, in what other ways have you used your phone? Timing of patient vital signs…..1 Calling supervisor……………..2 Calling friends…………………3 Others (Specify).......................4 .................................................... 10 Do you have access to health-related technical support when you need it? Yes ............................ 1 No.............................. 2 11  If no, please explain: Then skip to Q13 12 If yes, please explain the type of support you have received in the past month in response to a phone call or SMS: Diagnosis…………………………… ….1 Treatment…………………………… …2 Emergency evacuation of patients…...3 Referral for something other than an emergency……………………………. .4 Answers to technical questions on FP or other health issues………………….…5 Resolution of stock outs………………6 Responses to eLearning questions……………………………… .7 Notifications of events (alerts)………..8 Others(specify)……………………… …9 ………………………………………… … 86 13 Who provides you with technical support most of the time? HSA……………………………………. 1 ADEHO……………………………….. .2 DEHO…………………………………. 3 Medical Assistants……………………4 Nurses………………………………… 5 Clinician………………………………. 6 District health Coordinators………………................ 7 Doctors…………………….…………. 8 Others(specify)………………………. 9 14 How far from your residential area is the person providing the technical support? ………………………………... 15 Are you satisfied with the quality of technical support you receive from this person? Yes……………………………1 No…………………………….2 16 If no, please explain: ………………………………………… ………………………………………… ………………………………………… ………………………………………… ………………………………………… ………………………………………… 17 How do you usually contact the person providing the technical support? Phone Call…………………… 1 Public transport (in-person visit)………………………….. 2 SMS…………………..……… 3 Hire push bike ……………… 4 Use own push bike ………… 5 Regular supervision visit ….. 6 Others(Specify)……………... 7 18 How much does it cost you to get in touch with this person in one occasion using the means described by you in Q 17? ………………………………… 19 On average, how much time does it take for you to contact this person using the means described by you in Q 17? ………………………………… 87 20 On average, how many times per month do you contact this person using the means described by you in Q 17? ………………………………… 21 How much does it cost you to use public transportation in one occasion to seek technical support? ………………………………… 22 How much does it cost you to use your own push bike in one occasion to seek technical support? ………………………………… 23 How much does it cost you to hire a push bike in one occasion to seek technical support? ………………………………… 24 How much does it cost you to make a phone call in one occasion to seek technical support? ………………………………… 25 How much does it cost you to send an SMS in one occasion to seek technical support? ………………………………… 26 On average, how many clients do you visit each month? ………………………………… 27 On average, how often do you visit the same client in a month? ………………………………… 28 On average, how many referrals do you make in one month? ………………………………… 29 On average, how much time does it take you to report important events or seek support through the SMS system? ………………………………… 30 On average, how much time does it take you to receive feedback on issues or questions you have reported through the SMS? ……………………………….. 31 Are you satisfied with communications between CBDAs/HSAs and the district level? Yes……………………………1 No…………………………….2 32  please explain your answer in Q 31 above ………………………………… 33 Are you satisfied with way the front line SMS system is operating in your district? Yes……………………………1 No…………………………….2 34  please explain your answer in Q 33 above ………………………………… 35 What are some other benefits that have come because of having a cell phone? Personal communication......1 Phone charging fee..............2 Reduced cost of travel..........3 Reduced travelling time........4 Others(specify).....................5 The end, please thank the respondent. 88 Appendix G. LQAS questionnaire for Control District K4HEALTH/MALAWI FLSMS PROGRAM ENDLINE STUDY – JUNE 2011 LQAS QUESTIONNAIRE FOR CONTROL DISTRICT MODULE 0: RESPONDENT IDENTIFICATION AND INTERVIEW SUMMARY District Name: ______________________________________ CODE: |____| (See code list) Name of Supervision Area: ___________________________ CODE : |____|____| (See code list) Name of Health Center: ______________________________ CODE : |____|____| (See code list) Position of Respondent: _____________________________________ Duration on Position: _______________________________________ Sex of Respondent: ___M ___ F Age of Respondent: _____ Years Date of interview: |____|____||____|____||___|___| D D M M Y Y Enumerator (Name) __________________________________ CODE: |____|____| (See code list) TO BE COMPLETED AFTER INTERVIEW HAS BEEN DONE Name of supervisor__________________ Date checked |____|____||____|____||___|___| D D M M Y Y Is questionnaire complete? Yes |____| No |____| Date of data entry |____|____||____|____||_1_|_1_| D D M M Y Y 89 INTRODUCTION Hello, Good Morning/Afternoon. My name is……I am working for Management Sciences for Health/Knowledge for Health Project as a research assistant and you have been randomly chosen to participate in a survey. We are trying to learn more about how you access health related information and technical support for your work, especially in the areas of HIV/AIDS and Family Planning/Reproductive Health. If you can answer our questions as honestly as possible it will help in future related developments. You should not hesitate to say you do not understand a question, or do not know the answer. Your answers will be kept completely confidential. This interview takes about 30-40 minutes. Would you be willing to talk to me? Thank you. DID THE RESPONDENT CONSENT TO PARTICIPATE? Yes |____| MODULE 1: Health Communication through SMS technology # QUESTION RESPONSE CODES 01 Do you use a cell phone to access information or get technical support for your work? (If YES, END the interview) Yes………………………1 No……………………… 2 02 Do you have access to health-related technical support when you need it? Yes............................. 1 No .............................. 2 03  If no, please explain: Then skip to Q 05 04 If yes, please explain the type of support you have received in the past month: Diagnosis……………………….1 Treatment………………………2 Emergency evacuation of patients………………………….3 Referral for something other than an emergency……………………..4 Answers to technical questions on FP or other health issues………….5 Resolution of stock outs……….6 Responses to eLearning questions………………………..7 Notifications of events (alerts)...8 Others (specify)………………..9 90 05 Who provides you with technical support most of the time? HSA…………………………..1 ADEHO………………………2 DEHO………………………...3 Medical Assistants………… 4 Nurses………………………..5 Clinician………………………6 District health Coordinators………………….7 Doctors…………………….…8 Others(specify)………………9 06 On average, how much time does it take you to report important events or seek support from the person who provides you with technical support most of the time? ………………………………… 07 On average, how much time does it take you to receive feedback on issues or questions you have reported? ……………………………….. 08 Are you satisfied with the quality of technical support you receive from this person? Yes……………………………1 No……………………………..2 09 If no, please explain 10 Are you satisfied with communications between CBDAs/HSAs and the district level?? Yes……………………………1 No…………………….……….2 11  please explain your answer in Q10 above 12 How far from your residential area is the person providing the technical support? ……………………………………. 13 How do you usually contact the person providing the technical support? Phone Call………………………… 1 Public transport (in-person visit…..2 Hire push bike ……………………. 3 Use own push bike ………………. 4 Regular supervision visit ………... 5 Others(Specify)…………………... 6 14 How much does it cost you to get in touch with this person in one occasion using the means described by you in Q 13? ………………………………… 91 15 On average, how much time does it take for you to contact this person using the means described by you in Q 13? ………………………………… 16 On average, how many times per month do you contact this person using the means described by you in Q 13? ………………………………… 17 How much does it cost you to use public transportation in one occasion to seek technical support? ………………………………… 18 How much does it cost you to use your own push bike in one occasion to seek technical support? ………………………………… 19 How much does it cost you to hire a push bike in one occasion to seek technical support? ………………………………… 20 How much does it cost you to make a phone call in one occasion to seek technical support? ………………………………… 21 On average, how many clients do you visit each month? ………………………………… 22 On average, how often do you visit the same client in a month? ………………………………… 23 On average, how many referrals do you make in one month? ………………………………… The end, please thank the respondent.