FINAL PERFORMANCE EVALUATION PMI MOZAMBIQUE SOCIAL AND BEHAVIOR CHANGE COMMUNICATION PROGRAM Final Report JULY 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Pinar Senlet (Team Leader), Ritva Parviainen and Carlos Lauchande on behalf of Management Systems International (MSI), A Tetra Tech Company. FINAL PERFORMANCE EVALUATION PMI MOZAMBIQUE SOCIAL AND BEHAVIOR CHANGE COMMUNICATION PROGRAM Draft Report Contracted under AID-656-C-17-00002 Mozambique Monitoring and Evaluation Mechanism and Services DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS ACKNOWLEDGMENTS...............................................................................................................ii ACRONYMS..................................................................................................................................iii EXECUTIVE SUMMARY.............................................................................................................. 4 Findings and Conclusions............................................................................................................................................................ 4 Lessons Learned and Recommendations................................................................................................................................ 8 I. INTRODUCTION.................................................................................................................... 11 Malaria in Mozambique ..............................................................................................................................................................11 Partnership Between NMCP and the PMI ............................................................................................................................12 Activity Description....................................................................................................................................................................12 II. EVALUATION PURPOSE, OBJECTIVES AND QUESTIONS.......................................... 15 III. EVALUATION METHODOLOGY...................................................................................... 16 Evaluation Timeline.....................................................................................................................................................................16 Team Composition, Responsibilities and Roles..................................................................................................................16 Evaluation Methodology ............................................................................................................................................................17 IV. FINDINGS AND CONCLUSIONS: EFFECTIVENESS OF INTERVENTIONS ............. 23 Effectiveness in Improving Malaria Knowledge....................................................................................................................27 Effectiveness in Behavioral Change for Malaria Prevention and Treatment ................................................................30 V. FINDINGS AND CONCLUSIONS: SUCCESSES, BARRIERS AND COORDINATION WITH OTHER STAKEHOLDERS ............................................................................................ 31 Successes During Implementation ..........................................................................................................................................31 Barriers and Challenges.............................................................................................................................................................31 VI. FINDINGS AND CONCLUSIONS: EFFECTIVENESS OF THE M&E SYSTEM............ 36 Design of the M&E Plan.............................................................................................................................................................36 Data Collection System and Data Quality............................................................................................................................37 Data Analysis and Use of Data to Improve Performance ................................................................................................38 Effectiveness of Monitoring of Community Radio Messages ...........................................................................................38 VII. FINDINGS AND CONCLUSIONS: GENDER ANALYSIS.............................................. 41 VIII. SUMMARY OF EVALUATION CONCLUSIONS........................................................... 44 IX. LESSONS LEARNED AND RECOMMENDATIONS FOR FUTURE USAID ACTIVITIES................................................................................................................................. 45 ANNEX A: SCOPE OF WORK ................................................................................................. 48 ANNEX B: ANALYTICAL FRAMEWORK: GETTING TO ANSWERS MATRIX............... 61 ANNEX C: LIST OF DOCUMENTS REVIEWED ................................................................... 64 ANNEX D: LIST OF KEY INFORMANTS INTERVIEWED ................................................... 66 ANNEX E: KAP SURVEY PROTOCOL ................................................................................... 69 ANNEX F: QUALITATIVE DATA COLLECTION TOOLS .................................................. 74 ANNEX G: FINDINGS/CONCLUSIONS/RECOMMENDATIONS MATRICES ................ 100 ANNEX H: RESULTS FROM KAP SURVEY.......................................................................... 123 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | ii ACKNOWLEDGMENTS The evaluation team is grateful to the more than 1,500 individuals who generously gave their time and insights for this report. Throughout this evaluation, the team listened to and benefited from the contributions and experiences of dozens of Ministry of Health (MOH) officers, PIRCOM representatives and other implementers involved in malaria-related interventions at the central, provincial and district levels. In addition, the team received insight from community leaders, PIRCOM supervisors and volunteers as well as project beneficiaries (women and men) across the communities in which PIRCOM interventions were implemented. As such, this report represents a collective effort and incorporates experiences, opinions and observations of a large community of individuals. The team would especially like to thank the USAID/Mozambique Malaria team members for their guidance and support throughout the evaluation process. In particular, the evaluation team gives thanks for assistance received from Flavio Wate, the activity AOR. The evaluation team hopes that these observations, analysis, and recommendations will help all who are committed to fight against malaria in Mozambique, and that further progress will be realized through collective efforts. ACRONYMS ADRA Adventist Development and Relief Agency ACT Artemisin-based Combination Therapy CRs Community Radios FCR Findings/Conclusions/Recommendations FGD Focus Group Discussions IHO Integrated Health Office ITNs Insecticide-Treated Mosquito Nets IRS Indoor Residual Spraying IPs Implementing Partners IPT Intermittent Preventive Treatment of Pregnant Women JHU John Hopkins University KAP Knowledge, Attitudes and Practices KII Key Informant Interview M&E Monitoring and Evaluation MMEMS Mozambique Monitoring and Evaluation Mechanism and Services MOH Ministry of Health NMCP National Malaria Control Program NPDs Non-project Districts PIRCOM Interreligious Program Against Malaria PMI U.S. President’s Malaria Initiative PDs Project Districts PSI Population Services International RDT Rapid Diagnostic Test RLs Religious Leaders SBCC Social and Behavior Change Communication SPSS Statistical Package for Social Sciences USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 4 EXECUTIVE SUMMARY The purpose of the SBCC (Social and Behavior Change Comminication) Program Final Performance Evaluation is to assess the effectiveness of the malaria messages at influencing target audience malaria knowledge and practices as implemented by PIRCOM (Interreligious Progran Against Malaria). The results of the evaluation are intended to inform future USAID activity designs. The evaluation questions include touching upon the effectiveness of PIRCOM malaria messages at influencing target audience knowledge and practices; the successes and barriers during implementation; and the effectiveness of the PIRCOM M&E system. The evaluation – performed under the Mozambique Monitoring and Evaluation Mechanism and Services contract – was conducted between February and May 2018 and covered the period from the start of the current USAID award in November 2015 through December 2017. The methodology combined collection and analysis of quantitative and qualitative data. Data collection techniques included 1) review of existing secondary data; 2) semi-structured key informant interviews; 3) focus group discussions; and 4) a knowledge, attitudes, and practices (KAP) survey. The team visited three of the five PIRCOM implementation provinces to collect qualitative and quantitative data. Qualitative data was collected in three districts, and KAP survey was conducted in six implementation and six non-intervention districts. Analysis of complementary quantitative and qualitative data led to findings that support reliable and evidence-based conclusions. FINDINGS AND CONCLUSIONS EFFECTIVENESS OF INTERVENTIONS: PIRCOM used three approaches to reach communities with malaria prevention and treatment messages: 1) sermons and community meetings by religious leaders (RLs); 2) home visits by volunteers; and, 3) dissemination of malaria messages via community radios (CRs). Through these channels, PIRCOM disseminated five proven and highly effective malaria prevention and treatment messages. While all three means of message transmission were potentially effective and complemented each other, dissemination of malaria messages via RLs appeared to be the most effective approach based on the analysis of quantitative and qualitative data. The numbers of home visits conducted by volunteers was low, resulting in only a small percentage of households which benefitted from volunteer visits. Analysis of data collected did not support the hypothesis that community radio messages were effective in disseminating malaria messages. Broadcasting coverage was low, and it was not possible to estimate the number of community members reached by the messages. Throughout the qualitative data collection process, the evaluation team found that the PIRCOM activity is highly regarded by respondents who are familiar with it and who have been exposed to its malaria messages. The evaluation triangulated and verified the qualitative evidence collected on the effectiveness of PIRCOM interventions with the findings from the KAP survey. The KAP survey results showed that the knowledge on malaria prevention and treatment for all survey questions was significantly higher in PIRCOM intervention districts compared to non-intervention districts. The survey analyzed knowledge on 1) how malaria is transmitted; 2) perception of malaria risk; 3) groups at highest risk for malaria; 4) 5 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV that malaria is a preventable disease; 5) how to prevent malaria; 6) the importance of antenatal care to prevent and treat malaria; and, 7) symptoms associated with malaria. Practice and behavior changes achieved, theoretically as a result of malaria knowledge, however, were not found to be statistically different in intervention and non-intervention districts, except for the practice of indoor residual spraying (IRS). Sleeping under mosquito nets, pregnant women seeking antenatal care, households seeking immediate treatment for malaria, and completing malaria treatment were all high in both intervention and non-intervention districts. While improvements were slightly greater in areas where PIRCOM worked, those differences were not found to be statistically significant. Practice of IRS was an exception; the percentage of households who had IRS was very low in both PIRCOM and non￾intervention districts, but the practice was significantly higher in PIRCOM districts. All data from the KAP survey were disaggregated by gender and the findings were generally the same for males and females. The team concluded that PIRCOM interventions have been effective, but the effectiveness of these interventions in changing participant behavior was not high. There have been significant improvements in knowledge, but not so much in practices and behaviors. Malaria knowledge and positive behaviors appear to have been high throughout the provinces studied, allowing little room to improve, with IRS being an exception. SUCCESSES AND BARRIERS DURING IMPLEMENTATION AND COORDINATION WITH STAKEHOLDERS: Successes: PIRCOM is a well-known and respected organization in Mozambique. It has been working to bolster its institutional capacity since its establishment in 2007. The leadership is committed to improving activity performance and further developing its capacity as an organization. The activity helped PIRCOM to grow and strengthen its organizational and technical capacity, although there is room for further improvement. PIRCOM was successful in implementing its work plan, as it trained large numbers of delegates, RLs and volunteers on dissemination of malaria messages. Field interviews and discussions with PIRCOM staff and other stakeholders indicated that training programs have been successful and that PIRCOM staff were well trained on malaria messages and knew how to disseminate them. Barriers and Challenges: Despite PIRCOM’s progress in building on its organizational capacity over the years, there is room to grow as an organization, particularly in project management, management of human resources and monitoring and evaluation. The evaluation did not specifically analyze PIRCOM’s management, but during field trips the team observed challenges: the organizational structure, particularly at the district level, seemed to be complex and somewhat “loose.” Decision-making processes were not always well understood, and flow of information was complex. An overall finding/conclusion on PIRCOM management is that the organization is highly cost-conscious, opting for the cheapest option during activity implementation. This strategy to cut costs does not appear to be paying off well in the long run. Several examples are explained in this report, including decisions to rely on unpaid volunteers for critical work, purchasing low quality bicycles, and deciding not to introduce collection of monitoring data through smartphones. Although some volunteers did receive bicycles, PIRCOM volunteers did not receive direct monetary incentives, leading to low motivation to carry out home visits. Volunteers working in the same provinces USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 6 and districts for other IPs are paid incentives, which further complicates the issue. PIRCOM cannot pay its current volume of volunteers under its current budget because of the large number of volunteers. Transportation within the communities, and between the communities, districts and provinces continues to be a challenge due to long distances and costs. Volunteers note the long distances between the households in the communities while supervisors and delegates complain that they have to pay out of their pocket to travel to the communities. PIRCOM provided bicycles to RLs and volunteers to ease transportation and as an incentive. Regrettably, bicycles purchased were of poor quality: most of them broke down during the first year of use. Finally, PIRCOM’s geographic focus area poses a challenge. The activity is being implemented across a large number of provinces and districts, spreading limited resources thinly, which limits the intervention’s effectiveness and efficiency. This strategy to reach out too many districts, while coverage is very low in target districts, may be a factor limiting effectiveness. Coordination with Stakeholders: PIRCOM’s coordination with the Ministry of Health (MOH) at the central, provincial and district levels was generally good, but success appears to have largely depended on individual MOH initiative. In provinces and districts where the MOH officials were keen on coordinating with IPs, PIRCOM was always responsive. In others, where MOH was not as proactive approaching the IPs, coordination was not as strong. At the central level, the MOH was content with PIRCOM’s reporting on activities, coordination and responsiveness. PIRCOM participates in coordination meetings, is an active member in technical group meetings and regularly reports to MOH. PIRCOM coordinates closely with community leaders within the communities. Community leaders interviewed are well informed and interested in PIRCOM interventions. They were particularly instrumental in introducing PIRCOM volunteers, mobilizing communities during campaigns for distribution of mosquito nets and IRS. Community leaders have many strengths and could play a significant role in disseminating malaria messages. Not involving community leaders more directly in the PIRCOM activity is a missed opportunity. PIRCOM’s coordination with Abt Associates in Zambézia on IRS was instrumental in mobilizing communities and organizing households for spraying. Similarly, PIRCOM’s work with UNICEF to roll out training on health promotion, nutrition, child protection and education priority behaviors and with Johns Hopkins Univeristy Center for Communication Programs (JHU) to promote RLs’ involvement in health promotion and education program have been successful. EFFECTIVENESS OF THE M&E SYSTEM Design of the M&E Plan: The activity’s current M&E Plan includes 12 quantitative process indicators and each process indicator has a matching output indicator defined as the proportion of the total. PIRCOM reports on all process indicators but on none of the output indicators because the information on the “denominators” is not available. The evaluation team found that three of these process indicators were not useful, because they were either subject to double counting, or not meaningful, or accurate data were not available. The M&E plan is a useful tool to monitor timely implementation of planned activities and inputs; it shows that PIRCOM is making progress against planned activities. But, it does not evaluate or measure the results of interventions. 7 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Data Collection System: PIRCOM volunteers and RLs report on the number of home visits conducted and number of community members attending sermons/informational meetings on a monthly basis by manually filling in forms. Forms are collected by supervisors and sent to district delegates and finally to the PIRCOM provincial coordinators. Provincial coordinators enter data into access databases which are sent to PIRCOM headquarters. The data collection system is labor intensive and time consuming, due to the large number of forms and the frequency of reporting. The option to use smartphones for data entry in the field was not adopted due to cost implications. Data Quality, Analysis and Use: Quality of the data collected is questionable, due to low community capacity and inadequate capacity at the community level to monitor quality of reporting. The evaluation team did not find evidence of quality assurance measures in the field. Supervisors are supposed to check on data quality, but they only check on the completeness and timely submission of forms. PIRCOM is aware of the weaknesses of data quality and is currently conducting trainings to improve quality of data. Data collected through the M&E system is analyzed at PIRCOM headquarters. The analysis involves comparing indicator values against set targets for each indicator. The fact that data are not analyzed at lower levels is a missed opportunity to use data for decision-making and improving performance. PIRCOM was, however, able to make changes in implementation to improve the activity’s performance over the years. These improvements were not necessarily based on the analysis of the routine monitoring data; they were largely based on the feedback from and observations in the field. Monitoring of Community Radio Messages: PIRCOM does not directly monitor the CRs’ messaging but has confidence in the radio stations’ quality. The messages are centrally developed and approved by the MOH and USAID; they are broadcasted in Portuguese and the local language most commonly spoken in each province, and no issues were reported related to the quality of translations. Interviews with radio stations’ staff and radio communication experts revealed concerns about the stipulations of PIRCOM’s contracts with the stations regarding timing and format of malaria messages. The timing of airing messages is not optimal: mostly the messages are aired during mid-morning, which is not the peak time, and means they are missing out on the working young adults and the youth. The format used to air the messages most of the time is lecturing for 15- 20 minutes, which might become tedious. Experts commented on other innovative and more participatory formats to air malaria messages to attract more attention and interest from the audience. The evaluation team concluded that PIRCOM did not seek technical guidance from radio communication experts while developing the contracts with the community radio stations nor to monitor the effectiveness of the radio messaging. GENDER ANALYSIS AND CONSIDERATIONS Initially, the PIRCOM activity design did not include a gender analysis and a clear strategy on how to reach men and women. However, PIRCOM benefitted from the results of the capacity development assessments conducted by JHU and Eurosis, which included specific recommendations to raise gender awareness in the organization, and mainstreaming gender in community-level interventions. As a result, PIRCOM developed a policy and guide on how to approach gender issues and initiated implementation in April 2018. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 8 In the absence of specific orientation/guidance on how to address gender issues, PIRCOM’s approach to men and women largely depended on the individual volunteer or the religious leader. The team observed that PIRCOM staff were using SBCC materials developed by previous activities to reach men and women, and the findings were encouraging in the sense that the RLs and volunteers addressed the gender aspects very well. Malaria messages were taken seriously by both men and women and practices for prevention and treatment of malaria appeared to be widely followed by. Men and women generally received the same messages. However, regarding the antenatal care and seeking for malaria prevention and treatment for pregnant women and children under 5 years, men were specifically addressed in terms of their role as head of the family and their responsibilities. During the initial year of the activity, male volunteers were not always welcome and able to address women at the households when husbands were not at home. However, it appears that the sex of the volunteers is not as binding a constraint as previously was the case, as both male and female volunteers are introduced by the community leaders and RLs as PIRCOM activists and are well recognized and accepted by both male and female community members. There are relevant gender dynamics among the religious congregations: few Muslim women attend the Friday prayers at the mosques, while there are more female than male Christian community members who regularly attend the church. To increase access to malaria messages among Muslim women and Christian men, RLs reported that they are encouraging women and men to attend informational community meetings. The evaluation team did not find evidence that the gender norms in decision-making had a negative impact on women’s practices. Although men are the primary decision-makers, it doesn’t adversely affect women’s behavior change. PIRCOM has succeeded in taking care of gender issues in an appropriate way as both male and female respondents confirmed that the malaria messages were understood, accepted and followed correctly by both men and women. LESSONS LEARNED AND RECOMMENDATIONS The evaluation team provides the following lessons learned and recommendations to inform future USAID activity designs. To increase the effectiveness of interventions: • Promote the dissemination of malaria messages or other public health messages via PIRCOM RLs. PIRCOM has the capacity to reach out to broader communities through its extensive network of RLs, who are influential in dissemination of information and promoting effective practices. • Explore the possibility of broadcasting malaria and/or other health related messages via the national public radio instead of CRs. Future designs of messaging through radios should ensure appropriate timing of broadcasting and use of more participatory and attractive formatting to reach out to the intended audiences. 9 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV In cases where USAID decides to engage volunteers, home visits should be strategically planned and standardized to reach targeted audiences. Volunteers should receive comparable monetary incentives as provided by other organizations, their numbers should be reduced significantly, and home visits per volunteer should be increased to reasonable levels to cover a significant proportion of households in communities. To build on successes and overcome barriers: • Consider directly involving community leaders in dissemination of malaria messages or other public health messages. Community leaders have close ties within the communities and have a high potential for mobilizing communities to promote healthy practices. • Consider involving PIRCOM in future programs to increase IRS knowledge and coverage, as there is ample evidence that the organization has been successful in informing and mobilizing communities to increase IRS practices. • Continue to support PIRCOM organizational capacity development efforts, particularly in the areas of strategic management, management of human resources and strengthening of its M&E system. • Focus resources on a smaller number of high priority provinces and districts and strategize on how to reach a significant proportion of communities in each district in order to increase effectiveness and efficiency of future activity designs. • Be aware that pursuing inexpensive options, such as employing unpaid volunteers, purchasing low quality bicycles, and not using more advanced technologies due to cost considerations does not always pay off in the long run. There should be a balance between efforts to contain costs and maintaining the efficiency and effectiveness of the interventions. • Activity M&E plans should have fewer process indicators and a reasonable number of output/outcome indicators to improve the effectiveness of the M&E system. All indicators should be selected based on the availability and feasibility of collecting accurate data on indicator values. Data should be disaggregated by sex and M&E plans should include gender indicator(s) to monitor and evaluate the effectiveness of the activity on gender issues. • Consider use of more advanced technologies, e.g., smart phones, to improve timely submission of data, and ensure data quality at the community level. Pilot test use of smart phones. • M&E designs should incorporate measures to allow analysis of and use of data for decision￾making at the field levels. To address gender considerations: • Include clear strategies, policies and guidance on how to approach gender issues and plans on how to address the men and women in targeted audiences. Strategies should consider differing religious practices and include an organizational analysis on gender balance and explore ways to increase the number of female staff. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 10 • In community based SBCC activities, gender-specific messages could be strategically included while developing and disseminating malaria or other public health related messages. 11 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV I. INTRODUCTION MALARIA IN MOZAMBIQUE Malaria is endemic throughout Mozambique, with its entire population of 28.8 million at risk. Most of the country experiences year-round malaria transmission, with a rainy season peak from December to April. Mozambique is also prone to cyclones and floods, which have likely contributed to increases in malaria transmission in recent years, particularly in low-lying coastal areas and along major rivers. Malaria is considered among the most important public health problems in Mozambique, accounting for 29% of all deaths, followed closely by AIDS at 27% (2008 Post-Census Mortality Survey). Among children under five years of age, malaria accounts for 42% of deaths. Plasmodium falciparum accounts for 90% of all malaria infections, with P. Malariae and P. Ovale responsible for about 9% and 1%, respectively.1 1 Mortality in Mozambique: Results from a 2007-2008 Post-Census Mortality Survey. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 12 The last national cross-sectional survey to measure community parasitemia prevalence was the 2015 combined Immunization, Malaria, and HIV/AIDS Indicator Survey (IMASIDA). This survey showed that under-five parasitemia by Rapid Diagnostic Test (RDT) varied from 2% in the capital, Maputo, to 68% in Zambézia Province. Malaria is a rural disease in Mozambique: prevalence in rural areas was over two times higher than in urban areas (47% versus 19%, respectively). Malaria cases reported through routine health information systems increased from 2013 through 2015. This disappointing increase may be partly due to the increased quality of community-based care, and increased demand for medical care and better diagnostic procedures. Other factors may have contributed to this increase, such as the way bed nets are used, vector insecticide resistance, or exposure outside of bed nets. Despite the increase in total cases of malaria, the number of deaths due to malaria decreased by 24% (from 2014-2015), possibly due to the increased quality of community-based care, increased demand for medical care and better diagnostic procedures. 2 PARTNERSHIP BETWEEN NMCP AND PMI The goal of the President’s Malaria Initiative (PMI) is to reduce malaria-related mortality in high burden countries through scale-up of four proven and highly effective malaria prevention and treatment measures: 1) Insecticide-treated mosquito nets (ITNs); 2) indoor residual spraying (IRS); 3) accurate diagnosis and prompt treatment with artemisin-based combination therapies (ACTs); and, 4) intermittent preventive treatment of pregnant women (IPT). 3 Mozambique was selected as a PMI focus country in 2007. Since then, PMI has been working closely with the National Malaria Control Program (NMCP). PMI develops its implementation plans in consultation with the NMCP and with the participation of national and international partners involved in malaria prevention and control in the country. Activities supported by PMI are intended to mesh with NMCP strategies, and those of other partners, to improve and expand malaria-related services, including the Global Fund to Fight AIDS, Tuberculosis, and Malaria. Access to measures for malaria prevention, diagnosis and treatment has been increased through efforts led by the NMCP and supported by PMI and the Global Fund. However, barriers to utilization persisted. In particular, there was a need for increased community-based efforts to enhance understanding and adoption of malaria tools to optimize the impact of these investments. The high prevalence of malaria throughout the country, particularly in children under five years of age and pregnant women, coupled with the increase in access to malaria prevention and treatment tools, underscored the need for strengthening of social and behavior change communication (SBCC) to encourage malaria prevention and treatment utilization. ACTIVITY DESCRIPTION In 2015, USAID/Mozambique’s Integrated Health Office (IHO) awarded PIRCOM (Interreligious Program Against Malaria), a three-year cooperative agreement, valued at $1,574,247. Implementation 2 Inquérito de Indicadores de Imunização, Malária e HIV/SIDA em Moçambique (IMASIDA) (Immunization, Malaria and HIV/AIDS Indicator Survey, 2015) 3 President’s Malaria Initiative, Mozambique, Malaria Operational Plan FY 2017. 13 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV runs from November 2015 to November 2018. PIRCOM, a faith-based organization based in Maputo, is committed to improving the lives of the Mozambican population by mobilizing communities to strive for the elimination of malaria. PIRCOM’s main objective with respect to malaria is to contribute to the PMI and NMCP goals of reducing morbidity and mortality due to malaria, particularly among pregnant women and children under five years of age, through communication activities for behavioral change. Since its inception in 2007, PIRCOM has been receiving assistance from PMI to promote communication for social and behavioral change for the prevention and treatment of malaria at the community level. Between 2007-2010, PIRCOM received PMI funding through the Adventist Development and Relief Agency (ADRA) to implement such activities. From 2010 - 2012, PMI funded PIRCOM through the C￾Change Project and between 2012 - 2014, the organization continued to receive PMI support through Population Services International (PSI). Over the years, PMI’s average financial assistance to PIRCOM has been around $ 350,000 per year. PIRCOM aims to: • Increase access to information about malaria prevention and treatment in Mozambican communities, especially among at-risk populations. • Increase capacity of faith leaders at the national, provincial and district levels to lead, manage and sustain malaria prevention activities in their communities. • Make use of its extant multi-religious network to mobilize all leaders designated by local communities as faith leaders, • Disseminate malaria messages and promote malaria discussions through sermons at religious￾based places of worship, interpersonal communication in the communities, community-based home visits and through dissemination of SBCC mass media messages on malaria prevention and treatment using CRs. The activity’s main objective is to contribute to achieving the PMI and NMCP goals to reduce morbidity and mortality due to malaria, particularly among pregnant women and children under five-years, through community-level communication activities for behavioral change. PIRCOM increases capacity of RLs to lead and sustain malaria prevention activities in their communities. PIRCOM makes use of its existent religious network to individuals designated by local communities as RLs. PIRCOM disseminates malaria messages through interpersonal communication in the communities through home visits conducted by volunteers and disseminates mass media messages on malaria prevention and treatment using community radio. PIRCOM is active in 29 priority districts distributed among five provinces, as Figure 1 details. Figure 2 indicates the areas of analysis for this evaluation. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 14 FIGURE 1: PIRCOM IMPLEMENTATION PROVINCES AND DISTRICTS Province Priority districts Gaza Guijá, and Chòkwé Inhambane Inharrime, Jangamo, Panda, Massinga, Maxixe, and Inhambane City Sofala Búzi, Caia, Marromeu, Cheringoma, Maríngue, Dondo, and Nhamatanda Zambezia Ile, Inhassunge, Lugela, Morrumbala, Mocuba, Namacurra, and Nicoadala Nampula Moma, Angoche, Ribaué, Monapo, Murrupula, Nacala-Porto, and Malema FIGURE 2: PIRCOM IMPLEMENTATION DISTRICTS AND DISTRICTSINCLUDED IN THE EVALUATION Map subtitles: • Yellow – PIRCOM geographic focus • Orange – selected sampling districts 15 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV II. EVALUATION PURPOSE, OBJECTIVES AND QUESTIONS The SBCC Final Performance Evaluation is intended to assess the effectiveness of the malaria messages at influencing target audience malaria knowledge and practices and to inform future USAID activity design. Use of the evaluation’s results are intended to inform USAID’s social and behavior monitoring and evaluation (M&E) systems. The intended audiences for this report include USAID/Mozambique, PMI/Mozambique, PIRCOM, Mozambique MOH and other international and national organizations involved in malaria prevention and treatment in Mozambique. The evaluation SOW is found in Annex A. Figure 3 lists the evaluation objectives and questions. FIGURE 3: SBCC EVALUATION OBJECTIVES AND QUESTIONS Evaluation Objectives Evaluation Questions A. Determine the main barriers and successes that were identified during implementation I. What were the principal barriers and successes during implementation of the PIRCOM scope of work? a. How were the barriers addressed by the program? b. In what ways did PIRCOM build upon successes to improve implementation? c. How did PIRCOM work with other stakeholders (MISAU and other IPs) in coordinating interventions in the same geographic area? B. Understand the effectiveness of the malaria messages at influencing target audience malaria knowledge and practices 2. How effective were the PIRCOM malaria messages and delivery at influencing target audience malaria knowledge and practices? C. Determine the effectiveness of the PIRCOM M&E system 3. How effectively did the PIRCOM M&E system function? a. How could the PIRCOM and other similar projects’ M&E systems be improved? b. How effective was monitoring of community radio PIRCOM messages? Are there ways to improve such monitoring in future programs? USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 16 III. EVALUATION METHODOLOGY EVALUATION TIMELINE The evaluation was conducted between February and May 2018, covering the period from the activity’s inception in November 2015 through December 2017. In-country evaluation activities were conducted between 26 February and 7 April 2018. TEAM COMPOSITION, RESPONSIBILITIES AND ROLES The evaluation team was comprised of short-term consultants, fieldwork staff, and MMEMS full-time staff. MMEMS provided required office space, equipment, support staff and transport for all parties. MMEMS Senior Evaluation Officer, Luis Reves, provided oversight throughout the evaluation. He was responsible for the overall client management and providing quality assurance and technical oversight. The core team included Pinar Senlet (Team Leader) and Ritva Parviainen (Senior Health Sector Specialist). The team was supported by statistician Carlos Lauchande, MMEMS Gender Advisor Gaia 17 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Segola, and MMEMS Data Manager, Rohit Jayenda, who was responsible for the automating the quantitative instrument through data collection tablets, using the Fulcrum platform, and ensuring integrity and consistency of the electronic data collection process. The team was accompanied by two translators/interviewers to assist collection of qualitative data. A team of six field supervisors supervised quantitative data collection conducted by 35 field enumerators. All supervisors and the Inhambane enumerators were trained by Mr. Reves in Maxixe. Two supervisors in Nampula and two in Zambézia then replicated the training for enumerators in those provinces. The supervisors ensured quality data collection with oversight from the Maputo office through daily data review. EVALUATION METHODOLOGY OVERALL APPROACH The evaluation methodology began with completion of the Getting to Answers table, presented in Annex B. The table indicates the sources and methods to gather data and the analytical techniques applied to answer the evaluation questions. The resulting mixed-method approach is detailed in this section. Analysis of complementary quantitative and qualitative data lead to findings that are based on facts and evidence, supporting valid conclusions; which in turn helped the team to provide PMI and the IP recommendations. The evaluation methodology included different data collection techniques such as 1) review of existing secondary data; 2) semi-structured key informant interviews; 3) focus group discussions; and, 4) a knowledge, attitudes, and practices (KAP) survey. The focus of all data collection activities was to answer the evaluation questions formulated by PMI and included in the SOW. The evaluation involved the systematic integration of various kinds of data, drawn from the same activity through different methods. By approaching the same question from more than one perspective, the evaluation team could compare and contrast the results from these different methods, a process known as triangulation. The evaluation model followed USAID Evaluation Policy and performance evaluation best practices. DATA SOURCES QUALITATIVE APPROACH AND DATA SOURCES: 1. Secondary sources included PIRCOM records, reporting documents and other useful technical references. The core team began by conducting a detailed desk review of activity documents and expanded to additional documents to perform proposed analyses. The list of documents reviewed is in Annex C. 2. Key Informant Interviews (KIIs): KIIs provided insights into the effectiveness of PIRCOM approaches, successes, and gaps, with related interventions. Semi-structured interviews with a wide range of stakeholders and beneficiaries were conducted in Maputo, in selected provinces, districts, and communities. PMI, PIRCOM, MOH staff and staff of other malaria stakeholders were USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 18 interviewed in Maputo. At the provincial and district levels, PIRCOM, MOH staff, and community radio stations were among key informants. In the communities, community leaders and PIRCOM supervisors were interviewed. The list of KIs interviewed is found in Annex D. 3. Focus Group Discussions (FGDs): The field survey team followed a standardized FGD protocol with various cadres of PIRCOM activity beneficiaries in communities to gain an in-depth understanding of their experiences with activity interventions. Information on the effectiveness of interventions, successes, barriers, and gaps was collected through FGDs. Four groups of FGDs were conducted in each of six targeted communities, with: i. Pregnant women and mothers or female caretakers of children under 5; ii. Husbands of pregnant women and fathers or male caretakers of children under 5; iii. Community leaders; and, iv. PIRCOM volunteers. QUANTITATIVE APPROACH AND DATA SOURCES 1. The evaluation team analyzed quantitative data from PIRCOM’s activity M&E system and annual and quarterly reports to determine the trends and progress achieved toward the activity’s objectives. The analyses examined data collected on process indicators from the inception of the activity to the most recent quarter available at the time of the evaluation. 2. The quantitative approach followed knowledge, attitudes, and practices (KAP) survey principles to measure the effectiveness of activity interventions to improve the KAP of target audiences to reduce malaria mortality and morbidity. To evaluate the effect of the messages, two groups were compared, namely the beneficiaries of the activity, and a group of households that were not exposed to messages. The KAP survey also analyzed any misconceptions, and gender issues that may present obstacles to the implementation and potential barriers to behavior change. The KAP survey protocol describing all steps of the survey, including the sampling plan, is in Annex E. Figure 4 depicts total number of respondents contacted by KIIs, FGDs and the KAP Survey. FIGURE 4: TOTAL NUMBERS OF RESPONDENTS BY KIIS, FGDS AND KAP SURVEY Respondents Total Male Female KIIs 54 45 9 FGDs w/ Women 48 48 FGDs w/Men 30 30 FGDs w/Community Leaders 34 33 1 FGDs w/PIRCOM Volunteers 25 13 12 Total qualitative 191 121 70 19 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV KAP Survey Respondents in Project Districts (PDs) 658 267 391 KAP Survey Respondents in Non￾Project Districts (NPDs) 658 236 422 Total quantitative 1316 503 813 SELECTION OF PROVINCES, DISTRICTS AND COMMUNITIES The activity works in 29 priority districts, distributed among five provinces with particularly elevated levels of malaria prevalence. In consultation with PMI, the evaluation team focused its data collection on three of these provinces: Nampula, Zambézia, and Inhambane. Nampula and Zambézia were selected because they are two priority provinces for the PMI. Inhambane was included to seek insights, based on its diversified socio-religious spectrum in the south and PIRCOM’s long history of implementation in the province. The sampling design for the KAP survey chose two districts in each selected province for data collection, for a total of six districts to study where PIRCOM intervened. The selection criteria for these “intervention districts” included the following: • Typology (both rural-urban districts); • Location (both coastal and inland districts); • Existence of community radio (at least one district has access to broadcast of community radio); and; • Number of households (inclusion of districts with both large and small numbers of households). To measure the effectiveness of activity interventions, six other districts where PIRCOM is not active were selected to allow comparison. These “non-PIRCOM districts” where chosen for their similar typology and location as the intervention districts. The evaluation used qualitative approaches in the same provinces, but in fewer districts and communities where the interventions took place. The same criteria were applied for the selection of districts and communities. Figures 5 and 6 show the districts and communities selected for qualitative and quantitative (KAP survey) data collection, respectively. FIGURE 5: SELECTED PROVINCES, DISTRICTS AND COMMUNITIES FOR QUALITATIVE DATA COLLECTION Province District Communities Nampula Murrupula Murrupula Sede Chinga USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 20 Province District Communities Zambézia Mocuba Muaquiua Munhiba Inhambane Inharrime Nhanombe Chacane FIGURE 6: SELECTED PROVINCES, DISTRICTS AND COMMUNITIES FOR QUANTITATIVE DATA COLLECTION Provinces PIRCOM Districts (PDs) Communities Non-PIRCOM Districts (NPDs) Communities Inhambane Maxixe Malelane 2, Malelane 3, Maxixe-Sede, Nhamaxaxa Vilanculos Belane, Mapinhane, Município de Vilanculo, Vilanculo Sede Inharrime Chocane, Dongane, Nhanombe, Nhanonbe Morrumbene Gotite, Mucodoene, Sitila, Malaia Zambézia Mocuba Muaquiwa, Mugeba, Munhiba, Namajavira, Vila Sede Gurué Gurue, Incize, Mepuagiua, Mugaveia, Tetete Inhassunge Bingagira, Chirimane, Gonhane, Mucupia Pebane Malema, Mulela, Chichanga, Vila de Pebane Nampula Morrupula Cazuzu, Chinga, Mulhaniua, Murrupula Sede, Nacocolo, Namiope, Namitotelane, Nihessiue, Rovuma 2, Vila Sede Eráti / Namapa Alua Sede, Muanona, Namapa Sede, Namiroa Sede, Odinepa, Samora Machel, Vila de Namapa Moma Jagoma, Mirrupi, Mirupi, Moma Sede, Naicole, Pilivili Mossuril Lunga Sede (Ampita), Matibane Sede, Nacucha, Namitatari, Vida Nova, Vila de Mossuril (Chocas Mar) DATA COLLECTION TOOLS AND INSTRUMENTS The team developed data collection tools and instruments for all data sources, including a questionnaire for the KAP survey, semi-structured KII guides for different audiences, and FGD guides for specific groups. All data collection tools were focused on relevant key evaluation questions. They were translated into Portuguese, pre-tested and revised prior to data collection. Data collection tools are found in Annex F. DATA ANALYSIS METHODS For qualitative analysis, the core evaluation team collected data from the KIIs and FGDs with assistance from two translators/interviewers. The team filed notes from all meetings and recorded all interviews and FGDs. The notes were shared among the team members and checked against the recorded 21 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV interviews and discussions for consistency and accuracy. The core evaluation team analyzed the findings from the KIIs and FGDs, along with the information gained from secondary sources (PIRCOM records, reporting documents and other useful PMI references) to determine before and after effects of the interventions and the extent to which PIRCOM has contributed to improving malaria prevention and treatment. Any inconsistencies between the data sets and gaps to respond to evaluation questions were identified, examined and finalized during the analysis. The KAP survey analysis included an estimation of the PIRCOM activity’s effect. Statistical tests were carried out to determine whether the differences between the intervention and non-PIRCOM groups were statistically significant. The data analysis was done using Statistical Package for Social Sciences (SPSS) 23. Essential KAP survey indicators were derived as specified in the tables in Annex H. Chi￾square tests were used to determine whether the differences between the two groups (PIRCOM and non-PIRCOM) were statistically significant. To analyze 2x2 tables, we used p-values derived from the Fisher’s Exact Test.. 4 Estimation of Cohen's d was carried out to measure the effect size.5 Since this is a mixed-method evaluation, integrating both quantitative and qualitative data, the analysis triangulated information from the different datasets. The results from different methods were compared, contrasted and validated. For the process of triangulation, the team used the approach known as parallel combination. In this approach, each data collection method was carried out in its entirety and analyzed separately. Then, the results were triangulated/synthesized as appropriate. Upon completion of the initial analysis of all data, the core team integrated results into a Findings/ Conclusions/Recommendations (FCR) matrix for each evaluation question, summarizing what was learned in the effort. The matrices included all relevant conclusions, detailed the findings on which those conclusions were based, and recommendations offered by the team – with recommendations clearly linked to specific conclusion. The findings, conclusions and recommendations were presented separately to facilitate understanding what the team learned with respect to each of the evaluation questions included in the evaluation SOW. The FCR Matrices are the analytic basis for the evaluation report. The team made sure that there were no recommendations in the report that could not be traced back to previously￾stated conclusions and underlying findings. Important differences in men’s and women’s participation and benefits were considered in a separate matrix for gender analysis. The FCRs are found in Annex G. Readers should visit that Annex if they want greater detail regarding particular conclusions or recommendations. LIMITATIONS OF EVALUATION METHODS The evaluation team acknowledges some limitations to the evaluation design. First, because non￾probability sampling methods are used for selecting districts, facilities, and communities for qualitative data collection, the evaluation will not be able to generate findings that statistically represent the larger population from which they are drawn. Second, because the key informants and focus group participants 4The Fisher Exact is a test of significance that is used in the place of chi square test in 2 by 2 tables, especially in cases of small samples. 5 Cohen's d is an effect size used to indicate the standardised difference between two means. Effect size is a way of quantifying the difference between two groups and emphasises the size of the difference. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 22 constitute primary sources of qualitative data, that information will be infleunced by personal perspectives. The KAP survey revealed what respondents report, but gaps can exist between what is reported and actual behavior. That is, for example, in some cases, respondents may claim to use bed nets, when, in fact, they do not. It should also be noted that the results from the KAP survey were representative of only the six districts; they do not represent other districts in which the activity is being implemented. Also, while the typology of non-PIRCOM districts was similar to those of intervention districts, it was not possible to find an exact match. GENDER CONSIDERATIONS The evaluation incorporated the gender dimension throughout all stages and analyzed the differences between the planned versus actual interventions. The evaluation approach followed USAID guidelines for “integrating gender equality and female empowerment in USAID program cycle” (ADS 205.3.8.). The team aimed to identify, understand, and explain existent gaps, as well as the relevance of gender norms, roles, and responsibilities, access and control over resources and power relations specific to the implementation of activities. Specific attention was given to gender considerations while collecting and analyzing quantitative and qualitative data. Both women and men were interviewed and participated in KIIs and FGDs, with one FGD including only women in each community visited. All survey data were sex-disaggregated and qualitative efforts proactively probed to understand gender dynamics within the intervention to determine how the activity affected – and how future similar activities could affect – both women and men. The evaluation examined the extent to which gender equality was promoted through the activity as well as any unexpected or unintended results (positive or negative) affecting females, males or both. The evaluation provided recommendations for future similar projects regarding the key gender inequalities or needs for female empowerment. ETHICAL CONSIDERATIONS AND CONFIDENTIALITY The team obtained verbal consent from all respondents according to USAID Evaluation Policy guidelines. Interviewees were given the option to opt out of particular questions or the whole interview. Information provided as part of interviews or discussions are not linked to any specific person in the reports or raw data sets shared with USAID. Written signed consent forms were collected from all household respondents while implementing the KAP survey. 23 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV IV. FINDINGS AND CONCLUSIONS: EFFECTIVENESS OF INTERVENTIONS Communication Channels and Malaria Messages: The PIRCOM activity implements three interrelated approaches to reach communities with malaria prevention and treatment messages. • Sermons and community meetings by religious leaders: PIRCOM trained religious leaders to disseminate malaria messages during sermons and conduct community meetings to inform community members. • Home visits by volunteers: PIRCOM trained volunteers to conduct home visits and interact with household members to disseminate malaria messages. • Dissemination of malaria messages via CRs: PIRCOM collaborated with community radio stations in producing and airing radio programs broadcasted in Portuguese and local languages. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 24 Via these channels, PIRCOM disseminates the following five malaria prevention and treatment measures, considered by PMI to be proven and highly effective: 1. Sleep each night under a mosquito net treated with insecticide, especially pregnant women and children under 5 years old. 2. Allow teams to enter and carry out IRS inside houses to kill the mosquitoes that transmit malaria. 3. Pregnant women should go to the ante-natal clinic where they will receive IPT at least three times, to prevent and/or treat malaria for her and the baby. 4. Seek immediate treatment at a health clinic during the first 24 hours after the first signs of malaria, such a fever, headache, lack of appetite, shivers, nausea, diarrhea, convulsion, and pain in the back or in the joints. 5. People infected with malaria should take medicine as advised by a nurse/doctor. The evaluation team found that all three approaches implemented by PIRCOM were potentially effective and complemented each other. In practice, however, qualitative data collected by the team indicated that dissemination of malaria messages via RLs was shown to be more effective than the other approaches: RLs: RLs are highly respected and trusted individuals within communities. Along with their well￾established networks and strong ties with communities, RLs have the potential to promote and sustain positive changes in the social norms, attitudes, and behaviors of their communities. According to PIRCOM records, since the activity’s inception, PIRCOM has trained over 528 RLs from a variety of faiths on malaria prevention and treatment. These RLs have, in turn, reached congregants across PIRCOM districts/communities with the life-saving messages. Through PIRCOM, the RLs established a powerful community network - one that can complement national and multinational efforts against malaria. Another aspect relevant for behavior change is the consistency of the reception of the messages – those going to church/mosque most of the time have a consistent participation in the communication event, thus allowing for a “buildup approach.” In all FGDs conducted by the evaluation team, participants mentioned the RLs at the churches and mosques as source of malaria information. In addition to the sermons, RLs also conducted “informational meetings” with community members to discuss and disseminate malaria messages. The evaluation team did not find any significant differences between the religions of the RLs (Christians versus Muslims, or between the different sects of Christian leaders) in their effectiveness in disseminating malaria messages. The evaluation team was impressed that PIRCOM Muslim and Christian RLs appeared to work together in harmony, regardless of their faith. Interviews with MOH officers revealed that while RLs were reported to have been highly effective in disseminating malaria messages, the potential of the network is not fully utilized under the current activity. Currently, RLs disseminate malaria messages only in districts and communities where the activity is being implemented, while the network has the capacity to reach out to broader audiences. 25 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV “The RLs are very instrumental in transmitting messages, perhaps the most efficient channel. There should more of them- and more of the churches. PIRCOM is the representative of the churches and they can reach out to more.” — Central MOH Official Volunteer Home Visits: Home visits conducted by volunteers are also a potentially effective peer-to￾peer interpersonal communication channel to reach intended audiences. To date, PIRCOM has trained about 1040 volunteers to conduct home visits. FGD participants whose homes were visited by the volunteers identified these visits as a useful source of information about malaria. However, the team found that the effectiveness of home visits is limited, due to their low coverage. The team could not identify a standard and strategic pattern in how home visits were planned and organized in the communities. PIRCOM delegates, supervisors and volunteers interviewed could not explain how the households (HH) were selected, or the coverage of HHs in a community. Volunteers were required to make 8 visits per month, and during the last year of the activity, the number of visits were reduced to three per month.6 The rationale for the reduction is that during the last year of the activity, volunteers are required to revisit only the HHs with pregnant women and children under 5. However, there is no standard approach to how these revisits are planned or organized. Reportedly, volunteers each were required to visit between 10 to 40 HHs during the lifetime of the activity, with the number varying greatly across the communities. The team attempted to estimate the proportion of HHs visited by volunteers in communities. However, information on total number of HHs in the communities was not available. Instead, we estimated coverage within the districts. It is estimated that 2.8% of HHs are visited in Murrupula, 2.9% in Inharrime and 7.2% in Mocuba. 7 A common finding in all districts visited by the team, and voiced by virtually all respondents, is that the motivation of volunteers is low, due to lack of monetary incentives or stipend. The challenge is discussed in detail below. Community Radios: PIRCOM has partnered with the following community radio stations (CRs) since the inception of the activity: Mocuba, Inharrime, Sofala and Guija community radios; Radio Progresso in Maxixe; Licungo Radio in Ile; and HAQ Radio in Nampula. The evaluation had interviews with Radio Progresso, HAQ Radio and Mocuba CRs during field trips. CRs are an important media platform for disseminating information, with over 50% of rural households owning at least one radio. However, the team was not able to find evidence that CRs have been effective in disseminating malaria messages. Although PIRCOM claims that more than 700,000 people have been reached with malaria messages during the life of the activity, the team was not able to verify this figure. According to IREX, 8 none of the CRs have conducted an audience research to determine 6 Munhiba community in Mocuba, Zambezia is an exception. In Munhiba, reportedly, volunteers are visiting 15-25 houses per month. 7 These are rough estimates, based on the number of volunteers multiplied by visits per volunteer, divided by total number of HHs in the districts using the 2017 census projections (CITE). Censo 2017, IV Recenseamento Geral da População e Habitação, Divulgação dos resultados preliminares, IV RGPG 2017, Instituto Nacional de Estatística, 30 Dezembro 2017. 8 IREX implements the Media Strengthening Program (MSP) funded by USAID. One of IREX’s tasks is to provide technical training to the community radios to serve their audiences more efficiently. They are currently working with 40 community radios in Mozambique, in the northern and central parts of the country. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 26 their exact coverage nor who is actually listening to them. Such research is expensive and beyond the capacity of the CRs. In principle, a community radio broadcasting covers an area of 60-75 km. in radius. In actuality, however, coverage is likely less than 40 km. Stations’ reach declines as broadcasting equipment wears out. For example, Progresso Radio in Maxixe can be accessed only in Maxixe town and some communities surrounding the town. Radio HAQ in Nampula is an exception because it can reach a larger audience than the other CRs. In FGDs conducted by the team, participants in six FGDs mentioned that they have heard malaria messages from the CRs, in another six FGDs, they had heard the messages from the national public radio. In one FGD, no participants had radios. It should be noted, however, that some participants were not able to differentiate between the radio stations they have been listening to or could not remember from which station they heard about malaria. Monitoring of CRs is discussed in more detail under the effectiveness of the PIRCOM M&E system in the following sections of this report. Throughout the qualitative data collection process, the evaluation team found that PIRCOM malaria work is highly regarded by all respondents who are familiar with the activity and who have been exposed to malaria messages. When asked about specific examples of the effectiveness of the interventions, respondents provided valuable insights. FGDs with community leaders, volunteers, men and women revealed that malaria preventive and treatment practices have improved because of PIRCOM interventions. In all FGDs, participants mentioned that practices – including sleeping under mosquito nets, accepting indoor spraying, demand for ante-natal care and IPT, and seeking immediate treatment – have increased. FGDs provided valuable information on how community members benefitted from PIRCOM messages. “Before PIRCOM, we had the mosquito nets but we didn’t know they were important and how to use them. We got the nets from the hospital —- PIRCOM explained us how to use them.” — Women’s FGD in Muaquia. The positive responses regarding the effectiveness of PIRCOM interventions from the communities were echoed in interviews with PMI, MOH and PIRCOM staff at higher levels. Respondents also indicated that the activity has been effective in improving malaria knowledge and treatment behaviors in communities. However, some respondents also shared their doubts about their opinions due to lack of hard evidence and reliable data on the effectiveness of PIRCOM interventions. “I believe that PIRCOM was effective targeting the specific groups but I am not sure as we don’t have the exact data.” — USAID staff “PIRCOM is successful in disseminating messages on malaria but are the messages leading to behavior change within the communities? We don’t really know. There needs to be a special study to examine if the efforts are actually leading to behavior change.” — MOH Staff, Inhambane. The evaluation team triangulated and verified the anecdotal evidence collected on the effectiveness of PIRCOM interventions from the analysis of qualitative data with the findings from the KAP survey. The 27 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV following are findings from the KAP survey on the effectiveness of PIRCOM messages on improving malaria knowledge and behavior change for prevention and treatment of malaria. EFFECTIVENESS IN IMPROVING MALARIA KNOWLEDGE FIGURE 7: KNOWLEDGE ON HOW MALARIA IS TRANSMITTED AND PERCEPTION OF MALARIA RISK The knowledge that malaria is transmitted by mosquitos was significantly higher in PIRCOM implementation districts compared to Non-PIRCOM districts. (P<0.001). Similarly, a significantly higher percentage of respondents in PIRCOM districts knew that everyone is at risk for malaria, compared to non-PIRCOM districts. (P=0.039). When disaggregated by gender, the findings were the same for males and females. FIGURE 8: KNOWLEDGE OF GROUPS AT HIGHEST RISK Knowledge that pregnant women and children under 5 are at highest risk for malaria was significantly 86.3 85.3 69.5 80.9 0 10 20 30 40 50 60 70 80 90 100 How Malaria is transmited Perception of Malaria risk PIRCOM Non-PIRCOM % 22.2 51.2 16 44.2 0 20 40 60 80 100 Pregnant mother Children under 5 Non-PIRCOM PIRCOM USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 28 higher for men (but not for women) in PIRCOM Districts compared to non-PIRCOM districts. For men, the difference was significant (P=0.022) but for women it was not significant. (P=0.149). FIGURE 9: KNOWLEDGE THAT MALARIA IS A PREVENTABLE DISEASE A significantly higher proportion of respondents in PIRCOM districts knew that malaria is a preventable disease. (P=0.001). The finding was the same for males and females. FIGURE 10: KNOWLEDGE OF HOW TO PREVENT MALARIA Knowledge of two important malaria prevention measures, sleeping under mosquito nets and spraying one’s house with insecticides, was significantly higher in PIRCOM districts. (P=0.018). It should be noted that while knowledge on mosquito nets was very high in both groups (95.8 and 92.5%, respectively), 9 knowledge of IRS was very low (10.1 and 4.3% respectively) both in PIRCOM and Non-PIRCOM districts. The findings were same for males and females. 9It should be noted that while the difference is statistically significant, the effect size is very small (0.14). According to Cohen’s d criteria, it would practically mean insignificant. 93.2 87.4 6.8 12.6 0 10 20 30 40 50 60 70 80 90 100 PIRCOM Non-PIRCOM Do Not Know Know Malaria is Preventable 95.8 10.1 92.5 4.3 0 20 40 60 80 100 Sleeping under mosquito net Spray inside of house with insecticide Non-PIRCOM PIRCOM % % 29 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV FIGURE 11: KNOWLEDGE ON ANTE-NATAL CARE INFORMATION Information that pregnant women should go to an ante-natal clinic to receive IPT to prevent and/or treat malaria was significantly more prevalent in PIRCOM districts. (P=0.003). This finding was the same for males and females. FIGURE 12: KNOWLEDGE ON SYMPTOMS ASSOCIATED WITH MALARIA Among several symptoms associated with malaria, the percentage of respondents who knew about high fever (the most common symptom) was significantly higher in PIRCOM districts. Knowledge of loss of appetite (a less specific symptom) was higher in Non-PIRCOM districts. (P=0.014). There was no significant difference between the two groups on knowledge of other malaria symptoms such as headache, shivers, nausea, diarrhea, convulsion, and pain in the back or in the joints. The findings were the same for males and females. 91.3 86 8.7 14 0 10 20 30 40 50 60 70 80 90 100 PIRCOM Non-PIRCOM No Yes 89.4 21.3 85 27.2 0 20 40 60 80 100 High temperature/fever Loss of appetite Non-PIRCOM PIRCOM % % USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 30 SOURCE OF INFORMATION The KAP survey confirmed the finding from qualitative data that PIRCOM messages were not the sole source of information on malaria prevention and treatment in PIRCOM districts and communities. In fact, the majority of the KAP respondents learned about malaria from other sources. An analysis of the “other” category revealed most of the community members learned about malaria from health care workers and facilities, followed by the national public radio, family members, and schools. These findings were same for both male and female respondents. EFFECTIVENESS IN BEHAVIORAL CHANGE FOR MALARIA PREVENTION AND TREATMENT In this section, the team examined findings from KAP survey on behaviors associated with the five PIRCOM malaria messages. FIGURE 13: BEHAVIORS ASSOCIATED WITH FIVE MALARIA MESSAGES The practice of sleeping under mosquito nets was high in both PIRCOM and Non-PIRCOM districts, and the difference was not significant. Similarly, pregnant women who sought ante-natal care, HHs who sought immediate medical treatment and those who have completed treatment were also high in both groups. However, the differences between the PIRCOM and Non-PIRCOM districts were not statistically significant. The practice of allowing teams to carry out indoor spraying was an exception. Although the proportion of respondents who allowed IRS was very low in both groups, the percentage in PIRCOM districts was significantly higher than Non-PIRCOM districts (5.9 and 2.1 percent, respectively). (P=0.001). 95.1 5.9 72.9 94.8 94.8 95.6 2.1 71.3 92.5 97.8 0 10 20 30 40 50 60 70 80 90 100 Sleep under mosquito net Allow teams to carry out indoor spraying Pregnant women who sought ante￾natal care HH who have sought medical treatment for malaria in the last 6 months (within 24 hours) HH who have completed medical treatment in the last 6 months PIRCOM Non-PIRCOM 31 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV V. FINDINGS AND CONCLUSIONS: SUCCESSES, BARRIERS AND COORDINATION WITH OTHER STAKEHOLDERS SUCCESSES DURING IMPLEMENTATION PIRCOM is a well-known and respected organization. It has been working to improve its institutional capacity since its establishment in 2007. The organization’s leadership is committed to improving activity performance and further developing its capacity as an organization. In addition to receiving support from PMI, PIRCOM is now receiving funding from other international organizations. While PIRCOM achieved significant progress since its inception 10 years ago, it is still a young organization. Institutional capacity assessments conducted by JHU10 and Eurosis11 reveal that despite progress over the years there is room for PIRCOM to grow as an organization. This activity helped PIRCOM to grow and strengthen its organizational and technical capacity. PIRCOM leadership is aware of its shortcomings and committed to growing as an institution. PIRCOM was successful in training large numbers of delegates, RLs and volunteers on dissemination of malaria messages. PIRCOM set-up and organized training programs in which first delegates were trained as trainers, and, in turn, they trained supervisors and volunteers. Following the initial training programs and based on observations from the field, PIRCOM staff had refresher trainings. Field interviews and discussions with PIRCOM staff and other stakeholders indicate that training programs have been successful. Individuals were well trained on malaria messages and knew how to disseminate them. PIRCOM has good technical knowledge and expertise in the design of interventions for social and behavior change, using this knowledge in the design of this activity. “PIRCOM does not need to be taught on SBCC because not only does its staff have technical expertise and was able to demonstrate that all major instruments have been designed based on SBCC.” — Evaluation of the Institutional Capacity of the PIRCOM in SBBC, JHU. As discussed in detail under the previous section of this report, PIRCOM has been successful in improving knowledge of malaria prevention and treatment within the communities where the interventions were implemented. Regarding behavior change to prevent malaria, PIRCOM’s most significant success was mobilizing communities to accept IRS, which led to an almost three-fold increase in intervention districts compared to non-intervention districts. BARRIERS AND CHALLENGES ORGANIZATIONAL AND MANAGEMENT CHALLENGES In 2017, Eurosis conducted a needs assessment of PIRCOM to support opportunity identification for institutional capacity development under USAID’s Mozambique Local Partner Support Program. The assessment analyzed the organization’s strengths, weaknesses and challenges, leading to the development 10 JHU Center for Communication Programs, funded by USAID, works in Mozambique in the field of communications to adopt healthy behaviors. 11 Eurosis is a local business management consultancy and training organization. Eurosis receives funding from USAID for local capacity development services. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 32 of the PIRCOM Institutional Capacity Building Action Plan. The assessment identified opportunities for intervention to strengthen organizational capacity, particularly in the areas of financial management/internal control, project and monitoring and evaluation management, management of human resources, and gender issues and awareness. At the time of the evaluation, Eurosis was assisting PIRCOM in implementing a capacity development plan. The evaluation did not specifically analyze PIRCOM’s management, but during field trips the team observed challenges in line with the Eurosis institutional capacity assessment. The provincial coordinators were the only individuals in the provinces to oversee province-wide implementation, and they were overwhelmed with the task of entering data from paper forms into Access databases. The organizational structure, particularly at the district level, seemed to be complex and somewhat “loose.” Roles and responsibilities for delegates were not very clear and most individuals had multiple responisbilities. For example, all district delegates were also RLs. The team met with supervisors and volunteers, who were also RLs. Decision-making processes were not always well understood, and flow of information was complex. Challenges regarding the M&E Plan and functionality are discussed in detail in further sections of the report. “I coordinate all PIRCOM activities in the province, conduct monitoring visits and training. But a great part of my time, almost 90%, goes to processing of data received from the field.” — PIRCOM Provincial Coordinator An overall conclusion on PIRCOM management is that the organization is highly cost-conscious, typically opting for the least-cost options while implementing the activity. While this is an admiral trait in general, the strategy to cut down on costs does not appear likely to pay off well in the long run on this activity. Several examples are explained in this report, including decisions to rely on unpaid volunteers for critical work, purchasing low quality bicycles, and deciding not to introduce collection of monitoring data through smartphones. LOW VOLUNTEER MOTIVATION Volunteers do not receive monetary incentives, which leads to low motivation in carrying out home visits. In only one of the six communities visited in the qualitative sample did the team observe high motivation. PIRCOM volunteers work in the same districts where other IPs make incentive payments to their volunteers, further complicating the issue. Volunteers working for other organizations/activities were paid within the range of minimum wage in Mozambique. The team was told that well trained and experienced volunteers are quitting PIRCOM to work for other organizations. Meanwhile, it should be noted that PIRCOM volunteers’ workload is much less compared to volunteers working for other organizations. PIRCOM volunteers roughly work for an hour in a week (3-8 visits a month is about 1.5- 4 hours a month) while other paid volunteers are expected to work upwards of 40 hours a week. Inspite of the light workload, it appears that volunteers are reluctant to work without pay. “The volunteers are parents and have families to sustain. It would be very helpful and serve to improve motivation if you could provide a small incentive.” — JHU representative “My wife takes care of me and sends me to work every morning. In the evening I can’t bring anything back to her.” — Volunteer in Inharrime 33 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV PIRCOM cannot pay the volunteers under the current budget, due to the large numbers of volunteers. In total, PIRCOM has about 1,040 volunteers. Even a minimum amount of monetary stipend/incentive would require half of PIRCOM’s overall budget, according to the Eurosis assessment of PIRCOM financial resources. It should also be noted that not only volunteers are not paid: likewise, PIRCOM field staff, including RLs, delegates and supervisors, are unpaid. The only paid staff in provinces are the Provincial Coordinators. Transportation within the communities, and across the communities, districts and provinces, continues to be a challenge due to long distances and costs. Volunteers complain about the long distances while supervisors and delegates gripe that they must pay out of their pockets to travel to the communities. District delegates also must travel to the provinces to deliver reports. PIRCOM provided bicycles to RLs and volunteers to ease transportation and as an incentive in 2016 and 2017. In 2016, 558 bicycles were distributed to RLs and in 2017, 971 bicycles were provided to volunteers. Regrettably, bicycles purchased were of poor quality, and most of them broke down during the first year of use. In Murrupula, Nampula in the FGD with 23 volunteers, only 3 reported that they had functional bicycles. In Inharrime, Inhambane, volunteers said that they were unable to use the bicycles at all because they were of such bad quality. The situation in Zambézia was better, as most of the bicycles were still functional at the time of the evaluation. CONFUSION OVER THE SYMPTOMS ASSOCIATED WITH MALARIA One of the PIRCOM messages is, “To seek immediate treatment at a health clinic, during the first 24 hours after the first malaria signs, such as fever, headache, lack of appetite, shivers, nausea, diarrhea, convulsion, and pain in the back or in the joints.” While these are the most common symptoms of malaria, none of them is specific to malaria. All can be related to many other conditions/illnesses. Community members who experienced any of these symptoms report believing they had malaria and seeking malaria treatment. At the health facilities, they were not diagnosed with malaria and were not given malaria treatment. During field trips, the evaluation team heard many complaints from community members and volunteers that the clinics did not have malaria medications, and they were not treated. The team discussed the frequency of stock-outs with MOH officers. The MOH officers claimed that there are no issues with diagnosis of malaria at the health care facilities, and no shortage of malaria medication, except for probably in the cases of small pockets of hard to reach areas. According to MOH, the confusion stems from the message about malaria symptoms; people who have any of the malaria associated symptoms want to receive medication. The team was not able to clarify the situation since we did not visit health care facilities. It is possible that at least some of the misunderstanding is due to health workers not adequately explaining to patients that they do not have malaria and do not need medication. “Malaria medicines are not always available. We tell people to go to the clinics when they have symptoms, and they do, but cannot receive medicines. Communities ask us to bring them malaria medicines, which we can’t deliver.” — Volunteers, Murrupula “Our health system doesn’t have a problem with diagnosis of malaria and we have medicines in stock except for some isolated areas. The problem is with the message about malaria symptoms; people who USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 34 have any of the malaria associated symptoms want to receive medication. The message is misleading and should be revised.” — Central MOH Officer CHALLENGES REGARDING THE ACTIVITY DESIGN The PIRCOM activity budget is relatively small: $1,574,247 over a period of three years to implement interventions in 29 districts in 5 provinces. The large number of provinces and districts means the activity is spreading limited resources too thinly, constraining effectiveness and efficiency. The coverage of communities/ households reached by PIRCOM is low within the districts. The strategy to reach out to so many districts, while the coverage is so low in target districts, is not effective, and this fact is acknowledged by PIRCOM staff interviewed by the team. If the activity had focused on fewer high￾prevalence provinces and districts, and strategically planned to reach out to cover a significant proportion of communities in each district, its effectiveness could have been higher. “It’s a challenge for the program to reach out to more people. So far, we are covering few districts and coverage within the districts are low.” — PIRCOM staff, Nampula “PIRCOM interventions are effective but coverage is a challenge. There are 14 districts in Inhambane and PIRCOM works in only five of them, and does not cover the entire district, only some communities.” — MOH staff, Inhambane COORDINATION WITH MOH AND OTHER IMPLEMENTATION PARTNERS PIRCOM’s coordination with the MOH at the central, provincial and district levels was generally good but largely depended on individuals. In provinces and districts where the MOH officials were keen on coordinating with IPs, PIRCOM was reported to have always been responsive. In others, where MOH was not as proactive approaching the IPs, coordination was not as strong. In two provinces MOH officers stated that they would like to have stronger coordination and collaboration. When asked in what areas and how, it became clear that the officials actually wanted to have more “control” over the organization’s activities and resources. For example, one provincial officer wanted information on the PIRCOM’s daily activities and another wanted information on its financial resources. At the central level, MOH informants reported being content with PIRCOM’s activities, level of coordination and responsiveness. PIRCOM participates in coordination meetings and is an active member in technical group meetings. PIRCOM regularly reports to MOH. During qualitative field work, the evaluation team observed close coordination and collaboration between PIRCOM and the community leaders. 12 Community leaders interviewed were familiar with and interested in PIRCOM activities; they wanted to be trained and to disseminate malaria messages. Community leaders are highly respected senior community members and have close ties with the communities. They were particularly instrumental in mobilizing communities during campaigns for distribution of mosquito nets and IRS. Community leaders have many strengths and could have played a 12 Community leaders are traditional chiefs legitimized by the community members. Since 2000 (decree N. 15/2000) they are legal representatives of the State and recognized as part of the administrative decentralization process and receive a subsidy from the government. 35 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV more significant role in disseminating malaria messages as well if they were directly involved in PIRCOM activity design. Not involving community leaders directly in the PIRCOM activity is a missed opportunity. PIRCOM has been working with Abt Associates in six districts in Zambezia on IRS and was instrumental in mobilizing communities and organizing households for spraying. Abt Associates reported that awareness and acceptance of IRS is still very low in many communities in Zambézia, and PIRCOM’s efforts to disseminate information was valuable. To influence social change at the community level, UNICEF works with PIRCOM to roll out cascading training on health promotion, nutrition, child protection and education key priority behaviors. Coordination with UNICEF has been productive and PIRCOM now receives funding from UNICEF for training of trainers’ activities and training of volunteers. UNICEF also funds advocacy meetings with RLs at the highest level, with the objective of sensitization of those leaders to promote their involvement in the health promotion and education programs. The coordination between PIRCOM and JHU in Zambézia was also successful. JHU supported capacity building and monitoring of PIRCOM activities through a framework of community dialogue project in four districts from 2015 to 2017. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 36 VI. FINDINGS AND CONCLUSIONS: EFFECTIVENESS OF THE M&E SYSTEM DESIGN OF THE M&E PLAN The activity M&E Plan initially included 17 process indicators during the first year of implementation, which was reduced to 12 during the second year. 13 The reduction in number of indicators was a positive step toward simplifying the M&E plan because not all 17 indicators were relevant or useful and there were a few repetitive indicators. All of the current 12 process indicators are quantitative, providing data on such items as the number of RLs completing training, number of community members reached with malaria messages through home visits, number of IEC materials reproduced, etc. PIRCOM reports on all process indicators and tracks progress against set targets. 13 The evaluation team examined the Performance Framework tables which were attached to the Activity’s Year 1 Year 2 Workplans for FY 2016 and 2017. 37 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Each process indicator has a matching indicator, what PIRCOM refers as “outcome indicators”, defined as the proportion of the total. For example, the process indicator “number of volunteers completing training on malaria prevention and treatment” has a corresponding output indicator defined as “proportion of volunteers trained to disseminate malaria messages”. The activity does not report on any of the output indicators because the information on the denominators is not available. PIRCOM does not disaggregate data by sex in any process indicator, e.g., regarding community members reached by sermons or home visits. The form for RLs to complete after their sermons includes the number of men and women addressed and the form used by the volunteers has a part regarding how many pregnant women there are at a household. These data are neither collected nor reported. The only data disaggregated by sex is the number of male and female RLs and volunteers trained. The evaluation team noted that the indicator on number of community members reached through sermons is subject to double counting. Discussions with RLs confirmed that about 90-95% of church/mosque attendants every week are the same community members. The indicator on the number of households who have pregnant women and children under 5 who slept under a mosquito net the previous night is has limitations because the individuals might have been sleeping under a mosquito net all the time. It cannot be attributed to PIRCOM interventions. PIRCOM also reports on an indicator – defined as the number of people reached by the CRs–that was not among the indicators listed in the M&E plan. This indicator is problematic as the values reported by PIRCOM cannot be accurate because it is not possible to estimate the number of HHs or individuals reached by radio messages in the absence of audience research. The reported numbers are estimates made by the radio stations of people living within the reach of community radios, as if everyone within the estimated coverage actually have access to the radio stations and are listening to the malaria messages. The evaluation team concludes that PIRCOM’s M&E plan is designed to monitor timely implementation of planned activities and inputs, but it does not evaluate or measure the results of interventions. A review of PIRCOM’s annual reports against workplans reveal that the activity has been delivering what it had promised, such as training of PIRCOM staff, reaching households with malaria messages, and distribution of IEC materials. “PIRCOM routinely sends us performance reports. We see the annual activities and processes in the reports but we don’t see any results. There are no output indicators and rarely an analysis of what has been achieved as a result of activities.” — Central MOH Officer DATA COLLECTION SYSTEM AND DATA QUALITY Volunteers and RLs report monthly on number of home visits conducted and number of community members attending sermons/informational meetings by manually filling in forms. Forms are collected by supervisors and sent to district delegates and finally to PIRCOM provincial coordinators. Provincial coordinators were entering data into three separate Excel sheets during the first two years of the activity. With assistance from Eurosis, data are now entered into one Access database at the provinces and sent to PIRCOM headquarters in Maputo. There were delays in reporting initially, which improved USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 38 over time. Reporting was timely during the evaluation. In only one community visited by the team did the supervisors report delays. Eurosis estimates that about 50,000 forms are collected from the communities annually and the numbers have been increasing each year. Eurosis assisted PIRCOM to mainstream data collection and suggested using smartphones in the field. The option was dropped due to cost implications ($15,000 per year) and sustainability concerns. Instead, Eurosis assisted PIRCOM to improve the efficiency of the current system by combining three Excel data sets at the provincial level to automatically produce one Access database. Using the Access database, instead of Excel spreadsheets, helped mainstream reporting from the provinces to the main office, but the change did not ease the information flow between the lower levels. The data collection system is labor intensive and time consuming, due to the large number of forms, large numbers of indicators and frequency of reporting. The forms do not contain a lot of information, e.g., many boxes are checked N/A but each form needs to be entered into the database which is time consuming. Volunteers are required to complete a form for each household, increasing the number of forms sent to districts every month. The quality of the data collected is questionable, mainly due to low capacity of the volunteers in the field and inadequate capacity at the community level to monitor quality of reporting. The evaluation team did not find evidence of quality assurance measures in the field. Supervisors are supposed to check on data quality, but it is not happening fully. Supervisors interviewed reported that they check on the completeness of the forms – that each box is filled -- and timely submission of forms. PIRCOM is aware of the poor quality of its data. The issue was among one of the organizational weaknesses identified in Eurosis’ capacity assessment. PIRCOM reported that it is currently conducting intense trainings to improve data quality, with assistance from Eurosis. DATA ANALYSIS AND USE OF DATA TO IMPROVE PERFORMANCE Data collected through the M&E system is analyzed at PIRCOM headquarters in Maputo. The analysis involves comparing the indicator values against set targets for each indicator. Based on the analysis, PIRCOM reports on progress toward targets. The fact that data are not analyzed at lower levels is a missed opportunity to use data for decision-making and improving performance. PIRCOM was able to make changes in implementation to improve the activity’s performance. These improvements included streamlining the flow of information to allow more timely reporting, strengthening the training programs and rescheduling/reorganization home visits. The improvements in PIRCOM implementation are not necessarily based on its analysis of the M&E data; they were largely based on the feedback from and observations in the field. EFFECTIVENESS OF MONITORING OF COMMUNITY RADIO MESSAGES PIRCOM has contracts with seven CRs which provide detailed instructions on how to air malaria messages, including specific guidance regarding the timing and format of airing malaria messages. Contracts are usually for 4-5 months and are renewed as they expire. The budget of a contract is around $1,000. PIRCOM does not directly monitor the CRs but has confidence in the radio stations 39 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV doing a good job. Reportedly, the radio stations themselves monitor the dissemination of malaria messages. The malaria messages are centrally developed and approved by the MOH and USAID. Radio stations translate messages into local languages. Stations have expertise in translations, no issues were reported related to the quality of translations. Stations broadcast in Portuguese and the most common local language spoken in the geographic areas within their broadcasting coverage. It should be noted that while there is only one local language spoken in Nampula, multiple languages are used in Zambézia and Inhambane. The messages were not translated to all languages in Zambézia and Inhambane. PIRCOM requires the radios to air messages at mid-morning during week days and also once during the weekends. Only in Mocuba are messages broadcast twice per week in the afternoon. The format is lecturing on malaria messages for 15-20 minutes. In Nampula and Mocuba, the stations air an “open broadcast,” where the audience can call in. Radio stations have concerns about the timing and format of the broadcasting, but they have to follow the instructions provided by PIRCOM in their contracts. Challenges cited by the stations included: • Timing of broadcasting messages: According to IREX and the CRs interviewed, the timing of airing malaria messages does not seem optimal. Mid-morning is not the peak time, and they are missing working young adults and youth, who would be most receptive to behavior change. The findings from the KAP survey confirm the concerns on timing of airing messages. Figure 14 shows that the peak time for listening the radio was between 3 pm and 6 pm in both PIRCOM and Non-PIRCOM districts. When disaggregated by sex, we found that more men than women listened to the radio during peak time. • Format of airing messages: The format used- lecturing for 15- 20 minutes- might be tedious. A combination of fewer lectures and shorter “spots,” such as advertisements for a few minutes throughout the day, might be more attractive for the intended audiences. There is room to explore other innovative and more participatory formats to air the malaria messages to attract targeted audiences. “Open broadcasting,” where listeners can call in and ask questions, appears to have been well received in Nampula and Mocuba. There are other ways to attract the audience to listen to the malaria messages. For example, broadcasting competitions on who knows best about malaria prevention/ treatments. Such formats are not used. “We are hearing that people get bored with lecturing and switch to other channels.” — Community Radio Staff USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 40 FIGURE 14: TIME-PERIODS FOR LISTENING THE RADIO The interview with IREX also confirmed concerns voiced by the radio stations that the timing and format of radio messages should be carefully considered, tailoring to the habits of targeted audiences. Other USAID IPs have requested technical assistance from IREX with training and monitoring of CRs and while developing their contracts with radio stations and IREX responded to such requests. PIRCOM did not request assistance from IREX and thus they do not have a working relationship. 6:00-9:00 9:01-12:00 12:01-15:00 15:01-18:00 18:01-21:00 21:01-23:00 41 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV VII. FINDINGS AND CONCLUSIONS: GENDER ANALYSIS In addition to addressing the three evaluation questions, the team collected and analyzed qualitative and quantitative data on gender dimensions throughout the evaluation process. Women and men were interviewed and participated in KIIs, FGDs and the KAP survey. KAP survey data were sex-disaggregated and qualitative data collection efforts proactively probed to understand gender dynamics within the interventions to determine how the activity affected both women and men. This section summarizes the findings and conclusions from a gender perspective. Initially, PIRCOM activity design did not include a gender analysis and the activity did not have a clear strategy on how to reach men and women in targeted communities, nor the main target group – pregnant women and children under 5. However, PIRCOM benefitted from the JHU capacity development assessment conducted in 2016, which included specific recommendations to raise gender awareness and the concept of gender in the organization, and mainstreaming gender in community level interventions. The results from the JHU assessment on gender were further reinforced by the findings and recommendations from the capacity development assessment conducted by Eurosis. PIRCOM has now developed a policy and guide on how to approach gender issues, to be initiated in April 2018. The USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 42 plan follows the recommendations of the capacity development assessment conducted by Eurosis. At the time of the evaluation, PIRCOM was preparing to launch gender training program. In the absence of specific orientation/guidance for the volunteers or RLs on how to address gender issues, PIRCOM’s approach to men and women largely depended on the individual volunteer or the RL in question. The evaluation team observed that PIRCOM staff were using SBCC materials developed by previous projects to reach communities. For example, PIRCOM used a job aid/poster called ´Role of the Father.´ developed by the C-Change project. The poster is used specifically to approach male community members and inform them about the importance of accompanying their wives to the antenatal care and to receive IPT. Some RLs interviewed were taking specific action to reach the pregnant women and children under 5. For example, if there are many pregnant women present at church, the RL might attend to them after the sermon. Some Muslim RLs also mentioned that they encouraged pregnant women to participate in community meetings. The qualitative evaluation findings are encouraging in the sense that the RLs and volunteers address the gender dimensions well. For example, messages regarding the importance of pregnant women and children under 5 sleeping each night under the mosquito net, and of women attending the antenatal care and receiving IPT were well understood by both women and men and appear to have been followed. The team also observed that these messages were supported by the community leaders and health care providers. Specifically, community leaders were keen to addressing women’s needs. Whatever channel used, the evaluation team found anecdotal evidence that malaria messages were taken seriously by both men and women and good practices for prevention and treatment of malaria were followed by community members. “Woman is our priority as she is our mother.” — RLs in Munhiba “Previously there were prejudices and myths against IPT. Now men understand that they have to take their pregnant women to the clinics for IPT and they do so. Men have changed their behaviors.” — PIRCOM Provincial Coordinator KAP survey results showed largely equal levels of knowledge on malaria prevention and treatment among men and women. The only differences were that more men than women listen to the radio during the peak-time and knowledge that pregnant women and children under 5 are at highest risk for malaria was significantly higher for men (but not for women) in PIRCOM districts compared to non￾PIRCOM districts. However, the community radio showed to be a less important channel for dissemination of malaria messages, in comparison to the information given by the RLs, and by volunteers through household visits, for both women and men. During the initial year of the activity, male volunteers were not always welcome and able to address women at the households when husbands were not at home. The issue was important because, in many communities, men are working abroad and are absent for long periods of time. It appears, however, that the sex of the volunteers is not important any longer, as both male and female volunteers are introduced by the community leaders and RLs as PIRCOM activists and are well recognized and accepted by the community members. 43 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV “Male volunteers are able to enter the household even if there are no men present at the moment. I don’t hear of any problems.” — PIRCOM supervisor “[Laughing] It was difficult for the male volunteers to visit homes with no men around. It’s not an issue now because everyone knows the volunteers.” — Women volunteer As to the community members´ access to malaria messages by RLs, the evaluation found that there is an interesting gender dynamic within the religious congregations: Muslim women don’t have the same opportunity as Muslim men to receive information during sermons, as few Muslim women attend the Friday prayers at the mosques. On the other hand, there are more female than male Christian community members who regularly attend church, and, thus, they have more opportunities to get information than the men who are not churchgoers. To increase access to malaria messages among Muslim women and Christian men, RLs reported that they are encouraging women and men to attend informational community meetings. In Mozambique, it is a well-known fact that traditional gender roles for men and women may affect women´s participation in decision-making and thus adversely influence women’s ability to practice malaria prevention and treatment options. The evaluation team did not find evidence that the gender norms in decision-making had a negative impact on women’s practices related to malaria prevention and treatment. Although men are the primary decisionmakers, it does not adversely affect women’s behavior change. PIRCOM has succeeded in taking care of the gender issues in an appropriate way as both male and female respondents confirmed that the malaria messages were understood, accepted and followed correctly by both men and women. In Zambézia women´s participation in decision-making seems higher compared to the other provinces, and women have more power in decision-making. ¨The head of the family is the woman – she has the command and the husband follows her options.” — Supervisor, Munhiba Among PIRCOM leaders, coordinators, delegates and supervisors there are more males than females; there are only three women in leading positions at the headquarters, and only one female district delegate out of 28 delegates. In the communities visited by the team, out of 38 RLs, 29 were males and 9 were females. Similarly, more than half of the volunteers in the communities were men. In the communities included in the qualitative inquiry, there were a total of 70 volunteers, 59% male and 41% female. In one community, there were no female volunteers. Despite the organizational gender imbalance, the evaluation team did not find evidence that the situation has led to gender differences; gender equity was promoted throughout the activity. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 44 VIII. SUMMARY OF EVALUATION CONCLUSIONS PIRCOM’S RESULTS FRAMEWORK AND SUMMARY OF EVALUATION CONCLUSIONS Result: Contribute to reduce morbidity and mortality from Malaria in Mozambique PIRCOM contributed to reduce morbidity and mortality from malaria in communities where implemented by increasing knowledge on prevention and treatment methods, and by increasing practice of IRS. SO1 - Increase knowledge and utilization of malaria prevention and treatment in religious communities, especially among pregnant women and children under 5. PIRCOM was effective in increasing knowledge on malaria prevention and treatment. Its effectiveness on utilization of practices was limited. SO2 - Increase the capacity of faith leaders at national, provincial and district levels to lead, manage and sustain malaria prevention activities in their communities. PIRCOM has been successful in increasing the capacity of faith leaders at the district and community levels to lead, manage and sustain malaria prevention activities in their communities. PIRCOM was not involved in increasing the capacity of faith leaders at the national or provincial levels. IR 1.1. Increased knowledge of malaria prevention, diagnosis, and treatment among religious communities PIRCOM has been effective in increasing knowledge of malaria prevention and treatment regarding all five malaria messages. IR 1.2. Improved use of effective malaria prevention and treatment methods in targeted communities PIRCOM has been effective in improving use of IRS in targeted communities. It has not been effective in improving use of other preventive and treatment methods. The practice of other methods was already high in targeted communities, allowing little room to improve except for small pockets of hard to reach communities. Also, behavior change takes more time then the three-year life of the activity. IR 2.1. Enhanced leadership of malaria prevention activities by faith leaders in targeted communities PIRCOM has been effective in promoting malaria prevention activities via faith leaders in targeted communities. 45 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV IX. LESSONS LEARNED AND RECOMMENDATIONS FOR FUTURE USAID ACTIVITIES Since the PIRCOM activity ends in November 2018, this report does not have specific recommendations for PIRCOM to improve the subject activity. Rather, this section is meant to inform future USAID activity designs. It should be noted that the sequence of the recommendations is not prioritized; it follows the order of the evaluation questions. To Increase the Effectiveness of Malaria Messages at Influencing Knowledge and Practices: • The reach of malaria messages, or dissemination of other public health messages, can be expanded significantly through RLs. PIRCOM has the capacity to reach out to broad communities via its extensive network of RLs who are influential in dissemination of information and promoting effective practices. • Explore the possibility of broadcasting malaria a/o other health-related messages via the national public radio, instead of through community radios. The national radio has broader coverage and is accessible to a greater number of communities. • Future designs of messaging through radios should include consultation with radio communication experts prior to contracting with the stations to ensure appropriate timing of broadcasting and use of more participatory and attractive formatting to reach the intended audiences. • The evaluation team does not have a specific recommendation on whether to use volunteers to conduct home visits in future SBCC programs. In cases where USAID decides to engage volunteers, home visits should be strategically planned and standardized to reach targeted audiences. Volunteers should receive comparable monetary incentives as provided by other organizations, their numbers should be reduced significantly, and HHs per volunteer should be increased to reasonable levels to reach out to a significant proportion of households and communities. • USAID should consider directly involving community leaders in dissemination of malaria messages, or other public health messages, in future activities. Community leaders have close ties within the communities and have a high potential in mobilizing communities to promote healthy practices. To Build on Successes and Overcome Barriers/Challenges: • Consider involving PIRCOM in future programs to increase IRS knowledge and coverage, as there is evidence that the organization has been successful in informing and mobilizing communities to increase IRS practices. • Continue to support PIRCOM organizational capacity development efforts. PIRCOM has grown significantly since its inception 10 years ago but still needs assistance, specifically in the areas of strategic management, management of human resources and strengthening of its M&E system. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 46 • Similar activities should focus resources on a smaller number of high-priority provinces and districts and strategize to reach a significant proportion of communities in each district to increase interventions’ effectiveness and efficiency. • Realize that pursuing the most least-cost option does not always pay off well in the long run. There should be a balance between efforts to contain costs and maintaining the efficiency and effectiveness of the interventions. • Work with the MOH and PIRCOM to revise and clarify the message regarding malaria symptoms to avoid future confusion. The counseling skills of the health care providers who diagnose and treat malaria should also be strengthened to avoid misunderstandings among clients seeking advice and treatment. To Improve M&E Systems: • Activity M&E plans should have balanced mix of process indicators and a reasonable number of output/outcome indicators. Process indicators are helpful to monitor an activity’s inputs and progress toward achieving targets while output/outcome indicators are needed to appreciate results. Indicators should be selected based on the availability and feasibility of collecting accurate data on indicator values. Data should be disaggregated by sex and M&E plans should include gender indicators to measure the effectiveness of the activity on gender issues. • A robust activity design, with a lean monitoring system with fewer and more appropriate indicators can facilitate the flow information and quality of the data. Frequency of reporting can be reduced from monthly to quarterly reporting. Also, there is no need to complete separate forms for each household-which would significantly help to reduce the number of forms and the workload. • Where feasible, use advanced technologies, such as smart phones, to improve timely submission of data and data quality. Use of smart phones should be pilot-tested at the community levels to inform the design of future activities. • Future activity M&E designs should incorporate measures to allow analysis of and use of data for decision-making at the field level. To Address Gender Considerations: • Future activities should include a gender analysis, strategies, policies, guidance and training programs on how to approach gender issues and plans on how to address the men and women in targeted audiences. • The gender strategy should consider the religions of the audiences and include an organizational analysis on gender balance and explore ways to increase the number of female staff. • In community-based SBCC activities, gender-specific messages could be strategically included while developing and disseminating malaria or other public health related messages. 47 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV • Separate community meetings for men and women could be expanded in Christian and Muslim Communities to target Muslim women and Christian men, who do not attend sermons. • Regarding future partnerships with PIRCOM, the number of female volunteers should be increased in communities where the majority of volunteers are male. In particular, more Muslim women could be trained as volunteers. Additional community meetings for men and women could be expanded in Christian and Muslim communities to specifically target Muslim women and Christian men, who do not regularly attend the sermons at the churches or mosques. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 48 ANNEX A: SCOPE OF WORK OVERVIEW & PROGRAM INFORMATION 1.1 Overview Virtually the entire population of 28.8 million inhabitants in Mozambique is at risk of contracting malaria. Malaria is considered the most important public health problem in Mozambique; it is the leading cause of mortality in hospitals, with a rate of 29% of all registered deaths. In 2015, malaria was responsible for 42% of deaths among children under five years of age, with 40% of those deaths among children aged 6-59 months. Access to tools for malaria prevention, diagnosis and treatment has been increased through efforts led by the National Malaria Control Program (NMCP) and supported by the US President’s Malaria Initiative (PMI) and the Global Fund. However, barriers to utilization persist. In particular, there is a need for community-based efforts to enhance understanding and increase adoption of malaria tools to optimize the impact of these investments. Low utilization of malaria-related tools is associated with high morbidity and mortality from malaria. The high prevalence of malaria throughout the country, particularly in children under five years of age and pregnant women, coupled with the increase in access to malaria prevention and treatment tools, underscores the need for strengthening of social and behavior change communication (SBCC) to encourage malaria prevention and treatment utilization. 1.2 PIRCOM activity description The implementing partner is PIRCOM (Interreligious Program against Malaria), a faith-based organization, based in Maputo, with offices in each province in which it operates, is committed to improving the life of Mozambican population by mobilizing communities to strive for the elimination of malaria. PIRCOM’s main objective is to contribute the PMI and NMCP goals of reducing morbidity and mortality due to malaria, particularly among pregnant women and children under five years of age, through behavioral communication change activities. PIRCOM aims to: • Increase access to information about malaria prevention and treatment in Mozambican communities, especially among at-risk populations. • Increase capacity of faith leaders at the national, provincial and district levels to lead, manage and sustain malaria prevention activities in their communities. • Make use of its extant multi-religious network to mobilize all leaders designated by local communities as faith leaders, denominational networks and nondenominational free-standing houses of worship. • Disseminate malaria messages and promote malaria discussions through sermons at religious￾based places of worship, interpersonal communication in the communities, community-based home visits and through dissemination of SBCC mass media messages on malaria prevention and treatment using community radios. According to PIRCOM records, since the activity’s inception, over 528 faith leaders and 1,025 religious volunteers completed training in malaria prevention and treatment opportunities so that they could then disseminate malaria messages in worship places and home visits. Between faith-based sermons and 96 community radio programs from five community radios (Ile, Mobuca and Inharrime community radios, Progresso and HAQ radios), PIRCOM asserts that more than 700,000 people have been reached. 49 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Activity implementation PIRCOM was awarded by Integrated Health Office (IHO) from a USAID three-year project valued in $ 1,574,247.38. The implementation runs from 6 November 2015 to 5 November 2018. Thus, this final performance evaluation will likely occur near the end of implementation. 1.3 Geographic focus PIRCOM is active in twenty-nine priority districts distributed among five provinces, as detailed in Figure 1. Figure 1: Priority districts per province Provinces Priority districts Gaza Guijá, and Chòkwé Inhambane Inharrime, Jangamo, Panda, Massinga, Maxixe, and Inhambane City Sofala Búzi, Caia, Marromeu, Cheringoma, Maríngue, Dondo, and Nhamatanda Zambezia Ile, Inhassunge, Lugela, Morrumbala, Mocuba, Namacurra, and Nicoadala Nampula Moma, Angoche, Ribaué, Monapo, Murrupula, Nacala-Porto, and Malema Source: PIRCOM 1.4 Results framework PIRCOM’s results framework follows in Figure 2. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 50 Figure 2: PIRCOM’s results framework To contribute for the achieving of the PMI and the NMCP’s goals to reduce morbidity and mortality from Malaria in Mozambique, particularly among pregnant women and children under five years, by training and mobilizing faith leaders and faith groups for SBCC activities in selected provinces and districts. SO1 - Increase knowledge and utilization of malaria prevention and treatment in Mozambican religious communities, especially among pregnant women and children under 5 SO2 - Increase the capacity of faith leaders at national, provincial and district levels to lead, manage and sustain malaria prevention activities in their communities. IR 1.1. - Increased knowledge of malaria prevention, diagnosis, and treatment among religious communities participating in PIRCOM from 2015 to 2018 IR 1.2. - Improved use of effective malaria prevention and treatment methods in targeted communities IR 2.1. - Enhanced leadership of malaria prevention activities by faith leaders in targeted communities STATEMENT OF WORK 1.1. Evaluation Purposes The PIRCOM Final Performance Evaluation is intended to assess the effectiveness of the malaria messages at influencing target audience malaria knowledge and practices and to inform future USAID activity design. Use of the evaluation’s results will help to improve USAID social and behavior monitoring and evaluation (M&E) systems, and activities’ results on the usage of community radios for dissemination of social and behavior messages. 1.2. Evaluation Objectives & Questions The evaluation objectives and questions are presented in Figure 3. Figure 3: PIRCOM evaluation objectives and questions Evaluation objectives Evaluation questions A. Determine the main barriers and successes that were identified during the course of implementation 1. What were the principal barriers and successes during implementation of the PIRCOM scope of work? 51 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Evaluation objectives Evaluation questions a. How were the barriers addressed by the program? b. In what ways did PIRCOM build upon successes to improve implementation? c. How did PIRCOM work with other stakeholders (MISAU and other IPs) in coordinating interventions in the same geographic area? B. Understand the effectiveness of the malaria messages at influencing target audience malaria knowledge and practices 2. How effective were the PIRCOM malaria messages and delivery at influencing target audience malaria knowledge and practices? C. Determine the effectiveness of the PIRCOM M&E system 3. How effectively did the PIRCOM M&E system function? a. How could the PIRCOM and other similar projects’ M&E systems be improved? b. How effective was monitoring of community radio PIRCOM messages? Are there ways to improve such monitoring in future programs? METHODOLOGICAL APPROACH The evaluation will use qualitative and quantitative methods. The qualitative portion will collect and analyze secondary sources (PIRCOM records, reporting documents, and useful technical references), key informant interviews (KII), and group interviews. Qualitative data sources will include USAID, the implementation partner (PIRCOM), religious leaders and members from the community, as well as other relevant malaria stakeholders and partners at central, provincial and local levels. Local experts and specialists on media audiences will also be interviewed to understand monitoring of community radio messaging coverage. The identification of the principal barriers and successes during implementation will be gauged throughout the interviews with the key informants and by analyzing the MEL plan and related data, quarterly reports, and other secondary data. The effectiveness of PIRCOM’s M&E system will use the same approach, although focused in its scope to track results, including their timeliness, quantity, and quality. The quantitative approach will follow Knowledge, Attitude, and Practices (KAP) survey principles to measure understanding of general malaria concepts and specific knowledge, attitude, and practices of specific malaria subjects. The KAP survey will use probability methods to target households with pregnant women, children under age five, and among the religious communities from the sample districts. The evaluation team will identify what is known about various malaria messages (and resultant changes in behaviors), measuring the effectiveness of the malaria messages and other related health education activities in changing health-related behaviors. The KAP survey will also reveal misconceptions, misunderstandings and gender relations that may present obstacles to the implementation and potential barriers to behavior change. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 52 Some limitations need to be considered, as the KAP survey reveals what informants report, but gaps can exist between what is reported on a survey and actual behavior. The lack of time series data and control groups may also limit the interpretation and value of the data. The quantitative data will be complemented with qualitative data using focus groups (FGs) technique. A total of twelve FGs, four per each province, will be conducted to better understand the reasons for existing practices as result from the malaria messages, including looking into the different religious groups. Guided and open questions will be asked in an interactive group setting where participants will be encouraged to freely express their perceptions, opinions, beliefs, and attitudes. During this process, the researcher either takes notes or records the vital points he or she is getting from the group. 1.3. Survey protocol and sampling design The evaluation will develop a survey protocol for the KAP that will assess the malaria messages in influencing target audience malaria knowledge and practices in communities targeted by PIRCOM. The KAP survey protocol will be developed by a statistics specialist and will include a sampling plan reflecting sex disaggregation. Given the significant religious and cultural differences among the provinces, particularly between the southern provinces and the central and northern provinces, and within the regions, the evaluation will visit three of the five provinces. The selected provinces are Nampula and Zambezia (the two priority provinces for the PMI) and Inhambane. The assumption behind this decision is that cultural, religious and gender relations differences lead to different attitudes and behaviors; consequently, where differences are significant, we include them to capture their specific differences. In the south, the Inhambane Province will be considered for having a more diversified socio-religious spectrum compared to Gaza province. 1.4. Selection of districts and communities The sampling design considers two districts of each selected province in a total six of priority districts. The selection of districts considered elements such as typology (whenever possible one rural and one urban district per province), location (one inland and one coastal district per province), existence of community radio coverage (at least one of the selected districts should have access to the broadcast of community radio), the average dimension of the households (whenever possible significant different averages within a province). The different characteristics of the PIRCOM priority districts in the selected provinces are detailed in Figure 4. 53 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Figure 4: PIRCOM districts profiles in the selected provinces Provinces Selected districts Typology Location Community radio Total population Quantity of households Average of HH members Inhambane Inharrime Rural Inland Yes 123,605 27,919 4.4 Jangamo Rural Coastal No 105,478 25,458 4.1 Panda Rural Inland No 38,989 9,057 4.3 Massinga Rural Inland No 228,437 54,017 4.2 Maxixe Urban Coastal No 123,868 32,500 3.8 Inhambane Cidade Urban Coastal No 79,724 21,043 3.8 Zambezia Ile Rural Inland Yes 188,787 44,555 4.2 Inhassunge Rural Coastal No 91,653 23,456 3.9 Lugela Rural Inland No 188,659 41.742 4.5 Morrumbala Rural Inland No 380,189 91,914 4.1 Mocuba Rural Inland Yes 422,681 87,982 4.8 Namacura Rural Inland No 242,126 62,018 3.9 Nicoadala Rural Inland No 180,686 44,252 4.1 Nampula Moma Rural Coastal Yes (HAQ) 310,706 75,181 4.1 Angoche Rural Coastal Yes (HAQ) 399,092 97,399 4.1 Ribaue Rural Inland Yes (HAQ) 290,244 69,277 4.2 Monapo Rural Inland Yes (HAQ) 413,694 98,975 4.2 Murrupula Rural Inland Yes (HAQ) 215,208 55,822 3.9 Nacala – Porto Urban Coastal Yes (HAQ) 225,034 52,685 4.3 Malema Rural Inland Yes (HAQ) 223,791 54,562 4.1 Sources: INE Census 2017, preliminary results (2018); PIRCOM; MMEMS USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 54 Considering the above-mentioned criteria, the sample will consider the following districts per province as indicated below in Figure 5 and represented in Figure 6. Figure 5: KAP sampling districts per province Figure 6: Map representation of the sampling districts Map subtitles: • Yellow – PIRCOM geographic focus • Orange – selected sampling districts The KAP survey will target communities and/or neighborhoods covered by PIRCOM’s activities within the selected districts using probability sampling methods. To ensure that every element in the sample frame has an equal chance of being incorporated into the sample, all communities and/or neighborhoods will be considered and the households listed by PIRCOM will be randomly selected using random sampling procedures. PIRCOM is currently active in 26 communities and/or neighborhoods (area of influence of the worship places) distributed among the selected districts. The sampling frame is detailed in Figure 7. Provinces Sampling districts Inhambane Inharrime and Maxixe Zambezia Inhassunge and Mocuba Nampula Murrupula and Moma 55 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Figure 7: PIRCOM communities per district in the selected provinces Provinces Selected districts Communities/neighborhoods Inhambane Inharrime Inharrime Sede, Nhanombe, Dongane, Mahalambe Jangamo Cumbane, Bambela, Jangamo Sede Panda Panda Sede, Massavane, Chivalo Massinga Rovene, Guizuco, Mangonhe, Guma, Lioinzuane, Malamba, Queme Maxixe Maxixe Sede, Bembe Inhambane Cidade Bairros Balane 2, Chalane, Tofo, Marrambone, Conguiane, Muele 1 e 3, Chalambe 2, Siquiriva, mahila, Josina Machel, Nhampossa, Mucucune, Chamane, Machanvenga, Aeroporto, Liberdade 1 e 2, Malamuane Zambezia Ile Ile sede, Mulevala, Nampevo, Mugulama, Socone, Mudubua, Namanda Inhassunge Bigagira, Gonhane, Chirimane, Mucupia Lugela Lugela Sede, Mubede, Alto Lugela, Mulide, Ibide, Mussangane, Murrauanha Morrumbala Morrumbala Sede Mocuba Mocuba Sede, Namanjavira, Muaquiua, Munhiba Namacura Muiebele, Mutange, Namacura Sede, Malei Nicoadala Nicoadala Sede, Nhafuba, Munhonha, Licuari Nampula Moma Naicole, Jagoma, Mirupi, Pilivili, Moma Sede Angoche Namitoria, Aube Sede, Angoche Sede, Namaponda, Sangage Ribaue Ribaue Sede, Namigonha, Mecuasse Monapo Carapira, Nacololo, Napala, Namapa Sede, Itocolo, Monapo Sede Murrupula Murrupula Sede, Nihissiue Sede, Namiope-Namacoma, Namiope Sede, Murrupula Murala, Nacocolo, Cazuzu Sede Nacala – Porto Mutiva Malema Nioce, Nataleia, Muralelo, Malema Sede, Mutuali, Chihulo Source: PIRCOM The target sample (i.e. households with pregnant women and children under five years old) will be randomly selected within the visited HH by the PIRCOM volunteers. Using random sampling, the likelihood USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 56 of bias is reduced. The quantity of households (HH) in each community and/or neighborhood to be selected using simple random sampling procedures will vary between sixteen to thirty-three households HH. The final quantities will be calculated considering the detailed lists of the visited HH of the selected districts. A sample plan will be developed taking into the account the representativeness of the HH visited by PIRCOM volunteers in the selected districts. Data from the National Institute of Statistics (INE) from two key sources (2017 population census, preliminary results; family budget survey 2014-15) and PIRCOM annual report supports the Figure 8. Figure 8: KAP HH sampling basis Program coverage14 Total Population Total Households Population listening radio HH with radio handset HH members reached by PIRCOM home visits HH reached by PIRCOM home visits Totals in the selected provinces 4,472,651 1,069,814 1,657,766 395,905 72,541 25,253 Totals in the selected districts 1,287,651 302,860 479,632 124,105 NA15 NA2 Inhambane PIRCOM coverage 700,101 169,994 248,536 60,348 17,577 6,390 Inhambane selected districts coverage 247,475 60,419 87,853 21,449 NA2 NA2 Zambezia PIRCOM coverage 1,694,781 395,919 696,555 162,723 25,588 9,917 Zambezia selected districts coverage 514,334 111,438 211,391 45,801 NA2 NA2 Nampula PIRCOM coverage 2,077,769 503,901 712,675 172,838 29,376 8,946 Nampula selected districts coverage 525,914 131,003 180,388 56,855 NA2 NA2 14 Program coverage assumes the whole population of the priority districts and not necessarily the PIRCOM’s active communities and/or neighborhoods. 15 To be supplied by PIRCOM for the development of the sample plan 57 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Sources: INE Census 2017, preliminary results (2018); INE Family Budget Survey (IOF) 2014-15; PIRCOM Taking into the account that the confidence intervals tells us how much we can expect the KAP survey results to reflect the views from the overall population, the HH sampling scenarios presented in Figure 9 will ensure that the margin of error (confidence interval) won’t be higher than 5% for the national sample. This determines how much higher or lower than the mean of the population of the study are we willing to let our sample mean to fall. Details of the KAP HH sampling scenarios are detailed in Figure 9 Figure 9: KAP HH sampling scenarios Program coverage16 HH members reached by PIRCOM home visits HH reached by PIRCOM home visits Sample Basis Sample 95% conf level, std dev .5% margin error 5% Sample 99% conf level, std dev .5% margin error 5% Totals in the selected provinces 72,541 25,253 555 600 664 Inhambane PIRCOM coverage 17,577 6,390 140 152 168 Zambezia PIRCOM coverage 25,588 9,917 219 236 261 Nampula PIRCOM coverage 29,376 8,946 196 212 235 As a limitation of the proposed sample, the extrapolation of the results to the provincial level must be done with a certain care, considering the high margin of error inherent to the low sample. The margin of error was calculated for the entire population and not for the target (HH households with pregnant women and children under five years old among the religious communities) due to insufficient statistical information. The non-response rate will be minimum, as the field team will be presented to the HH members by PIRCOM volunteers, however, at no time they will attend/assist the interviews to avoid any kind of influence. A more detail KAP survey protocol, including the sample plan will developed once PIRCOM lists will be available. 1.5. Data Collection and tools The focus of all data collection activities is to answer the evaluation questions formulated by USAID and included in the SOW. To do that, a mix of data collection methods will be applied. 16 Program coverage assumes the whole population of the priority districts. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 58 The qualitative data collection will include group interviews and one-on-one interviews of USAID key informants, the IP (PIRCOM), and relevant malaria stakeholders and partners at central, provincial and local levels. These methods will cover a broad range of stakeholders. The evaluation team will develop instruments to guide the group and individual interviews, guidelines to register the information collected in the interviews, a tally sheet (on which how many interviewed say what are kept registered in a tabular form) and a design matrix that will assist the compilation of data collected from primary sources. The data collection will be supported by transcripts of individual interviews and focus groups, field notes from observation of certain activities, copies of documents, and other record forms. MMEMS and the evaluation team will collect other relevant activity data and documents, such as MEL plan data, quarterly reports, technical products and other secondary data to judge performance and access effectiveness of the PIRCOM M&E system. Quantitative data collection instruments will follow the survey protocol. A closed and semi-open questions form will be developed to answer Evaluation Question 2. This form will be done in English, translated into Portuguese and adapted up to five different local languages (Echawabu, Elomwe, Emakhuwa, Bitonga, Citswa). The tool will be initially piloted in Maxixe (Inhambane) where the first training will take place and adjusted according to the feedback. Field teams will be trained in Maxixe, Quelimane and Nampula cities. The KAP survey will be collected electronically using electronic tablets through the Fulcrum on-line platform. An experienced fieldwork team of supervisors and enumerators with fieldwork and local language domains will ensure high-quality data collection. 1.6. Analysis methods and data disaggregation The PIRCOM evaluation will analyze and compare planned versus actual interventions within multiple dimensions, such as time, quantity and quality. Data collected will be disaggregated and analyzed using gender lenses. Quantitative data collected will be disaggregated by age, sex, level of education and, as may be relevant and useful, will track other profile dimensions, including religion. To the extent possible, special emphasis will be placed on applicable domains (a similar set of issues) of gender analysis considered by USAID (ADS 205.3.2). Accordingly, the evaluation team will aim to identify, understand, and explain existent gaps, identify the relevance of gender norms and power relations specific to the implementation context(s) while applying/adapting a suitable gender analysis framework. Innovative models or approaches to monitor community radio coverage will be highlighted to assist recommendations that support reflection on future USAID program design. Although limited to realistically meet the evaluation scope and time constraints, the evaluation team will also consider the analysis of unintended outcomes that may provide useful lessons that may help explain factors for failure and success. The qualitative data will be organized, a framework will be identified to group the data and then look for relationships, including descriptive analysis and other subsequent analysis. This inductive approach will support the qualitative analysis by analyzing text data, the primary message content, the evaluative attitude of the informants toward the message, and whether the content of the message is meant to represent individual or group-shared ideas. Finally, the analysis and triangulation will allow the identification of information gaps (as per evaluation questions), conduct reference/expert consultation, keyword distribution of responses, and results cross check. 59 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV RECOMMENDED COMPOSITION OF TEAM The evaluation team will be comprised of short-term evaluators, short-term fieldwork staff, and MMEMS full-time staff. MMEMS will also provide the required support staff and transport. The evaluation and level of effort assume that all tasks listed in the previous section will be conducted. • Project oversight (MMEMS Senior Evaluation Officer). An international professional experienced in managing USAID evaluation projects, including knowledge and expertise in M&E systems, KAP surveys, and media audiences. He will be responsible for the overall client management, provide quality assurance and technical oversight. • Principal investigator. An international researcher with deep experience in M&E health systems and conducting social and behavior change communication evaluations. He/she will be the technical lead responsible for the evaluation, design, and implementation of the methodologies to be applied at field level, overseeing data quality gathered at field level, providing analysis and leading report writing. • Health sector specialist. An international professional experienced in assessing the Mozambican health sector, particularly the area of social and behavior change. This professional will participate in data collection and analysis and will participate in reporting under the lead of the principal investigator. • Statistician. A Mozambican professional experienced in developing survey protocols, particularly sampling plans taking into the account the rural conditions. • Data manager (MMEMS Data Manager). Will be responsible for the design of quantitative instrument using the Fulcrum platform, and ensure integrity and consistency of the electronic data collection process. • Translators. One per each local language (Echawabu, Elomwe, Emakhuwa, Bitonga, Citswa) in each province with linguistic background and expertise. The role of the translators is to ensure the adequate translation of the KAP instrument to local languages. • Field supervisors. A set of four experienced field supervisors (one per each province) will coordinate and supervise the quantitative data collection, ensuring data quality control at field level. The field supervisors have knowledge and field expertise on social surveys, especially households (HH) surveys. • Enumerators. A set of thirty experienced field enumerators (ten per each province) will be responsible for the collection of the quantitative data, under the direct supervision and guidance of the field supervisors. • Field guides (one per each site). Local guides will present the field teams to the communities and the selected HH, facilitating the initial contact process with, avoiding misunderstandings among parts, and reducing the drop/refusal rates. The field guides will be selected from the local PIRCOM volunteers, however, at no time will the field guides attend/assist the interviews to avoid any kind of influence. PERIOD OF PERFORMANCE AND EXPECTED LEVEL OF EFFORT This SOW is designed for a four-month period, starting in January 2018 and finishing on May 2018. The consultant total level of effort by the evaluation team, for the period of duration will be 20 days for the USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 60 project oversight, 60 days for the principal investigator, 40 days for the health sector specialist, 15 days for the statistician, 20 days for the data manager, 5 days for each translator (total 25 days), 15 days for each field supervisor (total 45 days), 10 days for each enumerator (total 300 days), an average of 1 day for each field guide (total 30 days). 1.7. Deliverables The survey protocol, including the draft sample design of the KAP survey, is the first key deliverable. With the arrival in-country of the principal investigator, the team will conduct a Team Planning Meeting (TPM), resulting in a shared sense of the way forward and production of a work plan, evaluation methodology, implementation timeline, finalization of the survey protocol, qualitative evaluation tools and instruments, and KAP survey instruments. Other PIRCOM evaluation deliverables will follow: • KAP survey instruments (Portuguese and in local languages); • Findings, Conclusions, and Recommendations table (FCR matrix); • Draft report; • PowerPoint and oral presentations, including methodology, data analysis methods and data disaggregation, findings, conclusions, and recommendations; and • Final report. Details of the deliverables are listed in Figure 11. Figure 11: Deliverables table Evaluation phase Deliverables Responsible Dates Preparation Survey protocol, including the KAP sample design (draft and final versions) MMEMS February 2018 Overall planning Team planning meeting Work plan development In-brief session MMEMS MMEMS MMEMS Starting 26 February February 2018March 2018 Design of tools and instruments Tools and instruments (draft and final versions) Tools in local languages on-line MMEMS MMEMS March 2018 March 2018 Fieldwork Weekly evaluation field reports/ internal MMEMS March/April 2018 Reporting Preliminary FCR presentation Out-brief to Mission Draft report Comments to report Final report MMEMS MMEMS MMEMS USAID MMEMS April 2018 April 2018 April 2018 April/May 2018 TBD May Knowledge sharing Dissemination/learning activities/event MMEMS TBD May 61 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV ANNEX B: ANALYTICAL FRAMEWORK: GETTING TO ANSWERS MATRIX Data Required Data Sources Data Collection Methods Data Analysis Methods Evaluation Question #1: How effective were the PIRCOM malaria messages and delivery at influencing target audience malaria knowledge and practices? A. Data on changes in target audience’s malaria knowledge and practices in line with 5 messages disseminated by PIRCOM to reduce malaria morbidity and mortality 1. Sleeping under a mosquito net 2. Allowing teams to carry out indoor residual spraying inside the house 3. Attendance to antenatal visits to receive IPT for pregnant women 4. Seeking immediate treatment after the first malaria signs 5. Adherence to treatment B. Data on effectiveness of PIRCOM communication strategy C. Data on PIRCOM SBCCP PMP process and outcome indicators A. USAID staff, PIRCOM and MOH staff at the central, provincial, district and community levels, community radio stations, community leaders, pregnant women and women with children under 5, husbands/fathers of the targeted households. B. Pregnant women and women with children under 5, husbands/fathers from non￾PIRCOM districts C. PIRCOM records and reporting documents A. Key informant interviews B. Group discussions C. KAP Survey D. Focus group discussions E. Desk study of secondary data A. Analyses of qualitative and quantitative data B. Data triangulation C. Descriptive statistics and data visualization of key indicators of KAP survey D. Estimation of PIRCOM malaria message effect USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 62 Data Required Data Sources Data Collection Methods Data Analysis Methods Evaluation Question #2: What were the principal barriers and successes during implementation of the PIRCOM scope of work? A. Data on successes achieved during the implementation of the project B. Data on barriers met during the implementation C. Data on effectiveness of PIRCOM in coordinating activities with other malaria stakeholders A. USAID staff, PIRCOM and MOH staff at the central, provincial district and community levels, other malaria stakeholders at all levels, community leaders, pregnant women and women with children under5, husbands/ fathers of the targeted households B. PIRCOM records and reporting documents A. Key informant interviews B. Group discussions C. Desk study of secondary data D. Focus group discussions A. Analyses of qualitative data B. Data triangulation RQ#3: How effectively did the PIRCOM M&E system function? A. Data on PIRCOM M&E Plan B. Data on functionality of the M&E system regarding the reliability and quality of information collected A. USAID staff, PIRCOM staff at all levels including community supervisors, MOH staff at all levels, community radio stations A. Key informant interviews B. Group discussions A. Analyses of qualitative and quantitative data B. Data triangulation 63 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Data Required Data Sources Data Collection Methods Data Analysis Methods C. Data on usefulness of the M&E system to improve activity performance D. Data on effectiveness of community radio messages B. PIRCOM M&E plan, quarterly and annual reports C. Desk study of the PIRCOM M&E plan and quarterly and annual reports USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 64 ANNEX C: LIST OF DOCUMENTS REVIEWED PIRCOM Annual and Quarterly Reports FY2015, Annual report, 1st year of the project, October 2016 FY2016, Annual report, 2st year of the project, November 2017 Project Year2 FY 2016 3 Quarterly report April-June 2017 Project Year 3 FY2017 1 Quarterly report October-December 2017 Agreement, Strategies, Workplans and Manuals Copy of the Grant Agreement November 6, 2015 PIRCOM Milestones for 3-year project PIRCOM Capacity Statement Year 1 Workplan – FY16, 3rd December 2015 Year 2 Workplan – FY16, 20th October 2016 Year 3 Workplan – FY17, 13th November 2017 Análise comparativa de Líderes Religiosos Treinados por Género Análise comparativa de Voluntários Formados por Género Anual, Voluntário Actividade Visitas Domiciliárias Avaliação da Capacidade Institucional do PIRCOM em CMSC; JHU / CCP Capacitação dos Parceiros Locais da USAID Moçambique, Task Order #3; ANEXO 2 – Constatações do Diagnóstico de Necessidades de Capacitação Institucional Data collection tools: Ficha de recolha de sermão 2017, Ficha de visitas domiciliárias 2017 DB sample 2017 – Inhambane, Nampula, Zambézia DB targeted – Líder Religioso Actividade Palestra, Líder Religioso Actividade sermão, Reporte DB whole provinces – base de dados, Gaza, Inhambane, Sofala, Nampula, Zambézia Fact sheet - USAID Social and behavior change communication program – PIRCOM M&A: Fluxo de informação Malaria Behavior Change Communication (BCC) Indicator Reference Guide February 2014, Roll Back Malaria Malaria messages – As cinco mensagens Multifaith program against Malaria in Mozambique (PIRCOM) and adaptations in other countries Performance and financial review (Social and behavior change communication program) – FY2017 PIRCOM – Geographical coverage of the project Plano de Acção para a Implementação do Apoio Institucional ao PIRCOM, 18 de Maio 2017, Religious Confessions – Confições Religiosas Gaza, Inhambane, Sofala, Nampula, Zambézia National Documents: PNCM – Plano Estratégico da Malária 2012-2016 National Malaria Control Programme, Malaria Strategic Plan 2017-2022 IMASIDA – Inquérito de Indicadores de Imunização, Malária e HIV/SIDA em Moçambique, Relatório de Indicadores Básicos, 2015 USAID/PMI: U.S President´s Malaria Initiative Gender documents Gender and SBCC, Implementation kit 65 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV ADS Chapter 205, Integrating Gender Equality and Female Empowerment in USAID´s Program Cycle, USAID 04/27/2017 Gender Transformative Approaches, Health Communication Capacity Collaborative Others KAP tools- Previous surveys from: • Cameroon, Swaziland, Uganda, Mozambique (COWI 2012), • Knowledge, Attitudes and Practices Survey, Baseline evaluation in Aragatsotn, Armavir and Ararat Marzes (USAID), • Malaria Needs Assessments in Phuntsokling Tibetan Settlement, Orissa (John Hopkins University), • Inquéritos sobre Conhecimentos, Atitudes e Práticas, Doença do Virus Zika e Potenciais Complicações (OMS), • RPS, Avaliação sobre a posse e uso de redes mosquiteiras em Moçambique, 2014 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 66 ANNEX D: LIST OF KEY INFORMANTS INTERVIEWED MAPUTO USAID Flavio Wate, AOR, USAID Abuchahama Saifodine, PMI Resident Advisor for USAID Rose Zullinger, PMI Resident Advisor for CDC Maria Branquinho, Local Capacity Development Advisor, Health Office PIRCOM Cidia Monteiro, Communication Specialist and Program Advisor Sérgio Mahumane, Monitoring and Evaluation Officer Carlos Miguel Vilanculos, Program Manager MINISTRY OF HEALTH Baltazar Candrinho, Head Chief of the Malaria National Control Program Sergio Tsabet, Focal Point of Communication in Malaria National Control Program UNICEF Aida Mahomed, Communication for Development Specialist EUROSIS Abdul Sacaar, Chief of Party Pedro Inácio, Project Management Specialist / M&E Carlos Brito, Communication Technical Specialist ABT ASSOCIATES Lourdes Loch, COP for Malaria Indoor Residual Spraying NAMPULA PIRCOM Luis dos Santos Canleva, District Secretary Herminio Guifutela, Province Coordinator Abilio, District Delegate, Murrupula Central Morais Gonçalves de Oliveira, Volunteer and Deputy Secretary, Murrupula central Elias João, Secretary Delegate, Murrupula central MINISTRY OF HEALTH Marcelino Matias Adui, Head of the National Malaria Program, Health Provincial Director Brito Languitone Mozoland, Education Communication Specialist in Malaria Program, Health Provincial Director Azevedo Joao, Health District Director of Murrupula Central 67 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV IREX Ryan Forbes Morris, IREX Program Manager HAQ Sheik Abdul Magid, Deputy Director Issufo de Amade Nelson, Speaker and Producer of the Programs Fatima Sadá Assane, Secretary ZAMBEZIA PIRCOM Miguel Alberto, Provincial Coordinator Marcelino Raston, Delegate Secretary Bernardo Cinco Reis, District Delegate, Mocuba District Lourinho Fevereiro, Supervisor, Munhiba Community, Mocuba District Américo da Silva, Supervisor, Muaquiua Community, Mocuba District Lino Américo Serrote, Religious Leader, Muaquiua Nigula Community, Mocuba District Lucas Viagem Pacala, Religious Leader, Muaquiua Community, Mocuba District MINISTRY OF HEALTH Eurico Gemuçe, Focal Point of the National Malaria Program, Health District Director, Mocuba district JHU Alberto Viano, Communication official, John Hopkins MOCUBA COMMUNITY RADIO Ricardo Ossalo, head of broadcasting, Mocuba District Tome Carvalho Candieiro, coordinator of Mocuba Radio Station, Mocuba District Isabel Duarte, administrative, Mocuba District Rosalina Caitano, Jornalist, Mocuba District INHAMBANE PIRCOM Micaias Caessa. District Delegate, Inhambane City Benjamina das Dores, District Delegate, Inharrime District Mussa Cassomo, Secretary of the District Delegate, Inharrime District Adriano Vilanculos, Provincial Delegate, Maxixe District Mateus Fernando, Provincial Coordinator, Maxixe District Sauk Mangue, Religious Leader, Nhanombe Community, Inharrime District Felizarda Eugenio, Volunteer, Chacane Community, Inharrime District Veronica Francisco, Volunteer, Chacane Community, Inharrime District Augusto Naete Guambe, Supervisor and Religious Leader, Chacane Community, Inharrime District USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 68 Jose Manuel Chicambo, Volunteer and Religious Leader, Chacane Community, Inharrime District Damas Folidane Nhamombe, Religious Leader, Chacane Community, Inharrime District MINISTRY OF HEALTH Naftal Matusse, Health Provincial Director António Beula, Provincial Director of National Malaria Program Samuel Mucavele, Focal point of the National Malaria Program, Health District Director RADIO PROGRESSO Issufo Mussagy, Representant of the Radio Progresso, Inhambane 69 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV ANNEX E: KAP SURVEY PROTOCOL The PIRCOM Final Performance Evaluation is intended to assess the effectiveness of the malaria messages in influencing targeted audience a knowledge and practices related to Malaria. To evaluate the effect of the messages two groups will be compared, namely a group that has been exposed to the messages (the PIRCOM beneficiaries) and a similar group of people that has never been exposed. Both groups will be subjected to KAP surveys. The effectiveness will be measured by the differences on in terms of Malaria KAPs on the groups. The bellow picture depicts the process. Picture 1: Evaluation approach for measure the effectiveness of the malaria messages To apply this methodology, it is necessary to check for balance – the characteristics of the intervention and comparison groups are compared to test for balance. Ideally, there will be no significant differences in average observable demographic characteristics between the two groups. Once it has been determined that the PIRCOM beneficiary and the non-beneficiary groups are similar in observable characteristics, variance in the indicators of KAP survey between the two groups can be attributed to differences in effectiveness of the malaria messages. Additionally, we will apply the Cohen's d17 and the statistical test to determine whether the indicators differences between the two groups are statistically significant. Sample design The “best” sample design for an analysis is one that provides levels of sampling accuracy that are acceptable in terms of the main purposes of the activity, while simultaneously limiting cost, logistics, and procedural demands to manageable levels. Three major constraints were established prior to the preparation of this sample design. First, the target population of the KAP survey is the communities covered by PIRCOM in six districts, two from each province namely, Inhambane, Nampula, and Zambézia. Specifically, the target population are HHs (households) with pregnant women and children under five years old. Second, the sampling should conform to the accepted rules of scientific probability sampling. That is, the members of the defined target population should have a known and non-zero probability of selection into the sample. Finally, the number of communities and HHs to be selected should take into consideration the administrative and financial resources available for data collection. 17 Cohen's d -defined d as the difference between the means, M1 - M2, divided by standard deviation, s, of either group. According to Cohen, d > =0.8 is regarded as large; d between 0.5-0.7 medium; d<0.5 Small malaria messages KAPs PIRCOM beneficiaries Non-PIRCOM beneficiaries KAPs USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 70 Taking into consideration the above constraints, and the PIRCOM effectiveness assessment approach, the sample design includes two samples of households, one of the PIRCOM beneficiries and the other with non-PIRCOM beneficiaries. The need to match the characteristics of PIRCOM and non-PIRCOM beneficiaries is one of the critial issues in this sample design. The selection of PIRCOM intervention districts considered elements such as typology (whenever possible one rural and one urban district per province), location (one inland and one coastal district per province), existence of community radio coverage (at least one of the selected districts should have access to the broadcast of community radio). Similar criterion was used to select the districts of non PIRCOM beneficiaries. Table 1 shows the characteristics of the sampling frame of the two group of districts beneficries and non beneficiaries of the PIRCOM intervention. TABLE 1 ELEMENTS OF DISTRICTS TOPOLOGY AND LOCATION District characteristics PIRCOM beneficiaries Non-PIRCOM beneficiaries Inhambane Urban Maxixe (Municipality) Vilanculos (Municipality) Rural Inharrime Morrumbene Costal Maxixe Vilanculos Zambézia Rural Inhanssunge Pebane Urban Mocuba (Municipality) Gurué (Municipality) Costal Inhanssunge Pebane Inland Mocuba Gurué Nampula Rural Moma Irati Costal Moma Mussoril Inland Morrupula Irati It is important to stress that the PIRCOM sample will only be representative of the six districts. It does not represent all PIRCOM’s target districts. Consequently, all performance indicators will also represent the six districts. The estimation of the performance of PIRCOM malaria messages will reflect the possible changes in KAPs of the six districts. However, one could argue that if the level of activity of intervention in other districts under PIRCOM were similar to the selected districts, comparable changes in KAPs could be expected in these districts. Sample size Table 2 presents the sample size for KAPs survey. The sample size of KAP survey is 646 households for PIRCOM beneficiaries and 600 households for non-beneficiaries: (margin error 3.8 %, confidence interval 95%). The PIRCOM beneficiaries sample is stratified by districts and communities, based in proportional allocation to ensure that each beneficiary HH has an equal chance of being selected for the KAP survey. There was an increase in proportion of sample size to ensure the is large enough. 71 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 2 SAMPLE SIZE Sampling frame Sample PIRCOM beneficiaries District HH reached by PIRCOM visits Number of clusters /communities HH to be randomly selected Total 646 Inhambane Inharrime, Maxixe 432 8 1521 Zambézia Inhassunge, Mocuba 826 9 202 Nampula Moma, Murrupula 1,203 16 292 Non￾PIRCOM beneficiaries Districts HH NOT reached by PIRCOM Cluster/communities selected randomly (PPS) HH to be randomly selected (PPS) Total 646 Inhambane Morrumbene, Vilankulos 58,244 8 152 Zambézia Pebane, Gurúe 112,479 9 202 Nampula Mussorial, Erati 99,420 16 292 A two-stage sampling methodology will be employed for the KAP survey for non-PIRCOM areas; this ensures that each household in the area has an equal chance of being selected for the survey. The first stage is the selection of clusters (communities/ neighborhood) and the second is the random selection of the households within selected clusters (HH with pregnant woman and child under five). Tables 3 and 4 present the sample allocation by communities for PIRCOM beneficiaries and non PIRCOM beneficiaries respectively. The selection plan to be executed by the supervisors will include the communities/clusters, randomly selected and tables of random numbers to be used on household selection. In the clusters with no listing of HHs available, the second option, known as the random walk18 will be used. Data cleaning and data analysis Data cleaning, consistency and missing analysis will be carried out to ensure reasonable standards of data quality. For data cleaning, frequency distributions of variables will be analysed to assess the problems of outliers and values out of the expected range. The analysis of KAPs survey key indicators will include descriptive statistics, estimation of Cohen's d and the statistical tests to determine whether the indicator’s differences between the two groups are statistically significant. 18 This is a far less desirable method than the household list method. It involves the following steps: (a)Start at the market area of the cluster. If there is no market, start at the church, mosque, or temple. (b)Throw a pen or pencil in the air and allow it to fall on the ground. Begin walking in the direction indicated by the point of the pen or pencil. Each enumerator team should randomly determine the team’s starting point using the method described above. (c) Count the number of households between the starting point and the boundary of the community. Then use the random numbers to select the first household, USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 72 TABLE 3 SAMPLE ALLOCATION BY CLUSTER FOR PIRCOM BENEFICIARIES província Distrito Localidade visited Hoseholds % of HH population target by district % of HH population target by commuinity within district Sample of HH for KAP survey stratified by district Sample of HH for KAP survey stratified by community within district Bingagira 18 4.8 5 Chirimane 81 21.5 20 Gonhane 247 65.7 60 Mucupia 30 8.0 7 Total-district 376 45.5 100 92 Muaquiwa 106 23.6 26 Mugeba 77 17.1 19 Munhiba 187 41.6 45 Namajavira 72 16.0 18 Vila Sede 8 1.8 2 Total-district 450 54.2 100 110 826 100 202 Chacane 19 5.2 7 Dongane 7 1.9 3 Nhanombe 337 91.8 117 Nhanonbe 4 1.1 2 Total-district 367 85.0 100 129 Malalane 2 2 3.1 1 Malalane 3 2 3.1 1 Maxixe-Sede 53 81.5 18 Nhamaxaxa 8 12.3 3 Total-distric 65 15.0 100 23 432 100 152 Jagoma 117 14.5 28 Mirrupi 31 3.8 8 Mirupi 191 23.7 46 Moma-sede 214 26.5 52 Naicole 97 12.0 24 Pilivili 157 19.5 38 Total-distric 807 67 100 196 Cazuzu 36 9.1 8 Chinga 73 18.4 18 Mulhaniua 24 6.1 6 Murrupula-sede 148 37.4 36 Nacocolo 6 1.5 2 Namiope 27 6.8 6 Namitotelane 9 2.3 2 Nihessiue 51 12.9 12 Rovuma 2 7 1.8 2 Vila Sede 15 3.8 4 Total-district 396 33 100.0 96 96 Total province 1203 292 Moma Nampula Murrupula zambézia Inhambane Sampling framework sample of HH for KAP survey Total -province Total -province Mocuba Inhassunge Inharrime maxixe 73 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 4 SAMPLE ALLOCATION FROM NON-BENEFICIARIES localidade-clusters Number Households Comultive sum clusters sampled HH per cluster Pebane ALTO MAGANHA 5 528 5 528 IMPACA 1 777 7 305 MAGIGA 3 297 10 602 MALEMA 4 877 15 479 X 23 MIHECUE 1 626 17 105 MUCOCORO 798 17 903 MULELA 2 460 20 363 NABURI 6 755 27 118 X 23 NAMAHIPE 919 28 037 NAMANLA 2 263 30 300 NICADINE 3 249 33 549 QUICHANGA 3 597 37 146 X 23 TOMEIA 5 081 42 227 TXALALANE 593 42 820 VILA DE PEBANE 3 500 46 320 X 23 Total -distrito 46 320 96 Gure 12 738 12 738 X 22 Iinvinha 3 517 16 255 Incize 2 488 18 743 X 22 Lioma 4 071 22 814 Magige 4 945 27 759 Mepuagiua 7 914 35 673 X 22 Mualijane 1 638 37 311 Muaquia 5 381 42 692 Mucunha 2 100 44 792 Mugaveia 2 622 47 414 X 22 Muximua 3 222 50 636 Nicoropale 1 490 52 126 Nintulo 4 226 56 352 Nipive 881 57 233 Tetete 5 772 63 005 X 22 Vehiua 3 154 66 159 Total -distrito 66 159 110 vilanculos Belane 7 862 7862 6 Mapinhane 7 512 15 374 6 Muabsa 870 16 244 Quewene 1 621 17 865 Município de Vilankulo 7 674 25 539 6 Vilankulo-Sede 6 192 31 731 6 Total-distrito 31 731 23 Gotite 3 601 3 601 32 Mucodoene 5 536 9 137 32 Sitila 2 899 12 036 Cambine 3 493 15 529 32 Malaia 3 817 19 346 Morrumbene-Sede 7 167 26 513 32 Total -distrito 26 513 129 % of HH population target by commuinity within district Sample of HH for KAP survey stratified by community within district ALUA - SEDE 18 502 26.9 25 MUANONA 6 284 9.2 9 NAMAPA - SEDE 13 963 20.3 20 NAMIROA - SEDE 9 007 13.1 13 ODINEPA 9 829 14.3 14 SAMORA MACHEL 5 172 7.5 7 VILA DE NAMAPA 5 905 8.6 8 Total distrito 68 662 100.0 96 LUNGA - SEDE(AMPITA) 5 475 17.8 35 MATIBANE - SEDE 3 718 12.1 24 NACUCHA 5 689 18.5 36 NAMITATARI 7 566 24.6 48 VIDA NOVA 2 933 9.5 19 VILA DE MOSSURIL (Chocas Mar)5 377 17.5 34 Total distrito 30 758 100 196 PPS sampling approach Morrumbene Sampling framework HH sampled for KAP surevy Nampula Zambézia Inhambane Gurue Namapa erati Nampula proportional allocation was done due to the fact the the number os communities/cluster was equal or less than the correspodent PIRCOM beneficiaries Mussoril USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 74 ANNEX F: QUALITATIVE DATA COLLECTION TOOLS KII guide for USAID and Ministry of Health staff at all levels Introduction – Introduce the team members, purpose of the evaluation, confidentiality and right to refuse to answer questions or participate in the interview. Instructions A. This is a semi-structured interview guide. The actual questions will be tailored based on the individual (s) interviewed. B. The questions cover the implementation of PIRCOM activity and are intended to collect data for performance evaluation of this activity. 1. What do you think about the effectiveness of the malaria messages delivered by PIRCOM to the target audience (pregnant women and children under 5 years) regarding about malaria knowledge and practices? Probe a. Effective / highly effective – why? b. If not effective – why? 2. What do you think the barriers and successes during the implementation of the PIRCOM project? a. How were the barriers addressed by the project? b. In what way did PIRCOM build upon successes to improve implementation? c. How did PIRCOM work with MISAU and other stakeholders in coordinating intervention in the same geographic area? 3. How effectively does the PIRCOM M&E function? a. How could PIRCOM and other similar projects´ M&E systems be improved? b. How effective was PIRCOM in monitoring the community radio malaria messages? 4. Do you think there are ways to improve such monitoring in future programs? 5. In your opinion, how were the gender issues taking into consideration in PIRCOM activity? 75 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV KII guide for PIRCOM staff at all levels Introduction – Introduce the team members, purpose of the evaluation, confidentiality and right to refuse to answer questions or participate in the interview. Instructions A. This is a semi-structured interview guide. The actual questions will be tailored based on the individual (s) interviewed. B. The questions cover the implementation of PIRCOM activity, and are intended to collect data for performance evaluation of this activity. 1. What is your role and responsibilities within PIRCOM? 2. What do you think about the effectiveness of the malaria messages delivered by your project to the target audience (pregnant women and children under 5 years) regarding about malaria knowledge and practices? Probe a. Effective / highly effective – why? b. If not effective – why? 3. What do you think the barriers and successes during the implementation of your project? a. What were the barriers addressed by the project? b. In what way did you build upon successes to improve implementation? c. How did you work with MISAU and other stakeholders in coordinating intervention in the same geographic area? 4. How effectively does your M&E function? a. How could your and other similar projects´ M&E systems be improved? b. How effectively did you monitor the community radio malaria messages? c. Do you think there are ways to improve such monitoring in future programs? 5. In your opinion, how were the gender issues taking into consideration in PIRCOM activity? USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 76 6. How effective was PIRCOM considering the gender relations in the household and in the community to make sure that pregnant women were empowered to change behavior about malaria? 77 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV KII guide for community radios and media experts (IREX) Introduction – Introduce the team members, purpose of the evaluation, confidentiality and right to refuse to answer questions or participate in the interview. Instructions A. This is a semi-structured interview guide. The actual questions will be tailored based on the individual (s) interviewed. B. The questions cover the implementation of PIRCOM activity, and are intended to collect data for performance evaluation of this activity. 1. Have you worked and if yes, how did you work with PIRCOM activity? 2. Have you been monitoring how the malaria messages have been understood among the community members? a. If yes, how? b. If yes, what have been the successes and barriers in monitoring the malaria messages broadcasted by the community radio? c. If not, who is doing the monitoring? USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 78 FOCUS GROUP DISCUSSION GUIDE FOR PREGNANT WOMEN AND MOTHERS OF OR OTHER FEMALE CARETAKERS OF CHILDREN UNDER 5 YEARS We are here to conduct a study to help prevention of malaria among pregnant women and children under 5. This group discussion should not take more than 90 minutes and all answers will remain confidential. You may choose to not to answer any questions if you are not comfortable. We hope you will choose to answer as your responses will help to improve health on mothers and children. May we begin? Start with the introducing yourself and the group. Probe for the gender relations in the household, e,g. who is using the mosquito net at home, did the pregnant women follow the message about the IPT, if not, what are the challenges, which information channel they prefer to get the messages etc. Name of the interviewer Date Province / District / Community Number of individuals involved in this discussion 1. Did you hear about malaria on the community radio? volunteer home visit? religious leaders (imam, priest) 2. What did you hear about malaria from these different channels? (probe for what they heard from each communication channels) 3. Was the information you heard helpful? a. If yes, in what ways? b. If not helpful, why not? 4. What else could be done in order to reduce malaria in your community? 79 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV FOCUS GROUP DISCUSSION GUIDE FOR HUSBANDS OF PREGNANT WOMEN AND FATHERS OF OR OTHER MALE CARETAKERS OF CHILDREN UNDER 5 We are here to conduct a study to help prevention of malaria among pregnant women and children under 5. This group discussion should not take more than 90 minutes and all answers will remain confidential. You may choose to not to answer any questions if you are not comfortable. We hope you will choose to answer as your responses will help to improve health on mothers and children. May we begin? Start with the introducing yourself and the group. Probe for the power relations in the household, if their wife followed the message they got, and if not what were the barriers, etc. Name of the interviewer Date Province / District / Community Number of individuals involved in this discussion 1. Did you hear about malaria on the community radio? volunteer home visit? religious leaders (imam, priest) 2. What did you hear about malaria from these different channels? (probe for what they heard from each communication channels) 3. Was the information you heard helpful? a. If yes, in what ways? b. If not helpful, why not? 4. What else could be done in order to reduce malaria in your community? USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 80 KAP QUESTIONNAIRE QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Data: automático Hora de início: automático Inquiridor: Código: automático Hora do fim: automático Supervisor: Introdução ao questionário 1. Este questionário é destinado a mulheres grávidas ou ao adulto do agregado familiar que cuida de crianças até 5 anos de idade (0-5 anos). 2. Este questionário só deve ser utilizado para estes elementos. 3. Antes de iniciar a entrevista com uma mulher grávida ou o adulto da família, tentar validar o entrevistado e seus dados, em seguida, garantir o seu envolvimento, disponibilidade e consentimento para prosseguir com a entrevista. 4. Dada a natureza das perguntas, deve sugerir/aconselhar o entrevistado a responder às perguntas de forma privada e sem ruído, mas num sítio público. 5. Apresente-se ao entrevistado de acordo com o modelo indicado abaixo e confirme a permissão para realizar a entrevista. 6. Faça as perguntas, conforme indicado no questionário. Não parafraseie. Se o entrevistado não percebe, repita a pergunta ou explique uma palavra, mas não mude a questão. 7. Seja paciente e compreensivo. Deixe o entrevistado responder ao seu ritmo. 8. Pratique a escuta activa, confirmando, sempre que necessário, a resposta do entrevistado. Nunca deve reagir às respostas. Deve manter-se neutro. Não interfira ou influencie as respostas. 9. Esta entrevista é composto por 10 partes: i) listagem dos membros do agregado familiar, ii) hábitos de rádio, iii) religião, iv) conhecimentos sobre a malária v), prevenção da malária vi) redes mosquiteiras, vii) pulverização inter-domiciliária com inseticida, viii) cuidados pré-natais, ix) o tratamento da malária, e x) informações sobre a malária 10. Ao concluir a entrevista, agradeça ao entrevistado pela disponibilidade. Apresentação, envolvimento e confidencialidade da informação Bom dia/boa tarde, o meu nome é ___________________. Sou colaborador do mecanismo e estudos MMEMS que está a avaliar o programa PIRCOM, parceiro do Ministério da Saúde. Eu gostaria de entrevistar uma mulher grávida ou um adulto do agregado familiar que cuida de crianças até 5 anos de idade (0-5 anos) sobre assuntos relacionados com malária. Esta entrevista via perceber o grau de conhecimento, atitudes-percepções e práticas relacionadas com a prevenção da malária no seio da comunidade. A avaliação das mensagens de malária do PIRCOM será também um dos objectivos esta entrevista. Todas as informações e dados serão mantidas em sigilo, sendo utilizadas para aferir a eficácia da implementação do programa PIRCOM e em particular as suas mensagens. Apenas a equipa que analisa os dados terá acesso aos mesmos. A sua participação é voluntária, sinta-se livre de responder a quaisquer questões. Pode parar a entrevista ou mesmo retirar-se do estudo em qualquer momento, pode pedir clarificação para quaisquer questões ou pedir que repita algo que não tenha percebido. A sua honestidade e precisão nas respostas dadas será altamente apreciada e fundamental para melhor entendermos o quanto eficazes as mensagens de malária foram a influenciar o grau de conhecimento, atitudes/percepções e práticas visando um comportameto preventivo. A entrevista levará aproximadamente 45-60 minutos. Desde já agradecemos a sua colaboração e disponibilidade. Existe alguma questão que gostaria que esclarecesse? Gostaria de participar nesta entrevista? CONSENTIMENTO INFORMADO 81 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Consentimento informado Sim Resposta simples. Siga para a pergunta seguinte Recusado DISPONIBILIDADE Disponibilidade Sim Resposta simples. Siga para a pergunta seguinte O respondente selecionado /elegível está ausente Necessidade de remarcar /revisitar o agregado familiar COORDENADAS GPS DO AGREGADO FAMILIAR - recolha automática 1 Província Nampula Zambézia Resposta simples. Siga para a pergunta seguinte Inhambane 2 Distrito carregar a lista de opções da amostra Resposta simples. Siga para a pergunta seguinte Nampula - Murrupula, Moma, Erati, Ilha Mozambique Zambézia - Inhassunge, Mocuba, Gurue, Pebane, Inhambane - Maxixe, Inharrime, Morrumbene, Vilankulos 3 Comunidade/Bairro carregar a lista de opções da amostra Resposta simples. Siga para a pergunta seguinte Seguir o plano de amostra 4 Algum voluntário do PIRCOM visitou alguma vez o seu agregado familiar nos últimos 12 meses para falar sobre a malária? Sim Se a resposta for SIM para qualquer uma destas perguntas, continue para a pergunta seguinte. Se a resposta for NÃO para todas as perguntas, por favor agradeça ao entrevistado e termine a entrevista. Siga para o próximo agregado familiar. Na P4, se a resposta for NÃO, passe para a P6 Se a resposta for NÃO nos distritos de controlo (Erati, Mussuril, Pebane, Gurue, Vilankulos, Murrumbene) continue com a entrevista, passe directamente para a próxima secção (listagem dos membros do agregado familiar) PERGUNTA DE CONTROLO PARA AS VISITAS DOMICILIÁRIAS DOS VOLUNTÁRIOS DO PIRCOM Pergunta de controlo para : Nampula - Murrupula, Moma Zambézia - Inhassunge, Mocuba Inhambane - Maxixe, Inharrime Não 5 Quantas vezes o voluntário do PIRCOM visitou o agregado familiar durante os últimos 12 meses? 1 vez 2 vezes 3 vezes 4 ou mais vezes 6 Já ouviu falar sobre a malária na rádio comunitária? Sim Não USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 82 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA 7 Algum líder religioso lhe falou sobre a malária nos seus sermões ou palestras? Sim Não LISTAGEM DOS MEMBROS DO AGREGADO FAMILIAR Nº ID 1. Nome de todos os residentes, incluindo visitantes que dormiram no agregado familiar na última noite. Por favor, diga-me os nomes das pessoas que vivem habitualmente neste agregado familiar e dos visitantes que dormiram aqui na noite passada, começando pelo chefe do agregado familiar. (primeiro e último nome; nome e apelido) 2. Idade na altura da entrevista (Anos) Anos completos 0 = Se for menor de 1 ano (meses) 999= Não sabe 3. Sexo: (NOME) é de sexo masculino ou feminino? 1=Masculino 2=Feminino 4. Grau de parentesco em relação ao chefe do agregado familiar: Qual é a relação de parentesco entre (NOME) e o chefe do agregado familiar? 1=Chefe do agregado familiar 2=Cônjuge - Esposo(a) 3=Filho(a) 4=Genro/Nora 5=Neto(a) 6=Pai ou mãe 7=Sogro(a) 8=Avô(ó) 9=Irmão ou irmã 10=Filho adoptado 11=Amigo 12=Residente/visitantes (sem parentesco) 13=Outro (especificar) 98=Não sabe 5. Vive habitualmente nesta casa ou dormiu a noite passada aqui? 1= Vive habitualmente na casa 2= Dormiu a noite passada aqui 6. Encontra-se grávida presentemente? 1=Sim 2=Não 7. A mulher esteve grávida nos últimos 3 anos? 1= Sim 2= Não 8. Este membro do agregado familiar toma de conta de crianças até aos 5 anos (0-5 anos)? 1= Sim 2= Não 1 2 3 4 5 83 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA 6 7 8 9 10 Para assegurar que a listagem de membros do agregado familiar está completa... 1. Relevância - adultos (com mais de 18 anos), sendo mulheres grávidas, mulheres grávidas nos últimos 3 anos, homens e mulheres que tomam conta de crianças até aos 5 anos (0-5 anos) 2. Disponibilidade - estar disponível no agregado familiar aquando da visita ao mesmo 3. Ser capaz de dar a informação sobre o agregado familiar e a sua vivência/ interacção com os assuntos relacionados com a malária 8 Existem outras pessoas como crianças ou bebés que não foram listadas? Sim Se a resposta for SIM, acrescente a pessoa em falta Se a resposta for NÃO siga Não para a pergunta seguinte 9 Existem outras pessoas que não são familiares como empregados domésticos, inquilinos, ou amigos que vivem habitualmente nesta casa? Sim Se a resposta for SIM, acrescente a pessoa em falta Se a resposta for NÃO siga Não para a pergunta seguinte 10 Tem hóspedes, visitantes temporários, ou alguém que tenha dormido nesta casa ontem à noite e que não foram listados? Sim Se a resposta for SIM, acrescente a pessoa em falta Se a resposta for NÃO siga Não para a pergunta seguinte Verificação de eligibilidade 11 Quantidade de crianças até aos 5 anos (0-5 anos) Cálculo automático 12 Quantidade de mulheres com 15-49 anos Cálculo automático Não é eligibilidade, trata-se apenas de 1 referência a mulheres em idade reprodutora 13 Quantidade de mulheres grávidas Cálculo automático ELIGIBILIDADE - ADULTAS 14 Quantidade de mulheres grávidas nos últimos 3 anos Cálculo automático ELIGIBILIDADE - ADULTAS 15 Membros do agregado familiar, com 18 ou mais anos, adultos, que tomam conta de crianças até aos 5 anos (0-5 anos) Cálculo automático ELIGIBILIDADE - ADULTOS, HOMENS E MULHERES Os cuidadores de crianças com menos de 18 anos NÃO serão entrevistados USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 84 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Hábitos de rádio 16 O agregado familiar possui em casa rádio? Sim Resposta simples. Siga para a Não pergunta seguinte 17 Se não possui rádio no seu agregado familiar, tem acesso à rádio por outros meios como rádio no carro, num vizinho ou em algum outro local na comunidade? Sim Resposta simples. Siga para a pergunta seguinte Não 18 Costuma ouvir rádio? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P21 Os entrevistados podem ter acesso à rádio, mas não a ouvir. Não 19 Com que frequência ouve rádio? Diariamente Resposta simples. Siga para a pergunta seguinte Escolha a frequência que reflicta os hábitos médios de escuta do entrevistado. Várias vezes por semana Semanalmente Mensalmente Outra (especificar) 20 Quando é que usualmente ouve rádio? Manhã cedo (6:00-9:00) Possibilidade de múltiplos registos. Siga para a pergunta seguinte Verifique todas as opções que se aplicam. A meio da manhã (9:01-12:00) À tarde (12:01-15:00) Ao fim da tarde (15:01-18:00) Ao princípio da noite (18:01-21:00) Noite (21:01-23:00) Outra (especificar) Religião 21 Participa/frequenta alguma congregação religiosa? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não for NÃO, passe para a P25 22 Que tipo de congregação religiosa frequenta? Lançar a lista de opções incluindo outra (especificar) Resposta simples. Siga para a pergunta seguinte Listagem dos tipos de congregações religiosas na comunidade: BASE INE - Católica, Anglicana, Islâmica, Sião/Zione, Evangélica/Pentecostal, 85 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Sem religião (ateu, animista, agnóstico, ...), outra (especificar) 23 Qual é o nome da congregação religiosa que participa? Lançar a lista de opções incluindo outra (especificar) Resposta simples. Siga para a pergunta seguinte Listagem dos nomes de congregações religiosas na comunidade: Ver base de dados do PIRCOM Incluir outra (especificar) 24 Com que frequência participa nos actos religiosos? Diariamente Resposta simples. Siga para a pergunta seguinte Escolha a frequência que reflicta os hábitos médios de participação nos actos religiosos na sua congregação religiosa. Várias vezes por semana Semanalmente Mensalmente Anualmente Outra (especificar) Conhecimento de Malária 25 Quando você soube sobre a malária pela primeira vez? Não sabe Resposta simples. Siga para a pergunta seguinte Escolha a resposta que melhor represente a resposta dada pelo entrevistado. Muitos anos atrás No último ano Nos últimos meses Nas últimas semanas Nos últimos dias Hoje Outra (especificar) 26 Aonde é que ouviu falar da malária pela primeira vez? Rádio comunitária Resposta simples. Siga para a pergunta seguinte Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 27 Algum voluntário do PIRCOM visitou o seu agregado familiar para falar sobre a malária? Sim Resposta simples. Siga para a pergunta seguinte Pergunta de controlo. PARA TODOS Não aplicável - para os distritos de controlo Não Não sabe USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 86 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Não aplicável 28 Sabe como é que a malária é transmitida? Não sabe Possibilidade de múltiplos registos. Siga para a pergunta seguinte Verifique todas as respostas que se aplicam. Escreva em outra outra opção que não esteja incluído na lista. De pessoa para pessoa Mosquito Animal (rato, cão) Beber água contaminada Comer comida contaminada Outra (especificar) 29 Considera que a malária é uma doença perigosa? Sim Resposta simples. Siga para a pergunta seguinte Não Não sabe 30 Considera que você está em risco de contrair/apanhar a malária? Sim Resposta simples. Siga para a pergunta seguinte Não Não sabe 31 Quem no seu agregado familiar considera que tem mais risco de contrair/apanhar a malária? Mãe Possibilidade de múltiplos registos. Siga para a pergunta seguinte Verifique todas as opções que se aplicam. Mulher grávida Pai Todas as crianças Crianças até aos 5 anos (0-5 anos) Idosos Outra (especificar) Prevenção 32 Considera que a malária pode ser prevenida? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P35 Não Não sabe 33 Qual ou quais dos seguintes métodos considera que ajuda a prevenir e controlar a malária? Uso de repelentes Possibilidade de múltiplos registos. Siga para a pergunta seguinte Pergunta dirigida. Verifique todas as opções que se aplicam. Pulverização intra-domiciliária (dentro da casa) com insecticida Uso de redes mosquiteiras nas camas 87 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Uso das serpentinas contra os mosquitos (dragão) Queima de estrume de vaca/folhas Uso de redes mosquiteiras nas janelas Usando mangas compridas e/ou calças longas Outra (especificar) Nenhuma das opções 34 Que medidas você toma em sua casa para proteger contra a malária? Dormir debaixo da rede mosquiteira Possibilidade de múltiplos registos. Siga para a pergunta seguinte Verifique todas as opções que se aplicam. Escreva quaisquer outras medidas de prevenção em "Outras" Uso das serpentinas contra os mosquitos (dragão) Pulverizar dentro da casa com insecticida Usar mangas compridas e/ou calças longas Usar repelente de mosquito Queimar estrume de vaca/folhas Outra (especificar) Não sabe Redes mosquiteiras 35 Considera que as redes mosquiteiras podem reduzir o risco de malária? Sim Resposta simples. Siga para a pergunta seguinte Não Não sabe 36 Esta casa tem redes mosquiteiras que são usadas para dormir? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não for NÃO, passe para a P42 37 Quantas redes mosquiteiras este agregado familiar tem? resposta numérica Resposta simples. Siga para a pergunta seguinte USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 88 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA 38 Há quanto tempo tem as redes mosquiteiras? Rede mosquiteira 1: Resposta simples. Siga para a pergunta seguinte Forneça uma resposta para cada rede mosquiteira que o entrevistado tenha em casa. Colocar tantas opções quantas redes mosquiteiras o entrevistado reportar na P37 No último ano/12 meses 1-2 anos 3-4 anos 5 ou mais anos Rede mosquiteira 2: No último ano/12 meses 1-2 anos 3-4 anos 5 ou mais anos Rede mosquiteira 3: No último ano/12 meses 1-2 anos 3-4 anos 5 ou mais anos Rede mosquiteira 4: No último ano/12 meses 1-2 anos 3-4 anos 5 ou mais anos Rede mosquiteira 5: No último ano/12 meses 1-2 anos 3-4 anos 5 ou mais anos 39 E quantas redes mosquiteiras estão pendurados nas áreas de dormir? Posso ver? 1 rede mosquiteira Resposta simples. Siga para a pergunta seguinte Registre o número de redes mosquiteiras atualmente penduradas na casa POR FAVOR PERGUNTE SE PODE VER. SE NÃO OBTIVER AUTORIZAÇÃO, QUEIRA 2 redes mosquiteiras 3 redes mosquiteiras 4 ou mais redes mosquiteiras 89 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA REGISTRAR O QUE O ENTREVISTADO REPORTAR. 40 Quantos membros do agregado familiar dormiram debaixo de uma rede mosquiteira na última noite? Nenhum Siga para a próxima pergunta se houve um qualquer membro da família que não dormiu sob o a rede mosquiteira na noite anterior. Se a resposta é nenhum, siga para a P42. Se a resposta é todo o agregado familiar, siga para a P43. Registe o número de membros da família que dormiram debaixo das redes mosquiteiras na noite anterior. Confirme o número total de membros do agregado familiar com o reportado na listagem dos membros do agregado familiar. 1-2 3-4 5 ou mais membros Todo o agregado familiar 41 Quais foram os membros da família na casa que dormiram na última noite debaixo de uma rede mosquiteira? HH ID 1 Continuar para a pergunta seguinte Verifique todas as opções que se aplicam. Consulte a a listagem dos membros do agregado familiar para determinar a identificação HH para cada membro da família. LIGAR ESTA PERGUNTA À LISTAGEM DO DOMÍNIO - APARECER SOMENTE OS MEMBROS REPORTADOS (TBC) HH ID 2 HH ID 3 HH ID 4 HH ID 5 HH ID 6 HH ID 7 HH ID 8 HH ID 9 HH ID 10 42 Qual foi a razão para NÃO ter usado uma rede mosquiteira na última noite? A rede mosquiteira não estava em boas condições Possibilidade de múltiplos registos. Siga para a pergunta seguinte Mais de uma resposta pode ser aplicável se vários membros da família não dormiram debaixo do mosquiteiro na noite anterior. Escreva em ´outro´ quaisquer razões adicionais que não estejam listadas. Não havia espaço suficiente Não gosto do cheiro (químico) É quente e não confortável Não há mosquitos Cria erupções e irritação na pele Dor ardente Não dormiu em casa Não há redes mosquiteiras disponíveis É cara USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 90 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Outra (especificar) Não sabe 43 Tem no seu agregado familiar alguma rede mosquiteira que não é utilizada para dormir? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta oôr NÃO, passe para a P45. TODOS RESPONDEM Não 44 Porque razão tem guardadas redes mosquiteiras? Para quando um amigo/visitante nos visitar Possibilidade de múltiplos registos. Siga para a pergunta seguinte Selecione todas as opções que se aplicam. Para os trabalhadores/empregados De reserva Para a pesca Para a protecção do jardim Para a protecção dos animais Outra (especificar) 45 Você já ouviu alguma mensagem ou informação sobre o uso de redes mosquiteiras tratadas com inseticidade para prevenir a malária? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P48. Não 46 Aonde e por quem é que ouviu esta informação? Rádio comunitária Possibilidade de múltiplos registos. Siga para a pergunta seguinte Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 47 Quando é que ouviu esta informação? Na última semana Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da resposta temporal dada. No último mês Nos últimos 6 meses No último ano Há mais de 1 ano Pulverização de inseticidade 91 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA 48 Considera que a pulverização de inseticida dentro da casa reduz o risco de malária? Sim Possibilidade de múltiplos registos. Siga para a pergunta Não seguinte 49 Você já ouviu mensagens ou informações sobre a pulverização de inseticidas para redução do risco de malária? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não por NÃO, passe para a P52. 50 Aonde e por quem é que ouviu esta informação? Rádio comunitária Possibilidade de múltiplos registos. Siga para a pergunta seguinte Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 51 Quando é que ouviu esta informação? Na última semana Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da resposta temporal dada. No último mês Nos últimos 6 meses No último ano Há mais de 1 ano 52 No último ano, esta casa foi pulverizada? Sim Se a resposta for NÃO, siga para a pergunta seguinte. Se a resposta for SIM ou NÃO SEI, passe para a P54. Não Não sei 53 Porque é que esta casa NÃO foi pulverizada? Não sei Possibilidade de múltiplos registos. Siga para a pergunta seguinte Selecione todas as opções que se aplicam. Não era conveniente Não estava ninguém em casa Não gosto do cheiro Não confiava na pessoa/equipa Foi difícil de preparar A equipa não apareceu A casa estava cheia de pessoas USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 92 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA A pulverização não funciona/não acredito que funcione Tenho medo dos químicos A pulverização atrai mosquitos A pulverização causa comichão e irritação Tenho bébé ou pessoa doente em casa Outra (especificar) 54 Com que frequência considera que a sua casa deveria ser pulverizada? Cada 6 meses Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da resposta temporal dada. Cada 9-12 meses Cada 2 anos Não sabe Cuidados pré-natais 55 Você acha que os cuidados pré-natais fornecem informações úteis às mulheres grávidas para prevenir ou tratar a malária? Sim Resposta simples. Siga para a pergunta seguinte Não 56 Será ou não provável/possível que você ou mulheres grávidas em sua casa tenham cuidados pré-natais regulares durante a gravidez? Extremamente provável Resposta simples. Siga para a pergunta seguinte Provável Neutro Improvável Extremamente improvável 57 Para si, será ou não importante que as mulheres grávidas façam o tratamento preventivo contra a malária? Muito importante Se a resposta for ´muito importante´ ou ´um pouco importante´, continue a próxima pergunta. Caso contrário, passe para a P59. Resposta simples. Um pouco importante Neutro Não é muito importante Nada importante 58 1 93 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Se considera que tomar tratamento preventivo contra a malária é muito importante ou um pouco importante, quantos tratamentos acha necessários tomar durante a gravidez? 2 Resposta simples. Siga para a pergunta seguinte 3 4 ou mais 59 Já ouviu mensagens ou informações sobre a importância de tomar pelo menos 3 doses de tratamento preventivo da malária durante a gravidez? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P62 Não 60 Aonde e por quem é que ouviu esta informação? Rádio comunitária Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 61 Quando é que ouviu esta informação? Na última semana Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da resposta temporal dada. No último mês Nos últimos 6 meses No último ano Há mais de 1 ano 62 Durante a gravidez mais recente em sua casa, a mulher grávida recebeu cuidados pré￾natais? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P66 Explique que esta questão se refere aos cuidados pré-natais durante a gravidez mais recente que houve no seu agregado familiar. Não Não sei 63 E com que frequência essa mulher grávida recebeu os cuidados pré-natal durante a gravidez? 1 vez Resposta simples. Siga para a pergunta seguinte Explique que esta questão se refere à frequência de acesso aos cuidados pré-natais durante a gravidez mais recente que houve no seu agregado familiar. 2 vezes 3 vezes Mais do que 3 vezes Não sabe 64 Sim USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 94 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Durante a gravidez mais recente, esta mulher recebeu tratamentos preventivos de malária? Não Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P66 Explique que esta questão se refere aos tratamentos preventivos de malária durante a gravidez mais recente que houve no seu agregado familiar. Não sei 65 Quantos tratamentos preventivos contra a malária foram recebidos durante a gravidez mais recente? 0 Resposta simples. Siga para a pergunta seguinte Explique que esta questão se refere à frequência dos tratamentos preventivos de malária durante a gravidez mais recente que houve no seu agregado familiar. 1 2 3 4 ou mais Tratamento da malária 66 Que sintomas associa à malária? Temperatura alta/ febre Possibilidade de múltiplos registos. Siga para a pergunta seguinte Registe todas as respostas que se aplicam. Escreva/adicione em todos os sintomas que não estão incluídos na lista. Perda de apetite Dor de cabeça Dores no corpo Arrepios Fadiga Náusea ou vómito Diarréia Tontura Outra (especificar) Não sabe 67 Se você ou algum membro do seu agregado familiar apresentasse sintomas da malária, o que faria? Procuro tratamento Resposta simples. Siga para a pergunta seguinte Se outro tratamento, especifique em ´outro´. Aguardo para ver se os sintomas pioram Trato os sintomas em casa Não faço nada Outra (especificar) 68 Se procurasse tratamento para a malária, onde iria? Centro de saúde/posto de saúde/hospital/clínica Resposta simples. Siga para a pergunta seguinte ATENÇÃO: APENAS SE PRETENDE UMA ÚNICA RESPOSTA. QUAL A PRINCIPAL/PRIMEIRA ESCOLHA Agente comunitário de saúde DO ENTREVISTADO 95 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Curandeiro/médico tradicional Farmácia Auto-tratamento A lado nenhum Não sabe Outra (especificar) 69 Considera que precisa procurar tratamento médico para sintomas associados à malária? Sim Resposta simples. Siga para a pergunta seguinte Não Não sei 70 Quando deve procurar assistência médica para sintomas de malária? Imediatamente (nas primeiras 24 horas) Resposta simples. Siga para a pergunta seguinte 24-48 horas após os sintomas Mais de 48 horas após os sintomas Não sei 71 Já ouviu mensagens ou informações sobre os sintomas da malária? Sim Resposta simples. Siga para a Não pergunta seguinte 72 Já ouviu mensagens ou informações sobre a importância de procurar tratamento médico nas primeiras 24 horas à apresentação dos sintomas de malária? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P76 Não 73 Aonde e por quem é que ouviu esta informação? Rádio comunitária Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 74 Quando é que ouviu esta informação? Na última semana Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da No último mês resposta temporal dada. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 96 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Nos últimos 6 meses No último ano Há mais de 1 ano 75 Considera que precisa continuar o tratamento da malária quando os sintomas desaparecerem? Sim Resposta simples. Siga para a pergunta seguinte Não Não sei 76 Já ouviu alguma mensagem ou informação sobre a importância de completar o tratamento contra a malária? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não for NÃO, passe para a P79 77 Aonde e por quem é que ouviu esta informação? Rádio comunitária Líder religioso do PIRCOM Membros da comunidade Visita domiciliária por parte de um voluntário comunitário do PIRCOM Outra (especificar) 78 Quando é que ouviu esta informação? Na última semana Resposta simples. Siga para a pergunta seguinte Registe a opção mais próxima da resposta temporal dada. No último mês Nos últimos 6 meses No último ano Há mais de 1 ano 79 Você ou algum membro deste agregado familiar sofreu de malária nos últimos 6 meses? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não for NÃO, passe para a P89 80 Quais foram os membros do agregado familiar que sofreram de malária nos últimos 6 meses? HH ID 1 Continuar para a pergunta seguinte Verifique todas as opções que se aplicam. Consulte a a listagem dos membros do agregado familiar para determinar a identificação HH para cada membro da família. LIGAR ESTA PERGUNTA À LISTAGEM DO DOMÍNIO - APARECER SOMENTE OS HH ID 2 HH ID 3 HH ID 4 HH ID 5 HH ID 6 97 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA HH ID 7 MEMBROS REPORTADOS (TBC) LINKED TO HOUSEHOLD LISTING HH ID 8 HH ID 9 HH ID 10 81 Quais foram os sintomas que este membro(s) do agregado familiar apresentava(m)? Temperatura alta/ febre Possibilidade de múltiplos registos. Siga para a pergunta seguinte Registe todas as respostas que se aplicam. Escreva/adicione em todos os sintomas que não estão incluídos na lista. Perda de apetite Dor de cabeça Dores no corpo Arrepios Fadiga Náusea ou vómito Diarréia Tontura Outra (especificar) Não sabe 82 No momento dos sintomas, alguma desta(s) pessoa(s) estava(m) grávida(s)? Sim Resposta simples. Siga para a pergunta seguinte PERGUNTA DE CONTROLO Não 83 No momento dos sintomas, alguma desta(s) pessoa(s) era uma criança com menos de 5 anos (0-5 anos)? Sim Resposta simples. Siga para a pergunta seguinte PERGUNTA DE CONTROLO Não 84 Aquando da apresentação dos sintomas, essa(s) pessoa(s) procurou/procuraram atendimento médico? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta Não for NÃO, passe para a P89 85 Quanto tempo após a apresentação dos sintomas, este membro do agregado familiar procurou atendimento médico? Imediatamente (nas primeiras 24 horas) Resposta simples. Siga para a pergunta seguinte 24-48 horas após os sintomas Mais de 48 horas após os sintomas Não sei USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 98 QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA 86 Aonde é que este membro do agregado familiar procurou atendimento médico? Centro de saúde/posto de saúde/hospital/clínica Resposta simples. Siga para a pergunta seguinte Agente comunitário de saúde Curandeiro/médico tradicional Farmácia Não sabe Outra (especificar) 87 Este membro do agregado familiar recebeu tratamento anti-malárico? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, passe para a P89 Não 88 Este membro do agregado familiar completou o tratamento anti-malárico? Sim Resposta simples. Siga para a Não pergunta seguinte Informação sobre a malária 89 Considera que tem informações suficientes sobre a malária? Sim Resposta simples. Siga para a Não pergunta seguinte 90 Gostaria de ter mais informações sobre a malária? Sim Se a resposta for SIM, siga para a pergunta seguinte. Se a resposta for NÃO, agradeça ao respondente e termine a entrevista Não 91 Que tipo de informação adicional gostaria de ter? Informação sobre o tratamento Possibilidade de múltiplos registos. Siga para a pergunta seguinte Registe todas as respostas que se aplicam. Informação sobre controle de mosquitos Informação sobre prevenção Sinais e sintomas Informação sobre os efeitos da malária na gravidez Informação sobre os efeitos da malária em crianças Natureza da doença Qualquer informação Outra (especificar) 92 Membro da família Agradeça ao respondente e termine a entrevista Registe todas as respostas que se Amigo ou vizinho aplicam. 99 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV QUESTIONÁRIO SOBRE AS MENSAGENS DE MALÁRIA Onde ou de quem você gostaria de receber essa informação? Rádio Instituição/congregação religiosa Curandeiro/médico tradicional Cartazes/panfletos Escola Unidade de saúde Voluntário da saúde Visita domiciliária (porta-a-porta) de um voluntário da comunidade do PIRCOM Reunião da comunidade Outra (especificar) Terminar o questionário. AGRADECER A DISPONIBILIDADE PARA A ENTREVISTA. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 100 ANNEX G: FINDINGS/CONCLUSIONS/RECOMMENDATIONS MATRICES Evaluation Question # 1: How effective were the PIRCOM malaria messages and delivery at influencing target audience malaria knowledge and practices? Analysis of Qualitative Data Findings Conclusions Recommendations Lessons learned 1. Religious leaders are highly respected and trusted individuals. Respondents viewed the RLs as the most effective channel to disseminate malaria messages. 1. Disseminating messages via RLs seems to be the most effective channel to reach out to the communities with malaria messages. F 1, C2 1. In future activities, malaria messages via RLs can be expanded significantly. PIRCOM has the capacity to reach out to broader communities/individuals via its extensive network of RLs. C 1 2. PIRCOM is partnering with 7 community radio stations. Community radios are an important media platform available in most of the rural households with over 50% ownership and are important for disseminating info. However, the team was not able to find evidence that community radios have been effective in disseminating malaria messages. 2. The effectiveness of the radio messages is limited due to radio stations’ coverage. The timing of airing messages may not be appropriate to reach intended audiences. Detailed information on monitoring of community radios is provided under Evaluation Q#3. F 2,3,4,5,6 2. Future activities may explore the possibility of broadcasting malaria messages via the national radio channel. The national radio broadcast malaria messages along with other diseases but airing of malaria messages is infrequent. C 2 3. In principle, community radios cover an area of 60-75 km. in radius. However, the actual coverage is much lower, probably less than 40 km. The reach of the stations 3. The reach of home visits is limited due to low motivation of the volunteers. The number of HHs visited are low and number and frequency of visits/revisits are not standardized nor strategically planned. 3. Home visits conducted by volunteers should be increased, standardized and better organized. Volunteers should receive comparable incentives as provided by other organizations. Number of 101 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV declines as broadcasting equipment wears out. Home visits by volunteers may be an effective approach, but their coverage is low. F 7,8,9 volunteers should be reduced significantly and number of HHs per volunteer needs to be increased to reasonable levels. C 3 4. According to IREX, the exact coverage of community radios and % of people listening to radios cannot be known because none of the stations have conducted audience research. Audience research is expensive and beyond the capacity of the community radio stations. 4. There is anecdotal evidence to suggest that PIRCOM interventions have been effective in improving knowledge and behaviors. F 10 4. Community leaders should be involved in dissemination of malaria messages in future activities. C 5 5. IREX works with other USAID IPs and provides assistance in monitoring community radios when needed. IREX did not work with PIRCOM because PIRCOM did consult with IREX. 5. Community leaders have many strengths to play a significant role in disseminating malaria messages and mobilizing the communities. Not involving community leaders in the PIRCOM activity is a missed opportunity. F 12 5. Future activities should focus resources on a small number of high priority provinces and districts and strategize to reach a significant proportion of communities in each district to increase interventions’ effectiveness and efficiency. C 7 6. The radio stations interviewed broadcasted malaria messages 4 times a week for 15-20 minutes, during mid￾morning and some days late afternoon and Saturdays. Few respondents mentioned that they have been listening to community radios during KIIs and FGDs. National radio station was favored by several respondents. 6. It is not possible to measure the effectiveness of the interventions based on data reported by the activity. F 13 6. Future activities should include relevant quantitative and qualitative output/outcome indicators to measure the effectiveness of interventions. C 13 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 102 7. The number of HHs reached by volunteers is limited. Volunteers visit 3 houses per month. It is estimated that 2.8% of HHs are visited in Murrupula, 2.9% in Inharrime and 7.2% in Mocuba. The number of HHs visited is different in each province and even between communities of the same district. Reportedly, volunteers visit 10-40 HHs in a community. There are no standards on how to select HHs and how many HH each volunteer should visit each month. Only in one community were the HHs with pregnant women and children under 5 were selected strategically. There are no plans to roll out to other HHs. 7. The activity is being implemented across a large number of provinces and districts, spreading resources too thinly, which limits the interventions’ effectiveness and efficiency. F 11 7. Future activities should aim to increase IRS knowledge and coverage, and PIRCOM can play an instrumental role in increasing IRS knowledge and practices via mobilizing the communities. C 8,10 8. There are also differences among communities and districts regarding revisits. There is no standardization on how many revisits are required/planned. 8. The activity was effective in improving knowledge on malaria prevention and treatmentm F 14-21 9. Volunteers’ motivation is low because they are not paid. See barriers under EQ#2. 9. PIRCOM messages were not the sole source of information. The majority of respondents learned about malaria from “other” sources, including health facilities/healthcare workers, national radio and other family members. F 22,23,24 10. PIRCOM malaria activity is highly regarded by all respondents interviewed who know about the interventions and who have been exposed to the messages. Specific 10. While the activity was effective in improving knowledge on malaria, it was less effective in changing behaviors, except for IRS. Positive behaviors to prevent/ 103 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV examples on the effectiveness of the activity included improved knowledge and behavior change for prevention and treatment of malaria. Correct knowledge regarding malaria, use of mosquito nets, demand for antenatal visits and use of IPT have increased. treat malaria were already high, leaving little room for improvement. Remaining small groups are probably harder to reach and it takes more time to achieve behavior change in those communities. F 26-30 11. The activity is being implemented in 29 districts in 5 provinces. 12. Community leaders interviewed were highly interested in PIRCOM activities and wanted to be trained and disseminate malaria messages. Community leaders have close ties with the communities they work in. 13. Data on quantitative process indicators reported by PIRCOM measure progress towards set targets. PIRCOM has been successful in achieving targets. However, the data does not measure the effectiveness of the interventions. 14. There is significant difference between PDs and NPDs regarding knowledge on how malaria is transmitted. In PD, more people know that malaria is transmitted by mosquitos. The finding is the same for males and females. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 104 15. There is significant difference between PDs and NPDs on perception of being at risk of malaria. In PDs, more people were aware that they are at risk. The finding is the same for males and females. 16. There is significant difference between PDs and NPDs on knowledge about high-risk groups. In PD, more respondents knew that pregnant women and children under 5 are at high risk. The difference between PDs and NPDs was not significant for women. 17. The knowledge that malaria can be prevented was significantly higher in PDs (93.2%) compared to NPDs (87.4 %). (P<1%). The finding was the same for males and females. 18. The knowledge on ways to prevent malaria (sleeping under mosquito nets and spraying the house with insecticide) was significantly higher in PDs. (P<5% and P<1%). The finding was the same for males and females. 19. Knowledge on mosquito nets were high both in PDs and NPDs (95.8% and 92.5% respectively, knowledge of spraying was very low in both groups (10.1% and 4.3% respectively). 20. Knowledge that antenatal care provides helpful information to prevent and treat malaria was significantly higher in PDs. The finding was the same for females and males. 105 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 21. Knowledge on symptoms associated with malaria was not different between the PDs and NPDs except that more respondents associated high fever and loss of appetite with malaria in PDs. The finding was the same for females and males. 22. The percentage of respondents who heard about malaria from the community radios was low and not significantly different in PDs and NPDs (8.8% and 7.3% respectively). When disaggregated by gender, a higher percentage of males heard about malaria from the community radios both in PDs and NPDs. (12.0% and 10.2% respectively). 23. In PDs, 42.8% of the respondents learned about mosquito nets from community radios (2.9%), PIRCOM religious leaders (5%) or PIRCOM volunteers (34.9%). Remaining 57.2 % learned from other sources. The finding was same for males and females. Among the other category, health care facilities/workers family members, and the national radio were the most common sources of information in both groups. 24. 58.8% of the respondents learned about indoor spraying from community radios (14.6%), PIRCOM religious leaders (11.9%) USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 106 or PIRCOM volunteers (32.9%). The finding was same for males and females. Among the other category, family members, health care facilities/workers and the national radio were the most common sources of information in both groups. 25. Percentage of respondents who heard about spraying from community radios was not different between PDs and NPDs. 26. Sleeping under mosquito nets was high in both PDs and NPDs (95.1% and 94.8%) and the difference was not significant. The finding was similar for males and females. 27. Spraying inside the house was low in both groups (5.9% and 2.1%) but the difference was statistically significant (P<1%). The finding can be attributed to PIRCOM’s social mobilization efforts to facilitate spraying of houses. The finding was similar for female and male respondents. 28. The percentage of women who received antenatal care during most recent pregnancy was high in both groups (72.9% and 71.3%) and the difference was not significant. 29. The percentage of respondents who sought immediate medical care was high in 107 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV both groups (94.8% and 92.5%) and the difference was not significant. The finding was the same for male and female respondents. 30. The respondents who completed medical treatment was high in both PDs and NPDs (95.6% and 97.8%) and the difference was not significant. The finding was the same for male and female respondents. Evaluation Question # 2: What were the principal barriers and successes during implementation of the PIRCOM scope of work? a. How were the barriers addressed by the program? b. In what ways did PIRCOM build upon successes to improve implementation? c. How did PIRCOM work with other stakeholders (MISAU and other IPs) in coordinating interventions in the same geographic area? Successes Findings Conclusions Recommendations Lessons learned 1. While PIRCOM has achieved significant progress since its inception 10 years ago, it is still a young organization. The activity helped PIRCOM to grow and strengthen its organizational and technical capacity although there is room for further improvement. PIRCOM leadership is aware of its shortcomings and committed to grow as an institution. 1. PIRCOM is a well-known and respected organization and its institutional capacity has been improving since its establishment in 2007. The leadership is committed to improving the performance of the activity and further developing their capacity as an organization. F 1,2 1. PIRCOM is well positioned to play an important role in SBCC efforts for malaria prevention and treatment in future activity designs. C 1, 2 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 108 2. Institutional capacity assessments conducted by JHU and Eurosis reveal that despite progress over the years there is much room for PIRCOM to grow as a local organization. PIRCOM leadership is aware of its shortcomings and is motivated to address them. 2. PIRCOM has been successful in training large numbers of religious leaders and volunteers on malaria messages and how to disseminate them. F 3 3. PIRCOM was successful in training large numbers of delegates, religious leaders and volunteers on dissemination of malaria messages. Field interview and discussions with PIRCOM indicate that training has been successful, individuals were well informed on malaria messages and how to disseminate them. Barriers Findings Conclusions Recommendations Lessons learned 1. Volunteers do not receive stipends for their labor. They are aware that other staff are paid, as are volunteers supporting other activities. 1. Lack of stipends/incentives for volunteers reduces their motivation and is likely to be a barrier to expand coverage in the communities. F 1,2 1. Future activities should significantly reduce number of volunteers, aim to increase efficiency of home visits and provide comparable monetary incentives as offered by other organizations. Already discussed under Q#1 C 1,2 109 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 2. The fact that some other IPs working in the same provinces receive monetary incentives also has an effect on volunteers’ motivation. Trained PIRCOM volunteers are quitting to work for other organizations. Out of 6 communities visited, volunteers’ motivation was quite high only in one community (in Zambézia). 2. Providing monetary stipends/incentives for the volunteers is not feasible due to the large numbers of volunteers. F 3 2. PIRCOM needs to take a closer look into the issue of frequent breakdown and maintenance of bicycles. C 3 3. PIRCOM cannot pay the volunteers because of the large numbers of volunteers. In total, PIRCOM has about 1040 volunteers. Even a minimum amount of monetary stipend/incentive would require half of PIRCOM’s overall budget, according to a Eurosis assessment of PIRCOM financial resources. 3. Providing bicycles for the volunteers to ease transportation and as an incentive was a sound solution but the quality of bicycles is an issue. Transportation from the communities to the districts and provinces continues to be a challenge due to long distances and costs. F 4,5 3. PIRCOM should revise and clarify its message regarding malaria symptoms to avoid future confusion. C 4 4. Volunteers received bicycles to help transportation and also as an incentive. However, in Nampula and Inhambane almost all bicycles broke down during the first year. There are no funds for maintenance of bicycles and reportedly, the quality is so bad, it is not possible to repair them. The situation was better in Zambezia, where a majority of bicycles were still functional. 4. Confusion over the symptoms associated with malaria is a challenge. The message regarding malaria symptoms may need revising to clarify that those symptoms are not unique for malaria and may be associated with other sicknesses. F 6 5. Transportation between communities, districts and provinces is an important challenge given the long distances and costs. Supervisors need to carry the USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 110 forms to the district and districts transport them to the provinces, most of the time paying out of their pockets. Transportation within the communities, although a complaint brought up frequently by the volunteers, may not be that important. 6. The team heard complaints from the communities that malaria treatment was not available when individuals went to the clinics with symptoms of malaria. The issue was clarified with the MOH. Most of the symptoms associated with malaria are not unique to malaria. There was confusion/frustration when the individual thought he/she has malaria but was not treated for malaria. Lack of medication for treatment of malaria may be an issue only in few isolated areas, according to MOH officers. Coordination with MOH and other IPs in the same geographic area Findings Conclusions Recommendations Lessons learned 1. PIRCOM works together well with the MOH at the central, provincial and district levels. When requested, PIRCOM attends coordination meetings and is an active member of the technical coordination meetings. PIRCOM routinely reports to the central MOH. 1. PIRCOM coordinates well with the MOH at all levels and is responsive to their requests although level of coordination largely depends on the MOH’s interest in strengthening coordination with the IPs. There is room to strengthen more effective coordination: PIRCOM could 111 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV have been more active in approaching the MOH. F 1 2. A provincial officer interviewed wanted information from PIRCOM on daily activities and another officer requested information on financial resources. 2. In provinces where MOH officers were critical about the level of coordination with PIRCOM, they actually wanted to have more “control” over the organization’s activities and resources. F 2 3. PIRCOM worked with Abt Associates in Zambezia on indoor spraying. Abt stated that PIRCOM was instrumental in mobilizing and organizing communities for spraying. 3. PIRCOM’s coordination with other IPs has also been productive and satisfactory. F 3,4,5 4. UNICEF worked with PIRCOM to roll out cascading training on health promotion, nutrition, child protection and education key priority behaviors. 5. JHU supported capacity building and monitoring of PIRCOM activities through a framework of community dialogue project in four districts from 2015 to 2017. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 112 Evaluation Question # 3: How effectively did the PIRCOM M&E system function? a. How could the PIRCOM and other similar projects’ M&E systems be improved? b. How effective was monitoring of community radio PIRCOM messages? Are there ways to improve such monitoring in future programs? Design of the M&E Plan Findings Conclusions Recommendations Lessons learned 1. The activity includes 17 process indicators. All of them are quantitative, providing numbers such as # of religious leaders completing training, # of community members reached with malaria messages through home visits, # of IEC materials reproduced. etc. 1. The activity is reporting on too many process indicators but is not reporting on output/outcomes designed to measure the results of interventions The MEL plan is not evaluating the activity’s performance but it is a tool for internal monitoring of the inputs. F 1, 2, 3 1. Future activity M&E plans should have a balanced mix of fewer process indicators and a reasonable number of output/outcome indicators. Process indicators are needed to track the inputs provided to achieve intended results and output/outcome indicators to measure the achievement of results. Indicators should be selected based on the availability/feasibility of collecting data. C 1,2,3 2. PIRCOM reports on all process indicators and tracks progress against set targets. The activity does not report on output indicators because the information on output indicators is not available. 2. The output indicators are not appropriate because information for the “denominator” is not available for most, e.g., “proportion of target audience” and some others are not meaningful, e.g., “proportion of IEC materials reproduced.” 113 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV F 2,3 3. Each process indicator has a matching output defined as the proportion of the total, e.g., proportion of community members achieved by malaria messages through home visits. 3. The indicator on number of community members reached through sermons is subject to double counting. The indicator on the number of people reached by the community radios is neither accurate nor useful. F 4,5 4. Mostly, the same individuals attend churches/mosques on a weekly basis but are reported as if there is a new group attending the sermons every week. 5. It is not possible to estimate the number of HHs or individuals reached by radio messages because none of the CRs have conducted audience research. Data Collection System Findings Conclusions Recommendations Lessons learned 1. Volunteers and religious leaders report on a monthly basis by manually filling in forms. Forms are collected by supervisors and sent to district delegates and finally to the provincial coordinators who entered data into excel sheets during the first two years of the project. Three separate databases were sent to PIRCOM M&E officer. 1. Although reporting has improved over the years, the data collection system is labor intensive and time consuming, due to large number of forms and frequency of reporting. F 1,2,3 1. Frequency of reporting can be changed from monthly to quarterly reporting. This will significantly reduce the number of forms. C 1 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 114 2. There were delays in reporting initially, which improved over time. Reporting was timely during the evaluation. In only one community did the supervisor report that reports were late. 2. Using an Access database instead of three excel spreadsheets helped mainstreaming reporting from the provinces to the main office, but the change does not seem to ease the information flow between the lower levels. F 4, 5 3. Eurosis (a firm providing capacity building assistance to PIRCOM) estimated that about 50,000 forms were collected from the communities annually and the numbers are increasing each year. 4. Eurosis provided assistance to mainstream data collection and suggested using smartphones in the field. This option was dropped due to cost implications ($15,000 per year) and sustainability concerns. 5. Instead, Eurosis assisted to improve the efficiency of the current system by combining 3 excel data sets at the provincial level to automatically produce one access database. Thus, the headquarters receive one access database instead of three excel sheets from the provinces. 115 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Data Quality Findings Conclusions Recommendations Lessons learned 1. Quality of the data collected is questionable due to 1) large numbers of indicator data requested 2) large numbers of forms manually completed 3) capacity and motivation of volunteers using the forms, and 4) inadequate capacity at the community level to monitor the quality of data. Supervisors are supposed to check on data quality, but it is not happening. Supervisors interviewed reported that they check on the completeness of the forms and timely submission. 1. The quality and reliability of data is poor, although there have been recent improvements. PIRCOM is aware of the weaknesses of its M&E system and committed to improving the quality of the data collected. F 1, 2 1. A more robust activity design with fewer number of motivated volunteers to collect data, and leaner M&E system with fewer and more appropriate indicators would help to improve the quality of the data. Use of more advanced technologies e.g., smart phones, if possible, would also help to improve the quality of data. C 1,2 2. PIRCOM is aware of the weaknesses of data quality and is conducting intense trainings to improve it. For example, reporting is more regular. Double counting is recognized as a weakness and solutions are developed. Data Analysis and Use of Data to Improve Performance Findings Conclusions Recommendations Lessons learned USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 116 1. Data are analyzed at headquarters. The analysis at the headquarters involves comparing the indicator values against set targets for each indicator. Based on the analysis, PIRCOM reports on progress towards targets. 1. The fact that data are not analyzed at lower levels is a missed opportunity to use data for decision-making and improving performance. Provincial coordinators’ role would be crucial in data analysis but they are too busy with entering thousands of forms into the data bases. They don’t have the time nor the responsibility to analyze data. F 1 2. PIRCOM leadership was able to make changes in implementation to improve the activity’s performance. These improvements included streamlining the flow of information to allow more timely reporting, strengthening the training programs and rescheduling/reorganization home visits. 2. The improvements in PIRCOM implementation are not necessarily based on its analysis of the M&E data. The improvements were largely based on the feedback from and observations in the field. F 2 Effectiveness of Monitoring of Community Radio Messages Findings Conclusions Recommendations Lessons learned 1. PIRCOM has contracts with 7 community radios, which provide detailed instructions on how to air malaria messages. 1. The timing of airing malaria messages does not seem appropriate. They are missing out on the working young adults and also the youth who would be most receptive for behavior change. F 4,5,6 1. Future design of activities aiming to use community radios to disseminate malaria messages should consider 1) more flexible timing for airing the messages to reach working young adults and youth; 2) use of brief spots like advertisements throughout the day rather than lecturing for 15-20 minutes; 3) participatory formats to attract the attention of the intended audiences. 117 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV C 2,3,4 2. PIRCOM does not directly monitor the messages but has confidence in the radio stations doing a good job. The radio stations themselves monitor the dissemination of malaria messages. 2. The format used- lecturing for 20 minutes- might be tedious. A combination of lectures and shorter spots for a few minutes throughout the day might be more attractive for the intended audiences. F 6 2. Future activities should consult with radio communication experts on the most effective ways to reach out to the intended audiences prior to specifying the terms of their contracts with the community radio stations. C 2,3,4 3. Radio stations translate messages into local languages. Stations have expertise in translations, no issues were reported related to the quality of translations. Stations broadcast in Portuguese and the most common local language used in the districts/communities they cover. 3. There is room to explore other innovative/participatory formats to air the malaria messages to attract the intended audience. “Open broadcasting” when people can call in and ask questions was successful in Nampula and Mocuba. F 7 4. PIRCOM requires the radios to air messages at mid-morning during week days and once during the weekends. Only in Mocuba are messages broadcasted twice a week at in the afternoon. Message format is lecturing on malaria messages for 15-20 minutes. In Nampula and Mocuba the stations air “open broadcast” when audience can call in. 4. Not consulting with available technical resources on radio communication experts such as IREX while formulating the contracts with the CRs and monitoring of the messages is a missed opportunity. 5. Radio stations have concerns about the timing and format of the broadcasting, but they have to follow the instructions provided by PIRCOM in their contracts. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 118 6. Challenges cited by the stations: Majority of working people and youth do not listen to radios during mid-morning. The format (lecturing for 20 minutes) was boring. Stations would prefer shorter spots for a few minutes (like advertisements) spread out throughout the day but they are bound with the stipulations of the contracts with PIRCOM. 7. There are other ways to attract the audience to listen to the malaria messages. For example, broadcasting competitions on who knows best about malaria prevention/ treatments. Such formats are not used. 8. IREX commented that they can provide technical assistance in monitoring of radio messages, as they assist other projects/organizations. PIRCOM did not request such assistance from IREX. 119 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV Gender Analysis Findings Conclusions Recommendations Lessons learned 1. The activity does not have a clear strategy on how to reach the main target group – pregnant women and children under 5. However, PIRCOM has a plan to develop a policy and guide on how to approach gender issues, to be initiated in April 2018. The plan is based on the findings and recommendations of the capacity development assessment conducted by Eurosis 1. Community members report following the messages regarding specific care for the pregnant women and children, influenced specifically by the community and religious leaders, and by volunteers. F 1, 2, 3,4,5 1. PIRCOM should begin implementing the gender policy and guides as soon as they are available. Provincial coordinators and delegates need to be trained on gender approaches who will orient religious leaders, supervisors and volunteers in return. C 1-3 2. As there is no specific orientation/guidance for the volunteers or religious leaders on how to address gender issues, the approach to men and women depended on the individual volunteer or the religious leader. 2. The gender of the volunteers is not of importance any longer with respect to gaining access to households, as both male and female volunteers are introduced by the community leaders as PIRCOM activists and are well accepted by the community members. F 6 2. Future activities should incorporate a gender analysis, gender policies, strategies and training plans. Strategies should consider the religion of men and women in the communities. C 1,2,3,4 3. Based on the interviews/discussions in the field, some religious leaders take specific action to reach the pregnant women and children under 5, e.g., if there are many pregnant women present at church, the 3. Men are the primary decision-makers, but this doesn’t adversely affect women’s behavior change. The messages were understood, accepted and followed by both men and women. 3. In future programs, gender-specific messages could be strategically included while disseminating malaria messages. C 1,2 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 120 religious leader will attend to them after the sermon. F 7 4. PIRCOM male workers approach male community members using a poster called “Role of the Father.” The poster was developed by the C- Change project. Male and female respondents reported that community members follow the messages. FGD men in Munhiba: ´Previously we didn’t know how to treat the wife during the pregnancy but now we know´. 4. There is a gender segregation among the religious congregations. The female Muslims don’t have the same opportunity as Muslim men to receive information during the sermons. Few Muslim women attend the Friday prayers at the mosques while male Christian community members who are churchgoers get more information than the male members who don’t attend the church. F 9 4. The gender strategy should include an organizational analysis on gender balance and explore ways to increase the number of women delegates, religious leaders and volunteers. The number of female volunteers should be increased in communities where the majority or all of volunteers are male. In particular, more Muslim women could be trained as volunteers. F 8,10 5. The messages regarding pregnant women, e.g that husbands should accompany their wives to the antenatal care to make sure she is receiving the IPT, are fortified by the health care providers and community leaders. Specifically, the community leaders address women’s needs ¨We give the priority to the woman, she is our mother¨. PIRCOM Province coordinator Quelimane: ´There were a lot of prejudice previously – the pregnant women didn’t take IPT as there were those myths that they would miscarriage. Now the men are told to take their women to the clinic to antenatal care, by bicycle, wait for them there, and take her home. The men have changed their behavior – previously they prohibited their wives to go to the clinic. ´ 5. Separate community meetings for men and women could be expanded in Christian and Muslim communities to target Muslim women and Christian men, who do not attend the sermons. F 9 121 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 6. At the beginning of the activity home visits by male volunteers were not welcome when men weren’t present at the house. This is no longer an issue. Now, volunteers are well recognized in the communities. Community leaders also helped by informing and preparing communities to accept home visits made by male volunteers. As a supervisor in Muaquia noted, “The male volunteers are able to enter the HH even if there is no man present at the moment. I never heard of any problems.” 7. Traditional gender roles for men and women may affect the women´s participation in decision-making and practice behaviors learned from malaria messages. However, both male and female respondents confirmed that the messages are followed correctly. Both men and women in FGDs indicated that women are taken better care of during pregnancy. In Zambézia, women´s participation in decision-making seems higher compared to the other provinces. The supervisor in Munhiba noted that, ´¨The head of the family is the woman – she has the command. And the husband follows her orientation¨. 8. More than half of the volunteers are men; in some communities visited, there were no female volunteers. In 5 communities visited by the team, there were a total of 70 volunteers, 59% male and 41% female. USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 122 Also, among PIRCOM leaders, coordinators, delegates and supervisors, there are more males than females – there are 3 women in leading positions at the HQ but only one female district delegate out of 28. 9. Muslim religious leaders hardly have Muslim women at the mosque, and on the other hand, the Christian religious leaders address mostly women at the church. 10. Among both the Muslim and Christian religious leaders, there are more men than women. In the 6 communities visited by the team, there were a total of 38 RLs, 29 males and 9 females. 123 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV ANNEX H: RESULTS FROM KAP SURVEY 1-MALARIA KNOWLEDGE TABLE 1: KNOWLEDGE ABOUT MALARIA DISEASE When did you hear about malaria? PIRCOM Non PIRCOM P value Effect size n % n % DO NOT KNOW All 47 7,1 31 4,7 0,079 Many years ago* 538 81,8 591 89,8 <0,001 Last year* 43 6,5 22 3,3 0,010 0.14 In recent months* 30 4,6 14 2,1 0,020 0.13 DO NOT KNOW Female 27 6,9 23 5,5 0,465 Many years ago** 316 80,8 377 89,3 0,001 Last year** 29 7,4 11 2,6 0,002 0.22 In recent months 19 4,9 11 2,6 0,096 DO NOT KNOW Male 20 7,5 8 3,4 0,052 Many years ago* 222 83,1 214 90,7 0,017 Last year 14 5,2 11 4,7 0,839 In recent months 11 4,1 3 1,3 0,060 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 124 TABLE 2 KNOWLEDGE ABOUT HOW MALARIA IS TRANSMITTED Do you know how malaria is transmitted? PIRCOM Non PIRCOM P value Effect size n % n % Mosquito** All 568 86,3 457 69,5 < 0,001 0.4 Animal, drinking contaminated water, food other 121 18,4 86 13,1 0,010 Do not know 74 11,2 179 27,2 <0,001 Mosquito** Female 340 87,0 305 72,3 <0,001 0.37 Animal, drinking contaminated water, food other 77 19,7 49 11,6 0,002 Do not know 45 11,5 106 25,1 <0,001 Mosquito** Male 228 85,4 152 64,4 <0,001 0.5 Animal, drinking contaminated water, food other 44 16,5 37 15,7 0,903 Do not know 29 10,9 73 30,9 <0,001 *statistically significant difference P <5% **statistically significant difference P<1% 125 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 3 KNOWLEDGE ABOUT HOW MALARIA IS TRANSMITTED PER RELIGION Do you know how malaria is transmitted? PIRCOM Non PIRCOM P value Effect size n % n % Mosquito** All 568 86,3 457 69,5 < 0,001 Mosquito** Christia n 292 86.1 199 75.1 <0.001 Mosquito** Islamic 157 83.5 162 57.9 <0.001 Mosquito** Christia nFemale 166 86.5 138 78.4 0,053 Christia nMale 126 85.7 61 68.5 0,003 Mosquito** Islamic Female 99 84.6 100 59.9 <0,001 Islamic Male 58 81.7 62 54.9 <0,001 **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 126 TABLE 4 PERCEPTION ABOUT HOW DANGER IS MALARIA PIRCOM Non PIRCOM P value Effec t size n % n % Malaria is a danger disease All 642 97.6 640 97.3 0,862 Female 379 96,9 408 96,7 1,000 Male 263 98,5 232 98,3 1,000 Do you think that you are risk of malaria? All* 561 85,3 532 80,9 0,039 0,1 Female* 337 86,2 338 80,1 0,025 0.16 Male 224 83,9 194 82,2 0,635 *statistically significant difference P <5% **statistically significant difference P<1% 127 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 5 PERCEPTION ABOUT HOW DANGER IS MALARIA PER RELIGION PIRCOM Non PIRCOM P value Effec t size n % n % Do you think that you are risk of malaria? All* 561 85,3 532 80,9 0,039 0,1 Christian 290 85.5 255 84.9 0.90 Islamic 155 82.4 218 77.9 0.24 Yes Christia nFemale 168 87,5 154 87,5 1,000 Christia nMale 122 83,0 71 79,8 0,602 Yes Islamic Female 96 82,1 125 74,9 0,191 Islamic Male 59 83,1 93 82,3 1,000 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 128 TABLE 6 KNOWLEDGE ABOUT HIGH RISK OF PREGNANT WOMAN AND CHILDREN TO CONTRACT MALARIA Who in your household do you think is at highest risk of malaria at home PIRCOM Non PIRCOM P value Effect size N % N % Mother All 56 8,5 50 7,6 0,613 Pregnant mother** 146 22,2 105 16,0 0,005 0.15 Father 29 4,4 33 5,0 0,697 All children 322 48,9 327 49,7 0,825 Children under 5 years* 337 51,2 291 44,2 0,013 0.14 Eldery 30 4,6 31 4,7 1,000 Other 39 5,9 59 9,0 0,046 Mother Female 34 8,7 33 7,8 0,702 Pregnant mother 70 17,9 59 14,0 0,149 Father 16 4,1 15 3,6 0,717 All children 191 48,8 209 49,5 0,888 Children under 5 years 193 49,4 181 42,9 0,076 Eldery 15 3,8 19 4,5 0,727 Other* 21 5,4 41 9,7 0,024 Mother Male 22 8,2 17 7,2 0,739 Pregnant mother* 76 28,5 46 19,5 0,022 0.2 Father 13 4,9 18 7,6 0,265 All children 131 49,1 118 50,0 0,858 Children under 5 years 144 53,9 110 46,6 0,108 Eldery 15 5,6 12 5,1 0,845 Other 18 6,7 18 7,6 0,731 *statistically significant difference P <5% **statistically significant difference P<1% 129 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 7 KNOWLEDGE ABOUT HIGH RISK OF PREGNANT WOMAN AND CHILDREN TO CONTRACT MALARIA PER RELIGION Who in your household do you think is at highest risk of malaria at home PIRCOM Non PIRCOM P value Effect size N % N % Pregnant mother** all 146 22,2 105 16,0 0,005 0.15 Children under 5 years* 337 51,2 291 44,2 0,013 0.14 Pregnant mother Christian 76 22.4 48 18.1 0.22 Children under 5 years 175 51.6 112 42.3 0.02 Pregnant mother* Islamic 38 20.2 34 12.1 0.01 0.2 Children under 5 years 91 48.4 138 49.6 0.92 Pregnant mother Christia nFemale 32 16,7 29 16,5 1,000 Christia nMale 44 29,9 19 21,3 0,173 Children under 5 years Christia Female 98 51,0 72 40,9 0,060 Christia nMale 77 52,4 40 44,9 0,285 Pregnant mother Islamic Female 21 17,9 17 10,2 0,076 Islamic Male 17 23,9 17 15,0 0,172 Children under 5 years Islamic Female 52 44,4 81 48,5 0,546 Islamic Male 39 54,9 57 50,4 0,649 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 130 TABLE 8 SOURCE OF INFORMATION ABOUT MALARIA From where did you first hear of malaria? PIRCOM Non PIRCOM P value Effect size n % n % Community radio All 58 8,8 48 7,3 0,362 PIRCOM Religious leader 16 2,4 Door-to-Door visit by a PIRCOM community volunteer 87 13,2 Other ** 497 75,5 610 92,7 <0,001 Community radio Female 26 6,6 24 5,7 0,662 PIRCOM Religious leader 13 3,3 - - - Door-to-Door visit by a PIRCOM community volunteer 49 12,5 - - - Other ** 303 77,5 398 94,3 <0,001 Community radio Male 32 12,0 24 10,2 0,571 PIRCOM Religious leader 3 1,1 Door-to-Door visit by a PIRCOM community volunteer 38 14,2 Other ** 194 72,7 212 89,8 <0,001 *statistically significant difference P <5% **statistically significant difference P<1% 131 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 2-MALARIA PREVENTION TABLE 9 INFORMATION ABOUT MALARIA PREVENTION PIRCOM Non PIRCOM P value Effect size n % n % Malaria can be prevented All** 613 93,2 575 87,4 0,001 0.2 Female 356 91,0 374 88,6 0,297 Male ** 257 96,3 201 85,2 <0,001 0.4 *statistically significant difference P <5% **statistically significant difference P<1% TABLE 10 INFORMATION ABOUT MALARIA PREVENTION PER RELIGION PIRCOM Non PIRCOM P value Effect size n % n % Malaria can be prevented All** 613 93,2 575 87,4 0,001 0.2 Christian 317 93.5 235 88.7 0.04 Islamic 171 91.0 242 86.4 0.14 0.4 Yes Christian Female 174 90,6 160 90,9 1,000 Christian Male 143 97,3 75 84,3 0,001 Yes Islamic Female 106 90,6 146 87,4 0,451 Islamic Male 65 91,5 96 85,0 0,253 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 132 TABLE 11 KNOWLEDGE ABOUT WAYS OF PREVENTING MALARIA Which of the following do you think helps to prevent and control malaria PIRCOM Non PIRCOM P value Effect size n % n % Sleep under mosquito nets* All 587 95,8 532 92,5 0,018 0.14 Spray inside of house with insecticide** 62 10,1 25 4,3 <0,001 0.22 Other ** 255 41,6 187 32,5 0,001 No one of the above 5 0,8 12 2,1 0,086 Sleep under mosquito nets** Female 345 96,9 345 92,2 0006 0.20 Spray inside of house with insecticide** 33 9,3 12 3,2 0,001 0.25 Others 131 36,8 117 31,3 0,119 No one of the above 3 0,8 7 1,9 0,342 Sleep under mosquito nets Male 242 94,2 187 93,0 0,700 Spray inside of house with insecticide 29 11,3 13 6,5 0,102 Others* 124 48,2 70 34,8 0,004 No one of the above 2 0,8 5 2,5 0,249 *statistically significant difference P <5% **statistically significant difference P<1% 133 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 12 MEASURES TO BE TAKEN BY HH TO PREVENT MALARIA What measure do you take in your house to guard against malaria? PIRCOM Non PIRCOM P value Effect size n % n % Sleep under mosquito nets All 583 95,1 545 94,8 0,895 Spray inside of house with insecticide** 36 5,9 12 2,1 0,001 0.2 Others 199 30,2 128 19,5 <0,001 Do not know 0 0,0 0 0,0 - Sleep under mosquito nets Female 337 94,7 356 95,2 0,866 Spray inside of house with insecticide* 19 5,3 8 2,1 0,030 0.17 Others * 105 26,9 84 19,9 0,020 Do not know 0 0,0 0 0,0 - Sleep under mosquito nets Male 246 95,7 189 94,0 0,519 Spray inside of house with insecticide 17 6,6 4 2,0 0,023 Others 94 35,2 44 18,6 <0,001 Do not know 0 0,0 0 0,0 - *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 134 TABLE 13 MEASURES TO BE TAKEN BY HH TO PREVENT MALARIA PER RELIGION What measure do you take in your house to guard against malaria? PIRCOM Non PIRCOM P value Effect size n % n % Sleep under mosquito nets All 583 95,1 545 94,8 0,895 Spray inside of house with insecticide** 36 5,9 12 2,1 0,001 0.2 Sleep under mosquito nets Christian 300 94.6 223 94.9 0.99 Spray inside of house with insecticide* 23 7.3 7 3 0,036 Sleep under mosquito nets Islamic 162 94.7 224 92.6 0.42 Spray inside of house with insecticide 3 1.8 4 1.7 1,000 Sleep under mosquito nets Christian Female 164 94,3 153 95,6 0,625 Christian Male 136 95,1 70 93,3 0,756 Spray inside of house with insecticide Christia Female 12 6,9 5 3,1 0,139 Christian Male 11 7,7 2 2,7 0,227 Sleep under mosquito nets Islamic Female 101 95,3 135 92,5 0,440 Islamic Male 61 93,8 89 92,7 1,000 Spray inside of house with insecticide Islamic Female 2 1,9 2 1,4 1,000 Islamic Male 1 1,5 2 2,1 1,000 *statistically significant difference P <5% **statistically significant difference P<1% 135 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 14 SOURCE OF INFORMATION ABOUT SPRAYING INSECTICIDE Where or from whom did you hear information about spraying insecticide? PIRCOM Non PIRCOM P value Effect size n % n % Community radio All 54 14,6 32 14,0 0,905 PIRCOM Religious leader 44 11,9 Door-to-Door visit by a PIRCOM community volunteer 122 32,9 Other ** 260 70,1 208 91,2 <0,001 Community radio Female 24 11,9 17 12,2 1,000 PIRCOM Religious leader 26 12,9 Door-to-Door visit by a PIRCOM community volunteer 64 31,8 Other * 141 70,1 129 92,8 <0,001 Community radio Male 30 17,6 15 16,9 1,000 PIRCOM Religious leader 18 10,6 Door-to-Door visit by a PIRCOM community volunteer 58 34,1 Other * 119 70,0 79 88,8 <0,001 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 136 TABLE 15 SOURCE OF INFORMATION ABOUT MOSQUITO NET Where or from whom did you hear information about mosquito net? PIRCOM Non PIRCOM P value Effe ct n % n % size Community radio All 12 2,9 18 5,5 0,090 PIRCOM Religious leader 21 5,0 Door-to-Door visit by a PIRCOM community volunteer 147 34,9 Other ** 241 57,2 312 94,5 <0,001 Community radio Female 6 2,4 6 2,8 0,779 PIRCOM Religious leader 11 4,4 Door-to-Door visit by a PIRCOM community volunteer 74 29,6 Other** 159 63,6 207 97,2 <0,001 Community radio* Male 6 3,5 12 10,3 0,026 PIRCOM Religious leader 10 5,8 Door-to-Door visit by a PIRCOM community volunteer 73 42,7 Other** 82 48,0 105 89,7 <0,001 *statistically significant difference P <5% **statistically significant difference P<1% 137 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 3-MOSQUITO NETS TABLE 36 KNOWLEDGE OF MOSQUITO NET AS ONE OF THE PREVENTION METHOD PIRCOM Non PIRCOM P value Effec t size n % n % Mosquito net can reduce malaria All** 626 95,3 585 88,9 <0,001 0.23 Female * 370 94,9 376 89,1 0,003 0.21 Male * 256 95,9 209 88,6 0,002 0.27 This house has mosquito net All 630 95,7 634 96,4 0,672 Female 374 95,7 407 96,4 0,592 Male 256 95,9 227 96,2 1,000 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 138 TABLE 47 HOUSEHOLD USE OF MOSQUITO NET Number of household members who have slept under mosquito net last night PIRCOM Non PIRCOM P value Effec t size n % n % No one All 6 1,0 6 1,0 1,000 1-2 30 4,8 26 4,1 0,587 3-4 70 11,2 54 8,6 0,073 5 or more 43 6,9 36 5,7 0,418 Entire family 477 76,2 508 80,6 0,064 No one Female 4 1,1 5 1,2 1,000 1-2 16 4,3 21 5,2 0,615 3-4 42 11,3 35 8,6 0,231 5 or more 26 7,0 21 5,2 0,297 Entire family 284 76,3 323 79,8 0,260 No one Male 2 0,8 1 0,4 1,000 1-2 14 5,5 5 2,2 0,099 3-4 28 11,0 19 8,4 0,360 5 or more 17 6,7 15 6,7 1,000 Entire family 193 76,0 185 82,2 0,116 139 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 18 THE PRACTICE IN HH OF PREGNANT WOMAN AND CHILDREN TO USE MOSQUITO NET PIRCOM Non PIRCOM P value Effec t size n % n % Members household who are pregnant slipping under the net last night All 11 11,1 18 19,4 0,157 Female Male Children under five slipping under the net last night All 164 16,8 141 14,2 0,119 Female Male Other members* All 375 10,1 253 7,0 <0,001 0.11 Female Male USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 140 4-SPRAYING INSECTICIDE TABLE 19 MEASURES IN HOUSE TO GUARD AGAINST MALARIA What measure do you take in your house to guard against malaria? PIRCOM Non PIRCOM P value Effect size n % n % Sleep under mosquito nets All 583 95,1 545 94,8 0,895 Spray inside of house with insecticide** 36 5,9 12 2,1 0,001 0.2 Others 199 30,2 128 19,5 <0,001 Do not know 0 0,0 0 0,0 - TABLE 20 SPRAYING WITH INSECTICIDE INSIDE HOUSE IN IRS DISTRICTS District Type Prevention n % Maxixe PIRCOM 3 12.5% 19 Mocuba PIRCOM 16 15% 20 19 From the total of HH surveyed in Maxixe 20 From the total of HH surveyed in Mocuba 141 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 21 KNOWLEDGE AND PRACTISE OF SPRAYING INSECTICIDE METHOD TO REDUCE THE RISK OF GETTING MALARIA PIRCOM Non PIRCOM P value Effect size n % n % Spraying insecticide inside the house reduces the risk of malaria All** 406 61,7 287 43,6 <0,001 0.36 Female 230 58,8 177 41,9 <0,001 0.34 Male 176 65,9 110 46,6 <0,001 0.4 Have received message that spraying insecticide reduces the risk of malaria All** 371 56,4 228 34,7 <0,001 0.44 Female 201 51,4 139 32,9 <0,001 0.38 Male 170 63,7 89 37,7 <0,001 0.53 In last year was this house was sprayed All** 123 18,7 27 4,1 <0,001 0.47 Female 60 15,3 16 3,8 <0,001 0.39 Male 63 23,6 11 4,7 <0,001 0.56 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 142 TABLE 22 REGULARITY OF USING SPRAY INSECTICIDE TO PREVENT MALARIA How often do you think your house should be sprayed? PIRCOM Non PIRCOM P value Effect size n % n % Every 6 mouths All 84 62,7 20 48,8 0,146 Every 9- to 12 months 31 23,1 6 14,6 0,282 Every two years 2 1,5 0 0,0 1,000 Do not know ** 17 12,7 15 36,6 0,001 -0.57 Every 6 mouths Female 42 63,6 11 40,7 0,064 Every 9- to 12 months 14 21,2 4 14,8 0,573 Every two years 1 1,5 0 0,0 1,000 Do not know ** 9 13,6 12 44,4 0,002 -0.72 Every 6 mouths Male 42 61,8 9 64,3 1,000 Every 9- to 12 months 17 25,0 2 14,3 0,503 Every two years 1 1,5 0 0,0 1,000 Do not know 8 11,8 3 21,4 0,389 *statistically significant difference P <5% **statistically significant difference P<1% 143 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 5-PARENTAL CARE TABLE 23 KNOWLEDGE AND PRACTICES OF ANTENATAL CARE TO PREVENT AND TREAT MALARIA PIRCOM Non PIRCOM P value Effec t size n % n % Antenatal care provides helpful information to prevent or treat malaria in pregnant woman All* 600 91,3 566 86,0 0,003 0.16 Female* 357 91,5 367 87,0 0,042 0.14 Male* 243 91,0 199 84,3 0,028 0.20 You have heard about the message or information about the importance of taking at least 3 doses of preventive malaria treatment during pregnancy All 451 100,0 370 100,0 - Female 262 100,0 239 100,0 - Male 189 100,0 131 100,0 - During the recent pregnancy did the woman received prenatal care All 571 86,8 554 84,2 0,210 Female 339 86,7 357 84,6 0,424 Male 232 86,9 197 83,5 0,314 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 144 TABLE 24 ATTITUDES TOWARD ANTENATAL CARE FOR PREVENTING MALARIA How likely are you or pregnant woman in your HH to get regular antenatal care during the pregnancy PIRCOM Non PIRCOM P value Effect size n % n % Extremely likely* 411 62.5 372 56.5 0.033 Likely 209 31.8 225 34.2 0.37 Neutral 35 5.3 53 8.1 0.06 Unlikely 3 0.5 5 0.8 0.72 Not at all important *statistically significant difference P <5% **statistically significant difference P<1% TABLE 25 ANTENATAL CARE FOR PREVENTING MALARIA PIRCOM Non PIRCOM P value Effec t size n % n % During the recent pregnancy did the woman received preventive malaria treatment All 480 72.9 469 71.3 0.53 Female 285 72.9 301 71.3 0.63 Male 195 73.0 168 71.2 0.69 *statistically significant difference P <5% **statistically significant difference P<1% 145 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV 6-MALARIA TREATMENT TABLE 26 KNOWLEDGE OF SYMPTOMS ASSOCIATED WITH MALARIA What symptoms do you associated with malaria PIRCOM Non PIRCOM P valu e Effect size n % n % High temperature/fever* All 588 89,4 559 85,0 0,021 Loss of appetite* 140 21,3 179 27,2 0,014 Headache 432 65,7 399 60,6 0,067 Body aches 360 54,7 408 62,0 0,009 Chills 55 8,4 43 6,5 0,248 Fatigue 70 10,6 61 9,3 0,461 Nausea or vomiting 148 22,5 160 24,3 0,474 Diarrhea 59 9,0 76 11,6 0,146 Dizziness 58 8,8 31 4,7 0,004 Other (specify) 66 10,0 49 7,4 0,118 High temperature/fever* Female 351 89,8 347 82,2 0,002 Loss of appetite 81 20,7 106 25,1 0,156 Headache 252 64,5 261 61,8 0,467 Body aches 218 55,8 267 63,3 0,032 Chills 37 9,5 31 7,3 0,311 Fatigue 34 8,7 37 8,8 1,000 Nausea or vomiting 93 23,8 112 26,5 0,375 Diarrhea 36 9,2 59 14,0 0,038 Dizziness 36 9,2 18 4,3 0,005 Other (specify) 45 11,5 31 7,3 0,053 USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 146 High temperature/fever* Male 237 88,8 212 89,8 0,773 Loss of appetite 59 22,1 73 30,9 0,026 Headache 180 67,4 138 58,5 0,042 Body aches 142 53,2 141 59,7 0,150 Chills 18 6,7 12 5,1 0,457 Fatigue 36 13,5 24 10,2 0,272 Nausea or vomiting 55 20,6 48 20,3 1,000 Diarrhea 23 8,6 17 7,2 0,622 Dizziness 22 8,2 13 5,5 0,292 Other (specify) 21 7,9 18 7,6 1,000 *statistically significant difference P <5% **statistically significant difference P<1% TABLE 27 ATTITUDES TOWARD MEDICAL TREATMENT OF MALARIA PIRCOM Non PIRCOM P value Effec t size n % n % HH that have heard message about the importance of seeking immediate medical treatment All** 565 86,0 468 71,1 <0,001 0.36 Female ** 343 87,9 312 73,9 <0,001 0.36 Male** 222 83,1 156 66,1 <0,001 0.39 The need to continue the malaria treatment after the malaria symptoms are gone All 523 92,6 429 91,7 0,642 Female 322 93,9 285 91,3 0,232 Male 201 90,5 144 92,3 0,584 HH that heard message about the importance of completing the malaria treatments All** 617 93,8 569 86,5 <0,001 0.24 Female** 366 93,6 371 87,9 0,006 0.19 Male ** 251 94,0 198 83,9 <0,001 0.32 *statistically significant difference P <5% **statistically significant difference P<1% 147 | PMI MOZAMBIQUE SBCC FINAL EVALUATION USAID.GOV TABLE 28 PRACTICES OF HH TOWARD SEEKING MEDICAL TREATMENT FOR HH MEMBER WITH SYMPTOMS OF MALARIA PIRCOM Non PIRCOM P value Effec t size n % n % HH who have had members suffering from malaria in last 6 months All 483 73,4 492 74,8 0,615 Female 290 74,2 313 74,2 1,000 Male 193 72,3 179 75,8 0,416 HH who have had pregnant woman with malaria in last 6 months All 42 8,7 45 9,1 0,823 Female 25 8,6 31 9,9 0,674 Male 17 8,8 14 7,8 0,852 HH who have had children under 5 with malaria in last 6 months All** 304 62,9 364 74,0 <0,001 Female** 183 63,1 233 74,4 0,003 Male* 121 62,7 131 73,2 0,035 HH who have sought medical treatment for malaria in last 6 months (within 24 hours) All 458 94,8 455 92,5 0,149 Female 272 93,8 293 93,6 1,000 Male* 186 96,4 162 90,5 0,033 HH who have completed the medical treatment is last 6 months. All 438 95,6 445 97,8 0,093 Female** 259 95,2 289 98,6 0,024 Male 179 96,2 156 96,3 1,000 *statistically significant difference P <5% **statistically significant difference P<1% USAID.GOV USAID/MOZAMBIQUE SBCC FINAL EVALUATION | 148 TABLE 29 PRACTICES OF HH TOWARD SEEKING MEDICAL TREATMENT FOR HH MEMBER WITH SYMPTOMS OF MALARIA PER RELIGION PIRCOM Non PIRCOM P value Effect size n % n % HH who have sought medical treatment for malaria in last 6 months (within 24 hours) All 339 71,2 348 71,3 1,000 Christian 170 70,2 142 74,0 0,452 Islamic 103 71,5 154 67,8 0,490 Yes Christian Female 97 71,9 94 74,0 0,781 Christian Male 73 68,2 48 73,8 0,493 Yes Islamic Female 62 68,1 91 66,4 0,886 Islamic Male 41 77,4 63 70,0 0,437 HH who have completed the medical treatment is last 6 months. All 438 95,6 445 97,8 0,093 Christian 220 94 177 97.8 0.087 Islamic 134 96.4 203 97.6 0.52 Yes Christian Female 122 93,1 118 98,3 0,062 Christian Male 98 95,1 59 96,7 1,000 Yes Islamic Female 83 96,5 123 98,4 0,400 Islamic Male 51 96,2 80 96,4 1,000 *statistically significant difference P <5% **statistically