EVALUATION Mid-Term Evaluation of the USAID/Zambia Communications Support for Health Program January, 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). It was authored by Joseph Limange, Maurice Ocquaye, Moses Simuyemba, and Saviour Chishimba. ii COVER PHOTO Credit: Moses Simuyemba Caption: Women from Kapata Ward in the Samfya District of the Luapula Province returning from the clinic. MID-TERM EVALUATION OF THE ZAMBIA COMMUNICATIONS SUPPORT FOR HEALTH PROGRAM January 25, 2013 Order No.: AID-611-TO-13-00001 Requisition/Reference No.: REQ-611-12-000034 By International Business & Technical Consultants Inc.: Joseph Limange, Maurice Ocquaye, Saviour Chishimba, and Moses Simuyemba (Consultants) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iv ACKNOWLEDGEMENTS The International Business & Technical Consultants, Inc. (IBTCI) evaluation team would like to thank all those from the USAID Mission who facilitated this evaluation. We especially appreciate the support and guidance of the evaluation Contracting Officer’s Representative Christopher Foley, Project Development Officer. The IBTCI evaluation team appreciates the generosity and cooperation of the helpful staff of the Communications Support for Health (CSH) project including: Anne Fiedler, Chief of Party; Linda Nonde, Deputy Chief of Party; Christina Wakefield, Technical Director; and Kelvin Chilemu, Research, Monitoring and Evaluation (M&E) Director. The IBTCI evaluation team would also like to thank the informative staff members of the Zambian Ministry of Health (MOH), National HIV/AIDS/TB Council (NAC) and National Malaria Control Centre (NMCC). In particular, the evaluation team is grateful for the insights and support of George Sikazwe, Chief Health Promotion Officer (MOH), Douglas Hampande, Civil Society Specialist (NAC), and Mulakwa Kamuliwo, Deputy Director, Public Health and Research – Malaria (NMCC).    ii CONTENTS ACRONYMS.......................................................................................................................................... iv EXECUTIVE SUMMARY..................................................................................................................... vi EVALUATION PURPOSE AND RATIONALE.............................................................................. vi PROJECT BACKGROUND ............................................................................................................. vi EVALUATION METHODS, AND LIMITATIONS........................................................................ vi EVALUATION QUESTIONS AND RESULTS .............................................................................. vii CONCLUSIONS................................................................................................................................. x RECOMMENDATIONS................................................................................................................... xi 1.0 INTRODUCTION............................................................................................................................ 1 1.1 EVALUATION PURPOSE AND RATIONALE......................................................................... 1 1.2 EVALUATION QUESTIONS...................................................................................................... 1 1.3 PROJECT BACKGROUND......................................................................................................... 2 2.0 EVALUATION METHODS ............................................................................................................ 4 2.1 EVALUATION TEAM STRUCTURE ......................................................................................... 4 2.2 EVALUATION DESIGN ............................................................................................................ 4 2.3 DATA COLLECTION METHODS ............................................................................................. 4 2.3.1 Document Review................................................................................................................... 4 2.3.2 Key Informant Interviews........................................................................................................ 5 2.3.3 Online Survey .......................................................................................................................... 5 2.3.4 Focus Group Discussions........................................................................................................ 5 2.3.5 Systems Assessment and Data Quality Assessment.................................................................. 5 2.3.6 Mini Survey.............................................................................................................................. 6 2.4 ANALYSIS PLAN......................................................................................................................... 7 2.5 LIMITATIONS ............................................................................................................................. 8 3.0 FINDINGS........................................................................................................................................9 3.1 QUESTION CATEGORY 1: MESSAGE EXPOSURE & EFFECTIVENESS............................9 3.2 QUESTION CATEGORY II: CAPACITY BUILDING ...........................................................13 3.3 QUESTION CATEGORY III: EVIDENCE BASED PLANNING .........................................18 3.4 QUESTION CATEGORY IV: SUSTAINABILITY ...................................................................21 3.5 QUESTION CATEGORY V: RESULTS TO DATE .................................................................22 4.0 CONCLUSIONS ............................................................................................................................. 33 4.1 MESSAGE EXPOSURE AND EFFECTIVENESS ...................................................................33 iii 4.2 CAPACITY BUILDING ............................................................................................................. 33 4.3 EVIDENCE-BASED PLANNING ............................................................................................ 34 4.4 SUSTAINABILITY ..................................................................................................................... 34 4.5 RESULTS TO DATE.................................................................................................................. 35 5.0 RECOMMENDATIONS ................................................................................................................ 37 5.1 RECOMMENDATIONS TO CSH............................................................................................. 37 5.1.1 Results to Date ...................................................................................................................... 37 5.1.2 Message Exposure and Effectiveness..................................................................................... 37 5.1.3 Capacity Building................................................................................................................... 37 5.1.4 Evidence–Based Planning & Implementation ........................................................................38 5.1.5 Sustainability.......................................................................................................................... 38 5.2 RECOMMENDATIONS TO USAID......................................................................................... 38 5.3 RECOMMENDATION TO GOVERNMENT OF ZAMBIA ...................................................39 ANNEXES............................................................................................................................................40 ANNEX B: MAP OF ZAMBIA – PROVINCES VISITED FOR DATA COLLECTION ..............52 ANNEX C: EVALUATION TEAM BIOGRAPHIES .....................................................................53 ANNEX D: METHODOLOGY NOTES ........................................................................................ 54 ANNEX E: LIST OF PERSONS CONSULTED .............................................................................56 ANNEX F: EVALUATION MATRIX............................................................................................. 63 ANNEX G: ORAL CONSENT SCRIPT for MINI SURVEY..........................................................70 ANNEX H: KEY INFORMANT INTERVIEW GUIDE................................................................72 ANNEX I: SITE VISIT PROTOCOL .............................................................................................. 75 ANNEX J: FOCUS GROUP DISCUSSION GUIDE.......................................................................76 ANNEX K: DATA QUALITY ASSESSMENT QUESTION GUIDE ............................................80 ANNEX P: CONFLICT OF INTEREST STATEMENTS.............................................................102    iv ACRONYMS ANC Antenatal Care BCC Behavior change communication BCP Behavior Centered Programming CHAMP Comprehensive HIV AIDS Management Program CSH Communications Support for Health CSO Civil society organization DACA District AIDS Coordinating Advisor DQA Data Quality Assessment FGD Focus Group Discussion FP Family Planning GRZ Government of the Republic of Zambia HCRC Health Communication Resource Center HIV/AIDS Human Immunodeficiency Virus / Acquired Immunodeficiency Syndrome IBTCI International Business & Technical Consultants, Inc. ICFI ICF International IEC Information, education, and communication IPTp Intermittent preventive treatment of malaria for pregnant women IR Intermediate result IRB Institutional Review Board IT Information technology ITN Insecticide-treated net JICA Japan International Cooperation Agency KII Key Informant Interviews M&E Monitoring and Evaluation MCH Maternal and Child Health MCP Multiple and concurrent (sexual) partners MDG Millennium Development Goals MNCH Maternal, newborn, and child health MOH Ministry of Health NAC National HIV/AIDS/STI/TB Council NGO Nongovernmental organization NMCC National Malaria Control Center NZP+ Network of Zambian People living with HIV/AIDS PMI President’s Malaria Initiative PMTCT Prevention of mother-to-child transmission PVO Private Volunteer Organization PRISM Private Sector Mobilization Project for Social Marketing Project PSE Private sector engagement RH Reproductive Health SAF Strategic Activities Fund SAfAIDS Southern Africa HIV and AIDS Information Dissemination Service SLC Safe Love Clubs SMAG Safe motherhood action group SMGL Saving Mothers Giving Life STI Sexually transmitted infection TB Tuberculosis v UNFPA United Nations Population Fund UNZA University of Zambia UTH University Teaching Hospital VCT Voluntary counseling and testing YHM Your Health Matters ZISSP Zambia Integrated Systems Strengthening Program    vi EXECUTIVE SUMMARY International Business & Technical Consultants, Inc. (IBTCI) is pleased to present the Mid-term Evaluation Report for the United States Agency for International Development (USAID) funded Zambia Communications Support for Health (CSH) project. CSH is funded by USAID and is a four and a half year, $43 million contract implemented by Chemonics International under contract number GHS-I-007- 00004-00, Task Order number GHS-I-05-07-00004. CSH began on July 8, 2010 and is scheduled to be completed by December 13, 2014. CSH sub-contractors include the Manoff Group and ICF International (ICFI). CSH sub-grantees include Afya Mzuri, and the Comprehensive Human Immunodeficiency Virus/ Acquired Immunodeficiency Syndrome (HIV/AIDS) Management Program (CHAMP). This report includes the findings, conclusions, and recommendations that the evaluation team collected and developed during November – December 2012. EVALUATION PURPOSE AND RATIONALE The USAID/Zambia Health Office contracted IBTCI as an independent entity to perform a mid-term performance evaluation of CSH. The objectives of this mid-term evaluation were three-fold: Part A (Retrospective): The evaluation sought to help determine what progress CSH has made in achieving its life-of-project targets and whether or not they are likely to achieve them by the end of the program. Additionally the evaluation looked at what components of the CSH program were working well and to explain why this was so. If there were parts that were not working well the evaluation went further to find out why. Part B (Prospective): Based on the findings and conclusions of the retrospective study, the evaluation made recommendations for CSH project implementation through December 2014, including the optimal mix of activities and funding for achieving project objectives and for aiding sustainability. Part C: Using the findings and conclusions of the prospective and retrospective studies, the evaluation framed issues to discuss or resolve at a level higher than the project, specifically at the level of the Government of the Republic of Zambia (GRZ) and/or other donor organizations, where applicable. PROJECT BACKGROUND CSH is tasked with supporting the GRZ’s vision of “equity of access to assured quality, cost-effective, and affordable health services as close to the family as possible.” The GRZ through its Ministry of Health (MOH) is committed to achieving Millennium Development Goal (MDG) targets by improving the quality of health care services and providing greater and equitable health care access for its people. To support these objectives, USAID is providing technical assistance to the GRZ in strengthening national health communications activities. The aim is for GRZ health communications activities supported by CSH to translate into increased sustainable local capacity and positive behavior change that contribute to GRZ efforts in five focal areas: 1) HIV/AIDS, 2) malaria, 3) family planning/ reproductive health (FP/RH), 4) maternal, newborn, and child health (MNCH) and 5) nutrition. EVALUATION METHODS, AND LIMITATIONS Multiple methods were used to answer each evaluation question. The evaluation team used a six-phase approach, which included a document review, key informant interviews (KIIs), focus group discussions (FGDs), a mini survey, an online survey, and a Systems/Data Quality Assessment (DQA). Annex F provides an evaluation design matrix that demonstrates the types of data collection methods and sources that were used to answer each evaluation question. The team conducted 50 KIIs with respondents drawn from the CSH project, ZISSP, PRISM, NAC, the Ministry of Community Development Mother and Child Health (MCDMCH), MOH, NMCC, CHAMP, Afya Mzuri, Southern Africa HIV and AIDS Information Dissemination Service (SAfAIDS), the Christian Health Association of Zambia (CHAZ), the Planned Parenthood Association of Zambia (PPAZ), the United Nations Population Fund (UNFPA), the Japan International Cooperation Agency (JICA), the European Union (EU), and the Network of Zambian People living with HIV/AIDS (NZP+). vii The evaluation team also conducted an online survey that was aimed at key informants and was successful in obtaining 27 responses out of a total 60 requests that were sent, from CSH staff, sub grantees, GRZ representatives, personnel from other USG funded projects, and GRZ health partners. The team conducted 25 FGDs in 13 Districts across six provinces with 325 participants, and conducted a mini survey in 12 districts across six provinces, reaching 242 respondents. The evaluation faced some challenges and there were some limitations to the evaluation design and methodology related to the time frame for the evaluation, a small sample size for the mini survey, and a disproportionately larger number of women interviewed. Also, without the approval of the Institutional Review Board for the mini survey, the evaluation team did not ask questions that were sensitive and could potentially infringe on the rights of respondents. EVALUATION QUESTIONS AND RESULTS The evaluation team was required to answer questions categorized into five aspects of performance. The results follow, with the first question of the SOW presented last, because it derives from the other four evaluation questions. Question Category I: Message Exposure and Effectiveness To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with Information, Education, and Communication/Behavior Change Communication (IEC/BCC) messages in each of the five health intervention areas (HIV/AIDS, malaria, family planning/reproductive health (FP/RH), maternal child health (MCH) and nutrition)? Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages? The evaluation established that CSH has launched three campaigns namely the Safe Love Campaign for HIV/AIDS, the Mothers Alive Campaign for MNCH, and the STOP Malaria Campaign. The campaigns were much more effective reaching urban audiences (including the use of the Helpline Counselors) than rural ones. The messages were effectively perceived for some adverts, such as the importance of consistent condom use, but did not get across in others, such as one on the implications of multiple concurrent partners, and antenatal care (ANC) services. The STOP Malaria campaign was not remembered by respondents in relation to other campaigns running at the same time. It is pertinent that stakeholders perceived the Safe Love Campaign – the most successful campaign – as owned by CSH, not the GRZ. Safe Love Campaign: The Safe Love Campaign was the most commonly known of all the campaigns as evidenced by the mini survey results, which established that 67.3%1 of the general population, rural and urban clusters combined, had knowledge about the campaign. In terms of channels through which the messages reached the population by cluster, 63.5% of the urban population heard about the campaign through television while 50.9% heard through the radio. There is a marked difference in the rural cluster where 56.0% of the population heard of the campaign through radio compared to 9.3% who watched it on television. The inability of the campaign to effectively reach a wider population was largely attributed to the inadequate supply of IEC materials. Further, during FGDs in rural communities, respondents revealed that they would prefer these materials to be produced and distributed in the seven major local languages, namely Tonga, Lozi, Bemba, Kaonde, Luvale, Chewa, and Lunda. Mothers Alive Campaign: The Mothers Alive Campaign, which also covers FP and nutrition, has not effectively reached both urban and rural communities. The mini survey results showed that only 21.6% of the urban population heard about this campaign. This picture is not too different from results in rural areas where 26.6% of the population were aware of the campaign. Most of the respondents from organized groups working with CSH had not heard of this campaign.                                                              1 The figure is from a bi-variate analysis of people who heard Safe Love on radio and those who heard it on TV. It therefore presents the universal set of those who have heard of the Safe Love campaign.    viii STOP Malaria Campaign: Like Mothers Alive, the STOP Malaria Campaign was also not effectively disseminated to the people. It was also apparent that most of the people heard about other malaria campaigns, and not the CSH campaign, which has been launched neither on TV nor radio. FP/RH and Nutrition Campaigns: While CSH is covering aspects of FP/RH and nutrition in the Mothers Alive Campaign; it has not launched a specific campaign on these thematic areas. Very little is therefore being done on these two thematic areas. Question Category II: Capacity Building To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own? What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own? What else does GRZ need in order to independently plan, implement, manage, and evaluate national health campaigns? To what extent is the project adapting to the current changes in the GRZ structures particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re-organization of the National HIV/AIDS/ sexually transmitted infection (STI)/ tuberculosis (TB) Council (NAC)? CSH is mandated to build capacity of GRZ, especially at that national level through the provision of technical and financial assistance in development, implementation, and evaluation of health communications activities. Specifically, CSH is tasked with working with the three primary agencies — the MOH, the NMCC, and NAC. The CSH mandate to GRZ includes the provision of direct support in the planning, design, and implementation of communications campaigns and activities. The project used six main approaches to build the capacity of GRZ to implement IEC/BCC activities on its own: training, mentorship, monitoring and evaluation support, support to Technical Working Groups, coaching, and logistics support to local GRZ partners working in IEC/BCC. In spite of many laudable interventions, gaps exist in the CSH strategy for building capacity. For instance, two out of three institutions confirmed they lack understanding of the roles and responsibilities of the embedded staff. Competing time demands on government counterparts cause CSH to sometimes compromise on its strategy of working closely with GRZ, and roll out activities without active GRZ involvement. The policy of USAID not to pay any allowance to GRZ staff for participating in USAID funded activities apart from normal per diem appears to account partly for the low priority given to CSH’s planned activities. Given the currently known responsibility of the MCDMCH - to take over the responsibilities of MOH at the district and community levels - the Ministry’s geographical area of operation falls outside CSH’s mandate. The project has also not been given any document detailing the structure and mandate of the new ministry and USAID has not formally communicated to CSH that it is assigned responsibilities with the new Ministry. The project is, therefore, unsure of its role in relation to MCDMCH. Question Category III: Evidence Based Planning In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? In terms of evidence based planning and implementation, CSH has had noteworthy achievements. Among the noteworthy achievements: CSH developed a trainer’s and participant’s manual for BCP that established strong mechanisms to assist in gathering and using evidence in the development of products and tools relating to IEC/BCC activities; training GRZ partners to conduct formative research and pre￾testing during the BCP process to inform the development of communications products; assisting MOH, NAC and NMCC to prepare a set of guidelines to guide the process of pretesting and evaluating communication materials, and; CSH conducted M&E training, which included evaluation design, to build capacity of 20 GRZ staff at central, provincial and district levels. Other than for initial planning there is a lack of evidence collection through well-established feedback mechanisms to evaluate the messages after their initial design. M&E plans are in place for all the campaigns. However, KIIs and DQA show that although evaluation of exposure to and outcomes of IEC/BCC activities is planned for, it is yet to be implemented. Only the Safe Love Campaign has had a rapid survey so far. Further, there are no plans at CSH for periodically revising the materials produced based on evidence gathered. Additionally, a lack of M&E Systems for BCC programs at MOH, NMCC and NAC means that there are no plans for evaluating IEC/BCC activities there. ix Question Category IV: Sustainability Are IEC/BCC activities likely to continue without further United States Government (USG) investments and if not, what investments or approaches would better promote sustainability? KIIs revealed that the approach of building the capacity of GRZ institutions namely, MOH, NMCC and NAC, by way of developing communications strategies and other toolkits was a very effective approach and sustainable, but the embedding of staff was unsustainable and ineffective. Systems strengthening, as opposed to placement of CSH paid personnel, was viewed to be more sustainable considering the high GRZ staff attrition rate. Further, the evaluation established that GRZ is not yet fully on course in terms of reaching a level where IEC/BCC activities can continue without USG investments. Evidently, the online survey showed that 35% of respondents rated their confidence levels in GRZ’s ability to continue as medium, while only 26% said their confidence levels were high. Question Category V: Results to Date To what extent is the activity on track to achieve its intermediate results and meet its life of activity targets? To what extent are the indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)? What improvements can be made to better capture progress? What are the challenges to implementation and what can be done to improve the chances of the activity achieving its intended results and meeting its life of activity targets? The STOP Malaria campaign is aimed at stopping malaria through promoting testing for malaria before treatment, insecticide treated net (ITN) usage, and intermittent preventive treatment of malaria for pregnant women (IPTp). FGDs indicated that all women who attend ANC receive IPTp. RDT is also widely used in health facilities to test for malaria before treatment is given, except in instances when test kits are out of stock. The mini survey found that only 27.7% of children under five years of age currently sleep under ITN, a decrease from when the project began; this can be attributed to the project not conducting any media activities on malaria. The malaria component of the project is therefore not on track to achieving its objectives. The Mothers Alive campaign aims to reduce maternal mortality through a number of goals. Key among them is the promotion of ANC services. ANC attendance averaged 87% of pregnant women, with some 13% failing to attend clinic for antenatal services; most pregnant women went for ANC services when their pregnancy was advanced. A significant proportion of the pregnant women went for ANC services up to the recommended frequency. FGDs revealed that the Birth Plan was not widely known nor utilized by women. The Mothers Alive TV campaign has a target of reaching 15% of the population by the end of 2012 and is currently reaching 14% according to the mini survey. The TV campaign is therefore on track even though it needs more efforts to achieve its intermediate and long term results. The radio campaign, however, is not on track. The project PMP indicates a target of 30% reach by end of 2012 but it is only able to reach 16.7%. The Safe Love campaign aims at targeting three main drivers of the HIV pandemic in Zambia. The survey results indicate that the Safe Love TV advert is not having any effect on men’s practice of multiple and concurrent partners (MCP) whereas the radio advert is having a very strong effect on men in having less MCPs. Conversely, the TV adverts are having a stronger effect on women having less MCPs than the radio adverts. The Safe Love campaign is contributing significantly to people knowing the HIV status of their sexual partners both in urban and rural areas. The Safe Love TV adverts are having a marginal effect on condom use in urban areas but absolutely no effect in rural areas. The radio adverts, however, are influencing condom use both in rural and urban areas. The project targeted reaching 50% of the targeted audience with radio by the end of 2012. It has successfully exceeded this target by 2 percentage points. However, the TV campaign is far from on track. It is currently reaching 46.7% of the targeted audience, significantly below its target of 90% by the end of 2012. Safe Love adverts on TV are making significant achievement in both rural and urban areas with beliefs on circumcision. Higher proportions of those exposed to the adverts on TV, both in urban and rural areas, believe that circumcision reduces the chances of contracting HIV compared to those not exposed to the campaign.    x CONCLUSIONS Message Exposure and Effectiveness Activities on HIV/AIDS have reached the most people (63.5% urban and 56.0% rural) with intended audiences able to recall most of the messages which were appropriate and culturally acceptable. However, the audience did not really understand the intent of most of the Safe Love adverts as the adverts had no clear call to action. The other campaigns have reached fewer people, with MCH having the next best reach (18.6% urban 25.3% rural). The MNC campaign name is not known. The Change Champion advert was particularly misleading and was seen to be directed at the government to build more health facilities. The Safe Motherhood Testimonials advert was appropriate and well understood but the characters selected all had Eastern Province names, which did not sit well with some audiences. Capacity Building CSH’s capacity building activities, especially at the provincial and district levels, have improved skills and equipped GRZ at all levels with strategies in conducting research and planning and implementation of BCC activities through such things as BCP training. There has been very little mentoring and coaching to systematically transfer skills on the job for GRZ staff to be able to implement IEC/BCC activities on their own with embedded staff not playing their role of mentoring and coaching their GRZ counterparts adequately. CSH’s approaches and strategies have invested little in IEC/BCC systems strengthening within the framework of health service delivery, with CSH leading in all activities that are being implemented rather than GRZ. CSH is not clear with their mandate in relation to capacity building for the new Ministry of Community Development Mother and Child Health (MCDMCH). Evidence-Based Planning CSH has greatly assisted MOH, NAC and NMCC to use evidence based approaches to plan their IEC/BCC products and tools through the trainings that they have been given in M&E and BCP, the BCC guidelines that were developed, and TWG terms of reference. However, gaps still exist which include a lack of follow through in implementing M&E plans for the various campaigns beyond the launch phase at CSH; lack of plans for collecting feedback systematically and for utilizing such feedback to improve on products and tools at GRZ level; and lack of M&E systems for IEC/BCC activities that would enable effective and efficient collection of evidence at GRZ level. Sustainability There is a lack of investment in IEC/BCC systems strengthening within the framework of health service delivery and, contrary to the project design, current project implementation and management arrangements reveal that CSH has taken a lead in carrying out IEC/BCC activities rather than assisting GRZ. CSH support to Afya Mzuri and CHAMP is the most sustainable through current budgetary and infrastructure (physical and IT) development support. These CSOs will continue to carry out IEC/BCC activities without further support from USG investment. The creation of Save Love Clubs and Safe Motherhood Action Groups (SMAGs) through partner community health organisations does not guarantee continuity of community level IEC/BCC activities because CSH has not created links between clubs and Neighbourhood Health Communities (NHCs), which are the GRZ-recognised structures for health delivery at the local level. Results to Date Campaigns for FP/RH and Nutrition have not been launched and activities on these two areas are clearly not on track to achieving their intermediate results (IRs). The malaria campaign has been launched but design of adverts is not yet underway. The general population is confused with the name of the malaria campaign and the campaign is not on track to achieving its IRs. The MNCH campaign has also been launched and is currently being implemented. The MNCH campaign is also not on track to achieving its intermediate and life of activity targets. The HIV/AIDS campaign is the most known of all the campaigns. It is on track to achieving its objectives in terms of the radio campaign. The TV campaign is however not on track to achieving its objectives. The project has adequate impact indicators but the outcome and output indicators are inadequate. Most of the outcome indicators do not have output indicators tracking them and some impact indicators do xi not also have outcome indicators tracking them. The data collectors have no tools for collecting the data. However, there are adequate tools to measure the capacity of GRZ on BCC related indicators. CSH’s role in community level work is unclear, with ZISSP meant to assist at his level but having limited resources to do this and restricted to working in a few geographic areas. It is not clear if CSH has any mandate in working with the newly formed MCDMCH since the ministry has no national level responsibility. The GRZ is challenged with funding and therefore has not been able to practice much of the technical knowledge they acquired from CSH support. The USAID policy of not paying allowances to GRZ staff for attendance of meetings and workshops is having a detrimental effect on the level of enthusiasm of GRZ staff with regard to CSH. RECOMMENDATIONS Recommendations to CSH Results to Date: The project should facilitate the process of launching the campaigns on FP/RH and Nutrition. The campaigns should be integrated at the community level by training the various community groups on multiple thematic areas to ensure that they are able to effectively disseminate messages on each of the thematic areas to community members. Message Exposure & Effectiveness: There is the need to produce more IEC materials and include materials in local languages to ensure that community members have access to these materials to educate themselves. This should not be a replacement for the current TV and Radio adverts but should supplement the effort to ensure effective dissemination. The existing community clubs should be trained in all the thematic areas. Capacity Building: There is need for more emphasis on quality mentoring on both BCC and M&E. This would require CSH attaching highly experienced, preferably international BCC and M&E Experts to GRZ, with a minimum of post graduate qualification with over 10 years working experience in their respective specialties across national boundaries, to mentor and coach them to ensure that they build skills and not just knowledge. The capacity building should be centered on systems and not individuals. The human resource departments should also be involved to enable them to plan the human resources required for various levels of BCC activities. CSH should be allowed by USAID to provide funding support to GRZ in implementing BCC activities. This can be done in the form of sub granting to the BCC unit or financing proposals from GRZ on BCC related activities. Evidence – Based Planning & Implementation: The capacity of NMCC and NAC should be assessed to identify their BCC limitations before planning on the support to provide to them. The planning system should not end after the project design, but should include planning on improving performance even during implementation by making use of feedback from monitoring, evaluation and research. Sustainability: CSH should ensure that capacity building is structured to include the entire GRZ system including the Human Resource departments, operations department, M&E department, and other departments who would contribute toward sustaining the USG support. The Clubs at the community level should be linked to the community structures such as the Neighborhood Health Committees (NHC) to ensure sustainable support after CSH. The work of CHAMP and Afya Mzuri should be linked to the activities of GRZ to ensure that the CSOs can continue to get some supervisory support when CSH ends. Recommendation to USAID USAID needs to clarify the responsibilities of CSH. Currently, various documents state that CSH is a national program, meaning they are supposed to be operating only at the national and provincial levels. However, CSH has district and community level responsibilities as well. USAID further needs to clarify CSH’s role in supporting the MCDMCH and ensure that subsequent projects continue the implementation of campaigns that were started by previous projects. Recommendation to the Government of Zambia The Government should ensure that staff are available and should install measures to reduce the attrition rate. The Government should ensure that there is formal communication to its development partners on the exact role and structure of the new MCDMCH ministry. This would enable projects like CSH to plan effectively to support them. 1 1.0 INTRODUCTION International Business & Technical Consultants, Inc. (IBTCI) is pleased to present the mid-term evaluation report for the United States Agency for International Development (USAID) funded Zambia Communications Support for Health (CSH) project. CSH is a four and a half year, $43 million contract implemented by Chemonics International under Contract number GHS-I-007-00004-00, Task Order number GHS-I-05-07-00004. CSH began on July 8, 2010 and is scheduled to be completed by December 13, 2014. CSH sub-contractors include the Manoff Group and ICF International. CSH sub-grantees include Afya Mzuri, and the Comprehensive HIV/AIDS Management Program (CHAMP). This report includes the findings, conclusions, and recommendations collected and developed over the course of the evaluation from November – December 2012. 1.1 EVALUATION PURPOSE AND RATIONALE The USAID/Zambia Health Office contracted IBTCI as an independent entity to perform a mid-term performance evaluation of CSH. The objectives of this mid-term evaluation were three-fold: Part A (Retrospective): The evaluation sought to help determine what progress CSH has made in achieving its Life of Project targets and whether or not they are likely to achieve them by the end of the program. Additionally the evaluation looked at what components of the CSH program were working well and to explain why this was so. If there were parts that were not working well the evaluation went further to find out why they were not working well. Part B (Prospective): Based on the findings and conclusions of the retrospective study, the evaluation made recommendations for CSH project implementation through December 2014, including the optimal mix of activities and funding for achieving project objectives and for aiding sustainability. Part C: Using the findings and conclusions of the prospective and retrospective studies, the evaluation framed issues to discuss or resolve at a level higher than the project, specifically at the level of the Government of the Republic of Zambia (GRZ) and/or other donor organizations, where applicable. 1.2 EVALUATION QUESTIONS The evaluation team was required to answer the following questions: Question Category I: Results to Date  To what extent is the activity on track to achieve its intermediate results (IRs) and meet its life of activity targets?  To what extent are the indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)?  What improvements can be made to better capture progress? What are the challenges to implementation and what can be done to improve the chances of the activity achieving its intended results and meeting its life of activity targets? Question Category II: Message Exposure and Effectiveness  To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with Information, Education and Communication (IEC)/Behavior Change Communication (BCC) messages in each of the five health intervention areas (HIV/AIDS, malaria, family planning (FP)/reproductive health (RH), maternal and child health (MCH) and nutrition)?  Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages?    2 Question Category III: Capacity Building  To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own?  What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own?  What else does GRZ need in order to independently plan, implement, manage, and evaluate national health campaigns?  To what extent is the project adapting to the current changes in the GRZ structures, particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re-organization of the National HIV/AIDS/sexually transmitted infection (STI)/tuberculosis (TB) Council (NAC)? Question Category IV: Evidence – based Planning and Implementation  In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? Question Category V: Sustainability  Are IEC/BCC activities likely to continue without further USG investments and if not, what investments or approaches would better promote sustainability? 1.3 PROJECT BACKGROUND Zambia is challenged with poor health outcomes which have a significant effect on the economic power of it citizens. The country records a high HIV prevalence (14.3% among adults and 16.6% among pregnant women), a high malaria burden (3.2 million reported cases in 2009), one of the highest fertility rates in the world (6.2 total fertility rate), a high maternal mortality ratio (591/100,000), and 45% of children under 5 years of age are stunted. To overcome these significant challenges the Government of the Republic of Zambia (GRZ), through its Ministry of Health (MOH), has committed to achieving Millennium Development Goals (MDG) targets and improving the health of its population by improving the quality of health care services and providing greater and equitable access to health care. To support these objectives, USAID, through the CSH program, is providing technical assistance to GRZ, targeting systems and interventions that will impact provision of health services and mobilize communities to actively participate in the management of health programs. It is expected that improved capability of the GRZ to implement effective health communications activities will translate to changes in population behavior — it will result in a measurable reduction in the practice of risky behaviors and increased demand for and use of health care services. Implemented in conjunction and collaboration with efforts to increase access to and quality of health care services, this will enable and result in improved health outcomes. CSH is tasked with supporting GRZ’s vision of “equity of access to assured quality, cost-effective, and affordable health services as close to the family as possible.” In the context of CSH, GRZ refers to three primary agencies: Ministry of Health (MOH), National Malaria Control Center (NMCC), and NAC. The GRZ through its MOH is committed to achieving MDG targets by improving the quality of health care services, and providing greater and equitable health care access for its people. To support these objectives, USAID is providing technical assistance to the GRZ in strengthening national health communications activities. The aim is for GRZ health communications activities supported by CSH to translate into increased sustainable local capacity and positive behavior change that contribute to GRZ efforts in five focal areas: 1) Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome (HIV/AIDS), 2) malaria, 3) FP/RH, 4) maternal, newborn, and child health (MNCH) and 5) nutrition. CSH operates primarily at the national level, providing technical assistance to the GRZ in development, implementation, and evaluation of health communications activities. CSH provides direct support to 3 GRZ in the planning, design, and implementation of communications campaigns and activities. This was to be done consistently with a focus on capacity building and transfer of skills. CSH was to measure the GRZ’s management capacity in IEC/BCC through a capacity index tool developed by the CSH Monitoring and Evaluation (M&E) team. The tool measures GRZ’s technical, management, and M&E capacity specifically as it relates to IEC/BCC activities. The tool provides an overall score, as well as scores for BCC planning and design, BCC program implementation, and M&E for BCC programs. While CSH’s office is located in Lusaka, its communications campaigns are designed to reach all ten provinces through a variety of different media (radio, TV, newspaper, community groups, Safe Motherhood Action Groups (SMAGs)) with messages in the focus areas mentioned above and generally targeting adults ages 15-49. Each campaign has its own specific target groups. The Safe Love campaign and the Integrated Malaria, MNCH and Nutrition campaign focus on adults ages 15-49, and the Mothers Alive (safe motherhood) campaign specifically targets pregnant women. Also, CSH was to implement three “mini campaigns” under the larger Safe Love campaign (Gender Based Violence, Alcohol, and Youth) and these were to focus on their own respective target groups. The national level campaign rollouts were staggered, thus the HIV/AIDS campaign was launched in June 2011, the Integrated Malaria, MNCH, and Nutrition campaigns were launched in November 2011, and the Safe Motherhood campaign was launched in April 2012. Family planning and reproductive health messages are integrated into the Safe Motherhood and Integrated campaigns. CSH is contractually mandated to work closely with the USAID funded Zambia Integrated Systems Strengthening Program (ZISSP), where CSH implements national communications campaigns and ZISSP brings the same messages down to the community level through its support of SMAGs and Community Health Workers (CHWs). CSH and ZISSP meet regularly to discuss their collective communications and capacity building activities. In February 2012 they developed a joint malaria implementation plan for 2012 to ensure coordination and to minimize the duplication of efforts. In attempts to link messages seeking to increase demand for health services to actual service delivery, CSH often works with the Private Sector Mobilization Project for Social Marketing (PRISM) to offer male circumcision and voluntary counseling and testing (VCT) services at CSH sponsored events. As of May 2012, $21,103,684 has been obligated to the award across the five program areas mentioned above (HIV/AIDS, malaria, FP/RH, MCH and nutrition). There have been three modifications to the contract to date, two of which added incremental funding and the third reduced the total estimated cost of the contract. The reduction in funding was minimal ($11,749) and was to adjust the Total Estimated Cost, exclusive of fixed fee. A shift in programming occurred in October, 2011, when Zambia was chosen as a focus country for the Saving Mothers, Giving Life (SMGL) endeavor to reduce maternal mortality by 50% in four target districts (Kalomo, Lundazi, Nyimba, and Mansa) in Zambia. USAID/Zambia did not receive any supplemental funding for these efforts, so USAID asked its existing implementing partners to adjust their planned activities so that they focus on these four districts. CSH had already designed the Safe Motherhood national communications campaign, called Mothers Alive, so USAID/Zambia asked CSH to take these same messages and work with ZISSP to bring the Safe Motherhood messages to communities through SMAGs. To coordinate, monitor and evaluate these efforts, CSH hired two new staff, an SMGL Coordinator and an SMGL Monitoring and Evaluation Specialist. The predecessor to the Communications Support for Health program was the Health Communication Partnership (Johns Hopkins University Center for Communication Programs). HCP was a five-year $33 million cooperative agreement that launched in August 2004 and ended in December 2009. Contributing to USAID/Zambia’s approach of Zambians taking action for their health (intermediate result under the old Assistance Objective, Improved Health Status of Zambians), the program worked closely with the MOH, NAC and the NMCC on mass media, Information, IEC/BCC materials and community development to increase knowledge of health issues and promote changes in risky behaviors as well harmful gender and other socio-cultural norms.    4 2.0 EVALUATION METHODS The project has multiple thematic areas and is being implemented in multiple strategies. In view of this, the evaluation team adopted multifaceted approaches to effectively respond to key evaluation questions. The approaches included a document review, mini survey, key informants interviews, focus group discussions, and an online survey. Each of these approaches adopted a different methodology to ensure triangulation. 2.1 EVALUATION TEAM STRUCTURE The evaluation team was comprised of an international consultant serving as the Team Leader (Joseph Limange, MBA), an international consultant serving as the Senior BCC/IEC Advisor (Maurice Ocquaye, MFA), two Zambian Research Specialists (Saviour Chishimba, PhD and Moses Simuyemba, MD, MPH), and two Research Assistants (Sharon Mwangani, Dip and Chendela Masengu, BSc). 2.2 EVALUATION DESIGN The evaluation adopted a non-experimental design that involved three phases. During the first phase the team conducted extensive background research and analytical review of existing documentation and project data sets, including documents and data on MNCH, nutrition, FP/RH, malaria, and HIV/AIDS and key implementation factors provided by USAID/Zambia and CSH. These documents, quarterly and annual reports, and internal reporting were thoroughly analyzed by the team. Further, the team conducted initial consultations with USAID/Zambia, CSH, ZISSP and PRISM. The desk review and consultation meetings enabled the team to create appropriate data collection instruments for key informant interviews (KIIs), Focus Group Discussions (FGDs), and a mini survey. Annex F contains an Evaluation Design Matrix that presents the different data collection methods and sources the team used to answer each evaluation question. During the second phase, the team used a mixed-method approach for quantitative and qualitative data gathering with a purposive sample of key informants who have good knowledge of GRZ’s activities, CSH support to GRZ, or both. The purposive sampling criteria are explained in greater detail below. The key informants selected include GRZ employees, CSH staff, other USG projects, CSH sub grantees and GRZ health development partners (both local and multinationals). Following the KIIs, the team randomly selected and conducted FGDs with Safe Love Clubs, Radio Listening Groups, SMAGs, and Facilitators Groups formed by civil society organizations (CSOs) with funding support from CSH. These FGDs were conducted simultaneously with a mini survey targeting the general population including men and women between the ages of 15 – 49 years. The team also developed and conducted an online survey. The final step of the data gathering plan was a detailed Systems Assessment and Data Quality Assessment (DQA); the team also reviewed an earlier DQA. The third phase was data analysis and compiling the report. 2.3 DATA COLLECTION METHODS Five main data collection methods were used to evaluate the project. They included a document review, KIIs, Online Survey, Focus Group Discussion, Mini Survey and BCC Systems/Data Quality Assessment. 2.3.1 Document Review The evaluation team’s assessment of CSH’s overall strategy, goals, objectives, approach and results began with a thorough review of CSH’s foundational documents. Key among these document included quarterly project reports, the Performance Monitoring & Evaluation Plan (PMEP), formative research reports on HIV/AIDS, Safe Motherhood, and Malaria, the Safe Love Rapid Survey Report, CSH Self-Assessment Report, sub-grantee reports to CHS, and the scope of work for ZISSP. The team also reviewed other documents that have been developed by GRZ through the support of CSH and reports from various units in GRZ receiving technical and financial support from CSH. For a complete list of documents consulted, see Annex O List of References. Information gathered during this phase was used to develop 5 more detailed questions about CSH’s activities as well as output for the KII, FGD, BCC Systems/Data Quality Assessment and Mini Survey. 2.3.2 Key Informant Interviews The evaluation team conducted a stakeholder mapping exercise and identified over 60 key informants who have knowledge of GRZ health activities and some relationship with CSH. These key informants were contacted but were not available for interviews. The evaluation interviewed 50 key informants from various organizations including CSH, ZISSP, PRISM, NAC, MCDMCH, MOH, NMCC, CHAMP, Afya Mzuri, the Southern Africa HIV and AIDS Information Dissemination Service (SAfAIDS), Christian Health Association of Zambia (CHAZ), Planned Parenthood Association of Zambia (PPAZ), United Nations Population Fund (UNFPA), Japan International Cooperation Agency (JICA), EU, and the Network of Zambian People Living with HIV/AIDS (NZP+). Reference Annex E for a list of persons consulted and Annex H for the KII interview guide. 2.3.3 Online Survey Based on the KIIs conducted, an online questionnaire was developed to enable key informants to respond to specific questions and quantify the responses. The survey instrument was sent out to all key informants who had been identified, including those who could not be reached due to their schedule, totaling 60 people. Two separate emails were subsequently sent as reminders. On the final day of the survey, a phone call was placed to each key informant who had not responded. These efforts resulted in 28 key informants responding to the questions. The majority of the respondents from the online survey were CSH Sub Grantees, accounting for 35% of the total respondents. CSH staff accounted for 27% while GRZ staff comprised 15% of the respondents. Other USG Projects such as ZISSP and PRISM were 11% and GRZ Health Partners such as EU, JICA and UNFPA accounted for 12% of the total respondents. Reference Annex N to view the Online Survey questionnaire. 2.3.4 Focus Group Discussions The team developed and used a FGD guide to interview various groups of stakeholders and beneficiaries. The evaluation team conducted a total of 27 focus groups in six provinces. The evaluation team held separate FGDs for members of each group. The groups contacted include projects supported by Safe Love Clubs, Facilitator Groups, Women’s Groups, Men’s Groups, Radio Listening Clubs, Mothers Alive Clubs, as well as Radio Presenters trained by CSH. These FGDs were conducted in 13 districts: Kabwe, Kapiri Mporshi, Ndola, Luanshya, Mansa, Samfya, Lusaka, Kafue, Chongwe, Chipata, Lundazi, Livingstone, and Kalomo. In all, 325 people were interviewed: 177 women and 148 men. The team assessed the quality of the various IEC/BCC messages, especially the adverts, by showing the clips repeatedly to the audience and asking a series of questions. The responses to these questions were accordingly captured. The FGD guide used by the evaluation team can be found in Annex J. 2.3.5 Systems Assessment and Data Quality Assessment The evaluation design called for a System and DQA for CSH, MOH, and NAC. However, during the initial assessment for MOH and NAC, it was established that there are no M&E systems specifically for IEC/BCC Figure 2: Facilitator Group members reviewing adverts in Ndola during a FGD. Figure 1: Percentage of Online Survey Respondents by Organization    6 programs. Detailed assessment was therefore conducted only for CSH. In assessing the M&E system, the team evaluated CSH’s M&E plan, indicators and their definitions, data collection protocols, data collection tools, databases, data backup mechanisms, and reporting procedures. A systems assessment was also conducted for CSH, MOH, NAC, and NMCC. This assessment evaluated the systems in place for BCC planning, implementation, and management. The assessment included inspecting BCC plans, training documents, BCC structures and materials, and distribution systems. The DQA Question Guide is located in Annex K. 2.3.6 Mini Survey A mini survey was conducted as part of the evaluation to respond to questions related to the general population and to access the extent to which various campaign messages are reaching their intended audience both in urban and rural areas. The Mini Survey questionnaire is located in Annex M. Sampling Frame The country was divided into nine clusters, with each province being a cluster. Six of the nine clusters were then selected for the survey. Purposive sampling criteria were used in selecting these clusters. They include four provinces in which CSH has a measurable level of activities and two provinces from which CSH has a limited level of activities. The provinces sampled include Lusaka, Southern, Eastern, Central, Copperbelt and Luapula. In each of these provinces, two districts were randomly selected. To ensure a fair combination of rural and urban dimensions, the districts in the province were further clustered into rural and urban before the random selection. Interviewing therefore took place in six urban districts and 6 rural districts. Sample Size The evaluation interviewed 242 community members in all the twelve selected districts. The number interviewed in each District, however, depended on the proportion of the district population to the total population of the twelve districts. To determine this, the team used the population data from the Zambia 2010 population and housing census. Table 2.1: Percentage distribution of respondents by age-group, marital status and gender. Age Group/Marital Status Sex Total Male Female Age Group 15-19 5.8 8.3 34 (14.0%) 20-24 10.7 12.0 55 (22.7%) 25-29 4.1 14.0 44 (18.2%) 30-34 4.1 8.3 30 (12.4%) 35-39 2.9 10.3 32 (13.2%) 40-44 2.9 6.6 23 (9.5%) 45-49 3.7 6.2 24 (9.9%) Marital Status Single 19.0 17.8 89 (36.8%) Married 14.9 37.6 127 (52.5%) Divorced/ Widowed/ Separated 0.4 7.0 18 (7.4%) Living together but not married 0.0 3.3 8 (3.3%) Total 83 (34.3%) 159 (65.7%) 242 (100%) 7 Sampling Method In each of the selected districts, a community was selected at random with a rural community being selected from a rural district and an urban community selected from an urban district. In the selected community, the interviewer randomly selected a house from the first five houses to start the interview. From the selected households the interviewer systematically selected every five houses in urban areas and every other house in rural areas. The interviewer interviewed one person in each household until the required sample for the district was achieved. Data Collection and Analysis A single questionnaire was used to collect the data. The team then used Formic software that scans, extracts and enters hand-written information. This process eliminated manual data entry errors. Population Characteristics The sampled population included 65.7% women and 34.3% males despite conscious efforts to reach a balanced gender population. The majority of those interviewed were between the ages of 20-24 years accounting for about 22.7% of the total population, while those between the ages of 40-44 years recorded the lowest proportion at 9.5%. Most of the respondents were married, accounting for 52.5% of the respondents. Respondents who were single and those divorced/ widowed/ separated accounted for 36.8% and 7.4% respectively, with 3.3% living together but not married. Geographic Distribution The majority of the respondents were selected from urban locations, accounting for 69.0% of the sampled population while rural residents accounted for 31.0% of the respondents. Most of the respondents were selected from Lusaka, accounting for 43.0% of the total sample. About 17.4% were interviewed from Eastern Province, the second highest proportion, with the Southern Province recording the smallest proportion at 7.9%. The sample proportion was based on the district’s population in proportion to the total population of the 12 districts sampled. 2.4 ANALYSIS PLAN Data were collected from four main sources including the responses from the mini surveys and the online surveys. To ensure a high degree of accuracy in data analysis, the questionnaires were pre-coded and formatted for data entry into an SPSS template. The mini survey and online survey questionnaires included respondents’ biostatistics such as location, age group, and sex. The team reviewed all collected data before entry into the SPSS. The team reviewed collected data in the field at the end of each day in order to ensure that the correct sequence of unique identifiers were used for each data point, and that the right location, gender, and other socio-economic data were transcribed by using a simple coding system. The Research Specialists oversaw data collection and cleaning processes in the field and confirm that team members complied with the necessary protocols. The data questionnaires were then programmed into a data capture scanner with a designed SPSS template. The team scanned the data into the SPSS designed template to avoid data entry errors. This process was closely supervised by the Team Leader to ensure a high level of data quality, efficiency, and timeliness. The team used SPSS to analyze clean data and generated various frequency tables for results as appropriate. The inclusion of basic socio-economic variables of respondents on the mini survey and Table 2.2: Percentage distribution of respondents by province and urban/rural residence Province Residence Total Urban Rural Central 4.5 5.5 24 (9.9%) Copperbelt 9.9 3.3 32 (13.2%) Eastern 10.3 7.0 42 (17.4%) Luapula 3.3 5.4 21 (8.7%) Lusaka 38.0 5.0 104 (43.0%) Southern 2.9 5.0 19 (7.9%) Total 167 (69.0%) 75 (31.0%) 242 (100%) Figure 3: An interviewer conducting an interview in Kamkwiba.    8 online survey questionnaires enabled the team to conduct bivariate and multivariate data analysis and generate reports that were disaggregated by gender and location. The team assessed the effect that various campaigns were having on women as compared to men in the different geographic locations, as well as across education and income levels, relative to knowledge, attitude and behaviors. The inclusion of organizational affiliation enabled the team to assess the perceptions and level of involvement of key implementing partners. To ensure flexibility in the development of charts, the analyzed data was exported from SPSS into excel for the creation and labeling of charts for analytical reporting. 2.5 LIMITATIONS The evaluation was challenged with limitations, which could have some effect on the analysis and findings. The most significant among them include: Limited Time for Evaluation - The evaluation of such a complex and multifaceted project like the CSH is challenging and needs more time. The evaluation team used a polygonal approach to ensure every aspect of CSH activities was covered. This often required more time and the limited period at the team’s disposal was a major challenge. To achieve this in such a limited time frame the evaluation team conducted multiple phases simultaneously and was successful in accomplishing the task within schedule. Mini Survey Limitation - The resources and time available to the team meant the team had to conduct a mini survey rather than a comprehensive sample survey. Mini surveys have a limited sample size that may not be highly representative and the questions are usually fewer. This can have an effect on generalizing the results. To ensure that the results were as representative as possible, the evaluation universe was clustered and the survey conducted in selected districts. Gender Imbalances in Sampled Population - The evaluation team kept in the much needed efforts to ensure that it was able to get a representative sample, reflecting geographic proportions as well as gender balance. However, in all cases there was not enough gender balance. The mini survey interviewed more women than men because households that were sampled more often had women available for interviewing than men. Also, women were generally more hospitable to interviewers and more willing to respond to survey questions than men. In the KII and online surveys, samples were selected based on organizations and positions of responsibility. This meant that it was not possible to consciously cater for gender disparities. Poor Response to Online Survey - The online survey questionnaire was sent to about 60 key informants but just about half responded. Two consecutive email reminders were sent to key informants who had not initially responded. On the final day of the survey a phone call was placed to each person who had not responded at the time. Despite these efforts, less than half of the sample responded. Absence of Some Key Informants - The team initially selected over 60 key informants to interview. It was difficult to contact some of them, and of those who were contacted not all were available. The team made conscious efforts to reach more key informants even after the field work was completed, but some very important stakeholders could still not be reached. Absence of Institutional Review Board (IRB) Approval - The short evaluation period prevented application for IRB research approval, preventing the asking of sensitive questions. For example, the team refrained from asking questions on intermittent preventive treatment of malaria for pregnant women (IPTp) because it would require asking about their HIV status which could infringe on their right to confidentiality. 9 3.0 FINDINGS CSH has successfully planned, launched, implemented, and managed three major campaigns over the past two years. Two campaigns, one on nutrition and one on FP/RH, are yet to be launched. However, components of these two campaigns are covered in the Safe Love and Mothers Alive campaigns. The three campaigns currently being implemented include the following: Safe Love Campaign The Safe Love Campaign is focused on HIV and is a continuation of the “One Love Kwasila” campaign that was implemented before CSH. The Safe Love Campaign has three main goals: reducing the rate of multiple concurrent sexual partners (MCP), addressing the challenge to condom use; and increasing the utilization of prevention of mother to child transmission (PMTCT) services. To achieve these goals, CSH has established Safe Love Clubs, Radio Listening Clubs, and Facilitator Groups in targeted communities. These clubs conduct home visits to educate their community members on HIV. CSH has also sub￾granted to CHAMP to use 990 Talkline Counselors to respond to the concerns of callers. CSH has also sub-granted to Afya Mzuri to establish and furnish information resource centers across the country. The project also uses TV and Radio adverts to promote healthy behaviors. Mothers Alive Campaign This is a campaign to improve MNCH with the main aim of reducing maternal mortality. The campaign has some components addressing FP and Nutrition issues. The goals of the campaign are to improve the use of modern contraceptives, encourage early initiation of antenatal care (ANC), encourage completion of at least 4 ANC visits, completion of the birth plan, facility-based delivery and post-partum follow-up care. This campaign is being promoted mainly by Change Champions - a group of community leaders conducting MNCH education in their Chiefdom. The project also uses TV and radio adverts to educate the public, with some of the community level clubs taking it upon themselves to spread MNCH messages. STOP Malaria Campaign The malaria campaign was launched last year. Since then the project has been using community-level interventions and media education to spread malaria messages. The campaign seeks to achieve three main goals: promoting the testing for malaria parasites before treatment, IPTp, and the use of insecticide treated nets (ITNs). 3.1 QUESTION CATEGORY 1: MESSAGE EXPOSURE & EFFECTIVENESS To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with IEC/BCC messages in each of the five health intervention areas (HIV/AIDS, malaria, FP/RH, MCH and nutrition)? There are significant variations on the levels of exposure in urban compared to rural areas as well as across thematic areas. The Safe Love Campaign seems to be the most popular both in urban and rural areas. Figure 4: Mothers Alive Campaign    10 Exposure to Safe Love Campaign The Safe Love campaign is the most popular of all the campaigns. All the groups interviewed had very good knowledge of the HIV campaign and could remember the name as Safe Love. This popularity was not limited to urban areas but also included rural communities. About 67.3%2 of the general population has either seen a Safe Love program/advert on TV or heard of it on radio. The TV is the main source from which urban dwellers heard about Safe Love. About 63.5% of urban residents heard of Safe Love through the TV compared to 50.9% who heard through the radio. This situation is however different in rural communities where radio turned out to be the main source of information on Safe Love. About 56.0% of rural dwellers heard of Safe Love from radio compared to 9.3% of rural residents who saw it on TV. This could be due to the absence of electricity and other social facilities in most rural communities. About 53.3% and 57.3% of urban and rural residents, respectively, were aware of Community Facilitators educating people on health behavior change in their communities, but only 28.1% and 29.3% of urban and rural dwellers, respectively, indicated having benefited from any such education. The failure to reach more people was attributed to the limited numbers of IEC materials including posters, leaflets, brochures etc. Members of various groups responsible for educating the communities and distributing these IEC/BCC materials indicated that they only had enough for themselves and not for distribution. It was clear not only that the materials produced were not enough, but also that the distribution channel is weak, meaning that even when more materials are produce, the distribution channels need to be strengthened before the materials can get to the ultimate beneficiary. All the IEC materials were also developed in English, limiting the ability of most rural residents, who can only read their local languages. The team also found CSH’s monitoring of the distribution of these materials to be very weak, preventing GRZ and CSH from knowing whether these materials indeed get to the ultimate beneficiaries. All Community Groups (Safe Love Clubs, Radio Listening Clubs, and Facilitator Groups) had the impression that the Safe Love campaign was for six months and had ended. The vigor with which these groups pursued the campaign during the six months following campaign launch had therefore diminished, with group members using their initiative to periodically carry out community education. The Talkline Counselors are better known in urban areas, with 53.3% of urban dweller indicating they are aware of them compared to only 14.7% of rural dwellers. However, only 8.4% of urban dwellers are benefiting from their services. This is worse when compared to the rural dwellers, where 0.0% indicated having benefited from their services. Most of the Groups interviewed were confused between the One Love Kwasila campaign and the Safe Love Campaign. They indicated that both campaigns were running, but the One Love Kwasila campaign was about reducing MCPs while the Safe Love campaign was about condom use. The Safe Love campaign was also known by stakeholders to be owned by CSH and not GRZ. Whilst CSH had involved GRZ in most of the campaign activities, it was always led by CSH and therefore seen by stakeholders, including GRZ, as the property of CSH. CSH had deadlines to meet with the launch and                                                              2 This figure is from a bi-variate analysis of people who heard Safe Love on radio and those who heard it on TV. It therefore presents the universal set of those who have heard of the Safe Love campaign. Table 3.1: Level of Exposure by Campaign Campaign Urban Rural TV Radio TV Radio Safe Love 63.5 50.9 9.3 56.0 Mothers Alive 18.6 13.2 4.0 25.3 STOP Malaria 71.3 64.1 10.7 61.3 11 implementation of campaigns and was therefore moved to lead the campaign when the capacity and resources of GRZ may not have matched the pace required to meet these deadlines. Exposure to Mothers Alive Campaign The Mothers Alive campaign was found not to be popular among all the community groups promoting health behavior change with support from CSH. Whiles most of them indicated they were aware of a campaign on MNCH, none of them knew the name of the campaign. Most group members mentioned Safe Motherhood as the name of the campaign. These groups however had some knowledge on MNCH. The groups also did not have any materials on MNCH. About 21.6% of urban residents had heard of the Mothers Alive campaign with some 18.6% hearing it on TV and 13.2% hearing it on radio. Rural residents, however, mostly heard of it on radio. About 25.3% of the 26.6% rural residents who indicated they have heard of the Mothers Alive campaign heard of it on radio, with only 4.0% of rural dwellers hearing of it on TV. One major approach to the Mothers Alive campaign was the use of Change Champions. The Change Champion strategy was much more popular in rural areas than urban areas. About 37.3% of rural residents’ were aware of the Change Champions but only 24.0% had heard their messages. This was worse in urban areas where 17.4% were aware of them but only 11.4% had heard their messages. STOP Malaria Campaign The Stop Malaria campaign was also found not to have been popular in the communities. While every group claimed to be aware of this campaign, no single group was able to mention the name. Most groups referred to the campaign with expressions like “Malaria Free Future,” “No More Malaria,” “Let’s Kick Malaria Away,” etc. In Luapula province, every group referred to the Malaria campaign as “Malaria Consortium.” The claim by most groups that they were aware of the malaria campaign could have been because several organizations had been launching one malaria campaign after another, confusing audiences as to which was running. About 71.3% of urban dwellers claimed to have heard of the Stop Malaria campaign on TV while 61.3% of rural residents claimed to have heard of it on radio. However, since the launch of the Malaria campaign last year, no TV or radio programs have run on the campaign. This means these groups might be confusing the Stop Malaria campaign with other malaria campaigns. Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages? Various videos were shown to the target audience to seek their understanding of the messages conveyed by the adverts. Most respondents in urban areas had seen the Safe Love videos but not the Mothers Alive videos. Comments received on the videos are presented by videos and, where appropriate, related videos are grouped. These videos are categorized into Safe Love and Mothers Alive. Safe Love Adverts The background messages on these adverts made it clear that the young girl and boy embracing a much older person were not embracing their parents, however, most community members who did not speak English and could therefore not understand the background message, had various differing understandings. Most said the ‘the men and women in the cities hug their children’ and therefore it is normal. Others who could understand the English language also saw them to be friends, mates, relatives, etc. and therefore saw nothing wrong with it. Some were however quick to link it to Sugar Daddy and Sugar Mummy but with very little linkage to HIV since they did not see anything indicating that these pairs had sexual related activities. One community member summed up the thought of this group of people “they need money to pay their fees and have to rely on adults to get such money. That is how the economy has become.”    12 Figure 6: Scene from the Condom Girl advert. The Condom Girl advert was seen by many in a positive light with community members clearly understanding the message. People thought it was appropriate but would have preferred the lady bringing out a female condom and not a male condom. Most men perceive the lady to be promiscuous for keeping male condoms. There was also no sign that the gentleman accepted the condom. When discussing the “As a man, I can have many girl friends” and “MCP-guy” adverts, members of various Safe Love clubs were quick to identify both adverts as MCP. This could have been due to their knowledge of HIV￾related issues including MCP. However, when community members who are not members of any of the groups trained by CSH were interviewed, a lot of them thought the scenes in the “MCP-guy” advert could have been brought out more clearly to illustrate the point on MCP. On the other hand, the “As a man” advert was clear since the man was seen in bed with multiple girls at different times. The Change Champion advert produces misleading information. Whilst the intent is to encourage women to access ANC and postnatal services to stay alive, most people interviewed understood the advert as a message to government. The advert begins by narrating the experience of a woman who gave birth in the bush due to the distance to the nearest clinic. Since it is neither the responsibility of the woman nor members of the community to establish clinics closer to the community, but rather the responsibility of government, most people understood it as a message to government to ensure that there are clinics closer to the communities. Further, a man in the advert attributes the absence of maternal deaths in the community in the past two years to the Change Champion. This period predates the Mothers Alive campaign and could therefore not be factually accurate. If the campaign launch was effective, people could remember when it started and would feel misled by the advert. Figure 5 Scenes from 'Sugar Daddy' and “Sugar Mummy” adverts. Figure 7: Scene from 'as a man' advert. Figure 8: Scene from MCP advert. Figure 9: Scene from the "keeping mother alive" (Change Champion) advert 13 The advert on Safe Motherhood testimonials was acknowledged by many to be very good and educative as it reflects the circumstances of many women in the communities. The advert was also easy to understand as vocal in the background were audible. People were however quick to ask, “Why are all the women from Eastern province?” This was echoed in almost every province the team showed the video. The community members could identify from the surnames that they were from Eastern province. Another challenge identified from the video was that one of the women in the video advertising Mothers Alive was rather wearing a Safe Love chitenge (wrapper). This buttresses the point that there has been considerable concentration on Safe Love to the detriment of the other campaigns. A third concern raised by people on this advert was that it was quite long, with the women basically repeating the same thing. It therefore made the advert boring: by reducing the length of the advert, the project could save cost and improve quality. Some women groups raised concern that the advert seemed to put the entire responsibility of MNCH on women, rather than encouraging their spouses to get involved to support the women to attend ANC. There was also an error at the beginning of the narration, where the narrator states that “For every woman who dies in pregnancy or labor or just after, there are more than 30 who experience severe complications.” The statement is impossible and should surely be an error needing correction. Finally, some community members noted that the women kept indicating they went to the University Teaching Hospital (UTH), creating the impression that UTH was the only place such situations could be attended to. With most women in rural areas who cannot easily have access to UTH, that can be discouraging. 3.2 QUESTION CATEGORY II: CAPACITY BUILDING To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own? This section of the report presents the findings of the evaluation in relation to capacity building of GRZ to be able to effectively design, implement, manage and monitor BCC activities. Key among the mandates of CSH is the building of capacity of GRZ especially at that national level through the provision of technical and financial assistance. Specifically, CSH is tasked with working with the three primary agencies — the MOH, the NMCC, and NAC. The CSH mandate to GRZ includes the provision of direct support in the planning, design, and implementation of communications campaigns and activities. The project used six main approaches to build the capacity of GRZ to implement IEC/BCC activities on its own. These include the following activities and accomplishments. Workshop Style IEC/ BCC Trainings The project used a series of workshop-style meetings to build the capacity of key personnel for the three key institutions to familiarize them with the key objectives of CSH, identify immediate areas of support, and to lay out a longer-term strategy for project support. In terms of training in BCC approaches, CSH carried out an institutional assessment to identify gaps and to design products and tools to equip staff at national, provincial, and district levels with skills in IEC/BCC planning and programming at MOH only. Assessments for NAC and NMCC are planned for early 2013. The design of the Behavior Centered Programming (BCP) training tool kit as well as the adoption of a trainer of trainers approach was highly appreciated by GRZ staff and other beneficiaries as very effective in building capacity and transferring skills in planning and implementing BCC activities. Downstream BCC trainings using the BCP tool kit at the community level was being implemented by district facilitators with supervision from CSOs sub granted by CSH. In addition to training in how to Figure 10: Scene from Safe Motherhood testimonial advert.    14 design and implement BCP approaches, CSH supported the development and printing of National Communication Strategies for Malaria and HIV/AIDS. Both strategies are aligned to the new strategic frameworks for 2011-2015. CSH has equally provided technical and financial support for a national stakeholder workshop to develop the Male Circumcision (MC) communication strategy that is aligned to the Zambia MC operational plan 2009-2012. With all this training and support for GRZ, respondents of the online survey thought GRZ cannot implement BCC activities on its own. This is because GRZ does not have the resource to implement what they are being taught to become familiar with it: they have learned the theory and would need to practice to gain the experience. Explaining this point, a key informant quizzed “Where can GRZ get even 5% of $43 million to implement campaigns and become used to it?” BCC Mentorship To facilitate the process of integration and support to GRZ in the delivery of high quality BCC activities, CSH initiated discussions with the GRZ on placement of BCC Specialists at the MOH, NMCC, and NAC. CSH developed scopes of work with GRZ partners and recruited the staffers that were placed at MOH, NMCC and NAC. Apart from participating in IEC/BCC planning and programming at their corresponding institutions, the embedded staff members served as a link between CSH and the three institutions. The presence of the embedded staff at the three institutions and style of work in terms of their ability to follow through assigned task with little or no supervision served as a source of motivation and inspiration to both management and staff at each agency. They also identified gaps in BCC in all three institutions and linked them with the relevant resources and consultants who provided technical assistance in the implementation of BCC activities. For example, the three specialists, with support from the CSH team have continuously supported the MOH, NAC and NMCC to develop communications messages, materials, and products to support routine communications activities implemented by the GRZ, such as the Child Health Week, World AIDS Day, Africa Malaria Day, National VCT Day, traditional ceremonies and communications activities to support mass distribution ofITNs. Monitoring & Evaluation Support CSH conducted an institutional assessment to identify gaps in designing and implementing BCC activities, and as a result collaborated with ICF Macro to conduct a Formative Research Trainer of Trainers (TOT) workshop for GRZ staff to transfer skills on how to conduct formative research with the aim to strengthen local CSH and GRZ staff capacity in this field to influence design and implementation of BCC activities. Further, CSH provided technical support to GRZ on the revision of NAC’s activity reporting forms, aligning them to the current National HIV/AIDS M&E Plan 2011-2015 and including adequate BCC indicators. In addition, CSH is in the process of developing a database for MOH, NAC, and NMCC. These are part of strategies being used by CSH to support GRZ to set up their M&E system. CSH is however not providing any mentoring support on M&E to GRZ as in the case of BCC. Support to Technical Working Groups CSH has supported the MOH, NAC and NMCC to revise the selection criteria for IEC/BCC Technical Working Groups (TWGs) who have subsequently reviewed their own TWGs and appointed members to the national TWG based on the revised selection criteria. These groups are active and offering their technical expertise to the three institutions. CSH is represented in the TWGs to offer their technical expertise in the developments of technical documents. For instance, through the NAC IEC Technical Working Group, CSH provided technical input to conceptualize and design television spots to promote condom use among circumcised men. The TV spots were pre-tested and then aired on Zambia National Broadcasting Corporation and MUVI TV. BCC Coaching Similarly, CSH has collaborated with the NAC IEC TWG to design and develop a comprehensive HIV prevention campaign; Safe Love: Think. Talk. Act. that seeks to address MCP, low and inconsistent condom use, and mother to child transmission of HIV (MTCT). Similar campaigns have been developed for Malaria - Stop Malaria; and for MNCH - Mothers Alive. These campaigns are ongoing and several BCC 15 products, ranging from TV and Radio spots to print and community level activities are being rolled out with GRZ and partners at the provincial and districts levels. Most of these activities are in English language, which poses a challenge to rural folks who cannot read English. Due to the reporting requirement, CSH is sometimes compelled to provide more than coaching support, thereby actually taking the responsibility and leading the design of these campaigns. This has sometimes resulted in the design of complex campaigns and programs that GRZ agencies find difficult to implement and sustain. Currently, CSH is supporting GRZ to design a TV program called “Your Health Matters” (YHM). This program was initially designed and managed by GRZ but went off air after some time. With support from CSH, the program is being reviewed and some partners think CSH is ‘hijacking’ the review rather than supporting it. According to one key informant, CSH is “creating a monster” of the program and it would not be possible for GRZ to manage or sustain it. Logistic Support to Local GRZ Partners CSH has provided direct assistance to GRZ’s local partners by sub granting to CHAMP and Afya Mzuri for the expansion of the 990 Talk line and the Health Communication Resource Centre. With Afya Mzuri, CSH has supported the provision of infrastructure and logistics, expansion and decentralization of Dziwani satellite points to Southern, Eastern, Copperbelt, and Western provinces, development of a web portal, and advertising of satellite points. Support given to CHAMP also includes designing a web portal, recruiting and orientating more telephone counselors, and development of a short term marketing plan. CSH also engaged a consultant to develop a new and expanded 990 training curriculum for counselors to include nutrition, MNCH, FP and Malaria, enabling CHAMP to go beyond their original aim of providing HIV counseling services. These two institutions will be able to provide a broad spectrum of health information beyond information about HIV/AIDS. What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own? In pursuance of CSH’s mandate to build capacity of GRZ, CSH has embedded BCC specialists in the three institutions to facilitate easy access by GRZ, ensure the smooth implementation of CSH’s activities, link GRZ to technical and financial resources from CSH, and offer on-the-job mentoring and coaching to GRZ staff regarding the most effective and efficient ways of designing, implementing and monitoring BCC activities. Mentorship by CSH This approach has proved ineffective in building the capacity of GRZ because of the limited capacity of the CSH staff embedded in GRZ to mentor GRZ. Key staff at the three institutions, NAC, MOH and NMCC, appear not to have a clear understanding of the roles of the embedded staff, even though there is a memorandum of understanding (MOU). This is because the staff have no capacity to perform their responsibilities stated in the MOU, which is to serve as BCC advisors to GRZ. All three institutions confirmed that they lack understanding of their roles and responsibilities in their institution apart from the fact that they are CSH’s embedded staff and are responsible for ensuring the successful completion of CSH’s planned activities. Besides that, they do not see any added value for the placement in terms of building capacity in the implementation of BCC activities. Embedded staff have been very helpful and supportive in coordinating CSH activities at all levels. Nonetheless, they appear to lack the capacity to offer mentoring and coaching on the job to transfer needed skills in BCC design, implementation and monitoring in the three institutions. These staff members were initially placed in GRZ as BCC Advisors to mentor BCC staff in GRZ to enable them to effectively carry out IEC/BCC activities independently without support. The capacities of the embedded staff are perceived, however, to be lower than what already existed in GRZ. Commenting on their capacity, a key informant from GRZ asked “who are they going to mentor?” According to another key informant, the title of these staff was developing uneasiness among the GRZ staff who apparently felt more competent than the embedded staff. Since these staff could not mentor GRZ BCC staff, their titles were changed from BCC Advisors to BCC Specialists to enable them to play a coordinating role rather than a mentoring role. Therefore CSH is currently not providing any direct mentorship to GRZ as planned. Despite embedding staff in the three institutions, it    16 appears that planning with government counterparts to ensure CSH programs fit within planned government activities remains relatively weak. CSH is rolling out a number of activities at both national and provincial levels, in most cases working with GRZ counterparts. However, the need to meet project deadlines and deliverables, coupled with the fact that occasionally government counterparts are either engaged in other activities or have other priorities gives, room for CSH to sometimes compromise on closely with GRZ to roll out its activities without GRZ’s active involvement. Lack of Motivation for Trainings Training activities by CSH are given low priority by GRZ since GRZ staff do not seem motivated to attend. In some instances, CSH invites District Health Officers for training only to see Environmental Officers representing the health officers. The policy of USAID not to pay any allowance to GRZ staff for participating in USAID funded activities, apart from normal per diem, appears account for the low priority given to CSH’s planned activities. GRZ’s full participation in CSH planned activities has also been a challenge due to inadequate staff in GRZ units responsible for the activity. These same individuals from the GRZ also support other partners. This development is affecting ownership by GRZ for certain activities implemented by CSH. For example, BCC TOTs planned for 2011 did not achieve the targeted number of GRZ staff trained due to challenges in allowance payments. CSH has been very instrumental in developing the BCP tools and rolling out training across Zambia through a cascading approach to ensure that GRZ staff at all levels become conversant with the behavior change methodologies and approaches. Nonetheless, it appears staff only go through the training once with little follow up or no monitoring of what participants do afterwards with the skills acquired. This approach defeats the process of transferring skills as beneficiaries who do not have the opportunity to practice what they have learned will likely forget about the knowledge and skills acquired. Monitoring & Evaluation Currently, all three GRZ partners lack an M&E system for monitoring their BCC activities. The existing Health Information System (HIS) captures only clinical data without community level indicators. CSH is developing a database for these partners, which is not yet ready. Their BCC units do not have their own M&E plans, protocols and data collection tools. There is also no system for getting feedback from communities on BCC activities. BCC Material Distribution System CSH has churned out myriad English-language campaign products of very high quality that meet international standards. These products range from radio skits, radio spots, TV documentaries, Q&A booklets for health workers, stickers, birth plans, job aids, and chitenges (a wrapped piece of clothing), among others, for all campaigns thus far: Safe Love, Stop Malaria and Mothers Alive. With the exception of TV and radio, which appear to have received higher viewership and listenership, print materials produced by CSH and partners remain invisible in the provinces and districts. CSH has a mandate to work at the national level; it is unable to get most of these materials to the lower levels where they are needed mostly because there is no system of material distribution in GRZ. However, there are existing structures in the three institutions to get materials to the communities. It appears these channels have not been extensively explored by CSH to create a system through which materials can get to the grassroots. What else does GRZ need in order to independently plan, implement, manage, and evaluate national health campaigns? In order to fully realize the intended objective of building GRZ capacity to independently plan, implement, manage, and evaluate national health campaigns, CSHs needs to do the following in the short term: Mentoring Support Revisit the discussion about the role of the embedded staff in the three institutions with GRZ to ensure that embedded staff are actually providing on-the-job training, mentoring, and coaching instead of their present perceived role of only seeing to the successful completion of CSH planned activities. To succeed 17 with this, CSH has to redefine the roles of the embedded staff and develop their job description with the heads of the three institutions. Most importantly, CSH needs to attach a new set of BCC advisors, highly capable and experienced in BCC, who can provide the technical support required by GRZ. The responsibility of these staff should include coaching and mentoring so that they are qualified to improve the skills and strengths of GRZ staff in BCC. GRZ Ownership CSH needs to take a back seat role in the implementation of campaign activities and put GRZ in the forefront in all BCC activity design, implementation, managements and evaluation. CSH also needs to consult with relevant stakeholders. CSH further needs to facilitate GRZ to spearhead the planning. CSH needs to allow GRZ to design all campaigns in their own style and not be prescriptive. This will enable GRZ to design campaigns and programs that they can manage and will engender ownership of any interventions supported by CSH. Material Distribution System CSH should explore creative ways and channels within MOH, NAC, and NMCC to ensure campaign materials reach the intended audience in a timely manner by supporting GRZ to create a system out of the existing structures from national level to community level. Currently, GRZ has the necessary structures for this material distribution but this is not working as a system. CSH should support GRZ to develop a material distribution system out of their current structures, starting from the national level to the community level. This would ensure that materials developed actually get to the community level and to the intended audiences. Monitoring & Evaluation CSH need to help GRZ to develop an M&E system for IEC/BCC programs. Currently, they have been trained on conducting formative research, which is very important for GRZ. CSH should provide technical support to GRZ on M&E, probably by assigning highly competent M&E professionals to mentor them on M&E. Support from these experts should include developing BCC M&E Plans, M&E protocol, data collection tools, databases, setting up the M&E teams, and ensuring a complete and running M&E systems for each of the three GRZ components. Resources GRZ has no resources even to practice what they are learning. Rather than CSH designing campaigns, CSH should be allowed to sub grant GRZ whenever they have been able to build the capacity of GRZ to implement a campaign, enabling GRZ to use the funds to design the campaign whilst receiving mentorship from CSH. This would ensure that GRZ receives not only the theoretical skills but also the experience in implementing to be able to sustain BCC activities. To what extent is the project adapting to the current changes in the GRZ structures particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re-organization of the National HIV/AIDS/STI/TB Council? CSH is not clear with their mandate in relation to the new Ministry of Community Development, Mother and Child Health (MCDMCH). The project currently has no direct communication with this new Ministry and only communicates to the Ministry through the MOH. CSH has three main challenges: Roles of MCDMCH There has not been any formal communication on the structure and role of the MCDMCH. The project has also not been given any document detailing the structure and mandate of the new ministry. The project is therefore not very sure of the exact role of the ministry to its work . Currently, what the project knows about the new ministry is that it’s to take over the mandate of the MOH at the District and Community level but was not able to identify any document that stipulates this mandate.    18 Design of CSH Project CSH is designed to work as a national program. This means it is expected to operate at the National and Provincial levels, with no responsibility to the District and Community level. Given the currently known responsibility of the MCDMCH, which is to take over the responsibilities of MOH at the district and community levels, the Ministry’s geographical area of operation falls outside CSH’s jurisdictional mandate. Formal Communication on Responsibility USAID has not formally communicated to CSH that it is assigned responsibilities with the new ministry. This has prevented the project from communicating directly to the new ministry and can only communicate to them through the MOH. This is not enabling CSH to provide any support to them either technically or financially. The only support provided to this ministry currently is the support provided to staff who have moved from MOH to MCDMCH. 3.3 QUESTION CATEGORY III: EVIDENCE BASED PLANNING In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? In terms of evidence-based planning and implementation, CSH has noteworthy achievements. At the same time, there are some gaps that could be bridged. The following are strides that CSH has made to enhance the use of evidence in planning and implementation of IEC/BCC activities: Behavior Centered Programming (BCP) There are strong mechanisms in place to assist in developing products and tools relating to IEC/BCC activities. These are mainly part of BCP, a process that consists of conducting project planning and strategy development, formative research, materials development, pre-testing, partnership development, and activities/communications development. Formative research and pre-testing during the BCP process are thus ways in which evidence is collected to inform the development of communications products. In order to strengthen BCP and enhance the capacity of GRZ partners in this area, CSH developed a trainer’s manual and participant’s manual with the purpose of supporting the training of program designers and managers of GRZ (MOH, NAC, NMCC) in Behavior Change and BCC. In addition to the primary GRZ partners the package was also meant for nongovernmental organizations (NGOs), private volunteer organizations (PVOs), and USAID partners. After development of the training manuals, BCP training was carried out for GRZ partners. Before the BCP training was conducted CSH carried out an assessment of MOH using the BCP Capacity Assessment Index. The BCP Capacity Assessment Index for NAC and NMCC is planned for January 2013. The MOH assessment showed that MOH was weak in BCP with an overall score of 54 percent (CSH, 2012). Figure 11: Survey results on CSH collection of information from end users 19 Online survey results indicated that over half of respondents (52%) strongly agreed that CSH collects information from end users to inform the design of various IEC/BCC materials and campaigns. A further 33 percent agreed with the statement. All CSH primary partners interviewed during the KIIs indicated their capacity has been built by CSH on BCP and this has enhanced their ability to plan their IEC/BCC activities. They have been empowered in identification of key behaviors, conducting a behavioral analysis, and clearly defining a responsive strategy and program through the BCP training. For example, formative research, which is part of BCP, was carried out by CSH and GRZ partners for the HIV, Malaria and MNCH Campaigns. Development of BCC Guidelines for Pretesting and Evaluating Communications Materials MOH, NAC and NMCC, in partnership with CSH, prepared a set of guidelines for pretesting and evaluating communication materials. GRZ partners expressed that these gudielines are assisting them in prestesting and evaluating IEC/BCC activities. Development of Technical Working Group Terms of Reference CSH assisted in the development of terms of reference for the NAC IEC/BCC TWG and one of the functions of this group is to aid in the planning, monitoring, and evaluation of IEC/BCC activities. One of the functions of the National Health promotion TWG, under the MOH, is the provision of guidance and systems to monitor and evaluate IEC/BCC materials and programs. The Malaria IEC/BCC TWG under NMCC is also tasked with assisting the NMCC health promotion section to identify, plan, monitor and evaluate IEC/BCC activities. KIIs revealed that although the TWGs are not meeting as often as they should, these TORs have assisted in improving the composition of the TWGs and in clarifying their roles. This is therefore a positive step in terms of enhancing the use of evidence in planning and implementation should the TWG fulfill their functions. M&E Training CSH developed an M&E training guide and has conducted M&E training to build capacity of 20 GRZ staff at central, provincial and district levels. The training included evaluation design and M&E indicators. KIIs revealed that this training has helped the GRZ staff to be more familiar with evaluation principles relating to IEC/BCC activities. Feedback from target audiences is thus adequately gathered during the design stages of the development of IEC/BCC campaigns and development of communication tools, as all relevant stakeholders are consulted in the development stage. KIIs revealed unequivocally that CSH has greatly assisted MOH, NAC and NMCC in planning their IEC/BCC products and tools. All three GRZ institutions expressed that their capacity had been enhanced in terms of planning and in identifying ideal behaviors, the gaps in current behaviors, and ensuring that their campaigns address those gaps. They derived these benefits from the various trainings conducted by CSH and not from the embedded staff. Other than for initial planning, there is a lack of evidence collection through well-established feedback mechanisms that track the reach of messages, the reactions of the intended audiences to the messages, and the outcomes of communication programs. Implementation of IEC/BCC Evaluation at CSH M&E plans are in place for the STOP Malaria Campaign, Safe Love Campaign, Safe Motherhood Campaign and SMGL Campaign that aim to track campaign inputs, processes, exposure to campaigns (Campaign Outputs) and outcomes (Knowledge, Attitudes, Self-efficacy, and Behavioral). However, KIIs show that although tracking exposure to and outcomes of IEC/BCC activities are planned for, this is yet to be done. This finding is also supported by the DQA that was carried out during this evaluation. Further, there are no plans at CSH for periodically revising the materials produced based on evidence gathered. There is also no established system in place at CSH to reduce errors in collecting information on the number of people reached through the different campaigns.    20 It is important to note however that in April 2012 CSH conducted a rapid survey of the Safe Love campaign. This survey aimed to assess exposure to the products and a report, which included recommendations on how the campaign could be improved, was produced in May 2012. This report was disseminated to NAC as the main partner on HIV & AIDS, although NAC itself was not involved in conducting the survey due to time constraints and having limited staff. The findings have so far not been utilized in improving the implementation of the Safe Love Campaign. Another rapid survey of the YHM television program was conducted and the findings were commendably utilized in drawing up a strategy for the YHM Multi-Media Program. The M&E team also developed a draft comprehensive Safe Love impact evaluation plan. The evaluation of the campaign, planned for June 2013, will assess the impact of Safe Love on the target audience’s knowledge, beliefs/attitudes, self-efficacy, interpersonal communication, intentions, and behaviors related to having multiple sexual partners concurrently and using condoms. That said, there have not been any evaluations of the campaigns that are currenlty running. Collection of evidence for evaluation of campaigns remains a challenge as there are currently no effective mechanisms to track exposure and outcomes of campaigns and their effectiveness in promoting behavior change. Feedback is anecdotal and seems to be obtained without deliberate effort. For example, it was revealed that a pastor had written to CSH to complain about a certain billboard that he thought was inappropriate and changes were made to the message based on this complaint. Whilst this was a good development such information should ideally be collected systematically and deliberately. Evaluation Plan for MOH, NAC and NMCC IEC/BCC Activities No GRZ partners have plans for evaluating the IEC/BCC activities they carry out and neither do they have any plans for revising the materials based on feedback and evidence obtained. The reporting systems for IEC/BCC activities in these institutions are integrated into the organizations’ overall M&E system and data collected is mainly restricted to numbers of IEC materials produced and distributed and occasionally exposure to the different messages. This is the case with campaigns during Child Health Week, for example, where such information has been collected by MOH and used to target different communities with the most effective communication channels. However, exposure tracking is limited and not implemented across all campaigns by MOH. Further, at all three institutions the data collected is not detailed enough for evaluation purposes. The flow of information is usually one way, proceeding from central level to community level through the provincial and district structures. Thus feedback is not obtained from target audiences to aid revision and improvement of campaigns. Online survey results support the finding that GRZ does not have adequate capacity to evaluate IEC/BCC activities as 48 percent of respondents’ rated GRZ ability to evaluate IEC/BCC activities without any support as medium while 35 percent rated it as being low. Only 9 percent thought such capacity was high. All GRZ partners, therefore, fall short of adequately obtaining evidence that could be useful in improving their communication products and tools. Figure 12: GRZ’s Ability to Evaluate IEC/BCC Activities 21 M&E Systems for BCC programs at MOH, NMCC and NAC A functional M&E system is required to collate evidence useful in implementation of IEC/BCC ativities. The BCP Capacity Assessment Index for MOH carried out by CSH revealed that M&E systems for BCC programs did not exist and that this was the weakest area in terms of IEC/BCC programming. The DQA carried out during this evaluation confirmed these findings and went further to show that the same findings were applicable to NAC and NMCC. One of the recommendations of the MOH BCP Capacity Assessment Index report was the need for an integrated M&E system that captures and stores information and monitoring data on all BCC programs and interventions. While M&E training has been conducted by CSH, the challenge of having no system in which practical application occurs means that the knowledge gained is not adequately utilized. Online survey results show that there is a reasonably strong perception (61%) that GRZ has a system for assessing feedback from end users on their IEC/BCC activities. However, KIIs and the DQA revealed that such systems do not exist at NAC, NMCC and MOH. It is therefore expected that such systems would be there, while in reality they are not. It is evident through KIIs, mini survey results, and the DQA that feedback is adequately collected during the planning and development stages of the different communication campaigns, both at CSH and GRZ partner levels, through formative research and pre-testing of communication material. However, once campaigns are launched the systems in place to collect feedback that can be used in improving the campaigns are weak at CSH level and non-existent at GRZ partner level. 3.4 QUESTION CATEGORY IV: SUSTAINABILITY Are IEC/BCC activities likely to continue without further USG investments and if not, what investments or approaches would better promote sustainability?    Taking cognizance of the multidimensional nature of sustainability, a comprehensive analysis of the design, implementation, management, and evaluation aspects of CSH was conducted to ascertain whether or not CSH-supported IEC/BCC activities would continue without further USG support. In the context of the current evaluation, GRZ’s ability to continue IEC/BCC activities was assessed in terms of sustained campaigns at the level that CSH supports. National Level Sustainability Basically, strategic networking through the TWG on IEC/BCC activities and the placement of BCC staff in GRZ institutions namely, MOH, NAC, and NMCC constitute the national level model for the promotion of sustainability. This approach is premised on the assumption that technical competences in IEC/BCC will be built among GRZ staff who will eventually continue to apply the acquired skills beyond the life of CSH. Interviews with key informants revealed that while the idea was logical, the technical competences of GRZ staff were perceived to be higher than those of the staff CSH had placed in some of the GRZ institutions. Consequently, the supposed technical advisors have ended up working as support staff or mere links between CSH and GRZ partners for programmatic convenience. It was clearly established that the national model for building sustainability in its current form was not satisfactorily Figure 13: GRZ has a system tor assessing feedback from end users on IEC/BCC activities    22 21.70% 34.80% 26.10% 8.70% 8.70% Confidence levels on whether GRZ will continue IEC/BCC activities without USG support Low Medium High Very High Don’t Know Figure 14: Confidence levels on whether GRZ will continue IEC/BCC activities without USG support sustainable. This can be deduced from the figure below, which shows that 35% of respondents rated their confidence levels in GRZ’s capacity to continue with IEC/BCC activities beyond CSH as medium, while only 26% said their confidence levels were high. This means stakeholders do not have confidence in GRZ’s ability to sustain BCC activities. Community Level Sustainability At the local level, CSH has created Safe Love Clubs and SMAGs whose primary function is to carry out grassroots campaigns on HIV/AIDS and MNCH, respectively. This local level ownership building is viewed as a strategic approach for sustainability of IEC/BCC activities beyond the life of CSH. Findings revealed that while these clubs are functional in some places, there are some communities in which they are no longer working, such as Kamanga Township in Lusaka. Additionally, the community facilitators who were trained to mobilize these groups were put on six-month contracts which expired around June 2012. They indicated that contracts will be renewed in 2013. While the role of facilitators is important, there is no evidence of strategies to sustain the payment of monthly allowances to them once USG funding ends. From the community level sustainability standpoint, all functional clubs revealed that resource constraints were a major drawback in their work. However, they did indicate that they will continue to exist as groups beyond CSH. CSO partners Apart from GRZ, CSH has built the capacity of CSOs through building physical and IT infrastructure. Infrastructural improvements have been made to resource centers under Afya Mzuri and the CHAMP 990 Talkline. Respondents revealed that these centers will continue because they were in existence well before CSH was launched. Evidence from site visits shows that support from CSH has actually enhanced the work of partner CSOs. Project Management The evaluation of the CSH looked at the extent to which the existing project management and coordination mechanisms support provided by CSH to GRZ had become entrenched. From KIIs, the evaluation established that while CSH has been conducting some skills building programs, GRZ has not necessarily taken a lead in developing, implementing, managing and evaluating national IEC/BCC campaigns under CSH. Partners view this as a fundamental departure from the design of the project. Project Evaluation The BCC M&E system of GRZ has not been set up even though CSH has been providing M&E support to GRZ; thus, there is little to be sustained. Even though GRZ has started conducting formative research before the design of their projects, it’s not clear if they can continue without CSH since they may not have the resources. However, GRZ staff indicated that they have the capacity to conduct formative research. This knowledge is sustainable but beyond the research, GRZ would not be able to effectively monitor or evaluate their BCC activities. Further, CSH has not developed the Sustainability Assessment Matrix (SAM) as a tool for continuous assessment of GRZ’s capacity to continue with IEC/BCC activities when USG support is no longer available. The implication is that CSH has not been able to systematically establish the extent to which the project is on course, which would be the basis for devising strategies that would promote sustainability. 3.5 QUESTION CATEGORY V: RESULTS TO DATE To what extent is the activity on track to achieve its intermediate results and meet its life of activity targets? 23 The extent to which activities are on track to achieve their intermediate results and meet their life of project activity targets are discussed below according to thematic areas and major components within each thematic area. The discussion is based on the three launched campaigns (Malaria, MNCH and HIV/AIDS). The other two campaigns have not yet been launched and are clearly not on track to achieving their intermediate results. Stop Malaria Campaign The CSH performance monitoring plan (PMP) does not have quantitative targets for malaria indicators. However, the baseline indicates that at the start of CSH about 49.9% of children under five years of age slept under ITN. The mini survey found that this figure has reduced to 27.7%. The major reason given as to why children did not sleep under ITN was the warm weather. The STOP Malaria campaign started last year with training events and media activities, but various adverts are still under production. The campaign is aimed at stopping malaria through promoting testing for malaria before treatment, ITN usage, and IPTp. The mini survey found that about 50.4% of the households had one or more ITNs, while almost half of the population (49.2%) was without ITNs. About 45.5% of urban households have ITNs compared with 61.3% in rural areas. A large proportion of the population (69.0%) also indicated that their household has not been sprayed in the last 12 months. The 29.3% who have their households sprayed to prevent mosquitoes included 28.1% of households in urban areas and 32.0% in rural areas. Of the children under five in the 50.4% households with ITNs, only 55% slept under ITN the night before the survey. A large proportions (44%) of children under five years old living in households with ITNs still do not sleep under net. Additionally, about 32% of adults living in households with ITNs did not sleep under the net the previous night. Some 68% percent of adults in households with ITNs however did sleep under net the previous night. Mothers Alive Campaign The Mothers Alive campaign aims to reduce maternal mortality through a number of goals. Key among them is the promotion of ANC services. The project had a target of reaching some 30% or the population with messages on mothers alive through radio and some 15% through TV. The mini survey however found the MNCH messages only reached 16.9% of the population through radio and 14% through TV. Whiles the TV campaign target is on track with barely a percentage short of the target, the radio campaign is not on track, falling short of about 13.1% The mini survey found that a significant proportion of the women who gave birth within the last 12 months went for ANC services, averaging 87% of pregnant women, with some 13% failing to attend clinic for antenatal services. Figure 15: Percentage of children <5 in household with ITNs sleeping under net Figure 16: Percentage of respondents with ITN in their household and those who benefited from IRS.    24 However, most pregnant women went for ANC services when their pregnancy was advanced. Only 17.2% of pregnant women went for ANC before the third month of pregnancy. The majority of pregnant women (37.1%) went for the first time when their pregnancy was in the third month. Some 45.7% of the pregnant women, however, went when their pregnancy was four months or older. A significant proportion of the pregnant women went for ANC services up to the recommended frequency. The baseline from the CSH PMP indicates that in 2007, 60.3% of women went for ANC at least for times before birth. The mini survey however shows a decline to 57.1%. Some 34.3% however went three times before delivery. Only 8.8% went for ANC two or less times before delivery. The FGD also showed that community members are not aware of birth plans. All the women who were pregnant and those who gave birth in the past few years did not use birth plans and are not planning to use them because they are not aware of them. Safe Love Campaign Activities The Safe Love campaign aims at targeting three main drivers of HIV pandemic in Zambia. These are: to reduce the rate of MCP, increase condom use with regular and casual partners, and promote the utilization of PMTCT services among HIV positive expectant mothers. The survey found that the Safe Love radio campaign is on track to achieving its objectives. The project targeted reaching 50% of the population with through radio by the end of 2012. The mini survey found that this target has been exceeded as 52% of respondents indicated they have head of the campaign on radio. The TV campaign is however not on track. Even though the project intended reaching 90% of the population through TV by the end of 2012, only 46.7 were reached, falling short by about 43.3% The survey results indicate that the Safe Love TV advert is not having any effect on men’s practice of MCP because 19.1% of respondents who have not seen the adverts practice MCP compared to 22.2% of those who have seen the advert. Thisfurther suggest that there are other factors influencing the reduction of the practice of MCP among men which are more effective than the Safe Love TV adverts. The radio advert, on the other hand, is having a very strong effect on men. About 10.8% of men who listened to the radio advert on Safe Love practice MCP compared to 28.3% of those who have never heard the advert on radio. Figure 17: Online survey results for questions pertaining to ANC visit frequency and ages for expectant mothers 25 Table 3.2 Effect of Safe Love Campaign on practice of Multiple and Concurrent Partnerships Percentage of males’ and females’ level of exposure to the Safe Love campaign and practice of multiple concurrent partnerships. Sex Saw Safe Love Advert on TV Has Not Seen Safe Love Advert on TV Heard Safe Love Radio Advert Has Not Heard Safe Love Radio Advert Male 22.2 19.1 10.8 28.3 Female 3.9 9.8 6.7 7.2 Conversely, the TV adverts are having a strong effect on women. Only 3.9% of women who have watched the Safe Love advert on TV practice MCP compared to 9.8% of those who have not seen the advert on TV. The radio adverts, however, does not seem to have significant effect on women, with 6.7% of women who listen to the radio advert practicing MCP compared to 7.2% of those who have not heard the advert on radio. Both the TV and radio adverts are achieving results in encouraging people to test for HIV and receive their test results in the urban areas, with radio having a higher effect than TV. In urban areas, 63.2% of those who watched the Safe Love advert have tested and received their test results compared to 57.4% of those who have never watched the advert, a 5.8 percentage point difference. The difference is even more significant with radio, which recorded 69.4% of listeners testing and receiving their test results compared to 52.4% of those who have never heard the advert, a 17 percentage point difference. Table 3.3 Effect of Safe Love Campaign on Testing for HIV and Receiving Test Results Percentage of respondents’ exposure to the Safe Love campaign and practice of testing for HIV and receiving test results in Urban and Rural localities. Location Seen Safe Love Advert on TV Not Seen Safe Love Advert on TV Heard Safe Love Radio Advert Not Heard Safe Love Radio Advert Urban 63.2 57.4 69.4 52.4 Rural 42.9 60.3 57.1 60.6 Both TV and radio adverts are having no effect in the rural areas on HIV counseling, testing and receiving test results. About 60.3% of rural residents who have never seen the Safe Love adverts on TV tested for HIV and received their test results compared to 42.9% of those who have watched the adverts. Also 60.6% of rural dwellers who have never heard the radio advert have tested for HIV compared to 57.1% of those who have heard the adverts on radio. This also indicates that in the rural areas there are factors influencing the testing for HIV more effectively than the Save Love campaign. Table 3.4: Effect of Safe Love Campaign on Knowing Sexual Partner’s HIV Status Percentage of respondents’ exposure to the Safe Love campaign and knowledge of sexual partner’s HIV status disaggregated by residence. Location Seen Safe Love Advert on TV Not Seen Safe Love Advert on TV Heard Safe Love Radio Advert Not Heard Safe Love Radio Advert Urban 46.2 24.6 41.2 35.4 Rural 71.4 39.7 47.6 36.4 The Safe Love campaign is contributing significantly to people knowing the HIV status of their sexual partners both in urban and rural areas. In urban areas, about 46.2% of those who have seen the adverts know the HIV status of their partners compared to 24.6% of those who have not seen the advert. A similar effect is experience with radio both in urban and rural areas. The most significant effect is with TV    26 viewership in rural areas. About 71.4% of rural residents who watch the Safe Love TV adverts know the HIV status of their sexual partners compared to 39.7% of those who have never watched the TV adverts, recording a 31.7 percentage point difference. The Safe Love TV adverts are having marginal effect on condom use in urban area but absolutely no effect in rural areas. In rural communities, a significant proportion (40%) of those who have not seen the Safe Love adverts indicated they use condoms correctly and consistently. This is significantly higher than those who have seen the Safe Love adverts on TV, of which just about 25% use condoms correctly and consistently. Table 3.5: Effect of Safe Love Campaign on Condom Use Percentage of respondents’ exposure to the safe love campaign and consistent use of condom in Urban and Rural localities. Location Seen Safe Love Advert on TV Not Seen Safe Love Advert on TV Heard Safe Love Radio Advert Not Heard Safe Love Radio Advert Urban 35.3 33.3 42.9 28.6 Rural 25.0 40.0 41.7 28.6 The radio adverts are, however, influencing condom use both in rural and urban areas, recording 42.9% and 41.7% of urban and rural listeners, respectively, using condoms consistently compared to 28.6% of both rural and urban residents who have never heard the campaign on radio using condoms consistently. Table 3.6: Effect of Safe Love Campaign on Demand for Condom Use at All Times Percentage of respondents’ exposure to the safe love campaign and practice of demanding that sexual partner uses condom at all times disaggregated by residence. Location Seen Safe Love Advert on TV Not Seen Safe Love Advert on TV Heard Safe Love Radio Advert Not Heard Safe Love Radio Advert Urban 42.9 31.2 70.0 20.0 Rural 50.0 40.0 41.7 50.0 Exposure to the Safe Love campaign on TV is also influencing both rural and urban dwellers to demand that their partners use condoms at all times. About 42.9% of urban residents who have seen the Safe Love advert would demand that their partners use condoms at all times compared to 31.2% of those who have not seen the advert. In rural areas, the condom use was demanded at all times by 50% of those who have seen the adverts on TV compared to 40% for those who have not seen the advert. Radio is also having a very significant effect on people demanding that their partners use condoms at all times in the urban areas, but not in the rural areas. In urban areas, 70% of those who heard the Safe Love radio advert demand consistent condom use but only 20% of those who have never heard the adverts on radio do so. However the radio advert is not having any effect in rural areas in influencing people to demand condom use at all time. Whiles 50% of those who have not heard the radio advert would demand consistent condom use, only 40% of those who have heard the adverts on radio would demand condom use. 27 Figure 18: Chart on Beliefs and Circumcision & HIV by exposure to Safe Love campaign on radio About 72.5% and 62.7% of urban and rural residents respectively have no doubt that that being circumcised could reduce the chances of contracting HIV. However, the Safe Love campaign is significantly contributing to this in urban areas. In urban areas, people exposed to the campaign on radio and who believe circumcision reduces the chances of contracting HIV are slightly higher (72.9%) than those not exposed to the safe love radio campaign (72%) but this difference is not significant. This is not so in rural areas, where more of those not exposed to the campaign (66.7%) believe that circumcision can reduce the chances of contracting HIV compared to those exposed to the campaign (59.7%). However, in rural areas, many more of those exposed to the campaign (35.7%) have moved from thinking circumcision cannot reduce the chances of contracting HIV to not being sure whether circumcision can reduce the chances of contracting HIV. A significant proportion of those not exposed to the campaign (15.2%) remain resolute that circumcision cannot reduce the chances of contracting HIV. Figure 19: Chart on Beliefs on Circumcision & HIV by exposure to Safe Love campaign on TV It would appear that Safe Love adverts on TV are making significant achievement in both rural and urban areas, influencing beliefs about circumcision. Higher proportions of those exposed to the adverts on TV, both in urban and rural areas believe that circumcision reduces the chances of contracting HIV compared to those not exposed to the campaign. In rural areas, 85.7% of those exposed to the adverts are sure of the effect of circumcision on likelihood of contracting HIV but about 30.9% of those not exposed are not sure whether circumcision has any relationship with chances of contracting HIV. This group of people, when exposed to reliable and    28 convincing information, is most likely to make the decision of believing that circumcision can reduce the chances of contracting HIV. To what extent are the indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)? What improvements can be made to improve data quality and better capture progress? A complete DQA was conducted on the CSH M&E system to identify the system’s effectiveness and whether it’s tracking the right indicators with the right tools and efficiently reporting results. Eight data quality criteria were assessed, including validity, reliability, precision, integrity, timeliness, accessibility, confidentiality, and data security. Each indicator under a criterion can score a maximum of four points. The findings from the DQA are presented below: No Indicators Score Comment Validity Score: 8.5 1 Are the right indicators being measured? 2.5 The PMP has sufficient impact and outcome indicators but there are no output indicators to most of the outcome indicators. There is the need to set more output indicators. The plan should ensure that each impact indicator is achieved by one or more outcome indicators and each outcome indicator is achieved by at least one output indicator. 2 Are the indicators clearly defined? 2.0 The impact indicators taken from USG documents (such as PEPFAR Hand Book) had definitions but the outcome indicators and the few output indicators set by the project have no definition. Each indicator needs to be defined. 3 Do the data collection tools disaggregate the data? 4.0 The various data collection tools effectively disaggregate the data in various dimensions including gender, month, age, indicator, title, location, contact number, type of training, topics treated, and materials distributed during training. Reliability Score: 10.3 4 Is the same method being used to collect and analyze the same data? 4.0 The design proposes the same method of data collection and analysis for data of the same kind. 5 Are different groups collecting the same data using the same data collection tools? 4.0 The same data collection tools are used by Facilitators across Districts, Provinces and Partners to collect data on the same indicator. 6 Is the database efficient enough to produce accurate data? 1.5 The project uses an SQL database which is Web-based but is not designed to input the raw data. The M&E team therefore has to manually tabulate data on each indicator and input the aggregated figures into the database. Since the data is already aggregated before entry, the system is not able to disaggregate data for location, cadre, date etc. 7 Is there an established system to reduce 0.8 The project staff review the data and manually calculate what to input into the database. However the Club members who conduct the education at the community level do not have data collection 29 errors? tools. They have to memorize the number of people they are educating (disaggregated by gender, and topic discussed) whenever they carry out education until they meet with the facilitator in two weeks and ‘narrate’ the data to the facilitator for him/her to record on the facilitator’s data collection forms. With each facilitator overseeing three or four groups, these Facilitators aggregate the data for these clubs and forward to the District level facilitator who also aggregates before it gets to CSH. By then, it’s too late to identify any error. 8 Are the data collectors trained on the tools and protocols to enable them to collect reliable data? 0.0 The data collectors who are members of the Safe Love Clubs, Radio Listening Clubs etc. do not have data collection tools to use and could therefore not be trained. Precision Score: 7.5 9 Do other methods of counting result in the same quantities with what is reported in the database? 1.5 There are data collection tools (for facilitators) and data collection protocols in place. However, since the data collectors have not been trained and they have no data collection tools, the data in the database is less than satisfactory. 10 Do the data collection tools eliminate double counting? 0.0 The data collection tools for community level education don’t make room for the status of the person being educated (whether the person has been educated on the same topic before) and therefore people are likely to be double counted. Also the training data collection tools do not make room for status and therefore people trained are likely to be counted in multiples if they participate in more than one training. 11 Are data precise enough to enable decision making at policy and operational levels? 2.0 Data collected from community level is disaggregated at two levels before its gets to CSH and therefore difficult for it to be used for operational decisions. However there has been some research that can be used for decision making. An examination of the Summary Report of Pretesting various IEC/BCC materials showed that some of the reports were precise, rating various indicators quantitatively. Others, though, used expressions like some and most, which are ambiguous and could therefore not be used for decision. 12 Are the data managers trained to enable them effectively manage the database? 4.0 Staff of the M&E team spent a month receiving training to enable them to effectively manage the database. Also whenever there is an upgrade of the database, M&E staff are oriented to keep them up to date with database management. Integrity Score: 8.0 13 Are there measures in place to ensure that management cannot manipulate the data? 4.0 The database has various levels of permission. The M&E Director has read, write and edit permissions whiles the Chief of Party (COP) and Deputy Chief of Party (DCOP) have only read permissions. This means that management cannot manipulate the data. 14 Is the data management system such that only the Other M&E team members have read and write permission but not edit or delete permissions and can therefore not manipulate the data once it’s entered. The IT team managing the server only has read    30 right person(s) can read, add, delete, and/or edit the data? 4.0 permission and therefore cannot manipulate the data. Accessibility Score: 4.5 15 Is the data available to management and other staff (including USAID staff) for decision making? 3.5 Data is available for management decision making because CSH management have read permission and can therefore access the data wherever they are and whenever it’s required as long as they have internet access. The data is also available for some USAID staff. The USAID Contract Office Representative (COR) for CSH, Daniel Vershneider, has password and read permission to the data. This means he can access the data but cannot manipulate it. The USAID M&E Team does not have access to the data and that could prevent other USAID staff from getting information on CSH since the CTOR is not responsible for supplying data to other staff. 16 Is the process of analyzing and retrieving the data efficient to ensure that management and staff can have easy access to various analyses promptly? 1.0 While staff have access to the data, they cannot easily retrieve it because the system is entirely web based. With slow internet speeds prevalent, it is very difficult for them to access the data. Moreover the database does not allow the staff to conduct any analysis from the data but can only view the data. Timeliness Score: 8.0 17 Does the time of reporting make data available for CSH reporting to USAID? 4.0 The M&E protocol document clearly states that data should be submitted by the 10th of each ensuing month. Data is therefore available on a monthly basis and should be available for reporting to USAID. 18 Does the frequency of reporting make data available for GRZ and CSH management decision making at the right time? 4.0 The M&E protocol document clearly states that data should be submitted by the 10th of each ensuing month. Data is therefore available on a monthly basis and should be available for GRZ and CSH management decision making at the right time. Confidentiality Score: 7.0 19 Are the individuals reported on protected to ensure information about them does not get into the public domain? 4.0 The organization seeks IRB approval before conducting research and respondents’ consent is sought before the interviews. The name of individuals who may provide sensitive data is also protected, e.g. Helpline Counselors do not take the real names of people who call￾in to protect their identity. 20 Are hard copies of the data kept in a safe place and secured to prevent the leakage of the identity of client and respondents? 3.0 Hard copies are stored without the names of the respondents, thereby preventing their names from leaking to the public. However some hard copies had been kept in an office unsecured. Security Score: 4.0 31 21 Is data saved in multiple locations to ensure they are secure in all eventualities? 1.0 The data is saved in a single location. This means if there is a disaster or damage to the storage device, all data could be lost. 22 Are hard copy data properly stored to prevent it from being misplaced? 3.0 Hard copy data is safely stored in a store room but some were found in the office and not properly protected and could therefore be misplaced. 23 Is there backing storage for all data? 0.0 The project has no backing storage for their data. The M&E team would therefore have no way of tracing the data should a disaster strike the server hosting the data. What are the challenges to implementation and what can be done to improve the chances of the activity achieving its intended results and meeting its life of activity targets? CSH Project Design CSH is designed as a national level project and is thus not expected to work at the community level. This poses several challenges for the organization. Firstly, ZISSP is supposed to work with CSH to deliver communication messages to the community level, but ZISSP has limited resources to do this effectively. Furthermore they are restricted to working only in certain geographic areas, which limits their reach throughout the country. Secondly, based on needs that were identified by the project, CSH went ahead with capacity building at provincial and sometimes district levels. Part of this approach involved funding local CSOs who then work with CBOs at community level to implement activities such as Safe Love. However, there is little reporting to CSH from these CSOs and CBOs on IEC materials distribution, and the quality of community level BCC activities reporting is very low. This means it is difficult for CSH to monitor and evaluate the community level interventions since it has no direct contact with the CBOs. It can be argued that if CSH is providing funding to these organizations then it should also take responsibility for monitoring and evaluating them in some way in order to demonstrate value for the investment placed in these organizations. In one occasion ZISSP developed community based materials for use in the community and had expected CSH to reproduce them on mass scale for distribution in the communities. CSH does not have all the needed resources to meet the request by ZISSP and had concerns about the kind of materials that should be used for the needed impact to be achieved. Lastly, the new MCDMCH works at district and community levels, while MOH is now working at central and provincial levels. Thus, in order to reach the district and community levels in the health sector, CSH will now need to work with MCDMCH. Its current mandate therefore needs to be examined where community level work and working with MCDMCH are concerned. Relationship with MCDMCH The relationship of CSH with the newly formed MCDMCH needs further clarification on exactly how the two parties should work together. Part of the ambiguity lies in the fact that the new ministry is still in the process of more clearly defining its role in the health sector. It was found that one staff member from the Health Promotion Unit at MOH has been moved to MCDMCH and serves as the contact with CSH. Through this contact there has been involvement of MCDMCH in various MNCH activities. However, CSH works with MCDMCH through their relationship with MOH. Thus there are no formal ties between CSH and MCDMCH and CSH is unclear as to how much support they should give MCDMCH given that this is not officially their mandate. GRZ and USAID need to discuss and agree on how this    32 relationship should work and whether to formally bring MCDMCH on board as a fourth GRZ partner for CSH. It was suggested during KIIs that an MOU should be signed between CSH and MCDMCH to formalize and clarify their working together. Funding to GRZ CSH support to GRZ has mainly been in the area of building technical capacity of staff and providing communication materials for further dissemination. CSH has no mandate to provide any funds to MOH to assist with IEC/BCC activities. As discovered during KIIs and the online survey, one of the major challenges MOH faces is inadequate funding. That means that much of the technical capacity built is underutilized. The CSH program should therefore consider giving some funds to MOH to enable it to implement IEC/BCC activities and better utilize the capacity it has been given. When MOH, NAC and NMCC are able to demonstrate competence in planning, implementing, managing and evaluating BCC activities and are able to impact positively on the health behavior of the general population through its BCC activities, they would win the confidence of more donors as well as central government and would be able to source more funding to sustain their campaigns. Allowances for GRZ staff The USAID policy of not paying allowances for attendance of meetings and workshops has had a detrimental effect on the level of enthusiasm of GRZ staff with regard to CSH. It was found through KIIs that in some cases trainings and meetings are poorly attended as a result. Of more concern is that this is affecting the sense of ownership of IEC/BCC activities that CSH is involved in since GRZ staff may not be very interested in them due to the issue of allowances. There is a perception that CSH is a big program with a big budget and therefore can afford to pay allowances to GRZ staff. Community level challenges Through CSOs, CSH is disseminating communication messages to community level. The CSOs in turn work through CBOs. The FGDs conducted during the evaluation found that these CSOs face a number of challenges that are linked to the design of the project. For example, accessing IEC materials is a major challenge with most of them having received the materials only once since the campaign started and even then in quantities inadequate for distribution within the communities. Other challenges include lack of identification cards; inadequacy of branded clothing such as t-shirts and chitenges; difficulties related to not having money or transport to go to more remote areas; lack of protective clothing such as raincoats and boots to use in the rainy season; lack of incentives or money for refreshments, especially when going to more remote areas; and IEC materials and messages being hard to understand and explain due to language barriers since the messages are only in English. The structuring of the Safe Love campaign, which CSOs, and therefore CBOs, were told was a six month campaign for the first phase to be followed by a further six months in the second phase, has meant that the CSOs have become less active after the first phase ended sometime in June 2012. They are eagerly awaiting the second phase and although activities on Safe Love continue they are at a much slower pace than before. Compulsory HIV Testing People complain that they are forced to test for HIV when they go for ANC services at the health facility. A group of pregnant women are kept together and provided some information as counseling. After that, all of them are tested compulsorily. Those who refused are often castigated and embarrassed before all others at the ANC. Some are even refused services. Women who go for ANC without their husbands/partners are also scolded and most often turned away without services. Some are made to pay a fine for not being accompanied by their husbands/partners. This practice is wide-spread and most pronounced in rural areas. This leads some women to refuse to go for ANC if they do not want to be tested for HIV or if their spouses are not accompanying them. 33 4.0 CONCLUSIONS Based on the findings of the study, the evaluation presents conclusion on each evaluation question as follows: 4.1 MESSAGE EXPOSURE AND EFFECTIVENESS To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with Information, Education, and Communication/ Behavior Change Communication (IEC/BCC) messages in each of the five health intervention areas (HIV/AIDS, malaria, family planning/ reproductive health (FP/RH), maternal child health (MCH) and nutrition)? The extent to which activities have reached the intended audience in all parts of the targeted geographical areas varies according to the health intervention area. Activities on HIV/AIDS have reached about 63.5% of the urban population and some 56.0% of rural residents, with most of them knowing the name of the campaign despite confusion about the difference between the One Love Kwasila and Save Love campaigns. The situation differs from MCH where only 18.6% and 25.3% of urban and rural residents, respectively, have been reached. Almost all groups interviewed during FGDs did not know the name of the MNC campaign. Community members seem to be somewhat confused about the Malaria campaign. Even though about 71.3% of urban residents and 61.3% of rural residents indicated knowing the Stop Malaria campaign, most of them referred to it as Safe Love Malaria, Malaria consortium, Roll Back Malaria, etc. In fact, even though CSH has not yet started media adverts on Malaria, most of people indicated having known it through TV and radio adverts. This means people are confused between the Stop Malaria campaign and the other malaria campaigns that have been run in the past. The campaigns on nutrition and FP/RP are said to be at the planning stage and have therefore not reached the intended audiences. Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages? The intended audiences are able to recall most of the messages on the Safe Love campaign. Most of these messages were also seen to be appropriate and acceptable by the culture, however the audience did not really understand most of the Safe Love adverts. The adverts left the audience to identify the action point themselves base on their discretion, mostly by ending with a question rather than a call to action. Most people therefore misunderstood the action required. Most people have not seen the Mothers Alive campaign adverts. Most of the intended audience indicated seeing it for the first time. Whiles the campaign’s messages are appropriate culturally, the Change Champion advert was particularly misleading, with most of the intended audience seeing the message to be directed to the government and not to them. The message of the Safe Motherhood Testimonials advert is appropriate, which is well understood by the intended audience, but the characters did not reflect the culture of the entire country since all of the characters were from Eastern Province. 4.2 CAPACITY BUILDING To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own? CSH’s capacity building activities have improved skills and equipped GRZ at all levels, especially at the provincial and district levels, with strategies for conducting research, planning, and implementation of BCC activities. For instance, provincial and district level staff who benefitted from the BCP training are now systematically applying the BCP approach in conducting formative research to understand the key issues influencing behaviors, undertaking audience segmentation, as well as understanding what will influence the intended audience’s behavior to inform their design, planning and implementation of campaigns. These skills have been largely attributed to the innovations used by CSH in engaging GRZ staff and especially the training in    34 BCP. In terms of monitoring for continuous improvement and to access whether the project is on target, very little seems to have been done jointly by CSH and GRZ at all levels. There has been very little on-the-job mentoring and coaching to systematically transfer skills for GRZ staff to be able to implement IEC/BCC activities on its own. What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own? In terms of gaps, CSH’s approaches and strategies have invested little in IEC/BCC systems strengthening within the framework of health service delivery. CSH is currently leading in all activities that are implemented with GRZ, giving the erroneous impression that those activities are implemented by CSH instead of being activities implemented by GRZ with CSH relegated to provision of technical and budget support with monitoring for transparency and accountability. Embedded staff in GRZ are not playing their role of mentoring and coaching GRZ staff in how to conduct formative research, design a campaign, and delivering the right messages to the intended beneficiaries through the most appropriate and efficient channels. Again very little is done to provide skills in IEC/BCC campaign organization, coordination, communication, management and monitoring for continuous improvement. What else does GRZ need in order to independently plan, implement, manage, and evaluate national health campaigns? GRZ needs to be supported by CSH through technical assistance and budgetary support and must be made to lead all campaigns as was initially designed. The current strategy, in which CSH tends to lead in all campaigns, should reverse to GRZ leadership. This will enable GRZ to learn by doing and to be empowered to independently plan, implement, manage, and evaluate national health campaigns. This practice will engender ownership and sustainability for all activities implemented by CSH. To what extent is the project adapting to the current changes in the GRZ structures particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re-organization of the National HIV/AIDS/ sexually transmitted infection (STI)/ tuberculosis (TB) Council (NAC)? CSH is not clear with their mandate in relation to the new Ministry of Community Development, Mother and Child Health (MCDMCH). The project currently has no direct communication with this new Ministry and only communicates to the Ministry through the MOH. The issue is further compounded by the lack of formal communication on the structure and mandate of MCDMCH, especially with regard to CSH’s role at the district and community level; CSH only has a national mandate. 4.3 EVIDENCE-BASED PLANNING In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? CSH has greatly assisted MOH, NAC and NMCC to use evidence-based approaches to plan their IEC/BCC products and tools through the trainings that they have been given in M&E and BCP, the BCC guidelines that were developed and TWG terms of reference. However, gaps still exist, including a lack of follow through in implementing M&E plans for the various campaigns beyond the launch phase at CSH; lack of plans for collecting feedback systematically and for utilizing such feedback to improve on products and tools at GRZ level; and lack of M&E systems for IEC/BCC activities that would enable effective and efficient collection of evidence at GRZ level. 4.4 SUSTAINABILITY Are IEC/BCC activities likely to continue without further United States Government (USG) investments and if not, what investments or approaches would better promote sustainability? 35 Support to GRZ: The national level GRZ model for sustainability adopted by CSH does not so far show evidence that the project is on course in working towards future continuance of IEC/BCC activities without further funding from USG for two main reasons. The first reason is the apparent lack of investment in IEC/BCC systems strengthening within the framework of health service delivery. The second reason is that, contrary to the project design, current project implementation and management arrangements reveal that CSH has taken a lead in carrying out IEC/BCC activities. Ideally, if CSH allowed GRZ to lead while providing technical and budget support with close monitoring for transparency and accountability, IEC/BCC activities would eventually receive priority attention from policy makers. This kind of buy-in has potential to influence budgetary allocations within GRZ to IEC/BCC activities and stimulate new actions that guarantee sustainability. Support to CSOs – National Level: Of its two-pronged approach (national and community model) for sustainability, the CSH support to Afya Mzuri and CHAMP is the most sustainable. Budgetary and infrastructure (physical and IT) development support have played a role in the intensification of information dissemination for behavior change. These CSOs will continue to carry out IEC/BCC activities without further support from USG investment predominantly because the support they have been receiving is within their core mandates and they have been at the centre of implementation. This project ownership building is important for sustainability. It is important to note that the two CSOs were existed well before CSH started supporting them. Therefore, the support that they have been receiving is a stimulus for strengthening their work. Support to CSOs – Community Level: The creation of Safe Love Clubs and SMAGs through partner community health organisations does not guarantee continuity of community level IEC/BCC activities at the moment. This is because CSH has not created links between clubs and Neighbourhood Health Communities (NHCs), which are the GRZ-recognised structures for health delivery at the local level. Despite the fact that some of these clubs have created working relationships with local health centers, there is no evidence that they carry out their activities through or in partnership with NHCs. This makes sustainability of community level IEC/BCC activities difficult without support from these local level GRZ structures. 4.5 RESULTS TO DATE To what extent is the activity on track to achieve its intermediate results and meet its life of activity targets? Campaigns for two thematic areas (FP/RH and nutrition) have not been launched. There are therefore no major activities being undertaken on these two thematic areas. Activities on these two areas are clearly not on track to achieving their intermediate results. The Malaria campaign has been launched but design of adverts is yet to begin. Therefore, not much is being done in the implementation of the campaign. The general population is confused with the malaria campaign, with the actual name of the campaign not among those people that people used to refer to it. The night before the survey, only about 27.7% of children under five years of age slept under an ITN. This is a reduction from 49.9% recorded during the baseline (MIS 2010). This mean the malaria campaign is not on track to achieving its intermediate results. The MNCH campaign has also been launched and is currently being implemented. The TV campaign on MNCH has reached 14% of the general population, just 1% short of the 15% target for 2012. The radio campaign is far from being on track. It has reached 16.9% of the target population compared to the 30% it targeted for 2012. The project baseline indicated 60.3% of the target population went for ANC at least 4 times before birth. However, the mini survey found that at the moment the figure has reduced to 57.1%. This indicates that the MNCH campaign is also not on track to achieve its intermediate and life of activity targets. The HIV/AIDS campaign is the most known of all the campaigns. Currently the radio campaign on HIV/AIDS is reaching some 52% of the targeted audience, slightly above the 50% target. The    36 HIV/AIDS campaign on radio is therefore on track to achieving its objectives. The same cannot be said for the TV campaign, which is reaching only 46.7% compared to its 2012 target of 90%. This means the TV campaign is not on track. To what extent are the indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)? What improvements can be made to better capture progress? The indicators being tracked by the project are not adequate. The project has adequate impact indicators but the outcome and output indicators are inadequate. Most of the outcome indicators are not tracked by output indicators, and some impact indicators are not tracked by outcome indicators. The data collectors have no tools for collecting the data. They therefore have to mentally record the data and narrate every two weeks to a facilitator, who records the data on facilitator’s data collection tool. However, there are adequate tools to measure the capacity of GRZ on BCC related indicators. To better capture progress, the project needs to develop data collection tools for members of the various groups responsible for conducting the community level education and reporting on the message coverage and effectiveness. The project also needs to develop an efficient database that would enable disaggregation of data collected and ensure CSH can determine the performance in each geographic area and support the groups accordingly. What are the challenges to implementation and what can be done to improve the chances of the activity achieving its intended results and meeting its life of activity targets? CSH is designed as a national level project and is thus not expected to work at the community level. This poses several challenges for the organization. Firstly, ZISSP is supposed to work with CSH to deliver communication messages to community level, but ZISSP has limited resources to do this effectively. Furthermore they are restricted to working only in certain geographic areas, which restricts their reach throughout the country. Currently CSH is working with some community groups but there is little reporting from these CSOs and CBOs to CSH on IEC materials distribution, and the quality of community level BCC activities reporting is very low. However, CSH’s systems are not effective in monitoring these community level activities. The relationship of CSH with the newly formed MCDMCH is not clear to CSH. Currently most development partners are not clear about the roles and responsibilities of the MCDMCH. What they hear (without any documental evidence) is that MCDMCH is taking over the responsibilities of MOH at the district and community levels. With CSH being a national level project, it is not clear if they have any mandate in working with this ministry since the ministry has no national level responsibility. The GRZ is challenged with funding and therefore have not been able to practice much of the technical knowledge they acquired from CSH support. Currently, GRZ has acquired some knowledge on BCC from CSH but is not developing this knowledge into skills. Funding support to GRZ would enable them to practice what they have learned and build skills from the knowledge. The USAID policy of not paying allowances to GRZ staff for attendance of meetings and workshops is having a detrimental effect on the level of enthusiasm of GRZ staff with regard to CSH. It was found through KIIs that in some cases trainings and meetings are poorly attended as a result of this. Women are forced to test for HIV when they go for ANC services at the health facility. A group of pregnant women are kept together and provided some information as counseling. After that, all of them are tested compulsorily. Those who refused are often castigated and embarrassed before all others at the ANC. Some are even refused services. Women who go for ANC without their husbands/partners are also scolded and most often turned away without services. Some are made to pay a fine for not being accompanied by their husbands/partners. This practice is wide spread but most pronounced in rural areas. This is leading some women to refuse to go for ANC when they don’t want to be tested for HIV or when their spouses are not accompanying them. 37 5.0 RECOMMENDATIONS To ensure a comprehensive response to the challenges faced by CSH, these recommendations have been categorized into three broad sections: CSH, USAID and GRZ. The legend for the suggested time frame to complete the recommendation is: short term = within next three months; medium term = within next six months; and long term = by the end of the project. 5.1 RECOMMENDATIONS TO CSH Recommendations to CSH have been categorized by question categories: 5.1.1 Results to Date The project should facilitate the process of launching the campaigns on FP/RP and nutrition. The campaigns should also be integrated at the community level by training the various community groups on all the thematic areas to ensure effective dissemination of campaign messages on each thematic area at the community level. The project should also ensure that each advert is achieving the right results on the right persons. On Malaria, the project should produce radio and TV adverts to reach a larger portion of the targeted audience. There is also the need to develop more TV adverts and radio adverts on MNCH and broadcasts to ensure achievement of its intermediate results and meet its life of activity targets. On HIV/AIDS, the project needs to continue the radio advertisement at the current rate but intensify the TV adverts to ensure it also reaches many more people to achieve the intended results. The activity M&E plan needs to be reviewed to capture more output indicators. Each impact indicator must have at least one corresponding outcome indicator and each outcome indicator should have a minimum of one corresponding output indicator. All the output indicators should also be defined to ensure that all data collectors understand and count the same thing. CSH should also develop data collection tools for the various community level clubs responsible for education and data collection. These club members should also be trained on the data collection tools once they are developed. CSH also needs to redesign the current database to ensure that it can take the data from the data collection tools and generate the various analyses required. This will improve on the current practice of manually calculating the data before entering into the database. 5.1.2 Message Exposure and Effectiveness CSH needs to support GRZ to produce more IEC materials to ensure that community members have access to these materials to educate themselves. Some of these materials should also be produced in the popular local languages since some community members can only read their local languages, not English. The existing clubs, including Safe Love Clubs, Radio Listening Groups, Facilitator Groups, and SMAGs, should be trained in all thematic areas. Since CSH cannot establish each of these groups in every community, there is the need for each group to specialize in each of the five thematic areas to ensure that the same level of education is promoted everywhere. Safe Love adverts should be reviewed to ensure their meaning is clear not only for the members of the Safe Love Clubs, who have been trained, but the general public as well. While the slogan of Think, Talk, Act! Is highly appropriate, the adverts need to the make the desired action clear to ensure that people are thinking of the right actions. The Motherhood Testimonial Adverts on MNCH should also be reviewed and made shorter with the names of the characters silent or some characters replaced to ensure there is national representation. The Change Champion advert needs to be withdrawn because it conveys the wrong impression. Reviewing it would mean designing a new advert. CSH should also ensure that all adverts produced on the thematic areas, particularly malaria, FP/RP and nutrition, are pretested and well understood by the audience before airing. 5.1.3 Capacity Building More emphasis is needed on quality mentoring on both BCC and M&E. This would require CSH attaching highly experienced, preferably international, BCC and M&E Experts to GRZ to mentor and    38 coach them. This would ensure that they build skills and not just knowledge. GRZ should also be supported by these experts to set up an effective BCC system to ensure continuous and efficient production and distribution of IEC/BCC campaigns and materials. They should also be supported to establish effective M&E systems encompassing all their IEC/BCC indicators. Capacity building should be centered on systems and not individuals. CSH should avoid training people for the sake of getting the training going. In instances where District Health Officers are invited for training and an Environmental Officer comes for the training, that person should not be trained. The various departments should be considered for the training. For instance, the Operations Research Unit should be involved in research trainings and not just individuals from the BCC unit. The human resource departments should also be involved to enable them to plan the human resources required for various levels of BCC activities. USAID should allow CSH to provide funding support to GRZ in implementing BCC activities. This can be done in the form of sub granting to the BCC unit or financing their proposal. That would ensure that GRZ is given the opportunity to practice what they learn and enable them to build the skills to independently carry out such activities in the absence to USG support. 5.1.4 Evidence–Based Planning & Implementation The capacity of NMCC and NAC should be assessed to identify their BCC limitations before planning the support to provide them. The planning system should not end after the project design but should include planning for improving performance even during implementation. This can be done by making use of feedback from monitoring, evaluation and research. Deliberate plans should be made for collecting evidence to inform IEC/BCC planning and implementation. There is also a need for plans on revision of IEC products should feedback indicate such revision is required. 5.1.5 Sustainability The capacity built would remain with the benefiting person, but, with the current attrition rate, this increased capacity would be lost if not built on systems. CSH should therefore ensure that capacity building is structured to include the entire GRZ systems, including the Human Resource departments, the operations department, M&E department, and other departments who would contribute to sustaining the support. The Clubs at the community level should be linked to the community structures such as the Neighborhood Health Committees (NHC), with these community structures oriented on the responsibilities of the clubs. This would ensure that the NHC is able to supervise their activities and provide them with the support needed. This would also ensure that support and supervision to these clubs is sustained beyond USG support. CSH should support CHAMP to market the 990 Helpline and Afya Mzuri to market the resource centers. This would enable more people to patronize the services. The work of these two CSOs should also be linked to the activities of GRZ to ensure that they can continue to get some supervisory support when CSH ends. 5.2 RECOMMENDATIONS TO USAID There is the need for USAID to clarify the responsibilities of CSH. Currently, various documents state that CSH is a national program, meaning they are supposed to be operating only at the national and provincial levels. However, CSH also has district and community level responsibilities. This apparent contradiction allows CSH to engage in supporting community level activities but not to the fullest extent possible. USAID would need to either allow CSH to operate just a nation project only, ending support to CBOs, or USAID may allow CSH to take full responsibility for activities at the district and community levels, in which case CSH would have the responsibility of effectively supporting and monitoring the activities of these CBOs to ensure maximum performance at the community level. 39 USAID further needs to clarify CSH’s role in supporting the MCDMCH. Current available information indicates that MCDMCH shall only operate at the district and community levels. If this is the case, CSH may not have responsibility to support them if USAID decides that CSH should remain a national project. However, USAID may decide on specific support that should be provided by CSH to MCDMCH. This decision needs to be taken promptly to guide CSH on supporting MCDMCH. Currently, ZISSP does not have enough funds to produce IEC/BCC materials and take them to the community level for effective mobilization, even after CSH support to GRZ to launch a campaign. If ZISSP and CSH are to collaborate effectively, USAID needs to clearly define the role of each project. USAID also has to ensure that the project has enough budgetary allocation to implement the assigned level of effort. Currently, each new project is launching new campaigns on each thematic area and is harming prospects for sustainability and confusing the public. This has also prevented other stakeholders from getting involved in new campaigns. USAID should ensure that subsequent projects continue existing campaigns by indicating such as a requirement in the RFP. Currently, even clubs working on Safe Love are not sure of the difference between the Safe Love campaign and One Love Kwasila campaign. It would have been much easier for CSH to continue that campaign by including new areas that were originally left out than launching a new campaign. Similar consideration should be given to malaria, MNCH, nutrition, and FP/RH. USAID needs to give consideration to the policy on USAID projects that prohibit paying allowances to GRZ staff. USAID should discuss this issue with GRZ and agree on how to move this forward since the lack of stipends is demotivating staff, with some even refusing to attend USAID trainings. An amicable resolution to this issue would ensure that the right persons attend the trainings and knowledge is gained by the right individuals in the GRZ for sustainability. 5.3 RECOMMENDATION TO GOVERNMENT OF ZAMBIA The government should ensure that staff are available and put in measures to reduce the attrition rate. The staff strength at the provincial, district, and community levels is also weak and needs to be fortified. The government should also make budgetary allocation to BCC to sustain activities at all levels. Currently the role of the MCDMCH is not clear to a lot of stakeholders because they have not received any formal communication on it. Government should ensure that there is formal communication to development partners on the exact role and structure of this new ministry. This would enable projects like CSH to plan effectively in providing supporting.    40 ANNEXES ANNEX A Statement of Work ANNEX B Map of Zambia – Provinces Visited for Data Collection ANNEX C Evaluation Team Biographies ANNEX D Scope & Methodology ANNEX E List of Persons Consulted ANNEX F Evaluation Matrix ANNEX G Oral Consent Script – Mini Survey ANNEX H Key Informant Interview Guides ANNEX I Site Visit Protocol ANNEX J Focus Group Discussion Guide ANNEX K DQA Question Guide ANNEX L BCC Assessment Guide ANNEX M Mini Survey Questionnaire ANNEX N Online Survey Questionnaire ANNEX O List of References 41 ANNEX A: EVALUATION STATEMENT OF WORK I. PROJECT INFORMATION Project Title: Communications Support for Health (CSH) Project Number: Contract No. GHS-I-007-00004-00 Project Dates: July 8, 2010 – December 13, 2014 Project Ceiling: $43,337,946 Obligated Amount: $20,428,684 Implementing Organization(s): Chemonics International, Inc. Sub-Contractors: The Manoff Group and ICF International Sub – Grantees: Afya Mzuri; Comprehensive HIV/AIDS Management Program (CHAMP) Introduction/Program Context The Republic of Zambia has a population of just over 13 million and is growing at a rate of 2.8% per year3. Population density is sparse in rural areas and the Zambian health care system consists of a network of approximately 1,880 public and private health facilities, consisting of health posts, rural health centers, urban health centers, and level one, two and three hospitals4. Limited human resources have complicated Zambia’s efforts to provide most health services. Despite donor support for training and retention schemes, the MOH is only able to employ approximately 57% of the health care staff required to staff health facilities.5 The reality today is that rural health centers are often staffed by a single individual who has not had clinical training (e.g., the grounds keeper or an environmental health technician). Most Zambians are subsistence farmers and 80% of the population lives in poverty, of which 63% live below $1.25 per day6. In a country of this size (overall the size of California and Nevada combined), with limited infrastructure, many communities, especially those in rural areas, face significant barriers to reach health facilities such as impassable roads, long distances to health facilities, and seasonal flooding (some facilities are physically impossible to reach for months at a time). Additionally, Zambian societies are traditional and many foster patterns of health seeking behaviors that delay or limit beneficial contact. Social barriers such as gender inequities and cultural practices further complicate access. All of these factors together result in poor health outcomes for Zambians, such as high HIV prevalence (14.3% among adults and 16.6% among pregnant women)7, a high malaria burden (3.2 million reported cases in 2009)8, one of the highest fertility rates in the world (6.2 total fertility rate)9, a high maternal mortality ratio (591/100,000) and where 45% of children under 5 years are stunted10. To overcome these significant challenges the Government of the Republic of Zambia (GRZ), through its Ministry of Health (MOH), has committed to achieving Millennium Development Goals (MDG) targets and improving the health of its population by improving the quality of health care services and providing greater and equitable access to health care. To support these objectives, USAID, through the CSH                                                              3 Zambia Central Statistical Office, 2010 Census of Population and Housing Preliminary Report 4 Ministry of Health, List of Health Facilities in Zambia, 2010 5 National Health Strategic Plan, 2011 - 2015 6 USAID/Zambia Country Development Cooperation Strategy, 2011 - 2015 7 2007 Zambia Demographic and Health Survey 8 Ministry of Health, National Malaria Control Action Plan for 2010 9 2007 Zambia Demographic and Health Survey 10 2007 Zambia Demographic and Health Survey    42 program, is providing technical assistance to GRZ, targeting systems and interventions that will impact provision of health services, and mobilizing communities to actively participate in the management of health programs. CSH aims to strengthen the capacity of the GRZ to implement effective health communications activities. It is expected that improved capability of the GRZ to implement effective health communications activities will translate to change in population behavior — it will result in a measurable reduction in the practice of risky behaviors and increased demand for and use of health care services. Implemented in conjunction and collaboration with efforts to increase access to and quality of health care services, this will enable and result in improved health outcomes. For more information about the specific health challenges in Zambia, see ANNEX A. The current government of Zambia created the Ministry of Community Development, Mother and Child Health (MCDMCH) that will take over many of the community oriented activities once undertaken by the Ministry of Health. As the roles and responsibilities of this new ministry are defined it is expected that the Ministry of Health will limit its activities to service delivery while the MCDMCH will take an increasing role in mobilizing the community to improve health outcomes. Activity Description CSH is a four and a half year, $43 million contract that is tasked with supporting the Government of the Republic of Zambia’s (GRZ) vision of “equity of access to assured quality, cost-effective, and affordable health services as close to the family as possible.” The GRZ through its Ministry of Health (MOH), is committed to achieving the MDG targets by improving the quality of health care services, and providing greater and equitable health care access for its people. To support these objectives, USAID is providing technical assistance to the GRZ in strengthening national health communications activities. The aim is for GRZ health communications activities supported by CSH to translate into increased sustainable local capacity and positive behavior change that contribute to GRZ efforts in five focal areas: 1) HIV/AIDS, 2) malaria, 3) family planning/reproductive health, 4) maternal and child health and 5) nutrition. While CSH’s office is located in Lusaka, it’s communications campaigns are designed to reach all 10 provinces, through a variety of different media (radio, TV, newspaper, community groups, Safe Motherhood Action Groups) with messages in the focus areas mentioned above and generally targeting adults ages 15-49. Each campaign has its own specific target groups, where the Safe Love campaign and the Integrated Malaria, MNCH and Nutrition campaign focus on adults ages 15-49; the Mothers Alive (Safe Motherhood) campaign specifically targets pregnant women. Also, CSH wiimplement three “mini campaigns” under the larger Safe Love campaign (Gender Based Violence, Alcohol, and Youth) and these will focus on their own respective target groups. For a more detailed summary of CSH’s campaign coverage and target groups, see ANNEX A. The national level campaign rollouts were staggered, thus the HIV/AIDS campaign was launched in June 2011, the Integrated Malaria, MNCH, and Nutrition campaigns were launched in November 2011, and the Safe Motherhood campaign was launched in April 2012. Family planning and reproductive health messages are integrated into the Safe Motherhood and Integrated campaigns. CSH is mandated in its contract to work closely with the USAID funded Zambia Integrated Systems Strengthening Program (ZISSP), where CSH implements national communications campaigns and ZISSP brings the same messages down to the community level through its support of Safe Motherhood Action Groups (SMAGs) and Community Health Workers (CHWs). CSH and ZISSP meet regularly to discuss their collective communications and capacity building activities. In February 2012, they developed a joint malaria implementation plan for 2012, in order to ensure coordination and to minimize the duplication of efforts. In attempts to link messages seeking to increase demand for health services to actual service delivery, CSH often works with the Private Sector Mobilization Project for Social Marketing (PRISM) project to offer male circumcision and voluntary counseling and testing (VCT) services at CSH sponsored events. As of May, 2012, $21,103,684 has been obligated into the award, across the five program areas mentioned above (HIV/AIDS, malaria, FP/RH, MCH and nutrition). There have been three modifications to the contract to date, two of which added incremental funding and the third reduced the total estimated cost of the contract. The reduction in funding was minimal ($11,749) and was to adjust the Total Estimated Cost, exclusive of fixed fee. A shift in programming occurred in October, 2011, when Zambia was chosen as a focus country for the Saving Mothers, Giving Life (SMGL) endeavor to reduce maternal mortality by 50% in four target districts 43 (Kalomo, Lundazi, Nyimba, and Mansa) in Zambia. USAID/Zambia did not receive any supplemental funding for these efforts, so USAID asked its existing implementing partners to adjust their planned activities so that they focus on these four districts. CSH had already designed the safe motherhood national communications campaign, called Mothers Alive, so USAID/Zambia asked CSH to take these same messages and work with ZISSP to bring the safe motherhood messages to the communities through SMAGs. In order to coordinate, monitor and evaluate these efforts, CSH hired two new staff, a SMGL Coordinator and a SMGL Monitoring and Evaluation Specialist. The predecessor to the Communications Support for Health program was the Health Communication Partnership (Johns Hopkins University Center for Communication Programs). HCP was a five-year $33 million cooperative agreement that was launched in August 2004 and ended in December 2009. Contributing to USAID/Zambia’s approach of Zambians taking action for their health (intermediate result under the old Assistance Objective, Improved Health Status of Zambians), the program worked closely with the MOH, National HIV/AIDS/STI/TB Council (NAC) and the National Malaria Control Center (NMCC) on mass media campaigns, Information, Education, and Communication/Behavioral Change Communication (IEC/BCC) materials, and community development to increase knowledge of health issues and promote changes in risky behaviors and harmful gender and other socio-cultural norms. CSH operates primarily at the national level, providing technical assistance to the GRZ in development, implementation, and evaluation of health communications activities. In the context of CSH, “GRZ” refers to three primary agencies — the Ministry of Health (MOH), the National Malaria Control Center (NMCC), and the National HIV/AIDS/STI/TB Council (NAC). The project works with and supports other USAID assistance programs (ZISSP, PRISM) in behavior change communication for message consistency and efficiency, as well as to extend the reach of CSH to the district and community level. CSH provides direct support to GRZ in the planning, design, and implementation of communications campaigns and activities. This will be done consistently with a focus on capacity building and transfer of skills. CSH will measure the GRZ’s management capacity in IEC/BCC through a capacity index tool developed by the CSH Monitoring and Evaluation (M&E) team. The tool measures GRZ’s technical, management, and M&E capacity specifically as it relates to IEC/BCC activities. The tool provides an overall score, as well as scores for BCC planning and design, BCC program implementation, and M&E for BCC programs. Results Framework The results framework outlines CSH's project objective (capacity of GRZ to manage effective IEC/BCC activities strengthened) and its strategy to achieving it through the project’s intermediate results. "Effective" means that IEC and BCC activities implemented by the GRZ result in a measureable reduction in the practice of risky behaviors and/or an increase in demand for and use of health care services. It is important to note that these are outcomes and will not be measured by or directly attributable to this activity, but based on the development hypothesis, it is expected that this activity will contribute to the improvement in these top level indicators (example: % of men and women 15–49 years who have had two or more partners in the last 12 months), measured through periodic national level surveys: Zambian Demographic and Health Survey (ZDHS) and the Malaria Indicator Survey (MIS). Aligning the project objective with USAID's strategic objective for health ensures that project activities are designed to achieve mission results. The project will work toward four intermediate results, each intermediate result aligned with a defined contract task (for an expanded list of Performance Intermediate Results, please refer to the signed contract: Performance Intermediate Result 1 (PIR 1): National health communications campaigns strengthened. Performance Intermediate Result 2 (PIR 2): GRZ use of evidence-based health communications approaches (formative research, design, implementation, monitoring and evaluation of IEC/BCC campaigns) increased. Performance Intermediate Result 3 (PIR3): Local capacity to support sustained implementation of IEC/BCC activities strengthened. Performance Intermediate Result 4 (PIR 4): IEC/BCC activities institutionalized in health system expanded.    44 Each PIR (and task) uses a different approach to building local capacity - providing direct technical consultation, establishing a repository of available resources, providing training, and coordinating partnerships. The framework provides the foundation on which the activities presented are structured and establishes how activities feed into project results and outcomes. For the full results framework, please see ANNEX B. At the core of monitoring and evaluation for the CSH project is the Performance Management and Evaluation Plan (PMEP) which consists of a set of performance indicators that are divided into two categories (ANNEX C): Behavioral Outcome/Health Impact Indicators: These indicators are the behaviors and health impacts that are expected to result from the implementation of effective IEC/BCC activities with CSH support, addressing the project’s objective: to strengthen the Government of the Republic of Zambia’s capacity to manage effective IEC/BCC activities (sources are the ZDHS and MIS). The outcome/impact indicators are internationally recognized, standard indicators that are usually obtained from population based surveys conducted every 2–4 years. However, it is important to note that CSH is not collecting these indicators. They are collected by the surveys mentioned above (ZDHS, MIS). The timing of the ZDHS and the MIS do not coincide with the start and end of CSH, so USAID is following trends of these health impact indicators over time, while providing significant investments into health overtime. Output/Immediate Outcome Indicators: These indicators aligned with the project’s intermediate results and reflect the activities of the project. In most cases, the indicators are direct results of project efforts (e.g., number of campaigns supported). In some cases, the indicators reflect the performance of others when that performance has been a target of the project’s capacity building efforts (e.g., percentage of national IEC/BCC campaigns for which formative research was conducted). Since the CSH launch in July 2010, there have been notable achievements: CSH provided technical and financial support to NAC in the design and development of a comprehensive HIV campaign ‘Safe Love’ that addresses multiple concurrent partnerships (MCP), low condom use and mother to child transmission (MTCT). CSH provided technical and financial assistance to NMCC in the design and development of communication messages and materials for the national insecticide-treated net (ITN) mass distribution program. CSH further helped NAC and NMCC in the development of their national HIV and malaria communication strategies for 2011 – 2015 and provided technical assistance to NAC to develop 2011- 2015 National HIV/AIDS M&E Plan. CSH assisted the GRZ in the development of National HIV/AIDS, Malaria and Male Circumcision Communication strategies. These strategies are aligned to national strategic plans and will guide national implementers in designing communication messages and products. In addition to CSH’s direct assistance to the GRZ, they provided sub grants to CHAMP and Afya Mzuri for the expansion of the 990 Talkline, and Health Communication Resource Centre respectively. These two institutions will be able to provide a broad spectrum of health information other than just information about HIV/AIDS. Performance monitoring, discussions with staff and observations on site have nonetheless identified a number of ongoing or emerging implementation challenges: The unresolved issue of allowances for GRZ staff at trainings or conferences highly affects their participation in planned activities. Currently, it is against U.S. Government policy in Zambia to pay GRZ officials any allowances for attending trainings or meetings, other than normal per diem (lodging costs, meals and incidentals). However, the Zambian government’s policy is to pay “sitting allowances” to their staff or a cash bonus for attending a meeting or training. GRZ staff will sometimes protest the USG policy by not attending USG sponsored meetings or trainings. It has affected the project implementation of CSH and other projects, as they depend on numbers trained as an indicator for project performance. This issue has yet to be resolved. GRZ full participation in CSH planned activities has been a challenge due to inadequate staff in GRZ units responsible for the activity. These same individuals from the GRZ also support other partners. 45 There is inadequate local capacity to produce quality media products contributes to lengthy production processes and leads to implementation delays. CSH is working with local media firms to increase their capacity to deliver high quality products, but it adds more time onto the production process. Ethical review and approval of protocol and tools for formative research and for pretesting campaign products takes a long time and delays actual development of campaigns. Normally, CSH gets approval from the local Institutional Review Board (IRB) for its research, which takes approximately 10 days. However, the MOH requires research approval as well and this process usually takes much longer and delays implementation. II. EVALUATION PURPOSE AND RATIONALE The USAID/Zambia Health Office seeks an independent team to perform a mid-term performance evaluation of the CSH program. The objectives of this mid-term evaluation are three-fold: Part A (Retrospective): To help determine what progress CSH has made in achieving its Life of Project targets and whether or not they are likely to achieve them by the end of the program. What components of the CSH program are working well and why? If there are parts that are not working, then why not? (Level of effort: 45%); Part B (Prospective): Based on the above findings and conclusions, to make recommendations for CSH project implementation through December 2014, including the optimal mix of activities and funding for achieving project objectives and sustainability (level of effort: 45%); Part C: Using the above findings and conclusions, frame issues to discuss/resolve at a level higher than the project, specifically at the level of the GRZ and/or other donor organizations, if applicable. For example, if the capacity building portion of this project is not performing well due to staffing shortages in the MOH, perhaps CSH and other projects can re-direct their assistance in ways that would better suit the GRZ and their needs (level of effort: 10%). USAID/ Zambia will disseminate the report widely with relevant stakeholders and project beneficiaries. The findings will also be used in modifying the life of project targets and technical approaches based on the recommendations. EVALUATION QUESTIONS The Contractor shall answer the following evaluation questions: Results to Date To what extent is the activity on track to achieve its intermediate results and meet its life of activity targets? To what extent are the indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)? What improvements can be made to better capture progress? What are the challenges to implementation and what can be done to improve the chances of the activity achieving its intended results and meeting its life of activity targets? Message Exposure and Effectiveness To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with IEC/BCC messages in each of the five health intervention areas (HIV/AIDS, malaria, FP/RH, MCH and nutrition)? Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages? Capacity Building To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own? What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own? What else does GRZ need in order to independently plan, implement, manage, and evaluate national health campaigns? To what extent is the project adapting to the current changes in the GRZ structures particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re￾organization of the National HIV/AIDS/STI/TB Council?    46 Evidence – based Planning and Implementation In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? Sustainability Are IEC/BCC activities likely to continue without further USG investments and if not, what investments or approaches would better promote sustainability? III. EVALUATION DESIGN AND METHODOLOGY: SOURCES OF INFORMATION Sources of information will include, but is not limited to, the following: Signed contract All relevant contract modifications (3) Quarterly project reports (7) Monitoring and Evaluation Plan Formative research reports (HIV/AIDS, Safe Motherhood, Malaria) Portfolio review templates (3) Performance Based Management System (tracks M&E and financial data in real time) Safe Love Rapid Survey Report CSH Self-Assessment Report Sub-contractor/sub-grantee reports to CSH Scope of Work or Program Description for ZISSP Scope of Work or Program Description for PRISM METHODOLOGY USAID/Zambia is looking for creative approaches to conducting this evaluation and the Contractor shall provide a detailed explanation of the proposed methodology for carrying out the work. The methodology shall be comprised of a mix of tools appropriate to the evaluations’ research questions. These tools shall include a combination of the following: Document and data review (see list above) Key informant interviews Interviews Mini-surveys (in particular among targeted populations) Focus groups Direct observation Self-Assessment by CSH: Prior to the evaluation team beginning its work, CSH will have completed and submitted in August 2012, a self-assessment report to USAID. This report will be provided to the Contractor and shall be included in the Contractor’s document review. Document and Data Review: The Contractor shall start its work with a document and data review of all the sources cited in the “Sources of Information” section above. Key Informant Interviews: The contractor shall conduct qualitative, in-depth interviews with key stakeholders and partners (see preliminary list below, but the Contractor should add to this list as necessary). The Contractor shall conduct face-to-face interviews with informants. When it is not possible to meet with stakeholders in person, telephone interviews shall be conducted. Key informants shall include, but not be limited to: CSH program staff (Chief of Party, Deputy Chief of Party, Technical Director, M&E Specialist, BCC Advisors) ZISSP program staff (Chief of Party, Deputy Chief of Party) PRISM program staff (Chief of Party, Deputy Chief of Party) USAID staff (COR for CSH, COR for ZISSP) GRZ staff (MOH, NAC, NMCC, MCDMCH) at the national level 47 GRZ staff at the provincial and district levels in all provinces that CSH is currently working in. The Contractor shall also make sure to gather information from GRZ staff that is representative for both rural and urban areas. Interviews: The Contractor shall conduct one-on-one interviews with a sample of GRZ employees at the national, provincial, and district levels who have attended a CSH sponsored BCC training in order to assess the quality of skills, knowledge transfer and the sustainability of the program. Mini-surveys: The Contractor shall conduct mini-surveys in order to determine key message coverage and effectiveness amongst beneficiaries in both urban and rural target areas. For more information on the geographic scope, target audiences, and messages for each communications campaign, please see ANNEX D. Focus Groups: The Contractor shall facilitate focus groups with community members involved in the Safe Love Clubs, groups which are organized by Civil Society Organizations (CSOs) in Zambia, who have subcontracts with CSH. This is to better understand the message exposure and effectiveness. Since these activities are in the process of expanding, the team should note that we only have subcontracted seven CSOs, but plan on increasing this number. For a preliminary list, see ANNEX E. Direct Observation: The Contractor shall directly observe any capacity development training, if possible (it will depend on the logistics and timing of scheduled trainings). The Contractor shall directly observe Safe Love Club meetings in order to assess the message exposure and effectiveness. Also, the team could visit service delivery points in order to directly observe any safe motherhood and/or FP/RH activities. This will allow the team to identify any gaps or issues directly. Part of the Contractor’s team composition shall include a USAID/Zambia employee, who is not the Contracting Officer’s Representative for the CSH program. The COR will provide technical direction during the course of this evaluation. IV. EVALUATION DELIVERABLES The contractor shall deliver the following: 1. Work plan: The Contractor shall submit a detailed work plan, including plans for consultation with USAID and its partners, two weeks following the start of the contract. o USAID/Zambia will provide written feedback to the Contractor within five days following receipt of the work plan. 2. The final evaluation methodology and schedule, interview and site visit protocols and a draft evaluation report outline designating individual team member responsibilities shall be submitted to the Contracting Officer Representative (COR) for review and feedback at the beginning of Week 4. o USAID/Zambia will provide written feedback to the Contractor within five days following receipt of the final evaluation method and schedule, interviews and site visit protocols and a draft evaluation report outline. 3. Briefings: The Contractor shall organize and provide an entry, and final briefing for USAID/Zambia staff, other USG agencies and staff, implementing partners host government officials. 4. Interview notes and resource documents: The Contractor shall undertake extensive consultations with USAID, its partners and stakeholders. The Contractor shall provide summaries of all key meetings, workshops and discussions conducted in the course of the CSH mid-term evaluation. These summaries shall be submitted to USAID/Zambia along with copies of any relevant documents and reports gathered in the course of the evaluation in accordance with human subject provisions. 5. Draft mid-term evaluation report: The Contractor shall submit three hard copies and one electronic copy of the draft report to the Contracting Officer’s Representative (COR) two working days prior to the final de-briefing and Evaluation Team Leader departure from Zambia. In the report, the Contractor shall separate the findings, conclusions, and recommendations for each question. All recommendations included in the report shall be practical, specific, and action-oriented and designate the proposed implementer and timeframe. o The draft report will be peer reviewed by a selected team comprising of USAID and other USG staff members. The review will be managed by the Program Office. The Contracting Officer’s Representative (COR) will provide results from this review to the Contractor.    48 o USAID/Zambia will provide written feedback to the Contractor within 10 days following receipt of the work plan. 6. Final evaluation report: The Contractor shall submit three hard copies and one electronic copy of the final report to the USAID. The final evaluation report shall incorporate modifications requested by USAID, as agreed by both parties, unless the modifications are designed to alter the findings. However, if USAID identifies factual errors or can provide additional evidence/information to the evaluation team and they agree to amend the report, then the modification shall be accepted. o USAID will return the final evaluation report with these comments within 10 working days. EVALUATION REPORT FORMAT The Contractor shall prepare the draft and final evaluation reports in accordance with the following format: The evaluation report must be written in English. The evaluation report must be formatted for size A4 paper. Executive Summary (6 pages maximum length): Brief description of CSH Project key results/impacts, and evaluation’s major findings/recommendations and lessons learned Main body (40 pages maximum length): Description of the project: Drawing from the CSH Project, concisely describe the rationale of CSH’s Behavior Change Communication interventions, what constraints/opportunities it they were meant to address, and what, specifically, the program has been trying to accomplish. Specify the problem(s) the program is facing and propose ways to overcome these challenges. Evaluation purpose, methodology: Describe briefly, types and sources of evidence and methodologies employed to complete the evaluation SOW. Findings: Present findings, with supporting evidence, as related to the questions in the SOW and other pertinent matters that arise during the course of the evaluation. Conclusions: Present conclusions in relation to the findings. Recommendations: Present and synthesize pertinent recommendations related to ongoing planning, management and implementation of the CSH Program. Also address matters of long-term sustainability and impact. All recommendations shall be practical, action-oriented, specific, and designate the proposed implementer. Lessons Learned: Describe and document lessons learnt from the project to-date. Consideration should be given to internal project aspects (planning, design, management, and implementation) and external factors (e.g., policy contexts, other country/regional/global factors that have been constraining or supportive). The evaluation report shall also contain all the data collection instruments used in the evaluation in the appendices. When applicable, evaluation reports must include statements regarding any significant unresolved differences of opinion on the part of the funders, implementers and/or members of the evaluation team. V. TEAM COMPOSITION The Contractor’s team shall be comprised of two senior consultants, two local (Zambia-based) consultants and one local logistics assistant. A representative from the GRZ may be asked to participate. Below is a summary of the team composition: Senior Team Leader/Evaluation Specialist is experienced in evaluation design and implementation. Experience evaluating large health programs is desirable. This position is open to both expatriate and local expertise. Senior IEC/BCC Advisor is experienced in the design, implementation, monitoring and evaluation of large, health IEC/BCC programs. Research Specialists (2) are experienced in both quantitative and qualitative research methods. Experience with monitoring and evaluation of development programs is desirable. A logistics coordinator 49 Officials from the GRZ (MOH, NMCC, and NAC) will be invited to participate as observers during the planning and field portion of the evaluation as well. They will not have any formal responsibilities related to preparation of the final report (site visit related travel and per diem costs for these officials will be budgeted under this contract and managed by the contractor). The contractor shall observe current USG policies with regard to allowances payable to Government of Zambia staff members. Senior Team Leader/Evaluation Specialist The Senior Team Leader/Evaluation Specialist must have extensive evaluation experience. It is desirable that they will also have previous experience evaluating health programs. S/he will agree to fulfill his/her responsibilities in the negotiated time-frame, spending approximately ten (10) weeks in country, and will play the lead role in guiding the evaluation process. The Senior Team Leader/Evaluation Specialist shall hold conference calls with core team members and USAID/Zambia representatives prior to and following the visit to Zambia, in-brief with USAID/Zambia on arrival, debrief USAID/Zambia and Chemonics on evaluation findings, and produce a draft report to be left with USAID/Zambia prior to departure, followed by a final report for USAID/Zambia incorporating USG feedback. Qualifications for Senior Team Leader/Evaluation Specialist Education: An advanced degree from an accredited institution. Work Experience: Minimum 10 years of progressively responsible experience with organization(s) in the evaluation of development programs. S/he must have demonstrated technical expertise and skills in Monitoring and Evaluation (M&E), preferably with experience evaluating health programs. Skills and Abilities: Demonstration of strong analytical, managerial, and writing skills is very critical for this evaluation assignment. The Senior Team Leader/Evaluation Specialist should have demonstrated strong leadership, analytical, management and organizational, communication and interpersonal skills. In addition, they must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. The Senior Team Leader/Evaluation Specialist must be fluent in English and have proven abilities to communicate clearly, concisely, and effectively, both orally and in writing. The Senior Team Leader/Evaluation Specialist must be able to produce a succinct, quality document that assesses implementation successes and shortfalls and lays out actionable recommendations to guide and improve the CSH project during its final years of implementation. Senior IEC/BCC Advisor This team member must have extensive knowledge of and experience with designing, implementing, monitoring and evaluating IEC/BCC programs, in the realm of health. Previous experience with IEC/BCC programs in Southern Africa is desirable. Qualifications for Senior IEC/BCC Advisor Education: MD, RN, MPH, Ph.D., MA, MS, or equivalent from an accredited institution. Work Experience: Minimum 6 years of progressively responsible experience with organization(s) in the design, implementation, monitoring and evaluation of IEC/BCC programs, with demonstrated technical expertise and skills in health, preferably in Southern Africa. Skills and Abilities: The Senior IEC/BCC Advisor must have strong, demonstrated analytical, managerial, and writing skills. The Senior IEC/BCC Advisor must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. The specialist must be fluent in English and have proven abilities to communicate clearly, concisely, and effectively, both orally and in writing. Research Specialists Two (2) expert consultants with expertise in research design. Demonstrated technical expertise in quantitative and qualitative research methods. Experience with monitoring and evaluating development programs in Zambia is desirable. The Research Specialists must be local consultants, fluent in English. The Contractor shall be responsible for identifying expert consultants with the skills mix required to assess the full range of technical and managerial program priorities under CSH. Qualifications for Research Specialists    50 Education: MD, RN, MPH, Ph.D., MA, MS, or equivalent from an accredited institution. The Research Specialists will possess the range of technical competencies and expertise required to fully assess CSH implementation progress and issues. Work Experience: Minimum 6 years of progressively responsible, Zambia-based experience with organization(s) in research design, using both quantitative and qualitative methods. Skills and Abilities: The Research Specialists must have strong, demonstrated analytical, managerial, and writing skills. The Research Specialists must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. The Research Specialists must be fluent in English and have proven abilities to communicate clearly, concisely, and effectively, both orally and in writing. Logistics Coordinator The Evaluation Logistics Coordinator will be responsible for logistics, coordination, administrative support, and ensuring all aspects of the evaluation are carried out seamlessly. The Logistics Coordinator, in collaboration with the USAID/Zambia COR and Chemonics, will organize meeting space and materials, initial partner meetings, and site visit schedule and related logistics in advance of the in-country portion of the evaluation. VI. EVALUATION MANAGEMENT Below is a list of the specific tasks to be accomplished by the evaluation team, with an estimated level of effort for each task. Home (five day work week) Field (six day work week) STL SIBA RS1 RS2 Logistics Planning/ Document Review * Week 1 and 2 5 3 Team Leadership Prep/ Document Review Cont'd * 5 5 4 4 4 International Travel 2 2 Team Preparation and Production of Deliverable 3 * Week 3 6 6 6 6 6 Field Implementation * Week 4, 5, 6, 7 24 24 24 24 24 Data Analysis/ Draft Report/ Mission De Brief * Week 8, 9 12 12 12 12 6 International Travel Week 10 2 2 USAID Review Week 10 Reporting * Week 11, 12, 13 12 12 4 4 IBTCI Proposed LOE Total 68 66 50 50 40 SOW Estimate 68 66 46 46 34 STL: Senior Team Leader/Evaluation Specialist; SIBA: Senior IEC/BCC Advisor; RS: Research Specialists; LC: Logistics Coordinator *Will take place in home country and not in Zambia 51 Draft site visit protocols and draft report outline will be reviewed and cleared by USAID. The GRZ needs 4 weeks of advance notice to approve site visits and interviews with staff. The mid-term evaluation will take place over an approximate 13-14 week period. Total level of effort (LOE)—68 days of LOE for Senior Team Leader/Evaluation Specialist (including four international travel days); 66 days for Senior IEC/BCC Advisor and up to 100 days for two local Research Specialists; 40 days for the Logistics Coordinator. A six-day work week is authorized for work in Zambia. Premium pay is not authorized. The contractor shall be responsible for all logistics support required by the evaluation team, including office and meeting space and equipment, secretarial support, photocopying, international and local communications, international and local travel and transport, and preparation of the final evaluation report.    52 ANNEX B: MAP OF ZAMBIA – PROVINCES VISITED FOR DATA COLLECTION 53 ANNEX C: EVALUATION TEAM BIOGRAPHIES Joseph Limange, Team Leader, is an independent consultant and served as the Team Leader for this evaluation. He has over 16 years of experience in project planning, research, and monitoring and evaluation in Africa and is a long-term leader of the monitoring and evaluation team for a USAID funded health behavior change communication project in Ghana covering Malaria, Maternal, Neonatal, and Child Health (MNCH), Family Planning and Reproductive Health (FP/RH), Nutrition, and Water, Hygiene and Sanitation being implemented. Mr. Limange is a PhD candidate in SMC University and holds an MBA from the Paris Graduate School of Management and a Higher Degree in statistics from Tamale Polytechnic, Ghana. Maurice Ocquaye, Senior IEC/BCC Advisor, developed his expertise in BCC program management, training and community mobilization in over 9 years of work with various organizations in countries including the USA, Australia, England, Zambia, Zimbabwe, South Africa, Kenya, Togo, Sierra Leone, Angola, Rwanda, Benin, Burkina Faso and Nigeria. He was recently the lead Social Mobilization Technical Advisor on a USAID PMI malaria project in Ghana. Mr. Ocquaye is a PhD candidate at Walden University and holds an MFA from the University of Ghana. Moses Simuyemba, Research Specialist, is a Zambian Medical Doctor and Public Health Specialist with an 11-year background in research design, M&E, and management of HIV/AIDS and other related health programs. He brings technical expertise in quantitative and qualitative research methods and was formerly the National Health Program Coordinator for the Zambia Red Cross Society (ZRCS). Mr. Simuyemba received his Masters for Public Health from the University of Western Cape, South Africa, and two Bachelor’s degrees in Medicine and Surgery, and Human Biology from the University of Zambia. Saviour Chishimba, Research Specialist, brings 17 years of experience in HIV/AIDS and providing strategic planning consultancy services for public and private organizations in Zambia. He has an extensive background in research methodologies, training and policy formulation and is the former Chief Executive Officer for the Pan-African Academy for Health and Social Sciences, a research, training institute and consultancy firm specializing in public health, human rights and all social sciences. His major research works include a study on integrating HIV/AIDS in poverty reduction strategies, and a study on the effect of common childhood illnesses on OVCs’ educations in community schools. Dr. Chishimba holds a Ph.D. in Public Health from the Cosmopolitan University.    54 ANNEX D: METHODOLOGY NOTES Protection of Human Subjects In adherence with guidance on the projection of human subjects recommended by the United States National Institutes for Health, informed consent (Annex G) was solicited from all respondents before commencing to interview them. We will not ask for signatures since this would not be culturally appropriate and could also break the confidentiality agreement. Annex III is a copy of the Informed Consent form that will be used. A copy of the oral consent form, signed by the interviewer was given to the respondent before the interview. Evaluation Team Structure The evaluation was led by the Team Leader, Joseph Sineka Limange during the pre-evaluation planning, and initial consultation with USAID, CSH, ZISSP and PRISM. He also led the team during the development of question guides and questionnaires. After the development and approval of the question guides and questionnaires, the team divided into two: Team A and B. Each team consisted of three members including the Team Leader, a Research Specialist and a Research Assistant. The Sub-Team Leaders and Research Specialists conducted the FGDs whiles the Research Assistant collected data for the mini survey. The Team Leader/Evaluation Specialist led the team A and the BCC specialist lead team B. The Research Specialists had the responsibility of supervising the research assistants, reviewing the data collected each day and supporting in responding to quires from respondents. After the field level data collection, field data collection, the Team/Evaluation Specialist with conducted the DQA for CSH whiles the BCC Specialist conducted the BCC Systems assessment for CSH, NAC, NMCC and MOH. After the complete set of data had been collected the Team Leader/Evaluation Specialist then led the joint evaluation team in analyzing the data and developing the report. Table of Roles and Responsibilities of the Sub-team Members Team Member Role & Responsibility Team Leader Joseph Limange Carry out detailed project planning. Coordinate and assign tasks to the research team. Assign desk review assignments to core evaluation team members for the review of background research and project documents. Lead the development of site visit protocols, data collection tools, question guides, and online and mini-survey questionnaires. Work with a Research Specialist to enter quantitative data into a SPSS database and lead the quantitative analysis. Lead the Data Quality Assessment 55 Team Member Role & Responsibility component of the evaluation. Participate in field data collection and lead the report writing phase. IEC BCC Advisor Maurice Ocquaye Perform Evaluation Tasks, collect and analyze performance data, assist in the development, completion, and editing of deliverable, and other tasks or materials relevant to the project that as requested by the Evaluation Team Leader, IBTCI and the client. Lead the BCC Assessment component of the evaluation. Research Specialists: Moses Simuyemba Saviour Chishimba Perform evaluation tasks, collect and analyze performance data, assist in the development, completion, and editing of deliverables, and other tasks or materials relevant to the project as requested by the evaluation team leader, IBTCI and the client. Research Assistants: Sharon Mwangani Chandela Masengu Data collection activities you will be expected to perform include the following: Take notes, ask questions, interpret as needed and directed during Key Informant Interviews and Focus group discussions; Prepare field notes and transcripts of Key Informant Interviews and Focus group discussions; Enter and clean data in evaluation project databases and spreadsheets. Logistics Coordinator: ChembeNyendwa-Banda Responsible for logistics, coordination, administrative support, and ensuring all aspects of the evaluation are carried out seamlessly. The Logistics Coordinator, in collaboration with the USAID/Zambia COR and Chemonics, will organize meeting space and materials, initial partner meetings, and site visit schedule and related logistics in advance of the in-country portion of the evaluation.    56 ANNEX E: LIST OF PERSONS CONSULTED 1. PERSONS CONSULTED FOR KEY INFORMANT INTERVIEWS NAME POSITION ORGANISATION ANNIE FIEDLER CHIEF OF PARTY CSH LINDA NONDE DEPUTY CHIEF OF PARTY CSH KELVIN CHILEMU RESEARCH, MONITORING & EVALUATION DIRECTOR CSH FLORENCE TEMBO MULENGA CAPACITY BUILDING DIRECTOR CSH PATRICIA NAWA FINANCE & ADMINISTRATION DIRECTOR CSH TODD JENNINGS DIRECTOR- PRIVATE SECTOR ENGAGEMENT CSH CHRISTINA WAKEFIELD TECHNICAL DIRECTOR CSH KIZITO M. NG'ANDU INFORMATION TECHNOLOGY SPECIALIST CSH MICHELLE HUNSBERGER COMMUNICATION SPECIALIST CSH KATHLEEN POER CHIEF OF PARTY ZISSP NANTHALILE MUGALA DIRECTOR OF TECHNICAL SUPPORT ZISSP VERA MBEWE COMMUNITY AND SOCIAL MOBILIZATION SPECIALIST ZISSP MPUNDU MWANZA BCC ADVISOR ZISSP BENSON BWALYA MONITORING AND EVALUATION TEAM LEADER ZISSP ESNART M. JUUNZA DICTRICT COORDINATOR ZISSP KUYOSH KADIROV DEPUTY EXECUTIVE DIRECTOR SFH NICHOLAS SHILIYA RESEARCH, MONITORING & EVALUATION DIRECTOR SFH CHARLES KALONGA DIRECTOR- PROGRAMS OPERATIONS SFH EDFORD G. MUTUMA EXECUTIVE DIRECTOR PPAZ 57 SOPHIE BAUMGARTNER PROJECT COORDINATOR PPAZ MULAKWA KAMULIWO DEPUTY DIRECTOR PUBLIC HEALTH AND RESEARCH - MALARIA NMCC KENAN M. NG'AMBI EXECUTIVE DIRECTOR PRIDE SOPHIE N. KAMWATA PROGRAM MANAGER PRIDE GEOFFREY N. CHIKUNJIKA EXECUTIVE DIRECTOR THANDIZENI ROMAN K. MUKENDI SENIOR PROGRAM MANAGER BROADREACH CHRISPIN CHOMBA COUNTRY REPRESENTATIVE- ZAMBIA SAFAID LIZZY CHANDA DIRECCTOR OF PROGRAMS AFYA MZURI KUNYIMA LIFUMELA BANDA PROGRAMS OFFICER NZP+ GEORGE SIKAZWE SENIOR HEALTH PROMOTION OFFICER MOH MATILDA NKASHI FINANCE & ADMINISTRATION MANAGER SAFAID TAMARA SIMAUWA PROGRAMS MANAGER SAFAID PAUL KALINDA HEALTH ADVISOR EU CAROLINE PHIRI DIRECTOR MATERNAL & CHILD HEALTH MCDMCH DOUGLAS HAMPANDE CIVIL SOCIETY SPECIALIST NAC RITA KALAMATILA IEC OFFICER NAC BWALYA MUBANGA M&E SPECIALIST NAC CHRISTINA MUTALE LEARNING CENTRE MANAGER CHAMP CHISOMO ZULU IT OFFICER CHAMP OSCAR MWANSA FINANCE DIRECTOR CHAMP PAULINE WAMULUME IEC OFFICER NMCC EDFORD G. MUTUMA EXECUTIVE DIRECTOR PPAZ SOPHIE BAUMGARTNER PROJECT COORDINATOR PPAZ KENAN NGAMBI DIRECTOR PRIDE. CHARLES BANDA NATIONAL PROGRAMME OFFICER UNFPA MWAKA SIAMUTWA PROGRAMME ASSISTANCE, COMMUNICATION UNFPA LOVEMORE MWANZA RESEARCH& EVALUATION SPECIALIST CSH    58 KAORU OZEKI ASSISTANT RESIDENT REPRESENTATIVE JICA PRISCILLA LIKWASI CONSULTANT JICA MWEEMBA CHIYALA District AIDS Coordination Advisor (DACA) NAC VINCENT KAMUTAMBO DATF SECRETARY NAC 59 2. FOCUS GROUP DISCUSSION PARTICIPANTS NAME SEX Name Sex Name Sex    60 61    62 63 ANNEX F: EVALUATION MATRIX Category Question Data Collection Method Question Response Summary Records Inspection KIIs Online Survey FGDs Systems & DQA Mini - Survey Results to Date To what extent is the activity on track to achieve its intermediate results and meet its life of activity targets? The team first inspected various results that were available in the project database for output and outcome achievements Various surveys that have been conducted either directly to assess the CSH project or not directly on the project but featuring question on any of the five thematic areas of Malaria, MCH, Nutrition, HIV/AIDS and FP/RH and conducted during the CSH project implementation period were reviewed This information was supplemental other data to answer this question The key informants were the implementers and stakeholders and therefore contributed significantly towards what CSH has been able to accomplish and why they have not been able to accomplish The broad spectrum KIIs enabled diverse opinions to be gathered to augment other field results to effectively explain the extent to which activities were on track to achieve intermediate results and meet life of activity targets These responses were qualitative and were therefore analyzed using MaxQDA Data from the on-line survey enabled quantifying of key informants perceptions on the extent to which activities were on track to achieve intermediate results, especially in areas where there were diverging views from key informants SPSS was used to analyze the quantitative aspect of this survey whiles MaxQDA was used to analyze the qualitative component The FGDs with Save Love Clubs and Organized Groups were used to discuss the extent to which activities were on track to achieve intermediate results and how implementation could be improved These responses were mainly qualitative and were therefore analyzed using the MaxQDA software The DQA was used to assess the extent to which CSH has supported GRZ to establish robust BCC and M&E system This DQA assessed the extent to which the system established was able to ensure data Validity, Reliability, Precision, Timeliness, Integrity, Accessibility, Confidentiality and data Security The findings on each of these data quality dimensions were analyzed using SPSS The mini-survey aided in identifying the achievements of CSH and the extent to which they were on track to achieving the community level behavior change indicators, including exposure to various campaigns and adverts, improvement in knowledge or otherwise, attitude and behaviors influenced by various campaigns In analyzing this data with SPSS, the team cross-tabulated various behavior indicators with exposure indicators to identify whether the behavior was as a result of the exposure For instance the analysis compared number of people who have been exposed to the malaria campaign advert and were using LLINs to those who were not exposed and yet use LLINs to see if the malaria campaign was having any significant effect on LLIN usage MaxDQA was also used to analyze reasons on why people have adopted certain behaviors or refused to change to inform further strategies by CSH The question on the extent to which activities were on track was a major part of the exercise and an embodiment of the other components. Each strategy used in this evaluation therefore contributed to answering this question. These strategies included records inspection, KIIs, online survey, FGDs, Systems & DQA, and Mini Survey. To what extent are the The various data The DQA was The DQA and 64 indicators and tools used to monitor and measure progress towards results adequate (especially in measuring the capacity of the GRZ and also message coverage and effectiveness)? collection tools, indicators in the M&E Plan, M&E protocol, databases, and available data were inspected and analyzed to assess their adequacy to measure progress This gave the basic answer to this question conducted on the GRZ, and CSH M&E systems and mainly responded to this question Through the DQA, the M&E Plan of CSH was evaluated comprehensively including the definition of various indicators The databases, data collection tools, M&E protocol, M&E staff capacity etc were also analyzed in the DQA The analysis was used to assess the extent to which each of these structures in the system established was able to ensure data Validity, Reliability, Precision, Timeliness, Integrity, Accessibility, Confidentiality and data Security The findings on each of these data quality dimensions were analyzed using SPSS records inspection were the main approaches to answer this question. What improvements can be made to better capture progress? Upon identifying the challenges and limitations through the DQA, the team recommended on how the system could be strengthened to better capture progress in a timely manner to inform strategic decision taking The DQA informed the needed improvement. What are the challenges to implementation and what can be done to improve the chances of the activity Various project reports were reviewed These reports stated some of the challenges Challenges found in The key informants, especially the implementers, knew much about the challenges to implementation Online survey of key informants also contributed significantly toward identifying the challenges to implementation The FGDs were used to identify further challenges to implementation from community members’ point of view This Records inspection, KIIs, online survey of key informants and 65 achieving its intended results and meeting its life of activity targets? the reports were the starting point for identifying the key challenges to implementation Other key informants also contributed significantly towards identifying the challenges to implementation KIIs were therefore used to answer this question Responses to this question were analyzed using MaxQDA Analyzing this with SPSS enabled the team to identify the proportion of KIs who think a particular issue is a challenge and the premium placed on that challenge For instance if 90% of respondents indicated a particular factor was a challenge but when asked how intense that challenges is, these 90% indicated that it is a low priority challenge That was treated differently from a factor that 70% considered a challenge but categorized by these 70% as a very high priority challenge In addition to analyzing the quantities with SPSS, MaxQDA was used to analyze the qualitative question was analyzed using MaxQDA the FGDs were therefore the main sources from which this question was answered. This ensured that challenges identified were not limited to that of the implementers but also the beneficiaries. 66 Message Exposure and Effectiveness To what extent has the activity reached intended audiences in all parts of the targeted geographical areas, in particular rural areas, across Zambia with IEC/BCC messages in each of the five health intervention areas (HIV/AIDS, malaria, FP/RH, MCH and nutrition)? Records available on the proportion of the target population reached with each of these campaigns were inspected as the bases for developing questions for the FGD and the mini survey The presence of these records was helpful The FGDs were used to discuss the various campaign activities with the Save Love Clubs and Organized Groups to know the extent to which these activities reached them Whiles these were mostly PLHIVs, they were also targets for the other four campaigns and therefore very appropriate to access the reach of the Malaria, FP/RH, MCH and Nutrition campaigns The mini survey interviewed only Zambian residents between the ages of 15-49 years and therefore was very effective in assessing the exposure level of the various campaigns The survey assessed output, outcome and impact indicators on each of the five intervention areas For instance with Malaria, the survey assessed the proportion that had seen/heard various malaria campaign activities, the proportion that slept under LLINs and the frequency with which people slept under LLINs SPSS was used to conduct cross tabulation to identify if people who were exposed to the campaign were more likely to sleep under LLINs frequently Further analysis was conducted to identify the effect of each campaign on gender, and rural – urban dwellers Mianly the FGDs and the mini-survey were used to answer this question. This survey covered as many of the indicators in the M&E plans as practicable; whiles ensuring that the questionnaire was not too long. The results from the survey were compared with the targets set by CSH in their M&E plan. Are the messages appropriate for their intended audiences (do they resonate)? Are the intended audiences able to recall and understand the messages? The FGD analyzed the appropriateness of the messages Questions assessed included (1) Did the audiences understand the messages (2) Were the message culturally appropriate and welcoming (3) Were the channels of communication appropriate (4) Were the languages being used understandable and able to reach the target audience (5) What made them remember the Questions in the mini survey assessed the appropriateness of the messages including the questions asked in the FGD in response to this question SPSS was used to analyze the quantitative component whiles MaxQDA was used to analyze the qualitative component The responses from the FGD combined with those from the mini survey provided the strongest of answers to this evaluation question. 67 messages? And many more such issues The responses were analyzed using the MaxQDA software Capacity Building To what extent has the activity built the capacity of GRZ to implement IEC/BCC activities on its own? The team inspected the records of CSH and GRZ to know the level to which the capacity of GRZ had been built to implement IEC/BCC activities The team interviewed both CSH staff and GRZ staff to know the extent to which the capacity of GRZ had been built Due to the tendency of subjectivity, the team augmented these responses with the systems assessment and DQA The online survey was another source from which answers to this question were sought This helped quantify diverging views The DQA accessed the extent to which capacity building for GRZ had enabled them to put the necessary structures in place in terms of building strong IEC/BCC systems as well as M&E systems The KIIs, records inspection and online survey were therefore used to assess the output and outcome indicators whiles the Systems and DQA was used to assess the impact indicators. By this means, this question was effectively answered. What are the gaps in terms of capacity building that need to be addressed to ensure that the GRZ can implement quality national health communications campaigns on its own? Various capacity building reports were inspected, reviewed and analyzed and gaps identified These gaps were the starting point to accessing information to answer this question KIIs were used to identify the established gaps in capacity building Since the key informants included donors, implementers and beneficiaries, they were able to contribute to answer this question from diverse backgrounds with different lenses, giving the team a comprehensive answer to this question The online survey was used to identify the gaps in capacity building KIs indicated what they saw as the gaps and GRZ beneficiaries also indicated why they were not able to effectively handle certain aspects This enabled the team to identify more gaps The DQA established the real gaps Whiles certain aspects were not be seen by implementers or beneficiaries as gaps, the DQA identified each of them through the inspections and analysis of their records, IEC/BCC plans, databases, data collection tools and the entire system The combination of records inspection, KIIs, online survey and DQA assessed the gaps in capacity building and ensured the answering of this evaluation question. What else does GRZ need in order to independently plan, KII identified what CSH staff thought GRZ needed, what The online survey also established more of the things GRZ needed to The systems analysis and DQA identified the areas that GRZ The combination of 68 implement, manage, and evaluate national health campaigns? GRZ staff thought they needed as well as what other stakeholders thought GRZ needed in order to independently plan, implement, manage and evaluate national health campaigns independently plan, implement, manage, and evaluate national health campaigns This was analyzed with SPSS and weighted to ensure that recommendations in the final report were based on the most effective and important needs that could catalyze achievement of results was lacking in and that were required to make them independent in planning, implementing, managing, and evaluating national health campaigns This enabled the evaluation team to identify and make appropriate recommendation on what GRZ needed to operate independently suggestions from KIs through the KIIs, and online survey and findings from the DQA provided the most effective answer to this question. To what extent is the project adapting to the current changes in the GRZ structures particularly those related to the creation of the Ministry of Community Development, Mother and Child Health and the re-organization of the National HIV/AIDS/STI/TB Council? The strategies developed by the project to enable it adapt to current changes in the GRZ structures were inspected and further assessed to identify the extent to which the strategies were being implemented The interviews with CSH staff, and GRZ staff also enabled the team to identify the changes taking place as the project adapts Other stakeholders had the opportunity to share how they thought the project was adapting to these changes The online survey also sought more information on the adaptation that the project was undergoing as a result of these changes and the extent to which the project was undergoing this adaptation successfully The combination of records inspection, KIIs and the online survey along with records inspection were used to answer this question. Evidence – based Planning and Implementation In terms of communications products/tools, are there mechanisms in place to collect feedback from end users and if so, is this feedback incorporated into the future design of products or used to inform decisions about current products? All documents developed to collect feedback on communication products/tools were inspected These included questionnaires, data collection forms, or strategy plans on how this feedback was collected The KIIs be used to answer this question Interviewing of some key informants including the M&E Specialists and BCC Advisors of CSH and other implementation partners as well as GRZ technical staff shed more light on the strategies kept in place to collect feedback on communication products/tools The DQA also assessed mechanisms for feedback and the extent to which the systems support these mechanisms to ensure that the absence of an individual does not result in malfunctioning of the system but would continue to monitor feedbacks for whoever takes over the management of the system The DQA, records inspections and interviews of key informants were the main sources of information to answer this evaluation question. 69 Sustainability Are IEC/BCC activities likely to continue without further USG investments and if not, what investments or approaches would better promote sustainability? Sustainability is a fundamental pillar in every project CSHs plans and strategies to ensure that all the structures they were building were sustainable were inspected and analyzed to identify their feasibility KIIs also established what these informants thought of the sustainability of the project as well as what they thought could be done to sustain the project This online survey identified if the BCC systems established by CSH were sustainable and came out with suggestions on how they could be made more sustainable FGDs enquired from Save Love Clubs and Organized Groups about the community level sustainability and how community structures built by CSH could be sustained Systems Assessment and Data Quality Assessment revealed how effective the structure built for BCC planning, implementation, management and monitoring is and their level of sustainability This enabled the team to recommend on effective ways of sustaining the system The combination of records inspections, KIIs, online survey reports, FGDs, Systems Assessment and DQA all contributed to answering this very important evaluation question.    70 ANNEX G: ORAL CONSENT SCRIPT for MINI SURVEY Study Title: Communication Support for Health (CSH) – Mid Term Evaluation Study Team Leader: Joseph Sineka Limange - IBTCI Address: 8618 Westwood Center Drive, Suite 220. Vienna, VA 22182 USA Introduction Hello. My name is…………………………. and I represent the International Business & Technical Consultants Inc. who is conducting this research study. We are asking you to take part in this study. In order to be sure that you are informed about being in this research study, I need to read to you this consent form. At the end, I will ask you to tell me if you agree to be in the study. We will give you a copy of this form. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. Reason for the Research The purpose of this study is to learn about family planning, maternal and child health, HIV/AIDS, Nutrition and Malaria. Results from this study will help us to improve activities aimed at improving health. It will also help us learn about how well our project is working. Why you are being asked to participate You are being asked to take part in this study because you were randomly selected to participate and you are between 15-49 years. About 240 individuals will take part in this research in Eastern, Lusaka, Southern, Central, Luapula and Copperbelt provinces. Your Part in the Research If you agree to be in the study, you will be asked to answer questions about family planning, maternal and child health, HIV/AIDS, Nutrition and Malaria. You would not need to provide us with your contact information. Your part in the research will last about 25 minutes. Possible Risks There are no physical risks in participating in this study. There is a very small risk to your privacy; however we will do everything we can to keep your information private. That is why we would not like to take your name or contact to ensure no one can trace information provided by you. Some of the questions may make you feel embarrassed or uncomfortable. You do not have to answer any question that you do not want to. You are also free to end the interview at any time. Possible Benefits There is no direct benefit to you for taking part in this study. However this information will help to better understand issues related family planning, maternal and child health, HIV/AIDS, Nutrition and Malaria. This information will help us to plan better health communication programs that will help people living in your province. Payment 71 You will not be paid to be in this study. However, if you do take part in the study, you will be helping us to plan programs to improve health in your province. If You Decide Not to Be in the Research Nothing will happen to you if you choose not to be in the study. You are free to decide not to participate. Protecting Confidentiality We shall not take your name or contact information and we shall not directly contact you in relations to this research after this interview. You will never be named in a report. If You Have a Problem or Have Other Questions, please call: Mr. Joseph Limange at 09 737 090 or by email at Jlimange@ibtci.com VOLUNTEER AGREEMENT The above document describing the benefits, risks and procedures for the mid- term evaluation of the Zambia Communication Support for Health (CSH) Project has been read and explained to me. I have been given an opportunity to have any questions about the research answered to my satisfaction. I agree to participate as a volunteer. PERMISSION TO PROCEED Do you agree to participate in the interview? Yes / No I have read the consent form completely before the study participant and the study participant voluntarily agreed to participate in the study." Print name of Person Obtaining Consent Signature of Person Obtaining Consent Date    72 ANNEX H: KEY INFORMANT INTERVIEW GUIDE Province: District: Organization: Name: Sex: Position: Date: Time: Venue: Interviewer: Introductory remarks Thank you for agreeing to talk with us, and for your participation in this evaluation. International Business & Technical Consultants, Inc. (IBTCI) has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out this midterm performance evaluation of the USAID funded Communications Support for Health (CSH) project in Zambia. The purpose of this evaluation is to assess the achievements of the CSH project to date, the challenges it is facing and how these can be overcome. We believe that you are in a good position to tell us about your organization and what it is doing in relation to the CSH program, hence this interview. We anticipate the interview will last about an hour or less and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names Introduction Please give us a background of your organization and what it does? What do you know about the CSH project? Capacity Building for GRZ What do you think about the extent to which CSH is building the capacity of GRZ staff to enable them implement IEC/BCC activities? Are you aware of any such capacity building? In which areas of operation do these activities cover? What approach is CSH using in building the capacity of GRZ on IEC/BCC? How effective are these approaches? What are the challenges and limitations of these approaches? Are the capacity building activities of CSH enabling GRZ to achieve anything special on IEC/BCC other than what they already know? What are some GRZ achievements on IEC/BCC that you can attribute to the capacity building approaches of CSH? How do you think CSH can improve on the quality of capacity building to achieve the desired result? Gaps to Capacity Building 73 What are some other things that you think should be included for GRZ to build its capacity in IEC/BCC? What capacity building approaches do you think should be implemented? GRZ Independence in Conducting IEC/BCC Activities How do you think GRZ can be supported to enable them to effectively plan IEC/BCC activities in the near future without any support? How do you think GRZ can be supported to enable them to effectively implement IEC/BCC activities in the near future without any support? How do you think GRZ can be supported to enable them to effectively manage IEC/BCC activities in the near future without any support? How do you think GRZ can be supported to enable them to effectively evaluate IEC/BCC activities in the near future without any support? Support to MCDMCH Are you aware of the creation of a new ministry called the Ministry of Community Development, Mothers and Child Health? What do you think is the responsibility of this Ministry? What adjustment did you have to make with the introduction of this new Ministry? What support do you expect CSH to be providing to this new Ministry in regards to IEC/BCC planning? implementation? management ? monitoring and evaluation? Do you think CSH is providing this support? How effective would you say the CSH support to this Ministry is? What challenges are the new Ministry facing with IEC/BCC activities? What recommendation would you suggest to address these challenges? Feedback (only CSH and GRZ) Do you have a channel for accessing feedback on your IEC/BCC activities? What is this channel? How does it work? What is the frequency of accessing this feedback? What does the feedback you have collected in the past indicate about each of your campaigns? Do you think your mode of collecting feedback is enough? What are the strengths of the method you use to collect feedback? What are the weaknesses of the method you use? How can this method be improved? Does your organization use the feedback to inform decisions? Why do you think your organization uses the feedback effectively or does not use the feedback effectively? Can you give some instances when such feedback has been used to inform program or management decisions? Sustainability With each of the channels you mentioned, do you think you would continue to receive the information even when the CSH project ends? Where would you receive this information/education? How would you receive it?    74 Would this group continue to exist even when there is no support for you from any project? Would you know another source from which you would get support? Do you think the GRZ is in the position to continue supporting you if there are no funds from CSH? Are there other means of support for IEC/BCC activities that you know of? Could you access these means? Extent to which activities are on track Do you think CSH’s activities are on track to effectively building the capacity to GRZ partners to ensure they are able to implement and monitor IEC/BCC activities? Can you explain why you say so? Do you think the extent of support to GRZ by CSH has put them on the track to being able to independently plan, implement, manage and monitor their own BCC activities? Can you give some reasons to support your answer? Do you think CSH support to MCDMCH is on track to enabling them to function effectively and independently regarding the implementation of IEC/BCC activities, without relying on others for support? Can you explain what makes you come to this conclusion? Do you think the design of CSH IEC/BCC activities would ensure sustainability even when USG support comes to an end? Are the structures and design of IEC/BCC activities in GRZ such that it would ensure sustainability even when CSH stops supporting them? 75 ANNEX I: SITE VISIT PROTOCOL Pre-Interviewing (FGDs) Contact key people in the selected districts and local communities. Give an overview of what the mid￾term evaluation is all about and ask them to help in mobilizing attendees. Provide guidance to contact persons on how to mobilize individuals for interviews. Make prior arrangements for a venue of the meeting and communicate to participants the date and time of the meeting. Work with the contact person to make sure that the venue is ready for the interviews Prepare all necessary materials for the interviews including films, note pads, pens, recorders etc. Interview Session (FGDs) Introductions of evaluation team and ask participants to introduce themselves. Welcome all participants and explain the purpose of the interview Provide guidance on how the interviewing will be conducted. Assure attendees that all the information given will be treated as confidential. Obtain consent from all participants. In case of recording, explain why it is important. Start the discussion and facilitate to ensure that participants remain on the topic. Probe to ensure participants address the issues. Ensure that all the participants are given an opportunity to make contributions. Be impartial and guard against advancing personal opinions as the facilitator. After the FGD Thank the participants for taking time to be part of the discussion. Provide a brief refreshment.    76 ANNEX J: FOCUS GROUP DISCUSSION GUIDE Province: District: Interviewer: Number of Participants: Male: Female: Date: Time: Venue: Name of Group: Type of Group: Introductory remarks Thank you for agreeing to talk with us, and for your participation in this evaluation. International Business & Technical Consultants, Inc. (IBTCI) has been contracted by the United States government through the United States Agency for International Development (USAID) to carry out this midterm performance evaluation of the USAID funded Communications Support for Health (CSH) project in Zambia. The purpose of this evaluation is to assess the achievements of the CSH project to date, the challenges it is facing and how these can be overcome. We believe that you are in a good position to tell us about your organization and what it is doing in relation to the CSH project hence this interview. We anticipate the FGD will last for about an hour and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and will be coded and reported without names. Extent to which activities reached intended audiences in all parts of the targeted geographical area Safe Love Campaign What do you know about the communications campaign on HIV/AIDS? What is the name? Who is the intended target audience? What is the purpose of the campaign? How are you benefiting from the campaign? What is it encouraging people to do? Do people in this community know about the campaign? What do they think about it? 77 Do they appreciate the recommendations from the campaign? Is the campaign having the intended outcome, where people are reducing the number of sexual partners they keep at the same time? Is the campaign having any effect on the correct and consistent use of condoms? Is the campaign having the intended impact of causing more people to be tested for HIV and receiving their test results? STOP Malaria Campaign What do you know about the campaign on Malaria? What is the name of the campaign? Who is the intended audience? What is the purpose of the campaign? How are you benefiting from the campaign? What is it encouraging people to do? Do people in this community know about the campaign? What do they think about it? Do they appreciate the messages from the campaign? Is the campaign causing people to sleep under ITN consistently? Are people seeking early treatment for malaria because of the campaign? Do people in this community request a malaria test before treatment? Do the health workers use a Rapid Diagnostic Test (RDT) to diagnose malaria before treatment? Do women in this community take Intermittent Preventive Treatment for pregnant women (IPTp) to prevent malaria because of the campaign? Mothers Alive Campaign What do you know about the campaign on safe motherhood? What is the name of the campaign? Who is the intended audience? What is the purpose of the campaign? How are you benefiting from the campaign? What is it encouraging people to do?    78 Do people in this community know about the campaign? What do they think about it? Do they appreciate the messages from the campaign? Are the campaign causing people to seek family planning services? Are women in this community going for ANC early enough because of the messages from the campaign? How many times do women in the community go for ANC before giving birth? Do the women use birth plans? Do they deliver at the health facility? And after delivery do they go back to the hospital for further attention? Appropriateness of the Message: Interviewer, show the various videos to the group before this discussion How did you hear of the messages discussed above? In your opinion, were the channels effective? Which of the channels do you think is the most effective? What channel would you have preferred to get to message to you? and which channel is your second most preferred choice? In your opinion, have the messages been effective? Are they easy to understand? Are the messages appealing to the target audiences? Do the messages capture the perceived issues, concerns, attitudes, and practices that can facilitate behavior change? From the cultural perspective, have the messages been appropriate? If not, what do you consider as inappropriate in the identified messages? What are your recommendations on how to make some of the messages more culturally appropriate? Considering the numerous languages in Zambia, what language would you prefer to be used in transmitting messages for everyone’s benefit? Sustainability With each of the channels you mentioned, do you think you would continue to receive the information even when the CSH project ends? Where would you receive this information/education? How would you receive it? 79 Would this group continue to exist even when there is no support for you from any project? Would you know another source from which you would get support? Do you think the GRZ is in the position to continue supporting you if there is no funding or technical assistance from CSH? Challenges What challenges do you face in receiving the messages? What challenges do you face in disseminating the messages you receive? What challenges have you seen in knowledge and attitude towards the following: VCT, PMTCT and treatment seeking, multiple concurrent partners, and use of condoms? Modern contraception, early initiation of antenatal care, facility-based delivery and follow-up care after delivery family planning? Use of mosquito nets, early treatment, RDT, uptake of IPT? Do you remember any challenge that you would like to add? Extent to which activities are on track What do you think about the STOP malaria campaign? Would it be able to achieve its intended objectives? Why do you say so? What do you suggest should be done to enable it achieve its objectives effectively? What do you think about the Safe Love campaign? Would it be able to achieve its intended objectives? Why do you say so? What do you suggest should be done to enable it achieve its objectives effectively? What do you think about the Mothers Alive campaign? Would it be able to achieve its intended objectives? Why do you say so? What do you suggest should be done to enable it achieve its objectives effectively? Finally, would you say the activities are on track to achieving their intended objectives? What do you suggest should be done to enable it achieve its objectives effectively    80 ANNEX K: DATA QUALITY ASSESSMENT QUESTION GUIDE The data quality assessment (DQA) shall be based on structures and systems kept in place to ensure effective data collection, analyzing, storage and reporting. Structures to be assessed include: Monitoring & Evaluation Plan Monitoring & Evaluation Protocol Data collection Tools Databases Monitoring & Evaluation trainings for data collectors on tools and protocols. In assessing these structures, the teams shall examine how these structures answers the following questions: Validity: Are the right indicators being measured? Are the indicators clearly defined? Do the data collection tools disaggregate the data? Reliability: Is the same method being used to collect and analyze the same data? Are different groups collecting the same data using the same data collection tools? Is the database efficient enough to produce accurate data? Is there an established system to reduce errors? Are the data collectors trained on the tools and protocols to enable them collect reliable data? Precision: Do other methods of counting result in the same quantities with the database? Do the data collection tools eliminate double counting? Are data precise enough to enable decision making at policy and operational levels? 81 Are the data managers trained to enable them effectively manage the database(s)? Integrity: Are there measures in place to ensure that management cannot manipulate the data? Is the data management system such that only the right person(s) can add, delete or edit the data? Timeliness: What is the frequency of data collection? Does the time of reporting make data available for CSH and GRZ reporting? Does the frequency of reporting make data available for GRZ and CSH management decision making at the right time? Accessibility: Is the data available to management and other staff (including USAID staff) for decision making? Is the process of analyzing and retrieving the data efficient to ensure that management and staff can have easy access to various analyses promptly?    82 ANNEX L: BCC ASSESSMENT GUIDE Province: District: Interviewer: Number of Participants: Male: Female: Date: Time: Venue: Name of Organization: This BCC System assessment tool is designed to guide team members to assess the extent to which the BCC systems are functioning to ensure the needed behavior change and sustainability as set forth by the program implementers. The team will employ participatory monitoring and evaluation, (a process of evidence-based learning for action in collaboration with stakeholders) and will aim to improve an understanding of results while also strengthening local capacity, institutional development, as well as sustainability of efforts. BCC Planning and Design How did you plan the design of your BCC system? What data did you gather and analyze in order to plan the design of the BCC system? How did this data influence the design of the BCC system? Who were the stakeholders mapped during planning and design of your BCC? How did you identify them? How did you determine their roles? What competing messages from other donors and corporate entities did you have to contend with? Are they consistent or contradictory to your messages? Who are the stakeholders you established partnerships with during planning and design? How did you establish partnerships with these stakeholders? Are they still working with you in some way? How? Do you set clear behavioral targets during planning and design? Can we see these behavioral targets? (Get a copy.) What categories of segmentation did you identify among your target audience? Do you deliver one message about a topic to everyone in a particular segment? 83 If not, why not, what do you do about it? and how do you do it? Please show any materials that explain what you’re talking about. What do you know about key elements of BCC design such as appeals? Barriers? Benefits? Enablers? Do you use those in the design of your messages? In which messages did you use one or more of these key elements? How did you use these elements in the message design? Are messages reviewed by technical staff to ensure accuracy of health information? What is the role of technical staff in relation to developing the messages? Give some examples of the discussions that arise when messages are reviewed/? How did these recommendations influence the design? Give examples. Are channels of communication (individual, group, radio, drama, etc.) selected with input from target audiences? Are their preferences taken into account? If yes, how? How have the target audience preferences influenced the BCC design in the past? BCC Program Implementation Do you use multiple communication strategies and channels in your programs? What are the communications strategies and channels used in your program? Why and how was each of them used? What levels of the society do your communication interventions seek to influence? Have your campaigns had influence at different levels of society (individual, family, community, district, provincial and, national) ? Was this intentional? Give examples. Have there been instances when the campaign has failed to influence the desired target? What are some of these instances? Do you evaluate your communication approaches and materials? Under what instances have you conducted such evaluations? What methods do you use to conduct these evaluations? What are some of the findings from these evaluations?    84 What do you do with the findings from the evaluations? What IEC materials and tools (e.g. manuals, flip charts, counseling cards, scripts) are available to providers to support mass community and interpersonal communication interventions? Is there a plan to ensure their continuous production? Is there a channel for their distribution to ensure that they get to the grass root level? Is there a channel to access feedback on these materials? Who has been trained on IEC/BCC in this organization? Can we inspect the list of all those trained? Where did they get the training from? Who were the trainers? Do you conduct assessment of staff BCC competencies? How often do you conduct such assessment? Show me the competencies you use for assessment. What has been some of your findings on staff BCC competence? Has this influenced the organization’s policy on BCC capacity building for staff in any way? Is there a plan for strengthening of staff’s BCC competencies (basic BCC training, on-the-job training, etc.)? Is it being implemented? Can we inspect this plan(s)? What criteria do you use to select your BCC providers? Is there a written job description for BCC providers, including volunteers? What are the duties of BCC providers – permanent staff? Please provide a copy. What are the duties of BCC providers – volunteers? Please provide a copy. Does the structure of the BCC plan allow or require technical supervisors to visit BCC providers? How frequently does the plan require supervisors to visit BCC providers? What structures are in place to ensure that this plan is implemented? Is there a supervisory checklist to guide supervisory visits? If yes, please provide a copy. What indicators are on this checklist? Does it include indicators related to BCC service delivery? How is the information gathered in the checklist used? 85 ANNEX M: MINI SURVEY QUESTIONNAIRE Mini Survey Questionnaire Interviewer’s Name: Date: Province: Introduction My name is ___________ and I am from IBTCI, an independent research agency. We are currently conducting research on peoples’ perceptions on various issues. May I spend about 25 minutes with you going through a questionnaire? The information you give us will be treated with utmost confidentiality and that is why we would not ask you for your name. Section 1: Biographical Information of Respondent Gender: M/F District: Community: Please circle one: Urban/ Rural Marital Status (Please circle one): 1.Single 2. Married 3. Divorced/ Widowed/ Separated 4. Living together but not married Age Group: a. 15-19 b. 20-24 c. 25-29 d. 30-34 e. 35-39 f.40-44 g. 45-49 MNCH Q1. Have you/partner given birth in the last 12 months? (Please circle Yeas or No) one: Yes No (If no, please go to Q6) Q2. Did your wife go for antenatal care before delivery? Yes No (If no, please go to Q6) Q3. How old (months) was the pregnancy before she went the first time? a. One b. Two c. Three d. Four e. More than four Q4. How many times did you/partner go for ANC services during pregnancy? a. Once b. Twice c. Three times d. Four times e. More than four Q5. Did you/partner take any drug to prevent malaria during pregnancy? Yes No MALARIA    86 Q6. Has your house been sprayed to prevent mosquitoes over the last 12 months? Yes No Q7. Do you have an Insecticide Treated Net (ITN) in your household? Yes No Q8. If you have any children less than 5 years, did they sleep under an ITN last night? Yes No Not- Applicable Q9. Did you sleep under ITN last night? Yes No FP/RH Q9. Are you/partner using any modern contraceptive method to delay child birth or space your children? Yes No (If the answer is no, please go to question 11) Q10. What modern contraceptive method are you/partner using? Multiple Responses Allowed. Probe a. Sterilization b. Pill c. Condoms d. Injectable e. Lactational amen. f. Withdrawal g. other methods Q11. Over the past 12 months, have you discussed contraceptive use with your partner? Yes No Not-Applicable HIV Q12. Over the past 12 months, have you had more than one sexual partner? Yes No Q13. Have you tested and received your HIV results over the last 12 months? Yes No Q14. Do you know the HIV status of your partner(s)? Yes No Q15. Do you use condoms correctly and consistently with all sexual partners including regular, long term and trusted partners? Yes No Not-Applicable Q16. Do you demand that your partner(s) use condom at all times? Yes No Not￾Applicable Q17. Do you think being circumcised could reduce the likelihood of contracting HIV? Yes No Exposure Q17. Thinking back over the past 12 months, have you heard of the following campaigns: “Safe Love” campaign? Yes No 87 “Mothers Alive” campaign? Yes No “Stop Malaria” campaign? Yes No Q18. What are the following campaigns about? Please do not mention the answers i)Safe Love: a. Malaria b. MNCH c. HIV d. Nutrition. e. FP/RH f. Others g. Don’t Know ii)”Mothers Alive”: a. Malaria b. MNCH c. HIV d. Nutrition. e. FP/RH f. Others g. Don’t Know iii)”Stop Malaria”: a. Malaria b. MNCH c. HIV d. Nutrition. e. FP/RH f. Others g. Don’t Know Q19. Are you aware of the following health education opportunities? (i): 990 Talkline Counselors: A group of counselors who respond to issues on a host of health areas when you dial 990 on any network? Yes No (ii) Community Facilitators who educated people on a host of health issues? Yes No (iii) Change CHAMPIONS: Community opinion leaders who have devoted part of their time to educate people on selected health thematic areas? Yes No Q20. Have you benefited from education provided by any of the following groups during the past 12 months? (i): 990 Talkline Counselors: A group of counselors who respond to issues on a host of health areas when you dial 990 on any network? Yes No (ii) Community Facilitators who educated people on a host of health issues? Yes No (iii) Change CHAMPIONS: Community opinion leaders who have devoted part of their time to educate people on selected health thematic areas? Yes No    88 ANNEX N: ONLINE SURVEY QUESTIONNAIRE CSH Mid Term Performance Evaluation You have been contacted to complete this survey because of your contribution to the health sector in Zambia and/or relationship with the United States Agency for International Development (USAID) financed Communications Support for Health (CSH) project. USAID has contracted the International Business and Technical Consultants Inc. (IBTCI) to conduct a mid-term performance evaluation of the CSH project. You might have been interviewed by the team already. This questionnaire is of great importance to the evaluation team and shall enable the team to quantify some of the answers you have already given. If you have not been interviewed by the team, this is an opportunity to get your input into the evaluation. The questionnaire should not take more than 10 minutes to complete. Your responses will treated with utmost confidentiality. Data will be aggregated across all respondents and reported only in summary statistics. Your voluntary participation in the survey is vital to ensuring the quality of the evaluation and will be used to inform the future of CSH. Thank you for your participation. Joseph Sineka Limange Team Leader/Evaluation Specialist CSH Project Evaluation International Business & Technical Consulting, Inc. Jlimange@ibtci.com +233 20 13 555 37 PLEASE NOTE: In the context of this survey, “GRZ” refers to three primary agencies – the Ministry of Health (MOH), National Malaria Control Centre (NMCC) and the National HIV/AIDS/STI/TB Council (NAC) 1. How best would you describe your organization and its relationship with the CSH project or GRZ? a. CSH Project Staff b. GRZ c. Other USG Project d. CSH Sub Grantee e. GRZ health development partner 2. Are you aware of CSH capacity building programs for GRZ? a. Yes b. No c. Not Sure 3. What approaches do you know CSH has used in building the capacity of GRZ? Multiple answers allowed. a. Workshops b. Mentoring c. Coaching d. Don't know Other (please specify) …………………… 4. Currently, how would you rate GRZ ability to PLAN IEC/BCC activities without any support? a. Very Low b. Low c. Medium d. High e. Very high f. Don’t know 89 5. Currently, how would you rate GRZ ability to IMPLEMENT IEC/BCC activities without any technical support? a. Very Low b. Low c. Medium d. High e. Very high f. Don’t know 6. Currently, how would you rate GRZ ability to EVALUATE IEC/BCC activities without any support? a. Very Low b. Low c. Medium d. High e. Very high f. Don’t know 7. Currently, how would you rate GRZ ability to MANAGE IEC/BCC activities without any support? a. Very Low b. Low c. Medium d. High e. Very high f. Don’t know 8. What are some IEC/BCC capacity gaps in GRZ that you expect CSH to address to ensure that GRZ can implement quality national health communication campaigns? ……………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………… …………………………………………………………………………………………………………… ………………………………………………….. …………………………………………………………………………………………………………… …………………………………………………. Please indicate how best you agree with the statements from Q9 ­ Q15. 9. CSH collects information from the end user to inform the design of various IEC/BCC materials and campaigns. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree f. Don’t know 10. CSH pretests IEC/BCC materials, and adverts after they are developed to get inputs from end users before producing the final version. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree f. Don’t know 11. GRZ assesses feedback from end users on their IEC/BCC activities after implementation. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree f. Don’t know 12. GRZ has a system for assessing feedback from end users on their IEC/BCC activities. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree f. Don’t know 13. GRZ uses feedbacks from IEC/BCC monitoring to inform program level and management decisions on the implementation of on­going programs. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree    90 f. Don’t know 14. GRZ uses feedbacks from IEC/BCC monitoring to inform the design of new communication campaigns. a. Strongly agree b. Disagree c. Neither agree nor disagree d. Agree e. Strongly Agree f. Don’t know 15. How confident are you that GRZ shall continue IEC/BCC activities without further CSH support? a. Not at all b. Low c. Medium d. High e. Very High f. Don’t know 16. Please list three major challenges that can prevent GRZ from planning, implementing, managing and monitoring IEC/BCC activities on its own without further CSH support. a. Challenge 1: …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… ……… b. Challenge 1: …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… ……… c. Challenge 1: …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… ……… 17. What role can the following organizations play in addressing the challenge(s) you mentioned in Q16 above? a. CSH …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… ……… b. USAID …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… ……… c. GRZ …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… 91 …………………………………………………………………………………………………………… ……… d. Other Development Partners …………………………………………………………………………………………………………… .. …………………………………………………………………………………………………………… …………………………………………………………………………………………………………… …… 18. Do you have any other comments/contributions you wish to make regarding the CSH project? ………………………………………………………………………………………………………… …………………………………………………….…………………………………………………… …………………………………………………………………………………………………………… …………… Thank you very much for your time and contribution.    92 ANNEX O: LIST OF REFERENCES Chemonics International, (2012), “Communications Support for Health (CSH) – Quarterly Report, April-June, 2012.” Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF International and the Manoff Group. Chemonics International, (2012), “Communications Support for Health (CSH) – Quarterly Report, July - Sept, 2012”. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF International and the Manoff Group. Chemonics International, (2011), “Communications Support for Health (CSH) - Quarterly Report, April-June, 2011. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2012), “Communications Support for Health (CSH) – Quarterly Report, Jan-March, 2012. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF International and the Manoff Group. Chemonics International, (2011), “Communications Support for Health (CSH) – Quarterly Report, Jan-March, 2011. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2011), “Communications Support for Health (CSH) – Quarterly Report, Oct- Dec, 2011. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2011), Communications Support for Health (CSH), mid- term Self Assessment Results, Lusaka. Developed under terms of contract no: GHS-1-007-00004-00, ORDER NO. -1-05-07-00004 implemented by Chemonics International in association with ICF MACRO and the Manoff Group, 2012. Chemonics International, (2010), “Communications Support for Health (CSH) – First Quarterly Report Year 1, July -Sept, 2010. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS￾I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group, 2010. Chemonics International, (2011), “Communications Support for Health (CSH) – Quarterly Report, July -Sept, 2011. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2010), “Communications Support for Health (CSH) – Second Quarterly Report Year 1, October -December, 2010. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. 93 Diop, F. Seshamani, V. Mulenga, C. (1998), “Household Health Seeking Behavior in Zambia”. Technical Report No. 20. Partnerships for Health Reform, Abt Associates Inc. Bethesda, Maryland in collaboration with Development Associates, Inc. Harvard School of Public Health, Howard University International Affairs Center, University Research Corporation, 1998. USAID Washington, (2011), “Evaluation- Learning from experience” USAID Evaluation Policy, Washington DC. Chemonics International, (2011), “Communications Support for Health (CSH) – 2012 Workplan, November, 2011”. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2011), “Communications Support for Health (CSH) – 2011 Workplan, December, 2010, Revised April, 2011”. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International, (2010), “Communications Support for Health (CSH) – Year 1(5 Month) Workplan, August - December, 2010”. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. McCann Healthcare, (2011). “Short-Term Marketing Plan” CHAMP–990 TALKLINE. McCann Healthcare, (2011). “Short-Term Marketing Strategy”- AFYA MZURI – DZIWANI RESOURCE CENTER. Danya International, (2011). “CHAMP IT Audit Report.” Chemonics International Inc. (2012). “Strategic Activities Fund Manual” Zambia Communications Support for Health Program (CSH), Developed under terms of task Order No. GHS-I-05-07-0004, Contract No. GHS-I-007-00004-00. Chemonics International Inc. (2012). “National Malaria Prevention and Surveillance, Monitoring and Evaluation Plan 2011- 2015”. National Malaria Control Center. Simbao, K. (2011). “National Monitoring and Evaluation Plan (2011–2015)”, National HIV/AIDS/STI/TB Council. Chemonics International Inc. (2011). “Zambia Behavioral and Social Change Communication - Project Task Order GHS-I-05-07-00004 (Under TASC3 IQC no. GHS-I-00-07-00004-00).” . Chemonics International Inc. (2012). “Program Campaign and Product Development Procedure” USAID Communications Support for Health (CSH), 2012. Chemonics International Inc. (2012). “Program Event Planning Procedure” USAID Communications Support for Health (CSH), 2012. Chemonics International Inc. (2010). “Emergency Action Plan.”    94 Chemonics International Inc. (2010). “Branding Implementation Plan & Marking Plan (BIP/MP)”. Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and The Manoff Group. Chemonics International Inc. (2010). “Communication Plan” Communications Support for Health Program (CSH). Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and The Manoff Group. Chemonics International Inc. (2012). “Environmental Mitigation and Monitoring Plan (EMMP).” Lusaka. Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and The Manoff Group. Communications Support for Health (CSH), (2012). “Campaign Coverage Update November” Tracking Progress of President’s Malaria Initiative Behavior Change Communication Activities. Chemonics International Inc. Communications Support for Health (2012). “Baseline Evaluation Report.” Chirwa, B. (2012). “National HIV/AIDS Communication and Advocacy Strategy (2011–2015), Vision 2030.” National HIV/AIDS/STI/TB Council. Kasonde, J. (2012). “National Voluntary Medical Male Circumcision (VMMC) Communication and Advoacy Strategy (2012–2015).” Mwaba, P. (2011). “National Malaria Communication Strategy 2011-2014,” National Malaria Control Center. Chemonics International Inc. (2010). “BCC Capacity Assessment and action Plan for Ministry of Health, National Malaria Control Center and National HIV/STI/TB Council”. Communications Support for Health (CSH). Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International Inc. (2010). “Assessment and Action Plan for 990 Talkline”. Communications Support for Health (CSH). Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International Inc. (2010). “Review of National IEC/BCC Campaigns”. Communications Support for Health (CSH). Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007- 00004-00 implemented by Chemonics International in association with ICF MACRO and The Manoff Group. Chemonics International Inc. (2010). “Situational Assessment and Action Plan for the Health Communication Resource Center” Communications Support for Health (CSH). Developed under terms of Task Order GHS￾I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and The Manoff Group. Chemonics International Inc. (2010). “Review of National IEC/BCC Messages, Materials, and Approaches.” Communications Support for Health (CSH). Developed under terms of Task Order GHS-I-05-07-00004, Contract No. GHS-I-007-00004-00 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Chemonics International Inc. (2012). “Monitoring and Evaluation Plan Version 1.5.” Communications Support for Health (CSH). Developed under terms of contract no: GHS-1-007-00004-00, ORDER NO. - 95 1-05-07-00004 implemented by Chemonics International in association with ICF MACRO and the Manoff Group. Musonda, K. (2010). “Draft Integrated Action Planning Workshop Report”, 2010. Annual Work Plans: 2010 USAID/Zambia, CSH 2010 Work Plan-Gantt chart, Aug. - Dec. 2010 USAID/Zambia, CSH 2010 Work Plan, Aug. – Dec. 2010 2011 USAID/Zambia, CSH 2011 Work Plan-Gantt chart USAID/Zambia, CSH 2011 Work Plan, Dec. 2010 (Revised Apr. 2011) 2012 USAID/Zambia, CSH 2012 Work Plan-Gantt chart, Nov. 2011 USAID/Zambia, CSH 2012 Work Plan Final, Nov. 2011 USAID/Zambia, CSH Baseline Program Evaluation, March 2011 USAID/Zambia, Champ–990 Talkline, Short-Term Marketing Plan, Nov. 2011 USAID/Zambia, Champ Audit Report, Nov. 2011 USAID/Zambia, Afya Mzuri – Dziwani Resource Center, Short-Term Marketing Strategy, McCann Healthcare, Nov. 2011 USAID/Zambia, CSH Branding and Marking Plan, Nov. 2010 USAID/Zambia, CSH Communication Plan, Sept. 2010 USAID/Zambia, CSH Environmental Mitigation Plan, n.d. USAID/Zambia, CSH Strategic Activities Fund Manual, n.d. Zambia National HIV/AIDS/STI/TB Council, National HIV AIDS Communications and Advocacy Strategy 2011-2015 Republic of Zambia Ministry of Health, National Malaria Communication Strategy2011-2014 Republic of Zambia Ministry of Health, National Voluntary Medical Male Circumcision (VMMC) Communication and Advocacy Strategy 2012-2015 Republic of Zambia Ministry of Health, National Malaria Control Centre, National Malaria Prevention and Control Surveillance, Monitoring and Evaluation Plan 2011-2015 National HIV/AIDS/STI/TB Council, National Monitoring and Evaluation Plan 2011 – 2015, May 2011 Initial Assessment Reports:    96 USAID/Zambia, National Malaria Control Centre And National HIV/STI/TB Council, CSH BCC Capacity Assessment And Action Plan For The Ministry of Health, Oct. 2010 USAID/Zambia, Situational Assessment And Action Plan For The Health Communication Resource Center, Sept. 2010 USAID/Zambia, CSH Assessment and Action Plan for the 990 Talkline, Oct. 2010 USAID/Zambia, CSH Review of National IEC/BCC Messages, Materials, And Approaches, Oct. 2010 USAID/Zambia, CSH Review of National IEC/BCC Campaigns, Oct. 2010 Org Documents: Chemonics International, Approvals, notifications, consent, reports, and concurrence required by the contract for Zambia Behavioral and Social Change Communication Project TO GHS-I-05-07-00004 (Under TASC3 IQC no. GHS-I-00-07-00004-00), n.d. USAID/Zambia, CSH Branding Implementation Plan & Marking Plan (BIP/MP), Nov. 2010 USAID, CSH Program Campaign and Product Development Procedure, May 2012 USAID/Zambia, CSH Communication Plan, Sept. 2010 USAID, CSH Event Planning Procedure (doc), May 2012 USAID, CSH Event Planning Procedure (pdf), May 2012 USAID/Zambia, CSH Emergency Action Plan, Nov. 2011 USAID/Zambia, CSH Environmental Mitigation and Monitoring Plan, n.d. PEPFAR COP Narratives Chemonics International, Partner Implementing Mechanism Template, Country Operational Plan FY 2012, n.d. PEPFAR Semi Annual (SAPR) and Annual (APR) Reports USAID/Zambia, CSH Annual PEPFAR Report Oct. 2010 – Sept. 2011 USAID/Zambia, CSH Annual PEPFAR Report Oct. 2011 – Sept. 2012 USAID/Zambia, CSH Semi-Annual PEPFAR Report Oct. 2010 – March 2011 USAID/Zambia, CSH Semi-Annual PEPFAR Report Oct. 2011 – March 2012 USAID/Zambia, CSH Monitoring And Evaluation Plan Version 1.5, May 2012 USAID/Zambia, CSH Support to the National Malaria Control Centre (NMCC), Annual Presentation, Apr. 2011 USAID/Zambia, CSH Support to the National Malaria Control Centre (NMCC), Annual Presentation, May 2012 97 PMI, Tracking Progress of President’s Malaria Initiative Behavior Change Communication Activities: Tracking Tool for PMI Implementing Partners: FY 2010-2011 USAID/Zambia, CSH Portfolio Review Meeting Presentation, Nov. 2010 USAID/Zambia, CSH Portfolio Review Meeting Presentation, May 2011 USAID/Zambia, CSH Portfolio Review Meeting Presentation, May 2012 USAID/Zambia, CSH Portfolio Review Meeting Presentation, Nov. 2011 USAID/Zambia, CSH Portfolio Review Sheet, Nov. 2010 USAID/Zambia, CSH Portfolio Review Sheet, May 2011 USAID/Zambia, CSH Portfolio Review Sheet, Nov. 2011 USAID/Zambia, CSH Portfolio Review Sheet, March 2012 USAID/Zambia, CSH First Quarterly Report, Oct. 2010 USAID/Zambia, CSH Second Quarterly Report, Dec. 2010 USAID/Zambia, CSH Quarterly Report, Apr. 2011 USAID/Zambia, CSH Quarterly Report, July 2011 USAID/Zambia, CSH Quarterly Report, Oct. 2011 USAID/Zambia, CSH Quarterly Report, Jan. 2012 USAID/Zambia, CSH Quarterly Report, March 2012 USAID/Zambia, CSH Quarterly Report, July 2012 USAID/Zambia, CSH Quarterly Report, Oct. 2012 USAID/Zambia, CSH Capacity Building Plan, n.d. USAID/Zambia, Kennedy Musonda, CSH Draft Integrated Action Planning Workshop Report, Oct. 2010 Success Stories: USAID/ Zambia, National VCT Day, July 2012 USAID/ Zambia, Getting Safe Love into the Communities, n.d. USAID/ Zambia, I Am a Safe Lover, n.d. USAID/ Zambia, Radio creates Demand for Nets, n.d. USAID/ Zambia, Safe Love Goes to Prison, n.d. USAID/ Zambia, Safe Love at Evelyn Hone College, July 2012 USAID/ Zambia, Be a safe lover this Valentine’s Day, n.d.    98 USAID/ Zambia, Zambians Encouraged to Become Safe Lovers, n.d. Weekly Highlights: USAID/ Zambia, CSH Weekly Highlights (1), Dec. 10, 2010 USAID/ Zambia, CSH Weekly Highlights (2), Dec. 10, 2010 USAID/ Zambia, CSH Weekly Highlights, Dec. 17, 2010 USAID/ Zambia, CSH Weekly Highlights, Dec. 3, 2010 USAID/ Zambia, CSH Weekly Highlights, July 1, 2011 USAID/ Zambia, CSH Weekly Highlights, Dec. 2, 2011 USAID/ Zambia, CSH Weekly Highlights, Sept. 2, 2011 USAID/ Zambia, CSH Weekly Highlights, Nov. 4, 2011 USAID/ Zambia, CSH Weekly Highlights, Aug. 12, 2011 USAID/ Zambia, CSH Weekly Highlights, Oct. 7, 2011 USAID/ Zambia, CSH Weekly Highlights, July 8, 2011 USAID/ Zambia, CSH Weekly Highlights, Dec. 9 , 2011 USAID/ Zambia, CSH Weekly Highlights, Sept. 8, 2011 USAID/ Zambia, CSH Weekly Highlights, June 10, 2011 USAID/ Zambia, CSH Weekly Highlights, Nov. 11, 2011 USAID/ Zambia, CSH Weekly Highlights, July 12, 2011 USAID/ Zambia, CSH Weekly Highlights, Aug. 15, 2011 USAID/ Zambia, CSH Weekly Highlights, Oct. 14, 2011 USAID/ Zambia, CSH Weekly Highlights, Dec. 16, 2011 USAID/ Zambia, CSH Weekly Highlights, June 17, 2011 USAID/ Zambia, CSH Weekly Highlights, March 18, 2011 USAID/ Zambia, CSH Weekly Highlights, Nov. 18, 2011 USAID/ Zambia, CSH Weekly Highlights, July 22, 2011 USAID/ Zambia, CSH Weekly Highlights, Aug. 19, 2011 USAID/ Zambia, CSH Weekly Highlights, Oct. 21, 2011 USAID/ Zambia, CSH Weekly Highlights, Sept. 23, 2011 USAID/ Zambia, CSH Weekly Highlights, June 24, 2011 99 USAID/ Zambia, CSH Weekly Highlights, Nov. 25, 2011 USAID/ Zambia, CSH Weekly Highlights, July 29, 2011 USAID/ Zambia, CSH Weekly Highlights, Aug 26, 2011 USAID/ Zambia, CSH Weekly Highlights, Oct. 28, 2011 USAID/ Zambia, CSH Weekly Highlights, Sept. 30, 2011 USAID/ Zambia, CSH Weekly Highlights, Jan. 21, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 1, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 8, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 15, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 21, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 29, 2011 USAID/ Zambia, CSH Weekly Highlights, June 3, 2011 USAID/ Zambia, CSH Weekly Highlights, March 11, 2011 USAID/ Zambia, CSH Weekly Highlights, March 18, 2011 USAID/ Zambia, CSH Weekly Highlights, March 25, 2011 USAID/ Zambia, CSH Weekly Highlights, May 6, 2011 USAID/ Zambia, CSH Weekly Highlights, May 20, 2011 USAID/ Zambia, CSH Weekly Highlights, May 13, 2011 USAID/ Zambia, CSH Weekly Highlights, May 27, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 8, 2011 USAID/ Zambia, CSH Weekly Highlights, Apr. 15, 2011 USAID/ Zambia, CSH Weekly Highlights, Jan. 7, 2011 USAID/ Zambia, CSH Weekly Highlights, Feb. 4, 2011 USAID/ Zambia, CSH Weekly Highlights, March. 4, 2011 USAID/ Zambia, CSH Weekly Highlights, Feb. 11, 2011 USAID/ Zambia, CSH Weekly Highlights, Feb. 18, 2011 USAID/ Zambia, CSH Weekly Highlights, Jan. 28, 2011 USAID/ Zambia, CSH Weekly Highlights, Jan. 14, 2011 USAID/ Zambia, CSH Weekly Highlights, June 1, 2012    100 USAID/ Zambia, CSH Weekly Highlights, March 2, 2012 USAID/ Zambia, CSH Weekly Highlights, Aug. 3, 2012 USAID/ Zambia, CSH Weekly Highlights, Feb. 3, 2012 USAID/ Zambia, CSH Weekly Highlights, May 4, 2012 USAID/ Zambia, CSH Weekly Highlights, March 16, 2012 USAID/ Zambia, CSH Weekly Highlights, Oct. 5, 2012 USAID/ Zambia, CSH Weekly Highlights, Sept. 7, 2012 USAID/ Zambia, CSH Weekly Highlights, June 8, 2012 USAID/ Zambia, CSH Weekly Highlights, March 9, 2012 USAID/ Zambia, CSH Weekly Highlights, Aug 10, 2012 USAID/ Zambia, CSH Weekly Highlights, Feb. 10, 2012 USAID/ Zambia, CSH Weekly Highlights, May 11, 2012 USAID/ Zambia, CSH Weekly Highlights, Apr. 13, 2012 USAID/ Zambia, CSH Weekly Highlights, Jan. 13, 2012 USAID/ Zambia, CSH Weekly Highlights, July. 13, 2012 USAID/ Zambia, CSH Weekly Highlights, Sept. 14, 2012 USAID/ Zambia, CSH Weekly Highlights, June. 15, 2012 USAID/ Zambia, CSH Weekly Highlights, Aug. 17, 2012 USAID/ Zambia, CSH Weekly Highlights, Feb. 17, 2012 USAID/ Zambia, CSH Weekly Highlights, May 18, 2012 USAID/ Zambia, CSH Weekly Highlights, Oct. 19, 2012 USAID/ Zambia, CSH Weekly Highlights, Apr. 20, 2012 USAID/ Zambia, CSH Weekly Highlights, Jan. 20, 2012 USAID/ Zambia, CSH Weekly Highlights, July 12, 2012 USAID/ Zambia, CSH Weekly Highlights, Sept. 21, 2012 USAID/ Zambia, CSH Weekly Highlights, June 22, 2012 USAID/ Zambia, CSH Weekly Highlights, March 23, 2012 USAID/ Zambia, CSH Weekly Highlights, Aug. 24, 2012 USAID/ Zambia, CSH Weekly Highlights, Feb. 24, 2012 101 USAID/ Zambia, CSH Weekly Highlights, May 25, 2012 USAID/ Zambia, CSH Weekly Highlights, Apr. 27, 2012 USAID/ Zambia, CSH Weekly Highlights, Jan 27, 2012 USAID/ Zambia, CSH Weekly Highlights, July 27, 2012 USAID/ Zambia, CSH Weekly Highlights, Sept 28, 2012 USAID/ Zambia, CSH Weekly Highlights, June 29, 2012 USAID/ Zambia, CSH Weekly Highlights, March 30, 2012 USAID/ Zambia, CSH Weekly Highlights, Aug. 31, 2012 USAID/Zambia, CSH Campaign Coverage Update, Nov. 2012 USAID Evaluation Guidance: USAID, Checklist for Assessing USAID Evaluation Reports, n.d. USAID, How to Note, Preparing Evaluation Reports, n.d. USAID, Evaluation Report Template USAID, USAID Evaluation Policy, Jan. 2011    102 ANNEX P: CONFLICT OF INTEREST STATEMENTS 103    104 105