Midterm Evaluation Summary Report IMPACTProgram March 2013 Cover Photo Boys and girls in Mtapa Village, Lilongwe, learn about HIV and gender equality while participating in an “In Charge!” session sponsored by the IMPACT Program. Photo by Megan Collins for CRS. Recommended citation Catholic Relief Services Malawi. (2013). Integrated (HIV Effect) Mitigation and Positive Action for Community Transformation (IMPACT) Program Midterm Evaluation Summary Report. Lilongwe, Malawi. Produced by Catholic Relief Services on behalf of the IMPACT Program Consortium. March 2013. DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. IMPACTProgram Midterm Evaluation Summary Report March 2013 IMPACTProgram Midterm Evaluation Summary Report March 2013 ii | IMPA CT Program MIDTERM EVALUATION Table of Contents ACKNOWLEDGEMENTS .....................................................................................................................iv ACRONYMS ...........................................................................................................................................v EXECUTIVE SUMMARY........................................................................................................................2 Background and Program Overview.................................................................................................................................2 Methodology ...................................................................................................................................................................................2 Summary Results and Discussion........................................................................................................................................2 Strategic Objective 1: Wellbeing of 58,017 OVCs improved...............................................................................2 Strategic Objective 2: Access to treatment and care for 41,505 PLHIV is enhanced.................................................................................................................................................2 Cross-Cutting Areas......................................................................................................................................................................2 Recommendations.......................................................................................................................................................................3 Conclusion.........................................................................................................................................................................................3 background and program overview..................................................................................5 Partners and Geographic Focus...........................................................................................................................................5 Midterm Evaluation Purpose ..................................................................................................6 Methodology..................................................................................................................................7 Summary Results and Discussion.........................................................................................9 SO1: Wellbeing of 58,017 OVCs improved. ....................................................................................................................9 IR 1.1 OVC caregivers have practiced improved infant and young child feeding and caring techniques............................................................................................................................................................. 10 Care Groups: Child Feeding, Care, Hygiene and Sanitation ...................................................................... 10 Community Complementary Feeding and Learning Sessions (CCFLS)............................................ 11 IR 1.2 OVC are protected from abuse and exploitation. ..................................................................................... 13 Strengthening Child Protection Systems.............................................................................................................. 13 Birth Registration ................................................................................................................................................................. 15 In Charge! ................................................................................................................................................................................. 15 IR 1.3 OVC access to secondary school is improved............................................................................................. 17 Secondary School Bursaries and Performance Improvement Plans................................................... 17 Educational Drop-In Centers........................................................................................................................................ 18 School Savings Accounts................................................................................................................................................ 20 IR 1.4 OVC households have increased financial assets...................................................................................... 20 SO 2: Access to treatment and care for 41,505 PLHIV is enhanced. ............................................................ 23 IR 2.1 Community mobilization for HIV testing and counseling is increased. ...................................... 24 HIV Testing and Counseling Through Community Health Days............................................................ 24 MIDTERM EVALUATION IMPACT Program | iii IR 2.2 PLHIV remain in longitudinal care and initiate treatment in a timely fashion......................... 26 IR 2.3 Loss to follow-up of PLHIV on ART and in PMTCT reduced. ............................................................... 27 Support Groups.................................................................................................................................................................... 27 Expert Clients......................................................................................................................................................................... 28 Environmental Monitoring and Mitigation ......................................................................................................... 29 Cross-Cutting Areas................................................................................................................. 31 Targeting and Beneficiary Enrollment........................................................................................................................... 31 Monitoring and Evaluation and Learning................................................................................................................... 32 M&E and Management Information Systems.................................................................................................... 32 Gender Integration.................................................................................................................................................................... 34 Disability Integration................................................................................................................................................................ 35 Strategic Objective Integration.......................................................................................................................................... 36 WALA and IMPACT Integration........................................................................................................................................... 36 Project Management and Staffing................................................................................................................................... 37 Project Design............................................................................................................................................................................... 38 LIMITATIONS OF THE MIDTERM EVALUATION ............................................................................. 40 RECOMMENDATION S....................................................................................................................... 40 CONCLUSION..................................................................................................................................... 41 iv | IMPA CT Program MIDTERM EVALUATION ACKNOWLEDGEMENTS The production of this midterm evaluation summary report would not have been possible without the generous financial and technical support of PEPFAR and USAID Malawi. Catholic Relief Services (CRS) is grateful for resources provided through the IMPACT Program to support this activity. CRS recognizes the participation and contributions of the Government of Malawi (GOM) partners, including the Ministry of Health; Ministry of Gender, Children and Social Welfare; National AIDS Commission; and Office of the President and Cabinet, Department of Nutrition and HIV and AIDS. CRS sincerely thanks the members of the WALA and IMPACT Programs for their active participation in program implementation and the midterm evaluation process itself: • Africare • Chikwawa Diocese • Dedza Catholic Health Commission • D-tree International • Emmanuel International • Lilongwe Catholic Health Commission • Opportunity International Bank of Malawi • National Association for People Living with HIV and AIDS in Malawi (NAPHAM) • PCI (Project Concern International) • Save the Children • World Vision • Zomba Catholic Health Commission We extend our appreciation to the midterm evaluation team, which included Dr. Ruth Hope, our lead international evaluator; Mr. Andrew Joabe, the local consultant evaluator; Ms. Cynthia Mambo, IMPACT M&E Technical Quality Coordinator (TQC); and Mr. Joseph Simfukwe of CRS Zambia. We would also like to acknowledge the contribution of IMPACT partner staff who engaged in the long hours of field work that informed this report. Like the IMPACT Program itself, this report represents the efforts and work of many people. This includes everyone from district government officers to IMPACT partner staff and volunteers to community members who assisted the evaluation in Balaka, Dedza, Lilongwe, and Zomba and took time off from their busy days to provide thoughtful replies to our many questions. MIDTERM EVALUATION IMPACT Program | v ACRONYMS ART antiretroviral therapy CA cooperative agreement CCFLS community complementary feeding and learning sessions CGV Care Group Volunteer CHC Catholic Health Commission COP Chief of Party CPC child protection committee CPW child protection worker CRS Catholic Relief Services C&S care and support CSI Child Status Index CWD children with disabilities DCOP Deputy Chief of Party DHMT district health management team EDC (afterschool) educational drop-in center EGPAF Elizabeth Glaser Pediatric AIDS Foundation EI Emmanuel International FGD focus group discussion GOM Government of Malawi HTC HIV testing and counseling HSA health surveillance assistant IMCI integrated management of childhood illness IMPACT Integrated (HIV Effect) Mitigation and Positive Action for Community Transformation IPC infection prevention and control IR intermediate result KII key informant interview LoA life of activity LOE level of effort MCHN maternal, child health and nutrition M&E monitoring and evaluation MIP mother–infant pair MIS management information system MACOHA Malawi Council for the Handicapped MTE midterm evaluation NAPHAM National Association for People Living with HIV and AIDS in Malawi NCD noncommunicable diseases OIBM Opportunity International Bank of Malawi OVC orphans and vulnerable children PEPFAR President’s Emergency Plan for AIDS Relief PLHIV people living with HIV PMTCT prevention of mother-to-child transmission vi | IMPA CT Program MIDTERM EVALUATION SC Sub-chief SO strategic objective SOGs standard operating guidelines TOT training of trainers TNM Telecommunication Network of Malawi TQC Technical Quality Coordinator VSL village savings and loan VSU Victim Support Unit WALA Wellness and Agriculture for Life Advancement EXECUTIVE SUMMARY 2 | IMPA CT Program MIDTERM EVALUATION EXECUTIVE SUMMARY Background and Program Overview Integrated (HIV Effect) Mitigation and Positive Action for Community Transformation (IMPACT) is a four-year USAID-supported Global Development Alliance program expected to improve the quality of life for orphans and vulnerable children (OVC) and people living with HIV (PLHIV). As lead agency of the IMPACT Program, Catholic Relief Services Malawi brings together the Title II-supported Wellness and Agriculture for Life Advancement (WALA) consortium and faith-based implementing partners with private-sector, technical-assistance and information-technology partners to expand access to services, in partnership with the Government of Malawi (GOM). Methodology The midterm evaluation (MTE) was designed to assess the progress in achieving the two strategic objectives (SOs) during the first two years of program implementation (July 2010–June 2012). It was conducted in two phases by internal and external consultants in collaboration with program staff. Document review, field visits, key informant interviews and focus group discussions were triangulated with data from the project’s management information system (MIS). Summary Results and Discussion Strategic Objective 1: Wellbeing of 58,017 OVC improved At midterm, IMPACT had reached 30,177 vulnerable households and provided support to 38,662 eligible children, representing 67% of the life of activity (LoA) target. Three intermediate results under SO1 provide age-appropriate and innovative services supporting child nutrition, protection and education. A fourth intermediate result focuses on economic strengthening of OVC caregivers. The project provided nutritional support services to 17,691 children and protection services to 31,911 children. Secondary school bursaries and educational drop-in centers increased access to education for 42,419 children, 281% over FY2012 targets. Members of 141 faith-based partner village savings and loan groups saved over MK24,500,000, or about $70,000, and invested this money in productive assets as well as child education. In general, performance on SO1 has been technically strong, and the program is likely to achieve its LoA targets. Strategic Objective 2: Access to treatment and care for 41,505 PLHIV is enhanced At midterm, IMPACT had reached 26,978 of eligible adults with at least one care service, representing 65% of the LoA target. Community Health Days provided 5,116 people with HIV testing and counseling. In partnership with the National Association for People Living with HIV and AIDS in Malawi (NAPHAM), IMPACT has strengthened 252 PLHIV support groups and trained 155 Expert Clients to reduce loss to follow-up of pre-antiretroviral therapy (ART) and ART patients. IMPACT experienced some challenges under this SO2 related to its design and requirements to track supply-side indicators outside of its manageable interests. Cross-Cutting Areas As a community-based program, IMPACT was designed to provide multiple and complementary services to vulnerable households. IMPACT’s OVC Beneficiary Enrollment Tool identifies the most vulnerable households and adds transparency to the beneficiary selection process, especially on high-value activities such as secondary school sponsorship. Synergies exist between the SOs and MIDTERM EVALUATION IMPACT Program | 3 among the intermediate results. Similarly, community members in WALA-IMPACT areas benefit from the added services. In addition to age- and sex-disaggregated data that have been collected to facilitate analysis of program results by sex, male involvement strategies feature prominently across both SOs. Participation in “In Charge!” has engaged adolescents in thoughtful discussions around gender norms and for many increased self-efficacy. As a learning consortium, IMPACT is committed to continuous quality improvement. The joint WALA-IMPACT MIS and concomitant data quality procedures facilitate reporting and help to inform program management and direction. Operations research undertaken by the project contributes to improving practice in Malawi and elsewhere, as the program disseminates its work widely. IMPACT has made a concerted effort to address the needs of children living with disabilities or HIV; however, limited referral options have proved challenging. Recommendations 1. Continue learning from and adopting promising practices across all implementing partners to enhance consistency and impact. 2. Place greater emphasis on community system strengthening to extend the IMPACT Program’s results beyond the life of the project. 3. Strengthen the design of SO2 and the indicators for measuring performance to reflect the strategies and activities IMPACT is employing for SO2. Conclusion Over the course of the first two years of operation, IMPACT has accomplished most of what it set out to do, reaching or exceeding many of its most important targets. Systems have been put in place, capacities have been strengthened, and innovations and promising practices are being tested. These approaches will benefit not only the families, children and PLHIV being served by IMPACT, but also others in Malawi and elsewhere. Over the next two years, the program should focus on community system strengthening. Much of the work has already been done. What remains is to institutionalize these systems so they are able to continue to provide lasting benefits to Malawi’s most vulnerable populations. background and program overview MIDTERM EVALUATION IMPACT Program | 5 background and program overview The Integrated (HIV Effect) Mitigation and Positive Action for Community Transformation (IMPACT) Program is a four-year USAID-supported Global Development Alliance award worth $28,488,482, of which USAID contribution is 45%; the IMPACT alliance leverage is 50%; and the IMPACT alliance cost share is 5%. The program is expected to improve the wellbeing of 58,017 orphans and vulnerable children (OVC) and enhance access to treatment and care for 41,505 people living with HIV (PLHIV). IMPACT activities are consistent with the Partnership Framework signed between the Government of the United States of America and the Government of the Republic of Malawi to support the National HIV and AIDS Response. In addition, IMPACT directly contributes toward the achievement of the Malawi National Plan of Action for Orphans and Other Vulnerable Children as well as the Malawi HIV and AIDS Extended National Action Framework. Partners and Geographic Focus As lead agency of the IMPACT Program, Catholic Relief Services Malawi brings together the Title II-supported Wellness and Agriculture for Life Advancement (WALA) consortium and faith-based implementing partners with private-sector, technical-assistance and information-technology partners (Figure 1) for expanded and wrap-around service in 39 traditional authorities across central and southern Malawi (Table 1). The Government of Malawi (GOM), through various line ministries’ participation, is a key partner in the IMPACT Program. • Africare • Catholic Relief Services • Chikwawa Diocese • Emmanuel International • Project Concern International • Save the Children • World Vision • Dedza Catholic Health Commission (CHC) • Lilongwe CHC • Zomba CHC • D-Tree International • National Association of People Living with HIV and AIDS in Malawi (NAPHAM) • Peace Corps • Opportunity International Bank of Malawi • Telecommunications Network of Malawi (TNM) WALA partners Faith-based partners Technical￾assistance partners Private sector partners l- Priv rs par Figure 1: Overview of IMPACT Program partners. 6 | IMPA CT Program MIDTERM EVALUATION Table 1: Implementing partners and their program district and traditional authorities. Implementing Partner District Traditional Authority Africare Mulanje Nthiramanja, Chikumbu, Mabuka (Tea Estates), Laston Njema (SC) (Tea Estates) CRS Malawi and Chikwawa Diocese Chikwawa Mankwira, Ngabu, Lundu, Maseya, Katunga, Kasisi Dedza Catholic Health Commission† Ntcheu Mpando, Ganya, Makwangwala Emmanuel International Machinga Kawinga, Mlomba (SC), Mposa (SC), Chamba (SC) Zomba Kuntamanji, Mkumbira (SC), Malemia, Mwambo Lilongwe Catholic Health Commission† Lilongwe Malili Project Concern International (PCI) Machinga Sitola Balaka Nsamala, Kalemebo Save the Children Chiradzulu Chitera, Nchema, Kadewere Zomba Chikowi, Mlumbe, Mbiza (SC) World Vision Thyolo Nsabwe, Bvumbwe, Thukuta (SC), Kwethemule (SC), Kapichi, Nchilamwela, Thomas Zomba Catholic Health Commission † Zomba Mlumbe † IMPACT-only (Church) partner. Overview of Management Structure and Operations To promote synergy and provide close contact with partners, most of the IMPACT management team is co-located with the WALA team in Blantyre, with additional IMPACT staff based in Lilongwe. The management of IMPACT is coordinated through an advisory board that addresses issues of strategic importance and a core program quality team that includes both WALA and IMPACT staff to address issues of program integration, resolve bottlenecks and identify areas for operations research. Sector-specific technical working groups develop solutions to implementation challenges and provide opportunities for exchange and learning across consortium members (Figure 2). Quarterly meetings among consortium staff provide opportunities to address challenges, align approaches and ensure timely reporting. Midterm Evaluation Purpose The midterm evaluation (MTE) was designed to assess progress toward achieving the two strategic objectives (SOs) during the first two years of program implementation (July 2010–June 2012): • SO1: Wellbeing of 58,017 OVC is improved. • SO2: Access to treatment and care for 41,505 PLHIV is enhanced. Advisory board Core program quality team Technical working groups OVC, MCH, C&S Figure 2: IMPACT management and operations structure. MIDTERM EVALUATION IMPACT Program | 7 The MTE was also intended to guide the program team in making necessary course corrections in achieving its life of activity (LoA) program targets. In assessing the progress achieved to date on the program’s SOs, the MTE was designed to focus on design considerations, implementation and impact. Methodology The IMPACT Program MTE had two phases. Phase 1 was conducted between June 12 and August 3, 2012, by an external international consultant and a Malawian consultant, working in collaboration with the M&E Technical Quality Coordinator (TQC), the 14-member team of IMPACT Program staff and a CRS Zambia staff member on temporary duty. The MTE scope of work, methodology and report were developed by the IMPACT Program in collaboration with the Government of Malawi (GOM), USAID and consortium members. The MTE included: • Document review of the program description, quarterly reports, standard operating guidelines (SOGs) and other documents available from implementing partners and stakeholders. • Forty key informant interviews with program, district and community level stakeholders. • Twenty focus group discussions with project volunteers and program participants. • Field visits to observe key project activities such as educational drop-in centers (EDCs) and hygiene and sanitation activities. Communities visited were randomly selected from the catchment areas of two WALA partners (Emmanuel International [EI] and Project Concern International [PCI]) and two IMPACT-only (faith-based partners the Catholic Health Commissions of Lilongwe and Dedza) representing a spectrum of implementation capabilities. A total of eight villages were selected at random as evaluation villages from a complete list of IMPACT villages. The comprehensive list of suggested MTE questions provided by the IMPACT Program was adapted by the international consultant to form the evaluation framework and the overall MTE questions. These in turn served as the foundation for the key informant interview (KII) schedules and focus group discussion (FGD) guides. Limited time to test the KII schedules required the consultants to correct the schedules as the MTE proceeded, by including missing questions, eliminating repetitive questions and adjusting the flow of the questions. The FGD guides were pilot-tested in Machinjiri, Zomba District. Following a three-day workshop to review, finalize and pretest data collection tools, the evaluation team split into two groups. To avoid bias, program staff were assigned to evaluation sites other than their own. The teams were dispatched and after 11 days in the field, reconvened to compile their findings and delivered them to the international consultant for analysis and report writing. Following a review of the consultant’s report and comments from the Malawi USAID Mission, CRS decided to revise the MTE to reflect the perspectives of program participants and link the qualitative findings with quantitative indicators from the jointly managed WALA-IMPACT management information system (MIS). The program seconded a CRS-headquarters-based senior technical advisor (STA) with a background in OVC programming for this purpose. Prior to initiating analysis, the STA conducted an extensive review of the international consultant’s report, identifying opportunities for further analysis and/or triangulation. To ensure a robust understanding of the project and perspectives of the program participants, the STA reviewed select project documents, quarterly reports, and the KII and FGD notes. These notes were coded in Nvivo along with key themes for further investigation, such as gender mainstreaming, unintended consequences, WALA-IMPACT integration, complementarity of the two IMPACT SOs and program participant perspectives. Since transcripts from KII conducted by the international consultant were unavailable to the STA, she conducted nine interviews with IMPACT and WALA program staff to supplement her understanding of the program and collect additional information to address concerns raised by USAID and the program staff. summary results and discussion MIDTERM EVALUATION IMPACT Program | 9 Summary Results and Discussion IMPACT was designed as a wrap-around project delivering additional services to the most vulnerable households in WALA program areas and as a stand-alone project in Dedza, Lilongwe and Zomba Catholic Health Commission program areas. The program’s results framework for the project is shown in Figure 3. Goal: Improved quality of life and HIV impact mitigated on 99,522 orphans and vulnerable children (OVC) and people living with HIV (PLHIV) in Malawi. Intermediate Result 1.1 OVC caregivers have practiced improved infant and young child feeding and caring techniques. Intermediate Result 1.2 OVC are protected from abuse and exploitation. Intermediate Result 1.3 OVC access to secondary education is improved. Intermediate Result 1.4 OVC households have increased financial assets. Strategic Objective 2: Access to treatment and care for 41,505 PLHIV is enhanced. Intermediate Result 2.1 Community mobilization for HIV testing and counseling is increased. Intermediate Result 2.2 PLHIV remain in longitudinal are and initiative treatment in a timely fashion. Intermediate Result 2.3 Loss-to-follow-up of PLHIV on ART and PMTCT is reduced. Strategic Objective 1: Wellbeing of 58,017 OVC is improved. Figure 3: IMPACT results framework. SO1: Wellbeing of 58,017 OVCs improved. Strategic Objective 1 aims to improve the wellbeing of children by supporting families in the areas of nutrition, child protection, education and economic strengthening. At midterm, IMPACT had reached 30,177 households and provided support to 38,662 eligible children, representing 67% of the LoA target (Table 2). Table 2: SO1 indicator performance tracking table. Indicator Cumulative FY12 Target Q4 FY12* Total Ac hieved† 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of households that receive external support 15,597 30,177 193% Number of eligible children provided with a minimum of one care service, by sex 31,194 38,662 18,510 20,152 124% * Cumulative target equals the number of households or individuals expected to have been served from July 10, 2010, through September 30, 2012. † Total achieved equals number of households or individuals actually served from July 10, 2010, through June 30, 2012. 10 | IMPA CT Program MIDTERM EVALUATION IR 1.1 OVC caregivers have practiced improved infant and young child feeding and caring techniques. Table 3: IR 1.1 indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of eligible clients <18 years old who received food and/ or other nutritional services 10,296 17,691 8,560 9,131 172% Number of eligible adults and children provided with health care referral, by sex, by age: 0–17, 18+ 6,696 1,224 31 41 814 336 18% Ac tivity Indicators Number of Care Group volunteers conducting home visits (new IMPACT groups only), by sex 1,500 624 42% Number of Care Groups formed (new IMPACT groups only) 150 123 82% Number of caregivers of OVC 0–59 months participating in Care Group, by sex 26,920 12,994 48% Number of caregivers participating in Community-Led Complementary Feeding and Learning Sessions 24,928 7,522 30% Number of IMPACT Program promoters, HSAs, CGV trained on “CommCare” in pilot areas, by sex 79 49 28 21 65% Care Groups: Child Feeding, Care, Hygiene and Sanitation IMPACT uses the Care Group model to train caregivers of young children in infant and young child feeding. Care Group volunteers have traditionally been women, but in an effort to increase male involvement, IMPACT has encouraged couples to participate. Trained volunteers share their new knowledge and skills informally to the cluster of homes around their own home, bringing new messages every two weeks. By midterm, the program had completed two modules (hygiene and sanitation, and breastfeeding) and was working through complementary feeding. A heavy emphasis is placed on hygiene and sanitation to increase numbers of households with hygienic pit latrines; hand-washing facilities with “tippy taps” and readily available soap; improved, raised platforms for drying pots pans and plates; and domestic rubbish pits. These activities have been well-received by households, as illustrated by this female Care Group leader, “My household has improved; I am able to differentiate the way my house and surroundings looked before I MIDTERM EVALUATION IMPACT Program | 11 joined the Care Group and after I joined. Even my husband comments on the good sanitation of my home these days.” In some communities, the chief or headman was actively engaged in promoting the new domestic hygiene behaviors, and his household hygiene facilities were role models for his community, supporting spill over to households beyond the Care Group. A widower looking after three children, this male Care Group volunteer from Likalawe Village in Mulanje benefits from WALA and IMPACT Program activities. Here he demonstrates using a homemade hand-washing station called a tippy tap. Tippy taps have become very popular among the households in Likalawe. Community Complementary Feeding and Learning Sessions (CCFLS) To augment Care Group activities, IMPACT organizes two CCFLS per year, once during the lean season and once in the postharvest seasons. Each set of CCFLS lasts 12 days, and caregivers bring foods available from their homes to the sessions, where they learn to prepare nutritious foods with locally available ingredients. Members of PLHIV support groups (see discussion under SO2 below) are invited to participate in the mainstream sessions, or sessions are organized specifically for them. To increase access to locally available nutritious foods year round, promoters incorporated topics such as herbal gardens and kitchen gardens. CCFLS have assisted members in learning how to prepare new types of nutritious age￾appropriate foods. A male Care Group volunteer noted, “I do not give my wife money to buy Cerelac because I have learned how to make different types of nutritious porridge, like one made from sweet potatoes.” This 4-year-old boy finished all of his meal at a Community Complementary Feeding and Learning Session in Kawanga Village, Balaka. With support from the WALA and IMPACT Programs, caregivers actively learn to prepare highly nutritious foods for children under age 5 or people living with HIV. My household has improved; I am able to differentiate the way my house and surroundings looked before I joined the Care Group and after I joined. Even my husband comments on the good sanitation of my home these days. —female Care Group leader I do not give my wife money to buy Cerelac because I have learned how to make different types of nutritious porridge, like one made from sweet potatoes. —male Care Group volunteer MEGAN COLLINS FOR CRS MEGAN COLLINS FOR CRS 12 | IMPA CT Program MIDTERM EVALUATION Lessons Learned and Way Forward Challenges observed pertaining to IR 1.1 included high attrition rates in some Care Groups, often attributed to difficulties in providing food, fuel and equipment for cooking demonstrations, and an expectation that development programs will provide tangible benefits such as food handouts. Also, due to the higher-than-anticipated cost of educational flipcharts, some groups were asked to share materials, and this was not well-received by group members. Finally, many WALA￾partner Care Groups are registered on paper, but not included in the MIS (see p. 23), resulting in underreporting of this indicator. WALA partners plan to address this issue in FY2013. Perceptions from both partner staff and communities around the sustainability of the groups varied. Some suggested that Care Groups do not need to be institutionalized, because once they had done their work improving infant and young child feeding and care, the need would no longer exist. Others, seeing an ongoing need, are exploring linking Care Groups to existing Village Health Committees. A Care Group volunteer in Maziro II Village, Lilongwe, shares maternal health and nutrition information with her neighbors using pictorial flipcharts and training provided by the IMPA CT Program. MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 13 IR 1.2 OVC are protected from abuse and exploitation. Table 4: IR 1.2 indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of eligible adults and children provided with protection and legal aid services, by sex and by age (0–17, 18+) 25,262 31,911 15,337 16,706 127% Number of eligible adults and children provided with psychological, social and spiritual services, by sex and by age (0–17, 18+) 9,259 8,943 4,362 4,581 0 0 97% Ac tivity Indicators Number of vulnerable children 0–17 years registered in the village headmen registration system, by sex 24,812 31,911 15,269 16,642 129% Number of vulnerable children 0–17 years with birth certificate, by sex† 2,481 N/A N/A N/A N/A Number of children reporting child protection violations to OVC Committee 450 132 68 64 29% Number of OVC Committee members trained on Child Protection, by sex and by age: 0–17, 18+ 2,540* 1,532 899 633 60% Number of children completing "In Charge!" session, by sex 3,000 2,578 1,358 1,220 86% Number of children participating in Children’s Corner 5,479 6,118 2,887 3,231 112% * Represents an annual training target, not the total cumulative target or total achieved, which was 5,104 OVC Committee members. † This indicator is dependent on a GOM process that was not operational over this period. Strengthening Child Protection Systems IMPACT revitalized or established OVC Committees with responsibilities both for the welfare of OVC—monitoring children in vulnerable households using the Child Status Index (CSI)—and for addressing child protection in their communities in conjunction with the village headmen, child protection workers (CPWs) and, when necessary, the Victim Support Unit (VSU) and police. As a result of these efforts, OVC Committees are active and empowered; community awareness of children’s rights is growing along with the recognition that some traditional practices—such as burdening children with excessive domestic or farm work, discriminating against girl children in terms of access to schooling, child marriage and disciplining children by beating them—do not respect children’s rights and are exploitative. 14 | IMPA CT Program MIDTERM EVALUATION At the time of the MTE, 5,104 committee members had been trained since inception. Committees value these trainings and expect to be engaged in child protection work in the future. One OVC Committee member explained, “Our capacity as a committee has been built through the trainings and lessons, and we will be able to apply the issues learned even when IMPACT comes to an end.” There was almost universal acknowledgement of the success of IMPACT activities in supporting the Government of Malawi (GOM) system for protecting children from abuse and exploitation. Program staff attributed this success to twice yearly CSI home visits by family care volunteers, who used these visits as an opportunity to identify cases of abuse and counsel caregivers on what constitutes abuse. Most cases were able to be resolved in this manner. IMPACT has successfully worked with village headmen, GOM community CPWs and the VSUs to raise awareness and strengthen systems for reporting and taking action on child abuse. MTE respondents reported action being taken locally by the headman for some infractions of children’s rights. For example, an OVC Committee chairperson observed, “Ever since the chief offered a penalty to the caregivers who physically abuse the orphan, the community members’ attitudes changed as they fear to [be] charged with stiffer penalties. This helped set a good example, and since [then], no violations were reported in the community.” Success stories—beyond the establishment of functioning reporting systems—included girls rescued from child marriages and returned to their families and to school, and examples of successful prosecution and imprisonment of rape and sex crime perpetrators. Adolescent participants have also noted changes in their communities: “Child labor is common in this area. You find a mother giving her baby to her 5-year-girl child to [take care of ] while she is doing other household chores, thereby denying the child time to play with friends. The project is helping to address that.” This village headman in Nsomo Village in Chikwawa, sits with the OVC Committee as they discuss the needs of vulnerable children in their community. The IMPACT Program has promoted the involvement of traditional leaders to help reinforce the importance of child protection issues and support for vulnerable households. Our capacity as a committee has been built through the trainings and lessons, and we will be able to apply the issues learned even when IMPACT comes to an end. —OVC Committee member MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 15 Lessons Learned and Way Forward In Malawi, as in much of the world, a culture of silence surrounds many of the worst forms of child abuse, such as incest and rape, due to fear of retribution, loss of social status or economic consequences, particularly if the perpetrator is a male head of household. As in many settings, reporting child protection violations also has negative consequences in terms of stigmatization or retribution. For example, an adolescent girl raped by her stepfather was blamed by her family when the stepfather was jailed. In this context, every disclosure and every documented report of a violation is a success. Thus, while the program has only achieved 29% of its FY2012 target on the indicator Number of children reporting child protection violations to OVC Committee (Table 4), program staff believe that activities to date have been very successful. Communities are now sufficiently knowledgeable about what constitutes abuse and have reached a tipping point that will generate an increase in reporting and hopefully fewer violations. Despite the fact that CPWs recognize and value the work of the OVC Committees, maintaining the profile of protection issues and institutionalizing the reporting system by the project’s end will be seriously compromised by the inadequate numbers of GOM-supported CPWs. Birth Registration IMPACT partners worked closely with communities to support implementation of the GOM birth registration policy, ensuring that beneficiary children were registered with the headman. However, by June 2012, the GOM had not yet implemented the national birth certification mechanism,1 so the IR 1.2 indicator, Number of vulnerable children 0–17 years with birth certificate, by sex, was not applicable. In Charge! “In Charge!” was designed to help young people (primarily aged 10 to 17 years) increase their knowledge and understanding around HIV, improve their self-efficacy skills, reduce stigma and discrimination toward PLHIV and other vulnerable groups, and avoid risky behaviors associated with HIV infection. The four-hour “In Charge!” intervention also promotes discussions about gender roles as well as strategies that boys and girls can use to avoid sexual assault and other forms of abuse. Developed by CRS in Ethiopia, the tool is based on the highly participatory SARAR methodology (self-esteem, associative strength, resourcefulness, action planning, and responsibility) and was adapted to the Malawian context by IMPACT. At the time of the MTE, “In Charge!” had been used to reach 2,578 children in IMPACT-only catchment areas by trained by OVC Coordinators and OVC Promoters in collaboration with primary school teachers, primary education advisors and CPWs. Operations research into the effectiveness of the program was underway at the time of the MTE. Feedback from communities where “In Charge!” has been implemented was very positive, and the program plans to execute an operations research study to evaluate the impact of the intervention in detail. Boys and girls in Mtapa Village, Lilongwe, debate whether HIV can be transmitted from sharing food or from mother to baby in this “In Charge!” small group discussion. 1 GOM is making a renewed effort to have birth certificates issued through a birth registration process being piloted at selected health facilities. After the pilot, GOM plans to roll out this process nationally, and IMPACT expects that the GOM will expand this renewed effort to all health facilities within the IMPACT program areas. Child labor is common in this area. You find a mother giving her baby to her 5-year-girl child to [take care of] while she is doing other household chores, thereby denying the child time to play with friends. The project is helping to address that. —IMPACT OVC program participant MEGAN COLLINS FOR CRS 16 | IMPA CT Program MIDTERM EVALUATION Box 1: There is an App for That— IMPA CT’s Use of Mobile Technology to Improve Child Wellbeing For service delivery, IMPACT relies on HSAs (health surveillance assistants) and community volunteers. To support their work, IMPACT partner D-tree International developed and piloted two mobile applications, with a third anticipated in FY2013. The applications use the CommCare platform and run on JAVA-enabled feature phones. The community case management (CCM) application supports HSAs to administer the protocol approved by the Ministry of Health (MOH) for the integrated management of childhood illnesses (IMCI). IMPACT staff reported that HSAs using the application provided more thorough child assessments, with health facility staff observing more appropriate and timely referrals. Perceived quality of care at the community level had also increased, they noted. HSAs were able to learn the application quickly and have even started their own Facebook user community accessed through mobile phones. IMPACT plans to introduce a mother–infant pair (MIP) application to the HSAs during the second half of the project. The CSI is used by IMPACT volunteers to support child assessment and referral. Regular assessment of tens of thousands of children across 12 factors creates large amounts of data, whereas referrals rely on volunteer knowledge of resources available in their own areas. To improve CSI assessment, data quality, referrals and follow-up, a local-language CSI mobile phone application was developed and tested. Unfortunately, budget constraints prevented volunteers from having individually dedicated phones. Therefore children were assessed by volunteers using the CSI paper form while trained OVC Committee secretaries entered the CSI data into the application. Benefits of the application included more complete CSI data and reduced loss to follow-up among referred children. However, the use modification changed the primary purpose of the tool from a job aid for assessment and referral to a data-entry tool, thereby shifting the CSI data-entry burden from the program to the OVC Committee secretaries. In contrast to the HSAs, OVC Committee secretaries faced more technological and safeguarding challenges with the phones. At the time of the MTE, IMPACT was identifying lessons learned and determining next steps. An HSA uses the IMPACT Program’s CCM mobile phone application at a village clinic in M’gwata Village, Lilongwe. Here the phone guides the HSA through malaria diagnosis and treatment for a toddler, while also providing key counseling messages and transmitting data. MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 17 IR 1.3 OVC access to secondary school is improved. Table 5: IR 1.3 Indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of eligible children provided with education and/ or vocational training, by sex 15,074 42,419 23,302 19,117 281% Ac tivity Indicators Number of vulnerable children receiving secondary school support, by sex 1,574 1,333 599 734 85% Number of vulnerable households visited by OVC Committee 1,475 1,203 899 633 82% Number of children participating in educational drop-in center (EDC) sessions, by sex 13,500 41,752 23,002 18,750 309% Secondary School Bursaries and Performance Improvement Plans IMPACT is conducting a number of education-related activities, including secondary school bursaries, educational drop-in centers (EDCs), and school savings accounts. By midterm, 1,333 children (599 girls, 734 boys)2 (Table 5) from vulnerable households had received bursaries. Partners supplemented the bursaries with school bags, books, shoes, uniforms and supplies, depending on the situation of each child. Home and school visits to monitor performance were also included as key program activities. Bursary students would otherwise have dropped out of school, as this female recipient reveals, “I tried to do piece works but could not raise the K1,500 fee, so I just thought of dropping out of school. It is the project that took me back to school.” Lessons Learned and Way Forward During implementation, IMPACT Program staff found that some sponsored students were having difficulty performing well and staying in school, often due to unsupportive situations at home (heavy afterschool chores, early marriage, etc.) In response, the project introduced performance improvement plans for struggling students. Partner staff review progress reports and take action where academic achievement is faltering or below average. In some cases, the action might entail a meeting among the teacher, guardian and child to encourage the student to work harder. Other cases require a more considered intervention. In one case, a female student living with her elderly 2 The project intends to support an equal number of boys and girls. However, fewer girls make it to upper primary in Malawi, making it difficult to find girls to support. In one catchment area there was a project already supporting girls, so IMPACT supported only boys in that area. I tried to do piece works but could not raise the K1,500 fee, so I just thought of dropping out of school. It is the project that took me back to school. —female secondary school bursary recipient 18 | IMPA CT Program MIDTERM EVALUATION grandparents was failing because she was overburdened with domestic responsibilities. Staff intervened with the family, and an aunt was co-opted to assist with household chores so that the girl could have time to study. Another girl dropped out because of a forced marriage; the IMPACT partner and local community took action, and she was returned home and went back to school. During a key informant interview (KII), the head teacher remarked, “I can say that the project takes particular interest in the performance of the children and really involves a wide community of stakeholders in educating vulnerable children. The school is involved in distributing uniforms, counseling students, and monitoring behavior and performance of children, and the project staff is always available when needed by the school.” Some partners initiated career days for bursary recipients to motivate students and help them understand the educational and personal requirements of various career options. The financial sustainability of school bursaries is always an issue of concern for OVC programs, and IMPACT is no exception. It is uncertain if the GOM will be able to absorb needy students. IMPACT has addressed this in part by only enrolling children who will graduate by the end of the program and by encouraging vulnerable households that receive secondary school bursaries to join village savings and loan (VSL) groups. Educational Drop-In Centers The IMPACT baseline survey found that only 12% of OVC and 6% of non-OVC aged 13–17 years were in secondary school, indicating that most students were studying below their age/grade equivalent. In response, IMPACT consulted extensively with communities and established 285 afterschool educational drop-in centers (EDCs) to promote retention and graduation of primary school students; this activity was not part of the original program proposal. EDC sessions, supported by volunteer school teachers from the community, provide a place for children to do homework and receive mentoring to encourage good academic performance. EDCs have benefited 41,752 children, 300% higher than anticipated in FY2012—a testament to the popularity of this activity among teachers, parents and community leaders. In addition, EDCs have helped over 480 children who were out of school to re-enroll. A primary school boy in Nsomo Village in Chikwawa gets tutoring from a community educational drop-in center mentor. With training from the IMPACT Program, community mentors run weekly sessions for primary school students to get extra help on homework, enjoy a supportive study environment and access basic school supplies where needed. I can say that the project takes particular interest in the performance of the children and really involves a wide community of stakeholders in educating vulnerable children. The school is involved in distributing uniforms, counseling students, and monitoring behavior and performance of children, and the project staff is always available when needed by the school. —head teacher MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 19 Lessons Learned and Way Forward EDCs have struggled with high student–mentor ratios. Not all EDCs received books and equipment from IMPACT; IMPACT staff emphasized that books and equipment kits were intended as seed provisions, and implementing partners and communities would be expected to continue to contribute. In spite of these challenges, EDCs remain extremely popular, and communities may be able to find the space, supplies and mentor incentives necessary to sustain them. Box 2: Dropping In to Move Up—IMPACT Program Educational Drop-in Centers After school EDCs are a community-based solution designed to help primary school students succeed by providing an environment conducive for study, completion of homework, and mentorship and coaching by volunteer teachers, mentors and even secondary school students. EDC creation has six steps: 1. Identify community members willing to serve as drop-in session mentors (retired teachers, professionals or recent secondary school graduates not yet employed). 2. Identify a physical structure that can serve as a drop-in tutoring session (primary school during off-hours, a community-based childcare center, a church or other community meeting area). 3. Sensitize community members about the activity. 4. Organize weekly three-hour education drop-in sessions. 5. Provide educational supplies. 6. Conduct a three-day training facilitated by District Education Managers, Primary Education Advisors and District Social Welfare Officers for EDC mentors. A small study conducted by the OVC TQC in Chiradzulu after the 2012 primary school academic year end results were released found that the pass rates in three schools increased on average 33% after EDCs were opened in their communities (Figure 4). 0 20 40 60 80 100 Namalamba Ndunde Muluma 2010 Pass rate 2012 Pass rate Figure 4: Pass rates in three Chiradzulu district schools before and after EDCs were established (2010 and 2012, respectively). 20 | IMPA CT Program MIDTERM EVALUATION School Savings Accounts IMPACT partner Opportunity International Bank of Malawi (OIBM) is a commercial bank that targets economically active but marginalized Malawians living in underserved areas. During the first half of the project, the bank introduced Tsogolo Langa™ (My Future) accounts, whereby households were offered the opportunity to open interest-bearing school savings accounts. Access to accounts was through OIBM’s mobile bank program, with the idea that school fees would be transferred from accounts at OIBM directly to the schools. Lessons Learned and Way Forward Child savings accounts show promise in other contexts and the use of an existing institution has the potential to be sustainable. However, the product design of the Tsogolo Langa™ account did not consider that GOM schools may only have one account at one bank, most often not OIBM. As a result, the savings would also have to cover transfer fees, and OIBM has not yet resolved how to address this issue. During the first year of implementation, when OIBM provided bursary transfer services for implementing partners, there were several cases of delayed payment. As a result, some students were kept out of classes for short periods for nonpayment. OIBM’s mobile bank program faced financial challenges and certain routes were terminated, which raised alarm in community members who were no longer able to access their accounts. At the time of the MTE, OIBM was going through a period a staff turnover and a general restructuring. The new senior managers (interviewed during the MTE) stated that OIBM will get back on track and support future IMPACT programming, although mobile banking routes may not reach all the earlier community points. IR 1.4 OVC households have increased financial assets. Table 6: IR 1.4 indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Total savings (cumulative) deposited held by U.S. government– assisted microfinance institutions (VSL) $58,090 $69,227 119% Number of adults and children provided with economic strengthening services, by Age: <18, 18+ 6,410 9,200 29 15 7,302 1,854 144% Ac tivity Indicators Number of VSL groups formed and trained (new IMPACT groups only) 180 141 78% Number of VSL group members (new IMPACT groups only), by sex 4,500 2,377 6 2 1,881 488 53% Number of vulnerable household members in VSL groups* 6,410 7,471 26 15 5,949 1,481 117% * Represents vulnerable households from both WALA and IMPACT-only areas. MIDTERM EVALUATION IMPACT Program | 21 IMPACT uses CARE’s Village Savings and Loan Association methodology and Economic Activity Selection, Planning and Management manual to train VSL groups. VSL has been extremely popular, as evidenced by the continued increase in VSL members (Figure 5). Community leaders as well as program staff appreciate the tangible benefits of VSLs, such as access to self-financed microloans. In addition, VSLs are potentially self-sustaining, as one VSL focus group participant explained, “Now even if WALA or IMPACT ends, we will be able to continue our activities—we are empowered.” 0 10000 20000 30000 40000 50000 60000 70000 80000 500 1,000 1,500 2,000 2,500 FY11 Q1 FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 Savings (USD) IMPACT-Only VSL Members Total Members Vulnerable Members Cumulative Savings 0 Figure 5: VSL membership by cumulative US dollar value equivalent saved by IMPACT-only groups. VSL members reported a newfound ability to pay school fees and purchase school supplies and uniforms, farm inputs, household items, and food. This statement, from an IMPACT Q2FY2012 report, provides some insights into VSL’s popularity, “Life changed when I joined Tikondane Village Saving and Loan group in February 2011. [Since] I started realizing the benefits, my family life has opened a new chapter. Sleeping in a grass thatched house and being victims of food insecurity are stories of the past for my family.” During FGDs, even children noted the benefits that VSL participation had on improving household economic status. Program staff observed that VSL membership could be empowering for PLHIV who were able to make economic contributions to their households. Most VSL members are women, but men have been participating indirectly in some cases by providing money for shares or influencing the borrowing decisions of their wives. Men only become interested in joining VSLs after they see the benefits to the women’s group, as exemplified by this recent male VSL member, “I joined the group because of my wife. She came home with lots of money, and she managed to do a lot at home with the money which assisted us all in the family. I was convinced that this is the type of group I really need to join!” Now even if WALA or IMPACT ends, we will be able to continue our activities—we are empowered. —VSL participant Life changed when I joined Tikondane Village Saving and Loan group in February 2011. [Since] I started realizing the benefits, my family life has opened a new chapter. Sleeping in a grass thatched house and being victims of food insecurity are stories of the past for my family. —female VSL pa rticipant 22 | IMPA CT Program MIDTERM EVALUATION Sivingativute VSL group in Nsomo Village, Chikwawa, is now in its second year of saving together. Supported by the WALA and IMPACT integrated program activities, the group has 23 members, including vulnerable households hosting OVC and PLHIV. Their first share-out of group savings and profits, in 2012, was worth over MK660,000, about $2,000 at the time. Lessons Learned and Way Forward IMPACT’s VSL groups are performing well with occasional challenges around loan repayment, identification and management of viable economic activities and protection of accumulated funds toward the end of the cycle. Data from the indicator performance tracking table (Table 6) suggest that IMPACT-only partners have formed nearly 78% of their FY2012 targeted groups, although on average, these groups are smaller than recommended. Staff expressed two theories for why this might be occurring. First, the market may be saturated, meaning that many people in the catchment area may already be in a savings group. The second was that in IMPACT-only areas, OVC Promoters also served as community agents (CAs), which possibly reduced the amount of time dedicated to community mobilization. In 2012, the program introduced CAs, whom the promoters are charged with mentoring, in IMPACT-only areas. To ensure that communities maintain access to VSL services after the LoA, IMPACT will introduce CRS’ private service provider model in FY2013 and certify OVC Promoters and CAs with the right skills. Although it was intended that providing access to credit and formal savings opportunities through OIBM would be a significant activity contributing to IR 1.4, OIBM experienced significant organizational challenges during this period, and this approach was not implemented as planned. MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 23 SO 2: Access to treatment and care for 41,505 PLHIV is enhanced. PLHIV require a broad spectrum of services to ensure the best possible quality of life at all stages of infection. SO2 provides a continuum of care that increases access to HIV testing and counseling (HTC) for adults and children, supports Malawi’s transition to community-based pre-ART services, and reinforces the importance of adherence and follow-up for clients on ART and PMTCT. All services are community-based, with referral to health facilities and medical providers as needed. This strategy recognizes that the vast majority of Malawians living with HIV reside in rural areas, and an effective continuum of care will require an approach that maximizes sustainability and community ownership. Table 7: SO2 indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 18+ Years Percent Cumulative Target F M Ac hieved Number of eligible adults provided with a minimum of one care service*, by sex 20,290 26,978 20,290 7,932 133% Number of eligible pregnant or lactating women who received food and/or other nutritional services 1,800 1,481 1,481 N/A 82% Number of eligible clients (>18 years old) who received food and/or other nutritional services 7,200 15,108 10,673 4,435 210% Number of community health and para-social workers who successfully completed a preservice training program, by sex 5,749 1,800 1,032 768 31% * Care services include HTC, pre-ART services (community-based tuberculosis [TB] screening, promotion of cotrimoxazole preventive therapy, community-based nutrition assessment, and referral and prevention with positives), referral, and ART services. At midterm IMPACT had reached 26,978 of eligible adults with at least one care service, representing 65% of the LoA target of 41,505 adults. IMPACT did not meet its FY2012 target for Number of community health and para-social workers who successfully completed a preservice training program due to a delay in completion of the GOM curriculum that required IMPACT to postpone their training (Table 7). 24 | IMPA CT Program MIDTERM EVALUATION IR 2.1 Community mobilization for HIV testing and counseling is increased. Table 8: IR 1.2 Indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 Years 18+ Years Percent Cumulative Target Ac hieved F M F M 0–14 Years (HTC only) 15+ Years (HTC only) Number of individuals who received testing and counseling (HTC) services for HIV and received their results, by sex, by type of counseling/test: individual or couple 3,870 5,116 275 260 2,997 1,584 132% Number of individuals who tested positive for HIV, by sex and age 387 273 6 3 179 85 71% Number of HIV-positive individuals assessed for ART eligibility (through CD4 or clinical staging) within 6 months of testing positive 1,343 978 25 24 246 128 73% Ac tivity Indicators Number of people receiving at least one service at Tsiku la umoyo, by sex, by age: 0–17, 18+ 19,350 39,581 10,158 8,218 13,702 7,503 205% Number of mother– infant pairs (MIPs) referred for early infant diagnosis 3,285 120 4% HIV Testing and Counseling Through Community Health Days IMPACT’s strategy for community mobilization for HTC is holding Community Health Days (Tsiku la umoyo) for communities distant from government health facilities. IMPACT partners mobilize local health workers to provide the counseling and testing services at the Community Health Days, as well as a range of other services from malnutrition screening for children under age 5, hypertension screening, body mass index evaluation screening and blood pressure checks. Partner staff and volunteers undertake “edutainment” and awareness-raising activities such as cooking demonstrations and dramas about child protection. Some IMPACT partners have also hosted GOM health day activities such as those for World AIDS Day. Throughout the program, many partners have invited district officials to Community Health Days to demonstrate their importance to the community and to influence district-level budgeting decisions to ensure that Community Health Days continue after the program ends. Community Health Days have successfully increased access to care. However, during the first two years of the program, the MOH experienced widespread stock-outs of HTC kits, and as a result IMPACT partners were unable to provide HIV testing services. MIDTERM EVALUATION IMPACT Program | 25 Lessons Learned and Way Forward The MOH has opted for lifelong ART for positive pregnant or breastfeeding women. IMPACT partners assume that all positive pregnant women identified at Community Health Days who go to a health facility for prenatal care or delivery will receive ART, rendering IR 2.1 indicator, Number of HIV-positive pregnant women assessed for ART eligibility, no longer applicable. Follow-up and data collection for IR 2.1 indicator, Number of HIV-positive individuals assessed for ART eligibility (through CD4 or clinical staging) within 6 months of testing positive, proved challenging to collect. HTC clients who test positive are provided with a referral to a health facility. In tracking referrals of positive clients from Community Health Days to health facilities, some clients may not attend facilities in the program area, forget to bring their forms, or health care workers may not complete or file the form. All of these cases will lead to an underestimate of the proportion of clients completing referrals. Since partners are not MOH staff they are not authorized to assess ART eligibility nor do they have direct access to client medical records. To collect indicator data, IMPACT Coordinators accompany the district health management team (DHMT) during facility supervision visits. During these visits the DHMT compiles the client assessment records and supplies the de-identified data to the IMPACT Coordinator. One partner introduced an innovation of providing clinical staging of clients testing positive for HIV at Community Health Days, which could be further evaluated to determine its effect on care-seeking behavior. While joint visits have enhanced the collaborative relationship between the program and the district health office, tracking IR 2.1 indicator Number of HIV-positive individuals assessed for ART eligibility (through CD4 or clinical staging) within 6 months of testing positive as well as IR 2.2 indicator, Number of eligible individuals receiving a minimum of one pre-ART service, are either no longer relevant or beyond IMPACT’s manageable interest. The MTE recommends that these indicators be removed going forward. Box 3: Community Health Days Offer a One-Stop-Shop for HIV and Noncommunicable Disease Screening HIV is a known risk factor for noncommunicable diseases (NCDs) such as cardiovascular disease. Improved life expectancy due to antiretroviral therapy renders NCDs an important source of morbidity and mortality among people living with HIV. To address the increasing NCD burden and continue to expand access to HIV testing and counseling (HTC), the IMPACT Program integrated NCD screening into HIV services offered during Community Health Days. From October 2010 to September 2011, 19,200 people participated in Community Health Days: 2,409 people (11.4% children; 88.6% adults) underwent HTC; 288 people (12%) tested positive and were referred to health facilities for confirmatory testing and ART eligibility assessment. Women were more likely to undergo HTC (62.4%) than men (37.6%). Low stock levels of HIV test kits in-country limited the number of people tested. Over 800 people (4%) were referred to the health facility for follow-up care for NCDs, the most common being hypertension. IMPACT’s Community Health Days attracted a larger and more diverse population than HIV-specific events. Many attendees attracted by NCD screening ultimately opted to test for HIV while at the event, and many people living with HIV were screened for NCDs that might otherwise be overlooked in the HIV-specific care context. Integrating NCD screening with HIV services is an appropriate public health strategy in low-resource settings to address the dual challenges of HIV and NCDs. Source: Powell, A., Mambo, C., & Chibwana, A. (2012). One-stop shopping: Integration of non-communicable disease screening and HIV testing and counseling at Community Health Days in central and southern Malawi. Washington, DC: International AIDS Conference 2012. 26 | IMPA CT Program MIDTERM EVALUATION IR 2.2 PLHIV remain in longitudinal care and initiate treatment in a timely fashion. Table 9: IR 2.2 Indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of eligible individuals receiving a minimum of one pre-ART service**, by sex, by age: 0–17, 18+ 1,013 738 23 17 488 197 73% Number of PLHIV support group members in the pre-ART stage of care 1,350 803 59% Number of PLHIV support group members with 95% ART adherence in the previous 3 months 6,401 533 8% Ac tivity Indicators Number Expert Clients trained in pre-ART concepts and ART adherence, by sex 134 155 84 71 73% Support group facilitators trained on support group manual 424 184 116 68 59% Number of health care providers trained on Mother-Infant Pair Manual 217 81 30 51 8% ** Pre-ART services include community-based TB screening, promotion of cotrimoxazole preventive therapy, community-based nutrition assessment and referral and prevention with positives. MIDTERM EVALUATION IMPACT Program | 27 IR 2.3 Loss to follow-up of PLHIV on ART and in PMTCT reduced. Table 10: IR 2.3 Indicator performance tracking table. Indicator Cumulative FY12 Target Q3 FY12 Total Ac hieved 0–17 years 18+ years Percent Cumulative Target F M F M Ac hieved Number of HIV-positive pregnant women assessed for ART eligibility (through CD4 or clinical staging), by sex, by age: 0–17, 18+ 155 556 556 359% Number of HIV-positive (nonpregnant) individuals assessed for ART eligibility (through CD4 or clinical staging), by sex, by age: 0–17, 18+ 1,188 423 25 24 246 128 36% Ac tivity Indicators Number of adult PLHIV support groups supported 225 252 112% Number of adult PLHIV participating in support groups, by sex 6,750 8,586 6433 2153 36% Number of children's support groups formed 26 16 170% Number of children participating in support groups, by sex 80 247 117 130 32% Environmental Mitigation and Monitoring Indicators Number of HTC providers with sharps container in possession 84 52 62% Number of health facilities receiving used sharps from HTC providers 84 47 56% Number of health facilities with GOM IPC policy/guidelines 84 37 44% Support Groups IMPACT’s initial strategy to achieve IRs 2.2 and 2.3 was through community-based peer support groups for PLHIV. NAPHAM was brought into the program to provide technical assistance to support groups in nine districts. IMPACT provides funding for activities in five districts, while the National AIDS Commission (NAC) program is responsible for supporting the other four districts. Some support groups are an expansion of NAPHAM’s network of support groups; others were initiated by other IMPACT implementing partners. In collaboration with NAPHAM, IMPACT developed a support group manual in Chichewa focusing on support group formation and management. This material complements other technical content modules focusing on prevention with positives, positive living, and so on. NAPHAM has reported that the NAC funding is insufficient to cover all the IMPACT activities in those districts that may have contributed to delays in key program deliverables. Under IR 2.2, IMPACT is required to report on Number of PLHIV support group members with 95% ART adherence in the previous 3 months. The self-reported data are collected at support group meetings from public statements by support group members. This information is likely unreliable, which calls into question validity and utility of this indicator. The MTE recommends that this indicator be removed going forward. 28 | IMPA CT Program MIDTERM EVALUATION Lessons Learned and Way Forward Knowing that psychosocial services for HIV-positive children are virtually nonexistent in rural areas, IMPACT had planned to start support groups for HIV-positive children. In the absence of national guidance, CRS Malawi sought expertise from the Elizabeth Glaser Pediatric AIDS Foundation (EGPAF). Unfortunately, EGPAF guidelines and training for positive children’s support groups were found to require a level of effort in training and support in excess of IMPACT’s available resources. IMPACT has, therefore, been unable to achieve its target for children’s support groups (Table 10), and the MTE recommends that Number of children's support groups formed be removed going forward. Currently, caregivers of young HIV- positive children are encouraged to be members of a support group, while older children are encouraged to attend with their caregivers. These services are appreciated by program participants, as this grandmother of an HIV-positive, 9-year-old girl explained, “My granddaughter stopped getting sick so frequently; I would have lost her by now, I think, if I had not joined the support group.” Expert Clients A further strategy to strengthen referral networks and reduce defaulter rates involves the deployment of Expert Clients, based in health facilities, who provide peer education and peer counseling to HIV-positive clients, escort referrals between HTC and ART clinics, and between HTC and PMTCT clinics. By midterm, 155 Expert Clients had been trained (Table 9). A nurse shares requests for follow-up with an Expert Client—an experienced patient who works closely with health facility staff to support pre-ART and ART adherence. These requests will be delivered to the homes of patients who have missed their appointments, helping them return to treatment. Expert Clients are responsible for working with the health facility ART Team to identify and trace defaulters in collaboration with the relevant HSAs—the basic-level community health worker responsible for tracing defaulters. In practice, many HSAs are overwhelmed with responsibilities and have inadequate transportation resources for community follow-up. In these cases, IMPACT Coordinators deliver the tracer cards to PLHIV Support Group Community Facilitators to initiate follow-up (Figure 6). To ensure efficient use of resources and maximize coverage, IMPACT coordinates with other development projects with similar services ensuring catchment areas do not overlap. My granddaughter stopped getting sick so frequently; I would have lost her by now, I think, if I had not joined the support group. —grandmother of an HIV￾positive, 9-year-old girl MEGAN COLLINS FOR CRS MIDTERM EVALUATION IMPACT Program | 29 YES Defaulter list provided by ART Team to Expert Client in health facility. Expert Client reviews list. IMPACT Coordinator delivers tracer cards to Community Facilitators from PLHIV Support Groups and/or HSA in communities. Are defaulters in IMPACT catchment area? Expert Client writes tracer cards. NO Community facilitators/HSAs follow up defaulters in their homes. Senior HSA identies HSAs responsible for non-IMPACT catchment areas. HSAs follow-up defaulters in their catchment areas. Expert Client gives names to Senior HSA. Figure 6: Illustrative defaulter tracing pathway used by the IMPACT Program. Environmental Monitoring and Mitigation IMPACT has capitalized on its joint supervisory visits with the district health offices to deliver GOM infection prevention and control (IPC) policy/guidelines and sharps containers to health facilities. However, given the community-based nature of IMPACT’s activities, it is not advisable to report on supply-side indicators, such as Number of HTC providers with sharps container in possession, Number of health facilities receiving used sharps from HTC providers, and Number of health facilities with GOM IPC policy/guidelines, that are beyond IMPACT’s manageable interests (see Table 10). The MTE recommends that these indicators be removed going forward. Lessons Learned and Way Forward As a community-based program designed to support pre-ART and ART adherence, IMPACT had to cope with several supply-side factors that likely influenced program results. Most significantly, over the course of IMPACT implementation, widespread stock-outs of HTC test kits and cotrimoxazole were reported, especially during the second year of the program. Although less frequent, stock-outs of antiretroviral drugs were also reported. These stock-outs had serious implications on enrollment and retention in care and treatment across the country. Other factors that likely influenced client retention include health worker attitudes and distances to health facilities. All these factors are outside the manageable interest of IMPACT but do influence IMPACT’s ability to achieve its targets. As a result, the MTE recommends changes to the results framework and measurement of SO2 to reflect what is within the manageable interest of the program (see the Project Design section). cross-cutting areas MIDTERM EVALUATION IMPACT Program | 31 Cross-Cutting Areas Targeting and Beneficiary Enrollment The standard operating guidelines (SOGs) describe IMPACT’s four-step, family-centered approach for beneficiary selection (Box 4). A Chichewa language Beneficiary Enrollment Tool is used by volunteers to assess households across 12 domains and thereby prioritized for services. Using scores from the tool, households are classified into three categories of vulnerability: high, medium and low. Project staff reported confidence in the tool’s ability to identify the most vulnerable households and added that communities appreciated the transparency of the process, especially on high-value activities such as secondary school sponsorship. Once enrolled, the child is assessed using the CSI and referred for appropriate services. It is important to note that the project coordinates closely with District Social Welfare Office (DSWO) to ensure that lists, particularly of children in need of school support, are accurate and shared. Box 4: IMPACT’s Vulnerable Household Registration Process 1. Convene the OVC Committee or other leadership group at community level. Ask them to list the households that they feel are most vulnerable. Refer to a village register if available. 2. Visit the listed households and administer the IMPACT Beneficiary Enrollment Tool for verification purposes. 3. If household is confirmed vulnerable, enter the information for the entire household using the IMPACT Beneficiary Registration Form. This information may also simply be transferred from the Beneficiary Enrollment Tool. 4. Consult with the DSWO for a copy of all registered OVC. Verify that listed OVC in targeted communities have been included as registered beneficiaries. Source: IMPACT Program. (2012, January). Standard operating guidelines, version 1.2. Blantyre, Malawi: Catholic Relief Services. Adult program participants are not captured in this registration process but are registered through various program activities. For example, an adult living with HIV may attend support group meetings, or a pregnant woman with HIV may be identified through Care Group visits. 32 | IMPA CT Program MIDTERM EVALUATION Box 5: Pictorial to Participatory: Evolving Child Status Index (CSI) Use by the IMPACT Program Home visits by family care volunteers are an integral part of the IMPACT Program’s approach to OVC care and support. To assist in OVC assessment and referral, volunteers use the pictorial version of the CSI and record the results of their assessment onto an emoticon version of the tool developed by the program to aid less literate volunteers. Initially, CSI assessments were conducted monthly, but with more than 32,000 children in care and 12 CSI factors, enormous amounts of data were generated. In response, the program adjusted the frequency of CSI assessment to quarterly and then twice a year. However, even with these adjustments, large amounts of data continue to be generated. In addition, regular CSI assessment by program volunteers raised expectations among households, and with limited material support provided by the program and few referral options, households began to express their resentment to the volunteer data collectors about the absence of tangible benefit in exchange for the frequent probing. This was particularly acute in the food security domain, where IMPACT-only partners were unable to provide food supplements. In reflecting on this experience as part of the MTE, IMPACT technical staff resolved to modify the program approach. CSI data will still be collected via home visits every six months; however, instead of entering the information into the MIS, partners will support OVC Committees to continue case management for acute situations, while using the village-level data for community prioritization using a simple pile sorting methodology. This will enable OVC Committees to focus on two or three priority issues facing children in their communities over 6–12 months. This approach is also in keeping with the quality improvement efforts currently underway at national level. Monitoring and Evaluation and Learning M&E and Management Information Systems IMPACT has invested significantly in rigorous M&E and an MIS to facilitate reporting to key stakeholders, inform program improvement, document evidence for internal and external program accountability, and identify and document best practices to encourage learning. Data flow maps and data collection tools for each activity and indicator were developed during a workshop shortly after program initiation. An M&E manual documenting the process was developed and distributed to all partners. The manual includes all required Chichewa data collection forms. Data collection forms are reproduced centrally and provided to partners responsible for distribution to and collection from community-based promoters and volunteers. Data quality checks are carried out at all levels (household, community and district) of reporting as part of ongoing and spot supervision visits. Completed forms are entered by partner-level data-entry clerks into the Access-based MIS. Each month a file is emailed to the IMPACT M&E team, where it is merged with the data from the other partners. The use of unique identifiers and built-in validation rules by the MIS assists in checking for consistency and accuracy of data and allows users to stratify data by WALA and IMPACT households. IMPACT, like other community-based programs, faced the challenge of double-counting program participants because a single participant may benefit from several care services through multiple service provision points. In some cases, particularly where services are delivered independent of the household location (such as during Community Health Days or drop-in sessions), it is MIDTERM EVALUATION IMPACT Program | 33 not possible to use ID numbers to control for double-counting. In such instances, if services are counted and simply equated with people, this will result in inaccurate reporting. Therefore, IMPACT developed and tested a double-counting strategy and released a report in October 2012.3 The results were used to update IMPACT’s double-counting strategy and also helped to identify missed opportunities for children to benefit from multiple services. Lessons Learned and Way Forward To ensure accurate reporting, promoters and volunteers invest a significant amount of time completing forms that are sent to the project for reporting purposes, with limited opportunities for feedback to communities or health facilities. IMPACT staff are responding by exploring new approaches such as using the CSI as a participatory tool to help communities analyze and respond to OVC issues in their communities (Box 5). IMPACT staff will also attend a participatory M&E organized by WALA planned for February 2013. Overall, the M&E system is working well for tracking performance indicators for SO1, with the caveat that there is underreporting across several SO1 indicators. This is because the MIS did not become fully operational at the community level until Year 2 of WALA, after many Care Groups and VSL groups had already been established and registered on paper. This created a backlog of data entry into the MIS that, at the time of the MTE, WALA partners were still struggling to clear. In contrast, because IMPACT started around the same time as the MIS was being rolled out, IMPACT-only partners were able to register their groups at inception. As a result, the indicator performance tracking table data presented in Table 3 represents primarily the achievements of the IMPACT-only partners while underreporting the important achievements of WALA partners. SO2 indicators have been more problematic due to the selection of the indicators, which are not within the manageable interest of the program as designed. The MTE recommends these indicators be removed going forward. As described above, program staff strongly believe that the M&E system and the MIS are serving their intended purpose, which is specific to the needs of the WALA and IMPACT projects. Sustaining such a system after the projects end would not meet the informational needs of key stakeholders such as the GOM. That said, the processes used to develop the M&E system and MIS are replicable. The active participation of GOM and other Malawian staff during the systems’ development and the mentorship they received over the course of project implementation have and will continue to contribute to skills development and capacity strengthening that is sustainable and transferable. Learning The double-counting study discussed above is one of many learning initiatives undertaken or planned by IMPACT. Others include evaluations of the “In Charge!” methodology, Expert Clients, and user testing of the IMCI, CSI and MIP mobile applications. Operations research undertaken by the program contributes to improving practice in Malawi and elsewhere as IMPACT makes a concerted effort to disseminate its work widely. For example, IMPACT has shared several program abstracts at international and national conferences. The project uses field visits, reflection events at the community and partner levels, technical working groups, and joint meetings between WALA and IMPACT to share information, develop new approaches and continuously improve quality. Once lessons are learned, they are incorporated into the SOGs and M&E system. For example, the Family Service Tracking Tool, an initial M&E form used by the program, proved too complicated for most volunteers, so the form was dropped and the M&E strategy was modified. 3 IMPACT Program Team. (2012). Results of Double Counting Strategy and Test. Blantyre, Malawi: Catholic Relief Services. 34 | IMPA CT Program MIDTERM EVALUATION Gender Integration Issues pertaining to the home, pregnancy, breastfeeding, health, nutrition, hygiene, and children are considered by most Malawians to be a woman’s domain, as evidenced by the large numbers of women4 who participate in Care Groups, serve as volunteers, sit on OVC Committees and join support groups. While services are open to men, women have been the earlier adopters. IMPACT has sought to actively engage men by incorporating a chapter on male involvement strategies into the Care Group training of trainers (TOT) manual, encouraging couples to serve as Care Group volunteers together and promoting participation in the WALA Couples Conference, It Takes Two—Supportive Fathers, Healthy Families, designed to help men understand their roles and responsibilities for maternal and child health. Other examples include male Expert Client–led and male-focused health education talks, and community sensitization campaigns. Leading by example and verbal encouragement from traditional leaders for male involvement have also been used as strategies. Box 6: Yes Men Can!—Male Involvement in IMPACT A persistent challenge in Malawi and many other countries is poor health-seeking behavior by men and limited male involvement in antenatal care and PMTCT. Chief Chibwana, of Mlomba traditional authority in Machinga district, decided to take matters into his own hands by opening a Community Health Day and getting himself tested for HIV. He then challenged men in his community to participate in various health activities, including PMTCT. His actions had a clear and positive impact on the demand for HTC, especially among men. Of the 132 people who were tested that day, 60 were men—representing 45% of those tested. During the previous Community Health Day, only 26% of the 92 people tested were men. Leading by example, Chief Chibwana from Mlomba traditional authority in Machinga district participates in HIV testing and counseling (HTC) at the start of a Community Health Day. Source: IMPACT Program. (2011, October). Annual Report for October 1, 2010 – September 30, 2011 (Fiscal Year 2011). Blantyre, Malawi: Catholic Relief Services. 4  To help track differential program impacts on men, women, boys and girls, IMPACT’s MIS and data collection forms include both age and sex. JORAM CHOMBO FOR EMMANUEL INTERNATIONAL MIDTERM EVALUATION IMPACT Program | 35 IMPACT has achieved impressive results in increasing the total number of male and female HTC clients (Figure 7). However, men still lag behind women as a proportion of people tested, representing on average 33% of HTC clients, with an even smaller proportion of couples going for HTC. This challenge is widespread throughout Malawi, and IMPACT staff plan to test additional strategies in FY2013. 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 FY11 Q1 FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 Adult HTC Clients (>15 years) Female HTC clients Male HTC Clients Couples HTC Clients Figure 7: Total number of adults (>15 years) receiving HIV counseling and testing broken down by female, male and couples clients. In addition, to support gender-responsive programming among adolescents, IMPACT-only partners have used “In Charge!” as a way to help adolescents explore gender norms, challenge gender inequality and encourage boys to take an active role in protecting girls against sexual and gender based violence. Disability Integration IMPACT has made significant efforts to identify and address the needs of children with disabilities (CWD) and change the attitudes of the community toward this population. Activities have included active case finding during the baseline survey, integration of disability criteria in the Beneficiary Enrollment Tool, engagement of facilitators from the Malawi Council for the Handicapped (MACOHA) in the TOT on child protection, training of OVC Committees on the rights of CWD and including images of people with disabilities into the “In Charge!” methodology to reduce stigma and discrimination. IMPACT has had some notable successes. An OVC Committee member from an EI catchment area explained that after frequent visits and counseling, one child who never went to school due to fear of mistreatment started going to school and is still there. In other cases, CPW and teachers were aware of CWD in their area and were frustrated that more had not been done to provide assistance or access to education. Lessons Learned and Way Forward With limited expertise and resources to address the diverse needs of CWD, IMPACT has relied on referrals to district resources such as MACOHA, district hospitals and DSWOs. Unfortunately, the services available in Malawi are woefully inadequate, and all too often little can be done to help the children and their families. In the upcoming fiscal year, IMPACT plans to link with 36 | IMPA CT Program MIDTERM EVALUATION Development and Integration of Persons with Disabilities, an NGO with a multidisciplinary team able to address both intellectual and physical disabilities. Their primary activity will be to train community volunteers to provide basic services to CWD referred to them by volunteers. Strategic Objective Integration As a community-based program, IMPACT was designed to provide multiple and complementary services to vulnerable households. Synergies exist between the SOs and among the intermediate results. These synergies are supported by several cadres of trained volunteers working with different target groups that engage in cross-referrals to other services supported by the program in the community to (and from) government services, primarily health facilities, through the HSAs. A small study to test IMPACT’s double-counting strategy found that 84% of adults participating in Community Health Days also participated in one or more of the following: VSL, Care Groups or PLHIV support groups.5 Likewise, Care Groups provide an important opportunity to identify and refer MIPs to health facilities for early infant diagnosis services, while PLHIV support group members and their children have benefited from participation in cooking demonstrations and nutrition education conducted as part of CCFLS. The activities under each SO also support GOM initiatives; for example, the double-counting study found that 79% of children participating in “In Charge!” sessions also participated in Children’s Corners,6 a GOM strategy designed to provide psychosocial support to OVC. Inter-IR benefits include, but are not limited to, the increased ability of VSL participants to address the needs of the children in their care, such as paying school fees and purchasing school supplies. Synergies are reinforced at the management level where the OVC and care and support (C&S) TQCs are co-located. As evidenced by the double-counting study and remarks made by FGD participants, the program is by and large successful at identifying and capitalizing on opportunities for cross-SO collaboration. WALA and IMPACT Integration IMPACT was designed in part to provide HIV-related care and support services to seven of the eight WALA partners charged with reducing food insecurity in targeted communities. To meet its objectives, WALA supports Care Group structures and diverse activities including the promotion of fuel-saving stoves, nutritious crops/foods (e.g., soy, orange-fleshed sweet potatoes), appropriate crop cultivation strategies, improved agribusiness and marketing strategies, and VSL. Additionally, rations are provided through Food for Work/Assets, supplementary feeding, and a food safety net for 6 to 12 months for selected households meeting specific criteria. Among shared WALA partners, IMPACT Program participants are those selected using the IMPACT targeting approach described above. At the community level, the programs act synergistically with households and communities benefiting from services provided by both projects. Specifically, IMPACT has provided access to school bursaries, drop-in centers, Tsogolo Langa™ (My Future) accounts, facilitation of PLHIV support groups, training of HSAs, referrals and links to health facilities, training and implementation of Expert Clients, and periodic Community Health Days in WALA catchment areas. Benefits observed by program staff included a decrease in HIV-related stigma in WALA areas due to support group and Expert Client activities. IMPACT Program participants have appreciated the fuel-efficient stoves, access to nutritious crops, technical assistance to support engagement in agribusinesses, access to VSL services and food rations in some cases. 5 IMPACT Program Team. (2012). Results of Double Counting Strategy and Test. Blantyre, Malawi: Catholic Relief Services. 6 Ibid. MIDTERM EVALUATION IMPACT Program | 37 Members of the Tikondane Support Group in Chibwanansamala Group Village Head (GVH) in Balaka show their maize field, planted using improved techniques promoted by WALA. Members contributed their own funds and bought 10 kilograms of maize seed. They plan to sell the harvest during the lean season to generate funds for their 38 members (32 adults and six children). With support from IMPACT and WALA, the group has benefited from training and also operates its own village savings and loan (VSL) group. There is a strong working relationship between the WALA-IMPACT COPs and DCOPs, facilitated in part by shared office space and regular coordination meetings. The arrival of a WALA DCOP experienced in both PEPFAR and Title II management helped align WALA’s food aid recipient selection processes with IMPACT’s targeting mechanism, thereby improving access to food aid resources for food-insecure IMPACT Program participants. Lessons Learned and Way Forward IMPACT’s SO2 activities were designed to be implemented as part of WALA’s MCHN structure. To support integration MCHN/C&S Coordinators and Supervisors as well as the M&E Coordinator were provided with a 10% level of effort (LOE) to support IMPACT. The introduction of many new and innovative HIV-related activities from IMPACT created a steep learning curve for WALA staff; but more importantly, the IMPACT’s reporting requirements increased demands on M&E staff, requiring a lot for only a 10% LOE. As a result, IMPACT is perceived by WALA staff as an underfunded mandate. Program staff interviewed felt that the management challenges could be addressed by (1) increasing LOE available for IMPACT, especially on M&E; and (2) increasing collaborative leadership to the partners from the WALA MCHN TQC, IMPACT senior TQCs and technical staff. Importantly, these challenges are not apparent at the community level. Communities are benefiting from the complementary services provided by both WALA and IMPACT. Furthermore, staff interviewed from both programs saw value in their ability to provide a wide range of services to community members. Project Management and Staffing IMPACT is a learning consortium with regular technical and management meetings that take actions to improve the program. Partners remarked frequently about the responsiveness of the IMPACT management to their concerns. Although there are many meetings, respondents MEGAN COLLINS FOR CRS 38 | IMPA CT Program MIDTERM EVALUATION explained that the balance was right—there are not too many meetings but enough to support feedback and efficient activity. IMPACT has senior staff with both public health backgrounds and substantial OVC and HIV program experience. Inclusion of a key personnel staff member with clinical HIV experience would have further bolstered this team. Partners (principally NAPHAM) are providing support to PLHIV support groups, and Expert Clients were working with health facility staff and their community- based counterparts to increase treatment adherence. As IMPACT transitions to focusing on sustainability, NAPHAM is poised to continue its current role with the support groups and perhaps even manage the Expert Client program. However, for that to occur, substantial investments are needed to develop its technical and organizational capacity (principally fundraising to enable the organization to support the required staff ). Organizational development is not part of IMPACT’s mandate, and the approved staffing pattern does not call for an expert in organizational development. Moving forward, IMPACT would be well advised to place a heavier emphasis on organizational development of NAPHAM. Program implementation relies heavily on community volunteers. There are volunteers, such as OVC Committee members and Care Group members, who are committing a few hours a week to their communities and neighbors. There are other volunteers, including promoters and Expert Clients, who are expected to work more than 30 hours a week. In compensation for the long hours of project duties, promoters and Expert Clients receive small stipends, somewhat of a concern for institutionalizing IMPACT activities beyond the end of the project. Partners are addressing the challenges of motivating the unpaid volunteers, including clearly indicating to the people who are selected that they are volunteering to help their communities and providing small items such a cloth for zitenje (wrappers) for women that can be sewn by local tailors into shirts for male volunteers, as well as t-shirts and baseball caps. Turnover of volunteers was reduced following introduction of these changes. Project Design An evaluation of IMPACT’s results framework (see Figure 3) found that IRs 1.1–1.4 contribute to the wellbeing of OVC; however, IRs 2.1–2.3 do not create the conditions that are both necessary and sufficient to enhance PLHIV access to care and treatment. Community mobilization for HTC can only contribute to enhancing access to care and treatment if (1) there are appropriate, affordable, acceptable, available HTC and care and treatment services; and (2) there are strong links between the community mobilization and such HTC and care and treatment services. Many of the barriers to accessing care and treatment (and HTC) are supply-side (i.e., related to the Malawian health system), which are beyond the manageable interest of the IMPACT Program. Further, IR 2.2, PLHIV remain in longitudinal care and initiate treatment in a timely fashion, and IR 2.3, Loss to follow-up of PLHIV on ART and in PMTCT reduced, are not distinct IRs: Reduced loss to follow-up can be considered an indicator that PLHIV are remaining in longitudinal care. As a consequence, the IMPACT strategy for achieving IRs 2.2 and 2.3—through deployment of Expert Clients—is the same for both IRs. Furthermore, Community mobilization for HIV testing and counseling is increased and PLHIV remain in longitudinal care and initiate treatment in a timely fashion are not the necessary and sufficient conditions for enhanced access to care and treatment. Importantly, this hierarchy of results issues can be remedied without changing the program scope or activities. MIDTERM EVALUATION IMPACT Program | 39 limitations recommendations & conclusion 40 | IMPA CT Program MIDTERM EVALUATION LIMITATIONS OF THE MIDTERM EVALUATION The qualitative methodologies used in the MTE, combined with data from project documents and the IMPACTS MIS, were appropriate for an evaluation that sought to identify areas of project success in order to consolidate and build these gains over the remainder of LoA, and to identify challenges and opportunities for course corrections. While valuable, this approach did not allow the evaluators to compare progress on indicators collected during the baseline survey, as data for these impact-level indicators were not regularly collected through the project’s MIS. Quantitative data drawn from the project’s MIS did provide an opportunity for triangulation of qualitative and quantitative data. The quantitative data available were assumed to be accurate and of good quality, as a data quality audit was not part of the MTE. The large number of MTE questions, diverse activities in the field and limited amount of time for data collection during Phase 1 meant it was not possible to go into a great deal of depth in the KII or FGD. Furthermore, only one focus group per activity per partner was conducted; therefore any attempt to quantify findings from the FGDs to create generalizable statements about the success of specific activities across all partners could lead to erroneous conclusions. The quotes gleaned from FGD and KII do provide rich and valuable insights into stakeholder perspectives. RECOMMENDATION S 1. Continue learning from and adopting promising practices across all implementing partners to enhance consistency and impact. As shown above, IMPACT is broadly on track to deliver its SO targets. There is strong programming on many levels, with variations in implementation approaches and performance across partners. Management systems are in place both to identify areas for further strengthening and to assist partners in learning from one another. 2. Place greater emphasis on community system strengthening to extend the IMPACT Program’s results beyond the life of the project. The community social welfare systems that have been built or reinforced need to be institutionalized. Strengthened structures or new activities under SO1 needs to be integrated with the GOM line ministry structures or, when appropriate, with local government (Chiefs and the Village Development Committees) or with the GOM-mandated community-based organizations. Lines of supervision or oversight need to be agreed upon with supervision facilitated by these groups rather than IMPACT partners taking the lead on oversight. This applies especially to Care Groups, EDCs and strengthened village-level OVC Committees. IMPACT will need to work closely with WALA to develop a network of certified CAs using the private service provider model. Similarly for SO2, a new emphasis on building referral systems and client tracking in the community is vital to sustaining results. Importantly, Expert Clients need institutionalization; however, the MOH cannot do that and retain the nature of the cadre. Institutionalizing Expert Clients with NAPHAM would ensure that empowered, committed PLHIV continue to be Expert Clients and the focus remains on the needs of PLHIV. IMPACT should work with NAPHAM to enhance their financial, resource mobilization, human resource and management capacity, allowing NAPHAM to enhance its support to IMPACT partners to sustain the support groups and continue the innovative work of the Expert Clients. Moving forward, IMPACT would be well advised to place a heavier emphasis on organizational development of NAPHAM. 3. Strengthen the design of SO2 and the indicators for measuring performance to reflect the strategies and activities IMPACT is employing for SO2. It is unreasonable for IMPACT to report against health facility supply-side indicators when its activities are primarily in the community and demand-side. Figure 8 presents a revised results framework for consideration. MIDTERM EVALUATION IMPACT Program | 41 Goal: Improved quality of life and HIV impact mitigated on 99,522 orphans and vulnerable children (OVC) and people living with HIV (PLHIV) in Malawi. Intermediate Result 1.1 OVC caregivers have practiced improved infant and young child feeding and caring techniques. Intermediate Result 1.2 OVC are protected from abuse and exploitation. Intermediate Result 1.3 OVC access to secondary education is improved. Intermediate Result 1.4 OVC households have increased financial assets. Strategic Objective 2: Strengthened community-based systems deliver health care, support services, and referrals for people living with HIV. Intermediate Result 2.1 Communities are mobilized to facilitate HIV counseling and testing services and to refer individuals effectively for care and treatment. Intermediate Result 2.2 PLHIV in the pre-ART and ART stages of care are followed up and retained for better treatment outcomes. Intermediate Result 2.3 NAPHAM’s organizational and technical capacity reinforced to further assist community PLHIV support groups. Strategic Objective 1: Wellbeing of 58,017 OVC is improved. Figure 8: Revised IMPACT results framework for consideration. CONCLUSION Over the course of the first two years of operation, IMPACT has accomplished most of what it set out to do, reaching or exceeding many of its targets. Systems have been put in place, capacities strengthened, and a number of promising practices and new innovative approaches are being developed and tested. These approaches will benefit not only the families, children and PLHIV being served by IMPACT, but also others in Malawi and elsewhere. Despite these successes, there have been some important challenges, many of which are related to the operating environment that provides few referral options for the most vulnerable, including children with disabilities and children living with HIV. Persistent supply-side challenges endemic to the health system need to be addressed, but are not within IMPACT’s scope. To assist the project in consolidating gains, a shift should be made to focus on community system strengthening. Much of the work has already been done. What remains is to institutionalize these systems so they are able to continue to provide lasting benefits to Malawi’s most vulnerable populations. For more information contact: IMPACT Program Catholic Relief Services Malawi Tel: +265 1 822 800; 1 757 272 www.crs.org