ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY December 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Dr. Ruth Hope and Ms. Uvani Bodasing through the Global Health Technical Assistance Project. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 09-001-248) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY i ACKNOWLEDGMENTS The evaluation team is grateful for the time provided by all the respondents for answering our questions—we realize they all had to take time out from busy schedules—and for their thoughtful responses to our enquiries. We found the briefings by IntraHealth staff very helpful, as were the perspectives of MSH/HCSP and Pathfinder/IHFP staff who have had IntraHealth PMTCT sites, including Mothers’ Support Group (MSG) sites, transitioned to their management. The briefings added context to our understanding of the MSG strategy and our site visits. We particularly appreciate that the briefings came at a time when IntraHealth staff had closed out the PMTCT project and were busy with wrap-up activities, and also during a week that had both a U.S. federal holiday (Labor Day) and an Ethiopian National Holiday (New Year), making people’s office time at a premium. The evaluation report would not be so rich without the qualitative feedback from the mentor mothers and their MSG members, as well as the site coordinators and case managers visited in the Addis Ababa and the Oromia and Amhara regions. The evaluation team was fêted with coffee ceremonies and was helped immensely by all the respondents at the 12 PMTCT sites with MSGs that we visited. We thank Dr, Michael Dejene for efficient logistical and other arrangements; the schedule for the interviews and site visits was very efficient, and we are grateful for the smooth travels. Finally, we appreciate all the essential assistance provided by GHTech in Washington DC, without which we would have been unable to perform our duties. The Consultant Evaluation Team September 2009 ii ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY iii ABBREVIATIONS AND ACRONYMS AIDS Acquired immune deficiency syndrome ANC Antenatal care ART Antiretroviral therapy ARV Antiretroviral DHS Demographic and Health Survey DQA Data quality audit FP Family planning FY Fiscal year GHTech Global Health Technical Assistance Project HAPCO HIV/AIDS Prevention and Control Office HC Health center HCSP HIV Care and Support Program HCT HIV counseling and testing HEW Health extension worker HIV Human immunodeficiency virus HP Health post IFHP Integrated Family Health Program IGA Income-generating activity JHU Johns Hopkins University M&E Monitoring and evaluation M2M Mothers2Mothers [an international NGO founded in South Africa] MCH Mother and child health MM Mentor mother MoH Ministry of Health MSG Mothers’ support group MSH Management Sciences for Health OGAC Office of the Global AIDS Coordinator, USAID OVC Orphans and vulnerable children PEPFAR President’s Emergency Plan for AIDS Relief PMTCT Prevention of mother-to-child transmission of HIV PLWHA People living with HIV and AIDS RH reproductive health iv ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY SoW Scope of work TWG (PMTCT) technical working group USAID/E U.S. Agency for International Development/Ethiopia WFP World Food Programme ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY v CONTENTS ACKNOWLEDGMENTS .................................................................................................................... i ABBREVIATIONS AND ACRONYMS ............................................................................................. iii EXECUTIVE SUMMARY .................................................................................................................ix INTRODUCTION AND BACKGROUND .................................................................................ix EVALUATION FINDINGS ......................................................................................................... x RECOMMENDATIONS ...........................................................................................................xi I. INTRODUCTION ..................................................................................................................... 1 II. BACKGROUND ....................................................................................................................... 3 THE HIV EPIDEMIC IN ETHIOPIA ......................................................................................... 3 THE HEALTH SECTOR .......................................................................................................... 3 THE PRESIDENT’S EMERGENCY PLAN FOR AIDS RELIEF.............................................. 4 THE EVALUATION .................................................................................................................. 4 III. EVALUATION OBJECTIVES AND QUESTIONS ................................................................... 5 THE OBJECTIVES .................................................................................................................. 5 THE EVALUATION QUESTIONS ........................................................................................... 5 IV. EVALUATION METHODOLOGY ............................................................................................ 7 THE EVALUATION TEAM ....................................................................................................... 7 THE APPROACH TO THE EVALUATION .............................................................................. 7 THE METHODOLOGY ............................................................................................................ 7 LIMITATIONS .......................................................................................................................... 8 EVALUATION TOOLS ............................................................................................................. 8 SCHEDULE ............................................................................................................................. 8 V. FINDINGS ............................................................................................................................. 11 INTRODUCTION ................................................................................................................... 11 INTERVIEWS AND PUBLISHED DOCUMENTS .................................................................. 11 FROM ANALYSIS OF M&E DATA AND REPORTS ............................................................. 13 FROM SITE VISITS ............................................................................................................... 14 VI. HOW THE FINDINGS RESPOND TO THE EVALUATION QUESTIONS ............................ 19 PROGRAM MANAGEMENT ................................................................................................. 19 SERVICE DELIVERY ............................................................................................................ 20 SUPPORTIVE ENVIRONMENT FOR PMTCT CLIENTS AND THEIR HOUSEHOLDS ...... 21 CAPACITY-BUILDING AND SUSTAINABILITY .................................................................... 23 MONITORING AND EVALUATION ....................................................................................... 24 LESSONS LEARNED ............................................................................................................ 25 VII. RECOMMENDATIONS ......................................................................................................... 27 vi ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY APPENDIXES APPENDIX A. SCOPE OF WORK ................................................................................................ 29 APPENDIX B. PERSONS CONTACTED ...................................................................................... 41 APPENDIX C. EVALUATION TOOLS ........................................................................................... 43 APPENDIX D. REFERENCES ...................................................................................................... 49 TABLES Table 1. MSG Intervention Strategies ........................................................................................... 12 Table 2. Sites Visited for Evaluation and Respondents Met ......................................................... 14 Table 1A. MSG Intervention Strategies ......................................................................................... 31 FIGURES Figure 1. Map of Ethiopia ............................................................................................................... vii Figure 2. Number of MSGs by Region, October 2007–June 2009 ............................................... 13 Figure 3. The Experience of Mothers in ANC, October 2007–June 2009 ..................................... 13 Figure 4. Experience of Mothers’ Support Group Members, October 2007–June 2009 ............... 14 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY vii Figure 1. Map of Ethiopia IntraHealth introduced mothers’ support groups in Addis Ababa, Oromia, Dire Dawa, Amhara, Tigray, and Southern Nations, Nationalities and Peoples. The evaluation team conducted site visits in Addis Ababa, Oromia, and Amhara. viii ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ix EXECUTIVE SUMMARY INTRODUCTION AND BACKGROUND Starting in 2005 IntraHealth introduced mothers support groups (MSGs) modeled on the South Africa Mothers2mothers (M2M) program at its prevention of mother-to-child transmission of HIV (PMTCT) sites as a strategy to increase uptake of PMTCT services and behaviors so as to reduce vertical transmission of HIV. By June 2008 IntraHealth had expanded MSGs to 84 sites and continued to expand into 2009. Although during this period there were anecdotal accounts of successfully supporting positive women, no external evaluation was undertaken. Because its project was ending in 2009, IntraHealth transitioned its PMTCT sites to other PEPFAR partners in the first half of 2009: 52 MSGs were transitioned to the Management Sciences for Health (MSH) HIV Care and Support Program (HCSP) in three tranches. USAID/Ethiopia (USAID/E) requested technical assistance to undertake an external evaluation by September 2009 of the IntraHealth MSG strategy, with special attention to the 52 sites so that the findings and recommendations could be used in planned other PMTCT programs. The evaluation was undertaken by an international consultant team leader and a regional consultant PMTCT specialist, with an Ethiopian consultant providing logistics support. The USAID/E Psychosocial Support Adviser and two Ethiopian research assistants helped to collect the field data. The specific objectives of the evaluation were to look at whether the MSG strategy is achieving its objective of empowering positive mothers and increasing utilization of PMTCT/reproductive health (RH) services and behaviors; and assess the viability and collaborative advantage of facility-based MSG in rapidly scaling up access to quality HIV/AIDS services. Evaluation questions covered program management, service delivery, a supportive environment for PMTCT clients and their households, capacity building, and sustainability, monitoring and evaluation (M&E), and lessons learned. The evaluation used several methodologies: Review of published documents and reports Analysis of IntraHealth MSG and PMTCT M&E data Key informant interviews Structured discussions Focal group discussions. The methodology was designed to allow triangulation of findings whenever multiple sources were available. The team developed evaluation tools—an interview guide, discussion guides for meetings with health facility staff and mentor mothers (MMs), and a guiding framework for focus group discussions with MSG members—that would elicit the information needed. The evaluation team (1) interviewed key informants in Addis Ababa during the first and second weeks of the evaluation; (2) piloted the site visit tools at health facilities in Addis at the end of the second week; and (3) undertook site visits in Addis, Oromia, and Amhara during the third week. x ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY USAID/E selected the sites, using a sample of convenience that would maximize the number of sites visited within the brief time available. EVALUATION FINDINGS The timing of the evaluation was less than optimal because IntraHealth had already closed out its PMTCT and MSG activity and had lost some institutional memory. It was thus very difficult to evaluate the company’s management structure and efficiency. The situation was further complicated by HCSP changes to the management structure and processes for the MSGs. The Documentation The evaluation team was unable to ascertain how many MSG sites IntraHealth had founded; there were inconsistencies between documents, reports, and the M&E data. Some planned MSG sites were found to have too few HIV-positive pregnant women to justify establishing MSGs after staff were trained. The evaluation scope of work and the IntraHealth briefing suggested that the MSG intervention had four components: 1. Mother to mothers-to-be 2. Mothers to mothers 3. Mothers to community 4. Mothers’ creation – income generation/self-help savings group The IntraHealth publication Mothers Support Groups in Ethiopia (2008) described only the first three intervention strategies; the fourth was added after the publication. There is no institutional memory as to why the fourth strategy was added—it is not part of the original M2M program. Site Visits It was clear at all 12 sites visited that the MSGs have greatly and positively changed mothers’ lives. Health workers, MMs, and MSG members all agreed that HIV awareness and motivation for HIV counseling and testing have increased and that MMs provide much-needed psychological support to antenatal care (ANC) for those who have tested positive. Continuing psychosocial support and peer counseling ―relieves the stress‖ on positive mothers; and many MMs provide special advice and counseling for positive mothers in discordant relationships. The MSGs provide information and behavior change motivation for living positively and not transmitting HIV to others; based on their own experience the MMs provide information and advice about taking antiretroviral (ARV) drugs and their side effects that many health workers acknowledge is more effective than the advice they provide that is grounded in theory rather than experience. MMs make home visits to persons who have dropped out of PMTCT and ARV therapy clinic, and in the early postnatal period to women who deliver in the community. MMs also provide information on family planning (FP) and advocate for it and for use of condoms for safer sex at all times. Another impact on positive mothers’ lives was reduced stigma (not only stigma in the community and from health workers but also self-stigma). MMs report that they have increased social standing now that they are seen and respected as health facility staff, and are valued by community leaders for their work in awareness-raising and prevention. They report greatly enhanced feelings of self-worth, self-confidence, and self-respect. The MMs and MSG members overwhelming expressed a desire for male participation in MSGs—because fathers should be involved in PMTCT, because it is better that men and women test and get their results together, and because positive women report problems when they take ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY xi home information on HIV, PMTCT, and safer sex. Thus it is better that male partners learn about these things at the same time as the women. Although some of the 12 sites visited were in more affluent districts in Oromia and Amhara, in those areas HIV- positive MSG members are among the poorest of the poor. They are far more likely than the general public to be widowed, divorced, deserted, or cast out by their families, and almost all of them have dependent children. World Food Programme support was available to positive mothers in Addis and Oromia but not in districts visited in Amhara, where nutritional security for positive women and their children was an issue. In every site visited, lack of income￾generation activities (IGAs) was a major issue, although some sites had within the previous two months commenced micro-savings clubs or handicraft activities. None of these was likely to provide significant financing in the foreseeable future. For most MSGs the issue was lack of seed money or other support to establish an IGA. MSG members at one site in Oromia were involved in a gardening project on land negotiated by a local association of persons living with HIV and AIDS (PLWHA) from the local council. Many health facilities visited had unused land within or nearby. Challenges faced by MMs included allowances that did not cover their travel or coffee ceremony expenses; demands for financial and other material support from MSG members; and sometimes homeless and abandoned MSG members. RECOMMENDATIONS 1. USAID/E and the PEPFAR partners implementing an MSG strategy to support PMTCT should ensure that baseline data is collected before an MSG is set up at PMTCT sites; and should ensure that their M&E systems are robust enough to generate accurate data on outcomes and the impact of MSGs on PMTCT indicators and pediatric AIDS incidence. 2. If, as seems likely from the qualitative evidence, MSGs are an effective strategy, advocacy may be needed with the Ministry of Health (MoH) HIV and AIDS Prevention and Control Office (HAPCO) about what is necessary to implement the strategy. Implementation nationally requires a plan that addresses human and other resources, training, support supervision, coaching of the technical staff that manage MMs and the MMs themselves, and a schedule for the roll-out. Advocacy may also be needed to address remuneration for MMs, who under MoH/HAPCO policy would not receive a stipend or allowance: MSGs are unlikely to be successful without some form of remuneration to the MMs, who need to work at least 5 days out of 10 to be fully effective in a health center (HC) and up to 10 days out of 10 if undertaking home visits and coffee ceremonies in the community. To avoid conflict with the policy on volunteers, MMs could be considered lay health workers rather than volunteers. 3. If MoH/HAPCO wants to extend the reach of MSGs by having them at health posts (HPs), USAID/E may want to implement a well-documented pilot program at HPs to learn what is necessary for institutionalizing MSGs there. 4. Similarly, USAID/E may want to implement a well-documented pilot program for MSGs at HCs offering a reduced curriculum but permitting mothers to be referred for longer-term psychosocial support to satellite MSGs in their home kebeles. 5. Partners implementing MSGs should pool their expertise on such issues as how many positive ANC attendees are enough to sustain an MSG at an HC and how many are enough for decentralizing MSGs to kebele HPs. xii ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 6. Consideration should be given to changing the last module of the MSG training manual, which concerns IGAs, to a module concerned with building life skills, such as problem￾solving, gardening, negotiating with HC management and local leaders, and perhaps partnering with local organizations that provide income-generation and business opportunities. Links between MSGs and the urban gardening program should particularly be encouraged. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 1 I. INTRODUCTION Uptake of services for prevention of mother-to-child transmission (PMTCT) of HIV services in Ethiopia is often low, in part because many Ethiopians have little contact with the health care system. However, there are many other barriers that have not been fully studied, such as limited use of antenatal and postpartum services generally, HIV-related stigma and discrimination, the uneven quality of care in antenatal and obstetric settings, and insufficient partner and community involvement. Other barriers might be the psychosocial needs of the mother during pregnancy, delivery, and the early postnatal period, particularly where important cultural rituals and behaviors cannot be practiced; behavior changes, such as exclusive breastfeeding and taking drugs in pregnancy, that conflict with community norms, values, and beliefs; and economic barriers, such as the direct and opportunity costs of accessing PMTCT services. IntraHealth initiated mothers’ support groups (MSGs) in Ethiopia in 2005 and the Capacity Project has continued them as a strategy to increase the uptake PMTCT services at government health centers (HCs). The MSG program is the product of knowledge-sharing between IntraHealth and the pioneering Mothers2Mothers (M2M) program in South Africa.1 To meet the information and emotional support needs of HIV-infected pregnant and postpartum women, and to improve utilization of PMTCT services in South Africa, M2M was started in the Western Cape in 2001 and rolled out in KwaZulu-Natal. M2M employs and trains new mothers who are HIV-positive to become ―mentor mothers‖ (MMs) who offer psychological, emotional, and social support to other HIV-positive women through one-on-one and support group sessions in antenatal care (ANC) and maternal and child care (MCH) clinical settings. The program serves women from pregnancy through the first year of their children’s lives. 2 The M2M program3 empowers women who test positive by providing psychosocial support, promoting women’s economic independence, reducing stigma, and promoting disclosure in families and communities. In 2005 U.S. First Lady Laura Bush met with members of M2M when she visited the mentoring program for HIV-positive pregnant women and new mothers in Cape Town; the following year, six M2M members went to Washington, D.C. to highlight the strength and leadership of women in South Africa’s fight against HIV/AIDS. Presentations on M2M in 2004 at the 15th International AIDS Conference in Bangkok4 and in 2007 at the HIV/AIDS Implementers’ Meeting in Kigali, Rwanda were well-received. IntraHealth country staff visited M2M in Cape Town and at various sites, and M2M staff made an advisory visit to Ethiopia, helping to train trainers for IntraHealth’s MSGs and providing a training curriculum and materials for adaptation to Ethiopia and translation into Amharic. A major adaptation to the M2M model is that MSG MMs are volunteers who receive only a small stipend to cover transport costs5 whereas M2M’s MMs are paid staff.6 1 http://www.comminit.com/en/node/267100/cchangepicks (accessed 9/3/2009). 2 Ibid. 3 In South Africa m2m is supported by the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) http://www.pepfar.gov/press/75987.htm Accessed 09/03/2009. The program was cited by the Office of the Global AIDS Coordinator as an example of meaningful involvement of persons living with HIV. http://www.pepfar.gov/pepfar/press/79671.htm (accessed 09/03/09). It was also evaluated with PEPFAR funding by Horizons: see Baek, C, et al. (2007) Key Findings from an Evaluation of the mothers2mothers Program in KwaZulu-Natal, South Africa, Horizons Final Report ( Washington: Population Council). 4 http://gateway nlm nih.gov/MeetingAbstracts/ma?f=102281450.html (accessed 9/3/2009). 5 Viadro, C., et al. (2008) Mothers' Support Groups in Ethiopia: A Peer Support Model to Address the Needs of Women Living with HIV (Chapel Hill, NC: IntraHealth). 6 Baek, C., et al. (2007), Key Findings from an Evaluation of the mothers2mothers Program in KwaZulu￾Natal, South Africa, Horizons Final Report (Washington: Population Council). 2 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY The goal of the MSG strategy is to reduce mother-to-child transmission by empowering HIV￾positive mothers and mothers-to-be to make informed decisions about their reproductive health (RH) and the health of their babies. Both the M2M and MSG programs are based on the concepts that peer support is an optimal model for effective education and social empowerment, and that mothers are particularly well-suited to provide support to other mothers.7 HCs were selected to become MSG sites based on ANC and PMTCT flow, HIV prevalence, and the availability of space for support group meetings. HC staff, especially ANC HIV counselors, had to be willing to work with MSG program site coordinators and MMs. IntraHealth provided modest material assistance to the HCs and paid a monthly stipend to site coordinators to cover their time, transport allowances to the MMs, and a small sum for the purchase of coffee and fuel for the MSGs. After initial resistance, the MSGs were enthusiastically received once health workers at PMTCT sites realized that the MMs and MSGs reduced the burden on staff rather than adding to it.8 The enthusiasm of staff, MMs, and MSG members was transferred to staff of the Ministry of Health (MoH) HIV/AIDS Prevention and Control Office (HAPCO) who visited the sites. Several other PEPFAR partners have begun implementing MSGs at their PMTCT sites. The program’s appeal to both providers and participants has led the HAPCO to incorporate MSG into its national HIV/AIDS response strategy9 and its recently revised national PMTCT guidelines.10 The HAPCO vision is for MSGs to be decentralized to health posts (HPs) closer to the community, supported by health extension workers (HEWs) and helping to integrate of PMTCT into broader MCH/FP services.11 Although the Ethiopian MSGs began in 2005 in three12 PMTCT sites, scaled up to 34 sites in 2006–0713 and 84 sites by June 2008, and there were anecdotal accounts of successful support of HIV-positive women, the strategy had yet to undergo an external evaluation. Because its funding for PMTCT was ending, in the first half of 2009 IntraHealth transitioned its PMTCT sites to other PEPFAR partners; 52 sites with MSGs were transitioned to the HIV Care and Support Program (HCSP) operated by Management Sciences for Health (MSH), and other PMTCT/MSG sites14 were transitioned to the Integrated Family Health Program (IFHP) operated by Pathfinder. 7 Ibid. 8 Dr. Fikir Melese, UNICEF/Ethiopia, 9/10/2009 [she was previously head of the Family Health Department, Addis Ababa Health Bureau]. 9 HAPCO (2007),Accelerated Access To HIV/AIDS Prevention, Care and Treatment in Ethiopia (Addis Ababa: Federal Ministry of Health). 10 HAPCO (2007), Guidelines for Prevention of Mother-to-Child Transmission of HIV In Ethiopia (Addis Ababa: Federal Ministry of Health). 11 Director General, Federal Ministry of Health, 9/14/2009. 12 Viadro, C. et al. (2008), Mothers' Support Groups in Ethiopia: A Peer Support Model to Address the Needs of Women Living with HIV (Chapel Hill, NC: IntraHealth). 13 Ibid. 14 Please see [V. Findings] for the number of sites transitioned to IFHP. Determining the number of MSG sites established by IntraHealth was beset with difficulties because of loss of institutional memory. The numbers quoted here are those verified be contemporous documentation and differ from those quoted in the scope of work, which are inconsistent. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 3 II. BACKGROUND THE HIV EPIDEMIC IN ETHIOPIA With a population of more than 85 million,15 Ethiopia is the second most populous country in Africa and has the third largest number of persons living with HIV/AIDS (PLWHA). PLWHA number about 977,394 (41% males, 59% females).16 In 2007 an estimated 898,350 children had been orphaned by AIDS.17 National HIV adult prevalence for 2007 is estimated at 2.1%, of which 7.8% is urban and 1.0% rural.18 The highest prevalence occurs in those aged 15–24, and prevalence is higher among females than males in both urban and rural areas. In 2005 HIV prevalence among ANC attendees was 10.5% in urban areas, 1.9% in rural areas, and 3.5% nationally.19 Prevalence appears to have leveled off in urban areas but continues to rise in rural areas, where 85% of the population lives.20 THE HEALTH SECTOR The health status of the Ethiopian people is poor. Life expectancy at birth stands at 54 years (53 for men and 55 for women). The infant mortality rate is estimated to be about 77 per 1,000 births, and under-5 mortality is about 123 per 1,000. Poor nutritional status, infectious diseases, and a high fertility rate, together with little access to reproductive health (RH) and emergency obstetric services, contribute to one of the highest maternal mortality rates in the world, an estimated 673 per 100,000 births.21 The 2005 Ethiopia Demographic Health Survey (DHS) indicates that approximately 28% of pregnant women attend ANC at least once and an estimated 6% deliver in health facilities. In urban areas generally, 69% of women attend ANC and 45% deliver in a health facility.22 In Addis Ababa, nearly 90% of women attend ANC and over 75% deliver in a health facility. Although the PMTCT program was initiated in Ethiopia in 2003, uptake of PMTCT services has been consistently low.23 In 2008 there were a total of 1,069 hospitals, HCs, and private clinics in Ethiopia,24 with private clinics mainly located in major urban centers. There is one specialist maternity hospital in Addis Ababa. Referral hospitals are in larger population centers. In rural areas health care is provided through the public health system with HCs at the woreda level and HPs at the kebele level run by 15 CIA, World Factbook—Ethiopia, July 2009 est. https://www.cia.gov/library/publications/the-world￾factbook/geos/et.html (accessed 9/28/2009). 16 HAPCO (2007), Guidelines for Prevention of Mother-to-Child Transmission of HIV in Ethiopia (Addis Ababa: Federal Ministry of Health). 17 HAPCO (2007), Multisectoral Plan of Action for Universal Access to HIV Prevention, Treatment, Care and Support in Ethiopia 2007–2010 (Addis Ababa: Federal Ministry of Health). 18 HAPCO (2007), Accelerated Access to HIV/AIDS Prevention, Care and Treatment In Ethiopia (Addis Ababa: Federal Ministry of Health). 19 Davis, N., et al. (2009), Ethiopia Prevention of Mother to Child HIV Transmission (PMTCT) Portfolio Review and Recommendations (Washington: PEPFAR PMTCT and Pediatric HIV Technical Working Group).. 20 HAPCO (2007), Guidelines, n. 19 above. 21 Central Statistical Agency [Ethiopia] and ORC Macro (2006) ,Ethiopia Demographic and Health Survey 200. p.233 (Addis Ababa, Ethiopia, and Calverton, Maryland, USA: Central Statistical Agency and ORC Macro). 22 Central Statistical Agency [Ethiopia] and ORC Macro (2006) ,Ethiopia Demographic and Health Survey 2005 (Addis Ababa, Ethiopia, and Calverton, Maryland, USA: Central Statistical Agency and ORC Macro). 23 Davis, N, et al. (2009), Ethiopia PMTCT Portfolio Review and Recommendations, n. 22 above 24 Ibid. 4 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY HEWs, the lowest cadre of trained health workers. In Addis Ababa, where there are no HPs, a new cadre of urban HEWs—qualified nurses—is being introduced. The National PMTCT Guidelines were revised in 2007; the second edition of Accelerated Access to HIV/AIDS Prevention, Care and Treatment In Ethiopia was also published in 2007, as was the Multisectoral Plan of Action for Universal Access to HIV Prevention, Treatment, Care and Support in Ethiopia 2007–2010. THE PRESIDENT’S EMERGENCY PLAN FOR AIDS RELIEF PEPFAR was launched in 2003. Working in partnership with host nations, over 10 years PEPFAR plans to support treatment for at least 3 million people; help prevent 12 million new infections; and care for 12 million people, including 5 million orphans and vulnerable children (OVC). The U.S. Mission to Ethiopia’s HIV/AIDS interagency team began integrated HIV/AIDS programming in 2004 under the oversight of the Office of the Global AIDS Coordinator (OGAC) To support comprehensive HIV/AIDS prevention, treatment, and care programs, Ethiopia received more than $48 million in PEPFAR funding in FY 2004, more than $83.7 million in FY 2005, about $123 million in FY 2006, $241.8 million in FY 2007, and more than $354.5 million in FY 2008.25 USAID/E supports PEPFAR activities with numerous implementing partners. USAID/E assisted PMTCT has increased significantly, from $1,660,000 in 2005 to $12,053,975 in 2009. In 2005 the Mission initiated a four-year (2005–09) program with IntraHealth to establish the Ethiopia Mothers Support Group (MSG) program to address the special needs of pregnant and postpartum women who are living with HIV and caring for new babies. THE EVALUATION The USAID/E PEPFAR office requested technical assistance to design and implement an independent external end-of-project evaluation of the MSG program by September 2009 in order that its findings, conclusions, and recommendations could be used in the design of future PMTCT programs. The evaluation was to gather information about MSG implementation, progress, and challenges and assess whether the project was achieving its objectives. The intent was to enable the Ethiopian Government, USAID, and other partners to address topics of MSG management, quality of services, and sustainability and ultimately to decide if the MSG strategy should be scaled up nationally. 25 PEPFAR FY2008 Country Profile Ethiopia http://www.pepfar.gov/press/countries/profiles/116132 htm (accessed 09/29/2009). ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 5 III. EVALUATION OBJECTIVES AND QUESTIONS THE OBJECTIVES The specific objectives of the evaluation are to look at whether the MSG strategy is achieving its objective of empowering HIV-positive mothers and increasing utilization of PMTCT/RH services and behaviors, and assess the viability and comparative advantage of facility-based MSG in rapidly scaling up access to quality HIV/AIDS services. THE EVALUATION QUESTIONS The evaluation was designed to find answers to the following questions: Program Management Has the current MSG structure26 succeeded in providing appropriate and high-quality services as well as effective program management? If so, how? Service Delivery Has the MSG project demonstrated significant measurable success in increasing the availability, quality, and consistency of facility and community-based care and support to PMTCT clients and their families? If so, how? – Are the services in the community and accessible to the target population? – Is the MSG program increasing opportunities for new HIV-positive mothers and babies to access and effectively use antiretroviral therapy (ART)? – Are members of MSG more likely to disclose their status than nonmembers? – How does the MSG program affect the infant feeding choices of members? – Is the MSG program increasing opportunities for new HIV-positive mothers to access family planning? What are the factors that contribute to or hinder progress toward the outcomes, including those linked to MSG program design, management, and partnerships? Is the MSG program a best practice, as defined by UN international standards documents for best practices in service delivery and community mobilization, or is it a promising practice? Supportive Environment Has the MSG project demonstrated significant measurable success in creating a more supportive environment for PMTCT and women’s households through strengthened coordination, networking, and advocacy? If so, how? – How have MSGs contributed to reducing stigma and increasing awareness about HIV and services available to PLWHA? – Have MSGs affected the national MCH/PMTCT policy environment? If so, how? 26 The current MSG structure was taken to mean the IntraHealth MSG structure; in fact, literally, the current structure is the MSH/HCSP MSG structure, which is not the subject of the evaluation. 6 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY – Has the project demonstrated significant improvement in the quality of life of HIV￾positive women? If so, how? – Has the project demonstrated significant improvement in the economic status of HIV￾positive women and their families? If so, how? Capacity-Building and Sustainability Has the IntraHealth MSG program provided capacity-building to local partners and government institutions, including regional health bureaus? If so, how? How has the program strengthened health care services by training health care workers in MSG-related activities? Has the Capacity Project MSG program developed a feasible exit strategy that provides assurance that the partners will be able to continue providing PMTCT services after the project ends? If so, describe the exit strategy. Monitoring and Evaluation Is the monitoring and evaluation (M&E) system functioning properly as designed? In particular, are the existing database, data collection, and data quality systems able to report on MSG indicators? If so, how? Is the current M&E system able to measure progress toward set targets? If so, how? Lessons Learned What are the lessons learned from the implementation of the MSGs so far? What are the implications for future planning and MSG implementation? – Can we use the term MSG ―component‖ instead of ―project‖? – What are the issues and gaps that have significant implications for the sustainability of the MSG component and therefore need to be immediately addressed? – What are the strategies needed in order to further strengthen the efficiency, effectiveness, management, and sustainability of the MSG component? – Identify successful interventions that merit continuation or replication, better practices, and significant products and tools from the MSG program for possible dissemination and replication. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 7 IV. EVALUATION METHODOLOGY THE EVALUATION TEAM The team consists of an international consultant team leader, Dr. Ruth Hope, who is experienced in USAID/PEPFAR evaluations and has solid PMTCT experience and prior Ethiopia experience; a regional consultant PMTCT specialist, Ms. Uvani Bodasing, from South Africa, who has PMTCT clinical research experience; and a local consultant, Dr. Michael Dejene, who provided logistical and administrative support. For the fieldwork, Dr. Hope and Ms. Bodasing were joined by the USAID/E Psychosocial Support Adviser Mr. Tsegaye Tilahun and two local consultant research assistants, Ms. Bizuayehu Ayele Feye and Ms. Dashe Negewo. THE APPROACH TO THE EVALUATION The evaluation team’s approach to the MSG evaluation triangulated quantitative data presented in IntraHealth publications and reports with an independent analysis of the PMTCT and MSG M&E data collected by IntraHealth and with qualitative data collected during the field visits. The team emphasized the importance of having Ethiopian women participate in the evaluation and was grateful to receive permission to hire two local research assistants to facilitate focus group discussions with MSG members and interpret during structured discussions with the MMs and health facility staff. Key informants included senior personnel from the MoH, HAPCO, and the regional health bureaus. The evaluation team briefed the PMTCT technical working group (TWG) about the evaluation; the TWG has representatives from the MoH, PEPFAR agencies, and implementing partners and is chaired by HAPCO. However, it was not possible to include government health personnel on the evaluation team. Because of unavoidable scheduling difficulties (for example, two public holidays in the first week in Ethiopia), the team adopted a flexible approach, taking opportunities to be briefed and hold discussions when they arose and later completing the team planning processes, including the drafting of formal data-gathering tools. As IntraHealth staff were in their last month and had closed out their PMTCT activity, it was not easy for them or the evaluation team to obtain information after the initial briefing. THE METHODOLOGY The evaluation used several methodologies: Review of published documents and reports Analysis of IntraHealth MSG and PMTCT M&E data Interviews Structured discussions Focus group discussions. 8 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY The methodology was designed to allow triangulation of findings from multiple sources whenever these were available. USAID/E selected sites for visits from among the 52 sites IntraHealth had transitioned to MSH/HCSP. No sites transitioned to Pathfinder/IFHP were included. The selection was essentially a sample of convenience to permit as many site visits as possible within the limited time available. LIMITATIONS This evaluation was undertaken without baseline data for the sites before the MSGs were introduced. The 12 sites were selected to maximize the qualitative data collected from as many facilities as possible in only a week of fieldwork. Thus sites were restricted to Addis Ababa (3 sites), and sites on major transportation routes in Oromia (4) and Amhara (5). It was not possible to compare PMTCT sites with and without MSGs. Thus, it is only possible to demonstrate qualitatively the effect that the MSGs had on the lives of HIV-positive mothers and MMs; it is not possible to measure the outcome of the MSG intervention in terms of change in use of PMTCT services or impact on pediatric AIDS. EVALUATION TOOLS The evaluation tools, which be found in Appendix C, were drafted to elicit the information needed to answer the evaluation questions in the Scope of Work (SoW). Written notes were taken during interviews and structured discussions. Most of the structured discussions were conducted in the local language, with the research assistants interpreting. However, focus groups with MSG members were conducted by the research assistants in the local language and tape-recorded. After the site visits, the team typed up the written notes for each and the research assistants translated and typed up in English the focus group transcripts. The team reviewed all the typed notes in Addis and synthesized a full response from all the sites for each question. These synthesized responses were used in the presentation and analysis of the findings. IntraHealth M&E data were provided in electronic .MDB files, the data in which were extracted using ACCESS for export to EXCEL spread sheets, and in .SAV files, the data in which were extracting using SPSS for export to EXCEL. SCHEDULE The team began the document review before arriving in Ethiopia and continued thereafter. During the first week in Ethiopia, the team was briefed separately by staff from USAID, IntraHealth, the MSH/HCSP, and Pathfinder/IFHP. Discussions during the briefings were guided by familiarity with the evaluation questions in the SoW rather than formal discussion guidelines. Also during the first week, the team drafted interview guides and structured discussion guidelines for use with MMs and HC site coordinators and case managers. The team also drafted a structure for focus group discussions with MSG members and interviewed and appointed two research assistants. At the end of the first week, the team conducted key informant interviews in Addis Ababa and site visits to two HCs in Addis. After using the site visit tools (structured discussion guidelines and focus group framework) in Addis, the team adjusted them for use thereafter. In week two, the team conducted another site visit in Addis, key informant interviews in Addis and Bahir Dar, five site visits in Oromia, and four in Amhara. The team was unable to secure appointments at the Oromia Regional Health Bureau, although they did visit the bureau and spoke with the administrative staff. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 9 Week three started with Eid al-Fitr—although officially celebrated on the Sunday, many offices were closed on the Monday. During the week, the team held discussions with the Johns Hopkins University (JHU) staff in Addis about their experience with MSGs and held a team workshop to synthesize the qualitative data from the field visits before analyzing those data and the quantitative data that IntraHealth provided on a compact disk. In week four, the team made a debriefing presentation to USAID and to PEPFAR partners IntraHealth, MSH, Pathfinder and JHU, and then drafted the evaluation report and submitted it to the Mission. 10 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 11 V. FINDINGS INTRODUCTION Since the 52 MSG sites were transferred from IntraHealth to MSH/HCSP between January and July 2009, ideally the evaluation of IntraHealth’s MSG strategy should have been undertaken in December 2008. The timing of the evaluation made it virtually impossible to thoroughly assess IntraHealth’s MSG management structure and efficiency. At the time of the transfer the monthly allowance provided to each MSG for coffee ceremonies was reduced to ETB200.27 The reduction caused much ill-feeling against MSH/HCSP, which is likely to have influenced responses to questions about IntraHealth’s MSG structure and effectiveness. Several informants said that different PEPFAR partners that had introduced MSGs in the light of the favorable reports from South Africa and the early successes of IntraHealth’s MSGs in Ethiopia provided differing stipends and allowances. None of the sites had complained that other projects paid higher allowances, but informants suggested that the TWG should establish a standard set of MSG allowances for all implementing partners INTERVIEWS AND PUBLISHED DOCUMENTS The evaluators were unable to ascertain the number of MSG sites IntraHealth had founded. This may be a result of loss of institutional memory at IntraHealth in its last month, but inconsistencies were found between documents, reports, and the M&E data provided. The evaluation SoW stated that ―MSG was started at 4 sites (3 in Addis and 1 in Oromia) in 2005.‖ This was corroborated by IntraHealth at the evaluation team briefing.28 However, Mothers Support Groups in Ethiopia29 states, ―The MSG program began in 2005 in three sites.‖ IntraHealth informed the evaluation team that by the end of 2008 there were 63 sites.30 This tallies with a statement in the SoW that ―A total of 41 MSG sites have been transitioned over to HCSP project leaving IntraHealth with only 22 MSG sites,‖ although it is not possible to say on what date that was written. Mothers Support Groups in Ethiopia also states, ―[MSG] began in 2005 in three sites… [and] scaled up to 31 additional sites in 2006–2007 for a total of 34 sites….. As of June 2008 the MSG program has scaled up to an additional 50 sites.‖31 This tells us that there were 84 sites as of June 2008. The M&E data provided by IntraHealth included information from 71 MSGs for 2007–08 and 33 MSG sites (a subset of the 71 from the previous year) for 2008–09. IntraHealth staff said that some MSG sites were discontinued because there were not enough positive mothers to form an MSG. 32 They could not be specific about how many mothers form the critical mass for an MSG. 27 Different respondents reported different figures for the allowances. IntraHealth said it had paid ETB250 a month; MMs at Addis Ketema Health Centre said IntraHealth had paid ETB500 but MSH/HCSP reduced that to ETB350; other sites said the monthly amount for coffee ceremonies was reduced to ETB200 28 Briefing by IntraHealth staff ,09/07/2009. 29 Viadro, C., et al. (2008) Mothers' Support Groups in Ethiopia: A Peer Support Model to Address the Needs of Women Living with HIV (Chapel Hill, NC: IntraHealth). 30 Briefing by IntraHealth staff, 09/07/2009. 31 Viadro, C., et al. (2008), Mothers' Support Groups. 32 GHTech requested clarification on the numbers in the SoW and in the briefing by IntraHealth staff; USAID/E provided a spreadsheet , MSG Sites Supported by IH-E: October 1, 2007–September 30, 2008, which indicates that MSGs at 10 sites [6 in Amhara, 3 in Oromia, and 1 in Southern Nations] out of 78 were cancelled after site coordinators and MMs were trained ―due to zero positive mothers‖ 12 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY MSH/HCSP staff reported that by July 2009 they had received 52 MSG sites from IntraHealth: 20 in January 2009, 25 in a second tranche, and 7 in the third tranche. Pathfinder/IFHP staff reported that officially they received 33 MSG sites from IntraHealth but in practice they received only 1633: the other sites are not being actively followed up because the MSGs are not completely in line with Pathfinder’s approach, which is community-based and addresses HIV prevention only. Thus IntraHealth transitioned 85 MSG sites to other USAID partners. The SoW and IntraHealth in the evaluation team briefing describe four MSG interventions (see Table 1). TABLE 1. MSG INTERVENTION STRATEGIES Relationship Who Is Involved Purpose 1. Mothers to mothers-to-be Mentor mothers Mothers-to-be Counseling, peer support 2. Mothers to mothers Mentor mothers Postpartum mothers Counseling, peer support 3. Mothers to community Mentor mothers MSG graduates Community members Education, referrals 4. Mothers creation Members of MSG IGA (self-help saving group is one of its initial steps) However, Mothers Support Groups in Ethiopia describes only the first three. Seeking clarification, the team learned from IntraHealth staff that the fourth intervention, for income￾generating activity (IGA), was added in 2008 after Mothers Support Groups in Ethiopia was published. There is no institutional memory about why it was added. It is not a component of the original M2M program, training manual, or materials, but it is included in the training manuals and materials developed with IntraHealth technical assistance and published by MoH/HAPCO. A former IntraHealth staff member said it was introduced because the costs of attending MSGs was a problem for some mothers.34 IntraHealth reported difficulties in implementing IGA, including (1) lack of potential partners who can support IGA; (2) quotas of most IGA partners limiting the number of mothers who can participate; (3) need for identity cards, which many mothers do not have, to receive a loan; (4) lack of a permanent address for many mothers; and (5) mothers’ fear of debt. Several informants thought that IGA should not be included in the MSG program because IGA issues become the focus of the MSG rather than health promotion and peer psychosocial support. They recommended that the IGA module in the curriculum should be limited to awareness-raising and linkages to organizations such as local HAPCOs and kebeles that have IGA capacity. Several informants remarked that HC staff were initially resistant to training MMs. IntraHealth staff also reported that because the PMTCT site coordinators are MoH staff, they do not receive allowances and resent that the MMs do. Although this is probable, the evaluation team did not receive corroborating information on the site visits. 33 Briefing by Pathfinder/IFHP staff, 09/08/2009. 34 Ms. Eftu Ahmed, Academy for Educational Development/C-Chang,e 09/08/2009. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 13 FROM ANALYSIS OF M&E DATA AND REPORTS IntraHealth provided the evaluation team with data from its M&E database for FY 2008 and FY 2009 through June. The database gave the total number of IntraHealth MSGs during this period as 71, with the largest numbers in Amhara (25) and Oromia (19) (see Figure 1). Figure 2. Number of MSGs by Region, October 2007–June 2009 0 10 20 30 40 50 60 70 80 Addis Ababa Amhara DireDaw a Oromia SNNPR Tigray TOTAL Although the evaluation team constructed a cascade for the experience of mothers testing positive in ANC from October 1, 2007, through June 30, 2009, the data may not be as accurate as Figure 3 might suggest. Discussions with the IntraHealth Chief of Party at the end of the evaluation indicated that, for example, the HIV-positive mothers who joined support groups were not all referred from ANC; a significant number come from other outpatient sources. Similarly, at some sites positive mothers who are taking ART are included in the figure reported for mothers on antiretroviral (ARV) prophylaxis; at other sites they are not. Figure 3. The Experience of Mothers in ANC, October 2007–June 2009 1343 1233 895 577 455 430 0 200 400 600 800 1000 1200 1400 1600 Women in ANC Tested HIV positive HIV+ mothers referred MSG HIV+ mothers joined MSG HIV+ mothers disclosed to partners HIV+ mothers on ARV prophylaxis HIV exposed infants ARV prophylaxis 14 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY The accuracy of the M&E data for the MSGs is also in doubt in relation to Figure 4, which shows 887 mothers as having been referred to ART. However, a sizeable proportion of the referrals are likely to have come in the opposite direction, from ART to the MSG.35 It is also not clear whether the mothers reported as using FP are using condoms for safer sex or other modern methods of contraception. Figure 4. Experience of Mothers’ Support Group Members, October 2007–June 2009 236 567 887 83 644 0 100 200 300 400 500 600 700 800 900 1000 Mothers Exclusively Breast Feeding Mothers Replacement Feeding Mothers Referred to ART Mothers using FP Mothers Referred Care & Support FROM SITE VISITS TABLE 2. SITES VISITED FOR EVALUATION AND RESPONDENTS MET Region Site Mothers Met Mentor Mothers Met Site Staff * Addis Selam - 1 2 Addis Lideta 3 - - Addis Ketema - 3 - Oromia Adama Many* 3 1 Oromia Bishoftu 3 4 1 Oromia Kuyu 2 + 1 father 4 1 Oromia Wolenchitti 3 4 2 Amhara Addis Kidam Many* 4 2 Amhara Bure 8 3 2 Amhara Debramarkos 9 4 1 Amhara Modjo 1 3 1 Amhara Tilili Many** 3 1 * Site staff included PMTCT coordinators and counselors and MSH site coordinators. ** Where there were more than 10 women and many children present, it was not possible to determine how many mothers were referred through ANC/PMTCT services and how many from outpatient ART. In the larger groups women came and went during the discussions, which could not be facilitated as focus groups because of their size. 35 Patricia McLaughlin, 09/29/2009, personal communication. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 15 Impact on Positive Mothers’ Lives The HC site coordinators and case managers, MMs, and MSG members met at all the sites visited were universally enthusiastic about mothers’ support groups. If the impact of the project is taken as the difference MSGs make to positive mothers’ lives, including their use of PMTCT/RH services and behaviors, there was a general consensus that MSGs and the MMs raise awareness of HIV and motivation for HIV counseling and testing (HCT). – Reportedly, mothers are more informed, which has reduced the awareness-raising and health education burden on staff. – A small proportion of MMs also do awareness-raising and motivation for HCT for general outpatients. – Increased uptake of HCT in ANC has been reported They provide much-needed psychological support to ANC attendees who test positive. – Some MMs provide psychological support to other HCT clients [male and female] who test positive. – Reportedly there was been increased uptake of PMTCT services by positive pregnant women. They give continuing psychosocial support and peer counseling to positive mothers, which ―relieves the stress on the mothers‖ and ―allows them to be happy in the group.‖ They offer special advice and counseling for positive mothers in discordant partnerships. They provide information on living positively and guidance on reducing the risk of transmission. – Much of the advice on living positively seems to be from personal experience, but MSG members report that the MMs ―read to them‖ and ―make presentations‖ at coffee ceremonies. They offer information and advice on taking ARVs, including how to manage side effects. – The numbers of persons taking ART have reportedly increased. – There is said to be increased adherence to ARVs. – The burden on staff in the ART clinic has reportedly decreased. They make home visits to persons who have dropped out of PMTCT or ART services at the HC. – The lost-to-follow up rate is said to have decreased. They provide information on infant feeding, particularly the importance of exclusive breast feeding or replacement feeding. – Exclusive breast feeding for 6 months has reportedly increased. 16 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY They offer information on use of condoms by all persons who are living positively. – Reportedly this has increased the use of condoms. – But it is also said to have caused conflict with some male partners. They give information on FP and provide motivation to use it. – Most respondents reported increased FP. Additionally, many HC staff report that MMs are more effective than HC staff at awareness-raising, health education, and counseling about ART because they speak from personal experience in language the mothers can relate to, rather than from theory. Impact of MSGs on Stigma Another impact on positive mothers’ lives that was widely reported was reduction in stigma: Self-stigma was the most reduced, with a concomitant positive living outlook. Stigma in the community was frequently said to be reduced, although examples of residual stigma in relation to housing and employment were cited. At one site, stigma by HC staff was specifically mentioned as having lessened. Impact on Mentor Mothers’ Lives Widely reported impacts on the lives of MMs relate to increased status in the community: Many MMs carry out regular coffee ceremonies and undertake other awareness-raising activities when they do home visits. Several MMs specified that they are respected by community leaders. MMs are seen as important persons by their own friends and neighbors as well as by MSG members. They are respected by HC staff. They are seen as members of the HC staff. MMs widely report much-increased feelings of self-worth, self-confidence, and self-respect. MSG Membership and Male Involvement Overall, MSG members report feeling more relaxed and happy when with the group and intend to remain part of the group. Many MMs similarly report that they will always remain part of the group (even if the stipend ceases or of they move to better paid opportunities). There was no sense in the discussions that mothers would or should graduate from the groups. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 17 Another area where there was near universal concordance in the responses from MMs and discussions with MSG members concerned male involvement in MSGs: All respondents thought that men should be involved. – Except for some mothers at one site who were concerned about embarrassment when use of condoms is taught, all the MMs and MSG members thought that men should be in the same groups as the women. – Several mothers cited conflict at home—one mentioned that this can lead to violence— when women disclose their status to their partners and when they try to educate partners on using condoms. – There was wide concurrence that partners should be called into the HC to be tested together, men and women should learn their results together, and men should then be invited to join the MSG with their wives so that the male partners learn about PMTCT from HC staff and about positive living from the MSG rather than their wives. Environment for PMTCT Clients and Their Households Although the sites visited were in some of the more affluent woredas in Ethiopia, the members of the MSGs were among the poorest of the poor in those areas. The MSG members were far more likely to be widows, abandoned, or divorced than others in their communities, and except for a very small number who had had still births, all had dependent children. Food aid: Food aid from the World Food Programme is available at some sites visited in Oromia at none in Amhara. (Food aid is available in Amhara only in the 40 woredas that are not food￾secure.) Many of the mothers at the sites visited where food aid is not available reported that they do not have food security and need food aid. Income-generating activities: At every site visited, the lack of IGA was an issue. Some MSGs had within the previous two months begun micro-savings clubs, saving about ETB10 per mother per coffee ceremony attended. Others, also recently, had begun spinning cotton for weaving traditional white cloth and making winnowing trays from basketwork. One reason for beginning craftwork was to keep busy rather than just sit and talk at the MSG. All said that they would sell locally; some had been encouraged by HC staff; and those that were producing craftwork thought they had a local market. For most MSGs, the main problem was that they had no seed money or other support to start an IGA. Gardening: At one site in Oromia36 the local PLWHA association had negotiated with the kebele to provide land for MSG members to garden and seeds for planting. They were growing hartil, a local crop similar to cotton, that is used to make traditional fabric. Many of the HCs had land available that was not utilized—at one the grass was very high and could have been used for feeding livestock.37 Challenges for Mentor Mothers The transportation allowance: At every site the MMs complained that the ETB200 transportation allowance was inadequate. At one site two MMs had left specifically because the allowance was not increased, and other MM attrition may have been related to lack of remuneration. At most sites the MMs work full days: two days one week and three days the next. 36 Wolenchiti HC: both the MMs and the MSG focus groups independently reported this. 37 Bure Health center, Amhara. 18 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY At one site the MMS reported working only 2–3 hours per day. At several sites some MMs work in the community on their days off, making home visits, tracing defaulters, and doing awareness￾raising or holding coffee ceremonies. Thus MMs work substantial hours and they consider themselves to be HC staff. Further, some MMs spend more than their allowance on transportation and material support to mothers in dire need. Some MSG group members are jealous of the ETB200 transport allowance the MMs receive and ―argue loudly‖ with the MMs. The MMs state that they manage this by explaining to MSG members all the benefits they get from the MSG—particularly having an uninfected infant—and suggest that the benefits MSG members receive are worth far more than the MM allowance. Homeless and abandoned MSG members: Several MMs reported that they had taken positive mothers into their own homes after the mothers had been abandoned by their families. Two of these emergency placements had resulted in marriage brokering by the MMs or MSG members with positive male relatives. At one HC a man had been ill with tuberculosis for a long time and an MM suggested to him that he be tested. The man was positive, and once he started ART his health greatly improved. He asked to join the MSG because he wanted continuing support and advice from the MM. In the support group he met a mother who had been divorced by her husband when she disclosed her status—the husband declined HCT. The male group member married the divorced woman and took responsibility for her child. They had a child together, and both parents remained in the MSG. Many MSG members are divorced or abandoned with their children and find themselves in extreme poverty. They press the MMs to give them money or food. The MMs refer these mothers to local civil society organizations and PLWHA associations (where they exist) for assistance. ETHIOPIA–EVALUATION OF MOTHERS’ SUPPORT GROUP STRATEGY 19 VI. HOW THE FINDINGS RESPOND TO THE EVALUATION QUESTIONS PROGRAM MANAGEMENT Has the current MSG structure succeeded in providing appropriate and high-quality services as well as effective program management? If so, how? The MSGs visited provide appropriate high-quality care and support services for positive pregnant women and those with young infants. In some situations physical space for meetings may be cramped, but the quality of the activities is not in doubt. IntraHealth program management included staff dedicated to the MSG strategy at the national level and in support of MSGs,38 although the team was later informed that not all MSG sites had IntraHealth regional staff support.39 MSH/HCSP does not have dedicated MSG staff: any supervision and mentoring is provided by PMTCT staff. In contrast JHU has a dedicated team that goes into a new site for start up, including a nurse case manager40 who remains on site until PMTCT service and MSG are established before moving on to set up another site. However, most JHU PMTCT sites are at in referral hospitals rather than HCs and have much larger client flows. Because IntraHealth’s MSGs have been fully transitioned to partners, it is not possible to evaluate the internal management structure except to note that the staff required to introduce a new program, particularly where the concept is new, is generally much higher than for ongoing management of a program. IntraHealth’s model provides for nursing staff (the site coordinator and PMTCT coordinator) to supervise and technically support the MMs. Since the MMs are not from a technical background, and many have only a very basic background education, they certainly need on-site supervision and back- up from clinical staff. They could not, for example, be supervised by case managers, who often have little education themselves and have not received the kind of PMTCT training the MMs have received. There is currently the perception amongst MSH/HCSP staff that the MSGs are a ―project‖ that IntraHealth could support because it was not supporting comprehensive AIDS treatment as MSH is. The evaluation team discussed this with MSH/Amhara regional staff, emphasizing that MSGs are a strategy for improving PMTCT, and thus could help MSH/HCSP to meet its goals. This was clearly a new perspective to them. For established MSGs at sites where site coordinators and MMs have had refresher training and MMs are confident managing the coffee ceremony and other activities, the strategy will not need dedicated technical support from a PEPFAR partner. However, MMs will continue to require support supervision and mentoring, along with the HC PMTCT staff, during regular supervision visits (preferably as a joint activity with the regional MoH/HAPCO) from implementing partner mentors. 38 Briefing IntraHealth, 09/07/2009. 39 Patricia McLaughlin, 09/29/2009, personal communication. The situation is complex, with some site coordinators trained to support MMs without IntraHealth staff follow up. The mixed success of this approach resulted in IntraHealth undertaking further training at some sites. 40 Briefing JHU, 09/22/2009. Note that JHU case managers are nurses and have a very different role from MSH/HCSP case managers. 20 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY SERVICE DELIVERY Has the MSG project demonstrated significant measurable success in increasing the availability, quality, and consistency of facility and community-based care and support to PMTCT clients and their families? If so, how? The MSGs have clearly increased the availability of psychosocial care and support for positive mothers, mainly at HCs, although some MMs also make home visits. Although the IntraHealth description of the mothers’ graduation process states that each mother must participate in home￾based care training before graduating,41 this was not mentioned during site visits. Although IntraHealth’s intent is for women who graduate from the program to become community educators, in practice it is the MMs who are community educators. Current and former members are likely to be assets to their families and households; however, because site visits were short and focused only on HCs, the evaluation was unable to follow up graduated mothers to ascertain if they are assets in the general community. Are the services in the community, and accessible to the target population? Because all the MSGs are at HCs, they are distant from many kebeles where positive mothers live. A large proportion of mothers wanted MSGs in the kebele because of the time and cost of travel to the HCs, although some mothers wanted HC staff to be available to give presentations at MSG meetings—they noted that positive mothers have in any case to travel to the HC to pick up their ARVs. Most MMs want the MSGs to remain at HCs because of the support they receive from the nursing staff—and probably because of the status they have as ―members of staff‖ of HCs. One MM noted that holding the MSG at the HC was excluding mothers from rural areas who could not reach the HC. Mothers in Addis Ababa preferred to have the MSG at HCs both because there are no HPs and because they can hide their MSG membership by claiming to be visiting the HC for clinical services. Some mothers are excluded from the MSGs not by distance but by opportunity cost. In some towns women work as day laborers and their finances are precarious. Because these mothers need their earnings to buy food for their families, they cannot take time away from work unless they are compensated with food or money to buy food. Are members of MSG more likely to disclose their status than nonmembers? Is the MSG program increasing opportunities for new HIV-positive mothers and babies to access and effectively use ART? How does the MSG program affect the infant-feeding choices of members? Is the MSG program increasing opportunities for new HIV-positive mothers to access FP? Most MSG members reported that they had disclosed their status to their partners; anecdotal evidence from HC staff and MMs suggests that a higher proportion of MSG members than nonmembers disclose. Similarly, HC staff and MMs report that MSG membership has increased exclusive breast-feeding and FP, but there is no quantitative evidence to verify the claims. The MMs, sometimes with support from HC nursing staff, teach MSG members about infant feeding—particularly exclusive breast-feeding, since few if any mothers can choose a safe alternative. The MMs also make referrals to ART and FP, give out condoms, and support adherence to ARVs—whether for prophylaxis or ART. Thus it is not surprising that during 41 Viadro C., et al.(2008,) Mothers' Support Groups in Ethiopia; briefing by IntraHealth staff, 09/07/2009. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 21 qualitative data collection discussion HC staff and MMs report increased use of these services by MSG members. However, it was not possible to confirm this from the IntraHealth M&E database. What are the factors that contribute to or hinder progress toward the outcomes, including those linked to MSG program design, management, and partnerships? One design and management factor that promoted progress toward outcomes is that the MMs are supervised and technically supported on a day-to-day basis by HC nursing staff. Factors that impede progress are lack of refresher training, mentoring, and support supervision (especially since the transfer of sites from IntraHealth). Facilitating MSGs only at HCs excludes participation by women living far from the HC and laborers too poor to take a day away from their work. Although the evaluation team was not able to study the Amharic curriculum and thus is not privy to the details, it seems that the IGA component is currently raising expectations that there will be IGA opportunities, including training opportunities and seed money through the MSGs. This may be unwittingly encouraging dependence rather than self-sufficiency. (The evaluation team recognizes that it only visited relatively wealthy areas of the country during a time of agricultural prosperity after several years of adequate rains, and that there are parts of the country that are chronically food- insecure.) Is the MSG program a best practice, as defined by the UN international standards documents for best practices in service delivery and community mobilization, or is it a promising practice? The MSG strategy should probably be described as a promising practice. Although quantitative data are not available to support definition as a good practice, it is very likely to be one: It encourages increased use of PMTCT services and behaviors and of use and adherence to ART, as well as promoting MCH/FP services. That MMs do home visits to new mothers dovetails nicely with the latest thinking on best practice in newborn care and early infant survival, and the length of time during which mothers are MSG members increases the potential for testing infants and early diagnosis of HIV infection, as well as supporting complementary feeding, encouraging immunizations, and family use of basic prevention kits, good domestic hygiene, etc. SUPPORTIVE ENVIRONMENT FOR PMTCT CLIENTS AND THEIR HOUSEHOLDS Has the MSG project demonstrated significant measurable success in creating a more supportive environment for PMTCT and women’s households through strengthened coordination, networking, and advocacy? If so, how? – Has the project demonstrated significant improvement in the quality of life of HIV￾positive women? If so, how? The MMs have created a supportive environment for ANC clients to accept HCT and for positive mothers to join MSGs, receive continuing peer psychosocial support, and learn more about HIV, positive living, ―having a negative baby‖ (knowledge of PMTCT behaviors and services), infant nutrition, and FP. More affluent mothers are underrepresented in MSGs; they presumably think the disadvantage of living openly as positive—at least within the MSG—outweigh the potential benefits of the group. One exception noted was an obviously affluent young mother in Oromia. She was overweight—not nutrition insecure—and dressed in stylish Western clothes with prominent fashion accessories and a watch. For her the benefit of the group was in peer support for the difficulties in her life caused by a discordant relationship with her husband. The MSG is a refuge for many who feel more at ease being with the group everyday than being at home. Many MSG members are widowed, divorced, abandoned or cast out with their children by husbands or families. The simple coffee ceremony and snacks (commonly popcorn) as well as the knowledge-sharing by MMs and peer support from other mothers form a hugely important 22 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY supportive environment. Also, many MMs undertake home visits to sick members, mothers post￾delivery in the community, and mothers who default from the coffee ceremony, ANC/PMTCT, or ART. This also helps make the environment supportive. Helping mothers accept their diagnosis, disclose to their partners, live positively, and adopt good behaviors through use of PMTCT, ART, MCH, and FP services has a similar effect, as does reducing self-stigma, health worker stigma, and stigma in the community. However, evidence for increased uptake of services, including institutional delivery and reduced stigma, is qualitative and not well triangulated with quantitative data because of a lack of baseline data or comparable data from non-MSG sites: this is addressed in the recommendations. Has the project demonstrated significant improvement in the economic status of HIV￾positive women and their families? If so, how? Many MSG members are the poorest of the poor. In parts of Ethiopia where the WFP operates, MSGs links with it so that pregnant positive women in PMTCT and malnourished positive mothers on ART can get a regular household ration of food and OVCs can get an individual food ration. Households with income less than ETB230 a month are eligible for a household ration. The judgment is left to HAPCO.42 In practice mothers at sites visited in Addis Ababa and Oromia present referral letters from the HC to receive the benefit. The MSGs advocate for members to receive the benefit. The WFP does not operate in the woredas visited in Amhara, although mothers there are as much in need as the mothers in Addis and Oromia. This is a source of discontent among MSG members. However, the MSGs are missing an opportunity to advocate for local communities to establish grain banks, or local affluent households to act as benefactors to support the MSG.43 They are also missing opportunities to start local gardens or as a group raise poultry or small livestock like goats. The HCs visited had unused land available, and it is also likely that the local kebeles and municipalities could provide land for gardens if lobbied by the MSGs or negotiated by or on behalf of MSGs. The few small IGAs now in existence should be considered hobbies. They are not adequate to impact the socioeconomic status of the mothers, and MMs and their supervisors cannot be expected to lead IGAs, which need specialized microfinance and business skills expertise. How have MSGs contributed to reducing stigma and increasing awareness about HIV and services available to PLWHA? The MMs have helped reduce self-stigma by being role models for living positively and using their experiences to support other positive mothers, many of whom are now able to disclose their status to their partners. They have helped reduce stigma by health workers by reducing the burden of duties that fall on ANC staff in terms of counseling, providing health education, and encouraging acceptance of PMTCT services and adherence to recommended behaviors. Their effectiveness has won the MMs the respect of HC staff and overcome stigma against positive mothers. By living openly and positively and raising awareness in the community, the MMs have also helped reduce stigma in the community. However, this evaluation cannot be certain that reduced stigma is a result of the MSGs rather than factors external to the MSGs. Some women may have lived openly and positively without the MSG but with the encouragement of local PLWHA associations. However, MMs would not have been undertaking specific awareness￾raising about PMTCT service and behaviors without being MMs and might not have achieved the 42 Dr. Meherete-Sailassie Menbere, HIV/AIDS Team Leader, WFP, interview, 09/08/2009. 43 Both establishing local grain banks and drawing upon benefactors are culturally appropriate activities in Amhara. Advocating for local support to ―MSGs serving the poor‖ may be a better draw for benefactors than specifically raising money for ―MSGs for HIV-positive mothers‖. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 23 same levels of personal self-worth and self-esteem without the boost of being seen as HC staff members and the respect they receive from MSG members. Have MSGs affected the national MCH/PMTCT policy environment? If so, how? The program’s appeal to providers and MSG participants has led HAPCO to include MSG in its national HIV/AIDS response strategy44 and in its recently revised national PMTCT guidelines.45 CAPACITY-BUILDING AND SUSTAINABILITY Has the IntraHealth MSG program provided capacity-building to local partners and government institutions, including regional health bureaus? If so, how? In introducing MSGs into Ethiopia, IntraHealth began a process for increasing capacity to manage MSGs within the government health system. IntraHealth staff—many of whom have now moved on to other organizations, the UN, and PEPFAR and its partners—are a resource for MSGs in Ethiopia. IntraHealth helped create the MoH/HAPCO training curriculum and materials for MSGs, an important contribution to building health system capacity for MSGs, and conducted a training of MSG trainers for the federal HAPCO to create a national corps of MSG trainers. These now need to expand trainer training to the regional level to ensure a corps of trainers that can work closely with regional and HC staff. HC staff oversee MMs, and the MMs require regular in-service refresher training to keep them up-to-date in a field where policies and guidelines change frequently Because of attrition, new HC staff and MMs also need to be trained. The MSG process needs to be institutionalized so that regional MoH/HAPCOs can mentor HC staff on MSG oversight and give regular support supervision to MMs. There is not yet a costed and workable plan for rolling out the MoH/HAPCO policy for MSGs, so the strategy is not yet institutionalized. Further, HAPCO does not pay stipends or allowances for volunteers, and MSGs are unlikely to be viable without remuneration of MMs. Most MMs work at least half time and many full time; rather than being thought of as volunteers, they should be considered lay health workers. Without continued MM allowances or a small stipend spelled out within the MoH budget, the MSGs are not likely to survive. The costs of running an MSG are quite small—replacement of coffee ceremony equipment and recurrent coffee ceremony expenses. IntraHealth advises that start-up equipment costs are about ETB500 per MSG, so annual replacement costs are likely to be similar. While the costs are small per MSG, scaling up the strategy nationwide becomes a significant line item in annual budgets; however, it is likely to be an attractive proposal for a Global Fund grant. Thus there is still a huge need for advocacy for a realistic roll-out plan and national institutionalization of MSGs that spells out the required human and other resources, with a plan for mobilizing the human resources and a source of continuous funding. How has the program strengthened health care services by training health care workers in MSG-related activities? HC site and PMTCT coordinators have been trained in MMs and MSGs. Has the Capacity Project MSG program developed a feasible exit strategy that provides assurance that the partners will be able to continue providing PMTCT services after the project ends? If so, describe the exit strategy. 44 HAPCO (2007), Accelerated Access to HIV/AIDS Prevention . . . . f 45 HAPCO (2007), Guidelines . 24 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY IntraHealth trained partners before transitioning sites and has responded to requests for follow-up training after sites are transferred. This seems to be a reasonable exit strategy, but transferring sites and training transferee staff is far from transfer of ―ownership‖ of the MSG strategy. The issue of ownership of the strategy and sites transferred is beyond IntraHealth’s manageable interest. MONITORING AND EVALUATION Is the M&E system functioning as designed? In particular, are the existing database, data collection, and data quality systems able to report on MSG indicators? If so, how? Is the current M&E system able to measure progress toward set targets? If so, how? IntraHealth introduced PMTCT and MSG registers at its sites as the primary tool for data collection. Data are collected for PEPFAR reporting on PMTCT and for project reporting on MSGs. The denominator for the MSG, new attendees at ANC, is recorded in the PMTCT database, not the MSG database. However, it is not an accurate denominator because an unknown number of MSG members were enrolled through the ART clinic and outpatient clinics other than ANC, 46 and the registers do not identify such persons. The IntraHealth M&E system and databases have been beset by problems in part because of changing policies and protocols (for example, the introduction of combined ARV prophylaxis) and different interpretations of the indicators with differing criteria applied for reporting (for example, the indicator for mothers taking ARV prophylaxis sometimes includes mothers on ART and sometimes does not). Concerns about double-counting led some PMTCT sites to not count mothers on ART or ARV prophylaxis; other sites do not have the same concern. Other systemic database accuracy problems are caused by uneven updating and cleaning of data entered. IntraHealth did not manage the sites long enough for there to be useful data on HIV testing of infants at 18 months (or later if still breast-feeding). Thus, the IntraHealth MSG experience lacks rigorous, quantitative evidence for monitoring progress against targets, and the outcomes and impact of MSGs. Recognizing the issues after a data quality audit (DQA), IntraHealth undertook a study of sites it established and managed between August 2008 and February 2009; this has better data quality but is limited to a small number of sites that met rigorous inclusion criteria. However, IntraHealth support to PMTCT services and MSG services commenced concurrently at sites in the study and seemingly at all other sites since October 2007. (The evaluation team has not seen data from 2005–07 from the Hareg project, although IntraHealth still manages it.) Although the study had tight criteria for inclusion of sites and provides good quantitative evidence for services used and behaviors reported at MSG sites—information not available in the M&E database—the study, like the M&E database, does not permit comparisons with the situation before the MSGs were introduced or with comparable non-MSG PMTCT sites. Reportedly, MSH does have sites that have PMTCT M&E date from before the introduction of MSGs.47 46 Patricia McLaughlin, 09/28/2009, personal communication. The accuracy of the data provided for mothers referred to ART is in doubt because it is thought to include mothers referred from ART to MSGs . 47 Patricia McLaughlin, 09/29/2009, personal communication following debriefing for partners and MoH. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 25 LESSONS LEARNED What are the lessons learned from the implementation of the MSGs so far? Major lessons learned are that 1. More attention to M&E is needed both to accurately monitor the progress of MSGs toward client enrollment and other targets and to measure the outcomes and impact of MSGs on PMTCT uptake and pediatric AIDS. – HC staff and others in the data reporting chain require training and clear guidelines about completing registers and other M&E data tools, with written job aids and reference lists defining categories for inclusion, that specifically address known problems (such as whether women on ART should be included in PMTCT registers as receiving ARVs; and whether women referred to MSGs from ART should be entered into the register as mothers referred to ART). – Staff need regular coaching and mentoring on HC support supervision visits; there should be regular DQAs and refresher training—heightened if the DQA identifies problems. – To demonstrate that MSGs are effective for Ethiopia it is essential that baseline data on PMTCT M&E be collected before MSGs are set up. While there is qualitative evidence that MSGs make a difference in the lives of positive pregnant women and mothers, quantitative evidence of the outcomes and impact is still needed. 2. HC staff acknowledge that MMs can be more effective than they are at awareness- raising, health education, and counseling about ART because the MMs speak from personal experience in language the mothers relate to. 3. HC staff acknowledge that MMs and MSGs reduce the burden on them in terms of – HIV awareness-raising and health education – Pretest counseling – Post-test counseling – Adherence counseling – Defaulter tracking. 4. IGAs introduced through the health sector are unlikely to significantly improve the economic status of HIV-positive women and their families; including an IGA module in the curriculum for MSGs raises expectations and then demands for IGA opportunities and seed money, which distracts from the real purpose of the MSG. It might be better to include a module on skills for life training—including problem solving, gardening, and negotiating with health sector and community leaders—to cultivate an empowered independent approach rather than one that expects external support. What are the implications for future planning and MSG implementation? – Can we use the term MSG “component” instead of “project”? Because the evaluation team considers MSGs to be a strategy for improving PMTCT, they should be considered an integral component of support to PMTCT services. This would seem to be an important distinction from MSGs being a project in situations where, as for IntraHealth, one entity sets up MSGs and then transitions them to another. It is important that the new partner has ownership of the strategy. 26 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY What are the issues and gaps that have significant implications for the sustainability of the MSG component and therefore need to be immediately addressed? What strategies are needed to further strengthen the efficiency, effectiveness, management, and sustainability of the MSG component? For the MSGs that IntraHealth established and transitioned to other partners, the primary issue is ownership. To take ownership and move MSGs toward sustainability, partners need to work with the regional HAPCO to train a cadre of trainers for MSGs, draw up MSG support supervision checklists within PMTCT support supervision, and develop mentoring materials. undertake joint support supervision/mentoring PMTCT site visits that include the MSGs. address skills-training needs so that MSGs cease to first look to the partner or others for handout, but first look to what they can do locally to resolve issues without dependency. identify gaps, such as provision for women who live too far away to attend the MSG at the HC, and for day laborers who cannot afford to take time off work. – However, there are many obvious issues that need to be addressed if MSGs are to be organized at HPs: How will they be managed and supported? HEWs are already over￾extended and for most of their working day are away from the HP working in the community. Where at the HP would MSGs be held? Most are two rooms and one of them is living quarters for an HEW. How many positive women are required to establish a MSG? If prevalence is low and the MSG at the HC is small, there may not be enough positive women at kebele level to maintain a MSG. – Identify successful interventions that merit continuation or replication, better practices, and significant products and tools from the MSG program for possible dissemination and replication. MSGs merit continuation—possibly becoming family support groups where mothers want their male partners to be involved. The MSG in Oromia that is gardening is a very interesting development, one worth replicating by other MSGs. The IntraHealth adaptation of the M2M training manual has already been disseminated by the MoH. ETHIOPIA–EVALUATION OF MOTHERS’ SUPPORT GROUP STRATEGY 27 VII. RECOMMENDATIONS 1. USAID/E and the PEPFAR partners implementing an MSG strategy to support PMTCT should ensure that baseline PMTCT data are collected before introducing an MSG, and should ensure that their M&E systems are robust enough to generate accurate data on the outcomes and impact of MSGs in terms of PMTCT indicators and pediatric AIDS incidence. 2. If, as seems likely from the qualitative evidence, MSG are an effective strategy, advocacy may be needed to MoH/HAPCO concerning the requirements for implementing its MSG strategy. Implementation nationally requires a plan that addresses human and other resources, training, support supervision and coaching of the technical staff managing MMs on a day-to￾day basis and the MMS themselves, and a schedule for the roll-out. Advocacy may also be needed to address remuneration for MMs who under MoH/HAPCO policy are considered volunteers ineligible for a stipend or allowances. Instead, the mentor mothers might be considered lay health workers. 3. If MoH/HAPCO wishes to extend the reach of MSGs to HPs, USAID/E might implement a well-documented pilot program to determine what is necessary for institutionalizing MSGs at HPs. 4. Similarly, USAID/E may want to implement a well-documented pilot program for MSGs at HCs that offers a reduced curriculum and allows mothers to be referred for longer-term psychosocial support to satellite MSGs in their home kebeles. 5. PEPFAR partners implementing MSGs should pool their expertise to address such issues as how many HIV-positive ANC attendees are enough to sustain an HC MSG; and similarly the requirements in terms of numbers of positive mothers for decentralizing MSGs to satellite MSGs at kebeles/HPs. 6. The module of the MSG training manual on IGAs should be reviewed and perhaps changed to cover life skills, such as problem-solving, gardening, negotiating with HC management and local leaders, and perhaps partnering with local organizations that provide income￾generation and business opportunities. Links between MSGs and the urban gardening program should be encouraged. 28 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 29 APPENDIX A. SCOPE OF WORK USAID/Ethiopia President’s Emergency Program for AIDS Relief (PEPFAR) Final Statement of Work (SOW) for End-of-Project Evaluation Mothers Support Group (MSG) Program (GH Tech Revised: 08-10-09) PROJECT IDENTIFICATION DATA 1. Project Title: Positive Change: Mothers Support Group (MSG) 2. Project Number: Cooperative Agreement No. 663-A-00-04-00433-00 3. Project Dates: Sept. 29, 2005 – Sept. 28, 2009 4. Project Funding: $ over 4 years Was not funded independently but as a part of PMTCT 5. Implementing Organization: IntraHealth/Capacity Project, USA I. IDENTIFICATION OF THE TASK The USAID/Ethiopia (USAID/E) PEPFAR office requests technical assistance from three qualified consultants to design and implement an independent external end-of-project evaluation of the Mothers Support Group (MSG) Program. The MSG is a health center-based peer support group for HIV-positive pregnant and postpartum mothers addressing psychosocial and economic needs as well as medical and health behaviors. The evaluation team will include participation from the USAID HIV/AIDS and Health teams. The USAID/E PEPFAR office requests that the evaluation be completed by September 2009 in order that the findings, conclusions, and recommendations can be used in the planned redesign of future programs for Prevention of Mother to Child Transmission of HIV/AIDS (PMTCT). II. BACKGROUND USAID/E Response to HIV/AIDS: From 2004 through 2006, an estimated 288,000 Ethiopians died from HIV/AIDS-related causes. The 2007 federal estimate48 of national HIV prevalence is 2.1%: 7.7% in urban areas and much lower in rural areas at 0.9% (FHAPCO 2007). As of 2007, over 8% of pregnant women in Ethiopia were estimated to be living with HIV (Ethiopia COP 2007); almost one million (977,000) Ethiopians were estimated to be living with HIV; there were almost 900,000 AIDS orphans (898,350) (FHAPCO 2007). The U.S. Mission to Ethiopia’s HIV/AIDS interagency team, composed of the Department of State, the Department of Defense, the U.S. Centers for Disease Control and Prevention, and the U.S. Agency for International Development (USAID), began integrated HIV/AIDS programming in 2004 under the oversight of the Office of the Global AIDS Coordinator. Peace Corps joined the PEPFAR team in early 2007. The U.S. Mission collaborates with a number of Ethiopian government agencies: the HIV/AIDS Prevention and Control Office, the Ministry of Health, the Ministry of Finance and Economic Development, the Ministry of Youth and Sports, the Ministry of Women Affairs, the Ministry of Education, and the Ministry of Labour and Social Affairs. USAID responds to HIV/AIDS as part of PEPFAR in collaboration with the Ethiopian Government and numerous other partners. USAID supports prevention, care, and treatment activities with a combined FY 2006 program budget of over $122 million. 48 Federal HIV/AIDS Prevention and Control Office (FHAPCO), Single Point Estimates of HIV and OVC Indicators, April 5, 2007. 30 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY Prior USAID Assistance to PMTCT in Ethiopia: USAID Mission-assisted Prevention of Mother to Child Transmission of HIV/AIDS has increased significantly, from $1,660,000 in 2005 to $12,053,975 in 2009. USAID Assistance for MSG: In 2005, USAID/E initiated a four-year (2005–09) program with IntraHealth under its PMTCT program to establish the Ethiopia Mothers Support Group (MSG) program to address the special needs of pregnant and postpartum women who are living with HIV/AIDS (PLWHA) and caring for new babies. The Ethiopian MSG program was adapted from a successful program which was introduced in South Africa in 2001 as Mothers 2 Mothers (M2M). M2M came out as a result of the recognition of shortcomings and key gaps in PMTCT programs, often compounded by the special social, education, and economic needs of HIV–positive women and mothers. Some of these challenges include: Reluctance to undergo HIV testing because of stigma Inability to effectively deliver PMTCT-related antiretroviral therapy to pregnant women with HIV Non-exclusive infant feeding causing postpartum transmission of the virus to the baby Mothers failing to access health- and life-sustaining medical/HIV care after delivery for themselves and their babies Lack of understanding about and support for women living with HIV/AIDS Increased number of ―lost-to-follow up‖ cases Distance to the health center In addition to these challenges, Ethiopia has a markedly low antenatal care (ANC) use as most women prefer to deliver their babies in the community setting. The overall goal of the MSG program, therefore, is to reduce mother-to-child transmission by empowering HIV-positive mothers and mothers -to-be to make informed decisions about their reproductive health and the health of their babies. The MSG program has seven broad objectives: 1. To enhance access to and use of PMTCT services by building strong linkages between health care providers and peer support networks. 2. To ensure adherence to antiretroviral therapy among pregnant and postpartum women. 3. To lessen HIV-related stigma and discrimination. 4. To increase HIV- positive mothers’ understanding of infant feeding options. 5. To reduce the incidence of new sexually transmitted infections and HIV among girls and women. 6. To increase acceptance and use of family planning among postpartum women. 7. To build linkages with other programs and services that strengthen women’s health and decision-making (e.g., nutritional support, income-generating activities (IGAs), and skills training). To achieve these objectives the MSG program fosters several mutually reinforcing strategies for peer-to-peer contact. Health center-based peer support groups for mothers-to-be and mothers living with HIV are the core of the MSG program. In the groups, trained ―mentor mothers‖ help ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 31 their HIV-positive peers address their unmet psychological, social, medical, and limited self-help strategies and linkages to IGA to support economic needs. In addition, both the mentor mothers and mothers who graduate from the support groups are encouraged to reach out to their communities to provide prevention education, decrease stigma, and refer pregnant women for ANC and PMTCT services. TABLE 1A. MSG INTERVENTION STRATEGIES Relationship Who Is Involved Purpose 1. Mothers to mothers-to-be Mentor mothers Mothers-to-be Counseling, peer support 2. Mothers to mothers Mentor mothers Postpartum mothers Counseling, peer support 3. Mothers to community Mentor mothers MSG graduates Community members Education, referrals 4. Mothers creation Members of MSG IGA (self-help saving group is on its initial step) The MSG program is designed to reach women from the beginning of pregnancy through the first year of motherhood. Mothers-to-be and mothers may join a mothers’ support group following their first HIV counseling and testing visit at a health center, after any repeat ANC visit, following labor and delivery, or postpartum. Indicators currently reported by MSG include the following non-PEPFAR indicators: Number of MSG-supported sites Number of MSG members Number of new HIV+ pregnant women enrolled in MSG Number of newly enrolled HIV + pregnant women disclosed their status to partners Number of newly enrolled HIV+ pregnant women linked to ART MSG was started at 4 sites (3 in Addis and 1 in Oromia) in 2005. Gradually it was scaled up to an additional 30 sites by February 2008. The program was expanded to an additional 31 sites in the third phase. However, 15 sites were found to have zero HIV prevalence and as such the program was discontinued in those sites. A total of 41 MSG sites have been transitioned over to HCSP project leaving IntraHealth with only 22 MSG sites. As of September 2007, IntraHealth had trained 135 mentor mothers and 60 site coordinators in these sites, enrolled 1,566 mothers in the program, and graduated 234 women from the support groups. Some challenges faced in implementing MSG to facilities include initial resistance from health providers; getting a reasonable number of mothers to initiate and maintain a group; having space at the facilities; linkage to IGA and microfinance activities; addressing mentor mother attrition and site coordinator turnover and stigma. Additionally, distance from health centers is an impediment to frequent participation for women in the more rural areas. Also, misconceptions from MSG members over monetary and nonmonetary support has been a challenge. 32 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY At the policy level, the Ethiopian HIV/AIDS Prevention and Control Office (HAPCO) plans to include MSG in its national HIV/AIDS response strategy (or Road Map). HAPCO also included MSG in its recently revised national PMTCT guideline. The goal of IntraHealth’s MSG program is to offer MSG services at 114 ART health networks and enroll 2,300 HIV+ women in the MSG program. III. PURPOSE OF THE ASSIGNMENT USAID/E requires a team of three consultants to conduct an end-of-project evaluation of the MSG project. With the MSG project having existed for three years now and with IntraHealth ending this 2009, this evaluation team will collect information about MSG implementation, progress, and challenges, specifically looking at whether the MSG project is achieving its objectives. The evaluation team will assess the viability and collaborative advantage of facility￾based MSG in rapidly scaling up access to quality HIV/AIDS services. In addition, the evaluation is expected to help stakeholders understand successful approaches that should be emphasized for future programming. The evaluators will share programmatic, management, and/or financial obstacles and challenges affecting program implementation and recommend any changes in program or management strategies that would increase the efficiency and impact of the program. The recommendations will inform follow-on programs. The evaluation will cover the MSG program performance period of 2005 to 2009. Overall, the evaluation report will help the Ethiopian Government, USAID, and other partners to address topics of management, quality of services, and sustainability around MSG, and ultimately to decide if the MSG strategy should be scaled up nationally. Key Evaluation Questions The evaluation will answer the following illustrative questions: Program Management Has the current MSG structure succeeded in providing appropriate and high-quality services as well as effective program management? If so, how? Service Delivery Has the MSG project demonstrated significant measurable success in increasing the availability, quality, and consistency of facility and community-based care and support to PMTCT clients and their families? If so, how? Are the services in the community and accessible to the target population? Is the MSG program increasing opportunities for new HIV-positive mothers and babies to access and effectively use ART? Are members of MSG more likely to disclose their status than nonmembers? How does the MSG program affect the infant-feeding choices of members? Is the MSG program increasing opportunities for new HIV positive mothers to access family planning? What are the factors that contribute to or hinder progress toward outcomes, including those linked to MSG program design, management, and partnerships? Is the utilization of project funds efficient in relation to the outputs-outcomes observed so far? ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 33 Is the MSG program a best practice, as defined by the UN international standards documents for best practices, in service delivery and community mobilization, or is it a promising practice? Supportive Environment for PMTCT Clients and Their Households Has the MSG project demonstrated significant measurable success in creating a more supportive environment for PMTCT and women’s households through strengthened coordination, networking, and advocacy? If so, how? How have MSGs contributed to reducing stigma and increasing awareness about HIV and services available to PLWHA? Have MSGs affected the national MCH/PMTCT policy environment? If so, how? Has the project demonstrated significant improvement in the quality of life of HIV-positive women? If so, how? Has the project demonstrated significant improvement in the economic status of HIV￾positive women and their families? If so, how? Capacity-Building and Sustainability Has the IntraHealth MSG program provided capacity-building to local partners and the government institutions including the Regional Health Bureaus? If so, how? How has the program strengthened health care services by training health care workers in MSG-related activities? Has the Capacity Project MSG program developed a feasible exit strategy that provides assurance that the partners will be able to continue providing PMTCT services after the project ends? If so, describe the exit strategy. Monitoring and Evaluation Is the monitoring and evaluation (M&E) system functioning properly as designed? In particular, are the existing database, data collection, and data quality systems able to report on MSG indicators? If so, how? Is the current M&E system able to measure progress toward set targets? If so, how? Lessons Learned Can we use the term MSG component instead of project? What are the issues and gaps that have significant implications for the sustainability of the MSG component and therefore need to be immediately addressed? What are the strategies needed in order to further strengthen the efficiency, effectiveness, management, and sustainability of the MSG component? What are the lessons learned from implementation of the MSG component so far? What are the implications for future planning and MSG component implementation? Identify successful interventions that merit continuation or replication, better practices, and significant products and tools from the MSG program for possible dissemination and replication. 34 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY IV. EVALUATION METHODS The evaluation will be carried out by a team of three independent external consultants over a four-week period through multiple methods, including key informant interviews, field observation, and a review of IntraHealth reports, tools, and materials. The evaluation team will develop a valid sampling scheme to identify a small but representative subset of facilities by region. The evaluation team would also need to consider the gains in having comparison sites with no MSG activities to get a better sense of the impact of MSG support in PMTCT sites. Key Informant Interviews Key informants to be interviewed will include the following: USAID Mission staff, including relevant members from the HIV/AIDS Team, Office of Financial Management (OFM), and Program Office Capacity Project, JHPIEGO, and CU/ICAP staff Onsite staff , i.e., mentor mothers, site coordinators Government of Ethiopia representatives: regional HIV/AIDS Prevention and Control Organization (HAPCO), kebele leaders, and Regional Health Bureaus (Ministry of Health), woredas, and health center site coordinators Beneficiaries (both pregnant and nonpregnant HIV-positive women and their families and mentor mothers) Other PEPFAR partners. Team Planning Meeting The full team will have a three-day team planning meeting upon arrival in Ethiopia. The team planning meeting is an essential step in organizing the team’s efforts. During this meeting, the team will produce a work plan, timeline, interview instruments, and a preliminary draft outline of the report. Roles and responsibilities will be agreed upon, and the team will have an initial briefing from USAID. This meeting will allow USAID and the partners to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will clarify team members’ roles and responsibilities review and develop final assessment questions review and finalize the assignment timeline and share with USAID develop data collection methods, instruments, tools, guidelines and analysis review and clarify any logistical and administrative procedures for the assignment establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion develop a preliminary draft outline of the team’s report assign drafting responsibilities for the final report. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 35 Site Visits and Observations The evaluation team is expected to conduct site visits of targeted areas and will work closely with USAID/E and partners to identify key sites. Note: The number of sites to be visited will be discussed/negotiated and finalized with the mission during the TPM. V. INFORMATION SOURCES Consultants will be provided the following background documents in preparation for the assignment: IntraHealth Cooperative Agreement, including modifications IntraHealth PEPFAR Semi-Annual Report submissions IntraHealth 2005, 2006, 2007 and 2008 Annual Reports IntraHealth Quarterly Reports IntraHealth MSG Curriculum IntraHealth M&E Tools USAID trip reports summarizing past field visits to IntraHealth GoE Road Map for HIV/AIDS Prevention, Care, and Treatment PMTCT Guidelines for Ethiopia Draft PMTCT Implementation Manual for Ethiopia Promising Practices in PMTCT in Ethiopia VI. TASKS TO BE ACCOMPLISHED Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task: Review background documents 3 days Travel to Ethiopia (international consultants) 2 days Team planning meeting 3 days Develop evaluation methodology - Develop field visit and interview schedule (consult with CTO) - In-brief USAID/E HIV/AIDS technical staff, in-brief and program overview by IntraHealth/E staff - Conduct interviews and site visits in Addis 2 days Conduct field visits and interviews 7 days Core team synthesis/analysis of findings in Addis; prepare debriefs; draft report; follow up stakeholder interviews in Addis 5 days Conduct preliminary debriefings for USAID and PSP/FMOH/National PMTCT TWG members (separately) 1 day Complete and submit draft report to USAID/E in-country 2 days 36 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task: Travel home (international consultants) 2 days USAID/E and implementing partners review report (14 days) - Finalize report: Team leader and core team members incorporate Mission comments and submit report electronically to Evaluation Coordinator (TL:5/TM:2) 5 days/ 2 days GH Tech edits Final Report (30 days) Total LOE: 32 days of LOE for Team Leader and up to 29 days for other Team Members including travel days. A six-day work week is authorized for work in Ethiopia. VII. TEAM COMPOSITION AND PARTICIPATION USAID/E seeks three consultants: a Team Leader with experience evaluating USAID health programs, a PMTCT Specialist, and a local Evaluation Logistics Assistant. 1. The Team Leader will be an international consultant with extensive PEPFAR program implementation and evaluation experience. S/he will agree to fulfill his/her responsibilities in over four weeks, spending three weeks in-country, and will play a central role in guiding the evaluation process. The consultant will hold conference calls with core team members and USAID/E representatives before and after the visit to Ethiopia, in-brief USAID/E and Government of Ethiopia on arrival, debrief USAID/E, PEPFAR/E, and MSG partners on evaluation findings, and produce a draft report to be left with USAID/E prior to departure, followed by a final report for USAID/E. The Team Leader will Establish assignment roles, responsibilities, and tasks for each team member Ensure with assistance of IntraHealth, that the logistics arrangements in the field are complete – Facilitate the Team Planning Meeting or work with a facilitator to set the agenda and other elements of the TPM – Finalize and negotiate with client the team work plan for the assignment by completion of the team planning meeting – Take the lead on preparing, coordinating team member input, submitting, revising and finalizing the assignment report – Manage the process of report writing – Manage team coordination meetings in the field – Coordinate the workflow and tasks and ensure that team members are working to schedule – Ensure that team field logistics are arranged (e.g., administrative/clerical support is engaged, ensuring that payment is made for services, car/driver hire or other travel and transport is arranged, etc.) ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 37 Consultant qualifications: An advanced degree (MS, MPH or MD) from a reputable accredited institution in any medical or social sciences field pertinent to work with PMTCT/MCH. Minimum 10 years of progressively responsible experience with recognized organization(s) in the design, implementation and evaluation of PMTCT/MCH programs with demonstrated technical expertise and skills in HIV/AIDS. Demonstrated strong analytical, managerial and writing skills. Exceptional leadership in coordinating, assigning the team with the appropriate responsibilities, communication, and interpersonal skills is absolutely critical. Ability to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. Must be able to produce a succinct quality document that gives direction and facilitates improvement for the MSG PMTCT program 2. The PMTCT Specialist will be a local or, preferably, regional consultant with extensive PMTCT implementation and evaluation experience in Africa. Knowledge of HIV/AIDS programming and PEPFAR is essential. The consultant will be responsible for writing some sections of the report. The consultant will assist the team leader in the development of any qualitative instruments to be used during site visits as well as the analysis of any data collected. Consultant qualifications: MD, MS, or MPH from a reputable accredited institution in any of the social sciences pertinent to working with PMTCT/MCH. Minimum 6 years of progressively responsible experience with recognized organization(s) in the design, implementation and evaluation of PMTCT programs with demonstrated technical expertise and skills in HIV/AIDS in Sub-Saharan African countries. Strong knowledge on Monitoring and Evaluation also desirable. Demonstration of strong analytical, managerial and writing skills. Able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. Must be fluent in English. Proven ability to communicate clearly, concisely and effectively both orally and in writing. 3. The Evaluation Logistics Assistant will be a local consultant, preferably fluent in Amharic, with a demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision; ability to work collaboratively with a range of professional counterparts at all levels. 38 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY The Evaluation Logistics Assistant will be responsible for logistics, coordination and administrative support, and ensuring all aspects of the evaluation are carried out seamlessly. He/She will assist the Team Leader and the implementing agencies in facilitating meetings, coordinating logistics and organizing site visits. The Evaluation Logistics Assistant will collect and disseminate background documentation to the evaluation team. Consultant qualifications: MA, MS, MD or BA. Four years of work experience may be substituted for the degree. Minimum 6 years of progressively responsible experience within GOE and/or NGO work settings handling complex logistics, such as coordinating business travel and meetings. Demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision. Ability to work collaboratively with a range of professional counterparts at all levels, including those from host country governmental and non-governmental organization, U.S. Government agencies and other donors. Capacity for effective time management and flexibility. Must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. Must be fluent in English and preferably Amharic. Proven ability to communicate clearly, concisely and effectively both orally and in writing. One USAID staff and one or more GOE representatives may join the evaluation team during the team planning meetings, site visits, debriefings, and report preparation. MSG partner will accompany the team on site visits as appropriate, but will not be present during interviews with the local partners, stakeholders, or beneficiaries. The two USAID staff who will participate alternatively are Chinyere Omeogu PMTCT/MSH Advisor and Yoseph Woldegebriel PMTCT Specialist. Their role will be to compliment the work of the GH Tech consultants. VIII. SCHEDULE AND LOGISTICS The in-country phase of the evaluation will be conducted over a period of up to 22 days with a desired start date on or about 7 September 2009. The Evaluation Logistics Assistant, in collaboration with the USAID/E Evaluation Coordinator and IntraHealth, will arrange all of the partner meetings, site visits and debriefings in advance. Meeting space will be provided at USAID/E, but the agency cannot provide access to fax and email. All associated travel and per diem costs for non-USAID staff will be covered by identified mechanism under the technical directive of USAID/E. IX. PERIOD OF PERFORMANCE Work is to be carried out over a period of approximately twelve weeks, beginning on or about September 2009 and concluding o/a November 2009. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 39 X. FINANCIAL PLAN A budget plan agreement between the USAID/E and GH Tech will be reached and USAID/E will process a MAARD to transfer funding for the evaluation. XI. DELIVERABLES The team will complete the following deliverables: Evaluation Methodology, Field Visit Schedule, and Interview Schedule. In conjunction with the team members, the team leader will develop and submit an evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO and USAID/E Evaluation Coordinator before initiation of the key informant interviews and site visits. Debriefings and Draft Report Prior to departure: Team makes debriefing presentations to USG PEPFAR staff and to IntraHealth and other PMTCT partners, and Team Leader submits a draft report, in the exact format specified by the USAID/E Evaluation coordinator (See separate MS Word file for GH Tech Evaluation Report Guidelines), to USAID/E Evaluation Coordinator - two hard copies and one electronic copy on CD ROM or flash drive prior to team departure from country. The mission will provide one set of written comments to GH Tech within 14 days after draft report submission. Final Report After departure: Team leader submits final unedited report to USAID/E within one week of receiving comments from USAID/E. The report (not including annexes) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, and Recommendations in English in the exact format specified by the USAID/E Evaluation Coordinator. Upon final approval of the content by USAID/E, GH Tech will have the report edited and formatted. This process takes approximately 3-4 weeks. The final report will be submitted electronically to USAID/E Evaluation Coordinator and Contracting Officer in addition to XX printed copies of the report. It will be posted to the USAID Development Experience Clearinghouse (USAID/DEC). 40 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 41 APPENDIX B. PERSONS CONTACTED ETHIOPIA Amhara Regional Health Bureau Tilahua Yemaldu, Deputy Head RHB Megash Tesfu, Care & Support Officer [was previously PMTCT Officer] C-Change Ms Eftu Ahmed, ANC/PMTCT Communications Manager [Previously IntraHealth] Centers for Disease Control & Prevention/Ethiopia Dr Abdulhamid Isehak, Technical Officer PMTCT Federal Ministry of Health Dr Kesete, Director General Ms Mirhet Hilu IntraHealth Ms Patricia McLaughlin, Chief of Party Hanna Tessemz, MSG Team Leader Melaku Gebissa, Project Officer Mestamet Bezu, MSG Officer Aneleye Dessie, Project Officer (Amhara) Dr Lemma Kerewa, PMTCT Advisor Dr Tadesse Ayamew, Regional Manager Bezu Beslir, Project Officer Kidest Belete, Project Manager Johns Hopkins University Dr Solomon Zewdu, Country Director Dr Meg Doherty, Clinical Director, JHU-TSEHAI Ms Eyrusalem Akele, MSG Coordinator Dr Senait G/Yesus Dr Daniel Kinde Management Sciences for Health/Ethiopia Bud Crandall, Chief of Party Haile Wubneh, Deputy Chief of Party Gebremedhin K Mariam, Team Leader Prevention Dr Belkis Wolde Giorgis, NGO Capacity and Gender Advisor 42 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY Muluken Melese Program Director Treatment Care and Support Teferi Fitsior Asamimew Girma Lemma Ketema Kassa Tiraueh, Amhara Regional Health Advisor, HCSP Dr Sale Workneh, Amhara Regional Pediatric/PMTCT Advisor, HCSP Dr Yewuwen Kassie, Amhara Regional Integration Advisor Pathfinder International/Ethiopia Tilahun Giday, Chief of party Peter Eerens, Deputy Chief of Party Dr Wassie Lingerh, Maternal and Newborn Health Adviser Metiku Woldegiorgis, Operations Manager UNICEF Ethiopia Dr Lu Wei Pearson, Chief Health Section Dr Fikir Melese, Project Officer, PMTCT/Pediatric HIV Urban Gardens Program Ms Nancy Russell, Chief of Party USAID/Ethiopia Meri Sinnitt, Office Chief, Health AIDS Population Nutrition Officer Dr Chinyere Omeogu, PMTCT/MCH Advisor Ms Sophia Brewer, Evaluation Coordinator for Health, AIDS, Population, and Nutrition Ms Jean Rideout, Team Leader Health Dr Peter Gichangi, Snr HIV/AIDS Family & Community Service Health Officer World Food Programme Dr Meherete-Sailassie Menbere, HIV/AIDS Team Leader World Health Organization Dr Seblewongel Abate, National Professional Office, HIV/AIDS Dr Akram Atnafu [Previously IntraHealth] ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 43 APPENDIX C. EVALUATION TOOLS QUESTIONS FOR KEY INFORMANT INTERVIEWS 1. Program Management 1.1 Has the INTRAH MSG structure succeeded in providing appropriate and high quality services? If so, how? 1.2 [If appropriate to ask the key informant] The HCSP MSG structure differs from INTRAH’s MSG structure: how will this impact i. quality of services? ii. program management? iii. sustainability of MSGs? 2. Service Delivery 2.1 Has the MSG approach increased the availability, quality and consistency of facility and community-based care and support to PMTCT clients and their families? If so, how? 2.2 The INTRAH MSGs are at Health Centres: are they accessible to the target population? probe: Would it be better to facilitate MSGs at health posts? Why? 2.3 Are MSGs increasing opportunities for new HIV positive mothers and babies to access and effectively use ART? How? 2.4 Are members of an MSG more likely to disclose their status than non members? Why? 2.5 How do MSGs affect the infant feeding choices of members? 2.6 Do MSGs increase opportunities for new HIV positive mothers to access family planning? How? 3. Supportive Environment for PMTCT clients and their households 3.1 How are partners /husbands involved in the MSGs? In PMTCT? 3.2 How have MSGs contributed to reducing stigma? 3.3 How have MSGs contributed to increasing awareness about HIV and services available to PWHA? 3.4 Have the INTRAH MSGs demonstrated significant improvement in the quality of life of HIV positive women? If so, how? 3.5 Has the INTRAH MSG approach demonstrated significant improvement in the economic status of HIV positive women and their families? If so, how? 44 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 4. Capacity Building and Sustainability 4.1 Has INTRAH MSG approach provided capacity building to local partners and the government institutions including the Regional Health Bureaus? If so, how? 4.2 Has the MSG approached strengthened health care services? How? 4.3 Did INTRAH develop an exit strategy that has assured continuance of MSGs and PMTCT services with other partner support now the Capacity Project has ended? 5. Monitoring and Evaluation For Key informants with knowledge of the M&E systems 5.1 Did the INTRAH monitoring and evaluation (M&E) system function properly as designed? In particular, were the existing database, data collection, and data quality systems able to report on MSG indicators? If so, how? 5.2 Is the MSH M&E system able to measure progress toward set MSG targets? If so, how? 6. Lessons Learned 6.1 What else is needed to further strengthen the efficiency, effectiveness, management and sustainability of the MSG approach? 6.2 What are the lessons learnt from the implementation of the MSG strategy so far? 6.3 In summary, what are the successful aspects of the INTRAH MSG approach that merit continuation or replication elsewhere in Ethiopia? QUESTIONS FOR HCSP CASE MANAGERS/INTRAH SITE COORDINATORS 1. Introductions 2. For how long have you been a case manager/site coordinator? 3. How did you become a case manage/site coordinator? 4. What training have you had? Did it include training about MSGs? 5. What do you do as a case manager/site coordinator? Prompt: How does your work relate to the MSG? 6. Do you have any problems or challenges supporting the MSG? Please tell us about them. ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 45 7. What difference do MSGs make to HIV+ mothers and their infants? 8. Has the MSG enhanced access to and use of PMTCT services? If so, How? 9. Has the MSG enhanced adherence to ART among pregnant and post natal mothers? 10. What services are available nearby here for HIV+ mothers and their infants? Prompt: ART services? VCT for family members? Nutritional support? Family Planning? Economic support? / IGA?; Skills training? Psychosocial support? Home-based care? Mekdim? or Dawn of Hope? Or other association of PLHA? Other services? 11. How has the MSG fostered linkages with health care providers at the health center? With community service providers? 12. Has the MSG had any impact on HIV/AIDS stigma? How? 13. Has the MSG increased positive mothers’ knowledge of infant feeding options? Has the MSG changed infant feeding practices? How? 14. Has the MSG had any impact on the incidence of new STIs? What impact? How? 15. How has the MSG impacted uptake of FP services by positive mothers? 46 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY FOCAL GROUP DISCUSSION PLAN WITH MOTHERS GROUP MEMBERS 1. Introductions: everyone tell their name Ruth/Uvani/Tsegaye give explanation for holding discussion: o Wish to learn about the women’s experience of being a member of a mothers’ support group to improve the design of future mothers’ groups elsewhere in Ethiopia Following the introductions and translation of the evaluator’s statement, the FGD will be conducted entirely in the local language by the women consultant research assistants 2. How long have you been members of the group? 3. How did you learn about the group? How did you become members of the group? Why did you join the group? Probe for feelings of the HIV+ mothers at the time and how they felt about the MSG 4. Tell us what do you do in the group meetings Probe for details of how the meetings are conducted, who does what, the feelings of the participants about the meetings ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 47 5. Tell us about what have you learned from the group meetings Probe about the topics, new knowledge and skills 6. Are there any other benefits from group membership? 7. Have you recommended the group to women relations or friends? Will you recommend the group? 8. What do you think about involving male partners/husbands in the group? Why? Probe whether male partners/husbands should join the group or have their own fathers’ support group 9. What do you think about holding MSG meetings at the Health Post (at Kebele level) rather than at the Health Centre (Woreda level) 48 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY 49 APPENDIX D. REFERENCES Baek C et al. Key findings from an evaluation of the mothers2mothers program in KwaZulu￾Natal, South Africa, Horizons Final Report. Washington, DC: Population Council. (2007) Central Statistical Agency [Ethiopia] and ORC Macro. Ethiopia Demographic and Health Survey 2005. Addis Ababa, Ethiopia and Calverton, Maryland, USA: Central Statistical Agency and ORC Macro. (2006) CIA World Factbook—Ethiopia, Jul 2009 est. https://www.cia.gov/library/publications/the￾world-factbook/geos/et.html Accessed 9/28/2009 Davis, N et al. (2009) Ethiopia Prevention of Mother to Child HIV Transmission (PMTCT) Portfolio Review and Recommendations. Washington, DC: PEPFAR PMTCT and Pediatric HIV Technical Working Group. HAPCO (2007) Accelerated Access To HIV/AIDS Prevention, Care And Treatment In Ethiopia. Addis Ababa: Federal Ministry of Health HAPCO (2007) Guidelines For Prevention of Mother-to-Child Transmission of HIV In Ethiopia. Addis Ababa: Federal Ministry of Health HAPCO (2007) Multisectoral Plan of Action for Universal Access to HIV Prevention, Treatment, Care and Support in Ethiopia 2007 – 2010. Addis Ababa: Federal Ministry of Health PEPFAR FY2008 Country Profile Ethiopia http://www.pepfar.gov/press/countries/profiles/116132.htm accessed 09/29/2009 Viadro C et al (2008) Mothers' Support Groups in Ethiopia: A Peer Support Model to Address the Needs of Women Living with HIV. Chapel Hill, NC: IntraHealth 50 ETHIOPIA–EVALUATION OF THE MOTHERS’ SUPPORT GROUP STRATEGY For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com