ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) END-OF-PROJECT EVALUATION July 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by DeeDee Yates, Renee de Marco, Ayansa Gonfa Debela, and Chalachew Tuiruneh through the Global Health Technical Assistance Project. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) END-OF-PROJECT EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 08-001-100) is available online. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) i ACKNOWLEDGEMENTS The evaluation team would like to express its appreciation to all the Positive Change partners who helped facilitate and participated in the end-of-project review. Special thanks to Save the Children USA for flexibility and assistance in making the site visits possible through both extensive consultations about scheduling and providing individuals to give back-up support in the field. USAID/Ethiopia Mission staff were extremely helpful during both the advance preparations and the in-country briefing, providing context and the bigger picture. We would like to thank the logistics officer, Kuleni Berhanu, for her accommodating spirit, and our government counterparts for sharing their insights, experience, and good humor with us. To the dozens of PC3 partners, staff, and volunteers in the community who willingly gave up substantial amounts of their time to meet with us, sometimes on short notice, to answer our questions, and to inspire us with their commitment to and enthusiasm in addressing the needs of vulnerable children in their communities, we give our special gratitude, admiration, and thanks. DeeDee Yates Team Leader Renee DeMarco OVC Specialist ii ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) iii ACRONYMS AIDS Acquired immune deficiency syndrome APS Annual Program Statement BOLSA Bureau of Labor and Social Affairs CBOs Community-based organizations CSO Civil society organization CSSG Community self-help savings group CTC Community therapeutic care DHS Demographic and Health Survey EDHS Ethiopia Demographic and Health Survey FBO Faith-based organization FHI Family Health International GH Tech Global Health Technical Assistance Project GFATM Global Funds for AIDS, TB, and Malaria GOE Government of Ethiopia HACI Hope for African Children Initiative HAPCO HIV/AIDS Prevention and Control Office HCP Health Communication Partnership HIV/AIDS Human immunodeficiency virus/acquired immune deficiency syndrome ID Identification IR Intermediate result M&E Monitoring and evaluation MDG Millennium Development Goals MOE Ministry of Education MOH Ministry of Health MOJ Ministry of Justice MOLSA Ministry of Labor and Social Affairs MOWA Ministry of Women’s Affairs NGO Nongovernmental organization OSSA Organization for Social Service to AIDS OVC Orphans and other vulnerable children PC 3 Positive Change: Children, Care, and Communities PEPFAR President’s Emergency Plan for AIDS Relief PLWA People living with HIV/AIDS RAAAP Rapid assessment, analysis, and action planning SCUSA Save the Children United States of America SDPRSP Social Development and Poverty Reduction Sectoral Programme SNNPR Southern Nations, Nationalities and People’s Region SO Strategic objective SWOT Strengths/weaknesses/opportunities/threats UNAIDS United Nations Programme on HIV/AIDS iv ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) UNDAF United Nations Development Assistance Framework UNICEF United Nations Children’s Fund UNGASS United Nations General Assembly Special Session USAID United States Agency for International Development WFP World Food Programme WV World Vision ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) v CONTENTS ACKNOWLEDGEMENTS .................................................................................................................. i ACRONYMS .................................................................................................................................... iii EXECUTIVE SUMMARY ................................................................................................................ vii Program Design ....................................................................................................................... vii Monitoring and Evaluation ........................................................................................................ix Summary of Major Accomplishments .......................................................................................ix Major Challenges ....................................................................................................................... x Lessons Learned ........................................................................................................................ x Short-Term Recommendations ................................................................................................. xi I. BACKGROUND ........................................................................................................................ 1 Ethiopian Context ...................................................................................................................... 1 Rationale for Evaluation ............................................................................................................ 2 II. METHODOLOGY ...................................................................................................................... 3 List of Tools (see Appendix E) .................................................................................................. 4 III. FINDINGS ................................................................................................................................. 7 IR 1: Increased Availability, Quality, and Consistency of Community-Based Support Services for OVC and Families Affected by HIV/AIDS ............................................................. 7 IR 2: Improved Capacity of Ethiopian CSOs to Plan, Implement, Monitor and Evaluate, Manage, and Report on OVC Programs and Services ........................................................... 12 IR 3: More Supportive Environment for OVC and Their Households Developed through Strengthened Coordination, Networking, and Advocacy ........................................................ 17 Monitoring and Evaluation ...................................................................................................... 18 Program Management and Design ......................................................................................... 20 IV. CONCLUSIONS AND RECOMMENDATIONS ...................................................................... 25 Conclusions ............................................................................................................................. 25 Short-Term Recommendations ............................................................................................... 25 Recommendations for the Transitioning of PC3 ..................................................................... 26 Recommendations for Future Project Design ......................................................................... 26 APPENDICES APPENDIX A: SCOPE OF WORK ................................................................................................ 29 Project Identification Data ....................................................................................................... 29 I. Identification of the Task ...................................................................................................... 29 II. Background ......................................................................................................................... 29 III. Purpose of the Assignment ................................................................................................ 34 IV. Evaluation Methods ........................................................................................................... 35 V. Information Sources ............................................................................................................ 36 VI. Tasks to be Accomplished ................................................................................................. 36 VII. Team Composition and Participation ................................................................................ 37 VIII. Schedule and Logistics .................................................................................................... 39 IX. Period of Performance ....................................................................................................... 40 X. Financial Plan ..................................................................................................................... 40 XI. Deliverables ....................................................................................................................... 40 vi ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) APPENDIX B: PEOPLE INTERVIEWED ...................................................................................... 43 Children’s and Guardians Focus Groups................................................................................ 44 Tier II Partners ........................................................................................................................ 45 Tier III Partners ....................................................................................................................... 46 APPENDIX C: SCHEDULE ........................................................................................................... 51 Schedule of Site Visits for Two Teams ................................................................................... 52 APPENDIX D: TEAM SUMMARY ................................................................................................. 53 IR 1: Services .......................................................................................................................... 53 IR 2: Capacity of CSOs ........................................................................................................... 55 IR 3: Networking, Coordination, Advocacy ............................................................................. 56 Lessons Learned ..................................................................................................................... 59 Responses from Tier III Group Meetings ................................................................................ 61 Summary of Beneficiary Focus Groups .................................................................................. 62 APPENDIX E: TOOLS ................................................................................................................... 63 PC3 Evaluation Tier I Partner Meeting ................................................................................... 63 Evaluation of Positive Change: Children, Care, and Communities (PC3) .............................. 66 PCS Evaluation Form .............................................................................................................. 70 PC3 Evaluation ....................................................................................................................... 75 PC3 Evaluation ....................................................................................................................... 76 APPENDIX F: REFERENCES ....................................................................................................... 77 Agreements, Reports, and M&E Tools ................................................................................... 77 Resource Material Developed by the Program ....................................................................... 77 Other Documents .................................................................................................................... 78 FIGURES AND TABLES Most Important Contributions from Tier I According To Eleven Tier II’s ........................................ viii Summary of Site Visits and Key Informant Interviews Only ............................................................ 4 Number of OVC Served by Core Program Area and Age ............................................................... 8 Most Important Contributions from Tier I According to Eleven Tier IIs .......................................... 14 Perceived Gaps in PC3 ................................................................................................................. 22 President's Emergency Plan: Ethiopia Targets for Care and Support, 2004-2008 ....................... 32 Framework for PC3 Agreement and Subagreements ................................................................... 33 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) vii EXECUTIVE SUMMARY The umbrella Cooperative Agreement known as Positive Change: Children, Care, and Communities (PC3) was awarded in September 2004 to Save the Children/USA as prime recipient, with CARE International, Family Health International (FHI), Hope for African Children Initiative (HACI), World Learning International, and World Vision (WV) as partners. The goal of PC3 is to improve the well-being of 500,000 orphans and other vulnerable children (OVC) and families affected by HIV/AIDS. The PC3 team addresses the needs of OVC affected and infected by HIV/AIDS in seven regions of Ethiopia: Amhara; the Southern Nations, Nationalities, and People’s Region (SNNPR); Oromia; Addis Ababa; Afar; Dire Dawa; and Beneshangul. The original project agreement required that 75 percent of the total funding pass to community￾based organizations (CBOs) to ensure their capacity building. However, this was later reduced to 65 percent in recognition of the capacity-building needs of both local nongovernmental organizations (NGOS) and CBOs. The project takes a tiered approach: Tier I international partners (Save, CARE, World Learning, FHI, and WV) provide technical assistance to 35 Tier II national and local NGO partners, which in turn subgrant to 560 local implementing partners, women’s associations, and CBOs (Tier III partners). Service components that PC3 implements to support OVC and their families through community-based mechanisms include educational support, life skills, health and nutrition, psychosocial support, livelihood support, protection, and legal support. The project had three intermediate objectives, which are discussed below. USAID/Ethiopia through the Global Health Technical Assistance Project commissioned an end￾of-project evaluation of the PC3 project. With less than two years remaining in the project, the evaluation, which took place May 5–23, 2008, was tasked with collecting information about PC3 implementation, progress, and challenges and formulating recommendations for follow-on programs. The evaluation report will help USAID and Save the Children address topics of management, quality of services, and sustainability of the three intermediate results (IRs). The evaluation team consisted of one independent consultant, one OVC specialist from the USAID Africa Bureau, and two professionals from the Government of Ethiopia (GOE). Guided by a comprehensive Scope of Work (Appendix A), the team interviewed key informants from all five Tier Is, the Ministry of Women’s Affairs (MOWA), UNICEF, and the World Food Programme; made field visits to Tier II and Tier III partners at 10 sites, meeting 360 individuals; carried out seven guided discussion sessions with beneficiaries, both children and caregivers; and reviewed all key project documents (Appendices B and C). Based on the team’s findings and reflections, review with stakeholders, and the documentation, the following observations are highlighted. PROGRAM DESIGN The PC3 program was ambitious. It worked throughout Ethiopia with 560 different local community organizations, of which 239 were schools. This was possible through capacity development of 35 local NGOs. To reach such a large number of local CBOs, the program was designed to build on existing partnerships between international and local NGOs. The tiered approach enabled local partners to benefit from the expertise and specializations of all Tier I partners while at the same time enjoying more immediate interaction with local NGO partners. This approach required a longer lead time than anticipated, which meant that the project was pressured in terms of reaching its targets. The design required considerable consistency and homogeneity in approach. Perhaps the most outstanding management and design feature of PC3 is that it achieved the most extensively united OVC partner network in any African country. More than 500 entities have been engaged in continuing commitment to responding to the needs of OVC. viii ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Intermediate Result 1: Increased availability, quality, and consistency of community-based support services for OVC and families affected by HIV/AIDS. PC3 has had a positive impact on the lives of OVC in Ethiopia. Services offered by community organizations, with support from PC3, have allowed children to continue to attend school and to enter informal schools, early childhood development programs, and vocational training institutions. Families have been helped to participate in community self-help savings groups (CSSGs) and to start small income-generating activities. Children in families affected by HIV and AIDS have received visits from volunteers trained in psychosocial support, such as bereavement counseling and life skills. Children and families have been linked to assistance offered by the World Food Programme and others. The capacity of local organizations to respond directly to the needs of OVCs and their caregivers and to refer these children and families to other services has been limited by the high numbers targeted and the funding available for material assistance. Although CBOs were expected and helped to mobilize other resources, this took time and could not fully cover the varied requirements of some particularly needy families—the identified needs were greater than the available resources. Intermediate Result 2: Improved capacity of Ethiopian civil society organizations (CSOs) to plan, implement, monitor and evaluate, manage, and report on OVC programs and services. Local NGOs identified areas where they had received assistance from PC3 through a Tier I partner. Analysis of the questionnaires of 11 Tier IIs shows that organizational capacity development is valued more highly than financial assistance; and that helping organizations to diversify their funding base is valued as much as actual funding itself. TABLE 1. MOST IMPORTANT CONTRIBUTIONS FROM TIER I ACCORDING TO ELEVEN TIER II PARTNERS Most Important Contribution from Tier I Frequency of Response Organizational capacity building 8 Financial assistance 7 Assistance in diversifying funding 7 Training in monitoring and reporting 6 Coaching 6 Training for staff 6 Approaches to community development 5 Exposure 3 Training in financial record-keeping 3 Assistance in assessing gaps in program 3 In turn, Tier II partners were responsive to problems of their Tier III partners, offering training, coaching, and subgrants to Tier III partners. Tier III partners deliver services through volunteers to children identified as in greatest need. Capacity-building activities and direct subgrants to Tier IIs are generally about 70 percent of the total for each Tier I. In turn, Tier IIs subgrant to Tier IIIs. One sample exercise undertaken by a Tier I partner found that a Tier II used 49 percent of its grant to make subgrants to Tier IIIs and 36 percent for capacity-building activities. Many Tier I partners supplement administration costs from their own funds. Tier III partners have been successful in raising local funds, both financial and in-kind donations. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) ix IR 3: More supportive environment for OVC and their households developed through strengthened coordination, networking, and advocacy PC3 participates in networks at all levels. At the national level, the Tier I partners form an impressive network of agencies working in support of children. In addition, two of the PC3 Tier I partners are members of the National Task Force on OVC housed within the MOWA. PC3 Tier III partners have created their own networks through the PC3 iddir council, a partners’ network, and similar organizations. These are effective, but they need to be linked to or merged with other networks. Splintered responsibility for OVC among national government entities has impeded concerted efforts at all levels of policy and implementation to improve child wellbeing. PC3 has not maximized its potential to influence structures at the national or regional levels. At the woreda and kebele levels, Tier IIs and Tier IIIs are able to participate with increased confidence and presence in such networks as HIV/AIDS Prevention and Control Office (HAPCO) Coordinating Committees and the OVC Forums. This will enhance decentralization—strong CSOs can participate more fully in the government’s plans for OVC and in addressing HIV and AIDS. MONITORING AND EVALUATION PC3 uses a variety of monitoring approaches, including volunteer recording sheets, a database with unique identification (ID) numbers for each child, best practice and most significant change stories, joint monitoring visits, and regular review meetings at different levels. Project implementation is well documented, and data are available at every level, down to specific services received by a child or family. The changes in President’s Emergency Plan for AIDS Relief (PEPFAR) guidance and monitoring requirements meant that much energy and effort was expended on reformulating the monitoring system at the community level to reflect primary and supplementary support and address undercounting problems. The problems were addressed by giving each child a unique ID number in the database. Changes in monitoring formats that may seem straightforward to Tier I partners, USAID, and PEPFAR translate into a whole array of training and communication issues when they reach the Tier III level. This is especially true for a program like PC3 that relies, as many programs do, on volunteers to collect the data. SUMMARY OF MAJOR ACCOMPLISHMENTS PC3 local partners have reached 398,000 children over the past four years. Support has included early education, education assistance and tutoring, vocational training, counseling and psychosocial interventions, food supplements, health education, life skills, legal protective services, and income-generating opportunities. Families have been targeted with economic strengthening activities, such as savings groups. PC3 has encouraged local organizations, especially iddirs, to expand their role in the community to include a strong focus on OVC. In some cases, this required legal changes to an organization’s constitution or by-laws to allow for collecting monthly contributions from members to be used for services to OVC in the community. Over 500 local organizations, including 239 schools, now have a greater awareness of how to address the multiple needs of families and children affected by HIV and AIDS. Many of these organizations have become a haven for children, providing them a safe place to go to receive help and adult advice. x ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Local organizations have increased their capacity to raise funds and other resources from both within and outside the community and to manage these funds for the benefit of children. A database with a unique ID number for each child is in place, into which Tier II partners input information they receive from Tier III partners. The Tier III partners have copies of the database lists of children to help them with their planning and delivery of services. MAJOR CHALLENGES Local organizations require more time and more money so they can consolidate the gains they have made. Without some continued funding the momentum and structures that are now in place may diminish. PC3 needs to ensure that the phase-out does not discourage partners, especially newer and smaller ones. The need to meet high targets for numbers of children served can derail the project and its partners from delivering quality services, which imply holistic services, to a child. A focus on numbers of children reached can also de-emphasize the needs of the whole family. The need for food support can overwhelm the resources of community organizations. The concept of ―primary‖ and ―supplementary‖ services for OVC, as presented in the PEPFAR Guidance on OVC Programs, triggered a perception that everyone must offer everything. Partners with limited resources often end up very thinly spread, with overburdened volunteers working 14 or more hours a week in many locations. Meanwhile, the comparative advantage or expertise of an individual organization, such as a lawyers association or a youth association, was not fully exploited. Not all community committees have strong enough local networks to reach the diverse needs of children for services, especially health and protection services. Some local service providers may themselves be overwhelmed and unable to respond adequately. Close links with GOE programs and services, for example HAPCO, were not apparent in all regions and sites, although some sites show excellent examples of cooperation. Referrals are hampered by the lack of resources within any one organization. As one Tier II partner explained, organizations to which one might refer children are already serving children and will not take on additional clients from other programs. Organizations can refer children for government services, such as exemption from health fees and school fees, through the kebele. Where the kebele is mobilized and networked through PC3, this can work well; elsewhere, it continues to be a problem. LESSONS LEARNED Too tight a focus on serving the child has diverted attention from the needs of parents and guardians, even though it is acknowledged that the home is where children can most effectively and sustainably be given the support they need for emotional, cognitive, physical, spiritual, and social development. The seven service areas present a broad package of essential services for a child and caregiver. They do not provide a blueprint of what a program should deliver, but they do indicate what providers might need to focus on in assessing needs and targeting interventions. Some services, such as educational services, can be easily defined and given. Others, such as psychosocial support, cannot be given in the same way as a vaccine can be administered or a school fee can be paid. A more nuanced approach to and understanding of psychosocial support is needed, one that considers the whole environment of a child—family, school, community—and discerns where psychosocial skills can be secured. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) xi A tiered approach to capacity building takes longer to implement but can be more sustainable once local NGOs have the skills, materials, and approaches they need to continue. The mixed package of capacity development—training, materials, coaching, and subgrants— is stronger than any individual element. Full-time staff placed at the Tier III level at the right moment for an extended period (2–3 years) could further enhance the package. CBOs can be empowered to manage subgrants in a sustainable manner with a combination of training, coaching, and financial assistance. SHORT-TERM RECOMMENDATIONS Resources Increase the focus on local resource mobilization to alleviate fears that some children will no longer be served as PC3 phases out. Continue helping Tier II and III partners to team up to apply for resources. Use as examples of good practice for others the project sites that are currently working effectively with a number of different agencies and partners to coordinate care for children. For example, ProPride in Dire Dawa seems to have an extensive web of partnerships beyond HIV/AIDS programming. Tier II partners need to receive and then provide technical assistance on how to make practical use of community-mapping results to relieve overburdened local partners and expand service coverage. This would include an outline of next steps and guidance on action, especially relating to filling gaps as PC3 phases out. Consider allowing Tier III partners still within the project to use some of their subgrant to start income-generating projects in support of OVC, as did the schools under World Learning. Services Identify and recruit an individual or team with extensive experience in economic growth to review current activities aimed at increasing household assets and suggest alternatives and additions to the income-generating portfolio of activities. Immediately convene leadership from across the three tiers of partners to strategize on how to navigate the current food crisis. USAID and other donors are intensifying their food aid. The PC3 network is well placed to make good use of the increased external inputs to meet emergency needs. Review current promising practices from PC3 and formulate the essential care components needed within a community to provide comprehensive support to households affected by HIV/AIDS. Dissemination Expand upon communities of practice and exchange visits (both Tier II and Tier III partners found these to be ―highly valuable‖) to include leadership forums where lessons learned and strategies can be exchanged. Explore what is needed to establish a centers-of-excellence approach to increase country and even regional exposure to Tier II and III processes and results. Document best practices in human resource capacity development for use by others working to support OVC (e.g., leadership exchanges or communities of practice for program management across Tier II partners). All manuals and guidelines should be translated and disseminated to Tier II partners. xii ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Set up a meeting with PC3 management, staff seconded to MOWA/HAPCO, and the two GOE team members from this evaluation to discuss ways to disseminate findings and approaches of PC3. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 1 I. BACKGROUND ETHIOPIAN CONTEXT Ethiopia has a population of 81,021,000 (State of the World‟s Children, 2008), of which 43 percent is under 15. The Single Point HIV Prevalence Estimate of 2007 places the number of people with HIV/AIDS at 977,394—2.1 percent of the population; rural prevalence is 0.9 percent and urban 7.7 percent. Differences are also significant between men (1.8 percent) and women (2.6 percent). The same report estimates that there are 5,441,500 orphans, of whom 898,350 are due to HIV/AIDS. This, combined with poverty, creates a large number of children with extensive vulnerability. The Orphans and Vulnerable Children Rapid Assessment, Analysis, and Action Planning Report Ethiopia (RAAAP) produced by UNAIDS, WFP, UNICEF, USAID, and the GOE, referring to the 2000 Ethiopia Demographic and Health Survey (EDHS), speaks of 15,897,600 vulnerable children, of whom 2,558,100 have been affected by HIV; 18 percent of households are caring for an orphan. The vast majority of households in Ethiopia are rural (85 percent) and headed by males (75 percent), although in urban areas more households are headed by women. Although literacy rates have been improving, rates for females (27%) are far below those for males (50%).1 Schools offer vulnerable children a myriad of services and opportunities, including interaction with a peer group, current education, and future job possibilities; contact with a supportive adult; and HIV prevention information, to name a few. In Ethiopia enrollment rates for children are increasing, but there are still marked differences between girls and boys at the secondary level, between rural and urban areas, and between regions. In 2004/5 11.4 million children were attending primary school, for a gross enrollment of 79.8 percent—70.9 percent for females and 87.3 percent for males. Secondary school gross enrollment for the same year was only 29.2 percent (21.6% of females and 36.6% of males) (PASDEP, 2006). Getting children into school and keeping them there may be one of the best HIV prevention mechanisms and psychosocial support interventions. It is critical for both lowering the prevalence rate in Ethiopia and protecting orphans and children made vulnerable by HIV. Delaying sexual debut while providing life skills is a critical function of the school in addressing HIV/AIDS prevention. The GOE is implementing a quality improvement plan for education, but given the pupil-teacher ratio of 60:12 quality improvement has been elusive. Ethiopia achieved a 9.6 percent growth in gross domestic product (GDP) in 2006–07, but GDP per capita is still only US$172. A major negative driver of development in Ethiopia today is inflation. Officially at 10 percent but unofficially as high as 39 percent, inflation is clearly affecting daily life as food and transport prices rise and associated costs spiral. Food and nutritional insecurity is a consistent backdrop to any community-focused project in Ethiopia. With 47 percent of children under 5 experiencing severe to modest stunting3 , the need for food and nutrition strategies especially for OVC is clear. In general, rural children and children of uneducated mothers are more likely to be stunted, wasted, or underweight than other children. The regional variation in the nutritional status of children is substantial. 1 Ministry of Finance and Economic Development, Ethiopia: Building on Progress. A Plan for Accelerated and Sustained Development to End Poverty (PASDEP) 2005/06 – 2009/10. Volume 1. September 2006, Addis Ababa (PASDEP). 2 Garcia, M. Rajkumar, A.S., Achieving Better Service Delivery through Decentralization in Ethiopia, World Bank Working Paper No. 131, 2008 (World Bank, 2008). 3 UNICEF, State of the World‟s Children 2008: Child Survival, December 2007 (UNICEF, 2007); EDHS, 2005. 2 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) RATIONALE FOR EVALUATION USAID/Ethiopia awarded the umbrella Cooperative Agreement known as Positive Change: Children, Care, and Communities (PC3) in September 2004 to Save the Children/USA as prime recipient, with CARE International, Family Health International (FHI), Hope for African Children Initiative (HACI), World Learning International, and World Vision (WV) as partners. (HACI is now defunct.) The PC3 team addresses the needs of OVC affected and infected by HIV/AIDS in seven regions or city administrations of Ethiopia: Amhara, SNNPR, Oromia, Addis Ababa, Afar, Dire Dawa, and Beneshangul. The project is one of the largest USAID-funded projects in support of OVC in Africa with a budget of nearly US$20 million over five years. The PC3 program was initiated when it was recognized that there was an urgent need to scale up care and support services for OVC on a sustainable basis. Its major concern was to address the challenges Ethiopian communities face in assuring positive change among communities affected by HIV and AIDS. The strategy envisioned was (1) to provide community-based care and support to OVC, and while so doing (2) to increase the capacity of Ethiopian nongovernmental organizations (NGOs), community-based organizations (CBOs), and faith-based organizations (FBOs) to provide such care and support over time. The project had three intermediate objectives: IR 1: Increased availability, quality, and consistency of community-based support services for OVC and families affected by HIV/AIDS. IR 2: Improved capacity of Ethiopian civil society organizations (CSOs) to plan, implement, monitor and evaluate, manage, and report on OVC programs and services. IR 3: More supportive environment for OVC and their households developed through strengthened coordination, networking, and advocacy. The intent of the PC3 program was not to increase the number of NGOs/CSOs working with OVC but rather to scale up community-based responses to the needs of OVC and to assure that more children get more services and protection. The program was designed to comprise a group of organizations at least some of which have established relationships with CSOs in Ethiopia. The consortium of five international NGOs is known as the Tier I partners. The 35 local NGOs are called Tier II partners, and the 239 schools and 321 community organizations through which they work are known as Tier III partners. This end-of-project evaluation was undertaken May 5–23, 2008, 16 months before project completion; it was commissioned to design and implement an independent external evaluation. The evaluation was tasked with collecting information about PC3 implementation, progress, and challenges, and formulating recommendations for follow-on programs. The evaluation report should help USAID and Save the Children address topics of management, quality of services, and sustainability. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 3 II. METHODOLOGY A participatory evaluation methodology was used to engage the different stakeholders in reflecting upon the program, its achievements, their own roles, and the challenges and opportunities for their organization and for the program as a whole within the country. Given the composition of the team and the limited time available, the evaluation was primarily qualitative, based on a review of documentation, in-depth guided interviews, focus group discussions, and observations by nonparticipants. It collected primary and secondary information about PC3 implementation, progress, and challenges. The priority areas for the evaluation from the Scope of Work were 1. Review of documentation (see Appendix F) 2. Informant interviews with Tier I partners and other stakeholders, including government officials in Addis (see Appendix B) 3. Site visits to a representative sampling of Tier II and III partners using 4. Questionnaires for Tier II and Tier III partners (see Appendix E) 5. Focus group discussions for caregivers 6. Focus group discussions for children and youth beneficiaries 7. Debriefing and final consultations with USAID and PC3 stakeholders. The evaluation used a purposeful sample for site visits to Tier II and III partners. To be as representative as possible the following criteria were considered: The Tier I partner Regional representation (Addis Ababa, Amhara, SNNPR, Oromia, Dire Dawa) Type of Tier III partner (iddir, church, school, cooperative, youth association, women’s association) Exiting in June or not Size (large or small) The evaluation team consisted of one independent consultant, one OVC specialist from the USAID Africa Bureau, and two professional project specialists from the GOE Ministry of Health. An independent local consultant provided logistical assistance. Two teams were established, each consisting of a consultant and an MOH staff member. For quality assurance purposes the combined team participated in two site visits in Addis before departing separately for regional visits. A regional coordinator from a Tier I organization or a Save the Children OVC technical coordinator was available at each site to assist with the visits and arrangements (see schedule in Appendix B). The two teams visited 10 sites in five regions: Addis Ababa, Nazret/Adama, Walenchit, Awasa, Shashamane, Yirgalem, Dilla, Bahir Dar, Dire Dawa, and Hirna. Before the evaluation a questionnaire was sent to all 35 Tier II partners; 11 responded. A guided interview format was used during in-depth discussions with 8 of the Tier II partners. A total of 16 different Tier II organizations were surveyed by questionnaire or interview (three both completed the questionnaire and were interviewed). An observation tool and set of guiding questions was used during the site visits to 15 of the 560 Tier III partners. Each team also held focus group discussions with the beneficiaries, caregivers and children. Organizations were given assurances of informed consent. The children’s discussions were participatory, age appropriate, and hopefully fun. Care was taken not to personalize the discussion or to expose or delve into 4 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) sensitive information. To be as inclusive as possible, three group meetings were organized with representatives from all the Tier III partners active in a particular locality. In total 30 Tier III organizations were covered through site visits or group meetings (see Figure 1). LIST OF TOOLS (SEE APPENDIX E) Questionnaire for Tier I partners administered at a group meeting Questionnaire for Tier II partners sent out before the evaluation Semistructured guided interview format for Tier II partners Semistructured guided interview and observation form for Tier III partners Agenda and guiding questions for general meetings with Tier III partners Focus group discussion guidelines for caregivers Focus group discussion guidelines for children and youth (over 12 years old) Guidelines for key informant interviews The fact that the evaluation took place at the start of a sequenced phase-out period may account for the emphasis of Tier II and Tier III partners on the need for continuation, and the persistent message that more time was required to optimize the results of PC3’s efforts. The field visits were well organized; every meeting took place as scheduled. Unfortunately, there was no time to triangulate information with other local service providers, such as clinics or hospitals. Figure 1: Summary of Site Visits and Key Informant Interviews Only GROUP NUMBER OF ORGANIZATIONS FEMALE MALE TOTAL Tier I partners International NGOs: 5 10 19 29 Tier II partners NGOs interviewed: 6 Respondents to questionnaire: 11 FBOs interviewed: 2 Out of 35 6 23 29 Tier III partners— core group members Iddir: 1 Iddir coalitions: 5 Associations: 5 Schools: 4 Combined meetings: 3 Out of 560 56 112 (28, gender unknown) 196 Children Focus group discussions: 7 42 22 64 Parents/guardians Focus group discussions: 5 48 3 51 Other Key Informants USAID, WFP, UNICEF, HAPCO, MOWA, MOH (team) 4 7 11 Total 166 214 380 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 5 The report follows the indicative questions asked in the Scope of Work about program management, the three IRs, and monitoring and evaluation (M&E). For the IRs the report gives findings divided into achievements and challenges, lessons learned, and short-term recommendations. Recommendations for a follow-on program are given in the final chapter. 6 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 7 III. FINDINGS IR 1: INCREASED AVAILABILITY, QUALITY, AND CONSISTENCY OF COMMUNITY-BASED SUPPORT SERVICES FOR OVC AND FAMILIES AFFECTED BY HIV/AIDS “Lam Kendwa Aykebdatem”: A cow will manage to carry her horn. Findings PC3 stimulated increased community awareness of the comprehensive needs of children affected by HIV/AIDS and their families. Previously community services were not tracked, and volunteers focused on specific areas, not necessarily a combination of needs. The organization of volunteers into teams, each addressing one of a variety of services required by children and together creating a package of services, is evidence of a holistic approach. Children and households are generally given priority when they are extremely vulnerable (e.g., double orphan status). Support to community organizations and groups required innovation and flexibility to manage the resulting network of over 500 volunteer-led local entities that are now able to track service provision to over 300,000 children. Responsiveness to PEPFAR OVC Guidance has been systematic; it includes a database categorizing children receiving primary and supplementary services. Local resource mobilization and extensive links with education, early childhood development, and food aid programming have augmented PC3 inputs. However, no concrete linkages with child survival programming were noted. Overview OVC in seven regions or administrative areas of Ethiopia are receiving services through PEPFAR: Addis Ababa, Afar, Amhara, Benishangul, Dire Dawa, Oromia, and SNNPR. From the start of PC3, IR 1 focused on community provision of comprehensive care and support to OVC. This consists of education, psychosocial support, health and nutrition, household livelihood support, child protection, life skills, early childhood development, and legal assistance. Findings from this evaluation reinforce or support findings found in the OVC Service Mapping Report (Nov. 2007), e.g.: Psychosocial support is the number one service offered by PC3, primarily through trained volunteers conducting home visits, some recreational activities, and some counseling. Educational services take different forms, such as provision of school supplies or uniforms, free attendance at private or vocational schools or preparatory schools (informal) or early childhood development centers or kindergartens. Educational support is the second most common service offered; it is a priority of Tier III partners, children, and guardians. Food and nutrition support is provided in conjunction with the World Food Programme (WFP) urban HIV/AIDS project. WFP targets communities that can demonstrate that the child will receive more than just food. One of the objectives of PC3 is to ensure that children stay in school. The WFP project reported that the combination of food from WFP and educational and psychosocial support from PC3 partners proved a more powerful combination for ensuring that children remain in school than food alone. Community therapeutic care (CTC) also provides emergency nutritional supplementation for young children through health centers. 8 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Life skills are offered through a partnership with Health Communication Partnership (HCP). HCP has drafted three life skills curricula—Beacon Schools, Youth Action Kit, and Sports for Life—for different age groups. HCP trained Tier I and II partners on these materials, and they in turn have trained schools and communities. Economic strengthening is offered in two ways: Guardians and volunteers have been trained and helped to establish community self-help saving groups (CSSGs) that collect weekly or monthly contributions from members. This money is then available for lending to members. Tier I partner CARE is leading this intervention and providing specialists to work with staff in other Tier I partners to provide technical assistance to groups over a 12-month period. Although the income generated is low, it can be enough to help families get through economic shocks associated with illness, unemployment, rising dependency rates, and other household changes. So far 350 CSSGs have been established, with 405 men and 5,819 women as members. Upon graduating and with official registration, these groups can be assisted by government small and micro enterprise units. Other forms of economic strengthening are subsidizing vocation training, apprenticeships, and start up-costs for micro enterprises. The majority of these described to the team were in hair dressing, sewing, shoe shining, food preparation, and woodwork for older OVC or guardians. Protection services were identified as inadequate by a number of partners. Despite some success stories, partners expressed the need for more training and information on child protection. No Tier I partner had taken the lead on legal and protective services since an original partner, Hope for African Children Initiative (HACI), was disbanded. The annual reports of the PC3 program break out the numbers of children receiving specific services. Table 2 gives the breakdown for April 1, 2007–March 30, 2008. Table 2: Number of OVC Served by Core Program Area and Age Core Program Area Number Supported Food and nutritional support 37,151 Shelter and care 2,456 Protection 24,999 Health care 121,320 Psychosocial support 163,330 Education and vocational training 126,460 Economic opportunity/strengthening 12,223 Note: The number of children served should not be added up— most receive multiple services. Age Number supported Under 2 years 5,024 2–4 13,863 5–11 88,646 12–17 76,420 Total 183,953 Source: PC3 Annual Report April 1, 2007–March 30, 2008. Since inception PC3 has reached a total of 395,866 OVC (Annual Report, May 2008), of which 189,838 are male and 206,028 female. It has also given training or support to close to 30,000 caregivers. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 9 A cascade training approach was used to reinforce and expand the network of volunteers within communities to provide comprehensive care and support. No paid workers provide services to children, and volunteers are not supervised by paid staff. Local service providers apply skills acquired through PC3 training and mentoring to improve methods for mobilizing and organizing their efforts to support as many children as possible. Who Gets Services? Community core groups, consisting of representatives of local Tier III partners, identify children who need assistance and the type of assistance required. Priority for service is given to young double orphans; followed by young single orphans, then, if resources are sufficient, other OVC. For example, a 6-year-old double orphan would have priority for services over an 11-year old single orphan. PC3 communities regularly assess need, and most have registered the number of OVC needing support—as in other countries, demand far exceeds supply. The impact of HIV/AIDS combined with poverty creates extensive vulnerability for a large number of children. At the outset a target of 500,000 OVC was set for PC3 to reach. There seems to be some confusion about this target. USAID/Ethiopia, like other PEPFAR countries, stated that at the end of the project (September 2009) 500,000 OVC will be receiving services. Save the Children is operating on the basis that the target is cumulative, so that over the life of the project 500,000 OVC will have received services. Since only 15 months remain in the PC3 project, it is not realistic to expect a shift in the counting of beneficiaries. The follow-on activity needs explicit direction on whether targets are cumulative, with explanations about members of the target group graduating or aging out. Achievements The cascade or tiered approach engaged an extensive network of service providers able to offer a range of services and support to OVC. They have internalized the holistic nature of care and address many of the seven components within PC3. The community lens on what a child needs to thrive has been broadened. Children and guardians list a variety of sources from which they receive support, with iddirs topping the list (see Appendix E). Satisfaction is highest for life skills and educational support services, according to both children and their guardians, the focus group discussions revealed. Caregivers particularly appreciate inclusion of household economic strengthening; they stated a desire to be completely independent from external inputs. The Tier III partners appeared to offer services on a staggered basis, addressing the most pressing needs of a child or caregiver in an attempt to reach as many children as possible rather than providing all services to everyone regardless of priority. While many beneficiaries interviewed indicated they would like services to be more consistent, they agreed with the focus on equity even if this meant staggered support in order to reach more children. Extensive monitoring of services was evident through documentation, and the core groups use monitoring forms provided by PC3, such as the Parent/Guardian Support Service Provision Record, the OVC Care and Support Service Provision Record, and the Community Mobilization Record. Tier III partners seem to have overcome the challenge presented by the PEPAR OVC Guidance related to primary and supplementary direct services through use of the database with unique ID numbers. Local partners indicated that a focus on three or more services encouraged broader consideration of OVC needs and how to meet them. It appears that most of the subgrants given to Tier III partners are used to cover children’s educational costs—pens, notebooks, uniforms, fees, etc. Upon close consideration, the 10 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) rallying of community awareness and responsiveness to OVC has resulted in more emphasis on the right to education even for OVC. Tier III partners make extensive use of referrals to meet OVC needs. The mapping data are now being used to help expand referrals and better coordinate even with health facilities. PC3 has begun to apply draft service standards—developed with the leadership of PC3— in some sites to improve service quality. These provide guidance on what constitutes a reasonable package of services for OVC based on identified needs. Challenges Because baseline studies on availability, quality, and consistency of services were not conducted at the start of the project, partners focused on increasing the availability of services based on information from household need assessments. A pilot study on applying service standards has recently focused attention on quality. Given the time remaining for the project, it is not likely that data on service quality will be available from all seven PC3 regions. Perceived pressure to provide at least three services due to new guidance on OVC programming from PEPFAR has triggered a belief that everyone must offer everything, to the point where a lawyer’s association hired a nonlawyer to train and supervise Tier III partners in home-based care. It seems that there was slight pressure, real or imagined, to provide three services so as to be able to count children as receiving primary direct support according to PEPFAR Guidance. Partners with limited resources often end up very thinly spread, with overburdened volunteers working 14 or more hours a week in many locations. Of the 20,000 schools in Ethiopia, 239 are currently providing services to OVC through the PC3 program. Even though other programs reach other schools, the basic elements of the PC3 program need to be embedded in more school-based programs. What can be done to acquaint more school faculty with life skills education and the special needs of OVC? How can support to the MOE be increased to expand coverage while keeping the program sustainable? The PC3 approach of reducing external inputs while increasing the coverage and comprehensiveness of services is creating stress among local partners. Tier II and III partners stated that they will not be able to keep up the current level of services using only local resources. Communities are not asking to reduce OVC targets, they are only seeking more time to increase their ability to bring in resources separate from PC3. Community resources were mapped in mid-2007, and local entities are now determining how to maximize local potential and opportunities for mobilizing more resources. Baseline supply and demand data are not available (e.g., number of children affected by HIV/AIDS who are starving or lack basic material care). For example, such data would provide an estimate of community capacity (human and fiscal) to meet demand so that a budget amount could be set to fill gaps with immediate emergency relief. After emergency relief, projections for OVC needing services to mitigate the effects of HIV/AIDS could be calculated so as to negotiate with the community the targets to be set and link them to external support over time, with a graduation strategy. Budget figures were not set according to each IR and based on results of community supply and demand analysis. USAID indicated that at least 75 percent (later reduced to 65 percent) of budget was intended for local or community use. This amount has been interpreted to cover both provision of services and building the capacity of national and local entities. Therefore, it has not been possible to draw conclusions about what amount and combination of external and community-based resources (human and fiscal) are needed to meet demand. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 11 Referrals are hampered by the lack of resources within any one organization. As one Tier II partner explained, organizations to which children might be referred are already serving children and will not take on additional clients. Organizations can refer children for government services, such as exemption from health fees and school fees through the kebele. Where the kebele is mobilized and networked through PC3, this can work well; otherwise, it is a problem. Lessons Learned The process for identifying families and children in need of support can create expectations that cannot be met. The large numbers of children identified in the start-up phase of the project forced partners to give priority to the most vulnerable. Although this ensured that very vulnerable children would be reached, it precluded the project from reaching families before a crisis, when fewer and less expensive interventions might have been sufficient. Local partners give priority to young double orphans rather than households with chronically ill individuals. Also, it appears that nonbiological children in a household are selected for support even though the entire household has been affected by taking in children who have been orphaned. Too tight a focus on targets related to serving the child has drawn attention away from the needs of parents and guardians, even though it is acknowledged that the home is where children can most effectively and sustainably be provided with the support they need for emotional, cognitive, physical, spiritual, and social development. The seven service areas represent a broad package of essential services for a child and caregiver. They do not provide a blueprint of what a program should deliver, but they do indicate what providers might need to focus on in assessing needs and targeting interventions. This can only be determined in response to the priority needs of children and their families in a specific resource context. Some services, such as educational services, can be easily given. Others, such as psychosocial support, cannot be given in the same way as a vaccine can be administered or a school fee can be paid. A more nuanced approach to and understanding of psychosocial support is needed, one that considers the whole environment of a child—family, school, community—and discerns where psychosocial skills are developed and secured. An antipathy toward hand-outs can nurture self-reliance but may also place unrealistic burdens on resource-poor communities and families to provide all the care and services a vulnerable child or family may need. In any society at any given time a certain percentage of people will need social services support. What is required is a sense of the timing, type, and source of the external inputs and the strategies required for effectively stabilizing and graduating strong local entities. A baseline assessment of community caring capacity (e.g., a SWOT [strengths/weaknesses/opportunities/threats] analysis) would have made it easier to establish concrete markers of progress in service provision beyond numbers. Staff turnover at USAID and among Tier I and II partners likely contributes to confusion on how best to follow PEPFAR OVC Guidance. Technical integrity can be compromised when there is variation in how to navigate PEPFAR demands. Short-Term Recommendations for IR 1 Immediately convene leaders from all three tiers of partners to strategize on how to navigate the current food crisis. USAID and other donors are intensifying their food aid. The PC3 network is well placed to make good use of the increased support. 12 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Identify partners that best demonstrate coordinated care and multisector integration, and use them to provide guidance to others. For example, ProPride in Dire Dawa has an extensive web of partnerships beyond HIV/AIDS programming. Increase the focus on mobilizing local resources to alleviate fears that some children will no longer be served as PC3 phases out. Continue helping Tier II and III partners to team up to apply for resources. Convene an experienced economic growth team to review current activities aimed at increasing household assets. A point-in-time study on what is and is not working well and why can help refine current efforts (e.g., are vocational training and small business development activities market-driven and sustainable?) and inform follow-on efforts. Such a review was conducted in Uganda and Kenya by the USAID centrally-funded mechanism FIELD Support (Leader with Associate’s Award) of AED. Save the Children is the associate partner that conducted that review. Review current promising PC3 practices as a basis for identifying essential care components needed within a community to provide comprehensive support to families or households affected by HIV/AIDS. As with service standards, there is a need to convey objectively and assess a community’s capacity (e.g., strengths, gaps, untapped potential) to meet the needs of vulnerable children. Are core group structures, service mapping, early childhood development centers, scholarships, referral networks, advocacy and other components considered the essential minimum for communities in Ethiopia? Are they sufficient to ensure good enough care of vulnerable children? How can community competence be measured and used to inform concrete outcomes? Continue investing in quality improvement by piloting OVC service standards. Revise service standards based on the piloting results in Dire Dawa and Addis Ababa. Piloting considers both how to apply standards and how to refine the definition of quality services and quality improvements. More sites should be added. IR 2: IMPROVED CAPACITY OF ETHIOPIAN CSOS TO PLAN, IMPLEMENT, MONITOR AND EVALUATE, MANAGE, AND REPORT ON OVC PROGRAMS AND SERVICES “[Tier II partners] are at our service. We can call them whenever we need anything We get money from HAPCO but we get support from our Tier II.” Findings Tier II and Tier III partners of the PC3 project have increased institutional and technical skills and financial management ability, including subgrant management, training in technical areas, and monitoring and reporting. Tier III partners of the PC3 project have received continuing training, coaching, and mentoring from Tier I and Tier II partners and demonstrate capacity to mobilize resources in their community and from outside partners in support of OVC and their families. Despite relying solely on volunteers to manage and implement all aspects of the program, Tier III partners are able to utilize resources to provide services for OVC. They are most hampered by the fact that resources are minimal compared to the large numbers of children that need to be reached and they lack paid staff, full or part-time, to manage the project. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 13 Overview The project was designed to take advantage of existing activities, partners, and projects throughout the country. A unique network of community support exists within Ethiopia, the iddirs or burial societies. This well-organized and highly responsible grouping of community members was in place long before PC3. Also, a multiyear education initiative funded by the USAID/Ethiopia Basic Education Services Office equipped at least half the current education committees under PC3 with skills to address the within-school needs of OVC. Because Tier I partners already had partners who became part of PC3, it is not always possible to attribute all the capacity development evident within Tier II partners to the efforts of PC3. The capacity of all Tier II partners was assessed before a plan for capacity building was drafted. That tool has not been re-administered to assess current capacity, though regular review meetings determine gaps. The tool may not have been particularly useful, or the project may have already had a strategy for capacity development. The Tier I partners assembled and delivered a package of training on financial management, community mobilization, subgrant management, and organizational development using standardized manuals. Although PC3 appears to have a fairly unified structure and strategy, the variation among Tier III partners has demanded a flexible and nuanced approach. Tier III partners range from recently established youth associations and school PTAs to iddir coalitions with 50 years of history. These organizations have different comparative advantages, but all expressed appreciation for the training in community mobilization they receive from PC3. Achievements The PC3 approach to capacity building involved an innovative combination of training, coaching, mentoring, and subgrants. It combined organizational development with technical skills on OVC issues. The package, though time-consuming and labor-intensive, has enabled Tier II and Tier III partners to improve and expand the services offered to children and families affected by HIV and AIDS. The comprehensive life-skills training for adults and children, for example, must confront long-standing social norms, such as of adults not talking openly with children about reproductive health. Tier II partners have demonstrated capacity in financial management, M&E, and institutional development. Some Tier IIs have restructured to facilitate their work with CBOs and to accommodate the holistic approach of PC3. Tier II partners have written proposals and have received funds from other sources, including the EU, HAPCO, and other international partners. They have also become experienced in managing subgrants to community organizations. Tier IIs manage the database, collecting data from their Tier III partners. An analysis of the questionnaires and interviews of 16 Tier IIs shows that they value organizational capacity development more highly than financial assistance; and that helping organizations to diversify their funding base is as valued as actual funding itself. 14 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Table 3: Most Important Contributions from Tier I According to 11 Tier IIs Most Important Contribution from Tier I Frequency of Response Organizational capacity building 8 Financial assistance 7 Assistance in diversifying funding 7 Training in monitoring and reporting 6 Coaching 6 Training for staff 6 Approaches to community development 5 Exposure 3 Training in financial record-keeping 3 Assistance in assessing gaps in program 3 Tier II partners were responsive to problems of their Tier III partners. In the field it was reported that ―They are there for us. Tier II is at our service.‖ In 2007 Tier III partners were providing services to over 200,000 children. Through the PC3 project they have developed the capacity to identify OVC; prioritize their needs; mobilize resources to address the needs; facilitate community volunteers to undertake regular assessments; initiate income-generating or saving activities for caregivers; and monitor all of these activities. Many of the Tier IIIs already had capacity in some of these areas, but the particular package of activities they have undertaken through PC3 partners is impressive, comprehensive, and structurally sustainable. PC3 has produced a number of manuals and guidelines for project participants to use, among them manuals on – Governance and leadership – Networking and partnering – Psychosocial support for OVC and their caregivers – Strategic planning – Guidelines on managing volunteers – Quality assurance and improvement standards – Starting CSSGs – Coordinated care – Community mobilization (the community action cycle approach) – Documenting promising practices and most significant changes – Implementing an OVC program in schools Local NGOs can continue to use these facilitator guides both for their own development and for CBOs in their programs. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 15 The mapping exercise conducted in May 2007 with all partners has further built understanding of local resource mobilization opportunities. It has led directly to new partnerships with the private sector—hotels, for example—and with individuals, foundations, and other partner organizations. Challenges Staff turnover among Tier II partners is a constant problem. Tier III partner schools also experience turnover as teachers are redeployed. This does slow capacity development, but institutional restructuring, institutional memory, manuals, and excellent documentation means that despite turnover Tier IIs seem to be strong, organized, enthusiastic, and equipped to continue the work of PC3, given resources. Fortunately, there is little turn-over in iddirs, associations, and other Tier III partners Delays in disbursement of funds have limited Tier III ability to deliver timely services. When funds are received late, Tier III partners reportedly are forced to purchase materials for their activities quickly or to delay planned meetings or workshops. This means that scarce resources may not always be used in the most effective way. For example, one Tier III respondent reported, ―We had to go out into the market and just buy supplies without negotiating the best deals.‖ The volunteer nature of all Tier III organizations inhibits their capacity to expand their services to more children. Tier IIIs are totally volunteer-led, managed, and run. Schools may be an exception because some members of the committees are teachers, but the time they give to the committees is volunteered. The core groups, consisting of local iddir or association leaders and community representatives, volunteer up to 14 hours a week to manage PC3 activities, including mobilizing resources from other sources. The iddirs have expanded beyond their traditional field of operations and are working with larger communities than just their members. This has implications for what can be expected. Exclusive reliance on volunteers, no matter how committed, contributes to member burn-out. Additional options are essential, such as paid Tier III staff to recruit, train, and manage the pool of volunteers. Tier III partners, except the schools, were not given the opportunity to use any of their subgrant for income-generating activities. Most of the schools, on the other hand, at the behest of the core group and parent-teacher associations decided to use their small subgrant to initiate an activity that would generate continuing funds in support of OVC. Schools visited had a small shop attached to the school compound, which was rented out; bought a refrigerator, which was placed in the canteen and rented out; and started a major gardening initiative, which generated funds. This strategy, which was not considered for the other Tier IIIs, is a lost opportunity to bring in sustainable income for the needs of OVC and their families. Nevertheless, Tier III partners that were visited had sufficient financial and management capacity and community accountability for their projects. Lessons Learned Uniting multiple international NGOs into one project gave substantial value-added to capacity development; each offering a different area of expertise as part of a concerted whole. Management of such an arrangement is time- and labor-intensive. Moreover, the cost of capacity building is not yet fully understood in terms of types and levels of investment and the returns they generate. What, for example, are the critical minimum care and support functions that must be demonstrated by a community to convey that its capacity is good enough? When OVC partners share and understanding of what good enough capacity is, actions and targeting for building capacity can be more informed. A tiered approach to capacity building takes longer to implement but can be more sustainable because local NGOs have the skills, materials, and approaches they need to continue. 16 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) The mixed package of capacity development through training, materials, coaching, and subgrants is more effective than any single element. Full-time staff placed at the Tier III level at the right moment for an extended period (2–3 years) could enhance the package. Options for initiating movement in this direction include use of health extension workers via the training in para-social worker approaches that is nearly in place at Addis Ababa University. This is a six-day training followed by six months of supportive supervision. Also, the U.S. government can discuss with other donors and the GOE ways to achieve multistakeholder buy-in to support the GOE with local staff focused on the needs of households affected by HIV/AIDS. Community organizations can be empowered to manage subgrants in a sustainable manner through a combination of training, coaching, and financial assistance. Mobilizing local resources—human, financial, and in-kind—is possible in resource-poor settings and can be leveraged with external funds, training, and awareness-raising. Recognition by community and peers and satisfaction with one’s work are two of the strongest factors in keeping services sustainable. As one iddir member explained, ―We get satisfaction through knowing we are making a difference.‖ Short-Term Recommendations All Tier I partners should produce a detailed assessment of the future prospects for each of their Tier II and Tier III partners. This would make it clear whether or not a partner has been successful in obtaining other sources of funding; is linked to other support networks; has linked children and families to other quality service providers; etc. This should guide implementation of the phase-out strategy. Tier II partners need to receive and then provide further technical assistance on use of data from the community mapping exercise. This would include an outline of next steps to guide action, especially to fill gaps as PC3 is phased out. All levels should document best practices in human capacity development that can be expanded and shared with other NGOs, local and international, working to support OVC (e.g., leadership exchanges or communities of practice for program management across Tier II partners). Tier III partners still within the project could be allowed to use some of their subgrant to start an income-generating project in support of OVC. World Learning gave schools a small grant to start an income-generating activity for support of the OVC attending that school. Schools used the grants, e.g., to build and rent a shop or plant vegetables for sale, which gave them a continuing source of income. Use the Annual Program Statement (APS) process as much as possible to accommodate Tier II and Tier III partners, especially those that have demonstrated capacity to mobilize community responses. It may not be possible without additional funding for PC3 to continue operating at current levels and increase the number of children served while adding new initiatives (e.g., CTC) and improving quality. The USAID mission should convene a strategic planning session with PC3 management to work through the realities of the situation and plot optional response scenarios. Findings and recommendations from this evaluation report can be used to inform this session. Also consider gleaning discussion points on this issue from the multiday PC3 partner meetings that include representatives from all Tier II partners to inform scenarios and map actions for the next 14–15 months. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 17 IR 3: MORE SUPPORTIVE ENVIRONMENT FOR OVC AND THEIR HOUSEHOLDS DEVELOPED THROUGH STRENGTHENED COORDINATION, NETWORKING, AND ADVOCACY „Maqetatel‟: ignite or spark Findings PC3 participates in networks at all levels. At the national level, the Tier I partners alone form an impressive network of agencies working in support of children; two of them are members of the National Task Force on OVC housed within the MOWA. PC3 Tier III partners have created their own networks through the PC3 iddir council, partners’ network, and similar organizations. These are effective, but they need to be linked or merged with other networks. Splintered responsibility for OVC among GOE entities has hampered concerted policy and implementation efforts at all levels to improve the wellbeing of children. PC3 has not maximized its potential to influence structures at the national or regional level. At the woreda and kebele level, Tier II and Tier III partners are able to participate with increased confidence and presence in networks like the HAPCO Coordinating Committees and the OVC Forums. This will enhance government decentralization, because strong CSOs can participate more fully in GOE plans for OVC and addressing HIV and AIDS. Achievements The core group structure wherein a number of local organizations meet regularly to determine activities has helped create local networks. Because children and guardians are represented in these groups, they can influence decisions that will affect them. PC3 has optimized its potential to form complementary partnerships with a wide array of partners. The WFP reports that PC3 enabled it to have a successful HIV feeding program to keep children enrolled in school. Finding partners that can offer psychosocial support and school supplies was a selection criterion of WFP; PC3 partners were a perfect match. The document on quality standards that was developed with the leadership of PC3 is now being considered by the National Task Force on OVC housed within MOWA. World Vision and Save the Children both sit on this task force. The task force is replicated at the regional but not the woreda or kebele level, where local forums and networks for OVC have been established. These local networks have yet to be coordinated with the task force, or in some cases even with each other. PC3 has seconded staff to MOWA and HAPCO to assist with OVC efforts. There are now funds for OVC earmarked within HAPCO. Due to involvement in PC3, kebele-level partners say they increasingly want to work together; they seek the benefits of coordination. Coaching and training from PC3 have fostered or ignited a new way of organizing community members to work together to meet the needs of OVC. National entities need now to catch up with the local momentum and find ways to reinforce and expand it. Social mobilization successfully resulted in formation of an innovative core group structure that combines the influence and resources of several iddirs and other community entities, such as women’s associations and youth groups. This type of Tier III structure is well placed to take advantage of resource mapping opportunities and combine resources to maximize service coverage and types. Representatives from the core groups participate in meetings held by local HAPCO and any local OVC networks. More diffusion of innovation is possible, as is sharing data on who is vulnerable and what needs are or are not being met. In some communities, the core groups had to overcome political suspicion that they were organizing 18 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) to rally against the government. That they did this through staying committed to their mission is notable. Challenges Splintered responsibility for OVC among GOE entities has impeded concerted policy and implementation efforts to improve child wellbeing at all levels. Without adequate and united leadership, both government and civil society entities cannot fully mobilize or realize their full potential (―stand on their own two feet‖). Such weaknesses and gaps in the system have allowed donor agendas to drive the work. HAPCO has regional joint planning networks, and the MOWA has OVC networks and task forces. Individual organizations establish local networks. There is actually no dearth of networks, rather an abundance of uncoordinated networks, often with the same participants. PC3 has not maximized its potential to influence national or regional structures. It was not clear to the evaluation team how the U.S. government used its access to GOE leaders, other donors, and UNICEF to work closely with PC3 to strategize on options for influencing national and local structures. As an impressive and widespread program chosen by independent consultants to be presented to Parliament as a best practice, PC3 should have a more influential voice. The Tier I partner initially responsible for advocacy, HACI Ethiopia, was disbanded when collaborators like CARE Ethiopia and Save the Children USA changed their approach to the HACI consortium strategy. PC3 partners at all levels have been consumed by the need to increase the number of children getting service or support. People are compelled to respond to the overwhelming need to mobilize resources out of fear that some children will no longer get services or the majority of registered OVC will go without services. Advocacy has had to wait. Short-Term Recommendations Expand upon communities of practice and exchange visits (both Tier II and III partners considered these ―highly valuable‖) to include leadership forums. For example, representatives from core groups, regional HAPC, and the Bureau of Labor and Social Affairs (BOLSA) can be brought together to mobilize resources for OVC and achieve very specific results by September 2009. Lessons learned and strategies can be exchanged on such topics as proposal writing, use of data to inform decisions and prioritize, donor relations, compliance issues, and conducting SWOT analysis on community capacity to meet the needs of OVC and their families. Explore actions needed to establish a centers-of-excellence approach to increase country— and even regional—exposure to Tier II and III processes and results. Set up a meeting with PC3 management, staff seconded to MOWA/HAPCO, and the two GOE members of this evaluation team to discuss ways to disseminate PC3 findings and approaches. MONITORING AND EVALUATION Findings PC3 uses a variety of monitoring tools and approaches, including volunteer recording sheets, a database with unique ID numbers for each child, best practice and most significant change stories, joint monitoring visits, and regular review meetings at different levels. The project is well documented and data are available at every level, including specific services received by a given child or family. The outcomes of services are harder to capture in a consolidated format, apart from success stories in all quarterly reports and anecdotal evidence ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 19 from project sites. Schools, iddirs, and other local partners do have information on positive outcomes for children and families due to PC3 interventions, such as school attendance and improved household income, but this type of quantitative information is not amalgamated and reported on for the project as a whole. The changes in PEPFAR guidance and monitoring requirements meant that a great deal of energy was expended on reformulating the monitoring system at the community level to reflect primary and supplementary support and address problems with undercounting. The database was used to address problems of double counting. Achievements The monitoring of the PC3 program has been detailed, extensive, and thorough. The M&E formats changed up to four times in the course of the five-year program, partly in response to new guidance from PEPFAR on primary and supplementary services but also in response to perceived double counting and undercounting. Although Tier III partners did comment on the effort the formats require, no one had huge complaints. This may indicate that they are finding the information useful. Most impressively, monitoring seems to be valued by Tier III partners and used as a management tool and as an argument in mobilizing resources. A database with a unique ID number for each child is in place. Tier II partners input information received from Tier III partners, which goes to FHI for final compilation. Tier III partners also have lists of children in the database, which helps them with planning and service delivery. The M&E system is able to track which services an individual child receives, by gender and age of the child. The use of a ―most significant change‖ methodology that uses the collection of stories and their filtering through all levels of the organization for M&E is commendable because it provides a window on how people experience PC3 services. It is not being used to its fullest because selection of stories has not been built into the management processes for Tier I, II, and III partners. Joint monitoring visits are organized so that a number of Tier I and Tier II partners can undertake peer assessments of a different partner every quarter. The joint monitoring visits are both a networking and a learning opportunity for participants. Challenges The monitoring system relies on reporting from volunteers. This is an additional burden to place on the unpaid volunteers. Since the new quality standards are still being piloted, the quality of data in terms of what represents a service is variable. An education service may range from receiving a notebook and a pencil to receiving a bursary to attend a vocational training institution. The same applies to other services, such as psychosocial services. There are currently no mechanisms to capture diversity and relative merit. The USAID-specified reporting format, apart from success stories, does not call for information about service outcomes—for example, whether or not a child who receives a school uniform remains in school. This information is generally available at local institutions, but a system to track a few key outcomes would strengthen the monitoring. 20 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Lessons Learned Given the funding, the target set for PC3—to reach 500,000 children over five years—was ambitious. The PC3 budget works out to US$40 per child; the average cost per child for PEPFAR is US$79 per year. It was always assumed that the Tier I partners would share some costs, which they did, and that communities would mobilize their own resources, which they did; however, facilitating the latter sustainably took time and resources. Changes in monitoring formats that may seem straight forward to Tier I partners, USAID, and PEPFAR translate into a whole array of training and communication issues when they reach down to the Tier III level. This is especially true for a program like PC3 that relies, as many programs do, on volunteers to collect the data. In order to motivate volunteers to undertake the potentially thankless task of data collection, the value of the information to be collected must be clear, as was the case when organizations used their data to attract other funds. PROGRAM MANAGEMENT AND DESIGN “If you want to go fast, go alone. If you want to go far, go with others.” The ambitious original design for PC3 covered an extensive range of technical and institutional competencies (see Technical Application, July 9, 2004). The program used a cascade model with the five international NGOs as Tier I, 35 local NGOs as Tier II, and 560 local schools and CSOs as Tier III. The program relied heavily on Tier II, national NGOs, having the capacity to absorb and apply the training from the Tier I NGOs, which each have a particular area of emphasis (e.g., psychosocial support, economic strengthening, life skills). Then each Tier II partner was responsible for mentoring numerous community or kebele-level entities. The scale and complexity of the program required fairly rigid guidelines. Tier II and III partners repeatedly emphasized the value of the PC3 approach to social mobilization for meeting the needs of OVC and their families. The approach affirms that resources within a community can be harnessed for the benefit of vulnerable children, and that external support must complement and extend, and not undermine, the local resources. While in general the methods employed are not unique to PC3, it used structured and systematic methods to organize and strengthen the care capacities of communities. A consistent commitment to mentoring (―walking alongside‖) local entities is a PC3 hallmark. Partners at all levels stated that flexibility and responsiveness to community context are central to the success of PC3. The tiered approach may have slowed the start-up phase within PC3 because Tier I partners had to reach agreements with Tier II partners before they could begin building the capacity of those partners, which in turn took time to start working with Tier III entities. This delayed the delivery of actual service to children, intensifying the pressure to meet targets. This was somewhat mitigated by the fact that Tier I partners had often worked previously with Tier II partners. Such partnerships are now well established and can be used for future programming. Criteria for setting targets were not established based on number of OVC, extent of existing community (versus organizational) capacity, cost per child served, location or environmental constraints, or other resources available (resource mapping was completed in mid 2007). Specific targets for capacity building, quality of services, and supportive context were not yet set by mid￾term of the project. Both PC3 and USAID/Ethiopia needed to help set or clarify such targets. The maturing of PC3 brought many reality checks; implementation has been adjusted to be more practical. At times Tier I partners had to step in to fill gaps in Tier II capacities. For example, in the Afar region where there are no Tier II partners, it was necessary to devise a mechanism for ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 21 subgranting directly to kebele-level groups. Core groups were established apart from but linked to CBO leadership committees. These core groups enhanced networking but created a new organizational structure with which people had to become comfortable. Training, coaching, and subgranting to Tier III partners are of a caliber and sophistication usually reserved for larger NGO partners. This strategy has paid off in that Tier III partners are delivering services at unprecedented levels, given the extent of services provided and numbers of children reached, and have the capacity to continue to do so as resources become available. In the original agreement 75 percent of funds were supposed to go to Tier II and Tier III organizations. This was later negotiated to 65 percent. Several Tier I organizations supplement administration costs from their own funds: Capacity-building activities and direct subgrants to Tier IIs are generally about 70 percent of the total for each Tier I. In turn, Tier IIs subgrant to Tier IIIs. In one sample exercise, it was found that a Tier II used 49 percent of its grant to make subgrants to Tier IIIs and 36 percent for capacity building. Perhaps the outstanding management and design feature of PC3 is that it has achieved the most extensively united OVC partner network of any African country supported by PEPFAR. More than 500 entities have made some form of continuing commitment to the PC3 approach. It takes a tremendously flexible and elastic program to nurture such an expansive array of talent and momentum. Many design and management growing pains had to be overcome, but the interviews with participating organizations indicate that community capacities have increased and are ready to evolve even more, independent of the three-layered structure. Phase-Out Strategy PC3 was intended to mobilize community responses to OVC through a tiered approach to capacity development. Once that response was in place, it was assumed that the project could decrease its funding because the gap would gradually be filled from community resources. The strategy and approach have been justified. Communities are taking considerable responsibility for providing services for children, as evidenced by the success of local fund￾raising and the number of children receiving assistance. However, a few gaps remain: Between the numbers of children requiring support and the numbers CBOs can actually serve Between the need and what can be provided with the resources available Between the resources a CBO can mobilize from its own members and other external sources and the deficit when a subgrant is discontinued. Partners at all levels described their growing capacity to raise funds and offer services, the growing numbers of families needing assistance, and the shortfall that would be created once PC3 subgrants are discontinued. Over the five-year lifespan of the program (Figure 4, x axis) funds from PC3 rose as local resources were mobilized. Local resources alone, however, cannot fully meet the needs of children and families (Figure 4, the red line); the gap between needs and local capacity to meet them is a growing concern to Tier III partners. Figure 4 illustrates how the situation was described. 22 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Figure 4: Perceived Gaps in PC3 0 2 4 6 8 10 12 Year 1 Year 2 Year 3 Year 4 Year 5 PC3 resources Needs Capacity The project has put in place methods to try to ensure that services for children are uninterrupted: • Community action cycle for community mobilization • Livelihood activities • Resource mapping and mobilization • Organizational and technical capacity building. There was evidence of all these at the sites visited; but the organizational maturity of Tier II and III partners, the high needs in some communities, the capacity limitations of government institutions that supply services, and the short time for actually developing local capacity have meant that organizations do not feel ready to completely phase out from their partners. The phase-out strategy varies depending on the partners. The strategy looked at all possible means for linking Tier II and Tier III partners with other sources of funds and with other projects within a Tier I partner. Among them: 1. When a Tier II partner is phased out, its Tier III partners can be transferred to another Tier II. 2. A Tier I partner can directly fund some Tier III partners whose Tier II partner is phased out. The need to meet targets also requires that partners reaching or able to reach many children be kept on while smaller partners would be transitioned out earlier. Already three Tier II partners have been phased out for contractual reasons, and 13 were given notice that they would be phased out in June 2008. Lessons Learned • Phase-out strategies are never popular, but a more specific graduation plan for partners, including identifying other sources of funding for them, might have given the phase-out a more supportive tone. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 23 It is not always possible to both rapidly expand coverage and mobilize local resources in a short time. A fundamental weakness in the design of the program is now fueling stress and fear among Tier II and III partners, especially the volunteers who regularly interact with beneficiaries. One problem is that the number of OVC receiving direct support from PC3 is supposed to increase as inputs from PC3 taper off. Short-Term Recommendations Every Tier I partner should list each Tier II and Tier III partner and delineate its current status and planned position after the phase-out. The APS should target Tier II and Tier III partners of PC3. Every effort should be made to have a smooth and seamless transition to a new project or grant. 24 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 25 IV. CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS PC3 is an ambitious and innovative approach to using community structures to reach large number of orphans and children affected by HIV and AIDS and their families. The program benefited particularly from a community network created by burial societies, iddirs, in Ethiopia. It also benefited from the inclusion of community schools, which bring established institutions into the partnership. PC3 was able to build on these strengths and infuse a holistic child-friendly essence into its community activities by building the capacity of local NGOs, which in turn partnered with a variety of CBOs. Reaching over 500 local organizations through 35 Ethiopian NGOs required a systematic and structured approach, with formats for monitoring and reporting. The approach included coaching and mentoring as well as financial support. Partners consistently reported satisfaction with the support they received, as did children and guardians. The program has been faced with an almost overwhelming volume of children and families with pressing needs for food, health care, and education. Inflation and food shortages, combined with the effects of HIV and AIDS, are driving up demand. Community organizations in PC3 have limited resources to respond to these demands and have had to strategize and stagger their support to families. They have also increasingly looked to income–generating, savings, and credit schemes to support families. Despite some success at mobilizing resources both locally and externally, with the ending of the PC3 program imminent, local NGOs and community partners fear they will not be able to sustain even the current level of services. Some continued injections of funding are needed to continue the momentum and commitment evidenced by PC3 partners. National, regional, and local level advocacy is also required to boost coordination with government health, education, and protective services and with FBOs. PC3 offers a model of how to reach community organizations and support their efforts to care for OVC. The model could be enhanced with stronger networks and policies and a longer time-frame. SHORT-TERM RECOMMENDATIONS Resources Increase the focus on mobilizing local resources to alleviate fears that some children will no longer be served as PC3 phases out. Continue assisting Tier II and III partners to team up to apply for resources. Use as examples of good practice for others the project sites that are currently working with a number of different agencies and partners to coordinate care for children. For example, ProPride in Dire Dawa has an extensive web of partnerships beyond HIV/AIDS programs. Tier II partners need to receive and then pass on technical assistance on how to make practical use of community mapping to relieve overburdened local partners and expand service. This would include an outline of next steps and guidance on action, especially relating to filling gaps as PC3 phases out. Consider allowing Tier III partners still within the project to use some of their subgrant to start an income-generating project in support of OVC, as did the schools under World Learning. 26 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Services An experienced economic growth team is needed to review current activities aimed at increasing household assets. Convene leaders from all three tiers of partners to strategize on how to navigate the current food crisis. USAID and other donors are intensifying their food aid. The PC3 network is well placed to make good use of these external inputs to meet emergency needs. Based on a review of current promising PC3 practices, identify the essential care components a community needs to provide comprehensive support to families or household affected by HIV/AIDS. Dissemination Expand upon communities of practice and exchange visits (both Tier II and III partners found these to be ―highly valuable‖) to include leadership forums where lessons learned and strategies can be exchanged. Explore actions needed to establish a centers-of-excellence approach to increase country, and even regional, exposure to Tier II and III processes and results. Document best practices in human capacity development that can be expanded for use by others working to support OVC (e.g., leadership exchanges or communities of practice for program management across Tier II partners). All manuals and guidelines should be translated and disseminated to Tier II partners. Set up a meeting with PC3 management, staff seconded to MOWA/HAPCO, and the two GOE members of the evaluation team to discuss ways to disseminate PC3 findings and approaches. RECOMMENDATIONS FOR THE TRANSITIONING OF PC3 It is critical that the momentum of, trust in, and capacity of local organizations to support OVC be maintained. This will entail continued support to many Tier III partners. Every effort should be made to help them make a smooth and seamless transition to a new project or grant. Articulate as soon as possible to all OVC partners the plans for maintaining the services and support to OVC currently covered by PC3. Partners at all levels, especially volunteers, are fearful of having to stop services to children until other options open up. All Tier I partners should produce a detailed assessment of the future of each Tier II and Tier III partner, stating whether or not the partner has been successful in obtaining other sources of funding, is linked to other support networks, and has linked children and families linked to quality service providers. This should guide the phase-out and transition period. The current APS allocation should not only target Tier III and Tier II partners but also look at widening the net with similar partners. The consultants met a few organizations, not PC3 partners, who are doing very similar work, including CSSGs, income-generating activities for orphans, food security, etc. Now that the approach, the materials, and a set of knowledgeable partners are available, it makes sense to expand the network. RECOMMENDATIONS FOR FUTURE PROJECT DESIGN A coordinated, unified program will still be essential, for a number of reasons: – It could offer focused, demand-driven, technical assistance. – It would harness the capacity of PC3 Tier I partners to deliver quality training and coaching, building on the good work of the past five years. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 27 – It would build on the core groups, allowing Tier III partners in the same locality to specialize—e.g., women lawyers, youth groups—while retaining a case management system. – A coordinated approach would strengthen advocacy, providing a single voice on children’s issues. – Managing numerous small APS grants will be extremely time-consuming and labor￾intensive for the USAID Mission. – An RFA that can incorporate a coordinated and harmonized approach would be extremely helpful. In purely financial terms the 200 or so Tier III partners that are not schools could each receive a US$4,000/year grant for a total of US$800,000. Adopt an overarching strategy that focuses on household vulnerabilities, rather than just the children, to keep parents alive and economically viable and children free from HIV. This requires identifying households before they reach the point of extreme vulnerability and giving more attention to parenting skills, economic strengthening, and case management or coordinated care. Consider ways to continue to use the massive reach and credibility of many Tier III partners. Iddirs and churches, for example, have members that may be target beneficiaries as well as possible service providers. Develop or adapt a parenting package that has components on the needs of vulnerable children (e.g., bereavement counseling, communication, and health education) that could be rolled out by iddirs and churches with their members. This would be a relatively simple way to scale up both awareness of OVC issues and support to caregivers/guardians and parents of OVC. Iddirs and churches would have some natural advantages in reaching parents—their members. If 18 percent of households are caring for an orphan (RAAAP, quoting DHS 2000) many iddir members could benefit from information on addressing the special needs of children so affected. PC3 used iddirs to reach children outside their membership, but they could also easily reach their own members. Expand engagement with FBOs. Consider consultation with the World Conference on Religions and Peace (accessible using the AIDSTAR mechanism and USAID/Washington central funds or through the New Partners Initiative of PEPFAR). (During a previous site visit, the Bishop of Nazeret agreed to convene a meeting of donors working in his area to better organize activities to benefit OVC and their families.) Define at the outset concrete benchmarks for achieving good enough community capacity. One might be that the amount of external inputs needed for direct service provision does not exceed 35 percent of the total cost, or that a committee with school, health, local government, and community representation functions in support of OVC. This will likely require agreement on what constitutes sufficient community capacity and the types and extent of capacity building necessary to achieve it. Partners could outline the critical minimum of skills, knowledge, and practices or standards needed for families and communities to provide care and support that mitigates the impact of HIV/AIDS on children. Reaching consensus on what is good enough capacity within communities and families will inform the setting of destination points for capacity building by civil society and government partners. The work to date on establishing OVC service standards can be used to inform a similar process for community and family capacity. Changes will probably be needed in the 100 percent reliance on volunteer structures at the kebele level, which is already overstretched. The health extension worker model could be expanded or adapted to give attention to supervising volunteers working with families affected by HIV/AIDS. 28 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Use the results from the OVC service standards pilot test to draw up costing scenarios that encompass community and external inputs needed to offset the impact of HIV/AIDS on households caring for children. Look at capacity development for government kebele and woreda-level institutions, both within and outside MOE and HAPCO. Capacity building would include, among other things, subgranting, community mobilization, understanding quality services, coordination, and networking. In a few regions, consider seconding someone to HAPCO for the specific purpose of making and monitoring grants to local CSOs. Pay more attention to policy development and reform and advocacy. Without adequate and united leadership, both government and civil society entities cannot realize their full potential. Greatly expand the life skills component so that every child who participates in a program receives life skills training, and, to the extent feasible, continue to ensure that non-OVC also participate, especially caregivers. Caregivers remarked on the value of the training they received (e.g., how to do a household budget) for improving their caring capacity. Reinforce links with education projects so that OVC issues are mainstreamed into collaboration with schools. Education programs should be scaled up through regional and woreda bureaus. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 29 APPENDIX A: SCOPE OF WORK USAID/Ethiopia President’s Emergency Program for AIDS Relief (PEPFAR) Draft Statement of Work (SOW) for End-of-Project Evaluation Positive Change: Children, Care, and Communities (PC3) Program (Revised: 24-Mar-2008) PROJECT IDENTIFICATION DATA 1. Project Title: Positive Change: Children, Care, and Communities (PC3) 2. Project Number: Cooperative Agreement No. 663-A-00-04-00433-00 3. Project Dates: Sept. 29, 2004 – Sept. 28, 2009 4. Project Funding: $20,000,000 over 5 years 5. Implementing Organization: Save the Children, USA 6. Cognizant Technical Officer (CTO): Catherine Hastings I. IDENTIFICATION OF THE TASK The USAID/Ethiopia (USAID/E) PEPFAR office requests technical assistance from the Global Health Technical Assistance Project (GH Tech) to design and implement an independent external end-of-project evaluation of the Positive Change: Children, Care, and Communities (PC3) Program. The two main PC3 project strategies are to (a) provide community-based care and support to orphans and other vulnerable children, and (b) increase the capacity of Ethiopian community-based organizations (CBOs) to provide care and support. This external end-of-project evaluation will determine the success and impact of these two main strategies. The USAID/E PEPFAR office requests that the in-country activities for this evaluation be completed by May 27, 2008, in order that the findings, conclusions, and recommendations can be used in the planned redesign of future programs for Orphans and Vulnerable Children (OVC). II. BACKGROUND USAID/E Response to HIV/AIDS: From 2004 through 2006 an estimated 288,000 Ethiopians died from HIV/AIDS-related causes. The 2007 federal estimate4 of national HIV prevalence is 2.1 percent; 7.7 percent in urban areas and much lower in rural areas at 0.9 percent (FHAPCO 2007). As of 2007 almost one million (977,000) Ethiopians were estimated to be living with HIV; there were almost 900,000 AIDS orphans (898,350) (FHAPCO 2007). The U.S. Mission to Ethiopia’s HIV/AIDS interagency team, composed of the Department of State, the Department of Defense, the U.S. Centers for Disease Control and Prevention, and the U.S. Agency for International Development (USAID), began integrated HIV/AIDS programming in 2004 under the oversight of the Office of the Global AIDS Coordinator. Peace Corps joined the PEPFAR team in early 2007. The U.S. Mission collaborates with a number of Ethiopian government agencies: the HIV/AIDS Prevention and Control Office, the Ministry of Health, the Ministry of Finance and Economic Development, the Ministry of Youth and Sports, the Ministry of Women Affairs, the Ministry of Education, and the Ministry of Labor and Social Affairs. 4 Federal HIV/AIDS Prevention and Control Office (FHAPCO), Single Point Estimates of HIV and OVC Indicators, April 5, 2007. 30 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) USAID responds to HIV/AIDS as part of PEPFAR in collaboration with the Ethiopian Government and numerous other partners. USAID supports prevention, care, and treatment activities with a combined FY06 program budget of over $122 million. USAID Integrated Strategic Plan FY 2004-2008 and the 2007 Foreign Assistance Framework: HIV/AIDS programs, including efforts to provide care and support for OVC, were initiated under the USAID/E Integrated Strategic Plan (ISP) for the period FY 2004 to FY 2008 under strategic objectives (SOs) SO 14: Human Capacity and Social Resiliency Increased and SO 14.2 HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased. In 2007 SO14 was incorporated into an alternate Foreign Assistance Framework (FAF) for the USAID 2007 Operation Plan. OVC programs now fit under the FAF Objective, Investing in People, Health Program Area, Program Element HIV/AIDS, Program Sub-Element Orphans and Vulnerable Children. Prior USAID Assistance to OVC in Ethiopia: USAID Mission-assisted care and support of OVC affected by HIV and AIDS has increased significantly from supporting 550 in FY 2002 to 11,506 in FY 2003 to over 273,000 as of September 2007. Overview of the Positive Change: Children, Care, and Communities Program: The umbrella Cooperative Agreement known as Positive Change: Children, Care, and Communities (PC3) was awarded in September 2004 to Save the Children/USA as prime recipient, with CARE International, Family Health International (FHI), Hope for African Children Initiative (HACI), World Learning International (WLI) and World Vision (WV) as key partners. The PC3 team addresses the needs of OVC affected and infected by HIV/AIDS in six regions of Ethiopia: Amhara, SNNPR, Oromia (including Addis Ababa), Afar, Dire Dawa, and Beneshangul. The goal of PC3 is to improve the well-being of 500,000 OVC and families affected by HIV/AIDS. To achieve this goal, the program addresses three key intermediate results (IRs): 1. Increased availability, quality, and consistency of community-based support services for OVC and families affected by HIV and AIDS; 2. Improved capacity of Ethiopian civil society organizations (CSOs) to plan, implement, monitor, evaluate, manage, and report on OVC programs and services; and 3. A more supportive environment for OVC and their households developed through strengthened coordination, networking, and advocacy. The original agreement required that 75 percent of the total funding pass to CBOs to ensure their capacity building. The project is set up in a tiered approach with Tier I international partners (Save, CARE, WL, FHI, and WV) providing technical assistance to 35 Tier II national level local nongovernmental organization (NGO) partners, who in turn subgrant to 500 local implementing partners, women’s associations, and CBOs (Tier III partners). The key service components that PC3 implements to support OVC and their families through a variety of community based mechanisms include: educational support, life skills, health and nutrition, psychosocial support, livelihood support, and protection and legal support. A list of key background reading materials is found below. PC3 Program Design: The PC3 program was initiated out of recognition that there was an urgent need to scale up care and support services for OVC on a sustainable basis. The major focus of the PC3 was to address the challenges Ethiopian communities face in assuring positive change among communities affected by HIV and AIDS. The strategy envisioned was two-pronged: to provide community-based care and support to orphans and other vulnerable children and, while so doing, to increase the capacity of Ethiopian NGOs, CSOs, CBOs, and faith-based organizations (FBOs) to provide such care and support over time. There are four program components: ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 31 Component One: Mobilization of CSOs (NGOs, CBOs, FBOs) for Community-Based Response, with particular emphasis on care and support of OVC Component Two: Capacity-Building of CSOs Engaged in Community-Based Response Component Three: Support to Networking Among CSOs Providing Community-Based Response, and Between Government and Civil Society Addressing the Needs of OVC Component Four: Monitoring and Evaluation of Community-Based Response Programs The program was designed to complement ongoing initiatives within Ethiopia focused on improving prevention, care, and support for OVC, including development of an OVC policy under the joint leadership of the Ministry of Labor and Social Affairs (MOLSA) and the HIV/AIDS Prevention and Control Office (HAPCO) through an OVC task force that was reinforced after a National Conference on OVC and Alternatives to Residential Care held in May 2003. Key integrating elements in the program are children, communities, empowerment, and positive change. The program design encourages the use of technical and management implementing strategies that foster empowerment of individual children and adults (regardless of HIV status), households, extended families, and communities in ways that promote positive change in addressing the HIV and AIDS crisis they are individually and collectively facing. Another key integrating element is gender: to provide funding to activities only when local leaders have demonstrated that women are proportionately represented to reflect the composition of the community to be served. Geographic Coverage: PC3 was designed to assure programmatic coverage in areas of high HIV and AIDS prevalence, and to complement existing PEPFAR programs. Relationship with Other USG-Funded Implementing Partners: PC3 was developed in the context of a wide-ranging program of USG assistance to Ethiopia to address HIV and AIDS. In addition to working closely with Ethiopian public and private sector partners individually and through various task forces, networks, coordinating committees, technical working groups (TWGs), and other agencies, PC3 is expected to collaborate effectively with the other USAID￾funded partners. Broad Program Parameters: The PC3 program was developed within a number of specific parameters. The program will be consonant with and further the objectives of Ethiopia's Sustainable Development and Poverty Reduction Programme (SDPRP) and National AIDS Strategy. The program will fully support implementation of the Ministry of Health's Health Sector Development Program (HSDP) and the Health Extension Package (HEP), as they relate to HIV/AIDS and as they evolve. The program will fully support implementation of the Ministry of Labor and Social Affairs policies and strategies on AIDS-affected OVC, as they evolve. The program was developed and will be implemented in close coordination with the PEPFAR Team in Ethiopia, in accordance with statutory requirements of H.R. 1298 and evolving guidance provided by the Department of State/Global AIDS Coordinator (S/GAC). The program will support achievement of USAID's new Integrated Strategic Plan, with particular attention to S014 and its results related to HIV/AIDS. 32 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) The program draws upon USAID's lessons learned in Ethiopia and elsewhere in community￾based care and support of OVC and PLWA and their families, and will support application of "best practices" in such care and support. Component One: Mobilization of NGOs and CSOs (CBOs and FBOs) for Community￾Based Response. The broad objective of the HIV/AIDS Access and Quality Component is to increase mobilization of Ethiopian CSOs (NGOs, CBOs, FBOs) for community-based response to the HIV/AIDS epidemic. PC3 is designed to provide subgrants and other assistance (technical, skills, informational) to CSOs operating in areas with high HIV and AIDS prevalence, particularly PEPFAR catchment areas, to help them increase the numbers of AIDS-affected orphans they are assisting. PC3 is expected to assist the USG and Ethiopian partners to meet PEPFAR and United Nations General Assembly Special Session (UNGASS) targets for care and support to OVC, and to a lesser extent the targets for non-ART care of PLWA. The targets for these groups are shown in Table I. Table 1. President's Emergency Plan: Ethiopia Targets for Care and Support, 2004-2008 Target Area 2004 2005 2006 2007 2008 Care and Support: OVC 10,000 84,000 153,000 276,000 500,000 Care and Support: Palliative 32,000 61,000 114,000 211,000 388,000 Care and Support: Non-ART Care 50,000 68,000 91,000 121,000 162,000 The PC3 program is the lead project in Ethiopia for achievement of the "Care and Support: OVC" targets in Table l and as such has been expected to achieve the annual targets indicated above. Cost Sharing: At the time the PC3 program was awarded, the USG estimated it would commit US$20 million for the 5-year program, and it was recognized that this was very little money given the extreme level of need in Ethiopia (e.g., 4 million orphans, of which 1 million were estimated to be affected by HIV and AIDS) and the PEPFAR targets above. A cost-sharing agreement of US$2 million was negotiated with the implementing partner. The PC3 is intended to be as innovative as possible in leveraging other funding, e.g., implementing partner contributions, on￾going OVC programs, community-generated, private commercial sector, foundations, other donors, etc. Program Emphasis and Structure: The intent of the PC3 program was not to increase the number of NGOs/CSOs working with OVC but rather to scale up community-based responses to the needs of OVC and to assure that more children are getting more services and protection. The PC3 program was designed to comprise a consortium or similar group of organizations, at least some of which have established programs and relationships with CSOs in Ethiopia. The PC3 program allocates a given proportion of resources to partner CSOs of consortium/group members, who in turn provide subagreements through a larger number of partner Ethiopian NGOs/CSOs. The PC3 program was also to allocate a given proportion of resources to a "New Partners Grant Fund" to respond to new partner proposals. As stated above, at least 75 percent of all CA funding was expected to be provided to and/or through Ethiopian NGOs/CSOs (including CBOs and FBOs) through consortium/group arrangements and/or the Small Grants Fund. The framework for different types of subagreements is presented in Figure 1. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 33 Figure 1: Framework for PC3 Agreement and Subagreements Up to 25 percent of funding PC3 Agreement with ―Recipient‖ (=1 or more lead partner organizations) Recipient￾Partner A Recipient￾Partner B Recipient￾Partner C Recipient￾Partner …Etc. New Partners Grant Fund No less than 75 percent of funding Subagreements w Ethiopian NGOs, CSOs (CBOs, FBOs, associations etc.) Component Two: Capacity-Building of CSOs Engaged in Community-Based Response. The broad objective of the CSO Capacity-Building Component is to expand the number and coverage of CSOs engaged in care and support to OVC, and to assure uniform quality of programs. More specifically, the PC3 program is expected to 1. provide technical assistance (TA) and training to CSOs already working in OVC programs, to assure that they are providing a uniform "fixed menu" of care and support; 2. provide organizational and administrative management TA and training to established CSOs, to help them expand their coverage as well as diversify their resource base for a future "non-project" situation; and 3. provide "readiness" TA and training to interested new partner CSOs, to help them develop capacity to undertake OVC programs themselves. The PC3 program is expected to assist the USG and Ethiopian partners to expand their technical, managerial capacity - to respond to the needs for care and support to 0VCs in Ethiopia, with particular attention to assuring minimum quality standards and program sustainability over time. Component Three: Support to Networking Among CSOs Providing Community-Based Response, and Between Government and Civil Society Addressing the Needs of OVC. The broad objective of the "Support to Networking" Component is to help operationalize new OVC policy, or, in the absence of a formal policy, commonly accepted norms and standards (e.g., consistency) for OVC care and support among key stakeholders in the regions in which PC3 works. PC3 is expected to collaborate with the regional Bureaus of Labor and Social Affairs (BOLSAs) and regional HAPCOs to assure that policy, norms, and standards are understood by government, private sector, and NGO partners working with 0VCs in the target regions. At the time PC3 began, there were several networks and working groups devoted to OVC in Addis Ababa. These include the OVC Network formed by Pact, Inc.; an OVC Task Force for Addis Ababa nurtured by FHI with USAID funding; a UNICEF-led OVC task force, with the SAVE Alliance and MOLSA; and others around the country. It was not clear if these groups shared a common definition of "OVC" or common norms and standards as to what constitutes valued "care and support." The UNICEF-led group, with MOLSA, appeared to be taking the lead in helping the Government of Ethiopia (GOE) establish an OVC policy, which was quite encouraging. Helping to operationalize OVC policy in Ethiopia's nine regions and two city regions was acknowledged to be a significant challenge. The intent of PC3 is not to form yet more networks but to strengthen those that exist and to increase the likelihood of them continuing over time. The intent is to develop common approaches to common problems and to assure consistently high quality of care throughout network members. Where indicated, the possibility of merging existing networks is to be explored. 34 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Component Four: Monitoring and Evaluation of Community-Based Response Programs. The broad objective of the Monitoring and Evaluation (M&E) Component is to monitor and report on the quantity and quality of USG-supported OVC programs in Ethiopia. More specific objectives are to 1. provide semi-annual reporting on required PEPFAR indicators as they evolve over time; 2. provide annual reporting on process; and 3. develop, evaluate, disseminate, and apply best practices and state of the art knowledge in the area of quality OVC programming in Ethiopia. PC3 is expected to undertake data collection and verification strategies that ensure reliability and accuracy of progress toward expected accomplishments. PC3 is strongly encouraged to collaborate in monitoring efforts with other PEPFAR colleagues, the GOE, and other donor/partner programs, to assure that monitoring and evaluation systems are as cost-effective as possible. Indicators currently reported by PC3 include the following PEPFAR indicators (broken down by gender) Number of OVC served by OVC programs; Number of OVC who received primary direct support (a subset of Number of OVC served) [3 or more services]; and Number of OVC who received supplemental direct support (a subset of Number of OVC served) [1 or 2 services, either PEPFAR-funded or leveraged]. PC3 is expected to revise and modify its monitoring and evaluation (M&E) systems to harmonize with PEPFAR OVC guidelines. PC3 has recently updated its systems to accommodate new PEPFAR OVC guidelines, which classify the count of OVC receiving care and support as primary direct and supplemental direct beneficiaries and require information on the direct support given to providers/caregivers and the services indirectly benefiting OVC through their caregivers or through system strengthening activities at a higher level. III. PURPOSE OF THE ASSIGNMENT USAID/E requires a team of three consultants to conduct an end-of-project evaluation of the PC3 project. With less than two years remaining in the project, this evaluation will collect information about PC3 implementation, progress, and challenges. It will formulate recommendations for follow-on programs. The evaluation will cover the PC3 program performance period of September 2004 to December 2007. The evaluation report will help USAID and Save the Children address topics of management, quality of services, and sustainability. The evaluation will answer the following illustrative questions: Program Management Has the three-tiered structure succeeded in providing appropriate and high quality services as well as effective program management? If so, how? Service Delivery Has the PC3 project demonstrated significant measurable success in increasing the availability, quality, and consistency of community-based care and support to OVC? If so, how? ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 35 Has the program met the set targets while avoiding doubling-counting and/or undercounting? Is there any duplication of effort in the services provided to individual children? If so, how? Supportive Environment for OVC and Their Households Has the PC3 project demonstrated significant measurable success in creating a more supportive environment for OVC and their households through strengthened coordination, networking, and advocacy? If so, how? Quality of Care and Services Is the PC3 program implementing best practices, as defined by the draft OVC Standards of Service in Ethiopia and other international standards documents, in service delivery and community mobilization? If so, how? Capacity Building and Sustainability Has PC3 provided capacity building to Tier II and Tier III partners? If so, how? Has PC3 demonstrated significant measurable success in improving the capacity of Ethiopian CSOs to plan, implement, monitor, evaluate, manage, and report on OVC programs and services? If so, how? Has PC3 developed a plausible exit strategy that provides assurance that the Tier II and Tier III partners will be able to continue providing OVC services after the project ends? If so, describe the exit strategy. Impact Has the project demonstrated significant measurable impact on the quality of life for OVC and their households that can be attributed to PC3 community-based care and support to OVC? If so, how? Has PC3 affected the national OVC policy environment? If so, how? Monitoring and Evaluation Is the monitoring and evaluation (M&E) system functioning properly as designed? In particular, are the new database, data collection, and data quality systems now able to report on the new OVC indicators? If so, how? Is the M&E system able to measure progress toward set targets while avoiding doubling￾counting and/or undercounting? If so, how? Lessons Learned What are recommendations for improving the program and for new follow-on program development? Identify lessons learned, successful interventions that merit continuation or replication, better practices, and significant products and tools from the PC3 program for possible dissemination and replication. IV. EVALUATION METHODS The evaluation will be carried out by a core team of three independent, external consultants over a three-week period through multiple methods, including key informant interviews, field observation, and a review of PC3 M&E reports, tools, and materials. One or more USAID staff and three or more GOE representatives may join the evaluation team during the team planning meetings and in briefing, site visits, debriefings, and report preparation. PC3 Tier I partners will 36 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) accompany the team on site visits as appropriate but will not be present during interviews with Tier II and III partners, stakeholders, or beneficiaries. In view of the large number of CBOs (some 500), the evaluation team will need to develop a valid sampling scheme to identify a small but representative subset of CBOs by region. Interviews will include the following: USAID Mission staff, including the HIV/AIDS Team and staff from the Office of Financial Management (OFM) Save the Children USA and subagreement holders - Family Health International, World Vision, Care International, and World Learning in-country staff Tier II and III partners (for example, local iddirs, Mekedim, etc.) Government of Ethiopia representatives: regional HIV/AIDS Prevention and Control Organization (HAPCO), Kebele leaders, Regional Health Bureaus (Ministry of Health), and Woredas Beneficiaries (both children and caregivers) Other PEPFAR partners. V. INFORMATION SOURCES Consultants will be provided the following background documents in preparation for the assignment: PC3 Cooperative Agreement, including modifications PC3 PEPFAR Semi-Annual Report submissions PC3 2005, 2006 and 2007 Annual Reports PC3 Quarterly Reports PC3 Volunteer Guidelines PC3 M&E Tools USAID trip reports summarizing past field visits to PC3 GOE Road Map for HIV/AIDS Prevention, Care, and Treatment Draft OVC Standards of Service for Ethiopia VI. TASKS TO BE ACCOMPLISHED Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task (see Attachment 1: Level of Effort Timeline and Attachment 2: Planning Calendar for the exact schedule). Review background documents/develop evaluation methodology/ and complete field visit and interview schedule in consultation with CTO and Evaluation Coordinator (approximately one month prior to departure) 3 days Travel for international consultants 2 days Team Leader advance planning in country 3 days Participate in team planning meeting and in-briefing with USAID/E HIV/AIDS technical staff 2 days Two teams conduct field visits and interviews 6 days ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 37 Field visit teams meet in Addis to synthesize findings 2 days Core team PC3 stakeholder interviews in Addis, prepare debrief 2 days Conduct debriefings for USAID and PC3 (separately) 1 day Draft and submit report to USAID/E in-country 5 days Travel for international consultants 2 days Finalize report; Team leader incorporates Mission comments and submits report electronically to CTO 4 days Total LOE: 33 days of LOE for Team Leader and up to 21 days for other team members, not including four travel days each for the Team Leader. A six-day work week is authorized for work in Ethiopia. VII. TEAM COMPOSITION AND PARTICIPATION USAID/E seeks three consultants: a Team Leader with experience evaluating USAID health programs, an OVC Specialist, and a local Evaluation Logistics Assistant. One or more USAID staff and three or more GOE representatives may join the evaluation team during team planning meetings, site visits, debriefings, and report preparation. PC3 Tier I partners will accompany the team on site visits as appropriate but will not be present during interviews with the Tier II and III partners, stakeholders, or beneficiaries. 1. The Team Leader will be an international consultant with extensive PEPFAR program implementation and evaluation experience. S/he will agree to fulfill his/her responsibilities in over three weeks, spending two weeks in-country, and will play a central role in guiding the evaluation process. The consultant will hold conference calls with core team members and USAID/E representatives before and after the visit to Ethiopia, in-brief USAID/E on arrival, debrief USAID/E and PC3 on evaluation findings, and produce a draft report to be left with USAID/E prior to departure, followed by a final report for USAID/E. The Team Leader will: Finalize and negotiate with client the team work plan for the assignment. Establish assignment roles, responsibilities, and tasks for each team member. Ensure that the logistics arrangements in the field are complete. Facilitate the Team Planning Meeting or work with a facilitator to set the agenda and other elements of the TPM Take the lead on preparing, coordinating team member input, submitting, revising, and finalizing the assignment report. Manage the process of report writing. Manage team coordination meetings in the field. Coordinate the workflow and tasks and ensure that team members are working to schedule. Ensure that team field logistics are arranged (e.g., administrative/clerical support is engaged, payment is made for services, car/driver hire or other travel and transport is arranged, etc.) Consultant qualifications: An advanced degree (PhD, MA, MS, or MBA) from a reputable accredited institution in any of the social sciences pertinent to work with OVC. 38 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Minimum 10 years of progressively responsible experience with recognized organization(s) in the design, implementation, and evaluation of OVC programs, with demonstrated technical expertise and skills in HIV/AIDS. Demonstrated strong analytical, managerial, and writing skills. Exceptional leadership in coordinating, assigning the team with appropriate responsibilities, communication, and interpersonal skills is absolutely critical. Ability to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. Must be fluent in English and have proven ability to communicate clearly, concisely, and effectively both orally and in writing. Must be able to produce a succinct quality document that gives direction and facilitates improvement for the PC3 OVC program. 2. The OVC Specialist will be an international consultant with extensive OVC implementation and evaluation experience in Africa. Knowledge of HIV/AIDS programming and PEPFAR is essential. The consultant will be responsible for writing some sections of the report. The consultant will assist the Team Leader in the development of any qualitative instruments to be used during site visits as well as the analysis of any data collected. Consultant qualifications: MA, MS, MBA or BA from a reputable accredited institution in any of the social sciences pertinent to working with OVC. Minimum 6 years of progressively responsible experience with recognized organization(s) in the design, implementation, and evaluation of OVC programs with demonstrated technical expertise and skills in HIV/AIDS in sub-Saharan African countries. Demonstrated strong analytical, managerial, and writing skills. Able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. Must be fluent in English. Proven ability to communicate clearly, concisely, and effectively both orally and in writing. 3. The Evaluation Logistics Assistant will be a local consultant, preferably fluent in Amharic, with a demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multitask, work well in stressful environments, and perform tasks independently with minimal supervision; ability to work collaboratively with a range of professional counterparts at all levels. The Evaluation Logistics Assistant will be responsible for logistics, coordination, and administrative support and ensuring all aspects of the evaluation are carried out seamlessly. He/She will assist the Team Leader and the implementing agencies in facilitating meetings, coordinating logistics, and organizing site visits. The Evaluation Logistics Assistant will collect and disseminate background documentation to the evaluation team. Consultant qualifications: MA, MS, MBA, or BA. Four years of work experience may be substituted for the degree. Minimum 6 years of progressively responsible experience within GOE and/or NGO work settings handling complex logistics, such as coordinating business travel and meetings. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 39 Demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multitask, work well in stressful environments, and perform tasks independently with minimal supervision. Ability to work collaboratively with a range of professional counterparts at all levels, including those from host country governmental and nongovernmental organization, U.S. Government agencies, and other donors. Capacity for effective time management and flexibility. Must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. Must be fluent in English and preferably Amharic. Proven ability to communicate clearly, concisely, and effectively both orally and in writing. VIII. SCHEDULE AND LOGISTICS The in-country phase of the evaluation will be conducted over a period of up to 24 days with a desired start date on or about May 4 (See Attachment 1: Level of Effort Timeline and Attachment 2: Planning Calendar for the exact schedule). The Evaluation Logistics Assistant, in collaboration with the USAID/E Evaluation Coordinator and Save the Children, will arrange all the partner meetings, site visits, and debriefings in advance. Meeting space will be provided at USAID/E, but the agency cannot provide access to fax and email. All associated travel and per diem costs for non-USAID staff will be covered by GH Tech under the technical directive with USAID/E. Time Line 3 Days Review of background reading materials. Develop evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO and Evaluation Coordinator (by March 28, 2008, one month prior to departure) 2 Days Travel - En route to Ethiopia. 3 Days Team Leader in-country for advance planning. 2 Days Team planning meeting; in-briefing with USAID/E. 6 Days Two teams conduct interviews and field visits outside of Addis Ababa. 2 Days Two teams meet in Addis to synthesize and write up findings. 2 Days Interviews in Addis, core team begins write-up of field observations, and compiles notes from interviews. Prepares debrief of USG and PC3 partners (separately). 1 Day Debrief USAID/Ethiopia (am) and PC3 partners (pm) 5 Days Draft and submit report USAID/E. 2 Days Travel - Departure from Ethiopia 10 Days USAID/E review of draft report. 4 Days Final report completed by Team Leader and delivered to USAID/E 40 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) IX. PERIOD OF PERFORMANCE Work is to be carried out over a period of approximately eight to nine weeks, beginning on or about (o/a) March 25, 2008, and concluding o/a May 27, 2008. This does not include the additional time for USAID/Ethiopia to review the draft report (10 days), final draft of the report by team leader (4 days) and final editing of the report by GH Tech (three to four weeks). X. FINANCIAL PLAN A budget plan agreement between the USAID/E PEPFAR and GH Tech will be reached, and USAID/E will process a MAARD to transfer funding for the evaluation activity into the GH Tech Indefinite Quantity Contract (IQC). XI. DELIVERABLES Four weeks prior to arrival: Team Leader will develop an evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO and USAID/E Evaluation Coordinator. Three days after Team Leader arrival: Team meeting and in-briefing with USAID/E. USAID/E HIV/AIDS technical staff to review and comment on evaluation methods. Prior to departure: Team makes presentation to USG PEPFAR staff and a separate presentation to PC3 partners, and submits a draft report, in the exact format specified by the USAID/E Evaluation coordinator (see separate MS Word file for GH Tech Evaluation Report Guidelines), to USAID/E CTO—two hard copies and one electronic copy on CD ROM or flash drive. After departure: Team Leader submits final unedited content to USAID/E within one week of receiving comments from USAID/E. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, and Recommendations in English in the exact format specified by the USAID/E Evaluation Coordinator. Upon final approval of the content by USAID/E, GH Tech will have the report edited and formatted. This process takes approximately 3-4 weeks. The final report will be submitted electronically to USAID/E CTO and Contract Officer. GH Tech makes the results of its evaluations public on the Development Experience Clearinghouse and on its project web site unless there is a compelling reason (such as procurement sensitivities) to keep the document internal. Therefore, GH Tech will request USAID/E confirmation that it will be acceptable to make this document publicly available. If there are certain restrictions regarding specific parts of the report that should be removed from a public version due to procurement-sensitive information, GH Tech is able to produce a second version suitable for public availability. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 41 Attachment 1: SCUSA PC3 End of Project Level of Effort Evaluation Timeline Draft 2 - March 24, 2008 Activity Team Member(s) Total Team Days Period of Performance Supporting documentation received from SCUSA PC3. Mission sends background documents to evaluation team members. SCUSA PC3 and USAID/E Evaluation Coordinator 1 Day 25 March Team Leader review of documents; determination of site visit schedule and evaluation methods Team Leader, USAID/E in collaboration with SCUSA PC3/implementing partners 3 days 26-28 March Team Leader prep/pre-evaluation informational interviews with stakeholders Team Leader 3 days for planning 4-7 May USAID/E in-brief and TPM in country Full team 2 days 8-9 May Fieldwork Two sub-teams in four regions in collaboration with SCUSA PC3 4 to 5 days 10-15 May Information analysis and synthesis Two sub-teams 2 days for 2 sub-teams in Addis 16-17 May Meetings and interviews with key stakeholders/prep for briefing, begin draft report Team Leader and some team members 2 Days 19-20 May Oral debriefing of Mission staff Team Leader and team members Morning 21 May Stakeholders presentation Team Leader and some team members Afternoon 21 May 42 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Attachment 2: Planning Calendar for PC3 Evaluation: Contingent on Team Availability etc. Draft 2 March 24, 2008 Monday Tuesday Wed Thurs Fri Sat Sunday 28 April 29 30 1 May Day Holiday 2 3 4 Sunday Rest PC3 DDYates arrives in Addis. 5 Victory Day PC3-DD Yates Planning/ Renee Demarco arrives in Addis 6 PC3-DD Yates /Renee Demarco planning 7 PC3-DD Yates/Renee Demarco planning 8 PC3 TPM and USAID IN -BRIEF 9 PC3 TPM and final prep for site visits 10 PC3 TPM team travel day 11 Sunday PC3 Rest and team travel day 12 PC3 two teams in field 13 PC3 two teams in field 14 PC3 two teams in field 15 PC3 two teams in field; return to Addis in PM. 16 PC3 Team synthesis Renee departs in late pm. 17 PC3 Team synthesis 18 Sunday Rest 19 PC3 Stakeholder interviews in Addis 20 PC3 Stakeholder interviews in Addis/prep for debriefing 21 PC3 DEBRIEF USAID am DEBRIEF stakeholders pm 22 PC3 Team Leader report prep 23 PC3 Team leader report prep 24 PC3 Team leader report prep 25 Sunday Rest 26 PC3 Team leader report prep 27 PC3 Team leader report prep/delivers draft report/ departs 28 29 30 31 1 June Sunday ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 43 APPENDIX B: PEOPLE INTERVIEWED CARE Dawn Wadlow Program Director Dr. Assefa Amenu HIV/AIDS Program Coordinator Genet Kebede Project Officer Adam Tekeste Livelihood Technical Specialist Feven Tassew HIV/AIDS Program Manager Developing the Family Together Kidest Belete Executive Director Family Health International Dawit Abraham M&E Officer Channe Addisu M&E Officer Ministry of Women’s Affairs Dr. Bulti Gutema Women’s and Children’s Affairs Department Head Regional HAPCO Lemlem Bezabih Dire Dawa HAPCO Head Save the Children USA Margaret Schuler Country Director Samson Radeny Chief of Party Betelhem Tafese Regional Coordinator Alemtsehaye Greiling CTC Manager Dereje Shiferlaw Acting Sub-Office Manager Solomon Woide Tsadik Psychosocial Specialist Asayehegn Tekeste Community Mobilization Specialist Alemseged Gebru OVC Technical Coordinator Tigist Hailu OVC Technical Coordinator Kassaw Asmare OVC Technical Coordinator Wondwossen Hailu PC3 Project Manager Amano Erbo M&E Officer UNICEF Alessandro Conticini Senior Chief of Adolescent Development and HIV/AIDS USAID Sam Clark HIV/AIDS Evaluation Coordinator Catherine Hastings Cognizant Technical Officer PC3 Melissa Jones HIV/AIDS Director Yegomawork Gossaye OVC Technical Advisor Befekadu Gebretsadik Education Technical Advisor Meri Sinitt Office Chief, Health, AIDS, Population, Nutrition 44 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) World Food Programme Dr. Meherete-Selassie Menbese HIV/AIDS Team Leader World Learning Robert Gurevich Chief of Party Firew Ayalew M&E Specialist Aragaw Biru School Support Director Abrha Gebretsadik Finance Officer PC3 Adanech Kebede Deputy Country Director for Finance and Administration Tonja Joma Awasa Officer Tewodros Tilahum Awasa Officer World Vision Martha Rezene PC3 Manager Feben Demissie PC3 Coordinator Alemayehu Tadesse M&E Officer CHILDREN’S AND GUARDIANS FOCUS GROUPS Children Guardians Male Female Male Female Addis 2 6 6 Adama Boset School Anti-AIDS Club 4 4 7 Awassa 5 3 2 10 Ras Desta School Yirgalem 3 9 Dire Dawa–Hirna 4 8 1 14 Bahir Dar (2 groups) 4 12 11 Total: 22 42 3 48 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 45 TIER II PARTNERS NO. NAME Sex ORGANIZATION 1 Dawitn Getenc M Beza Lehiwot 2 Mickael Araya M Beza Lehiwot 3 Yared Gimikael M Beza Lehiwot 4 Macedeu Tilahun F Beza Lehiwot 5 Degetu Asfew F Medan ACTS (Awasa) 6 Abise Gudeta F Medan ACTS (Awasa) 7 Musfin Makrria M Medan ACTS 8 Sgashe Domise F Mulu-Wongel 9 Tewedros M Mulu-Wongel 10 Aberra Wandimu M Hope for Children 11 Girma Aberra M Hope for Children 12 Yewoinshet Meshasha F Hope for Children 13 Woldesenbet Emagnew M Hope for Children 14 Gobena Soboka M Mekdim 15 Teshome Lelisa M Mekdim 16 Addis Arogew M Mekdim 17 Shiferaw Regessa M Mekdim 18 Ibrahim M Mekdim 19 Tewodros Kossahun M Ossa Bahir Dar 20 Sekkur Nuru M Ossa Bahir Dar 21 Getachew Tilahun M Ossa Bahir Dar 22 Zemene Mangistu M Ossa Bahir Dar 23 Abraham Tura M Propride Dire Dawa 24 Kedir Aliye M Propride Dire Dawa 25 Faferi Abera M Propride Dire Dawa 26 Gezehagen Ay M Propride Dire Dawa 27 Alemseged Gebru M Propride Dire Dawa 28 Moltotal Makoria M Propride Dire Dawa 29 Tewabech Tesfalem F 46 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) TIER III PARTNERS NO. NAME SEX ORGANIZATION 1 Adisu Malke Mhret (MWH 201 Φ∩ ) 2 Akale Lessanwork Yimam (MWH 201 Φ∩ ) 3 Ato Mengesha Assfa 1.05 Fet no. Derosh Iddir 4 Ato Lema Ybsa 1.05 Fet no. Derosh Iddir 5 Ato Alemayehu Abera 1.06 Iddir 6 Ato Abraham Alaro 1.06 Iddir 7 Tsidu Lema 1-04 Iddir 8 Abu Sabure 1-04 Iddir 9 Getachew Kehede 2-03 Iddir 10 Kallay G.Egziabher 2-03 Iddir 11 Fikrlub Adii M Adama Boset School 12 Aster Gosa F Adama Boset School 13 Tefera Godaa M Adama Boset School 14 Lishan Kibret F Adama Boset School 15 Fiabrtee Sogoye F Adama Boset School 16 Teyerawork Getehun F Adama Boset School 17 Adenec Abebe F Adama Boset School 18 Genet Tedesse F Adama Boset School 19 Mekdes Ayelew F Adama Boset School 20 Arabu Abdul Mijil M Adama Boset School 21 Kelemua Bekele F Adama Boset School 22 Hailu Mulat M Adama iddir umbrella 23 Wandimu Tola M Adama iddir umbrella 24 Abebe Beleke M Adama iddir umbrella 25 Eshetu W/Silasse M Adama iddir umbrella 26 Siae Seboka M Adama iddir umbrella 27 Woldemariam Kore M Adama iddir umbrella 28 Tekebe Hailamoriam M Adama iddir umbrella 29 Makinea Maleku M Adama iddir umbrella 30 Gezehayn Mengesha M Adama iddir umbrella 31 Kassahun Kebede M Adama iddir umbrella 32 Demeku Asres Adis Zemen School 33 Cap. Asefa Mano Alamura Iddir 34 Getachew Demissie Awassa Tabor Kale Hiwot Charch 35 Algenesh W/Semayat F Awassa Tobor School 36 Kossu Tseyon M Awassa Tobor School 37 Teshale Desta M Awassa Tobor School ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 47 38 Zerihun Mangistu M Awassa Tobor School 39 Habtamu Debebe M Awassa Tobor School 40 Tamenesh Leka F Awassa Tobor School 41 Atinefa Tilahun M Awassa Tobor School 42 Faritu Shinko M Awassa Tobor School 43 Godane W. M Awassa Tobor School 44 Tamiru Massa M Awassa Tobor School 45 Mufid Kedrola M Awassa Tobor School 46 Amelework WG F Awassa Tobor School 47 Mesay W/Yohanis M Awassa Tobor School 48 Yeshodule Yilmu M Awassa Tobor School 49 Tiruwork Belayneh Bahil Adarash School 50 Mekedes Mesfin Berhan Gozu 51 Sentayehu Lemenh Betekehnt Primary School 52 Abezash Tebege Betekehnt Primary School 53 Fekadu Telia Dato Odake Primary School 54 Zeleka Taddessa M Dilla Mehal Arade 55 Bogale Hade M Dilla Mehal Arade 56 Mathiyos Mule M Dilla Mehal Arade 57 Salamon Fayisa M Dilla Mehal Arade 58 Msfin Nake M Dilla Ms Iddir 59 Tedesse Fisaha M Dilla Ms Iddir 60 Mule Shew Bekele F Dilla Ms Iddir 61 Bogalah Goda F Dilla Ms Iddir 62 Emabet Assefa F Dilla Ms Iddir 63 Hayimerot Sheferaw F Dilla Ms Iddir 64 Muuken Kabe M Dilla Ms Iddir 65 Fekera Gola M Dilla Ms Iddir 66 Byorgishu Dderaro F Dilla Ms Iddir 67 Abebe Biftu Edeget Fana School 68 Fkre Bonte M EMW ALDO 69 Nasfub Ergate M EMW ALDO 70 Yifraf Abeba F EMW ALDO 71 Daniel Wolde M EMW ALDO Medin 72 Diriba Kebede M EMW ALDS 73 Smuele Ehyes ERHC (Awosso Med) ECD Eskele 74 Lemlem Haile Gimbi Genet 75 Zinesh Tsegaye F Gulele Subcity Educat. Association 76 Hana Asseta F Gulele Subcity Educat. Association 77 Etaferaw Anderge F Gulele Subcity Educat. Association 48 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 78 Tsehayi Haile F Gulele Subcity Educat. Association 79 Samuel Mola Higher / Kebele 01 80 Mekebo-Melayo Iddir n1-03 81 Z/Gulilat Bemablena Iddir n1-03 82 Ato Asefa Kumoli Iddir n1-03 93 Ato Abere Era Iddir n1-03 84 Sisay Beshah K. / 1-02 Kebele 85 Belay Beyene K. / 1-02 Kebele 86 Alemu Kebede M Kebele 20 m. Iddir 87 Asha Ali M Kebele 20 m. Iddir 88 Tolasa Hunde M Kebele 20 m. Iddir 89 Kebebe Yimamu M Kebele 20 m. Iddir 90 Birke Bekele F Kebele 20 m. Iddir 91 Haregegn teshome M Kebele 20 m.Iddir 92 Kenkiden Arayya M Kebele 20 m.Iddir 93 Mohamed Aryin M Kebele 20 m.Iddir 94 Teshoma Zeleka M Kebele 20 m.Iddir 95 Fotelh Yufina F Kebele 20 m.Iddir 96 Tigest Beyena F Kebele 20 m.Iddir 97 Desta Tedesse M Kebele 20 m.Iddir 98 Albsi Mitike F Kebele 20 m.Iddir 99 Hailu G/Madin M Kebele 20 m.Iddir 100 Solomon Asgedom M Kebele 20 m.Iddir 101 Alemu Kebede M Kebele 20 m.Iddir 102 Simegn Mokonnen F Kebele 20 m.Iddir 103 Birke Kebede F Kebele 20 m.Iddir 104 Solomon Mengesha M Kebele 20 m.Iddir 105 Dereshey Makuria M Kebele 20 m.Iddir 106 Alem Abebe F Kebele 20 m.Iddir 107 Ahied Mohimad M Kebele 20 m.Iddir 108 Amelework Zeleke F Kebele 20 m.Iddir 109 Gemada Benya M Kebele 20 m.Iddir 110 Tigist Abera F Kebele 20 m.Iddir 111 Nazit Tamal M Kebele 20 m.Iddir 112 Mesfin Zewde Madan Tesfa 113 Fetene Negash M Magbare H/HBCS 114 Mekonnen Abebe M Magbare H/HBCS 115 Kosahun Admasu M Magbare H/HBCS 116 Aklilu Abuye Nigat Cokabel School 117 Zerefensh Zergaw School ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 49 118 Simagent Amselu F Sefere Selam iddir 119 Abrham Ergete M Sefere Selam iddir 120 Habatu Kibur M Sefere Selam iddir 121 Emenesh Azene F Sefere Selam iddir 122 Marie Destew F Sefere Selam iddir 123 Yenot Gebire F Sefere Selam iddir 124 Agerie Mokonin F Sefere Selam iddir 125 Yamirot Semegnew F Sefere Selam iddir 126 Elsebet Alamnew F Sefere Selam iddir 127 Ashew Shome F Sefere Selam iddir 128 Amsalu Mazaw M Sefere Selam m.Iddir 129 Muche Senshaw M Sefere Selam m.Iddir 130 Yenew Warkench M Sefere Selam m.Iddir 131 Beress Assege M Sefere Selam m.Iddir 132 Fatima Jebrel M Sefere Selam m.Iddir 133 Ereyew Bekele M Sefere Selam m.Iddir 134 Mamiru Ayelew M Sefere Selam m.Iddir 135 Tembria Ekkonnen M Sefere Selam m.Iddir 136 Getechew Ayele M Sefere Selam m.Iddir 137 Haile Yesus Wondimagon M Sefere Selam m.Iddir 138 Haile Yesus Wondimagon M Sefere Selam m.Iddir 139 Hailemoriam Mokonen M Sefere Selam m.Iddir 140 Ali Abdi M Sefere Selam m.Iddir 141 Gedere Genet M Sefere Selam m.Iddir 142 Atsede Sirak F Sefere Selam m.Iddir 143 Bankie Berhanu M Sefere Selam m.Iddir 144 Kasahun Admasu M Sefere Selam m.Iddir 145 Ayelew Mokonen M Sefere Selam m.Iddir 146 Amsalu Mazew M Sefere Selam m.Iddir 146 Kumelachow Endalu M Sefere Selam m.Iddir 147 Anerelash Zeru F Sefere Selam m.Iddir 148 Beyilu Woberoh M Sefere Selam m.Iddir 149 Selam Ayelcrch F Sefere Selam m.Iddir 150 Adagna Hongew F Sefere Selam m.Iddir 151 Shimalis Derneshe M Shashamenen Yonata Assoc. 152 Zenabu Bogala M Shashamenen Yonata Assoc. 153 Roza Bazebin F Shashamenen Yonata Assoc. 154 Niguse Gebeyo M Shashamenen Yonata Assoc. 155 Tirunesh Felek F Shashamenen Yonata Assoc. 256 Fikre Wanja M Shashamenen Yonata Assoc. 50 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 157 Selemon Assefa M Tulo Dire Dawa 158 Selemon G/Selam M Tulo Dire Dawa 169 Tesfaye Werku M Tulo Dire Dawa 160 Wendimegeifn Hashalo M Tulo Dire Dawa 161 Abdi Yesuf M Tulo Dire Dawa 162 Mehomed Kelir M Tulo Dire Dawa 163 Muktor Mume M Tulo Dire Dawa 164 Alemeyo Worku M Tulo Dire Dawa 165 Teshome Demeka M Tulo Dire Dawa 166 Aberesh Habte F Tulo Dire Dawa 167 Adis Hayimoanot F Tulo Dire Dawa 168 Hasen Abdela M Tulo Dire Dawa 169 Amee Adem M Tulo Dire Dawa 170 Shimeles Aschelew M Tulo Dire Dawa 171 Abdela Abraham M Tulo Dire Dawa 172 Yesuf Musa M Tulo Dire Dawa 173 Halima Yosuf F Tulo Dire Dawa 174 Emobet Assefa F Tulo Dire Dawa 175 Haymona Abdo F Tulo Dire Dawa 176 Azuza Mahamad F Tulo Dire Dawa 177 Kunuza Amere M Tulo Dire Dawa 178 Hisra Idris F Tulo Dire Dawa 179 Eshetu G/Sellaise M Tulo Dire Dawa 180 Mohamed Tsmok M Tulo Dire Dawa 181 Abebe Beleke M Tulo Dire Dawa 182 Saro Seroch M Tulo Dire Dawa 183 Hidriya Yosuf M Tulo Dire Dawa 184 Ephrem Reta M Walenchiti Anti-AIDS Club 185 Sirtayehu Geshaw M Walenchiti Anti-AIDS Club 186 Legesse Imirot M Walenchiti Anti-AIDS Club 187 Boyush Taddesse F Walenchiti Anti-AIDS Club 188 Zelelem Birhanu M Walenchiti Anti-AIDS Club 189 Fetene Negash M Walenchiti Anti-AIDS Club 190 Asamirew Ayelew M Walenchiti Anti-AIDS Club 191 Yononis Tesema M Walenchiti Anti-AIDS Club 192 Negesh Hussier F Walenchti Anti-AIDS Assoc. 193 Alemework Biratu M Walenchti Anti-AIDS Assoc. 194 Fantu Zenab F Walenchti Anti-AIDS Assoc. 195 Bogalech Euta M Walenchti Anti-AIDS Assoc. 196 Abebe Fontaye M Walenchti Anti-AIDS Assoc. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 51 APPENDIX C: SCHEDULE Calendar for PC3 Evaluation 2008 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 28 April 29 30 1 May Day Holiday 2 3 4 Sunday May 5 Victory Day PC3 – Arrival 6 PC3 DD Yates/ Renee Demarco/ Kuleni Berhanu Chalachew Tiruneh, Ayanssa Gonfa Team Planning Meeting 8:30 – 16:30 7 PC3 Team Planning Meeting 8:30–12:30 USAID in￾brief 14:00–16:00 Meet with SCUSA Director and PC3 Director 8 PC3 Team Planning Meeting 8:30–10:30 Meet with SCUSA 10:30–12:30 Group Meeting with Tier I Partners (Program Managers) 14:00–16:00 9 PC3 am – Combined team site visits in Addis pm – Debrief on-site visit 10 PC3 am – Teams depart for site visits (see separate schedule) 11 Sunday PC3 Rest and site visits 12 PC3 Team 1+ Team 2: as per attached schedule 13 PC3 Team 1 + Team 2: as per attached schedule 14 PC3 Team 1 + Team 2: as per attached schedule Return to Addis 15 PC3 Series of meetings with 4 Tier I partners 16 PC3 Team synthesis Renee Departs in late pm. 17 PC3 Write draft 18 Sunday Rest 19 PC3 Meet HPC Meet WFP Meet UNICEF 20 PC3 Meet with SCUSA PC3 staff Meet MOWA 21 PC3 8:30-9:30: debrief for USAID 15:00: debrief for PC-3 stakeholders 22 PC3 Team leader report preparation Meet Kidest Belete 23 PC3 Depart 24 PC3 Write draft 25 Sunday Rest 52 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) SCHEDULE OF SITE VISITS FOR TWO TEAMS Town/Region Tier I Tier II Type of Tier III FGD – Children FGD – Guardians Combined Team in Addis Ababa Addis May 9 Care FHI Hope for Children Beza Le Hiwot Kindergarten teacher association Iddir coalition 6 female 2 male 6 female Subtotals 2 Local NGOS 1 iddir coalition 1 association 6 female 2 male 6 female Team 1 in Oromia and SNNPR Adama/ Oromia May 10 Save World Learning Mekdim PLWHA Assn. Mekdim Regional Staff Iddir Coalition Anti-AIDS Club School 4 female 4 male 7 female Shashamene/ Oromia May 11 World Vision Mulu Wongel Church CBO – Needy Association Awasa/ SNNPR May 11–12 FHI World Learning EKHC - Medan ACTS Regional Staff OVC Club School 3 female 5 male 10 female 2 male Dilla/SNNPR May 12–13 World Learning FHI EKHC – Medan ACTS School Ras Desta Iddir 9 female 3 male Subtotal Total Team 1 5 INGOs 2 FBOs 1 NGOS 3 schools 1 iddir coalitions 1 iddir 3 CBOS 16 female 12 male 17 females 2 males Team 2 in Bahir Dar and Dire Dawa Bahir Dar May 10 + 11 FHI Care World Learning OSSA Ethiopian Women lawyers association Iddir coalition Youth association Dudemegn primary school 12 female 4 male (2 groups) 11 female Dire Dawa May 13 + 14 SAVE the Children Propride Hirna 02 iddir coalitions Kebele 08 iddir coalition 8 female 4 male 14 female 1 male Subtotals Team 2 4 INGOs 3 NGOs 1 school 3 iddir coalitions 1 association 20 female 8 male 25 females 1 male Evaluation Totals 5 INGOs 6 NGOs 2 FBO 4 schools 5 iddir coalitions 1 iddir 5 associations 40 females 24 males 7 groups 48 females 3 males 5 groups ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 53 APPENDIX D: TEAM SUMMARY IR 1: SERVICES Services Tier IIs have: Tier IIIs have: Achievements Provided technical information and training on psychosocial skills (PSS), food and nutrition, health and hygiene, life skills, income-generating activities (IGA), legal protection for vulnerable children to Tier IIIs Became capable of providing training and coaching to Tier III Assisted Tier III to compile and report data Learned to conduct supportive supervision Facilitated referrals Managed subgrants Benefited from exchange visits Established a database Improved coverage and prioritization of services to children and families Ensured that more services were available than before Created strength so that they can scale up PC3 into other programs (Christian Aid, European Union, Clinton Foundation, Rural Reach Ethiopia) Benefited from the technical knowledge of different Tier I partners Are able to effectively use the database for data recording and reporting Learned how to assess capacity Changed bylaws to allow them to deliver additional social services (iddirs) Provided vocational training Helped children to seem happy, comfortable, and well socialized Established or supported informal preparatory classes for children out of school and kindergartens for younger children Encouraged OVC to stay in school and improved their academic results Reintegrated children off the streets back into their homes Built a cadre of volunteers in the community Improved relationships with government bodies and networks, such as HAPCO See themselves as capable of continuing to deliver a range of services (nutritional, educational, psychosocial, health, life skills, livelihood, legal protection) Provided legal support Formed partnerships or links with schools, health centers, kebele, HAPCO, police, individuals, private local foundations, private businesses such as hotels and private schools, NGOs, and iddirs to help children (150b/child/month from an individual for 10 children) Increased community awareness of the needs of OVC and changed attitudes toward OVC Mobilized local community resources and increased contributions from the community Know how to identify OVC s and prioritize services Prioritized needs in order to make the best use of available resources Enabled OVC and guardians to generate income and lead a better life Helped bedridden patients to become mobile through food assistance Helped CSSGs to increase their income Became capable of implementing 54 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) projects Brought strategic change in financial utilization and data management Used data in the database as a baseline to mobilize extra resources Challenges High staff turnover within the organization, causing a shortage of skills Delayed disbursements of funds from Tier I Frequent changes in the M&E report formats Poor planning among Tier III partners Struggled with limited resources and large numbers of children needing help. Level of services will decline as resources decrease, because Tier IIIs have been given insufficient time to build their own resources to maintain or match external resources. Unable to meet all the needs of children to a sufficient degree. Support is often minimal, insufficient, and thinly spread. Numbers needing services are increasing while the budget decreases. Magnitude of the problem (severity, spread, and complexity) is increasing. Plans are upset by delays in grant disbursements. Had to cope with frequent changes in the M&E reporting formats. All members are volunteers who have their own activities to see to. Services provided in different ways by different volunteers and with different levels of quality (bias/style, knowledge, skills, approach) Volunteers leave or reduce effort due to time constraints (burnout); consistency of care may be lacking. Staff turnover in schools Inconsistent (on and off) provision of services by some volunteers Few literate members Some resistance from CBOs and churches to change their guidelines at first Registered many children, but had to choose who was going to get the service. Used age and orphanhood criteria: double, single, vulnerable. Example: 19,000 registered as orphans (double or single) could have been 27,000 if vulnerability was used; 9,000 served. Registration raised expectations among community that could not be met. People lost confidence in the program. Increasing inflation minimized service provision. Lack of transportation and other facilities Fear of the phase–out ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 55 IR 2: CAPACITY OF CSOs IR 2: Capacity Building Tier IIs Tier IIIs Achievements Have been restructured to better support local CBOs, with power shifting to the local level. Have sufficient capacity to train and coach Tier IIIs on each technical component and on organizational components (financial, M&E). Are competent in M&E including managing the database and writing reports. Can undertake capacity assessments. Know how to reach more people. Participated in exchanges to share experiences. Can mentor and coach Tier III organizations. Can write proposals and have secured other sources of funding (Pathfinder; HAPCO, EU). Mobilized resources. Have built relationships with government and other NGO stakeholders. PC3 model has been taken up by other donors and can expand. PC3 was chosen by consultants to be presented to the Ethiopian Parliament as a best practice. Are better structured for community development. They use groups and committees for each component, which are then represented in core groups, allowing for greater representation and attention to each technical area. Clear structure, anticipation of challenges, projections of spending Have improved community mobilization methods. Are mobilizing resources. Excellent management and use of finances (tracking, projection, planning) Excellent recording and handling of data Mapped services and resources. Were able to use M&E information from the database; found it helpful. Core group system used to monitor and reflect on how well the work is going. Developed proposals. Leadership–able to manage in core group Equipped with office furniture and stationery Increased strategies to generate and save money in groups (CSSGs). Enhanced planning. Changed bylaws to allow for delivery of additional social services (iddirs). Greater understanding, skills, and knowledge in PSS, life skills, health, nutrition, legal protection, food, hygiene, IGA, through training, coaching, and practice Challenges Delays in budget disbursement made implementation difficult. Some lack of coordination between Tier I partners at the beginning of the project. High staff turnover meant that trained individuals were difficult to replace. Expectation was for five years; building the capacity of Tier IIIs took time, leaving less time for Phase-out not fully communicated. Communication was slow (PC4). Budget was delayed and funding slow; funds often disbursed late. Tier IIIs are totally volunteer-run, who must do all work in their spare time. It was difficult to adapt to the seven services, the formulas, and the structure of PC3. M&E was time-consuming and 56 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) providing services before phase-out begins. Three stages of phase-out not fully understood Frequent changes in M&E requirements had to be communicated to Tier IIIs, which took time. Phase-out announcement caused uncertainty, even unwillingness; not understood; poor communication on the justification for it. Many Tier IIIs, staffed by are volunteers, are sometimes hard to meet with. changed too often. More children need to be reached as resources from PC3 are going down. The phase-out design is not quite working. Project has been too short—the first few years were spent on capacity building. They are frustrated: they now feel ready to operate but are having the rug pulled out. They want to maximize their potential. Volunteers get overburdened. Volunteers may lack necessary skills. There are insufficient resources for vocational training for volunteers and guardians. Too few people participate in the training. The CSSG training was too short. School staff changed; only two teachers were trained in PSS and they might be moved. IR 3: NETWORKING, COORDINATION, ADVOCACY Tier II Tier III Achievements Helped build relationships between government, NGOs, FBOs, CBOs, etc. The tiered approach enabled them effectively to use existing community structures: iddirs; women associations; youth. Created a sense of ownership on OVC work. Feel part of a national effort—―we are not alone.‖ Did mapping exercise with Tier IIIs. Helped minimize duplication of effort and increased coordination between sisterly organizations. Shared human resources. Tier IIs link up for stronger services— example: Ossa and Ethiopian Women Lawyers Association for legal support. Some links with PMTCT and VCT Service delivery better linked Children were part of the core groups. Created linkage with other partners. Increased coordination with similar organizations; partnership is increased. Government role is improved. Community ownership increased. Stigma and discrimination decreased: children are not ashamed to come to seek services. More local resources available Schools, health centers, kebele, HAPCO, police, courts, health facilities, individuals, private local foundations, NGOs, and iddirs mobilized to help children (150b/child/month from an individual for 10 children). Changes in bylaws of different associations and contributions are evidence of acceptance by ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 57 Good relationship with both tiers NGO partnership forums and networks strengthened Work with HAPCO Success of the work with Tier III helped Tier IIs get resources from other partners. ProPride has many partnerships and connected with many of their own partners on OVC efforts. Tier IIs already had partners into which OVC issues could be slotted (ProPride). Decentralization—mandate is given at woreda and lebele level for health services/education. Health and social affairs desk at Dire Dawa kebele level links with this. Community facilitator paid by HAPCO facilitates volunteer meetings. Strong CSOs ready to engage with decentralized GOE structures. Tier IIs can support and join decentralized government structures—fertile ground. community of need. Churches are giving PSS. Religious leaders are involved, though to varying degrees. Volunteer Days and other celebrations bring awareness of OVC issues. Other donors are attracted to give them subgrants. Created fertile ground at community level for further care and support. Attitudes toward orphans have changed. Participation of leaders—deputy mayor and others—helps break down stigma. Visible government coordination has started between Tier III and the kebele administration. CBOs get to know each other, so that resource duplication is minimized in the community. Challenges No information on numbers of OVC from national government Several partners use HBC opportunities to identify and reach children. Not all stakeholders are cooperative; referrals may not be recognized or served, as organizations have their own clients and priorities. Some government bodies give this less attention. NGOs have less awareness on child care and support issues. Some Tier II partners have their own priority and focus areas. Lack of local HAPCO capacity means it is not available for work with core groups and Tier IIs. Needs more capacity to engage/link/use PC3 organizations. PC3 must share its experience with and support government at different levels. Absence of facilities to which to refer, and poor quality of some referral points Little knowledge by the community about children’s rights Have not used church networks. 58 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 59 LESSONS LEARNED Lessons Learned Tier II Tier III IR 1: Services Start-up phase takes longer than expected, especially with a multipartner and multilevel project. Design of projects must take into consideration the startup time that will be required for building capacity when the structure is complicated. Staff turnover can lead to lack of common understandings and expectations. (USAID and Save staff changes may have led to some lack of consistency in understanding.) When expectations and requirements change (for whatever reason) communication must be consistent and constant. Need a shared understanding of what is expected from different partners. Expectations were not sufficiently shared. There must be continuing communication and clarification between all levels. The tiered approach taught them the feasibility of giving more comprehensive services. Tier IIs learned how to implement community-level projects. PC3 showed the capacity of FBOs and associations to give services. Does the identification mean that those are reached ―too late‖ and not early enough to make support easier? Who needs support and when? Identifying only those who are most vulnerable may mean that children and families are identified only at crisis times, instead of earlier to prevent or mitigate the impact of a shock. Intervening earlier can mean that only minimal support is needed to make a major difference. Get children before they are on the street. Find children in homes with chronically ill parents before they are orphaned. Then they can be linked with ART. Children should not be registered if there is no intent to offer some services. Creating awareness on and ownership of OVC and their issues is a critical first step in mobilizing community resources for them. Assisting OVC to join with other children may be the most effective PSS strategy. (tutorials/life skills). PC3 showed communities how to solve their problems by themselves. A community-based care and support strategy is fruitful for addressing OVC service needs. Capacity of some groups not tapped. Women’s lawyers not fully used. Rubric is monolithic, not enough flexibility. Needed because program was large and complex. How is it possible to have structural and reporting consistency while encouraging differences, strengths, and uniqueness? Management demands sometimes overshadowed other aspects Consider ways to fully optimize the potential within organizations—Tier II and Tier III—while balancing the need to implement program plans. Need to consider how services are relevant and accessible to different age groups. 60 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) IR 2: Capacity Tier II and Tier III partners can fairly quickly be capacitated to make and manage small grants. The rigorous M&E demands in the end gave Tier IIs and IIIs the ability to make informed decisions and access other resources locally. Phase-out strategy: Phase-out need not always be of the weakest. Consider a graduation approach that rewards organizations that have mobilized partners to a good enough level, and communicate this at the start. Possibilities of graduation need to be investigated at the beginning with definite goals and achievements articulated. Phase-out must include some hope of continuity. They can identify resources in the community. Since different groups are represented, they were able to share knowledge, experiences, and resources. They were enabled to solve their problems by themselves. Members in Tier III became better able to manage finances. They could use and handle data to monitor and evaluate progress. IR 3: Supportive Environment Coordinated project implementation could make a difference. They learned how to use existing community structures. The environment motivated a sense of commonality. At the start there were insufficient links with the private and the public sector. Need strong government partnerships from the start to ensure sustainability. CBOs found PC3 influential in creating conducive environment from the government side. PC3 created ownership in the community. Increased self-confidence in the community allows fruitful partnerships, but must be accompanied by government capacity. Did not take full advantage of networks within churches to reach people with OVC awareness and HIV prevention information. Need to strengthen links with prevention efforts—also a way of integrating OVC into mainstream activities. ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 61 RESPONSES FROM TIER III GROUP MEETINGS Response Frequency Achievements We learned that we are the key to solving our problems. 1 PC3 has created ownership. 2 Psychosocial support is very vital all over the world and should be expanded. 8 Quality education should be enhanced as a tool for the future. 6 We were able to provide services through our own resources. 3 We were able to identify OVC. 1 PC3 should be replicated for other African countries. 1 Created new guidelines for iddirs to allow them to undertake this work. 1 Livelihood training helped a lot, so that I could help the children with me. 1 Our problems were identified first. 1 Giving the responsibilities to the community is best. 2 Community volunteerism is enhanced and the labor of volunteers is recognized. 4 Saving and credit have been improved. 3 Livelihood training was important. 1 The training and capacity building activities were effective. 3 The number of children on the street has decreased. 3 Community mobilization has been effective. 1 A community-based program has been initiated. 1 Challenges and Recommendations Community needs to look for IGA for sustainability 2 Home visits should be given for PSS. 1 Health and education support for caregivers is needed 1 We try to reach high numbers of children with the little funds available. 2 We need more information and training on legal issues. 4 We need to know how to train new members. 1 62 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) SUMMARY OF BENEFICIARY FOCUS GROUPS Children Guardian Most important service Psychosocial support Educational Nutritional Most important service Food support and shelter Training on saving and credit Life skills and livelihood training Where did you get services? Iddirs Home-based care givers Teachers Youth associations Volunteers Church Nearby health facilities Police stations Private sectors Wealthy individuals in the community Where do you get services? Iddirs Home-based care givers Volunteers Kebeles Church Nearby health facilities Level of satisfaction with service Rated consistently as high: Tutorials Life skills Guidance provided by an adult Rated as fair: Help from the church Help from the police Help from neighbors and friends Changes needed (Recommendations) The support should focus on sustainability Orphans and vulnerable children should not be separated, but care and support should be provided to all children Project should stay longer Supplies should arrive in time Changes needed (Recommendations) More focus on IGA Amount of service should increase Care and support should be also given to caregivers Project should stay longer ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 63 APPENDIX E: TOOLS PC3 EVALUATION TIER I PARTNER MEETING Questionnaire to be filled in by each Tier I partner May 8 2008 IR 2: Improved capacity of Ethiopian CSOs 1. How did you determine baseline capacity? 2. Were there goals/end points established? Can you give examples? 3. What have you discovered are the best strategies for building this capacity? 4. How are you tracking progress? 5. How well do you think PC3 has managed to build institutional competencies in these illustrative categories from the technical proposal on management capacity? 1 = excellent 5 = weak Section A: Institutional Competencies Mission and Vision 1 2 3 4 5 Corporate Governance and Leadership 1 2 3 4 5 Financial Management 1 2 3 4 5 Program and Project Management 1 2 3 4 5 Strategic Planning 1 2 3 4 5 64 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Development of Systems and Procedures 1 2 3 4 5 Human Resource Management (Including Volunteer Management) 1 2 3 4 5 Institutional Sustainability Strategy 1 2 3 4 5 Monitoring, Evaluation, and Reporting 1 2 3 4 5 Networking and Partnering 1 2 3 4 5 Section B: Technical Competencies Nutrition and Food Security 1 2 3 4 5 Health and HIV/AIDS 1 2 3 4 5 Psychosocial Support 1 2 3 4 5 Education 1 2 3 4 5 Child Protection/Legal Services 1 2 3 4 5 Economic Strengthening 1 2 3 4 5 Shelter 1 2 3 4 5 Community Mobilization 1 2 3 4 5 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 65 Child Development, Including Age-Appropriate Programming 1 2 3 4 5 Gender Equity 1 2 3 4 5 Community and Social Mobilization 1 2 3 4 5 Parenting Skills 1 2 3 4 5 6. Please estimate the amount of time (out of 100 percent) that you as a Tier I spend on the following: Institutional Competencies (see A above) _________________________________ percent Technical Competencies (see B Above) ___________________________________ percent IR 3: More Supportive Environment for OVC and Their Households 7. What is your greatest achievement in helping build a supportive environment? 8. What is PC3’s greatest achievement? 66 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) EVALUATION OF POSITIVE CHANGE: CHILDREN, CARE, AND COMMUNITIES (PC3) Questionnaire for Tier II Partners May 2008 USAID with Save the Children (USA) is conducting an end-of-project evaluation of the Positive Change: Children, Care and Community Program. The project has less than 2 years to run, and this evaluation will help determine any changes in the program over the next 18 months and will make recommendations for future program. Your help in this is highly appreciated. 1. Name of Organization: _______________________________________________________________________ 2. When was your organization established? (year) ________________________________ 3. When did you become a Tier II partner in PC3? (year)____________________________ 4. What is your organization’s role in the PC3 project? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 5. Please list the three most important ways in which your Tier I partner has assisted you: 5.1_______________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 5.2_______________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 5.3_______________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 6. In which areas has the capacity of your organization been strengthened? Please list the most important areas for you. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 67 7. Please list any additional ways in which your Tier I partner could assist your organization to be stronger and more effective in addressing issues of orphans and vulnerable children in your country. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 8. How many Tier III partners does your organization support/supervise? __________________ Type of Tier III Partner Number of Tier III Partners 9. How were these partners identified? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ _____________________________________________________________________________ 10. How did you assess their capacity? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ _____________________________________________________________________________ 11. What has been done in response? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ _____________________________________________________________________________ 68 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 12. How do you track their progress? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ _____________________________________________________________________________ 13. What are the main activities of the Tier III partners? What do they do best? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 14. What are the main changes you have seen in the work of the Tier III partners since the PC3 program began? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 15. What are the challenges you face in working with Tier III partners? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 16. What are the main challenges faced by your Tier III partners in providing quality services to orphans and vulnerable children and care givers in the community? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 17. What are the main challenges you face as a Tier II partner in PC3? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 69 18. What strategies would you recommend for improving and strengthening community capacity to help orphans and vulnerable children? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 19. What other partners do you have? How do they assist you? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 20. What other projects do you run? How do they work with PC3? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 21. Do you belong to any networks? Which ones? What do they do? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 22. Is there anything you else you would like to tell the evaluators that you think would be helpful? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 70 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) PC3 EVALUATION FORM Questions and Observation Form for Tier III Site Visits May 2008 1. Date of visit: _____________________________________________________ 2. Location: Town and region __________________________________________ 3. Name of organization _______________________________________________ 4. Type of organization (iddir, church, school) _______________________________ 5. Tier II partner_________________________________________________________ 6. When did the organization become a Tier III partner? (month/year) ________________________ 7. Whom we met on the site visit: Name, position, and gender (use separate sheet) 8. Capacity: HR Total Male Female Under 18 Committee members Paid staff Volunteers For volunteers: Hours per week Supervision by whom Major responsibilities Incentives Capacity 9. Observe availability of equipment and supplies. Room Desk Chair Filing cabinet Telephone/computer/other 10. What positive changes have you seen in your community over the last few years (3–5)? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 71 11. What training have you received from your Tier II partner? What training was the most useful? What other training would you require? What other kinds of support do you receive besides training? What other support would you find useful? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 12. What other sources of support do you have? Probe: members contribution/government/ private sector ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 13. Describe your relationship with the Tier II partner (give name of partner)? Probe: What has been good and what has been difficult? How can it be improved? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 14. What changes in your work and organization have you experienced because of this relationship with your Tier II partner? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Services and Support 15. What are your biggest concerns regarding children in your community? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 72 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 16. Where do children and caregivers in your community get services (and support)? Refer to mapping exercise. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 17. Which services do you provide? Probe: prevention Service Yes or No Overall satisfaction with these services = Question 16 Good Fair Poor Education Health and prevention Food and nutrition Shelter Legal and child protection Economic strengthening Psychosocial support Other 18. What is your opinion of those services? Are they good enough to help the child and make a difference? Why or why not? 19. How could they (the poor ones) be made better? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 20. How could your services in particular be made better? What would you need to make them better? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 73 21. How involved are the children and youth and caregivers in any decision making about services (prioritizing, quality)? How could they be more involved? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ IR 3: Supportive Environment 22. What is your relationship with these other service providers? How do you work together? How could this be improved? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 23. What policies are already in place to support vulnerable children and what ones are needed? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Monitoring and Evaluation 24. Is there any information not being collected that you feel is important? That would help you and your organization? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ How many beneficiaries do you serve?___________________(male /female)___________ How old are most of these children? 0–5 ___________6–12_____________13– 18_____________ 74 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) Walk through on forms: (3 forms?) Probe: Who fills in these forms? What happens to the forms? What is done with the information? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 25. How could this system be improved? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Anything else you would like to add? Anything else you would like to tell us? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 75 PC3 EVALUATION Focus Group Discussion with Caregivers May 2008 1. Welcome 2. What challenges do you face in caring for children? Discussion. 3. Where do children and families get services in your community? 4. Which are the most needed? Discuss. 5. How satisfied are you with these services? 6. What changes have they brought to you and your community? 7. What can be done to make the poor ones better? 8. How have you helped the organization that invited you here today with deciding on which services to provide and how to provide them? Would you like to be involved? How could you be involved? 9. What services are provided by the organization that invited you here today? 76 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) PC3 EVALUATION Focus Group Discussion with Children May 2008 Welcome and Explanation 1. Where do children and families get services in your community? Draw a map in a small group. Share. 2. Which are the most needed? You have three stars – choose which three are the most important for children in your community. Discuss. 3. How satisfied are you with these services? Hold up card (good/fair/poor) Smiley faces. Three cards per kid 4. What can be done to make the poor ones better? 5. How have you helped the organization that invited you here today with deciding which services to provide and how to provide them. Would you like to be involved? How could you be involved? 6. What services are provided by the organization that invited you here today? ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) 77 APPENDIX F: REFERENCES AGREEMENTS, REPORTS, AND M&E TOOLS 1. PEPFAR Ethiopia In-Country Reporting System (IRS), Quarterly Reporting Template 2. Positive Change: Children, Care, and Communities (PC3) Program, M and E Formats. 3. Save the Children, USA, PEPFAR Ethiopia In-Country Reporting System (IRS), Annual Reporting 2007. 4. Save the Children Federation, Inc., Positive Change: Children, Communities and Care Program Technical Application, July 2004. 5. Save the Children, USA, Positive Change: Children, Care, and Communities (PC3) Program, Narrative Annual Report October 2005–September 2006 6. Save the Children, USA, Positive Change: Children, Care, and Communities (PC3) Program, Annual Report, April 2007–March 2008. RESOURCE MATERIAL DEVELOPED BY THE PROGRAM 7. Care, Contributing to Community Self Reliance: An Innovative Saving and Credit Experience, Ethiopia, 2008. 8. Positive Change: Children, Care, and Communities (PC3), Baseline Report, 2005. 9. Positive Change: Children, Care, and Communities (PC3) Program. Changing Communities: Health and HIV/AIDS Care and Support for Orphans and Vulnerable Children in Ethiopia. 10. Positive Change: Children, Care, and Communities (PC3) Program, Changing Communities: Psychosocial Support for Caregivers and Orphans and Vulnerable Children Infected and/or Affected by HIV/AIDS. 11. Positive Change: Children, Care, and Communities (PC3) Program, Governance and Leadership Training Manual. 12. Positive Change: Children, Care, and Communities (PC3) Program, Phase-out / Transition Strategy , Task Force of PC3 Program Partners CARE FHI SC USA World Learning World Vision, March 12, 2008. 13. Positive Change: Children, Care, and Communities (PC3) Program, Strategic Planning Training Manual. 14. Positive Change: Children, Care, and Communities (PC3) Program, Volunteer Guidelines for the PC3 Program, Final Draft, January 2007. 15. Save the Children, USA, OVC Services Mapping Report, Addis Ababa, November 2007. 16. Save the Children, USA, Positive Change: Children, Care and Communities (PC3) Program, Baseline Survey Report, Addis Ababa, 2005. 17. Save the Children, USA, Quality Assurance and Improvement Standards for Orphans and Vulnerable Children Programs in Ethiopia, USAID/PEPFAR, and January 2008. 18. Save the Children, USA, Setting the Standard for Quality Care and Support to Orphans and Vulnerable Children in Ethiopia: Promising Practice and Key Achievements, March, 2008. 78 ETHIOPIA POSITIVE CHANGE: CHILDREN, COMMUNITIES, AND CARE (PC3) OTHER DOCUMENTS 19. Central Statistics Agency, Ethiopia Demographic and Health Survey 2005, Addis Ababa, 2006. 20. Garcia, M. Rajkumar, A.S., Achieving Better Service Delivery through Decentralization in Ethiopia, World Bank Working Paper No. 131, 2008. 21. Ministry of Finance and Economic Development, Ethiopia: Building on Progress. A Plan for Accelerated and Sustained Development to End Poverty (PASDEP) 2005/06–2009/10. Volume 1. September 2006, Addis Ababa. 22. Ministry of Health, AIDS in Ethiopia, 6th Report. 23. Ministry of Health, Single Point HIV Prevalence Estimate, Ethiopia, 2007. 24. Rapid Assessment, Analysis, and Action Plan for Orphans and Vulnerable Children Ethiopia, UNAIDS, Government of Ethiopia, USAID, UNICEF, WFP. 25. UNICEF, The State of the World‟s Children 2008: Child Survival, December 2007. For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com