January 2013 This report was produced at the request of the United States Agency for International Development. It was prepared independently by Deborah McSmith, Jenny Hunt, and Beverley Sebastian for International Business & Technical Consultants, Inc. (IBTCI) under the Evaluation Services IQC, AID-613-TO-12-00001. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. END OF PROJECT EVALUATION USAID/ZIMBABWE’S “CHILDREN FIRST” ORPHANS AND VULNERABLE CHILDREN (OVC) PROJECT END OF PROJECT EVALUATION USAID’S “CHILDREN FIRST” ORPHANS AND VULNERABLE CHILDREN (OVC) PROJECT IN ZIMBABWE Evaluation Report Deborah McSmith Jenny Hunt Beverley Sebastian January 23, 2013 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of the Children First OVC Project for Zimbabwe i Table of Contents ACKNOWLEDGEMENTS ............................................................................................................................................. iii ACRONYMS ...................................................................................................................................................................... iv EXECUTIVE SUMMARY .................................................................................................................................................. 1 1. INTRODUCTION AND BACKGROUND ...................................................................................................... 7 2. PURPOSE OF THE EVALUATION ..................................................................................................................... 8 4. THE EVALUATION TEAM .................................................................................................................................... 9 5. EVALUATION METHODS .................................................................................................................................... 9 6. EVALUATION FINDINGS .................................................................................................................................... 9 6.1 Overall Program Performance - What were the quantity, quality and timeliness of project results? ........................................................................................................................................................................................... 9 6.2. What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? ........................................................................... 13 5.2.1 Bantwana School Integrated Program (BSIP) ................................................................................. 13 5.2.2 Out of School Study Groups/Children’s Learning Centers and Safe Parks (OOSG/CLASP) 16 5.2.3 Early Childhood Development Centers (ECDC) ......................................................................... 17 5.2.4 Integrated Management of Pediatric AIDS Care and Treatment (IMPACT) ......................... 18 5.2.5 Youth-Friendly Corners (YFC) and Expanded Youth Friendly Corners (EYFC) ................. 20 5.2.6 Community-Based Case Management (CM) ................................................................................. 20 5.2.7 Community-Based Counseling program (CBC) ........................................................................... 23 5.2.8 Child Protection Committees (CPCs) ............................................................................................ 23 6.3. Country Ownership and Sustainability: How effective was the CF project in developing local capacity for local partner organizations and communities to effectively meet the needs of OVC? ....... 24 6.3.1 Capacity Building for Volunteer Cadres ......................................................................................... 24 6.3.2 Capacity Building for Communities to Address Harmful Practices ......................................... 25 6.3.3 Building Community Economic Capacity to Support OVC ....................................................... 26 6.3.4 CB on Behalf of Children with Special Needs ............................................................................... 27 6.3.5 Meeting the Needs of Girls and Boys ............................................................................................. 27 6.3.6 Child Participation ................................................................................................................................ 28 6.4 To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? ................................................................................................ 28 7. EVALUATION RECOMMENDATIONS .......................................................................................................... 31 Performance Evaluation of the Children First OVC Project for Zimbabwe ii ANNEXES ......................................................................................................................................................................... 32 ANNEX A. STATEMENT OF WORK ............................................................................................................. 32 ANNEX B. EVALUATION DESIGN ................................................................................................................ 43 ANNEX C. EVALUATION TIMELINE AND LIST OF PEOPLE INTERVIEWED ................................. 55 ANNEX D. DATA COLLECTION INSTRUMENTS ................................................................................... 58 1. Interview Instrument for Ministries/Local Authorities/UNICEF 2. Interview with Primary Partner Organization Managers 3. Final Evaluation Focus Group Discussion Guide – Group Facilitators & Community Volunteers (e.g., CPC, CFPs, OOS Facilitators, Caregivers) 4. Final Evaluation Focus Group Discussion Guide – Parents/Guardians 5. Final Evaluation Focus Group Discussion Guide – Children and Youth (Including In and Out of School) 6. BSIP/VSIP Interview for Education, Health, Child Protection Personnel 7. Focus Group Discussion with CF Technical Field Officers ANNEX E. DOCUMENTS REVIEWED .......................................................................................................... 71 ANNEX F. CF TARGETS REACHED BY PEPFAR INDICATORS ........................................................... 73 ANNEX G. CHILDREN FIRST JOINT INSTITUTIONAL ASSESSMENT TOOL ................................. 76 ANNEX H. ADDITIONAL RECOMMENDATIONS .................................................................................... 86 ANNEX I. CHILDREN FIRST PARTNERS ACROSS 5 YEARS................................................................... 90 ANNEX J. CHILDREN FIRST & PEPFAR INDICATORS ............................................................................. 91 ANNEX K. CONFLICT OF INTEREST DISCLOSURE FORMS ................................................................ 92 Performance Evaluation of the Children First OVC Project for Zimbabwe iii ACKNOWLEDGEMENTS The IBTCI evaluation team for the USAID/Zimbabwe Children First OVC End of Project Evaluation wishes to thank the USAID/Zimbabwe colleagues who guided and supported our work in country, in particular Matthews Maruva and Collen Marawanyika, as well as the Children First staff in Harare and Bulawayo, who were forthcoming, timely in their responses to questions, and very helpful in arranging our field-based data collection. We offer our sincere thanks to key Government of Zimbabwe colleagues with whom we met in the Ministry of Labour and Social Services (MOLSS/DSS), Ministry of Education, Sports, Arts and Culture (MOESAC) and its Non Formal Education Correspondence School, Ministry Of Youth Development, Indigenization, and Employment (MOYDIE), the Ministry of Health and Child Welfare (MOHCW PMTCT Program and Chitungwiza Hospital), and Greater Harare City Health authorities (City Health). We also thank UNICEF for sharing its perspective as manager of the Child Protection Fund. Our thanks go as well to all the Children First current and former partners we were fortunate to meet during this evaluation, and to the groups of affiliated volunteers, parents/guardians and beneficiaries who shared their time, experience and suggestions with us. Performance Evaluation of the Children First OVC Project for Zimbabwe iv ACRONYMS AIDS Acquired Immune Deficiency Syndrome AMTO Assisted Medical Treatment Orders ART Anti-Retroviral Therapy BEAM Basic Educational Assistance Module BSIP Bantwana Schools Integrated Program CAB/C Child Advisory Board/Committee CATS Community Adolescence Treatment Supporters CB Capacity Building CBCP Community-Based Counseling Program CBO Community-Based Organization CBT Community-Based Trainer CDLP Child Rights CD Listener Programme CF Children First CHAI Clinton HIV/AIDS Initiative City Health Harare Municipal City Health Department CLCPC child-led child protection committee CPC Child Protection Committee CPF Child Protection Fund CPS Child Protection Society CSI Child Status Index CSW Council of Social Workers DCIZ Disabled Children’s Initiative Zimbabwe DIC Drop-In Center DSS Department of Social Services ECDC Early Childhood Development Center EGPAF Elizabeth Glaser Pediatric AIDS Foundation EYFC Expanded Youth Friendly Corner FBO Faith Based Organization FST Family Support Trust GOZ Government of Zimbabwe GRS Grassroots Soccer HBC Home Based Care HIV Human Immunodeficiency Virus HTF Health Transition Fund IGA Income Generating Activity IMPACT Integrated Management of Pediatric HIV and AIDS Care and Treatment ISAL Internal Savings and Lendings Scheme ISOP Integrated Skills Outreach Programme JAZ Junior Achievement Zimbabwe JCT Justice for Children Trust JIA Joint Institutional Assessment KRA Key Result Area M&E Monitoring and Evaluation MOESAC Ministry of Education, Sport, Arts and Culture MOHCW Ministry of Health and Child Welfare MOJLPA Ministry of Justice, Legal and Parliamentary Affairs MOLSS Ministry of Labour and Social Services MOYDIE Ministry of Youth Development, Indigenization, and Employment MTR Mid-Term Review Performance Evaluation of the Children First OVC Project for Zimbabwe v NACCW National Association of Child Care Workers (South Africa) NASW National Association of Social Workers (Zimbabwe) NAP National Action Plan NGO Non-Governmental Organization OPHID Organization for Public Health Interventions and Development OOSG Out of School Study Group OVC Orphans and Vulnerable Children (affected by HIV/AIDS) PAZ Pediatric Association of Zimbabwe PEP Post-Exposure Prophylaxis PEPFAR President’s Emergency Plan for AIDS Relief PMTCT Prevention of Mother to Child Transmission POS Program of Support PSS Psychosocial Support QI Quality Improvement REPSSI Regional Psychosocial Support Initiative SADC Southern African Development Community SDA/C School Development Association/Committee SHA School Health Assessment SMS Short Message Service SRH Sexual and Reproductive Health SRHBC Seke Rural Home-Based Care UAN Umzingwane AIDS Network UGM Umbrella Grant Management VCT Voluntary Counseling and Testing VFU Victim Friendly Unit Performance Evaluation of the Children First OVC Project for Zimbabwe 1 EXECUTIVE SUMMARY Introduction and Background Children First (CF) is a five-year, $17 million cooperative agreement between USAID and World Education International, funded by USAID under PEPFAR II. CF was launched in March 2008 as USAID’s primary OVC intervention in Zimbabwe, with an end date of 31 December 2012. CF sought to mitigate the impact of HIV and AIDS in Zimbabwe by developing and improving on effective models of care and support for OVC and leveraging the experience of national and community-based organizations to increase access to quality holistic services for OVC. Over the five-year project period, CF included 22 partner organizations that provided education, health, child protection services and advocacy for OVC. Project start-up in 2008 coincided with unprecedented economic difficulties and a hostile operating environment for non-government organizations (NGOs). The CF project was designed before Zimbabwe ‘dollarized’ to use of the US currency, and original targets were set based on a currency exchange rate that became meaningless. With adoption of the US dollar, costs of services increased and numbers of beneficiaries were reduced accordingly; CF and USAID changed project targets from 180,000 children served to 65,000 child-years of support1 for a specified cohort of children. This end of project evaluation examined how well CF had responded to Mid Term Review recommendations as well as evaluation questions regarding innovative, sustainable and effective models of service delivery; quantity, quality and timeliness of results; capacity development for partners and communities; and institutionalization of models within relevant government structures. Evaluation Purpose, Team, Methodology Purpose: The evaluation focused on CF’s key results areas (KRA):  KRA 1: Access to OVC Services through Community Initiatives Increased  KRA 2: Human Capacity in Local Community Structures to Meet Needs of OVC Strengthened  KRA 3: Community and National Level Advocacy for Social Protection of OVC Improved and sought to answer the following questions:  Has the project developed any innovative, sustainable and effective models of service delivery with potential for nationwide scale-up?  Overall Program Performance: What were the quantity, quality and timeliness of project results?  Innovation: What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building?  Country Ownership and Sustainability: How effective was the CF project in developing capacity of local partner organizations and communities to effectively meet the needs of OVC?  To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? Team: The three-person evaluation team included an international team leader with broad global HIV and MNCH experience; a Zimbabwean social worker and palliative care and bereavement specialist with extensive OVC experience in country; a Southern African regional M&E specialist, resident in Zimbabwe, with experience in OVC and the broader public health sector; and two Zimbabwean Shona and Ndebele translators. There were no conflicts of interest for evaluation team members. Methodology: The evaluation employed a mixed methods methodology that included: a) desk review, b) semi￾structured key informant interviews, (c) a ‘snowball sampling approach’ to Focus Group Discussions (FGD), (d) observational site visits; and (e) primary and secondary data analysis including thematic content analysis. Gender issues were woven into FGD tools for caregivers and beneficiaries. The team used experienced 1 If a child received a defined service every year for five years, this would equate to five child-years of support. Performance Evaluation of the Children First OVC Project for Zimbabwe 2 translators for FGD. Team members reviewed case files used to document beneficiary services, referrals and follow-ups. Evaluators conducted a briefing with USAID mission and final debriefs with stakeholders and mission colleagues. Time was the major limitation for the evaluation given the number of CF partners, volunteers and beneficiaries to be interviewed. A further limitation was the fact that while individual OVC were identified uniquely with respect to services received in a given year, individual beneficiaries were not identified uniquely so as to make it feasible to count the total number of individuals served over the life of the project. Findings What were the Quantity, Quality and Timeliness of Project Results? Quantity: Tracking quantitative outputs for CF has been challenging for multiple reasons. Per managers, no programming was done in 2008, due to setting up systems and politically related shut down of all NGO activities. Children were tracked by the number of services (1, 2, or 3 or more) provided rather than individually. No gender disaggregation was made for targets, and it is unclear why targets were set only for children receiving three services. No specific targets were set for discrete services such as Bantwana School Integrated Program (BSIP) tuition/levy support or School Health Assessments (SHA), number of birth certificates obtained, or number of children served by ECDCs, although some numbers, e.g. of youth who received SRH education through YFC, were counted. See Table 1in Section 5 for an overview of project performance by key result area and Annex F for targets reached by PEPFAR indicators. (Evaluators received various data tables from WEI that did not match in all cases and also received conflicting information about which unique services were and were not counted; enhanced data audits will be very important for the new program.) Although CF developed a database for partners, partners electing not to use it were unable to track children over time; therefore the project was unable to account for total unique children that were served over its lifetime. Quality: CF aimed to work with partners to develop OVC quality standards under each of the three broad areas (Education, Health Care and Child Protection Services) from FY10. CF supported the Council of Social Workers (CSW) to develop minimum standards for Community Child Care Workers and supported DSS to develop a Child Protection Committee protocol. No minimum standards were developed for Education and Health Care. CF provided phased capacity building (CB) to improve quality of partners’ service delivery through strategies (p. 19). Recommendations for strengthening partner CB are included in Annex H. The new program may benefit from aligning with the USAID Quality Improvement (QI) Initiative for OVC implementing partners. Timeliness: CF responded rapidly to requests for funding and technical support (especially in the areas of finance and M&E). Partners credited CF with initiating its operations during a period in Zimbabwe’s history that was fraught with social, political and economic instability. What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? The Bantwana School Integrated Program (BSIP) model was introduced in Harare, greater Harare and Matabeleland South Province using schools as service delivery points for care and support through (i) Block Grants to help OVC access education; (ii) School Health Assessments (SHAs); (iii) School Development Association/Committee (SDA/C) business training and small grants; and (iv) Child Rights CD Listener Program (CDLP). Most children on grant tuition lists achieved an 80% attendance rate. CF engaged with City Health Harare, district hospital and local clinic staff to conduct SHAs for students and out-of-school learners at ten Out Of School Study Group (OOSG) sites. SHAs included referrals for HIV testing and ART and treatment of communicable infections and dental caries. Transport support was provided to clinics, and transport and salary top-ups were crucial for nurses who conducted SHA. Without ongoing salary and transport support, SHAs will only continue in Grades 1, 3 and 7 in the Harare area, as was done Performance Evaluation of the Children First OVC Project for Zimbabwe 3 prior to CF. The new OVC program may wish to evaluate the degree to which SHAs improve child health and use findings to advocate for more funding in the national education or health budget. SDA/C members were provided with business training and start-up funds to support IGAs, with profits from these businesses intended to provide ongoing educational assistance to OVC after project’s end. A total of 98 SDA/Cs were trained in business, and a total of 152 IGAs were implemented. Business profits will not replace the level of tuition and levy support provided through School Block Grants. CF implemented the Child Rights CD Listener Program to raise awareness of children’s rights and responsibilities, reaching children and youth through radio media and school classroom programs. The evaluation elicited multiple reports of association between the listener program and increased demand among students for birth certificates, although this wasn’t counted. The program targeted school children without sensitizing parents, creating some tensions in homes. PSS teachers lacked motivation and expressed need for more training and mentoring support. Over 1,300 out-of-school learners were served by the OOSG/CLASP sites across five provinces, with 186 children reintegrated back into formal school. CF worked with the MOESAC Distance Education Correspondence School to develop an accelerated curriculum for learners who had been out of school for short periods and others who had never been to school. This initiative has responded to an important need, but it appears that one curriculum cannot successfully stretch across such disparate learning levels. Use of retired teachers and university students as facilitators was an innovative practice, although quality of facilitator skills varied. The structure of OOSG sites also varied; some lacked adequate ventilation or ramps for disabled children and others provided safe structures. To mitigate effects of an abrupt cessation of a wet feeding activity under the former Program of Support, CF expanded its ECD program to include ECD Centers (ECDC) that were no longer funded and assisted seven ECDCs to create self-sustaining structures, such as nutrition gardens or poultry projects, to reduce young children’s vulnerability to changes in donor funding. Most but not all CF centers provided a nutritious meal or snack during the day; however, at both OOSG and ECDC sites young children were observed to be without nutritional support for several hours. The Integrated Management of Pediatric AIDS Care and Treatment (IMPACT) model sought to stimulate demand for and access to pediatric ART; the model facilitated access to pediatric HIV testing, CD4 testing, ART, cotrimoxazole prophylaxis and treatment of opportunistic infections; and provided transport for children to district hospitals, adherence peer support and support groups, and psychosocial support (PSS), and community sensitization meetings. During IMPACT pilots, the average time for a child to be initiated on ART was reduced from three months to two weeks. Under the Community Adolescent Treatment Supporters (CATS) program, 30 trained peer counselors provided adherence and psychosocial support to children on ART. Partner Africaid piloted the model in 2009 and expanded it through CF support. (As a data challenge example, WEI reported that 1,822 children had received CATS support but later told evaluators number of children served weren’t reported.) This model has worked well, but within an extremely protective environment and with limited visibility of adolescent volunteers. The CPF will support its expansion in 2013. In response to MTR recommendations to strengthen its focus on youth between 12 and 18 years of age, CF initiated Youth Friendly Corners (YFC), a model that provides youth with spaces to meet for SRH education, life skills and livelihood activities, YFC were observed to work well at Howard Mission Hospital. The concept was modified to include Expanded Youth Friendly Corners (EYFC), out of door platforms to reach out of school youth with SRH education and sports activities. Partners trained 168 peer educators at EYFC. Evaluators found most participating youth to be over 18 and dispirited, with limited hope or opportunities to build positive economic futures. Performance Evaluation of the Children First OVC Project for Zimbabwe 4 Using a network of resources including Child Protection Committees (CPC), village registers, volunteer care workers, clinics and hospitals, schools, CBOs, paralegals, and a tool to assess child well-being, the CF Community-Based Case Management (CM) approach demonstrated impact in identifying and serving children, although demand for services largely outstripped supply. Nearly all interviewed volunteers reported a lack of emotional support and supportive supervision. Case files were incomplete and often out of date. The model showed promised, and DSS publicized its intention to roll it out as a national CM system under the National Action Plan 2 for OVC. CF partner Childline’s Community-Based Counseling program (CBC) demonstrated effectiveness as a crisis counseling model through drop in centers (DICs). Expansion of DICs is a replicable approach contingent upon adequate trained social workers and volunteer counselors to staff the sites. Under DSS, Child Protection Committees (CPCs) compile village registers to identify and manage the care of OVC. CF attempted to revitalize CPCs and link them with CM to promote the continuum of care. CF provided financial support to DSS to develop a CPC protocol, which defines quality, standards and responsibilities. CPCs visited during the evaluation lacked resources to operate effectively. How effective was the CF project in developing capacity of local partners and communities to meet the needs of OVC? Capacity Building for PSS and Counseling was provided by local training organizations CONNECT and CONTACT; a total of 268 people were trained in counseling across nine partners. However, the evaluation team heard community care workers and other volunteer cadres express lack of confidence about child counseling skills. The counseling training does not address loss and bereavement, both of which are key factors in OVC support. Child care workers were trained to use the Child Status Index (CSI) tool to standardize data collection CF supported Justice for Children (JCT) to train 12 volunteer Community Paralegals in Harare on laws that protect children to address legal bottlenecks such as birth registration, inheritance and custody, and increase community awareness on child legislation. Per CF reports, paralegals handled 2,927 cases. Evaluators had no opportunity to see this program underway. CF supported Children with Special Needs by initiating sign language training for teachers and police officers to enable better communication with deaf children and youth. The Braille Institute translated CDLP materials for vision-impaired children. CF brought together agencies serving children with special needs under the Disabled Children’s Initiative of Zimbabwe (DCIZ) to develop a collaborative framework. CF supported Emerald Hill and Nzeve Schools for Deaf Children to train teachers in Greater Harare to work with deaf students in general classrooms and equipped schools with furniture and other resources. A community support group for parents of disabled children facilitated by Kapnek Trust was found to be a good practice for replication where possible. CF introduced Small Grant mechanisms in FY 11 as an exit strategy to build communities’ economic capacity to continue to provide OVC support, and disbursed a total of $2,703,403 USD in grants through partners to 61 ECDC, FBOs and CBOs and 121 community structures. Evaluators did not hear of any plans to monitor the success and appropriate use of grants. Of 1,533 youth who received business training through CF’s Livelihoods strategies, 243 (16%) engaged in businesses as a direct result of the training. Across youth livelihoods strategies, success rates have been low. Evaluators observed that the most innovative livelihoods strategy has been the introduction of the CARE Internal Savings and Lendings (ISAL) model to improve livelihoods of caregivers, volunteers and some older OVC. CF addressed the needs and rights of girl children and adolescents through CDLP and Peer Educator trainings that emphasized gender equality and equal participation. Sports activities under YFC and EYFC encouraged Performance Evaluation of the Children First OVC Project for Zimbabwe 5 equal participation of female and male youth. CF promoted child participation through Child Led CPCs, the CATS model, Child Law Clubs2, YFC and EYFC, and the establishment of child advisory boards for CF and all partner organizations. To what extent were processes and products developed institutionalized within relevant government ministry structures? The BSIP model expanded the SHA program already in place with Harare City Health; an evaluation of SHA impact on children’s health might help to leverage additional GoZ funding for expansion to more grades and districts. The CDLP component has been well integrated into schools. The OOSG/CLASP model has achieved some measure of institutionalization through curriculum development and some success in reintegration of pupils into formal schools. The ECDCs supported at community centers, schools and clinics are aligned with national policy but have depended on CF and partner funding to supply learning resources. There is no indication that this model has been institutionalized at any meaningful level. Africaid’s CATS model has been nationally and regionally recognized as a best practice and its roll out will be supported by the CPF. The MoHCW has expressed interest in the IMPACT model but there is no funding support for its potential integration into the national pediatric ART program. The YFC model appeared to work well in relatively well resourced settings such as Howard Mission Hospital, but the EYFC model faces challenges, with no apparent donor funding targeted to youth above 18 years and no national consensus on how best to serve older youth. DSS has endorsed the CM model for national roll out with support from the CPF. The model will be institutionalized across ten districts each year in alignment with social cash transfers. Potential exists for institutionalizing the CBC model developed by Childline in collaboration with DSS, contingent upon resources and counselor training and supervision, which DSS cannot now provide. Under NAP2 CPCs are poised to take on the task of child protection within communities; capacity to carry out this role depends on financial and human resource support from DSS not yet in place, however the CPC Protocol supported by CF is part of DSS policy. Minimum standards for Child Care Workers have been developed and endorsed by the National Council of Social Workers. Overall, CF OVC models and strategies will require ongoing donor support to maintain momentum toward institutionalization within relevant government ministries. Recommendations: Key recommendations provided here are relevant for OVC program strengthening across all service domains. Additional, actionable recommendations, including recommendations to strengthen partner capacity building strategies and improve partnerships, are included as Annex H.  Work with partners to utilize appropriate data management systems that have the functionality to track and analyze longitudinal data, e.g. unique children and families served over time.  Explore the USAID Quality Improvement (QI) Initiative for inclusion in future OVC programming.  USAID work with relevant stakeholders to maximize all linkages for nutritional support that will benefit OVC and vulnerable households.  Embed a family-centered approach within case management and other child protection services that is cognizant of relationships, extended family network, custody and guardian issues. Review and revise case management tools to strengthen their family focus.  Implement supportive supervision for community level volunteer cadres that allows volunteers to learn from experienced colleagues; investigate the possible use of retired, certified social workers through CSW and NASW to provide supportive supervision. 2 These clubs were linked with the Community Paralegal cadre and referred to in reports; however evaluators had no opportunity to observe or interact with them. Performance Evaluation of the Children First OVC Project for Zimbabwe 6  Seek a careful balance between advocating for the rights of vulnerable children and encouraging them to use their voices, and safeguarding their rights to confidentiality and privacy.  Given CPF support for strengthening case management and child protection services, consider support for research to determine how well IMPACT and SHA models improve OVC health and use findings to leverage more GoZ support for health programs that serve vulnerable children and youth. Strategize with CPF how to best complement its support for the GoZ BEAM program with other essential Educational Assistance for OVC. Performance Evaluation of the Children First OVC Project for Zimbabwe 7 1. INTRODUCTION AND BACKGROUND Children First (CF) was a five-year, $17 million cooperative agreement [674-A-00-08-00021-00] between USAID and John Snow International (JSI)/World Education International (WEI) and is funded by USAID under PEPFAR II. CF was launched in March 2008 as USAID’s primary OVC intervention in Zimbabwe and extended through December 31, 2012. As a result of PEPFAR budget increases and plus up funds, the total US Government funding contribution to CF stands at $17,590,000. CF sought to mitigate the impact of HIV and AIDS in Zimbabwe by developing and improving on effective models of care and support for orphans and vulnerable children affected by HIV/AIDS (OVC), leveraging the experience of national and community-based organizations to increase access to quality care and support services for OVC. Working in Matabeleland South and Harare’s urban, peri-urban and surrounding rural areas across 21 out of 65 national districts,1 CF provided technical and financial support to its partners to provide services, undertook advocacy initiatives for OVC, and strengthened program sustainability by institutionalizing programs where possible at local and national levels. Over the five-year project period, CF included 22 primary partners, seven of whom worked with local sub￾partners. CF worked alongside the national Program of Support (POS) for OVC until its conclusion in December 2010.2 POS supported 33 NGOs and 150 local organizations in order to scale up support for the National Action Plan (NAP) 1 for OVC 2004-2010. Following recommendations from the 2010 evaluation of the NAP1, a NAP 2 2011-2015 was developed by the Government of Zimbabwe (GOZ). While NAP1 focused on increasing child participation; the number of children with birth certificates; strengthening nutrition, health and hygiene for OVC; and reducing the number of children who live outside a family environment, NAP2 focuses on building the capacity of government systems through four key pillars: 1) Strengthening the household economy; 2) Increasing children’s access to basic services; 3) Increasing access to child protection services and justice; and 4) strengthening program coordination and management. To help operationalize NAP2, the GoZ designed a multi-donor OVC funding mechanism called the Child Protection Fund (CPF). The CPF, managed by UNICEF, was designed to better achieve national scale by focusing on aspects of three of the NAP 2 pillars: 1. Reduce household poverty and economic disparity for extremely poor households 2. Enhance access to effective child protection services for vulnerable children 3. Improve access to basic services including education. The CF Project was implemented during a period in Zimbabwe’s history that was fraught with change and instability. Project start-up in 2008 coincided with unprecedented economic destabilization and a hostile operating environment for NGOs. World Education staff described how the country had few HIV services available to children and insufficient numbers of test kits and pediatric ARVs. According to FGD participants, during this time many communities were largely unaware that children as well as adults might be HIV-positive. A number of organizations that became CF partners were on the verge of closure due to funding challenges, high staff turnover, and inability to safely operate in target areas due to political instability. Several partners credited CF with enabling them to continue functioning and with having strengthened and expanded their services. 1 Harare, Chitungwiza, Epworth, Khami, Mzilikazi, Mazowe, Chegutu, Zvimba, Goromonzi, Ruwa, Nyanga, Mutasa, Mutare, Makoni, Chipinge, Norton, Chimanimani, Bulawayo Central, Marondera, Seke, Umzingwane. 2 An $81 million program jointly funded by Australia, New Zealand, the EC, Germany, Netherlands, Sweden and the UK and implemented by UNICEF that reached approximately 590,000 OVC from 2006-2010. “Without doubt we would have closed without Children First.” Performance Evaluation of the Children First OVC Project for Zimbabwe 8 The CF project was designed before Zimbabwe ‘dollarized’ to use of the US currency. The local currency was subject to hyperinflation and original targets had been set based on a currency exchange rate that later became meaningless. The project was initiated in January 2008, when emergency legislation prevented NGO partners from going into the rural areas, effectively confining the project to Harare in its first year. The Howard Mission Hospital became an important early partner as it was already operating in a rural area not far from Harare. Umzingwane District was chosen as an area of focus due to the Umzingwane AIDS Network’s longstanding presence and good relationship with District Authorities. Initially, because of difficulty registering as an NGO, WEI operated under the auspices of USAID, with USAID providing letters of invitation for any workshops that included Ministry representatives. NGOs had never before worked under such daunting constraints in Zimbabwe. CF and USAID agreed to change project targets from 180,000 children served with more than 3 services to 65,000 child-years of support for a cohort of children, and CF developed a database to assist partners to track services with a total of 20 PEPFAR and other indicators (Annex J). An internal mid-term review (MTR) of CF conducted in February, 2010 recommended that future USAID support to OVC programs in Zimbabwe align with NAP2 and include the following components:  Target more youth, particularly out-of-school youth between the ages of 12 and 18 years;  Ensure comprehensive health and education services during early childhood;  Reach most vulnerable children, e.g. children living in extreme poverty or outside of family care;  Include more services for children with disabilities;  Provide capacity-building to the social services system, particularly to strengthen the social service workforce at the district and provincial levels. 2. PURPOSE OF THE EVALUATION This end-of-project performance evaluation is intended to provide longitudinal information and add current recommendations that can find immediate utility in the early phases of the successor OVC project. Findings are intended to inform USAID/Zimbabwe’s design and implementation of more effective, efficient and sustainable country-owned programs. The evaluation scope was framed within CF’s key result areas:  Key Result Area 1: Access to OVC Services through Community Initiatives Increased  Key Result Area 2: Human Capacity in Local Community Structures to Meet Needs of OVC Strengthened  Key Result Area 3: Community and National Level Advocacy for Social Protection of OVC Improved Data collection instruments (interview and FGD tools) were designed to answer the evaluation questions identified in the Scope of Work (Annex A) and detailed below: Key Evaluation Question: Has the project developed any innovative, sustainable and effective models of service delivery with potential for nationwide scale-up? 1. Overall Program Performance:  What were the quantity, quality and timeliness of project results? 2. Innovation  What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? 3. Country Ownership and Sustainability  How effective was the CF project in developing local capacity to effectively meet the needs of OVC? Focus on capacity development for: o local partner organizations and o communities.  To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? Performance Evaluation of the Children First OVC Project for Zimbabwe 9 The primary audiences for the evaluation report are USAID, Government of Zimbabwe, development partners, implementing partners and key stakeholders, especially at sub-national levels. 4. THE EVALUATION TEAM The evaluation team included an American team leader, two OVC technical experts from Zimbabwe, and two translators. Deborah McSmith (Team Leader) has worked with HIV/AIDS since 1985 and has broad global HIV and MNCH experience, including work in Zimbabwe. Jenny Hunt is a Zimbabwean social worker and palliative care and bereavement specialist with extensive OVC experience in country. Bev Sebastian is a Southern African regional M&E specialist, resident in Zimbabwe, with experience in OVC and the broader public health sector. Rodwell Chaitezvi and Ivy Makeleni are Shona and Ndebele translators and experienced FGD facilitators who were contracted through Target Research Zimbabwe. The evaluation team had no conflicts of interest with respect to the CF OVC project. 5. EVALUATION METHODS The evaluation employed a “mixed methods” methodology that included: a) desk review of project and national documents (Annex C); b) semi-structured key informant interviews with relevant government officials, UNICEF as CPF Manager, municipal health authorities, a district hospital pediatrician, CF staff in Harare and Bulawayo, primary partner managers and partner staff working at community level; (c) a ‘snowball sampling approach’ to Focus Group Discussions (FGD) with female and male community volunteers, primary and secondary caregivers, and girl and boy children and youth beneficiaries; (d) observational site visits to 16 CF partners; and (e) secondary and primary data analysis including thematic content analysis. The team spent the first three days in-country piloting tools to ensure standardization and quality of data collection. Gender issues were woven into FGD tools for caregivers and beneficiaries. In addition to interviews and FGD, evaluators also reviewed case files, forms and procedures related to case management, including referrals and follow-ups, etc., and other partner-based documents related to the provision of OVC services. Notes were transcribed the same day to avoid recall bias. Triangulation was employed to validate data from interviews at ministry, donor, partner and community levels. The key limitation for this evaluation was a time frame limited to three weeks for data collection across 16 partners in multiple locations and the broad spectrum of service categories (education, health, child protection, advocacy, children and youth with special needs and economic strengthening interventions). As USAID raises its standards for carefully structured, quality evaluations, it may also wish to allow more time to enable increased rigor and validation in keeping with these standards. An additional limitation was the fact that CF did not keep track of specific unique beneficiaries as they aged so that the count of specific beneficiaries over the life of the project is not well defined. 6. EVALUATION FINDINGS 6.1 Overall Program Performance - What were the quantity, quality and timeliness of project results? (i) Quantity: Table 1 below summarizes the CF project performance according to its Key Result Areas. Tracking quantitative outputs for CF has been challenging for multiple reasons. Per managers, no programming was done in 2008 due to time spent setting up systems and the politically related shut-down of all NGO activities. When services began, children were tracked for having received 1, 2 or 3 or more care services; however targets were only set for children receiving 3 services. Per CF, no gender disaggregation was made for targets, although evaluators found gender-based percentage targets in reports. It is unclear why no specific targets were set for discrete services such as tuition/levy support, SHA, number of birth certificates obtained, number of children served by ECDCs, number of children served by the CATS model, although some numbers, e.g. of youth who received SRH education through YFC, were counted. Table 1: Children First Annual Targets and Reach Performance Evaluation of the Children First OVC Project for Zimbabwe 10 Outcome/Results Performance Indicators FY08 Actuals FY09 Targets FY09 Actuals FY10 Targets FY10 Actuals FY11 Targets FY11 Actuals FY12 Targets FY12 Actuals Key Result Area 1: Access to and Quality of OVC services through community initiatives increased More comprehensive OVC services in targeted areas promoted OVC provided with 3 or more services 12,867 13,402 55,000 47,356 65,000 75,608 75,000 84,495 OVC provided with 1 or 2 services 25,733 45,905 33,295 50,766 36,025 Total OVC served 59,307 80,651 126,374 120,520 # of eligible OVC who received food and/or other nutrition services 500 3,132 3,132 2,000 4,025 2,000 5,499 2,000 5,520 # of organizations and community initiatives receiving funding from Children First sub-grants 50 62 60 55 70 117 75 172 %of children showing improvement according to Child Status Index 50% 0% 50% 50% 43% 50% 17% # of USG assisted service delivery points providing Family Planning 10 8 8 8 8 8 7 10 12 Key Result Area 2: Human capacity and performance of local communities to meet needs of OVC strengthened Governance and financial capacity of community structures improved % of local organizations that have been provided with management and or technical training relevant to their OVC programs. 20% 100% 40% 100% 60% 83% 80% 105% Technical capacity to deliver quality OVC services improved # of providers /caregivers trained in caring for OVC. 2,500 2,500 1,438 1,900 1,074 1,000 4,399 1,000 5,533 # of people (health/ Para-health workers) trained in family planning/ reproductive health with USG funds 80 80 96 80 86 80 93 80 114 Key Result Area 3: Community and national level advocacy for social protection of OVC improved Child participation in OVC advocacy increased # of OVC participating in community, district and national level advocacy activities 3,500 33 5,000 3,159 5,000 5,240 5,000 6,485 Knowledge of OVC social protection policies improved # of community groups targeted with information about OVC needs and child rights 40 5 40 268 40 930 30 1,020 See Annex F for targets reached according to PEPFAR indicators. Evaluators received various data tables from WEI that did not match in all cases, with conflicting information about which unique services were counted; enhanced data audits will be very important for the new program. In FY10, funding delays led to the suspension of various activities and CF reached only 86% of intended target with a package of 3 or more services. The delayed disbursement of additional USG Population and Supplementary funds resulted in CF issuing sub-grants for community initiatives to 55 out of a targeted 60 organizations and groups. Otherwise, per reports, CF met or exceeded annual targets for number of children reached with 1 or 2 or 3 services. OVC programming attempts to address many aspects of child wellbeing and it is challenging to assess true reach or impact without effective data management systems. Based on an evaluation of partner M&E systems and discovery that 11 of (then) 15 partners lacked a database to manage program data, CF engaged a consultant to develop a user-friendly Access database. Partners that elected not to use it were unable to track individual OVC over time; as a result the project was unable to account for the total number of unique children that were served over its lifetime. For future programming it is strongly recommended that partners be guided to use data management systems that have the functionality to track and analyze longitudinal data. Further to reporting to donors and avoiding double counting, tracking individual children over time allows for identification of patterns of change; and other information to support implementation of holistic care. Performance Evaluation of the Children First OVC Project for Zimbabwe 11 (ii) Quality CF’s Performance Management Plan (PMP) detailed its strategy for quality improvement (QI) of its core services. The document specified that QI would be achieved through targeted capacity building of partner organizations and individuals providing care to OVC such as caregivers. CF aimed to work with its partners to develop OVC quality standards under each of the three broad areas (Education, Health Care and Child Protection Services) from FY10. In the area of Protection Services, CF supported the Council of Social Workers (CSW) to develop minimum standards for Community Child Care Workers and supported DSS to develop a CPC protocol. No minimum standards were developed for the other core areas of Education and Health Care. In terms of capacity building (CB) for partner organizations, CF conducted pre-award assessments as partners came on board in 2008 and developed a CB plan for each organization in 2009. CB components included quarterly partner meetings, M&E trainings and data verifications, and quarterly thematic cluster workshops (health, education, child protection, youth and livelihoods) for partners working in shared focal areas. CF conducted a partners’ resource mobilization workshop in 2011 and strategic planning workshop in 2012, and assisted Umzingwane AIDS Network to develop a five-year strategic plan and expand its focus from Home Based Care to a broader social protection context). In FY09, CF introduced the Joint Institutional Assessment tool (JIA)3 (Annex G) to assess partners’ readiness to implement OVC work across education, health, child protection areas and to conduct advocacy. This tool is well designed to reflect relevant organizational capacity, as it looks at leadership, governance and strategy, program and financial management, M&E, utilization of volunteers, and creation of an enabling and sustainable environment. The tool includes a rating system to help prioritize partners for targeted CB. Partner assessment scores indicated common CB needs across several domains. CF organized technical M&E and finance management workshops and operational guidelines to prepare partners for PEPFAR reporting. Workshops were followed by an innovative strategy whereby M&E and Finance interns were placed directly in nine partner organizations to provide hands-on technical support: Some partners later absorbed interns into full-time positions. All interviewed partners who received interns reported them as helpful for reporting compliance. Strengthening financial management also built partners’ capacity to manage sub granting relationships with local partners. It was not clear why nine partners were not assessed over the life of the project. In FY10, CF worked with partners to establish minimum standards for volunteer programs, including: a) volunteer polices with job specifications, code of conduct and volunteer selection criteria; b) security vetting of all volunteers working with children; c) volunteer incentives standardized with other partners in the local operating area; d) minimum sets of competencies in their technical area of operations; and e) volunteer service under no more than three programs at any given time. Training by Measure Evaluation on Routine Data Quality Assessments (RDQA) in FY11 further enhanced the quality of M&E reports, as CF in turn trained partners in RDQA and performed RDQA on a minimum of 25% of partners. Partner verifications were carried out three times in FY11 and showed improvement in partner M&E systems. Reported figures had less than a 5% margin of error, acceptable by USAID standards. CB for partners was also provided through information sharing meetings and structured exchange visits between partners implementing similar activities. Partners interviewed found the information sharing fora especially useful and reported that relationships built during meetings enhanced their ability to make successful cross referrals. CF used weekly communication emails to update partner directors between meetings and workshops. During the project’s final year, as part of exit strategies CF placed livelihoods interns within eight partner organizations to support a variety of IGA, including grants to schools and development of ISAL groups. The 3 Originally developed by the Uganda Network of AIDS Services Organizations (UNASO) in 2005 Performance Evaluation of the Children First OVC Project for Zimbabwe 12 project assisted 14 partners to source additional funding through support for proposal development and submission. Internal CB and employment development were woven through the project through the hiring of young adults as community based trainers, and of qualified, unemployed youth as interns to augment staff capacity. Evaluators learned about several challenges related to the quality of interactions between WEI and partners. Despite the JIA process, some partners felt that WEI had an incomplete understanding of their activities, tools, and relationships with ministries that could have been better leveraged. Evaluators observed no consistent criteria for ending partnerships and some partners harbored negative feelings about the ways in which partnerships were ended. While some partners experienced funding flexibility, others experienced sudden funding fluctuations without warning or explanation. Partners reported that having to develop annual proposals for CF created delays in funding, planning and implementation. Inconsistent and delayed feedback to partners was cited, particularly where a key CF contact changed over time. Some partners described a lack of professional courtesy in interactions with CF. Also some partners described being pressured to implement activities outside their core areas, while others felt that WEI had always honored the parameters of their scope of services. Evaluators suggest these strategies to assist satisfactory partnerships in the new OVC program.  Develop terms of engagement to support collaborative and mutually respectful relationships;  Ensure that quarterly check-in meetings occur and allow opportunity to resolve concerns and hold one another mutually accountable;  Ensure that all UGM staff with technical oversight over partners demonstrate professional demeanor and technical competence in their interactions;  Ensure that partners have right of approval for use of their organization’s materials, especially those that have been copyrighted. Publicly acknowledge partners for all models, tools and practices that originate with them.  Rather than having partners develop annual proposals, invite annual work plans that allow partners to adapt plans to changes in their operating environments;  Use these work plans, progress toward CB milestones, and other transparent performance criteria to determine partnership continuation and funding levels;  Ensure transparency with partners in terms of UGM reports to and feedback from USAID, e.g., mid-term and final evaluations. Evaluators noted differences in the quality of experiences across partners over time, which may indicate that later partnerships that were rapidly developed to fill gaps noted during the MTR may not have included the same depth of assessment and CB as earlier partnerships. We were struck by the emphasis throughout evaluation interviews that positive relationships are key at all levels – national, district, local, and inter-partner - and we noted the Umzingwane AIDS Network as a CF partner that demonstrated especially positive relationships at district and community levels.4 Additional recommendations to strengthen CB for partners are included in Annex H. It is recommended that in addition to the QI measures undertaken by CF, the Quality Improvement Initiative, a partnership of USAID and OVC implementing partners, be explored for inclusion in future programming5. QI builds on the concept that 'performance is a characteristic of care provision' and that improvement will occur only when changes are made in the system. Adding inputs to a system will only lead to improvement to the extent they can effect change in that system. QI is based on four core principles: 4 The UAN membership model, which joins many community members in an active consortium at a cost of $1 per person, may be worth exploring by other partners. 5 PEPFAR Care that Counts Initiative. Online: http://www.ovcsupport.net/s/index.php?c=22 Performance Evaluation of the Children First OVC Project for Zimbabwe 13 1. Being client centered - keeping in mind that the needs of the children being served are at the core of what is done; 2. Multi-disciplinary team approach - bringing together the range of individuals who make up the team responsible for service delivery efforts to children; 3. Focus on how care is actually provided: examining and modifying the systems and processes used to provide care; 4. Data-based decision making: using data to determine where we are, what are the gaps, what we want to do, and whether the program made a difference. (iii) Timeliness All partners agreed that CF responded rapidly to requests for funding and technical support (especially in the areas of finance and M&E). Partners credited CF with initiating its operations during a period in Zimbabwe’s history that was fraught with social, political and economic instability. During 2008 NGOs were operating in a hostile environment, yet CF managed to support its partners in a timely way as they in turn endeavored to respond to various emergency situations such as outbreaks of cholera. CF also demonstrated timeliness in attempting to adjust and expand strategies in response to MTR recommendations as well as findings from project and other situational assessments. 6.2. What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? This evaluation question relates to CF Key Result Area 1: Access to OVC Services Through Community Initiatives Increased. The evaluation team examined several innovative models of OVC support, which generally fell within the service provision areas of Education, Health or Child Protection, although some models spanned multiple service areas. 5.2.1 Bantwana School Integrated Program (BSIP) In 2010, CF introduced the Bantwana School Integrated Program (BSIP) model in more than 63 schools. The model links provision of education to other care and support interventions to provide holistic, community￾based support to OVC and uses schools as service delivery points for care and support through four key areas: (i) Block Grants to Access Education6; (ii) School Health Assessments (SHA); (iii) business development training and small grants for School Development Association/Committee (SDA/C); and (iv) the Child Rights CD Listener Program (CDLP). (iv) Block grants to support schools and help OVC access education CF partners negotiated with headmasters to waive fees and levies for OVC in exchange for resources needed by their schools such as classroom furniture and ECD playground equipment. According to CF reports, school block grants resulting in 51,749 child-years of support over a period of four years. While BSIP is an innovative strategy to support the retention of vulnerable children in school as well as strengthen school resources, evaluators observed no clear link between the selection process for BEAM for OVC tuition fees and how schools were chosen to receive BSIP grants. There was a clear intent however for BSIP to help fill the significant gap between BEAM’s 800,000 OVC target and actual 6 In the beginning the project made a clear distinction between block grants and direct school fees, however some partners paid out direct school fee and some schools received both. “[Block Grants are] very effective… [they] achieve two goals with one fight, [they] make resources available in schools and give children opportunity to learn. Few resources, big impact” - Partner Interview Performance Evaluation of the Children First OVC Project for Zimbabwe 14 number of students being served.7 Table 2. Block Grants and OVC Reached Year # of Block Grants Provided # of OVC Reached # of Females # of Males Total 2009 58 9,066 8,760 17,826 2010 63 9,232 9,480 18, 712 2011 65 4,009 3,879 7, 888 2012 72 3,664 3,659 7,323 Headmasters reported that resources provided from block grants were beneficial, and per CF reports most partners were able to ensure that children on the block grant tuition lists achieved an 80% attendance rate. The evaluation team believes this initiative needs to be continued until BEAM is able to meet the need of all OVC. To ensure that school block grants are achieving the intended purpose, it is recommended that partners collaborate with schools to monitor the retention of tuition-supported students in school over time. (ii) School Health Assessments (SHA) CF engaged and partnered with the City Health Harare, District Hospital and local clinic staff to conduct SHAs (‘head to toe’ examinations) for 220,000 school child years of support. The assessments included screening for common as well as life-threatening illnesses such as HIV and AIDS related infections. Due to the decline of service delivery systems, City Health in Greater Harare has, since the year 2000, provided SHAs only to Grades 1, 2 and 7; the CF assessments were an attempt to revive a government program that encompassed all primary school children. Block grants were provided to clinics to procure the necessary consumables. In exchange, children who were unable to afford user fees and other associated costs of medications were referred to clinics and received free treatment. In some cases CF directly procured supplies for clinics. Referral forms were collected to monitor how many children were linked with health services through SHAs. Benefits from the SHAs included children referred for HIV testing and ART and treated for communicable infections such as ringworm and for dental caries. However, partners reported that often children referred to hospitals for specialized treatment were not serviced and blamed the Assisted Medical Treatment Orders program (AMTO8) for failing to provide children with free access to treatment at the tertiary level. Partners emphasized that transport support was key to getting children to clinics after SHAs, and transport and salary top-ups were crucial for nurses who conducted the SHAs. Without ongoing salary and transport support for nurses, SHAs will not continue other than in three grades in the Harare City Health area and not at all in rural areas. Health assessments were also conducted for out-of-school learners at ten sites with support from local clinics after CF advocated with City Health. Data from assessments at these sites revealed higher vulnerability levels for out-of-school children than those in formal schools, although disease trends remained the same: scalp ringworm, dental caries and bilharzia. These findings suggest that beneficiaries at the OOSG sites have higher levels of vulnerability and poorer access to basic health care and nutrition than those in formal schools and are 7 A BEAM study carried out in Umzingwane District showed that of 823 children in need, 16.8% were assisted by BEAM, 23.2% by CF block grants, Capernaum and CAMFED put together, and 60% remained unassisted. 8 GoZ offers AMTOs at district or central hospitals to provide health services to vulnerable people five years and older. However, to access AMTO, children need to have been referred to the district or central hospitals from a primary healthcare facility. CF's clinic block grants covered user fees at these sites for children referred from school and ECDC health assessments. Less than 1% of children assessed through school health assessments required specialized treatment not covered under block grants. “They have covered where we have not managed to assist. They have managed to help us give basic treatment.” –City Health representative Performance Evaluation of the Children First OVC Project for Zimbabwe 15 relevant for future SHA programming. Table 3. Number of School Health Assessments and OVC Reached Year # of SHA # of OVC Reached # of Females # of Males Total 2009 86 12, 066 12,692 24,758 2010 91 22,514 24,766 47,280 2011 166 35,369 34,194 69,563 2012 185 41,111 39,585 80,696 (iii) SDA/C Business Development Training and Small Grants As an exit strategy for BSIP block grants, SDA/C members were provided with business training and start-up funds to support income-generating activities (IGAs) such as school tuck-shops and poultry projects. The profits from these businesses were intended to provide ongoing educational assistance to OVC after the CF project ended. CF partners trained a total of 98 SDA/Cs in financial and business management and marketing, and disbursed small grants for them to begin IGAs of their choice. A total of 152 IGAs were implemented by the end of FY11. A percentage of profits from the businesses were used for general school expenses; it was not clear what percentage was agreed on and how this was monitored. It is recommended that the percentage of profit allocated to meet school needs other than OVC fees be capped and monitored in future programming. Clauses to this effect can be incorporated into agreements such as MOUs between schools and partners during the grant negotiation phase. Overall, small grants to schools were an innovative exit strategy, however profits will not meet the future needs of all OVC nor will they cover the same level of school fees funded under the BSIP block grants. Given that this initiative is relatively recent (FY11), there is value in earmarking the IGAs started under this program as pilots and evaluating them for effectiveness prior to the allocation of funds for replication of this model under the new OVC program. (iv) Child Rights CD listener program (CDLP) CF implemented the Child Rights Campaign in 2009 to raise awareness of children’s rights and responsibilities among children and communities. The campaign started with two 26-episode radio dramas that were broadcast nationally on Radio Zimbabwe and subsequently adapted into CD listening sessions. CF provided 82 schools with user friendly robust solar powered units with in-built program chips The CDLP is administered by Psychosocial Support (PSS) teachers who received basic counseling training from CONNECT to equip them to facilitate discussions on issues of children’s rights and responsibilities. Sessions are also conducted by school-based child-led child protection committees (CLCPCs). Interviews with PSS teachers during the evaluation elicited concerns about their depth of training to manage such dialogues and situations that arise, but more importantly confirmed the lack of supportive supervision. The CDLP was credited by many respondents to have significantly raised awareness of children’s rights at both primary and secondary schools. The evaluation elicited multiple reports of association between the listener program and increased demand among participating students to obtain birth certificates. However, the program targeted school children without adequately sensitizing parents, and there were also accounts of increased awareness of children’s rights clashing with family and community expectations. We recommend that “To a certain extent, it is good to let children know of their rights but there is a need to manage that a bit because we got very serious complaints in the community of only teaching children about their rights. Some take it the wrong way. Some children did not want to do chores. It must be explained and managed very well. “ - Primary Partner Interview Performance Evaluation of the Children First OVC Project for Zimbabwe 16 CPCs, parents, guardians and key leaders be made aware of the content of listening sessions through community sensitization sessions before the listening program is initiated at a school. Table 4. Solar Radios and Number of OVC Reached Year # of Schools that Participated # of Schools Provided with Solar Radios # of Solar Radios Provided # of OVC Reached #Females # Males Total 2010 63 23,870 23,852 47,722 2011 126 31,496 29,698 61,194 2012 133 82 82 38,909 38,691 77,600 To provide further PSS support within the CDLP, suggestion boxes were installed at schools so that children could anonymously report issues of child protection. Evaluators found a lack of enthusiasm about this system and learned of one instance where a girl who had been abused complained through the suggestion box, was encouraged by PSS teachers to report the abuse, and then experienced pressure from Victim Friendly Unit (VFU) police and her parents to withdraw the complaint, an outcome that created awkwardness for both child and teachers. Evaluators found PSS teacher motivation lacking; continuing training is desired and needed. Despite information in the FY11 3rd quarter report that PSS teachers receive continuing training and mentoring, there was no evidence of mentorship at two visited sites, nor was it apparent who might provide it, although head teachers were available to support teachers. The CF audit in FY11 suggests that having two PSS teachers per school is not a sustainable practice and encourages peer educators and prefects to take on facilitation of the CDLP. The evaluation team would not endorse this without assurance of a careful peer educator selection process and qualified adult mentoring. We also recommend that the selection process for PSS teachers include children’s input. The listener program was expanded in FY11 to reach children with disabilities, with the content translated by the Braille Institute for the visually impaired and formatted into comic strips in the vernacular for the hearing impaired along with a program of sign language training. Evaluators learned that the comic strips were inappropriate as sign language is English. The team was unable to assess during the evaluation how the Braille version had been received. Any broader implementation of this program would benefit from a wide consultative process to determine appropriate content (there was feedback that some information is unsuitable for children of certain ages), and a guide that indicates age-appropriateness for sessions. 5.2.2 Out of School Study Groups/Children’s Learning Centers and Safe Parks (OOSG/CLASP) The UNICEF Humanitarian Action Report (2008) indicated that Zimbabwe's educational system was characterized by low enrollment rates, declining attendance and completion rates. The report also described low transition rates to secondary school and insufficient learning spaces, teachers, and learning materials and a continual decline in the percentages of children attending primary and secondary school from 2005 to date. Children are out of school for a multitude of reasons, including the inability of families to afford to pay school fees or levies and children not having birth certificates. In 2010 CF began the OOSG, a group study mechanism that could be implemented by community volunteers at existing structures such as community centers and churches. OOSG was intended as a stop gap measure until students could be reintegrated into schools. CF worked with the MOESAC Distance Education Program Correspondence School and curriculum development experts to develop a non-formal accelerated learning curriculum that was aligned with Zimbabwe’s formal school curriculum and relevant to the needs of out of school learners who had been out of school for relatively short periods, as well as those who had never been to school. Table 5. Out-Of-School Study Groups and OVC Reintegrated into Schools Year # of OOSG # of OVC Reached Number of OVC reintegrated into school #Females #Males Total # Females # Males Total Performance Evaluation of the Children First OVC Project for Zimbabwe 17 Sites 2010 4 132 2011 7 533 2 7 9 2012 14 6,80 6,68 1,348 177 Given multiple challenges described by CF staff in trying to merge these very different learning requirements, and based on reviewed student feedback, it appeared likely that one curriculum cannot successfully stretch across such disparate needs. It may be prudent to continue to improve the current curriculum for students with more formal school experience and to design a simpler curriculum that is better suited to students with minimal or no formal school experience. During review of OOSG curriculum materials, the evaluation team noted stigmatizing language including the use of terms ‘victim’ and ‘deaf and dumb.’ We recommend that curriculum review workshops focus on replacing stigmatizing language with empowering language such as “survivor” and “deaf”. The review team should also consider the socio-cultural context of learners and ensure that curriculum material is sensitive to and reflects OVC realities. For example, stating that using bush toilets are ‘bad’ is not useful if children have no other option. Given the number of out-of-school children in Zimbabwe, this initiative has responded to an important need. The use of retired teachers and university students to facilitate study sessions was an innovative practice. The evaluation team noted that the quality of facilitators varied and included unemployed youth. It is recommended that minimum standards for facilitator qualifications be put in place and that facilitators be monitored through visits to learning sites. Evaluators observed that OOSG learners ranging in age from 6 to 16 years were studying for up to four hours and walking to and from study sites with no nutrition provided during this period. It is strongly recommended that some nutrition support (e.g., the fortified drink “maheu”) be provided during OOSG daily study. This could perhaps be incorporated by one or more key nutrition partners in the new OVC program who could work across the districts where OOSG programs are operating. Some OOSG sites were attempting to evolve into Children’s Learning Centers and Safe Parks (CLASP) sites, based on South Africa’s Isibindi Project model. That model provides shelter structures that meet minimum standards and adult volunteers who teach and mentor children and youth and provide psychosocial support. The structure of OOSG sites visited by evaluators varied; some lacked adequate ventilation or appropriate facilities for disabled children such as ramps, and others were appropriate. It is recommended that minimum standards be developed and monitored for all new sites and, where possible, existing sites be brought in line. The new OVC program may wish to explore in-kind contributions from local businesses, as found in the Isibindi model (furniture, food or nutritious snacks, stationery) and ways to publicly acknowledge donations. Over 1,300 children were enrolled into OOSG/CLASP sites across five provinces, with 186 children reintegrated into formal school. An indicator for success of this program will be the number of OOSG students who pass the 2012 national Grade 7 examination. 5.2.3 Early Childhood Development Centers (ECDC) To mitigate the effects of an abrupt cessation of a beneficial wet feeding program during the transition from the POS to the CPF, CF expanded its ECD program to include those centers no longer covered by the POS and helped seven ECDCs create self-sustaining structures, such as nutrition gardens or poultry projects, that would reduce children’s vulnerability to changes in donor funding. Table 6. Early Childhood Development Centers and OVC Reached Year # of ECDCs # of OVC Reached “The teacher if they mark your book they never explain where you went wrong for you to understand. They say you know for yourself. If they explain our understanding will go up and we will pass.” – FDG, OOSG Youth Performance Evaluation of the Children First OVC Project for Zimbabwe # Females # Males Total 2009 28 1,081 1,085 2,166 2010 28 1,667 1,556 3,223 2011 40 1,405 1,288 2,693 2012 42 3,740 3,610 7,350 Through partners, CF supported community, clinic, and school based ECD models that provided toys, learning materials and a safe place for children under the supervision of trained caregivers. Most but not all programs provided a nutritious meal or snack. Kapnek Trust’s ECD program was documented under NAP1 as a “best practice,” partly because of its family-centered approach of parenting dialogues to improve caregiver knowledge of child behavior, care, and abuse and its prevention. OPHID’s clinic based program provided daily health assessments from clinic nurses and adherence support for children on ART. Evaluators recommend that future support for ECDC programs include a review of and attempt to integrate where possible best practices of the various partner ECDC models. Area of Emphasis: Nutrition The Zimbabwe Vulnerability Assessment Committee (ZimVAC) estimates that over 1.6 million people will be food-insecure between January and March 2013, peak hunger months in Zimbabwe. This is a 60% increase from the one million people who needed assistance at the beginning of 2012.9 Whether at an OOSG site, ECD center or during a clinic visit the evaluation team learned that many children were not receiving food and /or nutritional supplementation. This was concerning and especially so for children on ART. To its credit WEI attempted to advocate for OVC to receive food from government and other donors but without success. Evaluators heard that community gardens were operating effectively in several places, including at SRHBC; gardens warrant further investigation as an ongoing nutrition strategy. It is recommended that the future OVC program focus if possible on districts where food is provided by donors (e.g., DFID, WFP) to maximize collaborations to ensure a continuum of care. Evaluators recommend that USAID work with relevant stakeholders to maximize all possible linkages for nutritional support that will benefit vulnerable households including OVC. 5.2.4 Integrated Management of Pediatric AIDS Care and Treatment (IMPACT) Launched in 2008, the IMPACT model emerged from a partnership between WEI, CHAI, and CF partner Seke Rural Home Based Care (SRHBC). The program sought to stimulate demand for and access to pediatric ART services by mobilizing communities and utilizing volunteer networks. CHAI attended to supplies, contributing CD4 machines and other essential testing resources, and CF partners focused on public awareness and recruitment. Through SRHBC and later other partners (Umzingwane AIDS Network and Howard Mission Hospital), children who needed to be initiated on ART were identified and linked to health institutions for counseling, testing and other related services. Support included facilitating access to HIV testing for children, providing transport to district hospitals for consultations and treatment, CD4 testing, and facilitating access to ARVs and cotrimoxazole prophylaxis as well as treatment of opportunistic infections. The model included adherence peer support and support groups, psychosocial support, and community sensitization meetings. During the IMPACT pilots, the average time for a child to be initiated on ART was reduced from three months to two weeks. Notable good practices associated with this model included ART adherence support groups for children, bringing CD4 testing equipment to the community to address transport and clinic capacity constraints, providing a courier service for samples from community to clinic to support rapid test results,, delivery of ARVs to community clinics by community outreach nursing teams, and 9 http://www.wfp.org/countries/zimbabwe “The model that CF is using in terms of moving through communities, creating demand for testing and providing for children is very grass roots. We think that it is very important. Once people come to the clinic like in the model that they are using there is a link to the clinic. Once a child is tested and found to be positive then they go on treatment and CF addresses this loop.” – MOHCW AIDS and TB Unit Representative Performance Evaluation of the Children First OVC Project for Zimbabwe 19 provision of block grants to clinics in exchange for support for children to access free services. Noteworthy and positive outcomes of the model included the initiation of a high number of children on ART (a total of 1,982 children)10 and linking them with case management to ensure that they remained on treatment within a continuum of care. The IMPACT model has helped to highlight the number of children requiring ART and often not getting it. It is important to note the role of local champions at pilot locations who have helped the IMPACT model succeed. Community sensitization and the buy-in of nurses who conduct testing and deliver ARVs are critical for the model’s successful replication. Transport may be a challenge for nurses who wish to replicate delivery of ARVs and samples to community clinics. Evaluators encourage the new OVC program to explore this model further in close collaboration with the national MoHCW pediatric ART program so that children initiated on ART are tracked and clinically supported in a uniform manner. The new OVC program may wish to explore operational research to confirm that this model complements and strengthens national strategies and investigate workable strategies to address transport and nurse sensitization issues. In recent months, CF has made efforts to begin to sensitize pediatric ART stakeholders, particularly through the screening of a video about the IMPACT model at the PMTCT partnership forum chaired by the MoHCW AIDS and TB Unit. Table 7. Number of OVC initiated on ART due to IMPACT Year Targets # of OVC Initiated on ART due to IMPACT # of Females # of Males Total # of Females # of Males Total 2009 75 2010 250 230 480 126 141 267 2011 93 92 185 276 220 496 2012 117 113 230 205 202 407 6.2.5 Community Adolescent Treatment Supporters (CATS) CF supported the Africaid CATS model, which trains HIV-positive youth to use their life experiences and knowledge to identify and support HIV-positive children in their communities. Through home visits and community meetings, CATS volunteers provide children and caregivers with information and counseling to reduce stigma and assist children to adhere to treatment. Under this program, 30 trained peer counselors visits to children to provide adherence psychosocial support. Africaid piloted the model in 2009 and expanded it with CF funding support. By the end of FY12, 1,822 children11 had received peer adherence support. In FY11, the model was nationally recognized and further recognized by the Southern African Development Community (SADC) as a regional “best practice.” Opportunities exist for further scale-up through the new OVC program. To support country ownership and prevent vertical programming, DSS and MoHCW need to continue their support of this simple and inexpensive model. Africaid has received CPF funding for wider replication in 19 districts across three provinces next year. Rigorous evaluation of the roll￾out will provide evidence for sustainability. Important issues to consider during expansion are confidentiality of volunteer peer supporters and children. This model has worked well, but within an extremely protective environment with limited visibility of adolescent volunteers. Any rollout or expansion will need to carefully consider how and whether confidentiality regarding status of both volunteers and beneficiaries can be maintained, and s that 1,982 children. The disparities in reporting are of concern to evaluators. Evaluators were given these numbers but also told that the partner did not track these numbers. Data source is unclear. “Communities don’t know if you are positive. Some caregivers may know. It may be hard if a person is in denial - you may disclose but you do not have to. We don’t have to disclose at every house. It can help to talk more openly between us and the child.” – CATS Volunteer Performance Evaluation of the Children First OVC Project for Zimbabwe 20 how the quality of volunteer training and mentoring can be preserved. It is essential to engage with children about their comfort with visibility, especially regarding HIV status. Children initiated on ART through IMPACT are ideal candidates to receive peer support from CATS volunteers. 5.2.5 Youth-Friendly Corners (YFC) and Expanded Youth Friendly Corners (EYFC) One of the key MTR finding was that CF’s youth programming was relatively weak. The review team recommended that CF explore Youth Friendly Corners (YFC), a model that provides youth with safe spaces to meet and opportunities for sexual and reproductive health (SRH) education, life skills and livelihood activities. CF carried out a Knowledge, Attitudes, Beliefs and Practice baseline survey to assess SRH knowledge and practices among 99 youth from three YFC catchment areas in Umzingwane District. Based on findings, CF began to implement the YFC program through various partners. Youth Peer Educators trained in adolescent SRH education used creative ways to disseminate information, such as sports tournaments and school clubs. Hospital-based YFCs linked 95 youth (26 females and 69 males) in need of STI screening across six clinics with youth-friendly nurses, avoiding scrutiny in adult clinics. YFC facilitated by trained and supervised adolescent peer educators were found to be working well at Howard Mission Hospital despite lack of electricity in the designated hospital space to show films or DVDs. Peer Educators were confident and articulate, and expressed pride in their role and awareness of gender issues related to SRH. They utilized the Aunty Stella Kit, an educational package comprising attractive cards with age￾appropriate SRH education messages. An assessment was carried out by Island Hospice to determine the functionality of the YFC model. The results led to the modification of the model into the Expanded Youth Friendly Corner. CF used 2010 as a planning year by contracting technical experts for livelihoods projects and furnishing the centers with more youth-friendly material. Table 8. Youth Friendly Corners and OVC provided with SRH Education Year # of YFC Number of OVC provided with SRH education # of Females # of Males Total 2009 8 466 434 900 2010 8 0 0 0 2011 7 1,420 1,883 3,303 2012 10 585 512 1,097 EYFC usually take place in outdoor community gathering areas and offer a platform to reach out of school youth with health education. Partners implementing SRH education worked with the Zimbabwe National Family Planning Council and the Southern Africa HIV and Aids Information Dissemination Services (SafAids) to train 168 peer educators at EYFC. CF reported that in 2011, 16,848 child years of support were provided for abstinence, sexuality, STIs, and HIV prevention education. The evaluation team met with youth at a number of EYFC. In most cases, participants were young adults above 18 years of age who were meeting with a young adult facilitator, who provided the SRH education. A few sites offered sport activities. Overall, evaluators found the EYFC groups to be rather dispirited. While it is appreciated that sport activities do indeed draw youth and create education opportunities, there is no evidence as to whether SRH information provided in this way has resulted in behavior changes. Per CF reports, YFC programs provided 62,317 child years of support. CF’s former partner Grassroots Soccer reached 8,749 students through its ten-week coaching and mentoring program. However, this model has limited geographic reach and is not a likely candidate for broad national replication. During 2011, CF expanded its YFC strategy to include a livelihoods component (see p. 40). As youth over 18 years of age are not included in PEPFAR OVC targets, the new OVC program may wish to encourage other USAID funding streams to create livelihoods opportunities for older youth. 5.2.6 Community-Based Case Management (CM) “When the project started we used to get population funds (USAID) which allowed us to work with youth. But now we have another mechanism looking at family planning so that fund was withdrawn. Youth above 18 are not a focus.” - USAID colleague Performance Evaluation of the Children First OVC Project for Zimbabwe 21 Case management (CM) is a well-established social work methodology which aims to ensure that needy families and children are identified, referrals are made and a comprehensive package of services is delivered. When case management is effectively in place, linkages and referrals cut across judicial, social service, education and health systems to ‘catch’ vulnerable children and provide effective support to them, usually within the remit of the DSS. Acknowledging that many partners had already integrated CM in their programs, CF invited three partners – Howard Mission Hospital, Seke Rural Home Based Care (SRHBC), and Umzingwane AIDS Network (UAN) in partnership with Umzingwane DSS – to pilot the CM model in peri￾urban and rural settings. The pilot adapted best practices from the Isibindi model in South Africa, which is based on harnessing community resources and culture in the care of vulnerable children. Each community child care worker works with an average of ten families at any given time. Families that were case managed under this pilot program were identified from block grant beneficiaries and the village registers.12 Most households under the CM pilot were headed by grandmothers or other old aged caregivers, indicating that DSS should pay particular attention to these types of households as vulnerable and at-risk. CF supported partners to strengthen documentation and referrals to ensure that OVC remained tracked until at least their basic needs were met. Using a network of resources and models including Child Protection Committees (CPCs) and village registers, volunteer community care workers, clinics and hospitals (IMPACT), schools (BSIP), paralegals, and the CSI tool to assess child well-being, the CM approach demonstrated impact in identifying and serving children requiring child protection. Evaluators heard from community care workers who had successfully captured information on needy children, referred children and families to relevant authorities and received feedback that services had been received, especially for birth registration. Although many children did access services, the evaluation determined that the demand for services largely outstripped supply. Applications for BEAM assistance and referrals to hospitals for operations were more often than not unsuccessful. CM files reviewed by evaluators in Umzingwane District recorded referrals for medical and financial assistance, but no follow-on notes were found in the files, and care workers reported unsatisfactory results with no assistance provided. Feedback within the CM model was sporadic and inconsistent and often relied on informal relationships between community and staff members. All closed files seen by the evaluation team indicated that families had moved out of the area or that children had died. At other sites, files were incomplete and out of date. Consequently files have remained open but inactive, and the morale of both family and care worker is low from having identified needs that were then not met. In CM terms, this may be argued as still having value in that children were identified for assistance, but this is an indication that the CM system needs to also perform a results-based function if it is to ensure that services are obtained by needy families. Evaluators also recommend a review of all tools (including the Child Status Index (CSI) and My Life Now case management tool) to select the most appropriate tool for a family rather than individual child focus in keeping with the DSS and CPF emphases on strengthening vulnerable households. CM forms varied from site to site, quality of completion was erratic, and forms did not encompass the entire continuum of care, often simply indicating what the referral was for. Family information gathered on CM forms was minimal and added little to the development of a holistic care plan for the child. It is recommended that a family-centered approach be embedded in all CM that is cognizant of relationships, extended family network, custody and guardian issues. Intake and referral forms would benefit from use of a family tree or geneogram that addresses the needs of the child within a family construct. Emotional attachments, discipline patterns, financial responsibility and cultural/spiritual influence should all be noted. Training in use of the geneogram at 12The OVC area register is a DSS-led initiative that allows communities to create lists of their vulnerable children most in need of assistance. These children can then be easily identified for access to government programs, such as BEAM, as well as other NGO assistance. “A few were referred on to hospitals. We tried to lobby with hospitals and doctors together with CF but we did not get much of a breakthrough…the AMTO (free medical) did not work.” - Primary Partner Interview Performance Evaluation of the Children First OVC Project for Zimbabwe 22 community level in-country has been successfully undertaken by at least one non-CF partner13, and a manual developed by HOSPAZ has a geneogram training section that could be rolled out by all partners. It is further recommended that a review of referral forms and feedback loops be undertaken to identify all resources along the continuum of care so that the care worker has a ready checklist of all services that the child should be entitled to and may need.14 Counseling training institutes (CONNECT and CONTACT) should employ a family-centered approach in their training of all partners. For FY11, CF targets for CM were for 100% of children who required ART, 20% served by block grants and 10% by other programs. CF actually provided CM for all children under the IMPACT model and OOSG sites and 10% of children receiving educational assistance under block grants. DSS has publicized its intention to roll out this model as a national CM system under the second phase of the NAP 2 for OVC, with WEI funded by the CPF to collaborate with DSS through this process. The CM rollout, planned across ten districts each year in alignment with social cash transfers, urgently needs to include an emphasis on putting more referral resources in place. Evaluators who visited OVC service sites at Caledonia and Hopley Farm locations noted a disturbing lack of basic water and sanitation services that put children at risk. It is recommended that the new OVC program explore links with relevant authorities to address these problems on behalf of extremely vulnerable OVC living in these environments. Area of Emphasis: Supportive Supervision to Strengthen Child Protection The CM model being rolled out by DSS with WEI support is premised upon use of community volunteers to perform a range of child protection activities described within this report. Community care workers are likely to work with up to ten families at any time (CF final report) and reports from CPC members indicate that they may encounter a range of situations including child abuse in all forms, neglect, illness, poverty, disability, domestic violence, substance abuse, marital discord, depression and bereavement. This is no easy task, even for well qualified, experienced social workers. It is all the more serious and daunting when working in a disabling environment with minimal training, few resources, and without practical support in the form of effective referral networks. All care providers should receive meaningful support to prevent compassion fatigue that arises with cumulative exposure to distress and suffering of others. This in turn can result ultimately in burnout, a syndrome of physical and emotional exhaustion. The long-term effects of burnout can be permanent and prevent continuation of the care worker role. The evaluation team noted that there are no supervision guidelines embedded in the CF project for the benefit of partners, and supervision of care workers and professional health staff is dependent on partner protocols. Evaluators asked questions to assess the effectiveness of current supervision systems and explore how volunteer care workers avoid burnout. Findings suggested that most care workers receive some practical supervision in the form of ongoing training (such as how to use the CDLP) and that details of cases visited are noted and problems are discussed, usually in a group setting. Many care workers had difficulty understanding the concept of emotional support as part of supportive supervision and confirmed that this was rarely available and that they tend to seek support from church and family members. Volunteers attached to Africaid and Childline understood the concept of supportive supervision, and the latter receive regular debriefing sessions from line managers and from a consultant psychologist contracted by the organization. These examples offer good practice for future programming. Evaluators recommend that CF explore supervision and mentoring opportunities that can be integrated into all programs and that draw on modeling techniques that enable community child care workers to learn from experienced colleagues. The new OVC program is encouraged to contract with organizations that offer 13 FACT Mutare Palliative Care Project in 3 Sites in Manicaland 2012 14 Refer to the Zimbabwe National Home-Based Care Training Manual, p. 137, for a form that could be revised to include a stronger family focus, and to the Client Profile Form on p. 132. Performance Evaluation of the Children First OVC Project for Zimbabwe 23 supervision training and investigate the possible use of retired, certified social workers to provide supportive supervision. 5.2.7 Community-Based Counseling program (CBC) CF partner Childline provided crisis counseling services to children through drop-in centers (DIC), a hotline call center (24 hour toll-free phone-in service for child crisis reports or anonymous dialogue) and referrals from police, schools and the wider community. CF expanded the DIC as a pilot into the high-density area of Mabvuku and later into five other sites. The initial six sites were reduced to four sites for the last two years. Nevertheless, the increase in number of DICs is regarded by Childline as having raised awareness of child protection issues in communities where they are based, due to expanded volunteer activism and counseling. Annual targets were exceeded. CF’s promotion of this model recognizes that children and families who access DICs are afforded quality counseling by social workers and trained volunteers familiar with community networks. Alternative efforts made by CF to expand on the CBC concept with a separate call-in radio program and SMS service, drawing on counseling expertise from Childline, proved largely unsuccessful due to inadequate buy-in from relevant authorities. The CBC program experience underscores the necessity for a standard of counseling, as well as the value of partners remaining in their core areas of expertise. Expansion of DICs is a replicable approach contingent upon adequate capacity of trained social workers and volunteer counselors to staff the sites. 5.2.8 Child Protection Committees (CPCs) CPCs bring together and leverage the collective efforts of government institutions, the private sector, non￾governmental organizations, civil society and communities to monitor and respond to the needs of vulnerable children within their communities. Under the leadership of DSS, CPCs are responsible for driving the implementation of NAP2 child protection activities and have been mandated by DSS to compile village registers to identify and manage the care of orphans and other vulnerable children, including the disabled, in their designated areas. Due to the instability in the country in recent years, CPCs have failed to function effectively. CF partners attempted to revitalize these important committees and link them with CM to promote the continuum of care for vulnerable children. Despite good efforts to achieve this, CPCs visited during the evaluation lacked resources to operate effectively and some were not meeting regularly. In particular, CPC members reported lack of resources (transport, identification or stipends) to develop, manage or update village registers. Members reported that some disabled children appear on registers, yet are not receiving crucial services such as BEAM. These findings raised concern about whether the planned DSS case management roll-out strategy includes resources to update and manage village registers and put in place more referral services. One potential strategy to resource CPCs to perform their function is to adapt the ISAL model to include a contribution for CPC functioning. CF has provided financial support to the DSS to develop a CPC protocol. To help CPCs harness the collective efforts of key government ministries, NGOs and civil society to address the plight of OVCs. The protocol defines the quality, standards and responsibilities of CPCs and includes a code of conduct and ethics. The new OVC program could usefully support the rollout of this protocol through training and sensitizing CPC members, in conjunction with innovative strategies to resource the committees to enable their full function, including some form of identification. “We learnt that by crying they are emptying their anger so if you stop them crying then you are keeping the anger in”. - FGD participant trained in community based counseling. “Right now we are supposed to be compiling an OVC register to know how many orphans and vulnerable children are living [here], if they are going to school, if they have fees. The social welfare said that it’s a community thing. You have the data with no assistance. Right now we have failed to complete the register. They did not look at the area of focal persons to collect this information door to door.” – CPC Member Performance Evaluation of the Children First OVC Project for Zimbabwe 24 6.3. Country Ownership and Sustainability: How effective was the CF project in developing local capacity for local partner organizations and communities to effectively meet the needs of OVC? This evaluation question relates directly to CF Project Key Result Area 2: Human Capacity in Local Community Structures to Meet Needs of OVC Strengthened and Key Result Area 3: Community and National Level Advocacy for Social Protection of OVC Improved. CF selected partners who could bring technical expertise in particular OVC service domains, seven of whom sub-granted to smaller community-based or faith-based organizations. In this way, CF helped to ensure that resources, technical support and increased capacity were directed to the frontline service providers who are best positioned to assist OVC. CF engaged in widespread advocacy at community level that was inextricably linked with community capacity building. In addition to the school based CDLP, the project conducted a situation analysis of the needs of urban children, evaluated services available to children who come into conflict with the law, and formed Child Law Clubs at schools, strengthened relationships with the Registrar’s Office for birth certificates, and trained and capacitated multiple volunteer cadres and helped to revitalize CPCs. Community advocacy that enhanced program visibility included participation in commemorative events to mobilize communities and give children a platform to highlight their issues, such as International Day of the African Day Child and International Day of the Girl Child, engagement with radio and television through multiple advocacy campaigns; public launches for handover of sports equipment for the OOSG/CLASP program; and launch of a child friendly counseling room and PEP awareness song recorded by a popular artist for adolescents at the Gwanda Provincial Hospital. In its final year, CF launched a “Be the Change for Children” art and games campaign to raise awareness and resources for out of school children and youth. 6.3.1 Capacity Building for Volunteer Cadres (i) CB for Psychosocial Support (PSS) and Counseling: The CF 2008 situational analysis 15identified the crucial role of psychosocial support (PSS) in addressing children’s most pressing needs of love, care and respect. PSS can be understood as support for the emotional and social aspects of a child’s life in order that the child may live with dignity and hope, and is best provided by families and communities. Good communication and counseling skills are necessary for PSS to be effectively provided. CF incorporated partners with PSS experience, such as Kapnek Trust, Family Support Trust and Childline. Technical support for counseling was provided by local counseling training organizations CONNECT and CONTACT. A total of 268 people were trained in counseling across nine partners. However, the evaluation team heard community care workers, nurses and other cadres express lack of confidence about their child counseling skills, and concern about the lack of supportive supervision offered for such responsible work. Given the enormous demand for quality counseling training in this project, evaluators recommend that organizations geared to provide PSS, such as REPSSI, be included as technical partners in the new program to strengthen both counseling and supportive supervision. The majority of children served by the CF project were single (one parent deceased) or double (both parents deceased) orphans. Orphanhood creates significant loss and bereavement; evaluators were surprised that the project focused minimally on both. Apart from Africaid trainings on grief and a brief introduction to the topic provided in a training manual used by HOSPAZ16, the evaluation team found little evidence that the emotional needs of bereaved families and children had been specifically addressed and recommend that the new OVC 15 Ndlovu P., Ruparanganda W., Mutyambizi T. (2008) Children First Project. Situational Analysis and Mapping Survey on Street Children and Children Infected with HIV and AIDS in Harare. 16 HOSPAZ and Ministry of Health and Child Welfare (2006). Palliative Care for Children: A Training Manual for Communities in Zimbabwe. Harare: UNICEF. Performance Evaluation of the Children First OVC Project for Zimbabwe 25 program strengthen counseling materials for volunteer cadres to include state of the art bereavement counseling curricula (African Palliative Care Association and South Africa’s Hospice Palliative Care Association are good curriculum resources). (ii) CB for Child Care Workers: A training-of-trainers was provided for CF volunteer child care workers by South Africa’s National Association of Child Care Workers (NACCW), which has worked toward certifying a similar new paraprofessional cadre in South Africa. The training is intended to be cascaded through communities as part of a strategy to integrate components of the NACCW Isibindi Safe Park model into OOSG/CLASP model. No information on cascade progress was available to evaluators. To maintain the level of competency for child care workers, minimum standards for Community Child Care Workers were developed by the Council of Social Workers (CSW) with the support of CF. The new OVC program could strategize ways to support the implementation of these standards. Consideration is underway as to how to register this volunteer cadre to recognize their efforts and ensure accountability; South Africa’s example through NACCW may be useful. (iii) CB for Community Paralegals Children’s access to legal justice in Zimbabwe is hampered by prohibitive costs, as well as ignorance of laws and mechanisms in place to protect them. CF developed the Community Paralegals Program with partner Justice for Children (JCT) in 2010 as a community level initiative to strengthen access to legal services for OVC. With CF support, JCT trained 12 volunteer community paralegals in Harare on the basics of laws that protect children. Volunteers were selected from FBOs and CBOs working with children and were intended to be the first contact for children in need of legal services. Training modules derived from the STRIVE Project and included child maintenance, children with special needs, birth registration, child abuse and guardianship, custody and access. Paralegals provided services related to legal bottlenecks in the community such as birth registration, inheritance and custody, and also provided community awareness sessions on child legislation. Per CF reports, the Paralegal program worked well in assisting children to obtain birth certificates and making referrals to JCT lawyers for further assistance. The CF database indicates that 2,927 cases were handled by community paralegals, including 1,856 birth registration cases, 278 maintenance cases, 254 custody cases, and 161 abuse cases. Evaluators had no opportunity to see this program under way in the field. We did ask what support was in place at community level for paralegals and were informed that a District Social Services Officer employed by DSS would work directly with paralegals. Evaluators found the concept of paralegals innovative, although we were unable to observe the model’s effectiveness. JCT has expanded this program to four more provinces with funding from other donors and, per WEI, DSS is implementing a similar model in two districts within each province. Reportedly, discussion is underway with the Law Society of Zimbabwe to certify community paralegals. A comparative evaluation of the different models would be useful for future programming. Community volunteer cadres interviewed during the evaluation expressed positive appreciation for knowledge and skills received during various CB trainings. However, there was no evident training follow up or monitoring of demonstrated skills increase as a result of trainings. Post training follow up needs to be included as an essential CB component in new programming. 6.3.2 Capacity Building for Communities to Address Harmful Practices Evaluators were struck by the reported high incidence of child abuse including sexual abuse in Zimbabwe and urge the new OVC program to align with CPF to utilize opportunities for community mobilization models such as the UNDP Structured Community Conversations to address issues of child protection, including child sexual abuse at community level. Structured, facilitated community mobilization can assist community members to discuss compelling issues, first in age and gender specific peer groups and then more openly in community conversations that reach across traditional age, gender or social status barriers. Community mobilization goes Performance Evaluation of the Children First OVC Project for Zimbabwe 26 further than community awareness to create opportunities for communities to assess problems and solutions from multiple perspectives. Evaluators noted that Howard Mission Hospital was the only partner who had implemented a crisis foster care model for children who needed to be moved quickly out of dangerous home situations and encourage the replication of this protective service in the new program where possible, coupled with an emergency fund to support such crisis foster where it is most urgently needed. Evaluators heard reports of HIV testing being mandated as part of awareness raising associated with CF activities. We strongly encourage the new program to emphasize a rights based approach in community mobilization so that no one is coerced to test for HIV or disclose HV status. The team also noted that the early CF focus on urban street children, highlighted in the 2008 Situation Analysis, seemed to disappear as partners were able to move into communities outside of Harare. Given the ongoing reality of homeless and abandoned street children and youth in Greater Harare, we encourage the new program to restore a focus and services for these vulnerable OVC. 6.3.3 Building Community Economic Capacity to Support OVC CF introduced granting mechanisms through partners to community structures in FY 11 as an exit strategy to position communities to continue to provide OVC support. Small Challenge Grants offered targeted in-kind and cash support to community-based groups to help them with disability programming, interventions for out￾of-school children and strengthening of community care networks for OVC. CF’s rationale was that small grants could be more easily-absorbed and managed by small community groups including CPCs. By the end of FY12, CF had disbursed a total of $2,703,403 USD in grants through partners to 61 ECDC, FBOs and CBOs and 121 community structures. Evaluators did not hear of plans to monitor the success and correct use of grants, and recommend that grants of any size included in the new program include measures of accountability and funding effectiveness. Some livelihoods strategies were targeted to older youth, as an attempt to fill programming gaps and help them gain an economic foothold. In FY12, CF collaborated with Junior Achievement Zimbabwe (JAZ) to train in- and out-of-school youth in entrepreneurship and business management skills. The JAZ model encouraged youths to form small groups, save money and inject it into a business as shared capital, start up a common enterprise, and learn business principles while managing the enterprise. Per reports, 243 out-of-school youth received this training. None of the trained youth interviewed by evaluators had been able to secure the cash needed to carry out their business plans. The team also met with youth who had received small direct loans from partners to start businesses of their choice; only one appeared to be having success. Without exception youth complained that having a business plan without start-up funding led to failure. Groups that received resources like sewing machines but no maintenance support were unable to continue activities. The evaluation team noted that in CF’s evaluation of the MOYDIE ISOP model17 only 5% of youth trained through this national program go on to engage in entrepreneurial activities. In contrast, of the 1,533 youth who received business training through CF, 243 (16%) engaged in businesses as a direct result of the training. Across strategies, success rates have been low. The ZimWorks program about to be piloted by WEI may show more promise in that it attaches trained youth directly to businesses and/or CBOs for internship experience. Given that the model targets youth mostly older than 18 years, it is not directly relevant for new OVC programming, although USAID may wish to take pilot results into account for other economic strengthening initiatives. Evaluators observed that the most innovative livelihoods strategy for sustainability has been the introduction of the CARE Internal Savings and Lendings (ISAL) model to improve livelihoods of caregivers and some older 17 The Integrated Skills Outreach Programme (ISOP) is a Zimbabwean government initiative rolled out in 2006 to build the vocational and entrepreneurial skills of young people in rural and urban areas through 4 ministries. Performance Evaluation of the Children First OVC Project for Zimbabwe 27 OVC. This model, which is also underway through other non-OVC national economic strengthening initiatives, trains self-selected groups of up to ten members to pool money into a revolving loan fund that is returned regularly with interest. Evaluators met with a large number of ISAL clubs that are doing well, but also met with community care workers and CPC members who were unable to begin groups because they lacked the start￾up funds. As this model is intended to primarily benefit vulnerable children and households and volunteer caregivers, evaluators invited thriving clubs to suggest ways to make the model more inclusive of the most vulnerable community members. Of particular merit for future programming was the following recommendation: “Once a group is well established and has earned interest on its principal investment, the group can use that interest as a scholarship to invite a vulnerable person without cash to join the group. The pooled interest can be used both for the initial membership payment and also for the first loan to the new member. This initial loan might be for a smaller amount than the pooled amount that is circulating monthly. That way, if the new member is not successful for any reason the group will not have lost its core profits. More experienced members can mentor younger or less experienced members as they build their businesses.” The OVC program may wish to incorporate shared community funds for the benefit of OVC into the ISAL model, e.g., a portion of profits to be directed to support CPCs. An informal market analysis could help to prevent saturation of local markets; innovative local opportunities that meet the needs of OVC (e.g., provision of clean rags and laundry detergent for sanitary ware) should be explored. Table 9. Number of OVC supported with Livelihoods Year # of youth provided with business training # of youth that engaged in businesses as a result of the training # females # Males Total # Females # Males Total 2011 688 688 2012 1533 134 109 243 6.3.4 CB on Behalf of Children with Special Needs During FY 11, CF expanded its program for children with special needs through small grants to strengthen resource units in Neve and Emerald Hill Schools for the Deaf. Under CF’s program, the Braille Institute translated child rights materials for vision-impaired children. The CDLP was translated into a comic book format for deaf children. Under CF’s advocacy, agencies focusing on children with special needs were brought together under the Disabled Children’s Initiative of Zimbabwe (DCIZ) to develop a collaborative framework and strategic plan under which to pool resources, raise awareness and advocate for the interests of children with special needs. The project initiated sign language training for teachers and police officers to enable better communication with deaf children and youth. The project further supported Emerald Hill and Nzeve Schools for Deaf Children to train teachers in the Greater Harare area to work with Deaf students in their general classrooms. More teachers have requested this training from the schools, and the new OVC program could usefully support both the teacher trainings and follow up supportive visits to classrooms. The evaluation team observed a community support group for parents of disabled children facilitated by Kapnek Trust that was very helpful to parents, and recommend that this model be replicated at community level by any future partners with the capacity to implement such an intervention. 6.3.5 Meeting the Needs of Girls and Boys CF addressed the needs and rights of girl children and adolescents through multiple strategies. The CDLP and Peer Educator trainings emphasized gender equality and equal participation. Evaluators met with adolescent female peer educators at four sites (including CATS, YFC and child-led CPCs) who spoke with confidence and Performance Evaluation of the Children First OVC Project for Zimbabwe 28 demonstrated their peer education skills. Skills building strategies for Deaf children included sewing classes and Shona sculpture that were equally inclusive of girl and boy students. Some CF partners distributed sanitary ware for girl students, based on evidence that monthly menstrual periods reduced school attendance due to lack of this resource. Youth sports activities under YFC and EYFC encouraged equal participation of female and male youth. Evaluators recorded genders of FGD participants and noted a good balance between female and male children and youth.. CF aimed to reach 52% females and 48% males, and in fact reached 69% of females and 31% males. 6.3.6 Child Participation Article 12 of the United Nations Convention on the Rights of the Child (UNCRC Article 12) and the African Charter on the Rights and Welfare of the Child (ACRWC Article 7) make provision for involvement and consultation of children on matters that affect them. Both recognize that children and youth under 18 are capable of actively contributing to programs that advance their own protection and well-being. Through its partners, CF promoted child participation through Child Led CPCs, the CATS model, Child Law Clubs, YFC and EYFC peer educators, and the establishment of child advisory boards for CF and all partner organizations. The benefits of child participation are many, including giving children a voice in line with their rights. The evaluation team however identified associated risks, especially partners asking children to be the liaison between abused children and authorities. This responsibility may place children who ‘blow the whistle’ at risk of secondary abuse, hostility and discrimination. Expecting children to take on intensive counseling was also cause for concern given the absence of supportive supervision noted earlier. Before asking children to take on such roles, the OVC program should consider whether this is an age appropriate responsibility. The Africaid model was remarkable in its understanding of PSS impact on children and its commitment to use only highly qualified and skilled personnel in its activities while accommodating child participation and activism. Risk mitigation and protection need to be part of all strategies that invite children to report abuses and identify perpetrators. There is also a need to balance visibility that helps OVC access services with protection of child confidentiality and privacy. 6.4 To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? CF has worked closely with ministries and with MOLSS/DSS in particular to strengthen government social protection systems on multiple fronts and institutionalize project models and good practices and standards within relevant ministries where possible. The project hosted site visits to OOSG/CLASP site for MOESAC and DSS officials. The FY11 annual CF stakeholder’s workshop enabled partners to present OVC support models and to consider possible model consolidation and expansion with MOESAC, MOLSS and MOYDIE officials. CF also presented the IMPACT model at a PMTCT forum hosted by the MOHCW AIDS and TB Unit. A key institutional strengthening challenge experienced by CF was the lack of coordination between Ministries that play vital roles in meeting the comprehensive needs of OVC. CF’s accomplishments in organizing well attended multi-Ministry and stakeholder workshops18 were particularly noteworthy. Systems strengthening and institutionalization project efforts included:  Supporting a strategic planning workshop for DSS delegates, private consultants and CF representatives for development of a costed five-year strategic plan for DSS;  Supporting an institutional capacity assessment/skills audit for DSS;  Supporting DSS to meet critical skills and staffing gaps by enrolling district officers with social sciences qualifications in social work program courses at Women’s University of Africa (WUA). CF hoped this would effectively circumvent a recruitment freeze while placing social workers in priority districts to align 18 E.g., the 2011 CF Annual Stakeholders Workshop “Partnering with Government for Sustainable OVC Programming” Performance Evaluation of the Children First OVC Project for Zimbabwe 29 with the rollout of the harmonized social cash transfer and case management program. The placement goal is ten districts per year.19  Working with WUA to adapt its social work modules into an innovative block release format that enables DSS officers to pursue degrees without interrupting important work in districts;  Working in partnership with DSS to develop a national CPC protocol;  Working in partnership with the CSW to develop national standards for community counseling;  Facilitating a workshop with MOESAC to develop an ECDC policy and curriculum;  Working with the Government Correspondence School under MOESAC to revise Zimbabwe’s distance learning curriculum and provide an accelerated learning program for children and youth outside the school system (OOSG);  Collaborating with University of Zimbabwe's (UZ) Adult Education Program to develop a Master Training Manual for OOSG/CLASP sites. The Master Training Manual will provide a comprehensive curriculum for facilitators and on-site trainers. CF independently developed a Child Protection Manual to be incorporated into the Community Learning Centers Master Training Manual;  Resuscitating and extending the SHA program in urban and some rural districts including Umzingwane and Mhondoro, reaching 46,947 child years of support;  Supporting MOYDIE to evaluate and redesign ISOP to better meet skills development needs of unemployed youth. MOYDIE reported that the CF evaluation report opened program strengthening conversations with UNICEF, UNDP, and other ministries;  Engaging DCIZ to support the formation of a coalition to address issues regarding access to education for children with special needs  Setting up child advisory boards for CF and all partner organizations which meet regularly  Engaging relevant Ministries to understand challenges associated in accessing AMTO assistance  Presenting SHA findings to the Pediatric Association of Zimbabwe (PAZ) in a bid to sensitize pediatricians on areas where they could be of assistance With respect to institutionalization of CF models within government: The BSIP education model depends upon multi-sectoral financial and human resource capacity to maintain momentum for fee payments, health assessments and grants to sustain functioning SDAs. Without continuing support from a program such as CF this is not yet sustainable. The CDLP has been well integrated into schools and institutionalization of this model should continue with refinements in implementation per report recommendations. The new program should work with the CPF to see how its efforts to strengthen BEAM can best be complemented by OVC program support. The OOSG/CLASP initiative has achieved some measure of institutionalization exemplified by development of curricula and some success in reintegration of pupils into mainstream school. Strategies to recruit and monitor qualified facilitators are necessary to ensure sustainability. The curriculum may need to be divided into two sets of materials, one for students out of school for short periods who have fallen behind and another for learners who have never been in the formal school system. Communities may be able to provide more in kind support for learning sites, e.g. furniture, stationery places, nutritious snacks. The ECDCs supported at community centers, schools and clinics are aligned with a national policy but have depended on CF and partner funding to supply educationally stimulating toys, equipment and learning materials. Likewise, feeding programs have been partner-dependent and there is no indication that this model has been institutionalized at any meaningful level. 19With a nearly 50,000:1 children per DSS social worker ratio, major understaffing has affected DSS's ability to conduct community-level work and revitalize community-level structures such as CPCs. Performance Evaluation of the Children First OVC Project for Zimbabwe 30 While MoH has expressed interest in the IMPACT pediatric ART model, the financial and human resources required to integrate the model within national protocols are unlikely to be mustered by the GoZ in the near future. This jeopardizes institutionalization, although implementing partners appear dedicated to ensuring sustainability within their programs as far as possible. The CATS model developed by Africaid has been recognized as a regional ‘best practice’. This offers impetus for ongoing national ownership. Institutionalization of this effective peer support model is best guaranteed by recent CPF funding for further expansion. Quality retention and protection of adolescent treatment supporters during expansion is Africaid’s main concern. The EYFC model that has taken services for youth out of the clinic/hospital setting and into the community faces challenges. There is no clear donor funding stream for youth above the age of 18 years and no consolidated national focus on how best to serve older youth. The YFC model appeared to work well in relatively well resourced settings such as Howard Mission Hospital. The CM model was piloted by three CF partners. Based upon promising outcomes, the DSS will collaborate with WEI with CPF support to roll out this model as a national CM system under the second phase of the NAP 2 for OVC. The model will be institutionalized across ten districts each year in alignment with social cash transfers, however at present it is seriously under resourced in terms of adequate for children and supportive supervision for care workers. Potential exists for institutionalizing the CBC model developed by Childline in collaboration with DSS, however this is contingent upon the availability of adequate human resources and ongoing counselor training, mentoring and supervision, which DSS is not now resourced to provide. Under NAP2 CPCs are poised to take on the task of child protection within their communities. Capacity to carry out this role depends on financial and human resource support from the DSS not yet in place. Momentum accomplished through CF trainings and support for development of the CPC Protocol will require ongoing donor support to continue. Other Relevant Findings WEI engaged Innovative Minds to conduct a Linear Monitoring and Evaluation (LIME) analysis to assess the impact of CF programs on children's lives over the years. Data was collected on 2,562 children supported by the CF project under seven partners. Children were from Harare (54.3%), Umzingwane (25%) and Goromonzi (19.3%); 17.7% of supported children showed improved CSI ratings since 2009. Improvements were noted to varying degrees across categories of food and nutrition, shelter and care, protection, health care and education. Although the proportion of children facing food insecurity and malnutrition had improved since 2009, the analysis revealed a significant reduction in the nutritional status of many children, especially in Umzingwane District. This drop in nutritional status could be attributed to the end of POS support, which provided wet feeding in many schools and ECD centers. The assessment finding is in line with this evaluation team’s findings that many children on ART and at OOSGs and ECDCs were not receiving needed nutritional support. The CF project has provided partners and USAID with useful opportunities to explore a wide range of strategies for an effective continuum of OVC care. While the multiplicity of activities could be viewed as a scattered response, evaluators acknowledge numerous innovative attempts to find effective solutions within a fragmented environment and believe that all explorations at national, district and community levels have yielded useful learning for future programming. At the same time, we recommend that the new program implement a more strategic focus based on lessons learned, with diligent evaluation for all new, untried areas. Recognizing that the GoZ is not yet in a position to take up most of the services currently supported by the USAID OVC program, and given the meager budget available for broader social protection (through cash transfers and BEAM), we emphasize that the next five-year OVC funding phase is critical to maintain the momentum that the CF project has begun. Performance Evaluation of the Children First OVC Project for Zimbabwe 31 Evaluators note that CF program activities have attempted to reflect the PEPFAR II shift in focus from emergency relief to development strategies that are intended to reinforce the sustainability of national HIV programs, insofar as the national operating environment has made it possible to do so. The following strategies were found to be particularly innovative:  CATS Peer support role modeling for adherence (Africaid);  Support Groups for Parents of Disabled Children (KAPNEK);  Nurse assessments and ART adherence support at ECDCs (OPHID);  Foster Placements for crisis situations (Howard Mission Hospital);  Training of teachers to work with deaf students in general classrooms (Emerald Hill and Nzeve Schools);  Integrated programming across education, health, child protection areas (Howard Mission);  Lab in the community for CD4 testing which could be explored for other testing (Seke Rural Home Based Care and Chitungwiza District Hospital). Finally, we highlight these CF achievements as especially notable: • Exceeded all targets but one (% of children showing improvement according to the CSI); • Responsive to mid-term review recommendations; • Strong relationships with GoZ ministries, especially DSS; • Useful engagement with multiple ministries and stakeholders to find collaborative ways to strengthen systems and link ministries to promote a continuum of care; • Visible, measurable improvements in situation of OVC, particularly increased number of children with birth certificates, increased number of children initiated on ART and increased number of cases of child sexual and other abuse reported. 7. EVALUATION RECOMMENDATIONS Key recommendations provided here are relevant for OVC program strengthening across all service domains. Additional, actionable recommendations, including recommendations to strengthen OVC support and improve partnerships, are included as Annex H.  Work with partners to utilize appropriate data management systems that have the functionality to track and analyze longitudinal data.  Explore the USAID Quality Improvement Initiative for inclusion in future OVC programming.  USAID work with relevant stakeholders to maximize all linkages for nutritional support that will benefit OVC and vulnerable households.  Integrate a family-centered approach in Case Management that is cognizant of relationships, extended family network, custody and guardian issues. With DSS, review and revise case management tools and strengthen their family focus.  Implement supportive supervision for community level volunteer cadres that allows volunteers to learn from experienced colleagues; investigate the possible use of retired, certified social workers through CSW and NASW to provide supportive supervision.  Seek a careful balance between advocating for the rights of vulnerable children and encouraging them to use their voices, and safeguarding their rights to confidentiality and privacy.  Given the CPF support for strengthening case management and child protection services, consider support for research to determine how well IMPACT and SHA models improve OVC health and use findings to leverage more GoZ support for health programs that serve vulnerable children and youth. Performance Evaluation of the Children First OVC Project for Zimbabwe 32 ANNEXES ANNEX A. STATEMENT OF WORK Task Order No. AID-613-TO-12-00001 Page 5 of 21 SECTION C – DESCRIPTION/STATEMENT OF WORK C.1 OBJECTIVE AND GENERAL DESCRIPTION This evaluation is intended for both accountability and learning purposes and will generate knowledge about the magnitude and determinants of project performance with a particular focus on the benefits derived from application of GHI principles in project design and implementation. The evaluation will inform future USAID efforts in design and implementation of more effective, efficient and sustainable country owned programs through an informed application of GHI principles. The evaluation will identify and describe innovative, sustainable and effective models of service delivery with potential for nationwide scale up and provide important lessons on how to improve sustainability, effectiveness and promote country ownership through; institutionalizing project activities within relevant government department, building local capacity and promoting innovation in service delivery. The evaluation findings will be shared with the implementing organization and its sub-partners, the host government and other relevant national stakeholders. Within USAID, the evaluation will be shared Mission wide and with USAID Washington. The final evaluation report will be posted onto the USAID Development Exchange Clearing House website. The evaluation should be conform to the new USAID Evaluation policy guidelines, ensuring sound methodological design, independence and objectivity of evaluators, and high quality documentation of findings. The Children First (CF) project is a five-year cooperative agreement [number: 674-A-00-08- 00021-00] between USAID and World Education (WEI) and John Snow International. Initiated in January 2008, the five year project is the US President’s Emergency Plan for AIDS Relief (PEPFAR) primary OVC intervention in Zimbabwe. The project seeks to mitigate the impact of HIV and AIDS on OVC in Zimbabwe by improving and developing proven models for care and support of vulnerable children. CF leverages the experience of national and community-based organizations to increase and improve access to quality care and support services for OVC in Zimbabwe. Working in Matabeleland South, Manicaland and Harare’s urban, peri-urban and surrounding rural areas, CF provides technical and financial support to its partners to provide services and undertake advocacy activities for OVC. The menu of core services provided by CF partners includes nutrition, health, education, legal support, child protection, psychosocial support, livelihoods, advocacy, and program sustainability. Through these partnerships, CF aims to achieve the following key results: • Increased access to OVC services through community initiatives • Strengthened human capacity in local community structures to meet needs of OVC • Improved community and national level advocacy for social protection of OVC Task Order No. AID-613-TO-12-00001 Page 6 of 21 Available background documents for the evaluation team to use include: • The cooperative agreement document between USAID and World Education • CF project quarterly progress reports • PEPFAR semi-annual reports • USAID portfolio review documents • CF project annual work plans and reports • PEPFAR annual country operational planning documents • PEPFAR annual reports • The CF Mid-term Evaluation Report • National Action Plan for OVC (I&II) • Program of Support for The National Action Plan for OVC Evaluation • Child Protection Fund for NAP II Strategic Concept & Design C.2 STATEMENT OF WORK USAID Zimbabwe requires a contractor to perform an end of project evaluation of a USAID supported Orphans and Vulnerable Children (OVC) project in Zimbabwe over a period of four weeks. The five year project was initiated in January 2008 and will end in December 2012. USAID anticipates that the contractor will conduct the performance evaluation in October 2012. C.3 OVERALL RESULTS AND INDICATORS This section sets forth results (outcomes of contactor’s performance) requirements, and performance standards (minimum standards that the contractor must meet) that must be met to USAID’s satisfaction. The Final Evaluation Report shall be evidence-based and respond to the Key questions and evaluation areas outlined below. The key questions listed below are not exhaustive. Offerors are strongly encouraged to propose additional or alternate questions, but the study should at a minimum answer the following: Key evaluation question: Has the project developed any innovative, sustainable and effective models of service delivery with potential for nationwide scale up? Specific Evaluation Questions: 1. Overall Program Performance:  What was the quantity, quality and timeliness of project results? 2. Innovation:  What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? 3. Country Ownership and Sustainability:  How effective was the CF project in developing local capacity to effectively meet the needs of OVC? Focus on capacity development for: o Local partner organizations and o Communities  To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? C.4 REPORTS AND DELIVERABLES Task Order No. AID-613-TO-12-00001 Page 7 of 21 In addition to the requirements set forth for submission of reports in Sections I and J, and in accordance with AIDAR clause 752.242-70, Periodic Progress Reports, the Contractor shall submit reports, deliverables or outputs subject to the deadlines specified in Section F.4 of this RFTOP, as further described below to the COR (referenced in Section G.2). The contractor will also be responsible for submitting the following deliverables: The following deliverables and reports are required under the Task Order. All deliverables and reports will be in English unless otherwise noted. The Contractor and the Contracting Officer’s Representative (COR) have the authority to make small changes to the deliverables and reports specified below. Any such alteration must not change the basic substance of the deliverable, require funds beyond the amount obligated or exceed the firm fixed price or any budgetary limitation. Each deliverable shall conform to the performance standards as described in the Statement of Work, Section C. 1. Signed statements attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the project being evaluated by each evaluation team member. 2. Final evaluation design, work plan and timeline presented to USAID by the lead evaluator within two weeks of the award of the contract. The evaluation design will include a detailed evaluation design matrix (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final design requires USAID approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 3. The evaluation team will meet with USAID upon arrival in Zimbabwe and go through the evaluation work plan and timeline. The team will also provide an oral presentation of preliminary findings in PowerPoint format to USAID and other key stakeholders in separate meetings prior to the evaluation team’s completing its evaluation activities in Zimbabwe and departing Zimbabwe. 4. Draft evaluation report (meeting all the criteria below) delivered to USAID for review within 10 business days from the time of departing Zimbabwe and returning to the Offeror’s base offices. USAID will provide comments within 2 weeks of receipt of draft evaluation report. 5. The final report will be provided to the USAID/Zimbabwe in electronic form within 15 business days following receipt of comments from USAID. The evaluation report must address all evaluation questions included in the statement of work. It must represent a thoughtful, well-researched and well organized effort to address the evaluation purpose. Readers must have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability. Task Order No. AID-613-TO-12-00001 Page 8 of 21 The final report should not exceed 30 pages (excluding appendices) and must include the following sections: o An executive summary: 3-5 page that summarizes the key points (project purpose and background, key evaluation questions, methods, findings, and recommendations) o Background information on the project o Purpose of evaluation o Evaluation team: must be described with particular reference to the existence or lack thereof real or potential conflicts of interest relative to the project being evaluated o Evaluation methods: must be explained in detail and limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) must be disclosed in the report o Evaluation findings: must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence. When applicable, include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team. o Recommendations: need to be supported by a specific set of findings and must be action-oriented, practical and specific, with defined responsibility for the action o The final scope of work, evaluation tools and sources of information must be properly identified and listed in annex 6. All data and records from the evaluation must be submitted to USAID in an easily readable and organized electronic format along with the final report. C.5 OVERARCHING ELEMENTS AND IMPLEMENTATION MODALITIES C.5(a) Building Local Capacity The Offeror shall, to the maximum extent possible, use Zimbabwean staff, technical experts, and institutions in carrying out the evaluation of the OVC project under the resulting Task Order. C.5(b) Geographical Coverage Task Order No. AID-613-TO-12-00001 Page 9 of 21 The OVC project is implemented in Bulawayo, Matabeleland South, Manicaland and Harare’s urban, peri-urban and surrounding rural areas (see map above) C.5(c) Gender Considerations Equity should be addressed with a focus on gender and orphans and other vulnerable children. The evaluation should provide more details on the effect and results of the project interventions on men, women, girls and boys. C.5(d) Audience The primary audiences for the evaluation report shall be USAID, Government of Zimbabwe, development partners, implementing partners and key stakeholders especially at sub-national levels. END OF SECTION C Performance Evaluation of the Children First OVC Project for Zimbabwe 43 ANNEX B. EVALUATION DESIGN Following is the Evaluation Design and Workplan. Annex D contains the final Data Collection Instruments. Field Data are being separately provided to USAID/Zimbabwe USAID/Zimbabwe End of Project Evaluation of USAID’s Orphans and Vulnerable Children (OVC) project in Zimbabwe AID-613-TO-12-00001 Draft Workplan for Final Evaluation Submitted to: Mr. Matthews Maruva Alternate Contracting Officer’s Representative Submitted by: International Business & Technical Consultants, Inc. 8618 Westwood Center Drive, Suite 220 • Vienna, VA 22182 Tel: 1-703-749-0100, ext. 234 • Fax: 1-703-749-0110 • E-mail: EAllan@ibtci.com October 15, 2012 USAID/Zimbabwe: End of Project Evaluation of the Children First Project International Business & Technical Consultants, Inc. Table of Contents 1. Evaluation Design Overview ..................................................................................................................................... 1 2. Technical Approach ...................................................................................................................................................... 2 3. Overview of Mixed Methods Methodology........................................................................................................... 3 4. Data Analysis ................................................................................................................................................................. 5 5. Evaluation Limitations ................................................................................................................................................. 5 6. Timeline and Work Plan ............................................................................................................................................ 5 7. Data Collection Instruments ..................................................................................................................................... 6 Annex A. Draft Calendar Annex B. Draft Interview Questions for CF Managers and CF Partner Organization Managers Annex C. Draft Anonymous Survey for Government and Donor Partners 1 1. Evaluation Design Overview USAID/Zimbabwe’s “Children First” (CF) OVC project is a five-year project PEPFAR-assisted project that seeks to mitigate the impact of HIV and AIDS on Zimbabwean OVC by improving and developing proven models for care and support of vulnerable children. PEPFAR’s major OVC intervention in Zimbabwe, CF is being implemented by World Education and John Snow International, with some 13 Zimbabwean partners. It began in January 2008 and is scheduled to end in December 2012; a solicitation for a successor project has been issued, and USAID/Zimbabwe anticipates that there will be a couple of months of overlap following award of the new project for a smooth transition. In September 2012, through a competitive process under the Evaluation Services IQC, USAID/Zimbabwe awarded International Business & Technical Consultants, Inc. (IBTCI) a task order to conduct the final evaluation of the CF project. The period of performance of this IQC is three months – October 1, 2012 to December 31, 2012. We are fielding a team of three specialists in global health and HIV/AIDS and/or health issues in Zimbabwe – an American, Deborah McSmith, who is serving as team leader, and two Zimbabweans, Ms Jenny Hunt and an additional Zimbabwean currently being recruited – plus a consultant in financial analysis. This field team is being supported by IBTCI’s home office management. USAID’s Key Evaluation Question for the end of project evaluation is: “Has the project developed any innovative, sustainable and effective models of service delivery with potential for nationwide scale up?” Specific Evaluation Questions include: 1. Overall Program Performance:  What was the quantity, quality and timeliness of project results? 2. Innovation:  What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? 3. Country Ownership and Sustainability:  How effective was the CF project in developing local capacity to effectively meet the needs of OVC? Focus on capacity development for:  Local partner organizations and  Communities  To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? IBTCI will conduct this performance evaluation with a mixed-methods methodology that is informed by the fact that in February 2011 a detailed mid-term review was conducted by a team of USAID and other specialists. In addition to providing an independent, external evaluation of the performance of CF, this final evaluation will build on that mid-term review to provide longitudinal information; we believe that it should also find immediate utility in the start-up phases of the successor project. The approach involves: (a) background and project document review, (b) a survey of the stakeholders who were surveyed last year plus any new personnel who may now be filling comparable positions, (c) key informant interviews with project management staff, partners and other key stakeholders, (d) focus group discussions with beneficiaries and volunteers, including children and youth, and 2 (e) site visits to each of the CF partners, ideally at times when the members of the evaluation team can observe OVC-related activities taking place. Team members will also visit sites of activities implemented by other NGOs in order to obtain a comparative perspective. The survey, which will be amplified by questions appropriate for this final evaluation, will typically be administered by the evaluation team members to relevant informants in the course of key informant interviews; otherwise it will be administered by a Zimbabwean firm. To promote candor and in keeping with the Common Rule for the Protection of Human Subjects, results will be reported anonymously. Discussions with beneficiaries, especially younger ones, will be conducted with the sensitivity our team members have developed through decades of work, including counseling, for children who are likely to be quite fragile. We will use data from multiple sources – quantitative and qualitative – to develop an objective and valid picture of CF’s work, its legacy, and areas that a successor may wish to pay special attention to, such as services for out￾of-school youth and children. 2. Technical Approach In addition to providing an independent, external evaluation of the performance of CF, this performance evaluation can find immediate utility in the start-up phases of the successor project. The evaluation will be framed within CF’s key results:  Increased access to OVC services through community initiatives;  Strengthened human capacity in local community structures to meet OVC needs; and  Improved community and national level advocacy for social protection of OVC. Evaluation instruments will be designed to address the key evaluation questions identified in the RFTOP. Apart from the question on Overall Program Performance, which has quantitative aspects as well, the evaluation questions are primarily qualitative, and the primary data sources for these will be the interviews, focus groups, and questionnaires/surveys, which will be mutually validated by the reviews of documents with data on comparative effectiveness derived from meetings and site visits with entities external to USAID and CF. Overarching question: Has the project developed any innovative, sustainable and effective models of service delivery with potential for nationwide scale-up? Specific evaluation questions, in order of relative priority, include: Overall Program Performance: What was the quantity, quality and timeliness of project results? Innovation: What innovative approaches contributed to achieving outcomes, particularly approaches to effective and efficient service delivery and capacity building? Country Ownership and Sustainability:  How effective was the CF project in developing local capacity to effectively meet the needs of OVC? The evaluation team should focus on capacity development for local partner organizations and for communities.  To what extent were processes and products developed through this project institutionalized within relevant government ministry structures? The team will pay particular note to points raised by the mid-term review team, primarily with respect to the needs of youth, especially out-of-school youth, under 18 years old. One particular concern was the effectiveness and equity of delivery of health-related services to out-of-school children and youth since much of CF was school-based. We note from CF’s website that its count for total children and for most of the critical services is substantially lower for the first two quarters of FY12 compared to FY11. We will seek to determine whether this is simply a statistical artifact or whether it reflects a true drop in services provided and, if so, why. 3 The evaluation will examine how CF program activities have reflected PEPFAR II and GHI priorities and principles, especially PEPFAR II’s shift in focus from emergency relief to development strategies intended to reinforce the sustainability of national HIV programs by strengthening national health systems, building capacity at all levels of government and civil society, and promoting country ownership, and GHI’s emphasis on integrating HIV programming into broader health responses to increase the impact and cost-effectiveness of USG support. Findings are intended to inform USAID/Zimbabwe’s design and implementation of more effective, efficient and sustainable country-owned programs through application of GHI principles. They will also be structured to reflect Objectives 1, Build Strong Beginnings, and 2, Put Family Care First, of the draft “US Government Action Plan for Children in Adversity.” The evaluation will be conducted in accordance with USAID’s Evaluation Policy, ADS 203, and other relevant policies and guidance (e.g., the TIPS and the July 2012 PEPFAR OVC Programming Guidance, especially Chapter 12, Critical Issues in Monitoring and Evaluating OVC Programs. Given the strong emphasis on capacity development, our review will also take into consideration both the FY2012 PEPFAR Capacity Building and Strengthening Framework and the principles of USAID’s HICD policy and will be mindful of the requirements of 22 CFR 225, Common Policy for the Protection of Human Subjects and its implementation via the Interpretive Guide, and transition.usaid.gov/policy/ads/200/200mbe.pdf. 3. Overview of Mixed Methods Methodology 1. Review of Documents Document review will provide the team with broad contextual perspective on what CF has accomplished and how it has evolved over time, both in its own context and in the context of the conditions of OVC in Zimbabwe. Document review will inform both quantitative (performance vis-à-vis the results framework) and qualitative data analysis (project report narratives will highlight challenges, barriers, unexpected outcomes, good practices and successes, and indications for sustainability). It will also provide the national context in which the CF work has been conducted and follow on work will be conducted. 2. Anonymous Survey We assume that USAID/Zimbabwe has information as to the universe of types of people surveyed in the mid￾term review, if not necessarily to the specific individuals surveyed. We plan to administer the mid-term survey to substantially the same universe (e.g., to previous interviewees still involved with CF plus new personnel with similar responsibilities in the CF partners), with the addition of open-ended questions such as “If you took part in the previous survey, what changes, if any, have you seen? What changes do you think would still be helpful?” The survey will be designed both to follow up on key MTE findings and to focus on end of project questions that differ. This will help assure comparability of findings and, depending on the results, may highlight areas for special attention.1 If particular individuals – or people in the same positions – be among interviewees for the Key informant interviews, the survey will be administered by members of the evaluation team as part of the interview. Otherwise, we plan to have surveys administered by Survey Research, a highly experienced Zimbabwean firm. Findings from this survey will be reported anonymously. The survey will focus on the “macro” level: perceptions of performance quantity, quality and timeliness of CF project results; documented innovative, sustainable and effective service delivery models; and lessons learned and recommendations relevant to country ownership and sustainability. Are we envisioning an electronic survey sent as email attachment, filled out and returned or as a survey conducted face to face? 1 There is no publicly available information as to whether there had been any baseline surveys or any other surveys conducted prior to the one for the mid‐term review.  Building our survey on the mid‐term review will make it possible to provide some degree of longitudinal analysis as to perceived CF impacts and effectiveness.    4 3. Key Informant Interviews We will conduct semi-structured informant interviews with staff of USAID/Zimbabwe, World Ed/John Snow/CF, all 13 Implementing Partners, the Ministry of Health and Child Welfare (MOHCW) and possibly the Ministry of Labour and Social Services (MLSS), UNICEF and other key OVC donors who are aware of the Children First Project. For the CF partners, these interviews will include project managers and individuals directly involved in the delivery of services to OVC. During site visits, interviews will be conducted where possible with education/health/social services staff in the field, and implementers of OVC programs not associated with CF to understand the overall performance accomplishments of the CF project over its five￾year span, and to capture key lessons learned that have implications for future US PEPFAR OVC programs as well as for broader national OVC programs. Interviews will particularly focus on performance quality, quality and timeliness; innovative, sustainable and effective service delivery models; capacity building strategies, accomplishments and effective strategies; and learning that relates to country ownership and sustainability. However, questions will be more detailed and specific to various project components and strategies. Semi structured interviews will permit comparison of responses across respective groups while allowing for differences in project partnership structures and roles. To help address the evaluation question as to effective and efficient service delivery, we will ask interviewees to compile their perspectives as to the “best and sustainable practices,” especially those that strengthen households with OVC (as separate from only the OVC themselves) that they are making use of and to share them with the evaluation team members. Completed interview tools will be scanned and included as an annex in the final report. Interviews with USAID staff, MOHCW, and senior World Ed staff will be recorded with consent of the interviewees and transcribed. To promote candor and in keeping with the principles of the Common Rule, the interviews with other World Ed CF and IP staff will be represented by the anonymous questionnaires. 4. Focus Group Discussions We intend to conduct focus group discussions in conjunction with site visits to CF partners, during which we plan to observe as well as ask about OVC-oriented activities that may be taking place. Focus group participants will consist of (a) local staff, including volunteers, (b) beneficiaries old enough to speak on their own behalf, and (c) caregivers. We will take steps to encourage participation in all focus groups by both males and females, and will conduct gender specific focus groups where useful. Interviews with youth and children and FGDs in single-language settings will take place with translation support in Shona or Ndebele, as appropriate. (Evaluation team members are experienced in working with translators during evaluations.) With informed consent, focus groups will be recorded and transcribed, with notes transcribed in English and included as an annex in the final report. 5. Site Visits The team will make site visits to implementing partners with observation of program activities, to the extent feasible. The evaluation team will confer with USAID and CF management staff to finalize site visit selection across the 13 partners. Site visits are intended to provide a representative overview of the range of education, health, and social support interventions provided by the project to out-of-school adolescents, in school youth, and young children in the care of guardians and will provide opportunities to facilitate on site FGDs with beneficiaries as well as representative interviews with school administrators, health service providers and social support services providers. Site visits to all CF partners will be scheduled to allow evaluators to observe OVC-related activities. To provide time with each CF partner to make meaningful observations, it will likely be necessary for the team to split up. For comparative purposes, in consultation with USAID, we plan to make site visits to some OVC service organizations within the CF catchment area that have not participated in CF. The site visit protocol will be based on the July 2012 PEPFAR Guidance for Orphans and Vulnerable Children Programming and piloted during visits made by the whole team to sites in and around Harare; in addition to 5 refining the instrument, this will promote a common understanding of the evaluation methodology and will reduce the possibility of inter-evaluator differences. Data obtained from multiple sources should provide the range of information needed to develop valid and reliable findings, conclusions, and recommendations. These data will be analyzed and triangulated to help inform potential additional areas for inquiry and to test for mutual consistency and internal validity. While quantitative research will provide important inputs to the overall findings, the analyses will be primarily qualitative. Recognizing that mid-term reviews and final evaluations have separate purposes and, assuming that the partners are essentially the same, we plan to use some of the mid-term review instruments, with appropriate adaptations, as a base which provides perspectives on changes, if any, over time. 4. Data Analysis Data will be triangulated to identify possible inconsistencies in the data and to identify areas that may call for special attention. The team as a whole will have been selected to reflect individuals with somewhat different, but overlapping, areas of expertise; this provides a difference in perspectives that builds depth to the analysis and avoids “group think.” Frequently during the field work, team members will confer as to their observations and perspectives on interviews and site visits; while these discussions will help to identify frequent themes and possible outliers, team members will keep in mind that findings in general, and certainly recommendations, must await the end of the data collection. To further maintain objectivity and the degree of comprehensiveness available within the given resources, before submission to USAID, drafts of reports will be shared among team members for their own responses and also shared with the Project Director for his review and comment both for quality in keeping with the Evaluation Policy and as an observer familiar with the project but an “outside reader.” Reports will be prepared in accordance with USAID requirements, including appropriate attention to similarities and differences between different categories of beneficiaries. 5. Evaluation Limitations Apart from the overall limitation of time, the major technical limitation is likely to be the availability of community-level staff and volunteers of the IPs to be interviewed and opportunities to observe OVC-related activities. With respect to the financial analyses requested, the major limitation is going to be the availability (and quality) of data usable for the analyses requested. While the IPs may be keeping track of costs by essential service, given the reality that large numbers of OVC receive comprehensive services, disaggregation by essential service is not likely to be either feasible or meaningful. Further, different partners may be providing different services within common categories, e.g. “Psychosocial Support.” There is likely to be further confounding if, as is probable, the partners use quite different approaches to accounting for indirect costs. What we may be able to produce are estimates of unit costs per OVC, regardless of number and type of essential services received, by direct cost. Data validity is dependent upon on accurate project reporting. Apart from written project documents, the evaluation team will rely on the experience and perspectives of stakeholders and beneficiaries. 6. Timeline and Work Plan The following presents a broad overview of the project timeline. Annex A (separate Excel file) presents our draft calendar, which is illustrative of the overall organization of time and entities and types of meetings we intend to hold. The actual calendar is subject to revision, based on consultation with USAID and World Education. Upon Award  Begin review of documents and preparation of evaluation plan, including draft questionnaires.  Work with Mission’s COR to refine or clarify the Scope of Work as appropriate. Agree upon role, responsibility, and associated matters with respect to the USAID staff person participating on the evaluation team. 6  Recruit second Zimbabwean evaluation team member in accordance with USAID Forward principles. (Because of the conflict of interest restrictions and associated vetting and other constraints, it would not have been feasible to do so at the proposal preparation stage.)  Recruit Zimbabwean financial analyst. Because of the constraints on this analysis and the need for consultation with the mission, it was not feasible to recruit this person at the proposal preparation stage. Before Arrival  Establish contact with CF (World Ed) to introduce evaluators and to hold preliminary discussion on the evaluation, including asking them if they could identify for us appropriate opportunities to observe OVC￾related activities with all partners.  Develop draft daily schedule, based on feedback from CF. On Arrival and Start-Up  Meet with entire consultant team to review TOR and deliverables, clarify roles and tasks, and establish agreement for successful collaboration.  Obtain in-brief from Mission staff on the Mission’s portfolio, the context of USAID’s PEPFAR activities, the Children First project, and expectations for the evaluation.  Review draft evaluation plan, instruments, and timeline with Mission staff.  Obtain and start review of documents not previously available.  Hold initial face-to-face discussions with CF staff to schedule interviews and confirm site visit plan details and CF staff participation in site visits.  Conduct initial interviews in and around Harare to test survey, interview and FGD instruments.  Consultant team review results of initial evaluation activities and revise instruments as needed. During and After Evaluation  Conduct evaluation in accordance with the evaluation plan and the schedule of deliverables, providing a weekly check-in to USAID and IBTCI.  Deliver draft report and provide debriefings per RFTOP.  Finalize and submit report after receipt of USAID feedback on draft report. 7. Data Collection Instruments Annexes B and C contain DRAFT instruments for interviews and for a survey of the type of respondents who we believe were surveyed in the Mid-Term review. Although with appropriate changes, the types of questions suggested for CF and partner managers are similar to those that would be posed for other Key Informant Interviews and focus groups involved in delivery of services (e.g., technical people, volunteers). The Evaluation Team is in the process of developing FGD protocols for use with caretakers and beneficiaries. These and other instruments will be revised based on Mission feedback, and interview protocols are likely to be revised as initial interviews suggest other areas that may merit exploration, given the time available. Performance Evaluation of the Children First OVC Project for Zimbabwe 55 ANNEX C. EVALUATION TIMELINE AND LIST OF PEOPLE INTERVIEWED Date Day Time Time 1‐Oct Task Order in Place Team begins research Recruiting takes place for OVC Research & 8‐Oct Document review and drafting of data 15‐Oct DRAFT Work Plan due 22‐Oct Document review 27‐Oct Sat. Deborah travels to Geneva 3‐Nov Sat. Deborah travels to Harare from Geneva 4‐Nov SUN Evaluation team meeting 5‐Nov USAID am briefing and draft instrument review CF logistics, tools review 6‐Nov Tues DSS Stakeholder (to be determined) interviews in and around Harare to test tools 6‐Nov Tues 11‐1300hrs CF Managers 6‐Nov Tues 14 30pm Dr Kujeke Ministry of Education 7‐Nov Wed 0:00 CF Calendar Review Bronte ‐ USAID Collen, Translator, Tools Review Date Day Time Team 1 Time Team 2 8‐Nov Thurs 08 30 Family Support Trust interviews, Hre Hospital, FGD, team, Ivy 8‐Nov Thurs 14:00 Africaid interviews, Avondale, FGD, visits, team, Ivy 9‐Nov Fri 08 30 Mavambo interviews, FGD, visits, Ivy, JH, BS (M) 09‐11:00 Emerald Hill D 14 00 Dr Chitepo Ministry of Youth D 10‐Nov Sat Document Review, Paperwork, reading docs, transcribe tapes All 11‐Nov SUN Rest Rest Performance Evaluation of the Children First OVC Project for Zimbabwe 56 12‐Nov Mon City Health Mr Tumbare, Local Health Authority (reschedule) leave 11 am for 12 to 1 pm Dr. Gonah, Chitungwiza Hospital (team), OI Clinic 14:30 CF review OOS curriculum, informational interview with curriculum staff (team) 13‐Nov Tues taxi pick up at 7 45 for 08:00 Dr. Angela Mushavi, MOH, Mkwati Bldg, 5th Street, 2nd Fl, Rm 323 AIDS AND TB UNIT, J,B leave at 7 to arrive 08:30am Howard Mission interview, FGD with Adults, site visits D and R, f‐up with Dr. Paul Thistle 15 00 Bronte D 10 45, meet someone at HOSPAZ HOSPAZ sub‐partner Simbarashe (Mhondoro) FGD, visit‐ Community Leaders, Ivy, B, J CF revise arrival time for 10 45 08 00 Team meet at Bronte to check in 14‐Nov Wed 09 00 PAZ, D09 00 Kapnek Trust interview, office, FGD, Caregivers, visits Location? J, B 11 00 OPHID, Diana Patel, D 14 00 UNICEF (Laurin Rumbles), D 15 30 CF tech field officers FGD, D 15‐Nov Thurs 08 30‐16 30 Child Protection Society interview, ECD, FGD, visits D, I locations, contacts, close to CF office 08 30 Justice for Children interview, FGD, visits, J,B, R, transport with CF? 15 00 Sister Munyaradzi, City Health 16‐Nov Fri 05 30 Nzeve Deaf Children's Center. Overnight B, I 08 30 Seke Rural interview, FGD, visits J, D, R, 17‐Nov Sat Return to Hre B, I Quantitative document review and meet with HOSPAZ J D 18‐Nov SUN Rest Drive to Bulawayo D, J, I 19‐Nov Mon 09 00 Chiedza interview, visit B, R TBA 10 30 FST Gwanda PEP Awareness song event, D, J, I 20‐Nov Tues 08 30 Childline HQ and Drop in Center ‐ Mbare, Interviews, FGD Volunteers, B, R TBA Byo Catholic diocese youth prog FGD, visit, IGA, OOS, GRS interview, D, J, I 15 00 Meet with Tsitsi B 21‐Nov Wed TBA Away 08 30 Umzingwane AIDS Network FGD, visit‐  Youth Friendly Corner, Case Care Workers, CPCs 22‐Nov Thurs Away Return to Harare CF Managers and Munya 14 30 Meet with CF: Susan, Patience, Munya, others 23‐Nov Fri Data Analysis (team) Data Analysis 24‐Nov Sat. Data Analysis (team) Data Analysis Performance Evaluation of the Children First OVC Project for Zimbabwe 57 25‐Nov SUN Data Analysis (team) Data Analysis 26‐Nov Mon 14 00 Stakeholder Debrief  D,J,B,I,R includes translators 27‐Nov Tues am USAID Mission Debrief (D,J,B,I,R) includes translators Performance Evaluation of the Children First OVC Project for Zimbabwe 58 ANNEX D. DATA COLLECTION INSTRUMENTS 1. INTERVIEW INSTRUMENT FOR MINISTRIES/ LOCAL AUTHORITIES/UNICEF (*Questions to be used when time is limited to less than an hour) INTRODUCTION: USAID/Zimbabwe has contracted with IBTCI to conduct the final evaluation for its Children First OVC Project, being implemented by World Education Inc. in collaboration with some 13-19 primary partners. We are seeking to learn about how the CF project has been implemented and about innovative approaches implemented through this 5-year project that have contributed to achieving intended outcomes. We are particularly interested in learning about how well the project has built capacity and contributed to sustainable and innovative practices that may be scalable in future OVC programming in Zimbabwe. We very much appreciate your taking the time to talk with us and help us to understand project accomplishments and lessons learned that can be sustained through follow-on and complementary OVC programs and services. We estimate that this interview will take approximately one hour, but we can adjust that if necessary. SECTION 1: INNOVATIVE, SUSTAINABLE AND EFFECTIVE SERVICE DELIVERY 1. How has Children First increased access to OVC services through community initiatives? How can you tell? 2. In what ways has Children First improved quality of OVC services? 3. * In what ways has Children First focused on providing a comprehensive continuum of care for the most vulnerable children? 4. * In your view, has Children First played a role in strengthening technical capacity for OVC services within (Ministry or Local Authority)? If so, at what levels – national/provincial/district – and in what ways? 5. How well has Children First coordinated its activities with (Ministry or Local Authority)? 6. * By the beginning of 2012, CF had identified and developed 9 formalized models for OVC programming (see below). Of the models you are familiar with, which do you believe to be REPLICABLE across Zimbabwe? Ask probing questions regarding innovation and effectiveness. Children First Model: Are you familiar with model? Yes/No Do you believe model to be replicable? Yes/No Please explain your answer. 1. Integrated Model for Paediatric AIDS Care and Treatment (IMPACT) 2. Out of School Study Groups (OSSG)/ Community Learning Centers (CLC) Performance Evaluation of the Children First OVC Project for Zimbabwe 59 3. Expanded Youth Friendly Corners (EYFC) 4. Bantwana Schools Integrated Program (BSIP) 5. Community Adolescent Treatment Support (CATS) 6. Community Based Paralegal Program (CPP) 7. Community Based Counseling Program (CBCP) 8. Integrated Early Childhood Development (IECD) 9.Community Based Case Management Model 7. Have any of these models been institutionalized within [Ministry or Local Authority]? If so, which ones and how? How will [Ministry] utilize these models going forward? SECTION 2: COUNTRY OWNERSHIP AND SUSTAINABILITY 8. To what extent are Children First’s implementation approaches in line with (Ministry or Local Authority) approaches and structures? Please explain your answer. 9. To what degree has Children First improved national advocacy for OVC? In what ways? 10. * To what degree has Children First improved advocacy at community level for OVC? In what ways? 11. In your view, to what extent has Children First been effective in developing local community capacity to meet the needs of OVC? Please explain your answer. 12. To what degree has Children First been effective in institutionalizing strategies and resources developed within (Ministry or Local Authority) as a result of this project? Please explain. 13. * Now that Children First is drawing to a close, how will (Ministry or Local Authority) play a role in continuing to implement some of the services the project has initiated? 14. How can Government funding be leveraged to sustain some of these activities? Probe how Ministries fill gaps and complement each other Performance Evaluation of the Children First OVC Project for Zimbabwe 60 2. Interview with Primary Partner Organization Managers Name of Organization: Date and Time: Location: Interviewee Name(s)/Title(s): Interviewer Name(s) INTERVIEW INTRODUCTION: Hello, my name is ______________ and these are my colleagues--------------- We are here as representatives of an evaluation organization that has been asked by USAID Zimbabwe to conduct the final evaluation for the World Education Children First OVC Project. We would like to ask some questions that will help us assess how the CF project has met its goals over the last 5 years, and how the work that has been undertaken by CF partners such as yourselves may be replicated or continued once the project funding ends in December. We estimate that the interview may take up to 1 hour, but we can adjust that if necessary? SECTION 1: PERFORMANCE 1. Overall, how well did Children First (CF) meet its key project goals? 2. Have you managed to meet your targets for the Children First Project? If not, why not? 3. Which strategic areas (education, health, child protection, advocacy) and what key activities did your organization focus on as a Children First Partner?  Education  Health  Child Protection  Advocacy 4. Of your project services which were most successful in: a) reaching most children b) providing the best quality services? SECTION 2: STRATEGY 5. Looking back over the CF project’s 5 years, how well did your strategies work? How can you tell? 6. Based on your experience, do you believe these to be replicable strategies in Zimbabwe? Why or why not? 7. CF utilized 5 key access points…  early childhood development centers,  schools,  out of school centers,  clinics,  home based care Performance Evaluation of the Children First OVC Project for Zimbabwe 61 …as service delivery platforms. Do you think this was an effective strategy for a continuum of care? Please explain. SECTION 3: TIMELINESS 8. Thinking of the many project components and services, can you provide examples of how CF was best able to respond to the needs of OVC in a timely way? SECTION 4: VISIBILITY 9. What districts are you operating in? 10. How well known are your activities in the districts that you operate in? [i.e., to traditional leaders, NGOs, etc.]. How important is visibility? 11. What has CF done to make other project accomplishments visible to national stakeholders? SECTION 5: QUALITY 12. How do you measure quality of services? 13. Have your activities included case management for OVC? If yes, how effective has it been and how can you tell? SECTION 6: COUNTRY OWNERSHIP AND SUSTAINABILITY 14. What capacity building has CF provided to your organization [probe for specific technical areas]? 15. How useful has it been? 16. Was your organization involved in providing capacity to sub-partners? If so what kind of capacity building? 17. Has your organization participated in any learning exchange with other CF partners? How did you benefit, if at all? 18. Which of CF’s information sharing and dissemination of best practices has been most effective and why? (e.g., thematic round tables, annual reflection forum, final lessons learned forum, structured exchange visits for challenge grant partners, newsletter) 19. How well was CF able to build on efforts to strengthen Child Protection Committees? 20. In your view will these CPCs continue after the project closes? 21. Now that CF is ending how will you continue your activities funded under CF? 22. What are your main concerns about ongoing OVC support in Zimbabwe? SECTION 7: OTHER INFORMATION THAT COULD STRENGTHEN FUTURE OVC SUPPORT IN ZIMBABWE 23. What role, if any, did CF and your organization play in the development of a national OVC volunteer policy? Please explain. 24. What else would you like the evaluation team to know about the CF Project? Performance Evaluation of the Children First OVC Project for Zimbabwe 62 3. Final Evaluation Focus Group Discussion Guide – GROUP FACILITATORS & COMMUNITY VOLUNTEERS (e.g., CPC, CFPs, OOS Facilitators, Caregivers) Name of Organization/Program: Date and Time: Location: Kind of group: Ages: Gender: (F) (M) Interviewer Name(s) INTRODUCTION: Hello, my name is ______________ and these are my colleagues--------------- We are here as representatives of an evaluation organization that has been asked by USAID Zimbabwe to conduct the final evaluation for the World Education Children First OVC Project. Have you heard of Children First? We would like to ask some questions that will help us assess how the CF project has met its goals over the last 5 years, and how the work that has been undertaken by volunteers such as yourselves may be continued once the project funding ends in December. We estimate that the discussion may take up to 1 hour, but we can adjust that if necessary? (Translator will fill out age and gender details at end of each FGD.) 1. Please describe how you support children and/or youth as a volunteer. What are your responsibilities? How often do you visit children/households? How many children do you currently support? What is their age range? How many girls and boys? How did you become a volunteer? 2. What if any are the differences in your support for boys and girls? 3. Which of these activities do you think benefit children/youth most? How can you tell? 4. What kind of referrals do you make on behalf of children and/or youth? Probe for effectiveness and problems. 5. How do children or youth find out about this support? Are there children and youth who need support that are not getting it? How can you tell? 6. How do children and their families benefit from this support? How can you tell? Probe for family involvement in services. 7. What kind of training have you received from CF or a CF partner? Clarify whether training from CF or partner. 8. How useful has this training been for your volunteer activities? 9. In what ways do you receive emotional support as a volunteer? Do you receive formal or informal supervision? Is this enough? 10. What is the hardest part of the work you do? What do you do about that? Who helps? 11. How will the support you provide continue after Children First ends in December? Performance Evaluation of the Children First OVC Project for Zimbabwe 63 12. What do you like most about your volunteer work and what motivates you to continue? Wrap Up: Is there anything else you would like to tell us about Children First/ local partner, or your role? Do you have any questions for us? Performance Evaluation of the Children First OVC Project for Zimbabwe 64 4. Final Evaluation Focus Group Discussion Guide – PARENTS/GUARDIANS – will be piloted in Mutare or Bulawayo as required Name of Organization/Program: Date and Time: Location: Kind of group: Ages: Gender: (F) (M) Interviewer Name(s) INTRODUCTION: Hello, my name is ______________ and these are my colleagues--------------- We are here as representatives of an evaluation organization that has been asked by USAID Zimbabwe to conduct the final evaluation for the World Education Children First OVC Project. Have you heard of Children First? We would like to ask some questions that will help us assess how the CF project has met its goals over the last 5 years, and how the work that has been undertaken by CF community partners such as yourselves may be continued once the project funding ends in December. We estimate that the interview may take up to 1 hour, but we can adjust that if necessary? Translator will fill out age and gender details at end of each FGD. 1. How did you become involved in this project? 2. In what ways if any have you helped to design or develop the project? 3. What if any support do you and your family get from this project? (i.e. If home visiting, how often. If services, what kind and how are they delivered?) 4. How helpful has this support been? 5. Where else do you get support from? What kind of support? Probe for people, organizations, churches, government, etc.) 6. In what ways if any has this project helped to link you with any other services? 7. What else do you need to support your family? Is it available in your community? What needs to be done in order for that to happen? Are the needs different for older and younger children and for girls and boys? 8. How are OVC perceived in your community? (including children who are bereaved, sexually abused) 9. How has the project helped to decrease stigma and increase acceptance for your family? 10. What if any other community groups/structures support OVC and their families and how effective are they? (Probe for CPCs, advisory committees, etc.) 11. In what ways if any do you receive emotional support as a parent or guardian? Is this enough? What else do you need? 12. What advice do you have for future programs such as this? Wrap up: Is there anything else that you would like to tell us about Children First/local NGO or your hopes and needs for your children? Performance Evaluation of the Children First OVC Project for Zimbabwe 65 Do you have any questions for us? Performance Evaluation of the Children First OVC Project for Zimbabwe 66 5. Final Evaluation Focus Group Discussion Guide – Children and Youth (Including In and Out of School) Name of Organization/Program: Date and Time: Location: Kind of group: Ages: Gender: (F) (M) Interviewer Name(s) INTRODUCTION: Hello, my name is ______________ and these are my colleagues--------------- We are here as representatives of an evaluation organization that has been asked by USAID Zimbabwe to conduct the final evaluation for the World Education Children First OVC Project. Have you heard of Children First? We would like to ask some questions that will help us assess how the CF project has met its goals over the last 5 years, and how the work that has been undertaken by CF and its partners may be continued once the project funding ends in December. We estimate that the discussion may take up to 1 hour, but we can adjust that if necessary? Translator will fill out age and gender details at end of each FGD. 1. Please tell us how you take part in this project. For how long? How often does the group meet? How often are you able to take part? 2. In what ways, if any, do you benefit from this project? Please tell us a story or example. In what ways if at all have you connected with other useful support through this project? 3. Is there anything about this project that you don’t like or wish was different? Please explain. 4. What difficulties if any does the project face? Please tell us a story about that. 5. How could your community best support Orphaned and Vulnerable Children? (e.g., education, livelihoods, SRH information, emotional support, mentorship, legal support, safety). Are the needs of boys and girls different? If so, how? 6. How do you think that Orphaned and Vulnerable Children are perceived in your community? Can you share your own experience? 7. What needs to happen for communities to support OVC? 8. What do you understand to be your rights as a young person? 9. What needs to happen to have your rights met? 10. What advice do you have for future programs such as this? Wrap up: Is there anything else that you would like to tell us about Children First/local NGO or your hopes and needs for your children? Do you have any questions for us? Performance Evaluation of the Children First OVC Project for Zimbabwe 67 6. BSIP/VSIP Interview for Education, Health, Child Protection Personnel Name of Organization: Date and Time: Location: Interviewee Name(s)/Title(s): Interviewer Name(s) INTERVIEW INTRODUCTION: Hello, my name is ______________ and these are my colleagues--------------- We are here as representatives of an evaluation organization that has been asked by USAID Zimbabwe to conduct the final evaluation for the World Education Children First OVC Project. We would like to ask some questions that will help us assess how the CF project has met its goals over the last 5 years, and how the work that has been undertaken by CF partners such as yourselves may be replicated or continued once the project funding ends in December. We estimate that the interview may take up to 1 hour, but we can adjust that if necessary? SECTION I. EDUCATION SERVICES - For Educators, ask these questions: 1. Please describe your main responsibilities that relate directly to the Children First project. 2. Did your school receive a CF Block grant? 3. If yes, did the grant accomplish its intended purpose in terms of number of OVC retained in school? If not, why not? 4. What other benefits, if any, did your school receive from the Block grant? 5. How did CF and the school identify OVC to be supported through the Block grant? 6. Are there OVC who need this support who are not getting it? If so, please describe. 7. Did your school take part in CF School Based Health Assessments? If yes, how successful do you consider the health assessment support to have been? How can you tell? 8. Besides formal education classes, what support did your school provide to OVC (PSS counseling, clubs, role models, health education, OOS, linkages to GOV and services) and how effective has this been? 9. Of the CF activities that took place at your school, which do you believe will continue after CF ends in December and why? 10. Based on your CF experience, how can the unique needs of girl children and boy children best be met? 11. Based on your CF experience, what recommendations do you have for future school based OVC programs and services? SECTION II. HEALTH SERVICES - For Health Workers, ask these questions: 1. Please describe your main responsibilities that relate directly to the Children First project. 2. Please describe your main responsibilities that relate directly to the Children First project. 3. Were you directly involved with initiating OVC on ART and/or ART adherence support? Performance Evaluation of the Children First OVC Project for Zimbabwe 68 4. Were you directly involved with CF Health Care Assessments at schools? If so, how effective do you consider this strategy to be and why? 5. Were you involved with any other health support for OVC? If so, how effective do you consider this strategy to be and why? 6. Were you directly involved with CF Health Assessments at Early Childhood Development Center sites? If so, how effective do you consider this strategy to be and why? 7. Please describe your direct involvement with CF Adolescent Sexual and Reproductive Health (SRH) services, if any. How effective do you consider this strategy to be and why? 8. Please describe challenges, if any, with CF Adolescent SRH services and what steps CF took to overcome challenges. 9. Which of the health services that you participated in do you think will continue after CF ends in December and why? 10. Based on your CF experience, what recommendations do you have for future school based OVC programs and services? SECTION III. CHILD PROTECTION SERVICES - For social workers, Victim Friendly Police Officers, CPC members, ask these questions: 1. Please describe the child protection services you have provided as part of the Children First project. (Interviewers track which services are named: Nutrition, Economic Strengthening, Shelter support, Legal Support, PsychoSocial Counseling Support, Life Skills, Crisis intervention strategies, Case Management, Strengthening of CPCs and/or Child Advisory Committees, support for Child Led Households, Child Rights campaigns, support for caregivers, referrals, linkage with Community Health Insurance Framework, village OVC registers, Support for OOS youth, etc.) 2. In your view, how effective have they been? 3. What support did you receive from CF to be able to do your job effectively? What additional support do you need? 4. Which of the CF services that you were involved in do you believe will continue after CF ends in December and why? 5. Based on your CF experience, what recommendations do you have for future Child Protection Services? Performance Evaluation of the Children First OVC Project for Zimbabwe 69 7. Focus Group Discussion with CF Technical Field Officers Date and Time: Location: Interviewee Name(s)/Title(s): Interviewer Name(s) INTRODUCTION: You may be aware that USAID/Zimbabwe has contracted IBTCI to conduct the final evaluation for its Children First OVC Project, being implemented by World Education Inc. in collaboration with some 13-19 primary partners over time. We are seeking to learn about how the CF project has been implemented and about innovative approaches through this 5-year project that have contributed to achieving intended outcomes. We are particularly interested in learning about how well the project has built capacity and contributed to sustainable and innovative practices that may be scalable in future OVC programming in Zimbabwe. We very much appreciate your taking the time to talk with us and help us to understand lessons learned and project accomplishments that can be sustained through follow-on and complementary OVC programs and services. Interviewer will note number, age range and gender among field technical officers. SECTION 1: FIELD TECHNICAL OFFICER ROLES AND RESPONSIBILITIES 1. Please describe your main responsibilities as a CF Field Technical Officer (FTO). SECTION 2: COMMUNITY PARTNERSHIPS and COMMUNITY SYSTEMS STRENGTHENING 2. With which community organizations or leaders have you directly worked as a FTO, if any? Please list all. 3. How do you work with Child Protection Committees, if at all? What has been your role in helping to strengthen their capacity to support OVC and vulnerable households? 4. What has worked well with regard to the role of CPCs in supporting children? 5. What has not worked well with CPCs and why? What suggestions do you have to improve what has not worked well? 6. To what degree has CF partnered with other organizations to provide capacity building for local organizations? How effective has this capacity building been? 7. How were you involved, if at all, in selection of community groups to receive Challenge Grants or Social Protection Grants? How were you involved in monitoring Social Protection grant recipients? 8. Did communities you work in receive grants for community gardens? If so, what was the result? How has household food security been improved through these gardens, if at all? 9. Please describe any private partnerships initiated by CF at the community level. How effective were they? SECTION 3. PROGRAM SERVICES Performance Evaluation of the Children First OVC Project for Zimbabwe 70 10. How did the CF project identify OVC to be served in the communities where you work? How was it decided which services each OVC would receive? 11. Have you provided referrals for OVC? If so, for what services? How effective have they been? 12. In the communities where you have worked, to what degree have OVC participated in decisions that affect them? Can you give examples? How do children provide feedback to CF, it at all? 13. How do you think services and activities that started with CF and benefit OVC can be continued after the project ends? What main lessons have you learned over the project’s 5 years that could improve future OVC programming in Zimbabwe? Wrap up: Is there anything you would like to tell us about Children First? Do you have any questions for us? Performance Evaluation of the Children First OVC Project for Zimbabwe 71 ANNEX E. DOCUMENTS REVIEWED WEI/CF WEI Children First (Nov 2012). End of Project Performance Evaluation for the USAID/Zimbabwe Children First OVC Project: Briefing Book WEI Children First project Annual Report 2008 WEI Children First (Oct 2008). Patience Ndlovu. Situational Analysis and Mapping Survey on Street Children and Children Infected with HIV and AIDS in Harare WEI Children First project Annual Report Oct 1 2010-Sept 30 2011 WEI Children First Fiscal Year 2011. First quarterly report Oct 1 2010-Dec 31 2010 WEI Children First Fiscal Year 2011. Second quarterly report Jan 1 2011-March 31 2011 WEI Children First Fiscal Year 2011. Third quarterly report April 1 2011-June30 2011 WEI Children First. (Sept 2012). Ephraim Dhlembeu, Monica Mandiki, Wonder Phiri) Partner Capacity Evaluation Report WEI Children First Fiscal year 2012. Workplan narrative WEI CF (undated). Let’s Enjoy. General paper draft. Level 1 Term 3. Out of School Learner’s Workbook. WEI CF (undated). Let’s Enjoy. Mathematics draft. Level 2 Term 3. Out of School Facilitator Manual WEI CF (undated). Let’s Enjoy. Maths. Level 3. Term3. Out of School Learner’s Workbook WEI CF (undated). Let’s Enjoy. English. Level 3. Term 3. Out of School Learner’s Workbook WEI CF (undated). Let’s Enjoy. English. Level 3. Term 3. Out of School Facilitator’s Workbook WEI CF (undated). Child Maintenance Training Modules WEI CF (undated). Children with Special Needs & Birth Registration Training Modules WEI CF (2011). Findings from Joint Mid-Term Review of WEI/Bantwana Initiative’s Children First Project Government/National Institutions: Council of Social Workers of Zimbabwe (November 2011). The Minimum Standards for Community Childcare Workers in World Education Inc. Programme Areas Zimbabwe National Statistics Agency (ZIMSTAT) and ICF International Inc. (March 2012). Zimbabwe Demographic and Health Survey 2010-11. Calverton, Maryland: ZIMSTAT and ICF International Inc. Dept Social Services, Ministry of Labour and Social Services (2010). Institutional Capacity Assessment. Oxford Policy Management and Jimat development Consultants. Ministry of Labour and Social Services (2012). Rapid Assessment of Child Protection Committees in Zimbabwe Ministry of Health and Child Welfare (2010). National Community and Home Based Care Caregiver Policy Ministry of Health and Child Welfare (2007). National Community Home Based care Volunteer Handbook for Programme Implementers Ministry of Health and Child Welfare (2005). National Home Based Care Training Manual Ministry of Labour and Social Services (Revised edition 1996). Children’s Protection and Adoption Act. Chapter 5:06 and Amendment no. 23 (2001) Ministry of Education (Revised edition 1996). Education Act Ministry of Labour and Social Services. National Action Plan for Orphans and Vulnerable Children Phase II 2011-2015. Parliament Zimbabwe Research Department (Sept 2006). Fact Sheet. Child Abuse in the Education Sector. Ministry of Labour and Social Services, NAP for OVC (2010). Our Children, Our Future. Zimbabwean Good Practices Responding to the Needs of Orphans and Vulnerable Children. The “Zvandiri Program, Africaid; The Kapnek Trust Early Childhood Development Centres. Ministry of Labour and Social Services. Minimum Quality Standards for OVC Programming in Zimbabwe. An implementer's guide (2008) Performance Evaluation of the Children First OVC Project for Zimbabwe 72 USAID/UNICEF/Save the Children: USAID AIDS Support and Technical Assistance Resources/AIDSTAR-TWO Nov 15-18, 2010. Investing in those who care for children: Social Welfare Workforce Strengthening Conference Report. Unicef, MoLSS (undated ?2010). Child Protection Fund. In Support of the Government of Zimbabwe’s National Action Plan for Orphans and Vulnerable Children. Strategic Concept & Design. Save the Children (2009) Policy Brief. Making Children’s Rights a Reality PEPFAR OVC Guidance for OVC Programming (July 2012). Partners: JCT  Goodwell Gadzikano, WESS Consulting, Harare (Dec 2011). Assessment of the community paralegal project (community volunteer’s project). Africaid  Africaid (2006). Our Story written by Zvandiri Support Groups  Willis N., Mawodseke M. and Zvandiri Youth (2011). Red Ribbons and Roses.  Willis N., Mawodseke M. and Zvandiri Youth (2011). Relationships and Families Other NGOs  Plan Sept 2009. Research Report on Child Abuse in Schools. A baseline Study Report for the “Learn without fear” campaign.  Project Hope (2008). Best Practices. Income Generation in Support of OVC. Articles Masuka T., Banda R.G., Mabvurira V., Frank R. (2012). Preserving the Future: Social Protection Programmes For Orphans and Vulnerable Children (OVC) in Zimbabwe. In International Journal of Humanities and Social Science Vol. 2 No. 12 [Special Issue - June 2012] Loewenson R, Mpofu A, James V, Chikumbrike T, Marunda S, Dhlomo S, Milanzi A, Magure T (2008) Review of links between external, formal support and community, household support to Children affected by HIV and AIDS in Zimbabwe, NAC, TARSC, Harare Nupen D. (2006). Zimbabwe Parents of Handicapped Children Association (ZPHCA) Evaluation and Strategic Planning Report July 2006 Rusakaniko S., Chingono A., Mahati S., Mupambareyi P., Chandiwana B. (2006) Psychosocial Conditions of Orphans and Vulnerable Children in Two Zimbabwean Districts. HSRC Press, Cape Town. Accessed from www.hsrc.ac.za Grundmann N., Iliff P., Stringer J., Wilfert, C. (2011). Presumptive diagnosis of severe HIV infection to determine the need for antiretroviral therapy in children less than 18 months of age. Accessed from http://www.who.int/bulletin/volumes/89/7/11-085977/en/index.html Performance Evaluation of the Children First OVC Project for Zimbabwe 73 ANNEX F. CF TARGETS REACHED BY PEPFAR INDICATORS Type Indicator World Education Indicators FY08 Actual FY09 Target FY09 Actual FY10 Target FY10 Actual FY11 Target FY11 Actual FY12 Target FY12 Actual OGAC: Sexual Prevention - AB # of individuals reached with individual/small group interventions primarily focused on abstinence and/or being faithful. AB sexual prevention activities in school classrooms (30-45 children). 3000 3,493 2,150 5,118 2,150 18,082 5,000 5,464 OGAC: Prevention: PEP # of persons provided with post-exposure prophylaxis. By: Rape/Sexual Assault Victims Number of children provided with PEP after rape/sexual assault. 800 860 800 387 400 352 OGAC:Preve ntion: T&C # Of individuals who received testing and counselling services for HIV and received their results. # of children receiving C&T and receiving results through community based testing outreach activities. > By Male, <15 750 155 1000 592 1000 821 > By Female, <15 750 170 1000 736 1000 759 TOTAL 1500 325 2000 1,328 2,000 1,580 OGAC: Care Number of eligible adults & children provided with a minimum of one care service OVC provided with care service # of OVC (age: <18) provided with 3 or more services 13,402 55,000 47,356 65,000 75,608 75,000 84,495 > Female 7,549 28,600 24,815 33,800 39,599 39,000 43,260 >Male 5,954 26,400 22,541 31,200 36,009 36,000 41,235 # of OVC (age: <18) provided with 1 or 2 care services. 45,905 33,295 50,766 36,025 > Female 22,413 17,278 25,959 18,840 > Male 23,391 16,017 24,807 17,185 TOTAL 59,307 80,651 126,374 120,520 > By: Female 29,962 42,093 65,558 62,185 > By: Male 29,345 38,558 60,816 58,335 OGAC: Care: Support Care Number of eligible OVC who received food and/or other nutrition services << same 500 3,132 2,000 4,181 2,000 5,499 2,000 5,520 Number of OVC benefiting from 0 0 5737 113 5737 673 Performance Evaluation of the Children First OVC Project for Zimbabwe 74 Type Indicator World Education Indicators FY08 Actual FY09 Target FY09 Actual FY10 Target FY10 Actual FY11 Target FY11 Actual FY12 Target FY12 Actual income generating activities OGAC: Clinical care # of HIV-positive adults and children receiving a minimum of one clinical service (including CTX, TB etc.) # of HIV+ children receiving a minimum of one clinical care service > By age: <15: 1500 1,128 2,000 1,298 2,000 2,436 > female 750 529 1000 668 1000 1,218 > male 750 599 1000 630 1000 1,218 > By age: >15: 319 1,174 323 > female 164 554 183 > male 155 620 140 TOTAL 201 1500 1,447 2,000 2,472 2,000 2,759 OGAC: Clinical care # of HIV positive persons receiving CTX prophylaxis (subset of above) # of children <15 receiving CTX > By: Female 473 287 630 299 630 314 > By: Male 472 265 630 274 630 295 TOTAL 126 945 552 1260 573 1260 609 OGAC: Treatment # of adults & children with advanced HIV infection receiving ART [Currently on ART] # of children currently on ART > By Age: < 1 0 9 25 > By sex: Female 0 5 15 > By Sex: Male 0 4 10 > By Age: <15 (i.e. age 1-14) 555 609 740 831 740 1086 > By sex: Female 278 287 370 425 370 552 > By Sex: Male 277 322 370 406 370 534 > By age: >15: 293 469 740 205 > By sex: Female 151 237 370 106 > By Sex: Male 142 232 370 99 TOTAL 555 902 740 1,309 1,100 1,316 OGAC: Treatment # of adults & children with advanced HIV infection newly enrolled on ART # of children newly enrolled on ART (under 15) > Female 250 126 93 276 117 205 > Male 230 141 92 220 113 202 TOTAL 75 480 267 185 496 230 407 OGAC: HSS: HRH # of community health and para￾social workers who successfully completed a preservice training Number of providers/ caregivers trained in caring for OVC 2500 1438 1900 1074 2000 4,399 3,500 5,533 OGAC: HSS: HRH # of health care workers who 1200 37 15 66 100 149 Performance Evaluation of the Children First OVC Project for Zimbabwe 75 Type Indicator World Education Indicators FY08 Actual FY09 Target FY09 Actual FY10 Target FY10 Actual FY11 Target FY11 Actual FY12 Target FY12 Actual successfully completed an in￾service training program USAID FY09 Number of local organizations provided with technical assistance for HIV-related institutional capacity building. 10 19 20 17 20 15 20 22 Performance Evaluation of the Children First OVC Project for Zimbabwe 76 ANNEX G. CHILDREN FIRST JOINT INSTITUTIONAL ASSESSMENT TOOL WEI PARTNER CAPACITY EVALUATION Questionnaire for Organizational Capacity Building Evaluation Domain 1: Leadership, Governance and Strategy 1. Do you have a governing body (board, executive committee) that oversees the organization? Yes = 1 No = 2 Describe Board or executive committee: 2. Are the roles and responsibilities defined? Not defined = 1 Defined in constitution = 2 Defined in other = 3 Description of role / responsibilities of Board or executive committee: 3. Does the governing body play any role in policy setting, planning, fundraising, etc.? In policy setting = 1 planning = 2 fundraising = 3 in other = 4 Describe other: Describe how body plays the roles: 4. Do members have some experience in OVC services? Yes = 1 No = 2 Don’t know = 3 Other = 4 Brief description of experience that governing Body members have in OVC: 5. Are there documents that define the organization’s vision, mission and goals? Yes = 1 No = 2 Other = 3 Name the documents…….evaluator to have sight of documents: 6. Are the vision, mission and goal statements clearly understood by the board members and staff? By board members & staff = 1 By Board only = 2 By Staff only = 3 Other = 4 Evaluator’s assessment of level of internalization of vision, mission and goal: 7. Are your OVC programs in line with the organization’s vision, mission, and/or goals? Yes 2 No = 2 Yes and No (i.e. some) = 3 Other = 4 Describe any variances: 8. Is there shared leadership, i.e. from a director, governing body members, staff, and any others? Performance Evaluation of the Children First OVC Project for Zimbabwe 77 Yes = 1 In some cases = 2 No = 3 Explain answer: 9. Is the current leadership important for the survival of the organization? Very important = 1 To a limited extent = 2 Not important = 3 Other =4 Explain answer: 10. Does the staff have any role in decision making? Yes = 1 In some cases = 2 Not at all = 3 Other = 4 Explain the process for routine decision making in your organisation? 11. Do staff have any training in OVC programming and management? All staff = 1 Some staff = 2 None of the staff = 3 Other = 4 Explain answer: 12. Is the organization registered under local/national regulations? Yes = 1 No = 2 Other = 3 Give registration number: Evaluator to see registration certificate. 13. Are there any financial and legal benefits to registration? Yes = 1 No = 2 Other = 3 Explain the benefits, if any: 14. Is the organization in compliance with local reporting requirements? Yes = 1 No = 2 Other = 3 Explain who organisation reports to, how and frequency: Domain2: Program and Financial Management 15. Do you have any documents that describe the organizational structure? Yes = 1 No = 2 Other = 3 Explain answer and evaluator to see/review documents: 16. Are there management policies and are they implemented? Yes = 1 No = 2 Other = 3 Describe relevant policy documents and evaluator to review them: 17. How do you manage funds? Are they separate for each project? Performance Evaluation of the Children First OVC Project for Zimbabwe 78 Separate bank accounts = 1 Joint bank accounts for projects = 2 Other = 3 Explain how funds are managed: 18. Does the organization produce any financial reports (e.g. balance sheets, statement of operations, cash flows, etc.?) All the above reports = 1 Some of the reports = 2 None = 3 Indicate the types of reports produced and evaluator to review reports: 19. Does the organization have any budgeting process? If yes, is this reflected in the work-plan? Budgeting process reflected in work plan = 1 Budgeting not reflected in work plan = 2 No budgeting process = 3 Other = 4 Evaluator to review work plan samples. 20. Is there a regular financial audit? Is this done internally or externally? Regular external audit = 1 Irregular external audit = 2 No external audit = 3 Other = 4 Evaluator to review audit reports: 21. Do you have a system for administrative procedures, and is this system followed? (e.g. manual). Yes = 1 No = 2 Other = 3 Evaluator to review manual: 22. How does hiring work in your organisation? Guided by policy manual = 1 Not guided by a policy manual = 2 Other = 3 Evaluator to review manual. Brief description of process 23. Do you have a filing/recording system? Is this regularly maintained? Filing system regularly maintained = 1 Filing system not regularly maintained = 2 No filing system = 3 Other = 4 Evaluator to have sight of system if there is one. 24. Do you think your present staff strength is sufficient to support your present operations? Yes = 1 No = 2 Not sure = 3 Other = 4 Explain answer: 25. What specific OVC program skills do your staff possess if any? a) Performance Evaluation of the Children First OVC Project for Zimbabwe 79 b) c) d) 26. Do you use volunteers? If so, do you have a formal volunteer structure? Do you have enough /too many/too few volunteers? Enough volunteers with formal structure= 1 Insufficient volunteers with formal structure= 2 Enough volunteers without formal structure = 3 Insufficient volunteers without structure = 5 No volunteers = 4 If there are volunteers, explain how they add to the organizations’ goals? 27. How do you mentor volunteers and monitor their activities? 28. Does the organization develop annual work plans? Yes = 1 No = 2 Other = 3 If yes describe the process: 29. Does the staff participate or have input into the finalization of work plans? Yes =1 No = 2 Other = 3 Explain how: 30. Are the work plans useful, implemented, reviewed and used to track accomplishments? Yes = 1 No = 2 Other = 3 Explain answer: 31. Do you follow up recommendations from staff meetings? Yes = 1 No = 2 Other = 3 Explain answer: 32. Does the organization have any strategy for skill development/training of its staff, especially technical staff? Yes = 1 No = 2 Other = 3 Explain answer: 33. If yes, what type of training is offered? Formal training or informal training (e.g. participation in workshops and seminars, etc. including organizational capacity building e.g. financial skills and proposal writing.) Formal training = 1 Informal training = 2 Other = 3 Performance Evaluation of the Children First OVC Project for Zimbabwe 80 34. Has many staff been trained in OVC related services? Give examples. Number ……………………………….. and Examples: 35. Does the staff have access to relevant educational and resource materials? Yes = 1 No = 2 Other = 3 If yes, where? 36. What is the organization’s funding sources? Single or multiple; local or international? Single international = 1 Single local = 2 Multiple international = 3 Multiple local = 4 Multiple local & international = 5 37. Does the organization look to the private sector for funding? i.e. big supermarkets, companies, lotto, etc.? Yes = 1 No = 3 Other = 4 Explain answer: 38. Has the organization identified potential sources for future funding? If yes, how many? Yes = 1 No = 2 Other = 3 If yes, number……………………………………………… 39. What are your future plans for funding i.e. short-term, medium term and long term? Explain plans: 40. Does the organization have enough technical skills to write proposals? Yes = 1 No = 2 Other = 3 41. Who coordinates fundraising? CEO = 1 A Board member = 2 A dedicated fund raising person = 3 Other staff member = 4 Domain 3: Monitoring and Evaluation 42. What specific information do you collect on OVCs? a) b) c) d) Performance Evaluation of the Children First OVC Project for Zimbabwe 81 e) 43. Does the organization have a monitoring plan? Yes = 1 No = 2 Other = 3 44. If yes, who is it for and how it is done (ask about data collection, tools, analysis) The organisation = 1 Funding partner(s) = 2 Government = 3 Other = 4 Explain how done and tools: 45. Do you prepare any reports? If yes can you show some? Do you share these with community, government/ stakeholders, donors etc.? Yes &reports share = 1 Yes but reports not shared = 2 No reports = 3 Is shared explain who shared with: 46. Does the organization evaluate its programs? Yes = 1 No = 2 Other = 3 If yes, how is it done: 47. Do you have any feedback mechanism for your results/ achievements/ client satisfaction? Yes = 1 No = 2 Other = 3 If yes, what are the mechanisms? Domain 4: Comprehensive Services 48. Does the organization provide health care and/or prevention related services? Yes = 1 No = 2 Other = 3 49. If yes, what kind of services? Ask both about curative and preventive services. Curative = 1 preventive = 2 Other = 3 50. How are these services provided i.e. on-site or through linkages/networks and referrals? On site = 1 linkages/networks = 2 referrals = 3 Other = 4 Explain: 51. Does the organization have linkages/networks for improving access of OVC to these services? Yes = 1 No = 2 Other = 3 Explain: Performance Evaluation of the Children First OVC Project for Zimbabwe 82 52. Does the organization provide any kinds of nutritional support to OVC and /or households with OVC? Yes = 1 No = 2 Other = 3 Explain: 53. If yes, explain the services (Look whether they are on –site or though linkages/ networks e.g. school based food programs) On site = 1 linkages/networks = 2 referrals = 3 Other = 4 54. Do you provide safe drinking water? Yes = 1 No = 2 Other = 3 Explain: 55. How does your community participate in mobilizing to ensure that OVC have enough to eat? 56. Do you provide food for OVC from community sources? Yes = 1 No = 2 Other = 3 Explain: 57. Does the organization provide any educational support to OV? What is this? (Ask about basic assistance e.g. fees, school materials, boarding facilities etc.)? Fees = 1 School material = 2 Boarding facilities = 3 Other = 4 Explain: 58. Do you provide monitoring i.e. ensuring attendance, participation, minimizing drop-out etc.? Yes = 1 No = 2 Other = 3 Explain: 59. Are there some interventions for training the teachers in psychosocial care and special needs of OVC? Yes = 1 No = 2 Other = 3 Explain: 60. Do you provide non-formal education or vocational training services, please explain? Yes = 1 No = 2 Other = 3 Explain: 61. How is the organization linked to other education programs? 62. What do you understand by PSS? 63. What training has your staff received in counseling? Describe. Performance Evaluation of the Children First OVC Project for Zimbabwe 83 64. What strategies or interventions have you put in place to address the PSS needs of OVC? 65. What special support, if any do you provide to children on ARVs? Support for adherence to ARV drugs = 1 Counseling for care-givers = 2 Counseling for parents = 3 Other = 4 Explain: 66. Is the org. aware of children’s legal rights? Yes = 1 No = 2 Other = 3 What are you doing to address or promote children’s rights? 67. Do you have any linkages with other organizations that promote and protect children’s rights? Yes = 1 No = 2 Other = 3 68. What about legislative efforts? (Work with local or national government)? Yes = 1 No = 2 Other = 3 Explain: 69. Is your org. aware of or actively involved in accessing or assisting OVC to access social service grants provided by the government? Yes = 1 No = 2 Other = 3 70. Are there any programs or interventions to ensure socio-economic security of OVC or vulnerable households? Yes = 1 No = 2 Other = 3 If yes, please explain the programs and interventions? Specifically ask about the interventions to improve economic capacity? 71. Do you think that the efforts have achieved sustainable livelihood? To a large extent = 1 To a limited extent = 2 No = 3 Other = 4 If yes, explain how: 72. Do you support or have links to any micro financing /small credit /self help groups or similar schemes for OVC or households? Yes = 1 No = 2 If yes, explain. 73. Are there some initiatives /programs to address shelter needs of OVC? Yes = 1 No = 2 Other = 3 Performance Evaluation of the Children First OVC Project for Zimbabwe 84 Explain answer: 74. Does your org. work with care givers or child headed households to provide shelter for OVC? Yes = 1 No = 2 Other = 3 If yes, describe these initiatives 75. Does the org. have community support structures? Are these supported by community leadership and members? Community structures that are supported = 1 Community structures not supported = 2 No community structures = 3 76. Are there services to mobilize communities/ What activities are communities contributing to? How are they involved in OVC care and support? 77. How does the community participate in decision making? Domain 5: Creating an Enabling and Sustainable Environment 78. What initiatives or activities does your organization undertake to reduce stigma of OVC? (At community level, internally and with the children themselves?) 79. What do you know about national HIV & AIDS and OVC policies and strategies? 80. What coordination do you have with the government service providers? (e.g. ministries of social welfare, health, education, gender and youth and labour) 81. Do you exchange any information with local or national HIV or OVC programs or with policy makers and planners? If yes, please explain. 82. How does your current program relate to the national policy for OVC? Please highlight how you contribute to the National Action Plan for OVC. General Questions 83. Did you have a capacity building plan with WEI (Children First Project)? 84. In which areas/domains was WEI/CF supposed to build your capacity? 85. In which areas did you actually receive capacity building support from WEI/CF? 86. In what form did the capacity building support come from WEI/CF? 87. How beneficial was the capacity building support that you received from WEI/CF? 88. What made the support beneficial? Performance Evaluation of the Children First OVC Project for Zimbabwe 85 89. Is there any capacity building support that you got from WEI/CF that was not beneficial? 90. If yes to above question, why was the support not of any benefit to the you? 91. Did you receive capacity building support from any other sources besides WEI/CF in any of the following areas/domains? 92. Are you still in need of capacity building in any of the following areas/domains? 93. What plans do you have for developing your capacity in the areas/domains of need? Interview participant names and designations: Name Position/designation 1 …………………………………………………………………………. …………………………………………………………… 2 …………………………………………………………………………. …………………………………………………………… 3 …………………………………………………………………………. …………………………………………………………… 4 …………………………………………………………………………. …………………………………………………………… 5 …………………………………………………………………………. …………………………………………………………… Name of Evaluator: …………………………………………………… Date: ….……………………………………………………… Performance Evaluation of the Children First OVC Project for Zimbabwe 86 ANNEX H. ADDITIONAL RECOMMENDATIONS 1 Recommendations for Program Performance Data Quality Continue RDQAs and quarterly verifications on partner data. Support partners to strengthen their M&E systems to produce quality data for donor reporting and strategic decision making. CB for Partners  Incorporate an organizational capacity self-assessment into the pre- award assessment process  As part of initial capacity assessment and with partners’ permission, explore partners’ relationships with ministries and district and local OVC stakeholders for possible leveraging opportunities.;  Begin to develop CB plans right after awards are made. Inclusive CB workshops for all partners can be sequenced, scheduled well in advance and incorporated into work plans  Develop clear definitions for each of the three rating scores in the JIA tool, along with indicators to measure an organization’s progress from one level to the next.  Design phased CB strategies that match the capacity levels and needs of partners within each of the three rating scores. Ensure that all group workshops are followed up with individual on site coaching or mentoring1 that includes a standardized rating of partner progress  Develop clear and consistent graduation standards for all partners that define their readiness either to be directly funded by USAID or to effectively seek funding support from other donors, including public/private Corporate Social Responsibility (CSR) partnerships.2, 3  Contract with outstanding partners to share their area of expertise with others in a similar field? 2 Recommendations for Innovative Approaches: 2.1 BSIP Continue to engage with schools that received block grants or IGA support for SDA/SDC to monitor retention of tuition/levy supported students in school over time. Integrate community sensitization into the CDLP to make parents, guardians, community leaders aware of content before the program is initiated at a school. Include children’s input in the selection process for PSS teachers for the CDLP program. Ensure a wider consultative process to determine appropriate CDLP content (there was feedback that some information is unsuitable for children of certain ages), and a guide that indicates age-appropriateness for sessions. Establish a ceiling for percentage of profit through SDA/C grants that can be allocated to meet school needs other than OVC fees, and monitor in new program. Earmark the IGAs started under this program as pilots and evaluate them for effectiveness prior to the allocation of funds for replication of this model under the new OVC program. 1 Coaching: Intensive short-term CB focused on a specific area/topic, such as the financial and M&E requirements. Mentoring: Long-term CB focused on more developmental topics such as governance, resource mobilization, and strategic use of information for decision-making. 2 Evaluators learned that CSR is not well developed in Zimbabwe. Grassroots Soccer was the only partner who reported success in this area, in part because through its international presence. 3 Refer to the EGPAF OVCAT tool, http://www.pedaids.org/Publications/Toolkits/Transition/OCVAT/EGPAF-OCVAT. Performance Evaluation of the Children First OVC Project for Zimbabwe 87 Sensitize schools regarding potential stigmatization, e.g., not displaying lists of disabled and/or BEAM students. Consider assisting DSS and schools to attach student names to a unique identifier (anonymous code) rather than using the child’s name. 2.2 OOSG/CLASP Consider improving the current curriculum for students with more formal school experience and design a less accelerated curriculum that is better suited to students with minimal or no formal school experience. During curriculum review, focus on replacing stigmatizing language with empowering language. Also consider the socio-cultural context of students and ensure that the material is sensitive to and reflects children’s realities. Put in place minimum standards for the recruitment and qualifications of OOSG facilitators and monitor their teaching techniques. Develop minimum infrastructure/environmental standards for all new OOSG sites and, where possible, bring existing sites in line with these standards. Explore in-kind contributions from local businesses, as found in the Isibindi model (furniture, food or nutritious snacks, stationery) and ways to publicly acknowledge donations. 2.3 ECDC Review and attempt to integrate where possible best practice components of the various partner ECDC models. As noted above, look for partners that can provide nutritional support at ECDCs. 2.4 IMPACT Explore operational research to confirm that this model complements and strengthens national strategies and investigate workable strategies to address transport and nurse sensitization issues. 2.5 CATS For model’s expansion, carefully consider how and whether confidentiality regarding status of both volunteers and beneficiaries can be maintained, and how the quality of volunteer training and mentoring can be preserved. 2.6 YFC and EYFC: Encourage other USAID funding streams to create livelihoods opportunities for older youth. 2.7 CM, including supportive supervision: Support a review of intake and referral forms and feedback loops to identify all aspects along the continuum of care. Develop a complete checklist of all OVC services for use by community care workers. Consider contracting with organizations that offer supportive supervision and bereavement trainings for technical support. 2.8 CBC: Provide support for minimum standard of crisis counseling. 2.9 CPCs: Support the rollout of the CPC protocol through training and sensitizing CPC members, in conjunction with innovative strategies to resource the committees to enable their full function, including having some form of recognized identification. Consider provision of economic support to CPCs through ISAL group contributions. 3 Recommendations for Developing Local Capacity for Country Ownership Performance Evaluation of the Children First OVC Project for Zimbabwe 88 3.1 CB for PSS and Counseling: Explore additional counselor training technical partners such as REPSSI to provide the identified demand for PSS training; For all community volunteer cadres, post training follow up including supervision needs to be included as an essential CB component in new programming. Strengthen counseling training materials to include quality bereavement counseling. Integrate effective supervision structures to manage the emotional impact on care workers. Post training follow up needs to be included as an essential CB component in new programming. CB for Child Care Workers: Strategize ways to support the implementation of counseling standards for child care workers. CB for Community Paralegals: A comparative evaluation of the different paralegal models being piloted and their relative effectiveness would be useful for future programming. For all community volunteer cadres, post training follow up needs to be included as an essential CB component in new programming. 3.2 Capacity Building for Communities to Address Harmful Practices Align with CPF to utilize opportunities for community mobilization models such as the UNDP Structured Community Conversations to address issues of child protection, including child sexual abuse at community level. Emphasize a rights based approach in community sensitization and mobilization so that no one is coerced to test for HIV or disclose HV status. Restore a focus on urban street children and youth. 3.3 Building Community Economic Capacity to Support OVC: For grants of any size in the new OVC program include basic measures of accountability and funding effectiveness. Consider revision of ISAL model design to include support for vulnerable community members. Explore whether a portion of ISAL group profits could be directed to support CPCs or other community structures that support OVC. Incorporate an informal market analysis into ISAL strategy to prevent saturation of communities with competing similar individual businesses. 3.4 CB for Children with Special Needs: Support both teacher trainings for teachers who work with deaf children and follow up supportive visits to teachers’ classrooms. Recommend that the Kapnek Trust parent support group style be replicated at community level by any partners with the capacity to implement such an intervention in the interests of a family centered approach. Where possible, contract with partners who can train other partners in their good practice models, e.g. Kapnek Support Trust parent support groups. Continue to raise awareness of children and youth with special needs and assist more children with disabilities to access BEAM in order to attend school. Performance Evaluation of the Children First OVC Project for Zimbabwe 89 Explore with Kapnek Trust the use of their district survey tool to identify disabled children and link them with services. 3.6 Child Participation Carefully consider inherent risks of asking children to report cases of sexual abuse on behalf of other children and of asking children who are living with HIV to have their status be visible at community level. 4 Recommendations for Institutionalization of Models at National Level: Observe progress of NASW plan to integrate ten graduate social work interns at district offices and consider support for expanding strategy if results seem promising. Explore links with relevant authorities to address critical sanitation and hygiene problems on behalf of the most vulnerable OVC living in Caledonia and Hopley Farm. Support ministries to link models (SHA, IMPACT, CATS, ISAL) through case management referrals, and strengthened resources to promote a continuum of care within a family centered framework. Respond to any opportunity to enable links between BEAM (MOE) and AMTO (MOLSS) when both are sufficiently resourced. Consider how to support the implementation of new minimum standards for CCW developed by the CSW with the support of CF. Performance Evaluation of the Children First OVC Project for Zimbabwe 90 ANNEX I. CHILDREN FIRST PARTNERS ACROSS 5 YEARS PARTNER ORGANIZATION DURATION OF PARTNERSHIP FUNDING LEVEL AFRICAID 2008-2011 USD 323,514 Chiedza Child Care Centre (CCCC) 2008-2012 USD 575,869 Christian Community Partnership Trust (CCPT) 2008-2010 USD 122,604 Child Protection Society (CPS) 2008-2012 USD 463,120 Childline Zimbabwe 2008-2012 USD 363,168 Developmental Aids from People to People (DAPP) 2008-2010 USD 46,814 Farm Orphan Support Trust (FOST) 2008-2010 USD 105,260 Family Support Trust (FST) 2009-2012 USD 249,127 JF Kapnek Trust 2010-2012 USD 340,864 Mavambo Orphan Care 2008-2012 USD 1,582,388 New Dawn of Hope 2008-2010 USD 55,444 Nhimbe Trust 2008-2010 USD 88,081 Grassroots Soccer Academy (GRS) 2009-2011 USD 213,921 Hospice Association of Zimbabwe (HOSPAZ) 2008-2012 USD 377,872 Howard Hospital 2010-2012 USD 465,173 Justice for Children Trust 2008-2012 USD 363,677 NZEVE Deaf Centre 2011-2012 USD 147,196 Oasis Zimbabwe 2008-2012 USD 278,119 OPHID Trust 2012 USD 85,600 Scripture Union (SU) 2008-2010 USD 108,839 Seke Rural Home Based Care 2008-2012 USD 495,833 Umzingwane AIDS Network 2009-2012 USD 826,721 Performance Evaluation of the Children First OVC Project for Zimbabwe 91 ANNEX J. CHILDREN FIRST & PEPFAR INDICATORS 1. # of eligible children provided with a minimum of one care service. (CF/PEPFAR) a) OVC provided with 3 or more services b) OVC provided 1 to 2 services 2. # of eligible OVC who received food and/or other nutrition services (CF/PEPFAR) 3. # of organizations and community initiatives receiving funding from Children First sub-grants (CF) 4. % of children showing improvement according to the Child Status Index (CF) 5. # of USG assisted service delivery points providing family planning (CF) 6. % of local organizations that have been provided with management and or technical training relevant to their OVC programs (CF) 7. # of providers/caregivers trained in caring for OVC (CF proxy indicator for # of community health and Para￾social workers who successfully completed a pre-service training) 8. # of people (health/Para-health workers) trained in family planning/reproductive health with USG funds (CF) 9. # of OVC participating in community, district and national level advocacy activities (CF) 10. # of community groups targeted with information about OVC needs and child rights (CF) 11. # of individuals reached with individual/small group interventions primarily focused on abstinence and/or being faithful (PEPFAR) 12. # of persons provided with post-exposure prophylaxis. By: Rape/Sexual Assault Victims (PEPFAR) 13. # of individuals who received testing and counseling services for HIV and received their results (PEPFAR) 14. Number of eligible adults & children provided with a minimum of one care service (PEPFAR) 15. Number of eligible OVC who received food and/or other nutrition services (PEPFAR) 16. # of HIV-positive adults and children receiving a minimum of one clinical service (including CTX, TB, etc.) (PEPFAR) 17. # of adults & children with advanced HIV infection newly enrolled on ART (PEPFAR) 18. # of community health and para-social workers who successfully completed a pre-service training (PEPFAR) 19. # of health care workers who successfully completed an in-service training program (PEPFAR) 20. Number of local organizations provided with technical assistance for HIV-related institutional capacity building. (PEPFAR) Performance Evaluation of the Children First OVC Project for Zimbabwe 92 ANNEX K. CONFLICT OF INTEREST DISCLOSURE FORMS Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Deborah McSmith, MPH Title Consultant Organization IBTCI Evaluation Position XX Team Leader Team member Evaluation Award Number (contract or other instrument) SOL-613-12-000007 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Zimbabwe “Child First” 674-A-00-08-0021-00 I have real or potential conflicts of interest to disclose. Yes XX No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 30 August 2012