EVIDENCE FOR DEVELOPMENT E4D AGIR PF Performance Evaluation Findings Report International Business and Technical Consultants, Inc. July 2017 This report has been made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of International Business and Technical Consultants Inc. (IBTCI) EVIDENCE FOR DEVELOPMENT E4D AGIR PF Performance Evaluation Findings Report International Business and Technical Consultants, Inc. Disclaimer This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of International Business and Technical Consultants, Inc. (IBTCI) and do not necessarily reflect the views of USAID or the United States Government ACRONYMS AgirPF AgirPour la Planification Familiale AIBEF Association Ivoirienne pour le Bien Être Familial AIMAS Agence Ivoirienne de Marketing Social BCC Behavior Change Communication BPs Best Practices CBD Community-Based Distribution CHW Community Health Worker CoE Centers of Excellence COPE Client Oriented Provider Efficiency CSOs Civil Society Organizations CPR Contraceptive Prevalence Rate CPT Contraceptive Procurement Table CYP Couple Years Protection DHIS2 District Health Information Software 2 E4D Evidence for Development EH EngenderHealth FP Family Planning HIPs High Impact Practices HIV Human Immunodeficiency Virus HMIS Health Management Information Systems HPP Health Policy Plus HR Human Resources IP Infection Prevention IR Intermediate Result IUD Intrauterine Device JSI John Snow Incorporation LARC Long-acting reversible contraceptive LMIS Logistics management information system MCH Maternal and Child Health M&E Monitoring and Evaluation MOH Ministry of Health MOU Memorandum of Understanding NGO Non-governmental organization NRHP/PF National Reproductive Health Program OCAT Organizational Capacity Assessment Tool PAC Post-abortion care PACFP Post Abortion Care Family Planning PPIUD Postpartum intrauterine device PPFP Postpartum family planning PMP Performance Monitoring Plan PRB Population Reference Bureau PY Program Year QDAP Qualitative Data Analysis Program RAPID Recommend, Agree, Perform, Input, Decide RCPFAS Network of Advocacy Champions for Adequate Funding of Health RCPFASCI Network of Advocacy Champions for Adequate Funding of Health in Cote d’Ivoire REDI RH Rapport Building, Exploration, Decision Making and Implementing Decisions Reproductive Health RH Regional Health Office (USAID/West Africa) SBCC Social and Behavior Change Communication SRH Sexual and Reproductive Health SWT Site Walk Through TFR TOT UNFPA Total Fertility Rate Training of trainers United Nations Population Fund URBC Union des Religieux et Coutumiers du Burkina Faso USAID/WA U.S. Agency for International Development/West Africa USG United states Government WAHO West African Health Organization WHO World Health Organization WRA Women of Reproductive Age TABLE OF CONTENTS EXECUTIVE SUMMARY ................................................................................................................................................... i INTRODUCTION ............................................................................................................................................................ 1 Evaluation Questions ................................................................................................................................................... 3 Data Collection Methods and Instruments ............................................................................................................ 4 Sampling Strategy .......................................................................................................................................................... 6 Quantitative Data .................................................................................................................................................... 6 Qualitative Data ............................................................................................................................................................ 7 Data Quality and Analysis ........................................................................................................................................... 8 Quantitative .............................................................................................................................................................. 8 Qualitative ................................................................................................................................................................. 8 Data Issues and Limitations ................................................................................................................................... 9 Summary Profile of Evaluation Team ................................................................................................................ 10 PRIMARY QUESTIONS ................................................................................................................................................. 12 Primary Q1: How has AgirPF performed? ............................................................................................................ 12 Primary Q2: Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services] ........................................................................ 18 Primary Q3: To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? ...................................... 27 Primary Q4: For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? .................................................................................................................................................. 32 Primary Q5: What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? ..................... 37 Primary Q6: How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? ...................................................................................................................................... 41 RECOMMENDATIONS ................................................................................................................................................. 44 CONCLUSIONS ............................................................................................................................................................. 48 REFERENCES .................................................................................................................................................................... 49 APPENDIX 1: Evaluation questions and sub-questions ................................................................................ 50 APPENDIX 2: Number of intervention and control sites sampled for evaluation by Region and District ..................................................................................................................................................................... 52 APPENDIX 3: Full suite of data collection tools ............................................................................................ 53 ANNEX 1: Post-Evaluation Action Review Table…………………………………………………....97 ANNEX 2: Statement of Work……………………………………………………………………..104 ANNEX 3: Inception Report………………………………………………………………………..158 ANNEX 4: Desk Review Findings…………………………………………………………………..171 ANNEX 5: Presentation…………………………………………………………………………….203 EXECUTIVE SUMMARY INTRODUCTION The West Africa region has the lowest modern contraception use worldwide. Only 12% of married women of reproductive age were using a modern contraceptive method in 2016 compared to the global average of 56% (PRB, 2016). Consequently, the region has the highest total fertility rate (TFR) estimated` at 5.4 children per woman (PRB, 2016). This high level of fertility, among which 26% are unintended (Sedgh, Singh, & Hussain, 2014), coupled with high adolescent fertility (111 births per 1000 women aged 15-19 compared with 52 worldwide) and persistent decrease in child and maternal mortality rates, contribute to high population growth rates. In response to the high level of unmet need for family planning in Francophone West Africa, nine governments of Francophone West African countries and their technical and financial partners launched the Ouagadougou Partnership in February 2011 in Ouagadougou, Burkina Faso. This initiative includes the governments of Benin, Burkina Faso, Côte d’Ivoire, Guinea, Mali, Mauritania, Niger, Senegal and Togo. The current goal of the Partnership is to reach at least 2.2 million additional Family planning (FP) method users in the nine countries by 2020. At the outset in 2011, the national action plans of the nine countries set two objectives: 1) accelerate the achievement of their national goals for modern Contraceptive Prevalence Rate (CPR); and 2) reach at least an additional 1 million women by 2015. 1 Against this backdrop, USAID/West Africa Regional Health Office (RHO) funded the Agir Pour la Planification Familiale (AgirPF). The goal of AgirPF is to enable women of reproductive age (WRA) (15– 49) to make, and voluntarily act on, informed decisions about FP in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. The project works closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP. The approach is to: - leverage FP momentum, activating the “grassroots” to increase access to, quality of, and demand for FP, and working with the RHO and countries to adapt evidence-based practices (Result 2); - learn about these practices (Sub-Result 2.2); - feedback learning to national actors in the form of project/RHO advocacy for adoption and scale￾up, grassroots-led advocacy, and information that USAID can use to rationalize policies and contraceptive logistics (Result 3). - AgirPF strengthens public, private, and Non-governmental Organization facilities to provide a range of FP services (Result1), including integrated FP/maternal health services and services for youth/men (Sub-Result 1.1). The theory of change for the AgirPF activity is that if - the delivery of quality FP information, products, and services are strengthened and expanded; - evidence-based service delivery approaches selected, adapted, and implemented; and - efforts to remove policy barriers and improve contraceptive commodity security coordinated; then access to and use of FP services will increase in urban and peri-urban areas in Burkina Faso, Côte d’Ivoire, Niger and Togo (Mauritania is not included because of delayed project implementation). The purpose of the Performance Evaluation for AgirPF Project is to increase learning about the performance of the AgirPF activity in Togo, Côte d’Ivoire, Niger and Burkina Faso (in order of priority). 1 https://partenariatouaga.org/en/about-us/the-partnership/ Specifically, this evaluation will aim to answer: i. How has AgirPF performed (analysis of family planning service delivery statistics)? ii. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? iii. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? iv. For AgirPF IR 3: To what extent has, AgirPF contributed to removing policy barriers to FP access in the region? v. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? vi. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? See appendix 1 for full set of evaluation sub-questions METHODS This evaluation used a mixed methods approach, which included quantitative data from the program record files, AgirPF District Health Information System 2(DHIS2) database, as well as direct data collection on FP service delivery from a representative sample of AgirPF and comparison sites. Qualitative data in the form of purposive stakeholder interviews (including AgirPF staff, AgirPF trained providers, MOH partners, district and regional health managers, local partners and consortium partners) and analysis of quarterly reports were also used to triangulate and verify quantitative findings and to answer specific evaluation questions. Data collection and analysis took place from June 5-26th 2017. Additional follow-up data were collected from key informants in early July. Quantitative data were analyzed using STATA, a data analysis and statistical software tool and included descriptive statistics, trend analyses and statistical tests of significance to determine changes in family planning uptake between intervention and non-intervention sites. Qualitative data collected during key informant interviews were transcribed from recorded format and analyzed for content. The analysis followed the general thematic organization of the interview guides, and answers to specific inquiries of interest were extracted from each transcribed interview and organized by area of interest/theme. Within each theme, the content was once again analyzed for general trends related to the outcomes of interest. These findings were triangulated across different key informants, and analyzed in light of the quantitative data provided. The evaluation team for the AgirPF team was composed of an overall Team Leader and four in-country coordinators, one each representing Burkina Faso, Cote d’Ivoire, Niger and Togo where AgirPF activities are currently being implemented. The in-country coordinators were assisted in-country by a research team composed of data collectors and a supervisor who were all recruited from E4D’s Recipient Groups in the respective countries (except the Team leader in Cote d’Ivoire). In total, 25 data collectors and 3 supervisors were recruited and trained in Burkina Faso, Togo, Niger and Cote D’Ivoire to collect data, clean and conduct data entry. In addition, in-country coordinators from Burkina Faso and Togo were recruited from E4Ds RGs in-country. All in-country coordinators took part in the AgirPF methodology workshop which oriented them on data collection methods, tools and approaches for the evaluation. The involvement of RGs in the AgirPF performance evaluation is part of the “Learning by doing” strategy of the Capacity Building Component of the E4D activity which aim to build the research and evaluation capacity of RGs. Below are the details of the Evaluation team: I RESULTS 1. How has AgirPF performed (analysis of family planning service delivery statistics)? According to analysis of project indicator data, AgirPF is not on track to meet its pre-set goals of 700,000 new family planning adopters or its goal of 1,683,000 Couple Years Protection (CYP) by end of project. By the third quarter of its third year, the project has only reached 23% of its projected target for CYP (394,584 out of 1,683,000). As for new family planning users, the project has only achieved 29% (204,657 adopters) of their targeted goal by the end of the third quarter of year three. Using data collected from a sample of control and intervention sites, the evaluation team compared trends in family planning uptake by new users, returning users and specifically for LARC users in both AgirPF and control sites. AgirPF sites recorded increasing trends in new FP users over time, compared to control sites (p<0.01). However, there were no significant differences in trends of returning users for intervention versus control sites. LARC use trends did increase significantly at AgirPF sites compared to control sites over PY2 and PY3 (p<0.01). The team also examined trends in FP uptake across all AgirPF intervention sites using data from the DHIS2 database. AgirPF sites experienced a 67% increase in FP uptake among returning users, over the period under evaluation. Trends among new users increased by 29% in this same period. Additional analyses looked at trends in FP uptake among returning and new users by each country, which showed increases in new users in Cote d’Ivoire (210%) and Niger (19%), and gains in returning users in Burkina Faso (15%), Cote d’Ivoire (233%) and Niger (36%). Analysis of the raw increase in number of CYP and new adopters indicates that the project has not achieved their targets. However, this data alone belies the progress that AgirPF made on a number of fronts, including repositioning family planning within the West Africa region through targeted use of high￾impact practices, national-level policy advocacy and coordination of multiple regional and country-level actors. An analysis of additional quantitative data, including DHIS2 project data and a comparative data from control sites show that AgirPF did make significant gains in FP provision over the first three years of the project. 2. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services]. The AgirPF approach to developing and implementing HIPs was based on a regional strategy of evidence￾based decision-making and of building consensus and capacity among regional partners in order to deploy and adapt best practices at the country level. The key activity of AgirPF’s regional role in exchange, learning and dissemination of HIPs was the West Africa Health Organization (WAHO) 1st Conference on Good practices in Health (held in Ouagadougou in July 2015). This conference provided an opportunity to assess the effectiveness of HIPs at a regional level in order to facilitate the deployment of best practices across the region. The documentation and sharing of best practices from each country in the region led to the promotion of four key HIPs for adoption across countries: FP special days (mobile outreach), the post-partum family planning provision, the FP provision in post-abortion care, and the use of the RAPID (Recommend, Agree, Perform, Input, II Decide) model to address policy advocacy and socio-cultural advocacy with religious leaders. AgirPF played an instrumental role in the success of this regional exchange by providing targeted country-level support for development of the evidence-base as well as funding attendance of participants from target countries to attend the forum. In addition to the forum, AgirPF participated in a number of other initiatives to promote regional exchange including: Regional Advocacy Training - In collaboration with Health Policy Plus, Deliver and WAHO, AgirPF developed a coordinated training curriculum and workshops through with they supported stakeholders of the Network of Advocacy Champions for Adequate Funding of Health (RCPFAS) in Burkina Faso, Niger and Togo to develop advocacy strategies for improving access to FP. Regional Database (DHIS2) – AgirPF developed a regional family planning database through DHIS2 to track FP method use at the country and regional levels. Regional Social and Behavior Change Communication (SBCC) strategy – Through their partnership with the Camber Collective2, AgirPF developed a regional SBCC strategy to address the factors influencing FP use. Centers of Excellence (CoE) - Through training of trainers within Centers of Excellence in each country, AgirPF regional staff were able to build the capacity of CoEs in Burkina Faso, Togo, Cote d’Ivoire and Niger in key service delivery areas including facilitative supervision, FP service provision and Client Oriented Provider Efficiency (COPE). Besides the regional initiatives, AgirPF supported a number of HIPs across all countries, as evidenced by their programmatic reporting and verified by feedback from stakeholders. The following are a sample of HIPs promoted by AgirPF with corresponding examples of specific work in that area: ● Community health workers – Due to the work of AgirPF, an additional 981 community health workers were supported to provide family planning information or services in Cote d’Ivoire, Niger and Togo. CHW support was planned for Burkina Faso as well, but as of the 3rd quarter of PY3, no additional CHWs in that country were supported3. ● FP integration into Post Abortion Care (PACFP) units and postpartum wards (PPFP) – Through their provider training and facility-level support AgirPF, trained a number of providers in integration of family planning provision into post-abortion care and in the postpartum period. One-hundred and fifty-one providers were trained in postpartum provision of family planning, 180 were trained in postpartum Intrauterine Device (IUD) provision and 103 were trained in FP services integration into post-abortion care units across all countries through the third quarter of PY3. An additional 50 providers were trained in how to adapt PACFP for youth. 2 The Camber Collective is a strategy-consulting firm that takes a human-centered approach to tackling entrenched public health issues whose funders include the WHO, USAID and The Bill and Melinda Gates Foundation. 3 In Burkina Faso, Community based distribution for contraceptives distribution is not legal to date. However, AgirPF used CHWs for FP awareness (sensitization and Information provision) only throughout the project life in Ouagadougou, Bobo￾Dioulasso and Koudougou. In addition, pilot projects are implementing the approach and will deliver results at the end of year 2017. III ● Supply chain management/logistics training – Through a variety of training modalities, a number of providers at both the facility and district level were trained in techniques for improving supply chain and logistics management. Across all countries, 537 providers were trained in contraceptive logistics and logistics management information systems. An additional 97 providers were trained in COPE for contraceptive security, a technique designed to assist providers in planning for contraceptive supply at the facility level. ● Policy/advocacy support - AgirPF, through their regional and in-country advocacy work has made significative progress in improving policy support for FP and increasing financing for FP projects. For example, AgirPF worked with the Network of Champions in Advocacy for Adequate Funding of Health in Togo, Cote d’Ivoire, Burkina Faso and Niger to advocate for new policies that commit additional funding for FP services and also ease restrictions on access to FP for youth. In addition, the project engaged the religious leaders in Burkina Faso to adopt and implement a policy document aimed at promoting the responsible childbearing among their respective communities. Additional details of this work are covered in section 4, below. ● Family Planning Special Days (FPSD) –Through FP special days4, AgirPF was able to reach users at their Health Facilities and provide them with a wider range of free modern contraceptives including LARCs. By the end of the third quarter in PY3, AgriPF had performed 432 special FP days for the general population. ● Community engagement - AgirPF used the site walk-through method to engage communities with their local health facility FP service providers. They also worked with religious and community leaders to advocate for improved FP access. The full list of HIPs and corresponding countries are included in the matrix below: BPs/HIPs used to enhance family planning provision in each country COUNTRY BEST PRACTICE/HIGH IMPACT PRACTICE CHW PACFP Mobile Outreach PPFP FP - immunization integration Gender Integration Quality Improvement FP advocacy/ Policy Youth Friendly Burkina Faso N/A X X X X X X X X Cote d’Ivoire X X X X X X X X X Niger X X X X X X X X X Togo X X X X X X X X X 4 FP Special Days is typically three consecutive days event conducted at the service delivery point (SDP) to increase access to and use of FP services among women of reproductive age living in the peri-urban area of AgirPF intervention zones. During the course of these three days, a team of FP service providers from the district hospital and/or the neighboring Health facilities (HFs ) provides FP services, both Pills, injectable and Condoms (PIC) and long acting and reversible contraceptives (LARCs). Clients are informed in advance that services will be available. IV 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Overall, participants felt that the framework took a holistic approach to addressing improvements in access to family planning by including elements addressing service delivery, policy/advocacy reform and contraceptive security. Furthermore, there was an explicit relationship between the design of the program and the overall national strategic plans for FP, stemming from the Ouagadougou partnership. The overarching principles of the partnership emphasize increasing political commitment, leveraging evidence￾based best practices, strengthening national level FP plans and coordinating a diverse set of partners to increase impact. The AgirPF staff stated that the principles of the Ouagadougou Partnership guided the development of their framework. However, the framework lacks coherence in its logical structure and might benefit from revisions that focus efforts on clear steps towards achieving results. The framework also needs increased emphasis on demand generation at the community level and ways to address systemic resource challenges (such as human resources and facility infrastructure). Examination of the quantitative data in relation to this question (Section 1, above) suggests that although the indicated targets for CYP and new adopters were not met, the AgirPF project approach did enable increased access to and use of family planning when compared to control sites. In particular, the AgirPF project succeeded in significantly increasing trends in Additional users and, specifically, use of LARCs, when compared to control sites. The extent to which each element in the framework contributed to these results is difficult to disentangle, but based on feedback from key informants and project staff, the multi-pronged approach is critical to overall improvements. More specifically, stakeholders felt that FPSDs were the most efficient approach of addressing AgirPF’s overall objective. 4. For AgirPF IR 3: To what extent has AgirPF contributed to removing policy barriers to FP access in the region? Through their coordinated work with the Network of Advocacy Champions for Adequate Funding of Health (RCPFAS5) , AgirPF contributed to the strengthening of efforts to remove policy barriers to FP funding and access across the region such as: engaging faith-based organizations to advocate for the promotion of the responsible childbearing in Burkina Faso, in Cote d’Ivoire and Togo; engaging stakeholders from the Network of Champions in Advocacy for Sustainable Health Funding (RCPFAS) in all countries and building their capacity to advocate for the removal of policy and socio-cultural barriers to FP in those countries through targeted workshops and meetings; using the RAPID6 models specifically developed for the national authorities and for those based in the intervention cities of AgirPF countries. Activities included stakeholders from government offices, WAHO representatives, civil society organizations (CSOs) and donors, and prompted adoption of the RAPID model at the advocacy and grassroots level; and engaging youth organizations to advocate for youth-friendly sexual and reproductive health services. The policy change process is ongoing in most countries. There are several potential changes in the policy work. In Cote d’Ivoire, the advocacy work supported by AgirPF led to the allocation of 400 million CFA (about US $ 800,000) for contraceptives purchase in the 2016 budget. In addition, AgirPF advocated for 5 The RCPFAS has a presence in all AgirPF countries. Additional details on the development and role of the RCPFAS are included in the body of the report. 6 RAPID stands for Ressources pour l'Analyse de la Population et son Impact sur le Développement. V increased funding for FP and lower barriers to access through task-shifting of contraceptives distribution by CHWs and integration of FP into other Reproductive Health (RH) services. Advocacy for task-shifting of contraceptives distribution in Togo, which began under the AWARE II project7, was continued under AgirPF, leading to the adoption of a new community-based initiatives policy that allows CHWs to offer family planning methods, including injectables. Moreover, these policy initiatives led to a new cost-effective strategy for the MOH-Togo to continue implementing the approach without the support of a specific donor, as well as the allocation of 125 million CFA (US$ 250,000) for contraceptives purchase in the 2016 budget. In November and December 2015, AgirPF supported a series of meetings culminating in the validation and adoption of three new regulations (ministerial orders and decrees) to implement the RH Law in Togo: - List of products, methods and means of contraception legally authorized in Togo, - Decree on protection of service providers in RH services, - Decree defining the mission, organization, composition and functioning of a national inspection and control unit for reproductive health services in all facilities in Togo. Such a process supported by AgirPF in Burkina Faso led to the validation of two regulatory texts to translate RH law into practice in this country. AgirPF provided technical and financial support to the URCB8 that held advocacy meetings with Evangelical, Muslim and Traditional governing bodies for religious leaders to discuss responsible childbearing and prepare them for the adoption of a national policy document on responsible childbearing in Burkina Faso. 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? According to stakeholder input, AgirPF staff feedback, project document reviews and the evaluation analysis, the AgirPF project has several successes, challenges that could be used to advance future family planning programming: Successes: ● Use of evidence-based HIPs – One of the most important contributions of the AgirPF project was in not only identifying appropriate HIPs, but supporting countries to document and disseminate learning from application of specific HIPs so as to elevate the most important approaches in each context. The four key HIPs relevant for use in this region are FP special days (mobile outreach), post-partum family planning provision, FP provision in post-abortion care, and the use of the RAPID model to address policy advocacy and socio-cultural advocacy with religious leaders. In particular, stakeholders emphasized the use of mobile outreach FP days as a successful method of seeing immediate benefits in FP service delivery. ● Regional coordination through key partners, meetings, workshops and support – The AgirPF project worked through a number of mechanisms to coordinate regional collaboration and support for the project. This included collaborating with regional actors such as WAHO and other international consortium partners to develop strategic action plans in key program areas. ● Capacity building through training of trainers/Centers of Excellence – To promote sustainable capacity 7 AWARE stands for Action for West Africa Region-Reproductive Health 8 Union des Religieux et Coutumiers Burkina Faso VI building for trained FP providers, the AgirPF project identified Centers of Excellence in each country to participate in master FP training and supervision efforts. By investing in this training of trainers’ model, AgirPF created a more sustainable program for long-term FP training within countries. ● Strengthening advocacy efforts through workshops and applied tools – The AgirPF project, through their collaboration with local civil society groups and MOH counterparts, and the use of advocacy tools like the RAPID models was able to give policy advocates the tools and support they needed to engage with key groups (such as religious leaders) and push for policy reforms in each country. ● Updating of national policy, guidelines and strategic plans for FP – Through their coordination with MOH counterparts, and in alignment with the Ouagadougou partnership agreements, AgirPF worked at the national level in each intervention country to update FP guidelines to be inclusive of best practices, as well as ensuring that country FP strategic plans were aligned to meet goals. ● Logistics management coordination – Stakeholders noted the positive impact of AgirPF’s work to improve the commodities supply chain at facility and management levels. Additional work is needed to ensure adequate and continuous supply at the national level. Challenges: ● Lack of funding follow-through at national level – The project itself provided funding for programming, but participants feel that a long-term challenge to program success is in the follow-through of funding agreements made by the government. More advocacy is needed at the government level to ensure adequate budgeting and timely follow-through on commitments to funding FP services. ● Enduring socio-cultural barriers – Although the project did include SBCC aimed at changing cultural barriers to FP uptake, almost all respondents noted that this is an enduring challenge to program success. Additional, targeted efforts to influence socio-cultural norms and stigma around FP use, through a variety of media and community-based initiatives, could help improve demand generation and FP uptake among the target population. ● Lack of facility-level infrastructure – One of the primary challenges in providing an appropriate mix of family planning services was lack of infrastructure at the facility level. In particular, when integrating FP services into other services or adding new methods like IUDs, the lack of adequate space and privacy for patients was a notable barrier. ● Consistent, reliable commodities through the “last mile” – Despite advances in logistics management procedures of the project, stakeholders across several countries acknowledged persistent challenges to commodities stock-outs due to bottle-necks,break-downs in the supply chain system and more especially to insufficient resources allocation for procurement of FP commodities. ● Staff turnover at facilities – At the facility level, staff turnover is very high and thus staff that are trained in FP methods through the project may be relocated in a short amount of time. New staff coming may not receive specific FP training, resulting in a gap in service delivery at the facility level. Although the project attempted to address such issues by increasing the pool of master trainers through Centers of Excellence in each country, several respondents recommended also using regular on-site training or incorporating FP into provider pre-service training curricula to address this issue. VII ● Understaffing of AgirPF regional/country staff – Overall, the AgirPF staff felt that for the size and scope of the activities directed under this regional project, there were insufficient staff at all levels. There were delays in hiring and onboarding of staff at the beginning of the project, as well as frequent turnover and gaps in human resources that made it difficult to manage the number of activities and partners involved. Future regional projects should consider the needs of appropriately managing the multitude of activities on a project of this scope and create staff positions accordingly. ● Lack of harmonization between project activities and government initiatives - Burkina Faso in particular, the regional and district managers which the evaluators interviewed felt that AgirPF did not fully coordinate their activities through the existing government structure and were prone to pursuing interventions without passing through the appropriate authorities. They also mentioned a need for increased coordination with other existing NGOs working in that space. ● Delays in financial mechanisms at EH headquarter level – The AgirPF staff and local partners both noted that at times there were delays in financial disbursements from the project, which led to delays in activities. ● Heavy data reporting burden at facility level – Staff at facilities reported being burdened with the number and type of data reports owed to AgirPF project by the facility staff. In some cases, providers felt these were parallel systems not fitting with existing national indicator definitions and collection methods. That duplicated reporting efforts for already over-taxed staff. 6.. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Relationships between country and regional staff at AgirPF were positive, with reports of timely communication, team-oriented approach and good management procedures. However, during internal conversations with AgirPF staff, there was a theme around the project being understaffed at both regional Both the project and USAID have recognized this, and added a significant number of country staff from the time this report was written and country level. Overall, the local and consortium partners reported positive views of working with AgirPF staff. AgirPF staff reportedly had an open and communicative style, and provided appropriate support in their approach to working with partners. Here, again, partners noted some delays in funding disbursement that posed problems in activity planning and management. Anecdotally, the only major complaint in partnership management for AgirPF came from their municipal government counterparts in Burkina Faso, who felt that they were not adequately included in program planning and implementation9. A purposive sample of ten AgirPF sites per country were visited to determine environmental compliance during the course of field data collection in each country. Facility staff were asked about the ways in which the site disposed of both solid and liquid waste. 9 In 2014, AgirPF Burkina Faso started advocacy activities implementation with local municipal gouvernements in Ouagadougou, Koudougou and Bobo-Dioulasso using RAPID models. Unfortunately, the activities stopped in October 2014 due to the popular insurrection resulting in a destructuration of the communalisation with suppression of the mayors. It was not until 2016 after the elections and the setting up of the town halls to start again the work with communes. This caused a feeling of abandonment. VIII For medical waste management, 8 out of 10 facilities in Togo, 4 out of 10 facilities in Cote d’Ivoire and 5 out of 10 facilities in Burkina Faso reported collecting and milling waste outside, with only 2 out of 10 sites in each country reporting the use of an incinerator. In Niger, 5 out of 10 sites reported burning waste in an incinerator, while 3 burned waste in an open hole, 1 burned and buried waste and 1 collected and milled waste outside. These findings show no improvement over environmental compliance recorded at the baseline for intervention sites. For disposal of liquid waste, Niger was the only country that reported disposing of liquid waste in septic tanks (9 out of 10 sites). In Cote d’Ivoire, 4 out of 10 facilities reported disposing of liquid waste in an open hole, in Togo, 3 out of 10 facilities reported collecting liquid waste in containers and Burkina Faso facilities reported using open holes and wells for liquid disposal. Note that in all countries but Niger, in numerous facilities, respondents did not know the actual mode of liquid waste disposal. RECOMMENDATIONS: The following recommendations stem from the evaluation team’s overall synthesis of quantitative, qualitative evidence collected and analyzed over the course of the performance evaluation: ● Focus on “quick wins” through mobile outreach – While participants appreciated a multi-faceted approach to long-term FP improvement, many mentioned the importance of focusing efforts on providing immediate benefits to target populations. Namely, the use of FP special days and mobile outreach to provide methods for those in need was viewed as one of the most successful aspects of AgirPF’s work and should be replicated/promoted throughout the focus countries. ● Improve timeliness of finance and accounting structures – Both AgirPF regional/country staff and their counterparts in-country mentioned that the approval processes for purchase orders and disbursement were seen as an impediment to rapid response and action on the ground. Future projects should streamline this process to enable more nimble and reactive capabilities. ● Provide adequate project technical and administrative staff – For a project of this size and complexity, it is essential that the staffing be sufficient and in place as soon after agreement signing as possible. The AgirPF project struggled to staff up in the first year of its program, which led to delays in roll￾out of activities. ● Continue to promote and propagate high-impact practices – The use of high-impact practices was viewed as a success by many of the participants and should be continued in future programming. The use of forums like the one on Good Practices, where HIPs were tested in each country and lessons shared in a regional format, are of particular benefit to aiding the scale-up of best practices across the region. ● Continue to push for policy change – Though the gains in policy advocacy are less immediate and tangible than other areas of AgirPF’s work, both civil society and MOH partners view this as a critically important effort that must continue in order to have large-scale impact on FP service provision. In particular, countries should continue to engage important gatekeepers like religious leaders to demonstrate the urgency of responsible childbearing to the population’s survival. Future projects should also continue to push for inclusion of vulnerable and neglected groups like youth. ● Capitalize on positive achievements to encourage buy-in at national level - Overall, almost all of the participants the evaluation team interviewed viewed the work of AgirPF as essential and effective. IX As such, the evaluation team recommends that the successes of the project be clearly distilled and promoted at the national level in order to encourage government buy-in and continuation of activities beyond the life of the project ● Take into account the need for infrastructure improvements – Across all countries, actors noted the need to provide structural improvements to facilities in order to be able to provide adequate FP service improvements. Without critical basic elements such as private rooms, beds, chairs and cabinets, many facilities will remain incapable of providing access to adequate family planning services. ● Improve provider’s working conditions – Though beyond the scope of most projects of this nature, it is worth noting that several participants in this study felt that provider training alone was not sufficient to ensure their engagement and promotion of FP services. Creative solutions to enduring human resources for health issues must be incorporated into future models of change in this region. ● Devolve provider training to site-level and/or provide more regular FP training – Due to high staff turnover at facilities, many AgirPF trained staff were already transferred from the facilities that the evaluation team visited. This is a common issue where human resources are scarce. Thus, devolving provider training to on-the-job or on-site formats and/or providing regular training to a broader mix of facility staff can help ensure continuity and presence of trained FP providers. ● Institutionalize provider training in FP – Another suggestion made for improving and ensuring provider readiness for FP service provision was to institutionalize FP training into exiting health provider pre-service training curricula at the national level. AgirPF began this work by working with MOHs to establish national centers of excellence and providing training of trainers’ activities in each country and also granting Teaching Hospital for FP teaching curriculum in Cote d’Ivoire. However, these activities must be maintained diligently and ownership transferred successfully to national MOHs to ensure sustainability. ● Increase supervisory visits – Several providers noted the benefits of facilitative supervision visits and requested that such visits be more frequent. Through increased supervisory visits, programs can ensure trained providers are cementing their FP services skills and can further identify gaps at the facility level if/when trained providers are relocated. ● Provide continuity in future programming to continue gains made – As of the time of this evaluation, participants and AgirPF staff felt that the program was just getting into full swing. Future projects should carry forward the successful aspects of this program through a seamless transition to further funding and programming so as not to disrupt the potential gains stemming from current activities. E4D: Draft 2 AgirPF Performance Evaluation report 1 INTRODUCTION The West Africa region includes 21 countries with a population of approximately 359 million (Population Reference Bureau (PRB), 2016). The region has the lowest modern contraception use worldwide. Only 12% of married women of reproductive age were using a modern contraceptive method in 2016 compared to the global average of 56% (PRB, 2016). Consequently, the region has the highest total fertility rate (TFR) estimated` at 5.4 children per woman (PRB, 2016). This high level of fertility, among which 26% are unintended (Sedgh, Singh, & Hussain, 2014), coupled with high adolescent fertility (111 births per 1000 women aged 15-19 compared with 52 worldwide) and persistent decrease in child and maternal mortality rates, contribute to high population growth rates. The West Africa population is expected to increase from 359 million in 2016 to 515 million by mid-2030 and 800 million by mid-2050 (PRB, 2016). Such population volumes constitute a threat for the future of the region (available resources, economic growth and population wellbeing). In response to the high level of unmet need in Francophone West Africa, nine governments of Francophone West African countries and their technical and financial partners launched the Ouagadougou Partnership in February 2011 in Ouagadougou, Burkina Faso. This initiative includes the government of Benin, Burkina Faso, Côte d’Ivoire, Guinea, Mali, Mauritania, Niger, Senegal and Togo. The current goal of the Partnership is to reach at least 2.2 million additional family planning (FP) method users in the nine countries by 2020. At the outset in 2011, the national action plans of the nine countries set two objectives: 1) accelerate the achievement of their national goals for modern Contraceptive Prevalence Rate (CPR); and 2) reach at least an additional 1 million women using modern FP methods by 2015.10 These action plans mapped their priority steps for strengthening national FP programs. Against this backdrop, USAID/West Africa Regional Health Office (RHO) funded the AgirPF project. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. The project works closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP. The approach is to leverage FP momentum, activating the “grassroots” to increase access to, quality of, and demand for FP, and working with the RHO and countries to adapt evidence-based practices (Result 2); learn about these practices (Sub-Result 2.2); feedback learning to national actors in the form of project/RHO advocacy for adoption and scale-up, grassroots-led advocacy, and information that USAID can use to rationalize policies and contraceptive logistics (Result 3). AgirPF strengthens public, private, and NGO facilities to provide a range of FP services (Result1), including integrated FP/maternal health services and services for youth/men (Sub-Result 1.1). To bring FP services to underserved communities, AgirPF supports mobile outreach services; brings health fairs to industries and community sites; and offers “city-based services,” an adaptation of EngenderHealth￾managed Community-based Distribution (CBD) in Togo. To lower client cost, AgirPF provides dedicated FP services at low/no cost (special FP days) in each city. To solve logistics issues and estimate commodity needs, AgirPF assists facilities to use Client – Oriented – Provider - Efficiency (COPE) for Contraceptive Security and Ministries of Health’s Contraceptive Procurement Teams to use Reality Check for contraceptive quantification (Sub-Result 3.2). AgirPF also provides training and refresher trainings to healthcare providers in FP service delivery, including infection prevention (IP), FP counseling (using the rapport building, exploration, decision making, 10 https://partenariatouaga.org/en/about-us/the-partnership/ E4D: Draft 2 AgirPF Performance Evaluation report 2 and implementing the decision (REDI) framework), facilitative supervision and gender sensitization. AgirPF supports community leaders with FP advocacy activities. The theory of change for the AgirPF activity is that if a) the delivery of quality FP information, products, and services are strengthened and expanded; and b) evidence-based service delivery approaches selected, adapted, and implemented; and c) efforts to remove policy barriers and improve contraceptive commodity security coordinated, then access to and use of FP services will increase in urban and peri￾urban areas in Burkina Faso, Côte d’Ivoire, Niger and Togo (Mauritania is not included in this evaluation because of delayed project implementation). As such, this evaluation used the results framework below as a guide for the overall assessment. Figure 1: AgirPF Results Framework AGIR-PF Objective Increase access to and use of quality family planning services in selected urban and peri-urban areas of four francophone West African countries AGIR-PF Result 1: Delivery of quality family planning information, products, and services strengthened and expanded Sub-Result 1.1 Partners strengthened to implement evidence-based approaches and deliver quality family planning services Sub-Result 1.2 Local leaders, civil society, service providers, municipal government support and promote family planning AGIR-PF Result 2: Evidence-based service delivery approaches selected, adapted, and implemented Sub-Result 2.1 Efficiency and effectiveness of approaches enhanced through the adaptation and implementation process Sub-Result 2.2 Lessons documented and disseminated on learning from adaption and implementation processes and experiences AGIR-PF Result 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-Result 3.1 Operational policy barriers identified and new/revised policies adopted and implemented Sub-Result 3.2 Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees E4D: Draft 2 AgirPF Performance Evaluation report 3 The purpose of the performance evaluation for AgirPF Project was to increase learning about the implementation of the AgirPF activity in Togo, Côte d’Ivoire, Niger and Burkina Faso (in order of priority). The USAID/WA health office wants to know and document whether the AgirPF project is on track for achieving its intended results; has advanced select high impact practices; if its intermediate results were necessary and sufficient to achieve expected results and what the activity’s key successes, challenges and lessons learned are. It serves as a performance evaluation of AgirPF to determine the extent to which the AgirPF portfolio has met its overarching objectives of: (1) Strengthening partners to implement evidence￾based approaches and deliver quality family planning (FP) services; (2) Enhancing efficiency and effectiveness of FP delivery approaches through the adaptation and implementation process; (3) Identifying operational policy barriers and new/revised policies adopted and implemented; (4) Supporting local leaders, civil society, service providers, municipal government in promoting FP; (5) Documenting and disseminating lessons learned from adaption and implementation processes and experiences; and (6) Identifying and coordinating contraceptive commodity needs among partners and country commodity security and logistics management committees. This evaluation complements any evaluation efforts already implemented as part of the AgirPF project’s PMP. The environmental compliance evaluation was designed based on the original data collection tools employed by AgirPF at baseline; the evaluation team attempted to measure the same items as were assessed at baseline. The target audiences for the AgirPF performance evaluation are the USAID/WA Front Office; USAID/WA Regional Health Office (RHO); other USAID health offices in the region, the Governments of Togo, Côte d’Ivoire, Niger and Burkina Faso Ministries of Health, the implementing partner EngenderHealth and other donors in the health sector as well as stakeholders in family planning and reproductive health in West Africa. Evaluation Questions The performance evaluation aims to answer six specific questions about the performance of the AgirPF project. These questions are designed not only to answer the question of whether or not AgirPF is meeting its performance objectives, but also how the project is being managed and perceived across its implementation countries, partner organizations and stakeholders. Specifically, this evaluation will aim to answer: 1. How has AgirPF performed (analysis of family planning service delivery statistics)? 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? See Appendix 1 for the full list of sub-questions. Underlying these evaluation questions are a series of queries related to the “processes” and “outputs” of the AgirPF project. Namely, the evaluation will serve to clarify if activities were carried out as planned, how well they were conducted, and if expected changes are occurring in terms of improved access to E4D: Draft 2 AgirPF Performance Evaluation report 4 family planning and progress on changing policy barriers to family planning in each country and at regional level. Data Collection Methods and Instruments This section describes the specific data collection methods and instruments used in answering evaluation questions. Facility records review – The evaluation team compared health management information system data from a sample of AgirPF facilities and comparison sites in Burkina Faso, Cote d’Ivoire, Niger and Togo to examine impact of the project’s work on provision of family planning.The team collected monthly family planning data from 6 months prior to the onset of activities, through the most recent month at the time of data collection. Data specific to 1) total number of family planning users, 2) total number of new method users and 3) method mix of contraceptive captured by the AgirPF DHIS2 database were used to analyze trends at intervention sites. For comparison sites, the team collected data directly from facilities through monthly facility reports or FP data registries. The team selected a simple random sample of AgirPF and comparison facilities for facility health records review. For AgirPF sites, specifically, site level data was extracted by an independent data assistant from the DHIS2 database housed at the AgirPF headquarters in Lomé. Data from all AgirPF sites was extracted, and a sample of sites from among the total was used for comparison against non￾intervention sites (sampling plan is described further, below). After extraction was complete, the independent database was shown to the AgirPF monitoring and evaluation team to ensure its quality and completeness; the evaluation team also conducted a spot-check of the data to ensure its completeness. The evaluation team coordinated this data extraction process directly with the AgirPF team. Finally, the team purposively selected 10 AgirPF sites for an environmental compliance assessment in each country. These sites were selected from among the districts that the evaluation teams were visiting during comparison site data collection. Where possible, the evaluation teams worked with AgirPF staff to ensure they are only evaluating facilities in which AgirPF provided support for environmental compliance. The team selected ten sites on the basis of the maximum amount of facilities that they would be able to assess in the time period given. Provider interviews – The evaluation team interviewed health care providers - including doctors, nurses and auxiliary nurse midwives-from a sampled AgirPF facilities to understand their experience with training and support under the project. The team also queried them on the implementation of high-impact practices such as postpartum family planning and FP integration into post abortion care, as well as their experience with tools such as COPE and other relevant planning tools. Providers were also asked about perceived successes, challenges and lessons learned from working with the AgirPF project to advance family planning programming. Those who were involved directly in AgirPF trainings or facility-based intervention were prioritized for interview. In some cases, providers may have been trained by other providers previously trained by AgirPF; these individuals were also included for interview if no other provider who was directly trained by AgirPF is available. The evaluation team determined eligibility of providers for interview through consultation with the AgirPF training list and pre-selection screening at the facility level. A minimum of one provider from each sampled AgirPF facility was interviewed. E4D: Draft 2 AgirPF Performance Evaluation report 5 District health manager interviews– The team interviewed district and regional health managers (including reproductive health coordinators) from each intervention country, with whom AgirPF coordinated trainings, built capacity and engaged for change, to understand their experience with the program. Specifically, the team aimed to understand the effectiveness of AgirPF’s support with regards to training, coordination and implementation of high impact practices at the local level. Utility and sustained use of tools such as COPE, Reality Check, Logistics Management Information Systems (LMIS) training and other relevant approaches were explored from the perspective of the health management teams in each country. Health managers were also asked about perceived successes, challenges and lessons learned from working with the AgirPF project to advance family planning programming. At least one health manager from each district and region of AgirPF intervention areas in each country was interviewed. Ministry of Health interviews – The evaluation team interviewed stakeholders from the Ministry of Health to examine the effectiveness of AgirPF’s overall coordination with government partners at the national level. MOH Stakeholders were also asked about the role that AgirPF has played in removing policy barriers to family planning at the national level. The team interviewed one MOH stakeholder from each country. Partner organization interviews–In order to understand AgirPF’s performance in engaging with and managing activities with partner organizations, the team interviewed members of local and international consortium partners. The evaluation team asked participants about their experience working with AgirPF and about success and challenges of coordinating efforts with the project. A particular focus was put on staff at “centers for excellence”, in which AgirPF attempted to institutionalize training systems, as well as local advocacy organizations, with whom AgirPF engaged to advance policy changes in FP. AgirPF country staff interviews– The evaluation team used interviews with AgirPF country staff to understand the performance of headquarters and regional management staff with regards to overall conduct of the program. In particular, the evaluation team investigated issues related to planning, execution and management of implementation to understand the adequacy of AgirPF’s approach. At the country level, the country program manager and senior program officer were interviewed. At the regional level, heads of each department (programs, HR, finance, M&E, etc.) as well as the technical director/acting Chief of Party were interviewed. Policy review – All relevant country-level policies related to family planning were reviewed in the period of intervention to examine any changes or movement in reducing barriers to FP. Reports from MOH stakeholders informed the policy review. Project files review – The project files were previously audited as part of the desk review. However, the team re-examined them in light of the broader evaluation as a means of triangulating data on FP provision and program implementation within each country. E4D: Draft 2 AgirPF Performance Evaluation report 6 Sampling Strategy Quantitative Data The facility level records review took place in a subset of intervention facilities in which AgirPF carried out programs. These facilities were selected on the basis of a simple random sample of facilities in each district of AgirPF intervention. Approximately thirty intervention facilities in each focus country (Burkina Faso, Cote d’Ivoire, Niger and Togo) were selected for inclusion in the sample. The selection was made randomly at the regional level, from among a list of AgirPF sites; a best effort was made to include representation from all project districts. An equal number of non-intervention facilities in each region were selected as comparison sites. These comparison sites were also selected using a random selection process at the regional level. In cases where there are fewer than 30 comparison sites (for example, in Togo), then all available comparison sites were included. The evaluation team selected 30 sites per arm per country to enable a statistically significant trend analysis. Table 1 details the number of facilities to be selected from each country and region therein for both intervention and comparison sites. Table 1: Number of facilities to be selected from each country/ region for intervention and comparison sites. Country REGIONS Total number of AgirPF interventionsit es Number of AgirPF intervention sites for evaluation Number of AgirPF comparison sites for evaluation BURKINA Ouagadougou 34 18 14 Koudougou 9 5 9 Bobo 14 7 7 TOTAL 57 30 30 COTE D'IVOIRE Abidjan 1 Grands Ponts 38 11 9 Abidjan 2 45 19 21 TOTAL 83 30 30 NIGER Niamey 15 14 15 Maradi 21 19 15 TOTAL 36 30 30 TOGO Lomé 19 12 9 Sokodé 14 9 7 Kara 15 9 8 TOTAL 48 30 24 See Appendix 2 for a break-down of intervention and control sites by region and district for each country. Health Management Information System (HMIS) records from these facilities in each country were accessed directly at facility level, with the cooperation and assistance of district health managers. The HMIS records dated from 6 months prior to AgirPF training implementation (variable depending on country), up and through June 2016. In the case that HMIS data were not available, the evaluation team extracted facility-level data directly from monthly facility reports. Table 2 depicts the dates of data collection in each country. These dates are selected to correspond with the availability of data in AgirPF’s DHIS2 HMIS database. It should be noted that AgirPF staff have confirmed that facility-level interventions E4D: Draft 2 AgirPF Performance Evaluation report 7 usually did not begin until 6-12 months AFTER the original launch date due to the requirement by USAID to complete the data collection for baseline study. For example, if the launch date in Burkina Faso was October 2014, site-level activities would not have started before April, 2015. As such, the evaluation data collection start and end dates correspond with our desire to collect data 6 months prior to start of activities. Table 2: AgirPF start dates and corresponding evaluation start and end dates. Country Start of AgirPF Program Start of AgiPF data collection Start of Evaluation data collection End of Evaluation data collection BURKINA January 23, 2014 October 2014 October 2014 September 2016 COTE D'IVOIRE October 16, 2014 April 2015 April 2015 September 2016 NIGER January 21, 2014 October 2014 October 2014 September 2016 TOGO January 09, 2014 October 2014 October 2014 September 2016 Qualitative Data For all individual interviews, a purposive sample of participants were chosen from among the following groups (Table 3): ● Health providers: Ten AgirPF sites were randomly chosen from among the areas where comparison sites are being visited by evaluators. At least one provider per site was interviewed as to he/she experience with AgirPF. The team chose providers from the same ten sites that the team selected for the environmental assessment, and focused on at least one AgirPF provider due to their availability and time available for data collection. ● Local health managers: Per country, three regional/district managers were selected from among the districts/regions being visited for comparison site data collection. These local health managers were screened in advance to discern their availability and eligibility on the basis of their knowledge of an exposure to AgirPF programming. ● Partner organizations interviews: Individuals from partner organizations, defined as both international and local consortium partners and local policy-advocacy groups, were interviewed. Participants were selected with the guidance of in-country AgirPF staff, and selected on their basis of partnership and collaboration with AgirPF for implementation activities. For local advocacy groups, individuals who underwent training and planning sessions with AgirPF were selected. From each organization, a minimum of one staff members were selected for interview. When available and relevant, additional staff from each organization were interviewed. ● AgirPF country staff interviews – Key EngenderHealth staff from each AgirPF country program were selected for interview based on their experience with the management at EngenderHealth regional and headquarters level. This includes heads of each department in focus countries (finance, M&E, operation, programs, etc.), as well as the acting manager in each country. E4D: Draft 2 AgirPF Performance Evaluation report 8 Policy and project records review was completed using documents gathered from AgirPF and respective MOH in each country. Table 3: Sampling for qualitative interviews Country Environmental Compliance Provider Interviews MOH Interviews * Distrcit manager interviews Partner Interviews AgirPF staff interview s Burkina Faso 10 sites 10 1 3 5 (consortium) 4 (advocacy) 2 Cote d’Ivoire 10 sites 10 1 3 3 (consort) 1 (advocacy) 1 Niger 10 sites 10 1 5 3 1 Togo 10 sites 10 1 3 2 (consort) 1 (advocacy) 2 * MOH interviews included government partners referred by AgirPF and often included counterparts in MCH or public health departments; due to confidentiality issues, the team are not at liberty to specify their particular titles in each country. Data Quality and Analysis Quantitative Monitoring data provided by AgirPF were verified for completeness and comprehensiveness by the data entry person who did a random spot-check of 20% of the data and the Team Lead, who checked the data for any missing cells or obvious outliers. HMIS data were also verified at both by the data collection team and the in-country supervisors for completeness and comprehensiveness on a daily basis throughout the data collection process, by checking for missing data or outliers. All HMIS data were downloaded directly into excel, verified by in-country teams and transferred to STATA for analysis by the evaluation lead in each country. In the event that data were collected directly from monthly facility reports, they were cleaned and entered into excel by each in-country team and verified by a double data entry process at the country level. Any discrepancies found in the double data entry process were resolved by referring back to the original paper tools. Quantitative data were analyzed using STATA and included descriptive statistics, trend analyses and statistical tests of significance to determine changes in family planning uptake between intervention and non-intervention sites. The AgirPF baseline report only presents statistics related to facility characteristics and population-level family planning behavior, but provides no specific detail on site-level FP uptake. As there were no FP service delivery statistics in the baseline AgirPF report, the team created additional analyses using the totality of the AgirPF DHIS2 database. The team examined all monthly data from all AgirPF sties in each country to ascertain trends in FP provision for both returning and new users in each country. Qualitative Qualitative data collected during key informant interviews were transcribed from recorded format directly into Word documents. One member of each country team reviewed all transcriptions to ensure E4D: Draft 2 AgirPF Performance Evaluation report 9 completion and comprehensibility. If any areas lacked clarity, the evaluation team referred back to the original recording for resolution. These transcriptions, along with notes taken at the time of the interview, were collated and analyzed for content by the in-country team leads. The analysis followed the general thematic organization of the interview guides, and answers to specific inquiries of interest were extracted from each transcribed interview and organized by area of interest/theme. Within each theme, the content was once again analyzed for general trends related to the outcomes of interest. These findings were triangulated across different key informants, and analyzed in light of the quantitative data provided. All analyses were led by in-country senior evaluators, and verified/confirmed by the team lead. Each in￾country senior evaluator used the software of their choice for analysis (ranging from Word to Atlas.ti). Each interview guide was then analyzed a second time by the Team Lead to ensure the validity of the original analyses. The Team Lead used the QDAP open-source qualitative data analysis software from the University of Massachusetts to analyze interviews across all countries for trends. Only findings that were triangulated through more than one source were used as evidence for each thematic area. For any areas of discrepancy in understanding, the team leader worked with the in-country senior evaluators to clarify content through revision of the original transcripts/recordings, as needed. Data Issues and Limitations It is important to note up-front the limitations in the data collected for this evaluation. First, the AgirPF site level data was taken directly from AgirPF’s DHIS2 database and were not independently verified to original data sources. However, as these are the official government statistics for family planning, and do go through a process of entry and validation, we have reason to believe they are accurate. For comparison site data collected directly from facilities, in-country team leaders reported that some of the comparison sites had poor quality record keeping and missing data. In cases where data were missing for site monthly facility reports at comparison sites, in-country evaluation teams extracted the data directly from FP registers. Overall, the amount of missing data should not have an overall impact on the findings of this evaluation. While use of a simple random sample of control and intervention sites at the regional level in each country enabled project-level statistical inference, the samples are not sufficient to draw conclusions at the country level. Therefore, the team have presented all statistical trend analyses at the country level only. Qualitative data, while providing depth and context for the findings, are not generalizable across countries, and may only apply in the specific contexts in which stakeholders reside. Furthermore, some stakeholders were not available for interview within the data collection period, despite the team’s strident efforts. Finally, there is possibility of contamination of family planning activities in control sites. Below is an accounting of partners working on family planning and SRH in control areas in each country. This does not necessarily indicate presence of programming in specific control sites, but does give an indication of program coverage in these areas. Table 4: Partners working in Family Planning and SRH areas in non-intervention zones by country E4D: Draft 2 AgirPF Performance Evaluation report 10 Country Partners working in Family Planning and SRH areas in non￾intervention zones Burkina Faso  UNFPA  JHPIEGO  Marie Stop International  Pathfinder International  Population Council Cote d’Ivoire  UNFPA  AIBEF/IPPF Member Association  AIMAS  Terre des Hommes (Health Systems Strengthening and FP) Niger  UNFPA  MSI  ANIMAS SUTURA (au niveau communautaire) Togo  UNFPA  NGO Handicap International/AFD  Hope Through Health (HTH) Summary profile of evaluation team The evaluation team for the AgirPF team was composed of an overall Team Leader and four in-country coordinators, one each representing Burkina Faso, Cote d’Ivoire, Niger and Togo where AgirPF activities are currently being implemented. The in-country coordinators were assisted in-country by a research team composed of data collectors and a supervisor who were all recruited from E4D’s Recipient Groups in the respective countries (except the Team leader in Cote d’Ivoire). In total, 25 data collectors and 3 supervisors were recruited and trained in Burkina Faso, Togo, Niger and Cote D’Ivoire to collect data, clean and conduct data entry. In addition, in-country coordinators from Burkina Faso and Togo were recruited from E4Ds RGs in-country. All in-country coordinators took part in the AgirPF methodology workshop which oriented them on data collection methods, tools and approaches for the evaluation. The involvement of RGs in the AgirPF performance evaluation is part of the “Learning by doing” strategy of the Capacity Building Component of the E4D activity which aim to build the research and evaluation capacity of RGs. Below are the details of the Evaluation team: A. Team Lead Ghazaleh Samandari is the overall evaluation team leader. She was responsible for overall management of the in-country evaluation teams. She prepared the desk review report and finalized inception reports as well as final evaluation report. She also led methodology-training workshop for the training of in-country coordinators and presentation of evaluation findings to USAID/WA. Ghazaleh Samandari holds a PhD in Maternal and Child Health and is an independent contractor. She has over a decade experience in international research, advanced statistics and program/client management. B. Togo Paul Tekou led the in-country evaluation team in Togo. Paul Tekou is the Director of E4Ds RG in Lomé, Togo Cabinet d’Expertise et de Recherche-Action (CERA) since 2005. For the AgirPF evaluation, two of CERA’s members were recruited to be part of a field team of seven responsible for data collection, cleaning and data entry. E4D: Draft 2 AgirPF Performance Evaluation report 11 Paul Tekou is a Doctorate candidate in Sociology and a holder of MBA. He has more than 14 years’ experience in research, monitoring and evaluation in various areas such as health, education and children/women’s rights. He is a member of scientific and evaluation organizations such as AFREA (African Association of Evaluation), IDEAS (International Development Evaluation Association), IUSSP (International Union for the Scientific Study of Population) and UAPS (Union of African Population Studies). C. Burkina Faso Joseph Catraye led the in-country evaluation team in Burkina Faso. He was supported by one supervisor and 6 data collectors all recruited from E4Ds RG, BASP ’96 for evaluation activities in Burkina Faso. Joseph Catraye is the director for BASP ’96 and is responsible for the development and management of Public Health projects and studies in Africa. He has over a decade experience in consulting for international agencies like the World Bank, FHI 360, USAID, and UNICEF on health projects in countries across Africa etc. D. Cote D’ivoire Emmanuel Esso led the evaluation in Cote d’Ivoire. He was supported in the field by a team of nine people all from E4Ds RG; ASAPSU. Emmanuel Esso is a demographic statistician with over 10 years of experience in public health with expertise in HIV/AIDS, reproductive health and population and development in West Africa. Emmanuel Esso’s technical knowledge includes qualitative and quantitative research, capacity building, monitoring and evaluation, and the development of data collection tools and frameworks. Emmanuel Esso holds his PhD in Demography. E. Niger Jacques Emina led the in-country team in Niger supported by one supervisor and 9 data collectors from RGs in Niger namely CERMES and ONDPH for all evaluation field activities. Apart from coordinating evaluation activities in Niger, Jacque Emina was responsible for quantitative data analysis of all data collected from Burkina Faso, Togo and Cote d’Ivoire. He was a co-trainer for the AgirPF Methodology Workshop in Lomé, Togo. Jacque Emina is currently the Senior Monitoring and Evaluation Advisor for E4D project. E4D: Draft 2 AgirPF Performance Evaluation report 12 PRIMARY QUESTIONS Primary Q1: How has AgirPF performed? Sub Q 1.1: How has AgirPF performed against reaching top line indicators of 700,000 method adopters, yielding 1,683,000 Couple years of protection (CYP)? Is the Project on track to reach its targets? In order to explore this question, the evaluation team analyzed and compared data from a number of sources including 1) data collected directly from a sample of intervention and control sites in each country, 2) AgirPF DHIS2 database 3) project reports. As there was no FP service delivery data in the EngenderHealth baseline report, the evaluation team used various sources of data to depict the change in FP service delivery over the project period. As presented in Figure 2 below, the project fell substantially short of its CYP target for PY2, achieving only 30% of its intended goal of 641,729 couple years of protection. Performance in PY3 was improved, and by the end of the third quarter, they had achieved 58% of their intended target for that year. The cumulative achievement of CYP by the time of the performance evaluation (394,584) is 23% of the final CYP target (1,683,000). In order to reach the project end target of 1,683,000 CYP, the project would have to achieve an additional 1,288,416 CYP in the remaining project time. That would mean more than tripling the cumulative CYP achievements made in the first two project years between the last quarter of PY3 and the end of the project. As it stands, without a considerable amount of additional resources and effort, the project will not be able to meet its stated CYP target. Figure 2: Overall AgirPF CYP, results vs targets (Source: AgirPF record data) * based on 3 quarters of reporting Figure 3: Overall AgirPF FP Adopters, results vs targets (Source: AgirPF record data) E4D: Draft 2 AgirPF Performance Evaluation report 13 * based on 3 quarters of reporting In PY2, the project reached 44% of its target for new method users, while in PY3, performance against targets for new method users in PY3 was improved, reaching 63% of targets by the third quarter. Despite improvement in new FP adopters in the third year of the project, the combined performance from the start of the project until the third quarter of PY3 (204,657 adopters) still leaves a gap of 495,343 adopters needing to be reached in order to achieve the project goal of 700,000 FP method adopters. By the third quarter of its third year, the project has only achieved 29% (204,657 adopters) of their targeted goal. As such, the project is not on track to reach its targets for FP service provision. Using data collected from a sample of control and intervention sites, the team compared trends in family planning uptake by new users, returning users and specifically for LARC users in both AgirPF and control sites. Figure 4 shows that AgirPF sites recorded increasing trends in new FP users over time, compared to comparable control sites (p<0.01). However, there were no significant differences in trends of returning users for intervention versus control sites (Figure 5). LARC use trends did increase significantly at AgirPF sites (Figure 6) compared to control sites over PY2 and PY3 (p<0.01). Figure 4: Trend analysis of new users; AgirPF vs control sites E4D: Draft 2 AgirPF Performance Evaluation report 14 Figure 5: Trend analysis of returning FP users, AgirPF vs Control sites (facility records data) Figure 6: Trend analysis of new LARC users, AgirPF vs Control sites (facility records data) E4D: Draft 2 AgirPF Performance Evaluation report 15 Next, the team examined the trends in AgirPF FP service delivery performance using data from the project’s DHIS2 database. This database houses FP service delivery data from October 2014 through December 2016 for all AgirPF sites in all countries, no data points were excluded in this portion of the analysis. Figure 7 shows that AgirPF sites experienced a 67% increase in FP uptake among returning users, over the period under evaluation. Trends among new users increased by 29% in this same period. Figures 8 and 9 break down the trends in FP uptake among returning and new users by each country. These analyses showed increases in new users in Cote d’Ivoire (210%) and Niger (19%), and gains in returning users in Burkina Faso (15%), Cote d’Ivoire (233%), Niger (36%). Figure 7: Trends in FP uptake by AgirPF sites by type of user (DHIS2 data) Figure 8: Trends in returning users at AgirPF sites, by country (DHIS2 data) E4D: Draft 2 AgirPF Performance Evaluation report 16 Figure 9: Trends in new users at AgirPF sites, by country (DHIS2 data) Sub Q 1.2 How has AgirPF performed against targets for number of Human Immunodeficiency Virus (HIV) positive women who received comprehensive FP services We reviewed all project indicator data but this specific disaggregation was not available in the reporting data. Furthermore, due to the manner in which facilities record and aggregate FP monthly and register data (which does not connect FP service provision to HIV status), it was not possible for us to collect this information through our own data collection procedures. However, interviews with MOH and partner organizations did indicate that AgirPF did policy and service delivery work related to integrating FP into HIV services. In Togo, AgirPF staff and WAAF grantees worked with facilities and districts to integrate FP and HIV services at the facility level. In Cote d’Ivoire, local training partners noted the emphasis that AgirPF put on updating FP service delivery guidelines to include issues related to serving HIV-positive populations: "Improving the quality of FP services depends among other things on the training of health personnel with documents that take into account not only the needs of young people, but also of people living with HIV and integrating the concept of gender and the new WHO guidelines for FP. To do so…the national training materials in contraceptive technology were revised together with the facilitating supervision training documents and the supervision guide.” Sub Q 1.3 How has AgirPF performed against targets for number of Community Health Workers supported and supervised [Indicator 16 from PMP] The indicator on CHWs had two iterations during the life of the project. The original, “Number of CHWs supported and supervised” during PY1 and PY2 and “Number of additional USG-assisted community health workers (CHWs) providing family planning information and/or services during the year” E4D: Draft 2 AgirPF Performance Evaluation report 17 Under the first iteration of the indicator, which spanned PY1 and PY2, there were no results reported, according to the AgirPF project data files. However, for the second indicator, which was implemented in PY3, the project exceeded its training targets for the year within the first three quarters for all countries and the total. The exception was Burkina Faso, where 27 CHWs were targeted for training, but no training took place. Table 5: Number of additional USG-assisted community health workers providing family planning information and/or services during the year, by country Number of additional USG-assisted community health workers (CHWs) providing family planning information and/or services during the year (Target: 310) PY3 Q1 PY3 Q2 PY3 Q3 PY3 Q4 Total Burkina11 N/A N/A N/A N/A Cote d’Ivoire 30 110 175 315 Niger 211 40 40 291 Togo 0 0 375 375 TOTAL 241 150 590 981 Sub Q 1.4 How has AgirPF performed against targets for number of youth who participated in educational program on gender, Family Planning (FP) and Sexual and Reproductive Health activities (SRH) [Indicator 17 from PMP] The anticipated cumulative target for number of youth who participate in educational programs on gender, FP and SRH activities for PY2 and PY3 was 240,000 individuals. By the third quarter of PY3, the cumulative achievement by AgirPF on this indicator was 208,608, 13% below the combined target for the two years. Further, in PY3 alone, by the 3rd quarter, the project had only achieved 50% of their expected outcomes for youth programming in PY3. When examining results at the country level, Burkina Faso and Niger exceeded their cumulative targeted number of participants, while Cote d’Ivoire and Togo fell substantially short in both years. Cote d’Ivoire joined AgirPF in PY2. This delay in joining the project explains the reason way its Cote d’Ivoire seem to shortfall regarding its targets as compared to the other countries in this evaluation. Table 6: Number of youth who participated in programming, by country and PY Number of youth who participated in educational program on gender, Family Planning (FP) and Sexual and Reproductive Health activities (SRH) PY2 PY3 Cumulative (PY2 + PY3) Target Achievement Target Achievement Target Achievement Burkina Faso 15,000 50,465 45,000 47,892 60,000 98,357 Cote d’Ivoire 15,000 0 45,000 19,296 60,000 19,296 Niger 15,000 31,066 45,000 29,360 60,000 60,426 Togo 15,000 11,767 45,000 18,762 60,000 30,529 TOTAL 60,000 93,298 180,000 115,310 240,000 208,608 * based on 3 quarters of reporting 11 In Burkina Faso, Community based distribution for contraceptives distribution is not legal to date. However, AgirPF used CHWs for FP awareness (sensitization and Information provision) only throughout the project life in Ouagadougou, Bobo-Dioulasso and Koudougou. In addition, pilot projects are implementing the approach and will deliver results at the end of year 2017. E4D: Draft 2 AgirPF Performance Evaluation report 18 Sub Q 1.5 How has AgirPF performed against targets for number of Best Practices (BPs)/ High Impact Practices (HIPs) for family planning and maternal and child health and/or HIV/AIDS incorporated into local, district or national health protocols or standards [Indicator 19 from PMP] According to program records, the BP/HIPS integrated into protocols and standards in both years were on family planning, but no additional details on the precise practices were available in project records. As formal adoption of BP/HIPs into local protocols and standards can be a time-consuming process, the evaluators followed up with AgirPF staff to get a broader idea of all of the BPs/HIPs utilized to enhance FP practices in each country. Table 7 presents a matrix of each type of BP/HIP being applied by AgirPF in each of the evaluation focus countries. Additional information on each of these HIPs is provided in section 2, below. Table 7: BPs/HIPs used to enhance family planning provision in each country COUNTRY BEST PRACTICE/HIGH IMPACT PRACTICE CHW PACFP Mobile Outreach PPFP FP - immunization integration Gender Integration Quality Improvement FP Advocacy Policy Youth Friendly Burkina Faso N/A X X X X X X X X Cote d’Ivoire X X X X X X X X X Niger X X X X X X X X X Togo X X X X X X X X X Sub Q 1.6 How has AgirPF performed against targets for number of best practices piloted through operations research studies [Indicator 20 from PMP]. The only country to pilot any best practices specifically through operations research studies was Togo, with six studies conducted in PY3 (including on Post Abortion Care Family Planning (PACFP), Postpartum Family Planning(PPFP), FP integration into immunization, youth-friendly services, mobile outreach and quality assurance). Based on the available project document files shared with the evaluation team (including quarterly reports and indicator database files), no specific operations research activities in other AgirPF country took place through the third quarter of PY3. Q2: Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services] As detailed in Table 7 above, the AgirPF project has advanced, to varying degrees, seven high-impact practices across the intervention countries. The HIPs fall into three main categories of intervention, as classified by the AgirPF team themselves: E4D: Draft 2 AgirPF Performance Evaluation report 19 Table 8: Categories of high impact practices HIGH IMPACT PRACTICES Service Delivery Enabling Environment Enhancement f HIPs HIP1: Use of community health workers HIP2: Integration of FP into Mother Child and Neonatal Health (Provision of post-abortion care family planning, Postpartum family planning, with emphasis on LARCs, FP integration in immunization HIP3: Mobile outreach (health fairs, FP days, etc.) HIP4: Integration of human rights/gender in provider technical and counseling curricula HIP5: Quality improvement of health services HIP6: Galvanize commitment to family planning through advocacy and policy development to: - Support financing for family planning commodities and services at the national and local levels; - Develop an effective supply chain management system for family planning - Implement a systematic, evidence-based health communication strategy - Increase access to care for vulnerable populations through improved policies (for example with youth) HIP7: Build the capacity of service providers to offer adolescent and youth friendly services AgirPF supported a number of HIPs across all countries, as evidenced by their programmatic reporting and verified by feedback from stakeholders. The following is a list of HIPs promoted by AgirPF with corresponding examples of specific work in that area: ● Community health workers – Due to the work of AgirPF, an additional 981 community health workers were supported to provide family planning information or services in Cote d’Ivoire, Niger and Togo. ● Integration of FP into Post Abortion Care (PACPF) and postpartum (PPFP)– Through their provider training and facility-level support AgirPF, trained a number of providers in integration of family planning provision into post-abortion care and in the postpartum period. One-hundred and fifty￾one providers were trained in postpartum provision of family planning, 180 were trained in postpartum Intrauterine Device (IUD) provision and 103 were trained in FP integration in post￾abortion care (PACPF) across all countries through the third quarter of PY3. An additional 50 providers were trained in how to adapt PACFP for youth. ● Supply chain management/logistics training – Through a variety of training modalities, a number of providers at both the facility and district level were trained in techniques for improving supply chain and logistics management. Across all countries, 537 providers were trained in contraceptive logistics and logistics management information systems. An additional 97 providers were trained in COPE for contraceptive security, a technique designed to assist providers in planning for contraceptive supply at the facility level. ● Policy/advocacy support - AgirPF, through their regional and in-country advocacy work has made E4D: Draft 2 AgirPF Performance Evaluation report 20 notable progress in improving policy support for FP and increasing financing for FP projects. For example, AgirPF worked with RCPFAS-Cote d’Ivoire to bring together members of the government and local and international partners (USAID, EngenderHealth, UNFPA) to advocate for new policies that commit additional funding for FP services and also ease restrictions on access to FP for youth. Médecins du Monde, Health Policy +, FHI 360, Association Ivoirienne pour le Bien Être Familial (AIBEF), Agence Ivoirienne de Marketing Social (AIMAS) took part also in this advocacy work. Additional details of this work are covered in section 4, below. ● FPSDs - With FP special days, AgirPF provided FP services to important number of Additional users yearning appreciable number of CYP. By the end of the third quarter in PY3, AgriPF had supported 432 special FP days. ● Community engagement - AgirPF used the site walk-through method to engage communities with their local health facility FP service providers. They also worked with religious and community leaders to advocate for improved FP access. Table 9: Number of people trained in different training modalities by project year Type of Training # of people trained in PY2 # of people trained in PY3 Total Contraceptive Technology 390 126 516 Infection Prevention 255 105 360 Counseling REDI 271 86 357 PPFP including PPIUD 19 312 331 Contraceptive Logistics/LMIS 277 260 537 COPE 51 24 75 3Is 204 0 204 Advocacy 322 36 358 Maternal and Child Health(MCH) 27 0 27 Reality Check 8 54 62 COPE for Contraceptive Security 12 85 97 Facilitative supervision 42 28 70 TOT in FP on EH approaches, tools, resources & policies. 17 50 67 OCAT 60 0 60 Post Partum FP 0 495 495 Gender 73 324 397 AgirPF Health Information System 137 64 201 Train health service providers on group and individual FP counseling as part of other services 0 50 50 Post Abortion Care (PAC) and FP adapted to the youth and adolescents’ needs 0 50 50 Post Abortion Care (PAC) and FP 0 247 247 Training on Youth friendly service provision 0 290 290 ToT Minilap for Tubal Ligation in Togo 0 8 8 Total number of providers trained per Year 2165 2694 4859 E4D: Draft 2 AgirPF Performance Evaluation report 21 Sub Q2.1 What policies, norms, guidelines, protocols, etc. related to the selected HIPs have been advanced? The AgirPF project worked with each implementation country to develop work plans that incorporated HIPs into their proposed activities. This included working with the senior policy advisor at EngenderHealth to advance HIP policies and guidelines at the national level. The adoption of HIPs into official norms and guidelines is an ongoing process among AgirPF countries, but some current advances related to HIPs include:  In Burkina Faso, the AgirPF team integrated the EngenderHealth technical approach of providing LARCs into MOH guidelines as well as in the national FP training curriculum.  In Togo, nine guidelines were developed to improve access to and use of family planning and reproductive health services, emphasizing HIPs such as mobile outreach, quality improvement techniques and integration of FP into other RH services.  In Niger, with AgirPF’s assistance, the MOH introduced of a draft policy to allow CHWs to provide long-acting and reversible contraceptive methods.  In Cote d’Ivoire, the advocacy work supported by AgirPF has led to a plan for formal revision of the national SRH law that would increase funding for FP and lower barriers to access through task-shifting of FP service provision to CHWs and integration of FP into other RH services.  The advocacy work for task-shifting in Togo that was begun under AWARE II was continued under AgirPF, resulting has led to adoption of a new Community Based Initiatives policy that allows CHWs to offer family planning methods, including injectables. Sub Q 2.2 To what extent have these HIPs been scaled up in AgirPF focus countries? The AgirPF project is in various stages of piloting and scaling-up of a number of HIPs. The matrix below provides examples of these activities for each HIP that AgirPF is promoting. Table 10: Scale-up of HIPs in AgirPF focus countries HIGH-IMPACT PRACTICE Scale-up Use of community health workers to provide family planning services Piloted at Haho and Blitta Districts in Togo currently and the plan is to scale-up nationwide in 2018. PAC/PF There is a regional USAID funded project which is working in collaboration with AgirPF to revitalize and prompt countries in scaling up PACPF. The vertical (institutionalization) scale up is already done in Burkina Faso, Cote d’Ivoire, Niger and Togo FPSDs and Mobile services AgirPF has equipped countries with vehicles to provide these mobile services to hard-to-reach populations with FP services. In terms of scaling-up, partners such as IPPF-Affiliates in all AgirPF countries will continue using these vehicles after the end of the project for the activities not only in the intervention E4D: Draft 2 AgirPF Performance Evaluation report 22 HIGH-IMPACT PRACTICE Scale-up sites of AgirPF but also to their own affiliate sites beyond AgirPF. PPFP PPFP is already scaled up and institutionalized in all AgirPF implementation countries . FP Integration into MCH services including immunization units This is not yet in the stage of scaling-up, as AgirPF is still in the process of gathering evidence to demonstrate the effectiveness of this approach. The project team anticipates that the presentation of promising results will prompt adoption and scale-up of this activity across intervention countries. Build the capacity of service providers to offer adolescent and youth (AY) friendly services This HIP has yet to be scaled-up, but is an ongoing activity in several countries, most notably Cote d’Ivoire : Integration of human rights/gender in provider technical and counseling curricula. This activity has already a nationwide scope; as such, scale-up is complete. Quality Improvement (SWT, COPE for Family Planning, COPE for Commodity Security, and Facilitative Supervision for improvement of Clinical Services) WAHO selected the site walk-through (SWT) approach to be part of its list of best practices in health to promote in West Africa. AgirPF prepared an article for the WAHO BP Handbook to publish this year with the expectation that WAHO will support replication of the SWT approach in ECOWAS countries. Facilitative supervision has already achieved national scale in all evaluation countries. Institutionalizing Youth friendly services provision at facility level Youth and adolescents are a critical target population for the AgirPF project. Therefore, to improve youth Health service seeking and uptake, AgirPF trained Health providers in Youth friendly services provision in all four countries, organized workshop with MOHs to review and/or initiate the development of policies related to youth health improvement. This took place in Niger, Togo, Cote d’Ivoire and Burkina Faso with USAID funded E2A Project. In addition, to improve youth and adolescent knowledge on SRH information and services, AgirPF developed a comic book named « Assibi et Salifou la première fois ». Galvanizing commitment to family planning through advocacy and policy development This activity is already ongoing at a national and regional scale in all intervention countries. Sub Q 2.3: To what extent have these HIPs contributed to AgirPF’s results? The activities of AgirPF were expressly formulated around the promulgation of HIPs. In other words, HIPs touch almost every aspect of the work that AgirPF is doing in all intervention countries. As such, the advances made in AgirPF countries in terms of policy changes, service delivery and enabling environment stem from HIP activities such as policy advocacy (HIP 10), FP integration into existing health services such as post-abortion care, postpartum services and immunization (HIP 2, 4 and 5), and FPSD (HIP 3). As E4D: Draft 2 AgirPF Performance Evaluation report 23 detailed further in the sections below, many stakeholders credit HIPs pertaining to mobile outreach and quality improvement with an immediate impact on family service provision (shown in section 1, above). Moreover, the continued policy advocacy work aimed at increasing financing for FP and removing barriers to access is seen as critical to the long-term advancement of family planning provision in these contexts. Sub Q 2.4 What has AgirPF done to facilitate replication of these HIPs within the country and in other countries? As detailed in the matrix under Sub Q 2.2, AgirPF has played a technical, financial and coordinating role in facilitating the replication of a number of HIPs within and beyond target countries. By developing a multi￾level approach to program implementation that included development of regional initiatives translated into local approaches, AgirPF supported the advancement of HIPs in focal countries and created the potential for adaptation and scale-up across the region. AgirPF devised regional plans of action through strategic partnerships with WAHO, HPP, JSI/DELIVER and Camber Collective that were then translated into national implementation with the assistance of AgirPF regional and local staff. Examples include: ● Regional Advocacy Training - In collaboration with HPP, JSI/DELIVER and WAHO, AgirPF developed a coordinated training curriculum and workshops through with they supported stakeholders of RCPFAS in Burkina Faso, Niger and Togo to develop advocacy strategies to increase funding for FP at the national level.; AgirPF, also, developed advocacy strategies to advocate for task shifting of FP services to lower level providers; create a strong commitment of high level decision makers to supporting FP; and integration of FP in other RH services. These efforts were further supported at the country level through harmonized efforts of AgirPF regional staff. ● Regional Database (DHIS2) – AgirPF developed a regional family planning database through DHIS2 to track FP method use at the country and regional levels. By training staff at the country level to collect and enter data into a coordinated regional database, the project was able to track FP service delivery improvements at both national and regional levels. ● Regional SBCC strategy – Through their partnership with the Camber Collective, AgirPF developed a regional SBCC strategy to address the factors influencing FP use. The Camber Collective is a strategy-consulting firm that takes a human-centered approach to tackling entrenched public health issues. Their funders include a range of multilateral and bilateral development agencies and foundations including the WHO, USAID and The Bill and Melinda Gates Foundation. The effort began with formative research from Niger to identify the factors that encourage or discourage FP use at the individual/community level. This was validated after in the other countries (Burkina Faso, Cote d’Ivoire and Togo). Key messages stemming from this formative research were then tested among key target groups in each country. These findings were then used as the basis for the development of a regional strategy that could then be adopted for use at the country level. ● Centers of Excellence - AgirPF works with Centers of Excellence in each country to disseminate high-impact practices of FP service provision. Therefore, AgirPF regional staff built the capacity of CoEs (i) in Burkina Faso (at Bogodogo Health District in Ouagadougou), (ii) Togo (at ATBEF’s main clinic located in Lomé), (iii) Cote d’Ivoire (at Binger-Ville General Hospital in Abidjan) and (iv) Niger (at Centre National de Santé de la Reproduction in Niamey) in key E4D: Draft 2 AgirPF Performance Evaluation report 24 service delivery areas including facilitative supervision, FP service provision and COPE® (Client-Oriented Provider-Efficient. Through systematic work with national CoEs, best practices can be adopted at the national level in each country and ultimately lead to a regional impact. Sub Q 2.5 How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? The AgirPF approach to developing and implementing HIPs was based on a regional strategy of evidence￾based decision-making and of building consensus and capacity among regional partners in order to deploy and adapt best practices at the country level. The primary example of AgirPF’s regional role in exchange, learning and dissemination of HIPs centers around the WAHO 1st Conference on Good practices in Health (held in Ouagadougou in July 2015). This conference, borne directly out of the USAID West Africa Regional Development Cooperation Strategy, and to advance the Ouagadougou Partnership agenda, provided an opportunity to assess the effectiveness of HIPs at a regional level in order to facilitate the deployment of best practices across the region. AgirPF started by conducting a scan of best practices appropriate for adaptation in Burkina Faso, Niger and Togo and Cote d’Ivoire. The project targeted four HIPs selected from proven good practice (post￾partum family planning, PAC-FP) to promising practices (mobile Outreach services) and emerging practices (FP integration to immunization points, site-walk-throughs). In collaboration with WAHO, AgirPF provided technical and financial support for the organization of an SRH good practices selection and documentation workshop in Burkina Faso, Cote d’Ivoire, Niger and Togo. The 5-day workshops were aimed at supporting country MOH teams and their SRH partners and local organizations to analyze best practices for FP service delivery in their countries, selecting those that are considered proven, promising or emerging and documenting them so that abstracts could be submitted to WAHO for presentation at the 1st Forum on Good Practices in SRH in the West Africa region. In each of the four countries workshops were facilitated by WAHO and the local Division of Family Health of the MOH, with AgirPF support. Over 30 participants attended in each country, and abstracts were developed for submission to the Forum. This documentation of best practices, and the ultimate regional exchange through the forum, led to the promotion of four key HIPs for adoption across countries: FP special days (mobile outreach), the post￾partum family planning provision, the FP provision in post-abortion care, and the use of the RAPID model to address policy advocacy and socio-cultural advocacy with religious leaders. AgirPF played an instrumental role in the success of this regional exchange by providing targeted country-level support for development of the evidence-base as well as funding attendance of participants from target countries to attend the forum. 2.6 SECTION ON AGIRPF SUPPORT FOR HIPs 2.6.1 How has AgirPF supported exchange, learning and dissemination of HIPs among health care providers? Training and capacity building were the primary channels for disseminating high-impact practices among health care providers, and AgirPF carried out various trainings of providers across all countries. The training primarily provided health care personnel with knowledge of the different contraceptive methods, strategies to address clients and outreach techniques. E4D: Draft 2 AgirPF Performance Evaluation report 25 To ensure the quality of training, AgirPF created a national pool of FP trainers at the start of the project. These trainers, in turn, were able to disseminate knowledge to lower levels by holding training workshops for providers in each country. Training was provided not only to front-line providers, but also to district health managers, with an emphasis on managing FP budgets, logistics and developing local capacity. AgirPF used a variety of tools and techniques to improve FP performance through training, including the REDI counseling method (which promotes client-centered decision-making) and the COPE for contraceptive security improvement approach (which educated providers on how to improve logistics management systems). According to the interviewees, the trainings were very helpful: "The training allowed us to know the IUD methods, providing implants to the woman, doing quality counseling and improving the behavior change communication service.” – Provider, Cote d’Ivoire "The training allowed me to have a good knowledge of FP and a mastery of techniques and knowledge of FP methods. With the training, I can explain to women the advantages and disadvantages of these methods, I have a good knowledge of methods, I manage to take a woman in charge and Explain which FP methods to choose " - Provider, Cote d’Ivoire Training in the logistics management of contraceptives enabled the beneficiaries to know how to manage stocks through the use of different management tools. As one provider in Cote d’Ivoire reported: "It allowed me to make inventories, to know stocks. It makes it possible to fill in the inventory cards which was not done before. Training sessions have enabled us to learn about the various management tools, inventories, storage techniques and ordering. The training allowed us to have a knowledge in the evaluation of inventories and inventories." In addition to training, AgirPF has developed on-site coaching during supervisions visits to continue supporting providers in application of high-impact practices. 2.6.2 How has AgirPF supported exchange, learning and dissemination of HIPs among health care managers? AgirPF training involved not only FP providers but also FP focal points in health districts, including health managers at the district and regional levels. AgirPF provided targeted training and support to these stakeholders to improve their ability to coordinate and manage specific aspects of family planning service delivery such as supply chain management. Tools such as Reality Check, which allow district managers to forecast FP supply needs on the basis of service, were combined with collaborative partnerships with other partners like DELIVER (John Snow Inc.) in the districts/regions to coordinate improved service delivery. As one district manager in Cote d’Ivoire stated: "In the [logistics] system, there is no more problem. Before people did not know how to order the products, there were stock-outs. That corrected all that. A total of 191 logistics service providers were trained. AgirPF worked with Deliver to improve its logistics management system. Supplies and stocks are tracked. Henceforth, the stock-outs faced by certain health facilities have been reduced, enabling them to practice their FP activities.” AgirPF focused on strengthening the capacity of the district FP focal points by providing them direct training in a series of management tools and techniques including contraceptive technology, REDI E4D: Draft 2 AgirPF Performance Evaluation report 26 counseling, logistics management, facilitating supervision, infection prevention, 3I (Informer, Inspire and Involve) and data quality assessment at the district level. According to one district manager, the development, installation and training on the DHIS2 database system also greatly improved their ability to track performance and provide effective family planning services. "They were trained in contraceptive technology and REDI counseling and logistics management. All our sites have been trained. The capacity of Reproductive Health Officers and PF focal point of the ten districts was strengthened. They were all trained in contraceptive technology, REDI counseling, and in logistics management, facilitating supervision, and infection prevention." – District health manager, Cote d’Ivoire 2.6.3 How has AgirPF supported exchange, learning and dissemination of HIPs among partner organizations? The exchange, learning and dissemination of high impact practices at the level of consortium and advocacy partners can be summed up as support for advocacy for political, community and religious decision￾makers. As part of the project, partners were provided with tools to address national elected officials, Ministry of Health authorities, religious and community leaders and specific client groups (people living with HIV, sex workers) to be able to effectively engage these groups for improved FP advocacy and activities. “In terms of strategies AgirPF supports enormously in the reflection and the elaboration of the strategies and the formative documents.” – NGO leader, Togo "The different trainings (Clinical FP, Supervision, Counseling) have led to an improvement in the quality of services. There is also institutional strengthening in financial and accounting management ". – Consortium partner, Burkina Faso As part of their work, particularly with local partners, AgirPF also consistently applied the OCAT (organizational capacity assessment tool) to gauge their baseline readiness for FP service delivery and to identify area of need for capacity building. AgirPF administrated the OCATanalysis tool to all its partner institutions working in clinical area. This tool enabled the development of capacity-building plans for these institutions, thus making it possible to fill in their shortcomings in terms of the capacity to implement FP activities. Implementation partners interviewed confirmed this. “It's a very enriching technical support that has made it possible to identify problems and to look for solutions, to improve the performance of the interventions and the coordination of the implementers and the interventions." – local partner, Cote d’Ivoire Local partners report that because of their work with AgirPF, they have been better equipped to perform their work, particularly in the realm of advocacy: “Before the project the workforce was not there; in two years, we recruited less than 30, in four months we recruited 160, which means that we are relaunched on these issues. AgirPF's support has also increased the number of women screening for cervical cancer and so many women have been screened and are being treated. Were it not for sure, there were many women who were in need.” – Local advocacy partner, Togo 2.6.4 How has AgirPF supported exchange, learning and dissemination of HIPs among MOH stakeholders? E4D: Draft 2 AgirPF Performance Evaluation report 27 The actions carried out by the AgirPF project to support the performance of national Ministries of Health are multiple and generally aim at strengthening its managerial capacities, design and monitoring of interventions. AgirPF worked with MOH counterparts on a number of issues aimed at increasing the use and dissemination of HIPs and advocated for these changes at the level of national FP planning and implementation documents. By working side by side with MOH counterparts and being involved in national level technical working groups in each country, AgirPF aimed to have positive influence on national FP plans, and to coordinate their own activities to support and complement MOH plans. Where feasible, AgirPF has also included MOH partners in key regional network meetings with partners and has involved individual members of national MOH staff in specific training. By incorporating ministry staff into technical training and advocacy networking activities, the project has increased visibility of its work and improved the likelihood of sustained changes at the national level. “All that AgirPF does is to support the division; All the activities that AgirPF is leading is to support the government through the Ministry of Health and Social Welfare and through the technical division where we are. So all the activities carried out by AgirPF is to support us. Not only guided tours, training of providers, training of CHWs, monitoring of providers and CHWs, equipment provision, organization of Site Walk-Through visits and above all support to the health information system at district level.” – MOH counterpart, Cote d’Ivoire “AgirPF has trained the MOH staff. Some trainings are tailored to the health of young people and adolescents as trainers and as soon as AgirPF organizes this training, the division's staff participate. They trained me on the visibility strategy. They also trained on new approaches to formative supervision, facilitating supervision. So the staff has benefited and as soon as we organize these supervisions, we participate. They also adapted the data collection tools and we were trained on these tools. I personally I participated in the training on the site Walk-Through Approach.” – MOH staff, Togo Primary Q3: To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Investigating the extent to which the results are necessary and sufficient to achieve the objective of a project is tantamount to evaluating the relevance of the project. It is part of the project design evaluation. It is assessed using the planning method used (needs identification, vertical logic, logical framework matrix, competences of the implementing actors, etc.). In this section, we will focus only on the vertical logic of the project. Is the link between purpose, intermediate objectives, products and activities consistent? To assess this piece of the evaluation, the team used a number of approaches. First, the team examined the link between each IR and its sub-IRs in light of the related activities performed by AgirPF. In other words, did AgirPF support interventions that addressed each result and sub-results as a part of their programming? Next, the team solicited input from both AgirPF staff and local/regional partners on the effectiveness of each IR in relation to the overall goal of the project. The team wanted to understand, from the perspective of the stakeholders, what if anything was missing from the program logic that may impede full achievement of the overall objective. The team also did an independent assessment of the results framework in light of the ecological model of public health, which is an industry standard for holistic intervention (see Sub Q 3.4). Overall, participants felt that the framework took a holistic approach to addressing improvements in access to family planning. Furthermore, there was a notable relationship between the design of the program and the overall national strategic plans for FP, stemming from the Ouagadougou partnership. E4D: Draft 2 AgirPF Performance Evaluation report 28 “The relevance of the AgirPF project is fully proven because they intervene at different levels: they intervene at the community level, they also intervene at the level of the health units, they even intervene at the level of religious leaders and they intervene at the level of the local authorities. When you see the activities that are carried out in this AgirPF project, it is modeled on the plan of repositioning of the FP 2013-2017.” – MOH partner, Togo However, as the team will see below, the framework lacks some coherence in its logical structure and might benefit from revisions that focus efforts on clear steps towards achieving results. The framework also needs increased emphasis on demand generation at the community level and ways to address systemic resource challenges (such as human resources and facility infrastructure). Sub Q 3.1 Are the elements of IR1 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? The result R1 is: Delivery of quality family planning information, products, and services strengthened and expanded. Under this result, AgirPF conducted a number of activities through a combination of training and material support (Table 11). The first step in achieving this result was in engaging a variety of local actors (leaders, civil society, service providers, and municipal government officers) to promote family planning (sub result 1.2). This was done through both direct skills training activities as well as regional and local technical networking and workshops. AgirPF systematically trained master trainers (n=12) and supervisors (n=20) in each country, who then proceeded to train over 5,700 individual providers in enhanced family planning service provision. AgirPF also worked closely with municipal health managers to improve their ability to manage health systems delivery strategies such as FP data use for decision-making and supply chain management. The project also worked closely with local NGOs and civil society groups to improve their organizational management and service delivery capabilities to ensure long-term engagement and capacity in FP service delivery. By engaging this constellation of stakeholders through capacity-building activities, the project was able to strengthen these groups to implement effective approaches for delivering quality family planning services (sub result 1.1). In addition to this, where possible, AgirPF also provided material support such as basic FP equipment, training materials, and vehicles to support mobile services provision. This combination of efforts contributed to enhanced and strengthened delivery of FP information, products and services (Result 1), which in turn increased overall access to FP services. Table 11: Examples of Activities related to each Intermediate Result Intermediate Results Related Activities IR1: The supply of FP information, products and services of enhanced and extended quality. ● Conducted a baseline study in the five countries to inform additional programmatic needs, ● Trained a group of 12 FP trainers and 20 supervisors, in each country, ● Provided technical and financial support to 264 health facilities and 20 private clinics in four countries, ● Trained more than 5700 FP providers in various themes related to FP provision ● Purchased and delivered basic FP equipment at 264 response sites, as well as training materials and anatomical models, ● Trained more than 300 FP service providers in 60 sites on youth-friendly services, E4D: Draft 2 AgirPF Performance Evaluation report 29 Intermediate Results Related Activities ● Integrated FP in maternal and child health services and in HIV services ● 5 vehicles purchased and placed in Burkina Faso (1), Côte d'Ivoire (2), Niger (1) and Togo (1) to support mobile services; ● To date over 641 special FP days and about 100 mobile services organized; IR 2: Evidence-based service delivery approaches selected, adapted, and implemented ● Trained, equipped and supported community health workers to provide a wide range of FP methods ● Supporedthe organization of mobile outreach service delivery to provide a wide range of contraceptives, including long-acting reversible contraceptives ● Integrated sexual and reproductive rights and gender into technical and counselling curricula for providers ● Integrated FP services into postpartum services by building partner capacity to provide postpartum FP (PPFP) counselling and services ● Ensured service quality using EngenderHealth’s Site Walk-Through (SWT) approach and supporting Facilitative supervision visit in all 264 intervention sites and more than 5700 trained providers. ● Evaluated the trained providers competence in quality FP service provision and that 94% of the supervised providers are deemed competent to provide quality FP services. ● Developing and implementing AgirPF’s social and behavior change communication strategy in collaboration with Camber Collective ● Providing technical and financial support for the organization of an SRH good practices selection and documentation workshop aimed at supporting country MOHs team and their SRH partners and local organizations to analyze good practices ongoing in their countries, selecting those that are considered proven, promising or emerging and documenting them. ● Developed a Comic Book for Youth and Adolescent Sexual and reproductive Health and Rights education for Burkina Faso, Cote d’Ivoire, and Togo. NB: For sociocultural sensitivity reasons, Niger declined the Comic Book. IR 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated ● Engaging faith-based organizations to advocate for the promotion of the responsible childbearing in Burkina Faso and Togo ● Engaging stakeholders from advocacy the Network of Champions in Advocacy for Sustainable Health Funding (RCPFAS) in all countries and building their capacity to advocate for the removal of policy and socio-cultural barriers to FP in those countries through targeted workshops and meetings. ● Using the RAPID models specifically developed for the national authorities and for those based in the intervention cities of AgirPF countries. Activities included stakeholders from government offices, WAHO representatives, civil society organizations and donors, and prompted adoption of the model at the advocacy and grassroots level. ● Engaging youth organizations to advocate for youth-friendly sexual and reproductive health services, in specific countries. ● Training providers on COPE for Contraceptive Security, a client-oriented provider-efficient method of assessing quality of care with the specific intent of improving family planning commodities and supply chain systems. ● Training health managers on the Reality Check tool, a tool that provides estimates on commodity needs for desired contraceptive prevalence rates. Use of this system allows health management teams to estimate and budget E4D: Draft 2 AgirPF Performance Evaluation report 30 Intermediate Results Related Activities for the family planning method mix, as well as estimate necessary service expansion to meet targets. Sub Q 3.2 Are the elements of IR2 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? IR 2 of the AgirPF results framework is: Evidence-based service delivery approaches selected, adapted, and implemented. Under this IR, the AgirPF team used a series of evidence-based approaches to enhance access to and use of FP. As shown in Table 11, AgirPF engaged partners in a series of activities to promote HIPs including, using CHWs to deliver FP knowledge and services, supporting FP special days, engaging communities through site walk-through efforts, etc. In doing so, AgirPF used a variety of tools and approaches and worked closely with local counterparts to adapt and implement according to local needs. These efforts undoubtedly contributed to increased capacity in family planning provision. However, when examining the specific sub-results under IR2, what is missing is an effective process of vetting and prioritizing the most impactful approaches from among the variety of HIPs proposed. The current sub-results under IR 2, a) Efficiency and effectiveness of approaches enhanced through the adaptation and implementation process and b) Lessons documented and disseminated on learning from adaption and implementation processes do not explicitly relate to the selection and application of the most effective interventions. It may be true that each of the HIPs used in this project have been shown effective in one context or another, but the framework has no apparent process for determining which HIP was best applied in which country and for how long. The sub results here should be more prescriptive of the process by which appropriate evidence is gathered and analyzed for best practices. Furthermore, the relationship between the sub-results does not follow a logical path. For example, documentation and dissemination of learning (sub-result 2.2) is not a starting point for use of evidence-based approaches, but rather is the end-result of the learning gained by applying these approaches. Result 2 would benefit from including a preliminary step of assessing evidence-based approaches in light of each country context and with the help of local stakeholders to determine which approaches were most effective in each case. As one stakeholder in Togo said: “I think that ideally we should focus on what can really bring results, which has a direct link with results; There has been a lot of effort by the various teams in the implementation of the training and support activities, but I think if we could put the accent on the activities that are really good results. It could save us time and maybe show a lot more results.” Sub Q 3.3 Are the elements of IR3 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Intermediate Result 3 consists of: Efforts to remove policy barriers and improve contraceptive commodity security coordinated. To this end, AgirPF conducted a number of activities aimed at improving policies and enhancing commodities logistics systems. As table 11 shows, there were numerous networking, advocacy and partnership efforts made to improve outcomes at these two levels. For policy advocacy, AgirPF worked closely with civil society groups and national/regional advocacy groups and faith-based organizations to introduce efforts aimed at improving access to FP, particularly for vulnerable groups such as youth. At the same time, AgirPF engaged partners such as DELIVER and used tools like COPE for E4D: Draft 2 AgirPF Performance Evaluation report 31 contraceptive security, to improve logistics management systems at facility and district levels. In terms of its relationship to the overall project goal, it is clear that improvements in policy and logistics management would contribute to increased access to family planning services. However, it is unclear why these two components were combined as one intermediate result, as they are not logically linked in their origins or applications. Here, again, the logic of the model breaks down, when following the path from sub result 3.2 (commodities) to 3.1 (policy advocacy) to IR 3 (an arbitrary combination of commodities and policy). There is no explanation as to how these two items relate to each other, and further, there is no devolution of each item to explain how the result should be achieved (i.e., there is no explanation of the sub-activities under policy or commodities that would lead to the achievement of the overall objective). To their credit, AgirPF did implement a number of wide-reaching activities to address each of these two components of the IR3, but future programming would benefit from more explicit theoretical pathways for the best ways to promote policy and commodities management changes. Sub Q 3.4 How could the overall theory of change be improved to better address program design needs? Overall, participants felt that successful achievement of the three intermediate results could result in AgirPF reaching their overall objective. However, participants did note some outstanding areas of attention that could be enhanced in future program design. Specifically, stakeholders noted the need for increased material support for infrastructural changes at the facility level. They also noted the need for more mass communication to combat negative cultural beliefs around family planning and drive demand. Finally, one MOH stakeholder suggested a need to tilt the balance of activities towards efforts that yield immediate and effective results, such as FPSDs and Mobile services. “Activities related to advocacy are activities that are of utmost importance, yet it is difficult to see the effect at the end of a year, two years…I think we should take into account the really high-impact activities that make it possible to recruit the maximum number of clients… [such as] FPSDs, mobile services, SWTs in the communities are activities that allow to have many results.” – Local partner, Togo When examining the project theory in light of the public health ecological model, the theory of change explicitly addresses a few key areas of the model such as public policy, community factors and institutional factors. Yet, at the same time, the current AgirPF results framework does not directly reference the two lower levels of interpersonal and group processes that are essential to promoting behavior change and generating demand for family planning. Rather, these components are presumably implied within IR2, as part of evidence-based practices such as behavior change communication (BCC) or outreach. The objective of the AgirPF project is not only to increase access to family planning but also to increase use, which implies a focused effort at the level of potential individual users. Future results frameworks should be more explicit about efforts around each component of the model, they should include logical step-wise development from sub-results to results, and should endeavor to prioritize the most impactful processes (such as FPSDs and Mobile services) in order to create an effective balance of efforts aimed at improving family planning service delivery and uptake. Figure 10: The public health ecological model E4D: Draft 2 AgirPF Performance Evaluation report 32 Primary Q4: For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? As noted in section 2, above, AgirPF, through their coordinated work with the Network of Advocacy Champions for Adequate Funding of Health (RCPFAS) 12, contributed to the strengthening of efforts to remove policy barriers to FP funding and access across the region. They also worked with civil society and faith-based organizations in each country to advocate for improved access to family planning through removal of policy barriers. Notable achievements in this area include: ● Engaging faith-based organizations to advocate for the promotion of responsible childbearing in Burkina Faso, Togo and Cote d’Ivoire. These efforts have resulted in the acknowledgement of local religious leaders of the importance of responsible procreation in their communities. In certain cases, such as in Cote d’Ivoire, leaders have also pledged to advance the cause of responsible childbearing among their followers. ● Engaging stakeholders from advocacy the Network of Champions in Advocacy for Sustainable Health Funding (RCPFAS) in all countries and building their capacity to advocate for the removal of policy and socio-cultural barriers to FP in those countries through targeted workshops and meetings. RCPFAS is involved in resources mobilization for health financing. Its role is to get the government to keep their commitment to health financing and to mobilize resources from decentralized authorities, municipalities in the countries, donors and the private sector. In Cote d’Ivoire, the RCPFAS contributes to the review of the RH draft law on FP and any other advocacy activity on FP such as delegation of tasks. The work done through the network has elevated the family planning policy commitments in participant countries. ● Using the RAPID models specifically developed for the national authorities and for those based in the intervention cities of AgirPF countries. Activities included stakeholders from government offices, WAHO representatives, CSOs and donors, and prompted adoption of the model at the 12 A note on the origin of RCPFAS: The West African Health Organization, the ECOWAS health agency, held a meeting in February 2011 where parliamentarians and senior officials from the finance ministries of ECOWAS countries shared the findings of the study and identified suitable solutions for improving the allocation of Resources. One of the strong recommendations of this meeting was the creation of national champions’ networks in advocacy for adequate health financing (RCPFAS). Since then, WAHO created ten networks that are functioning more or less. All AgirPF implementation countries are a physical presence of RCPFAS. E4D: Draft 2 AgirPF Performance Evaluation report 33 advocacy and grassroots level. The use of the RAPID model has been credited with highlighting the negative impact of population growth on national development and prompting national policy makers to prioritize improved family planning policies. In the case of Burkina Faso, the RAPID model has been credited with advancing revision of article of family planning provision at the national level. ● Engaging youth organizations to advocate for youth-friendly sexual and reproductive health services. The involvement of youth-oriented civil societies has helped expand the advocacy networks dedicated to improving access to FP for vulnerable youth. In the case of Cote d’Ivoire, their participation aided in the advancement of new provisions aimed at extending access to contraceptives to women ages 16 and older. In Togo, these organizations are engaged in direct advocacy with the Togolese parliamentarians for youth-friendly sexual and reproductive health services. Below are specific examples of progress towards policy changes stemming from AgirPF’s advocacy work in each country. As previously stated, full policy change is a time-consuming process which is ongoing in each context. However, the examples provided here approximate advances being made in this area of AgirPF’s work. BURKINA FASO  AgirPF built the capacity of stakeholders from RCPFAS in Burkina Faso to create advocacy strategies for adoption of new policies to remove family planning policy barriers in Burkina Faso. This resulted in the validation of two application texts of Burkina Faso RH Law by the Minister of Health. The first text is related to clients’ FP rights and the second notifying the contraceptive range in Burkina Faso.  AgirPF in collaboration with the WAHO and HPP, supported religious leaders in Burkina Faso in the development of their advocacy tools and capacities for the promotion of healthy timing and spacing of births. Activities included identification of socio-political barriers to FP and development of advocacy plans by participants.  In collaboration with HPP, DELIVER and WAHO, AgirPF strengthened numerous stakeholders in Burkina Faso to advocate for the increase of country funding for FP, and task shifting of family planning provision to lower level provider cadres. They developed 4 advocacy strategies whose implementation will increase efforts for obtaining the increase of country funding for FP, and for the adoption of policy change towards the adoption of task shifting more widely in the region.  AgirPF in collaboration with the RCPFAS carried out an Advocacy activity towards the National Assembly and the Economic and Social Council of Burkina Faso to increase the domestic envelope allocated to contraceptive products from the sum of 175 million (USD 350,000) in 2015 to 500 million CFA francs (USD 1000,000) in 2017.  The Burkina Faso Evangelical Churches Federation’s President signed of a public declaration by in favor of FP  Achievement of 8 commitments signed by the BF's Traditional high authorities in favor of responsible procreation (namely The Dima of Boussouma, The King of Tenkodogo, The King of Ouahigouya, The Head of Traditional Authorities of Dedougou, The Emir of Gor-Gorom, The King of Gulmu and Bobo Dioulasso Canton Chief). E4D: Draft 2 AgirPF Performance Evaluation report 34 COTE D’IVOIRE  AgirPF built the capacity of several stakeholders from RCPFAS-CI to advocate for - The adoption of a national RH law, - The increase of country funding for FP at the national level, - The increase of funding for FP at the Abidjan city level, - Task-shifting to community health workers - Strong commitment of high level decision makers to FP, - The integration of FP in other RH services. The team further engaged stakeholders to launch an advocacy campaign for the development and adoption of a new SRH law. An MOH meeting attended by multiple advocacy groups including AgirPF, resulted in the formal decision by the MOH to revise and reintroduce a draft SRH law, an action plan outlining all the steps leading up to the introduction of a proposed law to the Ivorian Parliament for its adoption, and to the advertisement of the law once it is adopted.  The advocacy efforts resulted in increasing of the allocation up to 400 million CFA (USD 800,000) for the purchase of FP contraceptive commodities in the 2016 budget.  AgirPF collaborated with the Ministry of Health and Aids Control in Cote d’Ivoire to strengthen advocacy capacity of 25 high-level leaders of faith-based organizations (Catholic, Evangelical Protestant, and Muslim), as well as to engage these influential partners in the efforts to reposition FP, through the promotion of the concept of “Responsible Childbearing”. Participants developed advocacy strategies that could lead to the adoption of a national policy for the promotion of “responsible childbearing.” NIGER  In Niger, AgirPF provided technical support to RCPFAS and networks of religious leaders to finalize planning of advocacy activities targeting the removal of FP policy barriers and the promotion of healthy spacing and timing of responsible childbearing within faith-based organizations. TOGO  In Togo, AgirPF’s advocacy efforts led to the stakeholder validation of three new regulations (ministerial orders and decrees) to implement the RH law and the agreement of next steps to have the regulations signed by authorities, as well as the allocation of 125 million CFA for the purchase of contraceptives commodities in the 2016 budget.  AgirPF in collaboration with the WAHO and HPP, supported 34 religious leaders in Togo the development of their advocacy tools and capacities for the promotion of healthy timing and spacing of Responsible childbearing. Activities included identification of socio-political barriers to FP and development of advocacy plans by participants.  AgirPF collaborated with ATBEF’s Youth Action movement to organize in Lomé a workshop to mobilize actors and develop an advocacy document for the promotion of family planning among youth population. E4D: Draft 2 AgirPF Performance Evaluation report 35 Sub Q 4.1 To what extent did AgirPF support advocacy partners in removing policy barriers to FP? The AgirPF project provided critical support to advocacy partners through their work with the Network of Advocacy Champions for Adequate Funding of Health (RCPFAS) and through the deployment of the SPECTRUM and RAPID models in each country. The RAPID tools are advocacy tools to help decision makers analyze different scenarios and support policy dialogue on the effects of demographic factors on population health and socio-economic development. These tools were used with key stakeholders, advocacy networks and faith-based organizations to facilitate dialogue around the impact of population growth in each country and to underscore the need for action in family planning. The use of these tools, and the overall training support provided by AgirPF staff, empowered advocacy partners to engage government partners and religious leaders in change around family planning policy. While many decision makers initially believe that their countries need a large population to develop, the concrete facts in the developed advocacy tools help them to understand that FP can contribute to a demographic dividend and point the country in a more promising direction. The data presented demonstrate that a decrease in fertility is one of the key factors that can push a country toward emerging status. The facts clearly make the link between lower planned fertility and individual well-being and demonstrate how the demographic dividend improves the quality of life in the population, offering more opportunities for education, more investment in modern agriculture, and higher levels of savings and investments. In the area of FP socio-cultural environment, instead of basing its advocacy support to the religious leaders on the old concept of “birth spacing” which didn’t produce significant results for decades, AgirPF promoted the concept of “responsible childbearing”. Indeed, religious leaders were comfortable promoting “birth spacing” among married couples in the region, but this approach falls short of removing sociocultural barriers to FP, because it does not address the needs of those who wish to limit their births or of couples who do not agree on contraception, or the needs of all those at risk of an untimely pregnancy, regardless of age or marital status. The specific objective of advocacy to improve the social-cultural climate is that religious and traditional leaders promote “responsible childbearing” in their communities out of a conviction that this concept is compatible with their beliefs and that actions by policymakers to strengthen FP should be supported because of their importance to the country’s socioeconomic development. A decisive factor in choosing this objective is that religious and cultural leaders are concerned about the implications of contraception, child spacing, and family size limitation in the context of their faith and traditions. Furthermore, many are unaware of or uncertain about the relevance of FP programs to their country’s future. A visible and active movement to reposition FP will heighten their concerns unless they can identify with goals that they share. This concept well accepted by all parties permitted fruitful collaboration between advocates from RCPFAS and the religious networks within the countries. The following quotes are examples of removal of policy barriers and are representative of major themes found during the qualitative analysis. E4D: Draft 2 AgirPF Performance Evaluation report 36 “When the RAPID model was finalized, we used it to advocate with the high religious authorities to adopt responsible procreation. We met [with them] and they agreed to adopt a policy document in their community on the matter of responsible procreation. They believe that responsible procreation has its place…in the face of the disastrous consequences of non-responsible procreation on households.” Advocacy partner, Cote d’Ivoire. "There is a new change in the language of leaders. There has been a political commitment to FP. With the advocacy, there is a draft decree in progress for the application of the texts of FP. This draft order is inadequate to follow up, and also delays due to administrative movements ". – Advocacy partner, Burkina Faso. "AgirPF has played a major role. We started this advocacy and thanks to the support of the Network of Champions for Advocacy for Adequate Funding of Health, we started talking about [policy change] at the level of the national assembly. We have agreed that we must make contraceptives available to young people from the age of 16 ... it is already accepted, we must go and negotiate again. We made the plea with the National Assembly, AgirPF played a big role, we were appointed as spokesperson for the whole delegation. We used the RAPID model to go to the floor with concrete examples ". – Advocacy partner, Cote d’Ivoire Sub Q 4.2 To what extent did AgirPF support MOH partners in removing policy barriers to FP? The AgirPF project engaged MOH partners in various ways to promote the removal of policy barriers and promote elevation of family planning as a national priority. The AgirPF regional and country staff worked with appropriate MOH partners in each respective focus country to align and reposition national family planning strategies according to the framework of the Ouagadougou agreement. They also assisted national and municipal leaders in planning and budgeting for family planning, as a means of increasing support for these services. The project also drafted or updated key national family planning guidance documents to ensure promotion of the most up to date, evidence-based practices such as integration of the EngenderHealth technical approach of providing IUD and implants into the into MOH policy, norms and protocol as well as in FP training curriculum in Burkina Faso. Regarding direct policy change efforts, the AgirPF project worked with MOH partners to train and use effective policy decision-making tools such as RAPID models. These tools are designed to demonstrate the urgency of dedicated effort in family planning, and by engaging MOH partners directly in their training and use, the project was able to improve the dialogue around FP policy change. The AgirPF regional policy/advocacy leader also worked closely with all countries to draft policy strategic action plans aimed at reducing policy barriers to FP access. The plans were designed to aid stakeholders in each country to identify specific policies that hamper family planning access (such as restrictions for youth services) and craft approaches to remove these barriers. Finally, AgirPF coordinated with counterparts in the MOH to engage faith-based organizational leaders to promote responsible childbearing in countries like Cote d’Ivoire and Niger. From project Year three, AgirPF has worked to build a regional partnership under the umbrella of WAHO, with the ultimate goal of contributing to an improved FP regional environment by supporting and replicating region wide the successful advocacy practices of RCPFAS in implementing countries. The said successful advocacy practices led other organizations to join WAHO and AgirPF, including SWEDD and LMG/WA, to agree on additional joint activities aimed at further strengthening efforts to reduce barriers to FP and to secure the capture of the demographic dividend: - Support for documenting RCPFAS advocacy experiences in implementation countries of AgirPF; E4D: Draft 2 AgirPF Performance Evaluation report 37 - Organization of a regional workshop to share the documented experiences, and to formalize a regional mechanism for consultation and exchange on advocacy to promote FP and the demographic dividend. The workshop resulted in the Regional Framework of Consultation and Exchange of the Network of Advocacy Champions for a Demographic Dividend (CRCE￾RCPFAS-DD), which is now functioning and equipped with a work plan. - Institution of a platform for partner organizations including WAHO, AgirPF, SWEDD, UCPO, and HP+ to ensure synergy in supporting the implementation of the CRCE-RCPFAS-DD work plan and bring all partners together periodically, through workshops and skype call meetings to assess work plan progress (partners now consulting through periodic meetings and other communications); - Support for the replication of advocacy good practices region wide. Sub Q 4.3 Are there any tangible changes in FP policy contributed to by AgirPF support? As policy advocacy is a slow and arduous process, the process is ongoing in most countries. However, there several can be cited, with others in the works: ● AgirPF is currently assisting the introduction of a draft policy in Niger to allow CHWs to provide long-acting and reversible contraceptive methods. ● In Cote d’Ivoire, the advocacy work supported by AgirPF has led to a plan for formal revision of the national SRH law that would increase funding for FP and lower barriers to access through task-shifting to CHWs and integration of FP into other RH services. ● Advocacy for task-shifting in Togo has led to adoption of a new Community Based Initiatives policy that allows CHWs to offer family planning methods, including injectables. ● In November and December 2015, AgirPF supported a series of meetings culminating in the validation of 3 new regulations (ministerial orders and decrees) to implement the RH Law in Togo: (i) list of products, methods and means of contraception legally authorized in Togo, (ii) decree on protection of service providers in RH services, (iii) decree defining the mission, organization, composition and functioning of a national inspection and control unit for reproductive health services in all facilities in Togo. ● AgirPF provided technical and financial support to the URCB13 which held advocacy meetings with Evangelical, Muslim and Traditional governing bodies for religious leaders to discuss responsible childbearing and prepare them for the adoption of a national policy document on responsible childbearing in Burkina Faso. "The law has not yet been voted on. We work together but we are not far from having the law. AgirPF has been a pillar in advocating for the law of reproductive health to be put in place, AGIRPF has been a pillar for a budget line on FP. AgirPF has been a pillar for clarification of the value of FP, pregnancy in schools, all this ". MOH partner, Cote d’Ivoire Primary Q5: What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? The AgirPF project has a wide-range scope of work and operated on both regional and country/local levels. As a result, the project has yielded a number of lessons that could benefit future programming on family planning. 13 Union des Religieux et Coutumiers du Burkina Faso. E4D: Draft 2 AgirPF Performance Evaluation report 38 Sub Q 5.1 What were program successes useful for advancing family planning programming? According to stakeholder input and project document reviews, the AgirPF project has several successful elements that could be used to advance future family planning programming: ● Use of evidence-based HIPs – One of the most important contributions of the AgirPF project was in promoting the adoption of HIPs through regional and national-level efforts. The process involved not only identifying appropriate HIPs, but supporting countries to document and disseminate learning from application of specific HIPs to elevate the most important approaches in each context. Based on the results presented in the WAHO Forum on Best Practices, the four key HIPs relevant for use in this region are FP special days (mobile outreach), the post-partum family planning provision, FP provision in post-abortion care, and the use of the RAPID model to address policy advocacy and socio-cultural advocacy with religious leaders. In particular, stakeholders emphasized the use of mobile outreach FP days as a successful method of seeing immediate benefits in FP service delivery. ● Regional coordination through key partners, meetings, workshops and support – The AgirPF project worked through a number of mechanisms to coordinate regional collaboration and support for the project. This included collaborating with regional actors such as WAHO and other international consortium partners to develop strategic action plans in key program areas. They also conducted several key networking and capacity building activities such as the WAHO Forum on Good Practices and the work with the Network of Advocacy Champions for Adequate Funding of Health to devise regional strategies for increasing access to family planning. Furthermore, the use of regional technical experts in policy and programming helped AgirPF maintain a unified approach to improving FP service delivery within the region. ● Capacity building through training of trainers/Centers of Excellence – To promote sustainable capacity building for trained FP providers, the AgirPF project identified Centers of Excellence in each country to participate in master FP training and supervision efforts. In each country, AgirPF trained approximately 12 master trainers and 20 facilitative supervisors, who in turn were able to disseminate family planning training to target providers across the intervention areas. By investing in this training of trainers’ model, AgirPF created a more sustainable program for long-term FP training within countries. As one stakeholder in Niger said: “The nature of the AgirPF intervention is to “make do”, which means that the project reinforces local capacity to be able to take charge themselves.” ● Strengthening advocacy efforts through workshops and applied tools – The AgirPF project, through their collaboration with local civil society groups and MOH counterparts, and the use of advocacy tools like the RAPID models was able to give policy advocates the tools and support they needed to engage with key groups (such as religious leaders) and push for policy reforms in each country. Participants made specific reference to the utility of the advocacy tools and the capacity building workshops. Through their efforts, there has been movement on family planning policies in each target country. ● Updating of national policy, guidelines and strategic plans for FP – Through their coordination with MOH counterparts, and in alignment with the Ouagadougou partnership agreements, AgirPF worked at the national level in each intervention country to update FP guidelines to be inclusive of best practices, as well as ensuring that country FP strategic plans were aligned to meet goals. E4D: Draft 2 AgirPF Performance Evaluation report 39 ● Logistics management coordination – Stakeholders noted the impact of AgirPF’s work to improve the commodities supply chain at facility and management levels. Additional work is needed to ensure adequate and continuous supply at the national level. In addition to analysis of stakeholder interviews, the evaluation team asked AgirPF regional staff to provide additional thoughts on which aspects of the project were most successful. Specifically, the evaluators asked AgirPF staff to detail which elements of the current approach they would replicate if they were designing a regional project of this scale. The following are their suggestions, presented in the order in which they were relayed to the evaluation team: ● All current interventions are relevant to achieving the project’s objectives, but that there should be a vetting process by which specific approaches are scrutinized for relevance and impact within each country context. For example, demand generation may be a more critical endeavor in one country while post-partum IUD may be more relevant in another. These differences should be acknowledged and resources should be allocated accordingly. ● Promotion of high-impact activities such as task shifting, community health workers, mobile outreach, PPPF and PACPF, FP special days, and quality improvement were all viewed as essential activities to continue, as were behavior change communication activities ● Strengthening and expansion of the data collection system database (DHIS2) so as to include monthly LMIS statistics at the national level ● Facilitative supervision for strengthening providers' skills for quality FP services provision ● Provision of reproductive health services adapted to adolescents and young people in school and universities Sub Q 5.2 What were program challenges useful for adjusting family planning programming in the future? The respondents shared a number of challenges to family planning programming. Many of them were related to contextual issues, while there were some related to the current project structure or performance. The two categories have been separated here, and items listed in order of the frequency with which the themes appeared in the qualitative data. Context-related Challenges ● Lack of funding follow-through at national level – The project itself provided funding for programming, but participants feel that a long-term challenge to program success is in the follow-through of funding agreements made by the government. More advocacy is needed at the government level to ensure adequate budgeting and timely follow-through on commitments to funding FP services. ● Enduring socio-cultural barriers – Although the project did include SBCC aimed at changing cultural barriers to FP uptake, almost all respondents noted that this is an enduring challenge to program success. Additional, targeted efforts to combat stigma around FP use, through a variety of media and community-based initiatives, could help improve demand generation and FP uptake among the target population ● Lack of facility-level infrastructure – One of the primary challenges in providing appropriate family E4D: Draft 2 AgirPF Performance Evaluation report 40 planning care was lack of infrastructure at the facility level. In particular, when integrating FP services into other services or adding new methods like IUDs, the lack of adequate space and privacy for clients was a notable barrier to care. ● Consistent, reliable commodities through the “last mile” – Despite advances in logistics management procedures because of the project, stakeholders across several countries acknowledged persistent challenges to commodities stock-outs due to bottlenecks in the supply chain system. ● Staff turnover at facilities – At the facility level, staff turnover is very high and thus staff that are trained in FP methods through the project may be relocated in a short amount of time. New staff coming may not receive specific FP training, resulting in a gap in service delivery at the facility level. Although the project attempted to address such issues through creating master trainers at Centers of Excellence in each country, several respondents recommended using regular on-site training or incorporating FP into provider curricula to address this issue. ● Provider motivation – in at least two countries, participants noted the issue of provider motivation as a barrier to improving FP service delivery. Particularly in areas where FP is integrated into an existing service, providers may feel over-burdened with additional work and no additional reward. Further, when faced with shortages of supplies, materials or inadequate infrastructure, providers find it difficult to engage in improved FP service delivery. ● Male involvement – In at least two countries, stakeholders noted the lack of motivation and participation in family planning by male partners as a barrier to long-term uptake. Future programming should include explicit initiatives aimed at increasing support for family planning by male partners. Project-related Challenges ● Understaffing of AgirPF regional/country staff – Overall, the AgirPF staff felt that for the size and scope of the activities directed under this regional project, there were insufficient staff at all levels. There were delays in hiring and onboarding of staff at the beginning of the project, as well as frequent turnover and gaps in human resources that made it difficult to manage the number of activities and partners involved. Future regional projects should consider the needs of appropriately managing the multitude of activities on a project of this scope and create staff positions accordingly. ● Lack of harmonization between project activities and government initiatives - Burkina Faso in particular, the regional and district managers which the evaluators interviewed felt that AgirPF did not fully coordinate their activities through the existing government structure and were prone to pursuing interventions without passing through the appropriate authorities. They also mentioned a need for increased coordination with other existing NGOs working in that space. Although no other countries reported this issue with AgirPF, it bears noting as a challenge to future family planning programing. ● Improve timeliness of finance and accounting structures – Both AgirPF regional/country staff and their counterparts in-country mentioned that the approval processes for purchase orders and disbursement were seen as an impediment to rapid response and action on the ground. Future projects should streamline this process to enable more nimble and reactive capabilities E4D: Draft 2 AgirPF Performance Evaluation report 41 ● Heavy data reporting burden at facility level – Staff at facilities reported being burdened with the number and type of data reports owed to AgirPF project by the facility staff. In some cases, providers felt these were parallel systems not fitting with existing national indicator definitions and collection methods. That duplicated reporting efforts for already over-taxed staff. As with the “success” section above, the evaluation team asked AgirPF regional staff to provide additional thoughts on which aspects of the project were most challenging. Specifically, the evaluators asked AgirPF staff to detail which elements of the current approach they would change if they were designing a regional project of this scale. The following are their suggestions, presented in the order in which they were relayed to the evaluation team: ● It would be better to select all health facilities of selected district or city as intervention sites and choose health facilities in different geographical areas as control sites. When working only in a subset of sites within a district, it becomes impossible to provide aggregate results of the intervention to the MOH in a way that is meaningful to them (i.e., being able to provide entire district-level data would be more helpful). ● When selecting intervention sites, pay specific attention to the potential they have for improving family planning if given support. For instance, a facility with one consultation room and one or two providers taking care of all types of patients in a city with many other facilities will not make a sizeable impact on FP provision no matter how intensive the interventions in that facility are. ● Make sure the staffing is sufficient at the beginning of the project to avoid work overload on the staff. ● Create a limited, focused and easy-to-track set of indicators that are directly linked to the project interventions and their effects/impact. This is not the case with the early and current versions of the PMP. ● Invest more funds for the project to allocate SBCC activities sufficiently and allow for light structural renovation (i.e., compartmentalization of the rooms to ensure the privacy of clients, necessary equipment, furniture, etc) Primary Q6: How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Sub Q 6.1 How has AgirPF managed AgirPF staff in focus countries? (Regional AgirPF performance) Relationships between country and regional staff at AgriPF were positive, with reports of timely communication, team-oriented approach and good management procedures. However, during internal conversations with AgirPF staff, there was a theme around the project being understaffed both at regional and country level. Staff felt that given the scope of the project, the reach and volume of the different activities, they did not have sufficient human resources at the project level to E4D: Draft 2 AgirPF Performance Evaluation report 42 adequately meet the needs of the project. Staff felt they were forced to work overtime just to keep the project moving, and that they lacked proper technical and administrative counterparts between regional and country staff. This lack of adequate staff translated into real frustration both for the program staff as well as their partners and counterparts on the ground. “At the beginning, given the volume of work, we realized that [staff] was really insufficient. Fortunately, the donors are somewhat sensitive to this and we have asked funders to recruit additional staff… But until now if I am asked I will say that we must increase further.” – AgirPF staff . “Most people work beyond and so far, you will see that people come very early and leave late and during that time they work. There are really a lot of loads in terms of time, which is why I say that human resources have to be increased ... In my opinion, having no focal point, having regional staff and being in different countries has also contributed greatly to making it difficult to monitor the project over time.” – AgirPF staff. "I find that the operation of AgirPF is cumbersome. We cannot directly reach the coordinator, so things can drag on. There is a lot of time wasted, many activities scheduled without funding, failure to meet deadlines due to the burden of funding.” – Consortium partner, Burkina Faso. Sub Q 6.2 How has AgirPF managed AgirPF staff in focus countries? (Headquarter AgirPF performance) As mentioned above, the main issue between the AgirPF project staff and the coordinating body at EngenderHealth headquarters centered on the cumbersome financial and accounting procedures which slowed disbursement of funds to the project. No other major issues were noted. Sub Q 6.3 How has AgirPF managed partners in focus countries? The evaluation team spoke with a variety of local and consortium partners to AgirPF across the four countries to understand how they managed the relationship between the project and partners. Overall, the partners reported positive views of working with AgirPF staff. AgirPF staff reportedly had an open and communicative style, and provided appropriate support in their approach to working with partners. Here, again, partners noted some delays in funding disbursement that posed problems in activity planning and management. "Very very good collaboration. AgirPF is invited to the activities and they support us. It is our environment that is healthy, that is friendly. We want to see solidarity and support for ideas. For example, the day of contraception in Togo was never celebrated. The others came and put more money than we do.” – Local partner, Togo “In matters of communication they were very open and responsive [but] the accounting department is slow. Otherwise in terms of partnership, they are very open.” – Consortium partner. "Those who have evaluated us and trained are very competent. All the processes were well done, the trainings; everyone was satisfied.” – Local partner, Togo “What I can say is that AgirPF is an essential partner in supporting family planning activities in Cote d’Ivoire. The project was reviewed... It must be continued and improved. Improve it by adding additional sites because there are many sites that are neglected…So if AgirPF has additional sites, it will allow us to cover the country a bit.” Local partner, Cote d’Ivoire E4D: Draft 2 AgirPF Performance Evaluation report 43 The only major complaint in partnership management for AgirPF came from their municipal government counterparts in Burkina Faso, who felt that they were not adequately included in program planning and implementation. "We often feel that AgirPF wants to take over from us to do our job when it has to accompany us. They often go to the field directly without our collaboration. Interventions (allocations of resources) are carried out on the ground without the knowledge of the health authorities." – Regional health manager, Burkina Faso Sub Q 6.4 Are AgirPF supported sites maintaining environmental compliance? A purposive sample of ten AgirPF sites per country were visited to determine environmental compliance during the course of field data collection. Facility staff were asked about the ways in which the site disposed of both solid and liquid waste. For solid waste management, 8 out of 10 facilities in Togo, 4 out of 10 facilities in Cote d’Ivoire and 5 out of 10 facilities in Burkina Faso reported collecting and milling waste outside, with only 2 out of 10 sites in each country reporting the use of an incinerator. In Niger, 5 out of 10 sites reported burning waste in an incinerator, while 3 burned waste in an open hole, 1 burned and buried waste and 1 collected and milled waste outside. These findings show no improvement over environmental compliance recorded at the baseline for intervention sites. According to the AgirPF baseline report: “In Niger, the large majority of facilities reported that solid infectious wastes are burned in incinerators (64% of intervention facilities), as did about half of facilities in Burkina Faso and in Togo. Slightly smaller proportions of facilities in these countries (fewer than 30% of facilities in in Burkina Faso and about one￾third of intervention facilities in Togo, and about 10% of facilities in Niger) reported that they burn such wastes in an open pit. Solid infectious wastes were burned in an open pit or in an incinerator in a significant minority of facilities in Côte d’Ivoire.” Figure 11: Solid waste disposal practices E4D: Draft 2 AgirPF Performance Evaluation report 44 For disposal of liquid waste, Niger was the only country that reported disposing of liquid waste in septic tanks (9 out of 10 sites). In Cote d’Ivoire, 4 out of 10 facilities reported disposing of liquid waste in an open hole, in Togo, 3 out of 10 facilities reported collecting liquid waste in containers and Burkina Faso facilities reported using open holes and wells for liquid disposal. It should be noted that in all countries but Niger, and particularly in Burkina Faso, respondents answering the question on liquid waste at numerous facilities did not know the actual mode of liquid waste disposal. The evaluation team would thus caution against using these data as a point of comparison against the AgirPF baseline findings, which showed: “Liquid infectious wastes are collected in septic tanks at the majority of facilities in Burkina Faso and in Togo and at a significant minority of the facilities in Niger.” Figure 12: Liquid waste disposal practices RECOMMENDATIONS The AgirPF project is a multi-partner, multi-faceted regional project that aimed to address the issue of improved family planning service delivery through a variety of modalities. As such, it tackled everything from policy advocacy to provider training to logistics management and more. A project of this scope, while seemingly unwieldy, managed by AgirPF staff with attention to communication, synergistic partnerships and with evidence-based practices. The following recommendations stem from a synthesis of the overall qualitative and quantitative data collected and analyzed for the performance evaluation and are presented in no particular order: ● Focus on “quick wins” through FPSDs and mobile services – While participants appreciated a multi￾faceted approach to long-term FP improvement, many mentioned the importance of focusing efforts on providing immediate benefits to target populations. Namely, the use of FP special days and mobile outreach to provide methods for those in need was viewed as one of the most successful aspects of AgirPF’s work and should be replicated/promoted throughout the focus countries. ● Improve finance and accounting structures – Both AgirPF regional/country staff and their E4D: Draft 2 AgirPF Performance Evaluation report 45 counterparts’ in country mentioned the cumbersome nature of the project current financing and accounting procedures. The number and nature of approvals needed for purchase orders and disbursement were seen as an impediment to rapid response and action on the ground. Future projects should streamline this process to enable more nimble and reactive capabilities. ● Provide adequate project technical and administrative staff – For a project of this size and complexity, it is essential that the staffing be sufficient and in place as soon as possible after agreement signing. The AgirPF project struggled to staff up in the first year of its program, which led to significant delays in rollout of activities. Furthermore, even when staff were increased, both in country and regional staff felt overextended in their work. Partners noted an admirable work ethic and commitment by AgirPF staff, even so there were some participants that felt there were delays in communication due to lack of staff availability. ● Continue to promote and propagate high-impact practices – The use of high-impact practices was viewed as a success by many of the participants and should be continued in future programming. The use of forums like the one on Good Practices organized by WAHO, where HIPs were tested in each country and lessons shared in a regional format, are of particular benefit to aiding the scale-up of best practices across the region. The HIPs that were most notable in this project were FPSDs, community engagement through CHWs, integration of family planning into PAC and the postpartum period and engagement of religious leaders for policy change. ● Continue to push for policy change – Though the gains in policy advocacy are less immediate and tangible than other areas of AgirPF’s work, both civil society and MOH partners view this as a critically important effort that must continue in order to have large-scale impact on FP service provision. In particular, countries should continue to engage important gatekeepers like religious leaders to demonstrate the urgency of responsible childbearing to the population’s survival. Future projects should also continue to push for inclusion of vulnerable and neglected groups like youth. ● Capitalize on positive achievements to encourage buy-in at national level - Overall, almost all of the participants the evaluation team interviewed viewed the work of AgirPF as critical and fruitful. As such, they felt that the successes of the project should be clearly distilled and promoted at the national level in order to encourage government buy-in and continuation of activities beyond the life of the project ● Take into account the need for infrastructure improvements – Across all countries, actors noted the need to provide structural improvements to facilities in order to be able to provide adequate FP service improvements. These included mainly areas for private conversations and rooms for method insertion (for IUDs and implants). Some participants also noted the need for other basic material support such as locked cabinets, shelves, chairs, etc. Without critical basic functions such as these, many facilities will remain incapable of providing access to adequate family planning services. ● Improve provider working conditions – Though beyond the scope of most projects of this nature, it is worth noting that several participants in this study felt that provider training alone was not sufficient to ensure their engagement and promotion of FP services. Many of these providers are already facing high work burdens, and integrating new/improved services in their work should be joined by remuneration or additional recognition to improve provider motivation. Creative solutions to enduring human resources for health issues must be incorporated into future models of change in this region. E4D: Draft 2 AgirPF Performance Evaluation report 46 ● Devolve provider training to site-level and/or provide more regular FP training – Due to high staff turnover at facilities, many AgirPF trained staff had already been transferred from the facilities that the evaluation team visited. This is a common issue in countries, where human resources are scarce. Thus, devolving provider training to on-the-job or on-site formats and/or providing regular training to facility staff can help ensure continuity and presence of trained FP providers. ● Institutionalize provider training in FP – Another suggestion made for improving and ensuring provider readiness for FP service provision was to institutionalize FP training into exiting training curricula at the national level. This way, a foundation for knowledge on FP service provision can laid for each provider as part of their routine training. AgirPF began this work by creating national centers of excellence and providing training of trainers activities in each country. However, these activities must be maintained diligently and ownership transferred successfully to national MOHs to ensure sustainability. ● Increase supervisory visits – Several providers noted the benefits of facilitative supervision visits and requested that such visits be more frequent. Through increased supervisory visits, programs can ensure trained providers are cementing their FP services skills and can further identify gaps at the facility level if/when trained providers are relocated. ● Provide continuity in future programming to continue gains made – As of the time of this evaluation, participants and AgirPF staff felt that the program was just getting into full swing. Future projects should carry forward the successful aspects of this program through a seamless transition to further funding and programming so as not to disrupt the potential gains stemming from current activities. As with the “success” and “challenges” sections above, the evaluation team asked AgirPF regional staff to provide additional thoughts on essential elements for future regional projects. The following are their suggestions, presented in the order in which they were relayed to the evaluation team: ● More partnerships and leveraging of local and regional relationships and resources. Some interventions are critical for the success of the process but are not eligible to the project funding (e.g. renovation of consultation rooms, equipment, and construction of incinators, etc.). The project should be implemented in health facilities either where such basic needs are already met, or where investments for structural improvements can be made by the local government or by other partners. ● Future projects should include a strong and well-staffed knowledge management unit from the very start, to enable the efficient capture and dissemination of program learning throughout the life of the project. A KM unit at the start of the project. ● While it is important to maintain a regional presence to coordinate across ECOWAS countries and engage with regional actors such as WAHO, more decentralization of the regional offices into local country offices is necessary. At present, the AgirPF staff feel there is under-representation in each intervention area in-country. ● Although this specific project was focused on high-density urban areas and their unique challenges, a more comprehensive contribution equitable access and utilization of family planning services can only take place when the needs of the most vulnerable rural, populations are also addressed. E4D: Draft 2 AgirPF Performance Evaluation report 47 ● Strengthening and expanding the supply of reproductive health / FP services for key and vulnerable populations and people with disabilities in settings where they are located. E4D: Draft 2 AgirPF Performance Evaluation report 48 CONCLUSIONS AgirPF has contributed important support to the regional advancement of improved family planning access and utilization. This was done through a combination of regional training, networking and exchanges as well as harmonized in-country efforts aimed at training providers, supporting policy advocacy, creating or increasing domestic budgets dedicated to FP, improving logistics management and behavior change activities. AgirPF, through its collaborative initiatives played a catalytic role in re-positioning FP in its intervention countries, a feature that goes beyond mere quantitative targets. AgirPF was designed as a demonstration project aimed at testing specific evidence-based approaches (HIPs) within the Francophone context. The lessons learned by this project can contribute not only to future iterations of regional investment, but can also be absorbed and taken to scale at the national level in each intervention country. The AgirPF project followed several key elements of successful scale-up of global activities, including engagement of strong leadership, active management of partners and use of evidence-based practices adapted to local contexts (Yamin G. 2011). Per the suggestions of both stakeholders and AgirPF staff, regional family planning projects must recognize the diversity within individual West African countries, and tailor approaches to be as specific and relevant as possible for each local context, while still maintaining the overarching objectives of the regional project. D. Despite its successes, AgirPF has experienced difficulties in terms of human resources shortfalls and the constant MOHs staff turnover in its intervention countries. AgirPF also experienced delays in project implementation startup due to structural / contractual requirements (no technical activity before the completion of the baseline studies). In addition, contextual problems such as sociocultural barriers to FP, the insufficient number of providers and infrastructure were not supportive for the project. This situation requires AgirPF to be creative in order to find strategies to help it overcome these obstacles to move decisively towards achieving the project's results. Future projects should take the lessons of AgirPF into account to create more streamlined, targeted approaches that are well-supported by technical and administrative staff needed to execute on the multitude of activities needed to increase access and utilization of family planning services. E4D: Draft 2 AgirPF Performance Evaluation report 49 REFERENCES FY15Q1_PY2 - AgirPF_Report Oct-Dec 2015 Institut National de la Statistique et de la Démographie (INSD) et ICF International, 2012. Enquête Démographique et de Santé et à Indicateurs Multiples du Burkina Faso 2010. Calverton, Maryland, USA : INSD et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR256/FR256.pdf. Consulted January 2017. Institut National de la Statistique (INS) et ICF International. 2012. Enquête Démographique et de Santé et à Indicateurs Multiples de Côte d’Ivoire 2011-2012. Calverton, Maryland, USA : INS et ICF International. http://dhsprogram.com/publications/publication-FR272-DHS-Final-Reports.cfm. Consulted January 2017. Institut National de la Statistique (INS) et ICF International, 2013. Enquête Démographique et de Santé et à Indicateurs Multiples du Niger 2012. Calverton, Maryland, USA : INS et ICF International. Available at http://dhsprogram.com/pubs/pdf/FR277/FR277.pdf. Consulted January 2017. Population Reference Bureau. 2016. World Population Data Sheet with a Special Focus on Human Needs and Sustainable Resources. Available at: http://www.prb.org/pdf16/prb-wpds2016-web-2016.pdf. Consulted January 2017. Ministère de la Planification, du Développement et de l’Aménagement du Territoire (MPDAT), Ministère de la Santé (MS) et ICF International, 2015. Enquête Démographique et de Santé au Togo 2013-2014. Rockville, Maryland, USA : MPDAT, MS et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR301/FR301.pdf. Consulted January 2017. Sedgh G., S. Singh and R. Hussain 2014. Intended and Unintended Pregnancies Worldwide in 2012 and Recent Trends. Studies in Family Planning. 45: 301–314. doi:10.1111/j.1728-4465.2014.00393.x. Consulted January 2017. E4D: Draft 2 AgirPF Performance Evaluation report 50 APPENDIX 1: Evaluation questions and sub-questions Primary Q1: How has AgirPF performed? Sub Q 1.1: How has AgirPF performed against reaching top line indicators of 700,000 method adopters, yielding 1,683,000 Couple-years of protection (CYP)? Is the Project on track to reach its targets? [CYP Target]. Sub Q 1.2: Did family planning provision increase after AgirPF intervention? Sub Q 1.3 How has AgirPF performed against targets for number of HIV positive women who received comprehensive FP services [Indicator 11 from PMP]; Sub Q 1.4 How has AgirPF performed against targets for number of Community Health Workers supported and supervised [Indicator 16 from PMP] Sub Q 1.5 How has AgirPF performed against targets for number of youth who participated in educational program on gender, Family Planning (FP) and Sexual and Reproductive Health activities (SRH) [Indicator 17 from PMP] Sub Q 1.6 How has AgirPF performed against targets for number of Best Practices (BPs)/ High Impact Practices (HIPs) for family planning and maternal and child health and/or HIV/AIDS incorporated into local, district or national health protocols or standards [Indicator 19 from PMP] Sub Q 1.7 How has AgirPF performed against targets for number of best practices piloted through operations research studies [Indicator 20 from PMP]. Primary Q2: Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services]. Sub Q 2.1 What policies, norms, guidelines, protocols, etc.related to the selected HIPs have been advanced? Sub Q 2.2 To what extent have these HIPs been scaled up in AgirPF focus countries? Sub Q 2.3 To what extent have these HIPs contributed to AgirPF’s results? Sub Q 2.4 What AgirPF has done to facilitate replication of these HIPs within the country and in other countries? Sub Q 2.5 How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? Primary Q3: To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Sub Q 3.1 Are the elements of IR1 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Sub Q 3.2 Are the elements of IR2 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? E4D: Draft 2 AgirPF Performance Evaluation report 51 Sub Q 3.3 Are the elements of IR3 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Sub Q 3.4 How could the overall theory of change be improved to better address program design needs? Primary Question 4: For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? Sub Q 4.1 To what extent did AgirPF’s support advocacy partners in removing policy barriers to FP? Sub Q 4.2 To what extent did AgirPF’s support MOH partners in removing policy barriers to FP? Sub Q 4.3 Are there any tangible changes in FP policy contributed to by AgirPF support? Primary Question 5: What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? Sub Q 5.1 What were program successes useful for advancing family planning programming? Sub Q 5.1 What were program challenges useful for adjusting family planning programming in the future? Primary Question 6: How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Sub Q 6.1 How has AgirPF managed Agir staff in focus countries? (regional AgirPF performance) Sub Q 6.1 How has AgirPF managed Agir staff in focus countries? (headquarter AgirPF performance) Sub Q 6.1 How has AgirPF managed partners in focus countries? Sub Q 6.1 Are AgirPF supported sites maintaining environmental compliance? E4D: Draft 2 AgirPF Performance Evaluation report 52 APPENDIX 2: Number of intervention and control sites sampled for evaluation by Region and District Region Districts # of AgirPF sites # of control sites BURKINA FASO Bobo Dioulasso Dafra, Do, Accart Ville 7 7 Koudougou Koudougou 5 9 Ouagadougou Baskuy, Boulmiougou, Nongremassom, Signoghin 18 14 COTE D’IVOIRE Abidjan 1-Grands Ponts Adjamé-Plateau-Attecoubé , Dabou, Yopougon Est, Yopougon Songon 14 9 Abidjan 2 Abobo Est, Abobo Ouest, Anyama, Cocody Bingerville, Koumassi-Port Bouet-Vridi, Marcory-Treichville 16 21 NIGER Niamey Niamey 1, Niamey 2, Niamey 3 14 15 Maradi Maradounfa, Maradi Commune 16 15 TOGO Maritime District 1, District 2, District 3, District 4, District 5, Golfe 15 9 Centrale Tchaoudjo 9 7 Kara Kozah, Binah 9 8 E4D: Draft 2 AgirPF Performance Evaluation report 53 APPENDIX 3: Full suite of data collection tools Facility Service Statistics Data Extraction Sheet FACILITY IDENTIFICATION Name of the City: ______________________________ Name of District: ______________________________ Name of the facility_____________________________ Tel: _________________________________________ Type of Health Facility : 1 = Teaching Hospital 2 = Regional hospital 3 = District Hospital; 4 = Other Hospital 5 =Polyclinic 6 = Health Center 7 =Maternity Home 8 = Health post 9= Other ________________ Operating Authority: 1 = Public; 2 = Private for profit 3 = NGO 4 = Faith based 5 = Other (specify)_____________ Date: / / DAY / MONTH / YEAR Name of the interviewer______________________________ E4D: Draft 2 AgirPF Performance Evaluation report 54 Name of Facility/ District Supervisor_________________________________ 55 How many clients received the following services in 2013 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R a. Combined Oral Pill (Lo-Femenal) b. Progesterone Only Pill (Ovrette) c. IUD d. Injectables e. IMPLANT e1. Jadelle e2. Implanon e3. Sino Implant f. Male Condom g. Female Condom h. Spermicides i. Emergency Contraceptives j. Female sterilization 56 How many clients received the following services in 2013 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R k. Male sterilization How many clients received the following services in 2014 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R a. Combined Oral Pill (Lo-Femenal) b. Progesterone Only Pill (Ovrette) c. IUD d. Injectables e. IMPLANT e1. Jadelle e2. Implanon e3. Sino Implant f. Male Condom 57 How many clients received the following services in 2014 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R g. Female Condom h. Spermicides i. Emergency Contraceptives j. Female sterilization k. Male sterilization How many clients received the following services in 2015 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R a. Combined Oral Pill (Lo-Femenal) b. Progesterone Only Pill (Ovrette) c. IUD d. Injectables 58 How many clients received the following services in 2015 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R e. IMPLANT e1. Jadelle e2. Implanon e3. Sino Implant f. Male Condom g. Female Condom h. Spermicides i. Emergency Contraceptives j. Female sterilization k. Male sterilization 59 How many clients received the following services in 2016 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R a. Combined Oral Pill (Lo-Femenal) b. Progesterone Only Pill (Ovrette) c. IUD d. Injectables e. IMPLANT e1. Jadelle e2. Implanon e3. Sino Implant f. Male Condom g. Female Condom h. Spermicides i. Emergency Contraceptives j. Female sterilization 60 How many clients received the following services in 2016 (N=new clients and R = repeat clients) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec N R N R N R N R N R N R N R N R N R N R N R N R k. Male sterilization 61 INFECTION PREVENTION AND ENVIRONMENTAL MONITORING AND MITIGATION PLAN NO QUESTIONS CODING CLASSIFICATION GO TO 1101 What system do you have for solid infectious waste disposal? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected and disposed externally….…1 Burned in incinerator……………………2 Burned in open pit………………………3 Burned and buried………………………4 Buried…………………………………….5 Put in trash/open pit…………………….6 Put in pit latrine………………………….7 Other (specify)__________________ 8 Don’t Know…………………………….88 1102 What system do you have for liquid infectious waste disposal? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected in septic tanks…………..…1 Collected by municipality…………..….2 Facility connected to the sewage system………………………...3 Collected in container and burned…….4 Dispose in pit……………………………5 Put in trash/open pit……………………6 Flush down toilet………………………..7 Other (specify)_______________________ 8 Don’t Know…………………………….88 1103 How does this facility dispose of items such as syringes and bandages that may be contaminated? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected and disposed externally. …1 Burned in incinerator........................2 Burned in open pit…………………..3 Burned and buried…………………..4 Put in trash/open pit…………………5 Put in pit latrine………………………6 Other (specify)________________7 Don’t Know…………………………88 1104 Does the health facility/NGO have the national and international policies and protocols for the collection and processing of recyclable waste and handling and treatment / disposal of other waste? Yes……………………………………1 No……………………………………..0 Don’t Know……………………….….8 1104a If yes, ask to see them Observed…………………………….1 Not observed……..………….………2 1105 How many health facility/NGO staff are sensitized on how to avoid an adverse 62 NO QUESTIONS CODING CLASSIFICATION GO TO environmental impact from activities conducted in the workplace Number ____________ 1106 Are the facility training curricula revised to include training in infection prevention for service providers including medical waste management and environmental protection? Yes……………………………………1 No……………………………………0 Don’t Know………………………….8 1107 Has the facility conducted clinical training for services providers in IP including medical waste management and environmental protection? Yes……………………………………1 No……………………………………..0 Don’t Know………………………..…8 1108 If yes, how many staff members received this training? Number of staff ____________ 1109 How many copies of MOH medical waste management and environmental protection procedures were distributed to the health facility? Number of copies__________ 1110 Is the facility equipped with IP materials? Yes……………………………………1 No……………………………………..0 Don’t Know…………………..………8 1111 How many sterilizers does the Health Facility have? 1112 How many incinerators does the Health Facility have? 1113 If no incinerator, ask if the Health Facility has a double pit? Yes………………………………..….1 No…………………………………….0 1114 How many times in the last six months has a supervisor come to this facility to supervise trained providers in Infection protection including medical waste management and environmental protection? Number: ____________ 63 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Evaluation KEY INFORMANT GUIDE: SERVICE PROVIDERS Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of health facility: ___________________________________________________ Sex of informant: Male [___] Female [___] Designation of informant: 01=Medical Doctor/Officer 02=Assistant Medical Officer 03=Clinical Officer 04=Nurse (RN/EN) 05=Medical Attendant 06=Other (specify) _______________________ [___|___] Service Delivery Point 01=Family Planning Clinic 02=Antenatal Care Clinic 03=Comprehensive Care Centre 04=Child Immunization 05=Maternity/Postnatal Ward 06=Outpatient Department 07=Other (specify _______________________ [___|___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and IBTCI working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at increasing access to family planning in your country. In particular, we would like to know how the AGIRPF project has helped strengthen facility-based health systems to increase access to family planning methods. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in health care. 64 1. What is your professional qualification? 2. In what capacity do you currently work in health care? 3. How long have you worked in this role? 4. What are your primary responsibilities? B. General views about health systems issues in reproductive, maternal, newborn and child health services: Now I would like to ask you about issues related to providing family planning in this facility. 5. What are the main challenges to providing family planning services at this facility? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 6. What types of support would help the facility improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support C. Family planning training with AGIRPF Now I would like to hear about your experiences receiving specialized training or support in family planning from the AGIRPF project. 7. Have you received any training or orientations from AGIRPF related to improving family planning services in the past 3 years? a. If yes, go through each product in the table and ask for details: 65 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) Contraceptive Technology YES NO REDI FP counselling method YES NO Contraceptives logistics management YES NO Gender norms YES NO 66 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) COPE for contraceptive security YES NO Infection prevention and control YES NO Facilitative Supervision YES NO Youth-friendly family planning services? YES NO 67 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) Post-partum Family Planning YES NO Environmental Compliance YES NO 68 8. Please explain the ways in which these orientations, overall, supported you to improve family planning services in your district. Probe on FP counseling, data management, supply-chain management, outreach, methods-specific knowledge, etc. 9. Were there any topics on which you wish you had received additional support? Probe to obtain multiple answers. For each subject, probe on details of training they would need 10. Please describe any family planning policy/guideline/norm changes that have taken place as a result of AGIRPF’s support. a. Probe for details on each change mentioned. b. For each change mentioned, please probe on the exact role that AGIRPF played in facilitating the change 11. Overall, what were your impressions of this training activity? Probe on both successes and shortcomings of the training a. Did you feel that you were given adequate time and information on each topic covered? If not, on what topic would you have liked to have more time? b. What are your impressions of the facilitators? (Probe: Were they knowledgeable and helpful? Or could have done better? If so, in what ways?) c. In your opinion, what could have been done to improve the training? 12. After the initial training, what kind of follow-up activities did AGIRPF provide to ensure lasting improvements? 13. Besides this training, what other supports did you received from the AGIRPF project? 14. Are there any additional supports that you wish you could receive to help improve access to family planning at this facility? We have now come to the end of our discussion. 15. Would you like to add anything else about AGIRPF or issues in improving family planning services? THANK YOU VERY MUCH FOR YOUR TIME 69 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation KEY INFORMANT GUIDE: MINISTRY OF HEALTH STAFF Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of institution: ___________________________________________________ Sex of informant: Male [___] Female [___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and Evidence for Development working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at increasing access to family planning in your country. In particular, we would like to know how the AGIRPF project has helped coordinate local partners to improve access to family planning methods. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in health care. 16. What is your professional qualification? 17. In what capacity do you currently work in health care? 18. How long have you worked in this role? 70 19. What are your primary responsibilities? B. General views about health systems issues in family planning services: Now I would like to ask you about issues related to providing family planning in this country. 20. What are the main challenges to providing family planning services in this country? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 21. What types of support would help the country improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support C. Family planning support with AGIRPF 1. During your time in this role, what was the nature of the MOH’s work with AGIRPF? Probe on what activities they did together, what type of support AGIRPF provided, what specific family planning outcomes they pursued together, etc. 2. Did your organization receive any orientations from or participated in any events by AGIRPF related to improving family planning in your country? (If no, skip to #6) b. If yes what was the approximate date of event/orientation? Please also describe the format of the event (Probe for: on-site, workshop, number of days, etc) c. Please describe the content of the orientation or events. Probe on the content of the event; for each topic, ask for details on the topics provided. d. What was the final result of this orientation/event? Probe on what was learned, networks created, connections made, action plans, etc. 3. Please explain the ways in which AGIRPF’s work has supported your efforts to improve family planning services in your country. Probe on advances they have made in FP services since working with AGIRPF 4. Were there any areas in which you wish you had received additional support/ partnership from AGIPF? Probe to obtain multiple answers. For each subject, probe on details of support they would need 5. Please describe any family planning policy/guideline/norm changes that have taken place as a result of AGIRPF’s support. c. Probe for details on each change mentioned. d. For each change mentioned, please probe on the exact role that AGIRPF played in facilitating the change 6. Overall, what were your impressions of working with AGIRPF’s? Probe on both successes and challenges of the support a. Did you feel that communication with AGIRPF was timely and adequate? If not, please give an example of a time where lack of communication affected your work with AGIRPF? 71 b. What are your impressions of the AGIRPF staff? Probe: What were positive aspects of working with AGIRPF staff? Please give an example What could be improved about working with AGIRPF staff? Please give examples. c. In your opinion, what could have been done to improve the support given to you by AGIRPF? Probe on coordination, communication, preparation, follow-up support, etc. Now, I would like to ask you about the conceptual design of the AgirPF project 7. The AgirPF project’s main objective is to increase access to and use of quality FP services. It proposes doing so through 3 main results: a. Expanded delivery of FP products and services b. Use of evidence-based service delivery approaches (such as FP outreach and training community health workers) c. Efforts to remove policy barriers and improve contraceptive security In your opinion, are the three intermediate results I listed sufficient to meeting AgirPF’s objective of increased FP use? If yes, please explain why these are sufficient. If no, please explain what additional program elements should be included to ensure reaching AgirPF’s objective. We have now come to the end of our discussion. 8. Would you like to add anything else about partnering with AGIRPF or issues in improving family planning services in your country? THANK YOU VERY MUCH FOR YOUR TIME 72 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation KEY INFORMANT GUIDE: DISTRICT/REGIONAL HEALTH MANAGERS Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of district: ___________________________________________________ Name of institution: ___________________________________________________ Sex of informant: Male [___] Female [___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and Evidence for Development working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at increasing access to family planning in your country. In particular, we would like to know how the AGIRPF project has helped strengthen facility-based health systems to increase access to family planning methods. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in health care. 22. What is your professional qualification? 23. In what capacity do you currently work in health care? 24. How long have you worked in this role? 73 25. What are your primary responsibilities? B. General views about health systems issues in family planning services: Now I would like to ask you about issues related to providing family planning in this district. 26. What are the main challenges to providing family planning services in this district? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 27. What types of support would help the district improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support C. Family planning support with AGIRPF Now I would like to hear about your experiences receiving specialized training or support in family planning from the AGIRPF project. 28. Have you received any training or orientations from AGIRPF related to improving family planning systems in your district? e. If yes, go through each product in the table and ask for details: 74 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) Contraceptive Technology YES NO REDI counselling method YES NO Reality Check YES NO Contraceptives logistics management YES NO 75 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) FP Advocacy YES NO Gender norms YES NO COPE for contraceptive security YES NO Infection prevention YES NO 76 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) Facilitative Supervision YES NO OCAT YES NO Spectrum YES NO AgirPF Health Information System YES NO 77 Did you receive training in…? If so, when did you receive this training? (month/year) Did you feel that the training was adequate? Please explain yes/no Did you receive follow-up support of any kind? (mark yes/no) If yes, please describe. How did this training help your work in FP provision? How could this training/support have been improved? What are challenges to sustained, appropriate use of this tool? (probe on staff turnover, lack of follow-up, etc) Youth-friendly FP services YES NO Environmental Compliance YES NO 78 29. Please explain the ways in which these orientations, overall, supported you to improve family planning services in your district. Probe on FP counseling, data management, supply-chain management, outreach, methods-specific knowledge, etc. 30. Were there any topics on which you wish you had received additional support? Probe to obtain multiple answers. For each subject, probe on details of training they would need 31. Please describe any family planning policy/guideline/norm changes that have taken place as a result of AGIRPF’s support. a. Probe for details on each change mentioned. b. For each change mentioned, please probe on the exact role that AGIRPF played in facilitating the change 32. Overall, what were your impressions of this AGIRPF’s support? Probe on both successes and shortcomings of the support a. Did you feel that you were given adequate time and information on each topic covered? If not, on what topic would you have liked to have more time? 33. What are your impressions of the AGIRPF staff? (Probe: Were they knowledgeable and helpful? Or could have done better? If so, in what ways?) 34. In your opinion, what could have been done to improve the support given to you by AGIRPF? Now, I would like to ask you about the conceptual design of the AgirPF project 35. The AgirPF project’s main objective is to increase access to and use of quality FP services. It proposes doing so through 3 main results: a. Expanded delivery of FP products and services b. Use of evidence-based service delivery approaches (such as FP outreach and training community health workers) c. Efforts to remove policy barriers and improve contraceptive security In your opinion, are the three intermediate results I listed sufficient to meeting AgirPF’s objective of increased FP use? If yes, please explain why these are sufficient. If no, please explain what additional program elements should be included to ensure reaching AgirPF’s objective. 36. Would you like to add anything else about AGIRPF or issues in improving family planning services? THANK YOU VERY MUCH FOR YOUR TIME 79 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation KEY INFORMANT GUIDE: CONSORTIUM PARTNERS Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of institution: ___________________________________________________ Sex of informant: Male [___] Female [___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and Evidence for Development working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at decreasing barrier to family planning support in your country. In particular, we would like to know how the AGIRPF project has partnered with local and international organizations in your country plan family planning programming. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in family planning. 37. What is your professional qualification? 38. In what capacity do you currently work in health care? 80 39. How long have you worked in this role? 40. What are your primary responsibilities? B. General views about health systems issues in family planning: Now I would like to ask you about issues related to providing family planning in this country. 9. What are the main challenges to providing family planning services in this country? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 10. What types of support does your organization provide to improve family planning services? Probe to obtain multiple answers. 11. What types of additional support is needed to improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support? C. Family planning partnership with AGIRPF Now I would like to hear about your experiences partnering with AGIRPF project to improve family planning services in your country. 12. During your time in this role, what was the nature of your organization’s partnership with AGIRPF? Probe on what activities they did together, what type of support AGIRPF provided, what specific family planning outcomes they pursued together, etc. 13. Did your organization receive any orientations from or participated in any events by AGIRPF related to improving family planning in your country? (If no, skip to #6) f. If yes what was the approximate date of event/orientation? Please also describe the format of the event (Probe for: on-site, workshop, number of days, etc) g. Please describe the content of the orientation or events. Probe on the content of the event; for each topic, ask for details on the topics provided. h. What was the final result of this orientation/event? Probe on what was learned, networks created, connections made, action plans, etc. 14. Please explain the ways in which your partnership with AGIRPF supported your efforts to improve family planning services in your country. Probe on advances they have made in FP services since working with AGIRPF 41. Please describe any family planning policy/guideline/norm changes that have taken place as a result of AGIRPF’s support. a. Probe for details on each change mentioned. b. For each change mentioned, please probe on the exact role that AGIRPF played in facilitating the change 81 15. Were there any areas in which you wish you had received additional support/ partnership from AGIPF? Probe to obtain multiple answers. For each subject, probe on details of support they would need 16. Overall, what were your impressions of partnering with AGIRPF’s? Probe on both successes and challenges of the support a. Did you feel that communication with AGIRPF was timely and adequate? If not, please give an example of a time where lack of communication affected your work with AGIRPF? b. What are your impressions of the AGIRPF staff? Probe: What were positive aspects of working with AGIRPF staff? Please give an example What could be improved about working with AGIRPF staff? Please give examples. c. In your opinion, what could have been done to improve the support given to you by AGIRPF? Probe on coordination, communication, preparation, follow-up support, etc. 17. The AgirPF project’s main objective is to increase access to and use of quality FP services. It proposes doing so through 3 main results: a. Expanded delivery of FP products and services b. Use of evidence-based service delivery approaches (such as FP outreach and training community health workers) c. Efforts to remove policy barriers and improve contraceptive security In your opinion, are the three intermediate results I listed sufficient to meeting AgirPF’s objective of increased FP use? If yes, please explain why these are sufficient. If no, please explain what additional program elements should be included to ensure reaching AgirPF’s objective. We have now come to the end of our discussion. 18. Would you like to add anything else about partnering with AGIRPF or issues in improving family planning services in your country? THANK YOU VERY MUCH FOR YOUR TIME 82 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation KEY INFORMANT GUIDE: ADVOCACY PARTNERS Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of institution: ___________________________________________________ Sex of informant: Male [___] Female [___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and Evidence for Development working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at decreasing barrier to family planning support in your country. In particular, we would like to know how the AGIRPF project has helped advocacy groups in your country plan policy-related actions. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in family planning policy. 42. What is your professional qualification? 43. In what capacity do you currently work in health care policy? 44. How long have you worked in this role? 83 45. What are your primary responsibilities? B. General views about health systems issues in family planning policy: Now I would like to ask you about issues related to decreasing policy barriers to family planning. 46. What are the main challenges to removing policy barriers to family planning in this country? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 47. What types of support would help your organization better address policy issues around family planning? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support? C. Family planning policy support with AGIRPF Now I would like to hear about your experiences receiving specialized training or support in family planning policy from the AGIRPF project. 48. During your time in this role, what types of support have you received to improve family planning policy? Probe to obtain multiple responses. 49. Have you received any orientations from or participated in any events by AGIRPF related to improving family planning policy in your country? i. If yes what was the approximate date of event/orientation? Please also describe the format of the event (Probe for: on-site, workshop, number of days, etc) j. Please describe the topics on which you were oriented/planned. Probe on the content of the event; for each topic, ask for details on the training provided. 50. What was the final result of this orientation/event? Probe on what was learned, networks created, connections made, action plans, etc. a. Did you create a policy action plan as a result of your work with AGIRPF? b. If so, in what ways did the action plan support your policy work? 51. Please explain the ways in which these orientations/events supported your efforts to remove family planning policy barriers in your country. Probe on advances they have made in policy advocacy since AGIRPF events 52. Were there any topics on which you wish you had received additional support from AGIRPF? Probe to obtain multiple answers. For each subject, probe on details of support they would need 53. Please describe any family planning policy/guideline/norm changes that have taken place as a result of AGIRPF’s support. a. Probe for details on each change mentioned. b. For each change mentioned, please probe on the exact role that AGIRPF played in facilitating the change 84 54. Overall, what were your impressions of this AGIRPF’s support? Probe on both successes and challenges of the support a. Did you feel that you were given adequate time and information on the topics covered? If not, on what topic would you have liked to have more time? b. What are your impressions of the AGIRPF staff? (Probe: Were they knowledgeable and helpful? Or could have done better? If so, in what ways?) c. In your opinion, what could have been done to improve the support given to you by AGIRPF? Probe on coordination, communication, preparation, follow-up support, etc. Now, I would like to ask you about the conceptual design of the AgirPF project 19. The AgirPF project’s main objective is to increase access to and use of quality FP services. It proposes doing so through 3 main results: a. Expanded delivery of FP products and services b. Use of evidence-based service delivery approaches (such as FP outreach and training community health workers) c. Efforts to remove policy barriers and improve contraceptive security In your opinion, are the three intermediate results I listed sufficient to meeting AgirPF’s objective of increased FP use? If yes, please explain why these are sufficient. If no, please explain what additional program elements should be included to ensure reaching AgirPF’s objective. We have now come to the end of our discussion. 55. Would you like to add anything else about AGIRPF or issues in improving family planning policy? THANK YOU VERY MUCH FOR YOUR TIME 85 EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation KEY INFORMANT GUIDE: AGIRPF STAFF Date of interview: [___|___ /___|___/__2_|_0_|__/__ ] Time of interview: Start [___|___:___|___] End [___ |___ :___|___] Venue of interview: ___________________________________________________ Name of institution: ___________________________________________________ Sex of informant: Male [___] Female [___] INTRODUCTION Good morning/afternoon. Thank you for taking the time to speak with me today. My name is __________, and I’m here on behalf of USAID and Evidence for Development working on an evaluation of the AGIRPF project. We are conducting a series of discussions to learn about the performance of the AGIRPF project and understand the successes and challenges of its programs aimed at decreasing barrier to family planning support in your country. In particular, we would like to learn about your experience working for the AGIRPF project. Participation in the interview is completely voluntary and you don’t have to answer any question you don’t want to answer. None of your responses will be shared with anyone outside of the research team, especially your supervisors or colleagues. I will also be tape recording the discussion for future reference but will not be identifying the speakers on the tape. This interview should take no more than 30-45 minutes. A. Qualifications and experience working in health care: I’ll start by asking you about your qualifications and experiences working in family planning. 56. What is your professional qualification? 57. In what capacity do you currently work in AGIRPF? 58. How long have you worked in this role? 59. What are your primary responsibilities? 86 B. General views about health systems issues in family planning: Now I would like to ask you about issues related to providing family planning in this country. 20. What are the main challenges to providing family planning services in this country? Probe to obtain multiple answers. For each subject, probe on details of why/how it is a challenge 21. What types of support does AGIRPF provide to improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support 22. What types of additional support is needed to improve family planning services? Probe to obtain multiple answers. For each type of support, probe on: a. In your opinion, why does this support not currently exist? b. In your opinion, what could be done to provide this support? C. Working at AGIRPF Now I would like to hear about your experiences working at AGIRPF to improve family planning services in your country. 23. During your time in this role, what was the nature of AGIRPF’s work in family planning? Probe on what activities they did, what type of support AGIRPF provided, what specific family planning outcomes they pursued, etc. 24. Now I’m going to ask you about specific high-impact practices pursued by AgirPF 87 During your time here, did AgirPF provide support in…? What were successes of implementation? What were challenges of implementation? How did AgirPF ensure correct/ consistent application of this HIP? How did AgirPF ensure institutionalization of this HIP? What more could have been done? Post-abortion FP YES NO Adolescent FP YES NO Reducing policy barriers to FP YES NO 88 During your time here, did AgirPF provide support in…? What were successes of implementation? What were challenges of implementation? How did AgirPF ensure correct/ consistent application of this HIP? How did AgirPF ensure institutionalization of this HIP? What more could have been done? Community Health Worker training YES NO Supply chain/ logistics management YES NO Mobile outreach (FP days) YES NO 89 25. Now I’m going to ask you about specific AgirPF tools that you implemented During your time here, did AgirPF provide support in…? What were successes of implementation? What were challenges of implementation? How did AgirPF ensure correct/ consistent application of this tool? How did AgirPF ensure institutionalization of this tool? What more could have been done? Reality check YES NO AgirPF Health Information System YES NO COPE for contraceptive security YES NO 90 During your time here, did AgirPF provide support in…? What were successes of implementation? What were challenges of implementation? How did AgirPF ensure correct/ consistent application of this tool? How did AgirPF ensure institutionalization of this tool? What more could have been done? OCAT YES NO Spectrum YES NO Environmental Compliance YES NO 91 During your time here, did AgirPF provide support in…? What were successes of implementation? What were challenges of implementation? How did AgirPF ensure correct/ consistent application of this tool? How did AgirPF ensure institutionalization of this tool? What more could have been done? Other? 92 26. During your time at AGIRPF, please describe the relationship of the project to local government entities. Probe on the work they did together, success and challenges of that relationship. 27. During your time at AGIRPF, please describe the relationship of the project to local and international consortium partners in the country. Probe on the work they did together, success and challenges of that relationship. 28. During your time at AGIRPF, please describe the relationship of the project to local advocacy group. Probe on the work they did together, success and challenges of that relationship. C. Challenges in meeting AgirPF performance targets At this time, we would like to learn more about the challenges in meeting specific AgirPF performance targets. 29. What were the challenges in meeting targets for new family planning users? Probe on technical, administrative or operational challenges 30. What were the challenges in meeting targets for total family planning users? Probe on technical, administrative or operational challenges 31. What were the challenges in meeting training targets? Probe on technical, administrative or operational challenges 32. What were the challenges in meeting targets for FP outreach days? Probe on technical, administrative or operational challenges 33. Were there any other programs aims/targets that were challenging to meet? If so, which ones? Why was it challenging? 34. The AgirPF project’s main objective is to increase access to and use of quality FP services. It proposes doing so through 3 main results: a. Expanded delivery of FP products and services b. Use of evidence-based service delivery approaches (such as FP outreach and training community health workers) c. Efforts to remove policy barriers and improve contraceptive security In your opinion, are the three intermediate results I listed sufficient to meeting AgirPF’s objective of increased FP use? 93 If yes, please explain why these are sufficient. If no, please explain what additional program elements should be included to ensure reaching AgirPF’s objective. D. Successes and challenges of AGIRPF management Now I would like to hear your opinions on success and challenges related to AGIRPF management 35. Please explain the ways in which AGIRPF regional management supported your efforts to improve family planning services in your country. 36. Were there any areas in which you wish you had received additional support from AGIRPF regional office?? Probe to obtain multiple answers. For each subject, probe on details of support they would need 37. Please explain the ways in which EngenderHealth headquarters management supported your efforts to improve family planning services in your country. 38. Were there any areas in which you wish you had received additional support from EngenderHealth headquarters? Probe to obtain multiple answers. For each subject, probe on details of support they would need 39. In your opinion, what could have been done to improve the support given to staff at AGIRPF? Probe on coordination, communication, preparation, follow-up support, etc. We have now come to the end of our discussion. 40. Would you like to add anything else about partnering with AGIRPF or issues in improving family planning services in your country? THANK YOU VERY MUCH FOR YOUR TIME 94 Facility Service Statistics Data Extraction Sheet FACILITY IDENTIFICATION Name of the City: ______________________________ Name of District: ______________________________ Name of the facility_____________________________ Tel: _________________________________________ Type of Health Facility : 1 = Teaching Hospital 2 = Regional hospital 3 = District Hospital; 4 = Other Hospital 5 =Polyclinic 6 = Health Center 7 =Maternity Home 8 = Health post 9= Other ________________ Operating Authority: 1 = Public; 2 = Private for profit 3 = NGO 4 = Faith based 5 = Other (specify)_____________ Date: / / DAY / MONTH / YEAR Name of the interviewer______________________________ Name of Facility/ District Supervisor_________________________________ 95 INFECTION PREVENTION AND ENVIRONMENTAL MONITORING AND MITIGATION PLAN NO QUESTIONS CODING CLASSIFICATION GO TO 1101 What system do you have for solid infectious waste disposal? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected and disposed externally….…1 Burned in incinerator……………………2 Burned in open pit………………………3 Burned and buried………………………4 Buried…………………………………….5 Put in trash/open pit…………………….6 Put in pit latrine………………………….7 Other (specify)__________________ 8 Don’t Know…………………………….88 1102 What system do you have for liquid infectious waste disposal? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected in septic tanks…………..…1 Collected by municipality…………..….2 Facility connected to the sewage system………………………...3 Collected in container and burned…….4 Dispose in pit……………………………5 Put in trash/open pit……………………6 Flush down toilet………………………..7 Other (specify)_______________________ 8 Don’t Know…………………………….88 1103 How does this facility dispose of items such as syringes and bandages that may be contaminated? Interviewer: PROBE IF NECESSARY; CIRCLE ONE ONLY. Collected and disposed externally. …1 Burned in incinerator........................2 Burned in open pit…………………..3 Burned and buried…………………..4 Put in trash/open pit…………………5 Put in pit latrine………………………6 Other (specify)________________7 Don’t Know…………………………88 1104 Does the health facility/NGO have the national and international policies and protocols for the collection and processing of recyclable waste and handling and treatment / disposal of other waste? Yes……………………………………1 No……………………………………..0 Don’t Know……………………….….8 1104a If yes, ask to see them Observed…………………………….1 Not observed……..………….………2 1105 How many health facility/NGO staff are sensitized on how to avoid an adverse 96 NO QUESTIONS CODING CLASSIFICATION GO TO environmental impact from activities conducted in the workplace Number ____________ 1106 Are the facility training curricula revised to include training in infection prevention for service providers including medical waste management and environmental protection? Yes……………………………………1 No……………………………………0 Don’t Know………………………….8 1107 Has the facility conducted clinical training for services providers in IP including medical waste management and environmental protection? Yes……………………………………1 No……………………………………..0 Don’t Know………………………..…8 1108 If yes, how many staff members received this training? Number of staff ____________ 1109 How many copies of MOH medical waste management and environmental protection procedures were distributed to the health facility? Number of copies__________ 1110 Is the facility equipped with IP materials? Yes……………………………………1 No……………………………………..0 Don’t Know…………………..………8 1111 How many sterilizers does the Health Facility have? 1112 How many incinerators does the Health Facility have? 1113 If no incinerator, ask if the Health Facility has a double pit? Yes………………………………..….1 No…………………………………….0 1114 How many times in the last six months has a supervisor come to this facility to supervise trained providers in Infection protection including medical waste management and environmental protection? Number: ____________ Post-Evaluation Action Review Table: AgirPF Performance Evaluation July 2017 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Implementation Implementation Status Focus on “quick wins” through FPSDs and mobile services – While participants appreciated a multi-faceted approach to long-term FP improvement, many mentioned the importance of focusing efforts on providing immediate benefits to target populations. Namely, the use of FP special days and mobile outreach to provide methods for those in need was viewed as one of the most successful aspects of AgirPF’s work and should be replicated/promoted throughout the focus countries. Yes AgirPF May 2018 Ongoing Improve finance and accounting structures – Both AgirPF regional/country staff and their counterparts’ in country mentioned the cumbersome nature of the project current financing and accounting procedures. The number and nature of approvals needed for purchase orders and disbursement were seen as an impediment to rapid response and action on the ground. Future projects should streamline this process to enable more nimble and reactive capabilities. No Sub-awards require a long process of responsibility determination. Most local NGO lack financial capacity to speed up the process. Provide adequate project technical and administrative staff – For a project of this size and complexity, it is essential that the staffing be sufficient and in place as soon as possible after agreement signing. The AgirPF project struggled to staff up in the first year of its program, which led to significant delays in rollout of activities. Furthermore, even when No Initially the staffing was meant to be leaner to ensure EngerderHealth staff do not engage in work that is inherently the responsibility of government. However, later in the implementation AgirPF staff were increased, both in country and regional staff felt overextended in their work. Partners noted an admirable work ethic and commitment by AgirPF staff, even so there were some participants that felt there were delays in communication due to lack of staff availability. added at least two staff members per country to improve monitoring and support supervision of providers. Any staffing gap should have been filled using creative approaches such as short term technical assistance for carrying -out specific, periodic task. Continue to promote and propagate high -impact practices – The use of high - impact practices was viewed as a success by many of the participants and should be continued in future programming. The use of forums like the one on Good Practices organized by WAHO, where HIPs were tested in each country and lessons shared in a regional format, are of particular benefit to aiding the scale -up of best practices across the region. The HIPs that were most notable in this project were FPSDs, community engagement through CHWs, integration of family planning into PAC and the postpartum period and engagement of religious leaders for policy change. Yes AgirPF May 2018 Ongoing Continue to push for policy change – Though the gains in policy advocacy are less immediate and tangible than other areas of AgirPF’s work, both civil society and MOH partners view this as a critically important effort that must continue in order to have large -scale Yes AgirPF May 2018 The roadmap for the implementation of the parliamentarian declaration is under way. impact on FP service provision. In particular, countries should continue to engage important gatekeepers like religious leaders to demonstrate the urgency of responsible childbearing to the population’s survival. Future projects should also continue to push for inclusion of vulnerable and neglected groups like youth. Capitalize on positive achievements to encourage buy -in at national level - Overall, almost all of the participants the evaluation team interviewed viewed the work of AgirPF as critical and fruitful. As such, they felt that the successes of the project should be clearly distilled and promoted at the national level in order to encourage government buy -in and continuation of activities beyond the life of the project Yes AgirPF June 2018 Dissemination, learning and transition workshops to be conducted for the close of the AgirPF. Take into account the need for infrastructure improvements – Across all countries, actors noted the need to provide structural improvements to facilities in order to be able to provide adequate FP service improvements. These included mainly areas for private conversations and rooms for method insertion (for IUDs and implants). Some participants also noted the need for other basic material support such as locked cabinets, shelves, chairs, etc. Without critical basic functions such as these, Yes Amplify -FP By the end of implementation Year 1 Amplify -FP’s statement of work includes light infrastructure renovation. many facilities will remain incapable of providing access to adequate family planning services. Improve provider working conditions – Though beyond the scope of most projects of this nature, it is worth noting that several participants in this study felt that provider training alone was not sufficient to ensure their engagement and promotion of FP services. Many of these providers are already facing high work burdens, and integrating new/improved services in their work should be joined by remuneration or additional recognition to improve provider motivation. Creative solutions to enduring human resources for health issues must be incorporated into future models of change in this region. No No provision was made for other forms of provider incentives apart from capacity building. Devolve provider training to site-level and/or provide more regular FP training – Due to high staff turnover at facilities, many AgirPF trained staff had already been transferred from the facilities that the evaluation team visited. This is a common issue in countries, where human resources are scarce. Thus, devolving provider training to on-the-job or on-site formats and/or providing regular training to facility staff can help ensure continuity and presence of trained Yes Amplify-FP Year 1 into Year 2 of implementation Amplify-FP will utilize approaches such as whole site training to ensure training opportunities for more staff on site. FP providers. Institutionalize provider training in FP – Another suggestion made for improving and ensuring provider readiness for FP service provision was to institutionalize FP training into existing training curricula at the national level. This way, a foundation for knowledge on FP service provision can laid for each provider as part of their routine training. AgirPF began this work by creating national centers of excellence and providing training of trainers activities in each country. However, these activities must be maintained diligently and ownership transferred successfully to national MOHs to ensure sustainability. No Institutionalizing family planning training into existing training curricula is not within USAID/West Africa’s manageable interest. WAHO is well placed to effect this and shall be engaged. Increase supervisory visits – Several providers noted the benefits of facilitative supervision visits and requested that such visits be more frequent. Through increased supervisory visits, programs can ensure trained providers are cementing their FP services skills and can further identify gaps at the facility level if/when trained providers are relocated. Yes AgirPF May 2018 AgirPF has pursued a model of peer support supervision from a trained pool within the district. Provide continuity in future programming to continue gains made – As of the time of this evaluation, participants and AgirPF staff felt that the program was just getting into full swing. Future projects should carry forward the successful aspects of this program through a seamless transition to further funding and programming so as not to disrupt the potential gains stemming from current activities. Yes Amplify-FP Year 1 into Year 2 of implementation Amplify-FP is designed with several lessons learned from AgirPF implementation. More partnerships and leveraging of local and regional relationships and resources. Some interventions are critical for the success of the process but are not eligible to the project funding (e.g. renovation of consultation rooms, equipment, and construction of incinators, etc.). The project should be implemented in health facilities either where such basic needs are already met, or where investments for structural improvements can be made by the local government or by other partners. No See response to recommendation above on infrastructure. Future projects should include a strong and well-staffed knowledge management unit from the very start, to enable the efficient capture and dissemination of program learning throughout the life of the project. A KM unit at the start of the project. Yes Amplify-FP Year 1 into Year 2 of implementation Amplify-FP will draw lessons from AgirPF’s not so successful Knowledge Management activities. While it is important to maintain a regional presence to coordinate across ECOWAS countries and engage with regional actors such as WAHO, more decentralization of the regional offices into local country offices is necessary. At present, the AgirPF staff feel there is under -representation in each intervention area in -country. No Refer to the staffing recommendation above. Although this specific project was focused on high -density urban areas and their unique challenges, a more comprehensive contribution equitable access and utilization of family planning services can only take place when the needs of the most vulnerable rural, populations are also addressed. No AgirPF was designed for urban and peri -urban settings. Other stakeholders were expected to cover other geographic areas. Strengthening and expanding the supply of reproductive health / FP services for key and vulnerable populations and people with disabilities in settings where they are located. Yes AgirPF aimed to be disability inclusive but failed to carry -out activities that targeted the disabled. AgirPF will continue to work with female sex workers in hotspots as part of HIV/FP integration. Amplify -FP will continue HIV/FP integration activities including providing family planning services to female sex workers. E4D revised SOWAgirPF Performance Evaluation I. PURPOSE OF THE EVALUATION The purpose of the Performance Evaluation for AgirPF Project is to increase learning about the performance of the AgirPF activity in Togo, Côte d’Ivoire, Niger and Burkina Faso (in order of priority). The USAID/WA health office wants to know and document whether the AgirPF project is on track for achieving its intended results; has advanced select high impact practices; if its intermediate results are necessary and sufficient to achieve expected results and what the activity’s key successes, challenges and lessons learned are. It serves as a performance evaluation of AgirPF to determine the extent to which the AgirPF portfolio has met its overarching objectives of: (1) Strengthening partners to implement evidence-based approaches and deliver quality family planning (FP) services; (2) Enhancing efficiency and effectiveness of FP delivery approaches through the adaptation and implementation process; (3) Identifying operational policy barriers and new/revised policies adopted and implemented; (4) Supporting local leaders, civil society, service providers, municipal government in promoting FP; (5) Documenting and disseminating lessons learned from adaption and implementation processes and experiences; and (6) Identifying and coordinating contraceptive commodity needs among partners and country commodity security and logistics management committees. This evaluation will complement any evaluation efforts already implemented as part of the AgirPF project’s PMP. The target audiences for the AgirPF performance evaluation are the USAID/WA Front Office; USAID/WA Regional Health Office (RHO); other USAID health offices in the region, the Governments of Togo, Côte d’Ivoire, Niger and Burkina Faso Ministries of Health, the implementing partner EngenderHealth and other donors in the health sector as well as stakeholders in family planning and reproductive health in West Africa II. AGIR-PF SUMMARY INFORMATION Agir-PF is a five-year, five-country, $29 million activity launched in July 2013. The period of performance to be evaluated is July 2013 to December 2016. Summary of the AgirPF Project Activity/Project Name Agir pour la Planification Familiale (AgirPF) Implementer EngenderHealth Cooperative Agreement/Contract # AID-624-A-13-00004 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $ 29,000,000 Life of Project/Activity July 2013 - July 2018 Active Geographic Regions Burkina Faso, Côte d’Ivoire, Niger, Togo and Mauritania (no evaluation to be conducted in this country. Development Objective(s) (DOs) Utilization of Quality Health Services Increased through West African Partners USAID Office USAID/WA Regional Health Office STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 2 III. BACKGROUND A. Context and Description of the Problem, The West Africa region includes 21 countries with a population of approximately 359 million (Population Reference Bureau (PRB), 2016). The region has the lowest modern contraception use worldwide. Only 12% of married women of reproductive health (WRH) age were using a modern contraceptive method in 2016 compared to the global average of 56% (PRB, 2016). Consequently, the region has the highest total fertility rate (TFR) estimated` at 5.4 children per woman (PRB, 2016). This high level of fertility, among which 26% are unintended (Sedgh, Singh, &Hussain, 2014), coupled with high adolescent fertility (111 births per 1000 women ages 15-19 compared with 52 worldwide) and persistent decrease in child and maternal mortality rates, contribute to high population growth rates. The West Africa population is expected to increase from 359 million in 2016 to 515 million by mid￾2030 and 800 million by mid-2050 (PRB, 2016 ). Such population volumes constitute a threat for the future of the region (available resources, economic growth and population wellbeing). Therefore, researchers, policy makers and non-government organizations (NGO) advocated that the promotion of FP and ensuring access to preferred contraceptive methods for women and couples is essential to securing the well-being of children, women and families as well as the health and development of communities. In response to the high level of unmet need in Francophone West Africa, nine governments of Francophone West African countries and their technical partners and financial resources governments launched the Ouagadougou Partnership in February 2011 in Ouagadougou, Burkina Faso. This initiative includes the government of Benin, Burkina Faso, Côte d’Ivoire, Guinea, Mali, Mauritania, Niger, Senegal and Togo. The main objective of the Partnership is to reach at least 2.2 million additional FP method users in the nine countries by 2020. The national action plans of the nine countries encompass two objectives: 1) accelerate the achievement of their national goals for modern Contraceptive Prevalence Rate (CPR); and 2) reach at least an additional 1 million women by 2015. These action plans mapped their priority steps for strengthening national FP programs. In West Africa, compared to Anglophone and Lusophone countries, the Francophone countries have the highest rates of maternal and child mortality, the highest fertility rates, and lowest contraceptive prevalence rate. Against this backdrop, USAID/West Africa RHO funded the AgirPF. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. Indeed, though, overall women living in urban areas experience better reproductive health indicators (low fertility and unmet needs for FP and high modern contraception use), analysis of DHS data (2003 to 2007) from 26 Sub-Saharan countries shows that, women living in poorest households and slums areas in Sub-Saharan Africa are, on average, two and a half times less likely to use any contraceptive method (Ezeh et al, 2010). USAID/WA tasked the Evidence for Development (E4D) project to conduct an independent performance evaluation of the AgirPF activity to compare the achievements against targeted objectives over the evaluation period (July 2013-March 2016). The Evidence for Development (E4D) Project is a five-year activity with the overall objective of increasing the availability of evidence in health interventions to inform policy advocacy and program planning, including resource allocation. E4D is funded and directly supported by the United States Agency for International Development/West Africa (USAID/WA) Regional Health Office (RHO) with a focus on family planning (FP) and Human STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 3 Immunodeficiency Virus/Acquired Immune Deficiency Syndrome (HIV/AIDS). The activity addresses critical gaps in the research and evidence for program implementation for USAID/WA/RHO’s FP and HIV/AIDS programs. Capacity building (CB) activities are integrated throughout the life of E4D to build a cadre of personnel and institutions capable of conducting rigorous operations research (OR) and evaluation studies. E4D is available for all 21 countries supported by USAID/WA with six countries targeted for direct support, namely Burkina Faso, Cameroon, Cote d’Ivoire, Mauritania, Niger, and Togo. To achieve the overall objective, there are three intermediate results (IRs): (1) Best and promising practices in health tested and documented; (2) Capacity of regional and local institutions to implement operations research and evaluation strengthened; (3) Research and evaluation findings disseminated locally, regionally, and internationally. B. Development Hypothesis(es), and Theory of Change The theory of change for the AgirPF activity is that if a) the delivery of quality FP information, products, and services are strengthened and expanded; b) evidence-based service delivery approaches selected, adapted, and implemented; and c) efforts to remove policy barriers and improve contraceptive commodity security coordinated, then access to and use of FP services will increase in urban and peri-urban areas in Burkina Faso, Côte d’Ivoire, Niger and Togo. Mauritania will not be part of this exercise. C. Results Framework The results framework (schematic of theory of change) for activities carried out by the AgirPF activity is shown in the diagram below. PFAR￾s AGIR-PF Objective Increase access to and use of quality family planning services in select urban and peri-urban areas of four francophone West African countries AGIR-PF Result 1: Delivery of quality family planning information, products, and services strengthened and expanded Sub-Result 1.1 Partners strengthened to implement evidence-based approaches and deliver quality family planning services Sub-Result 1.2 Local leaders, civil society, service providers, municipal government support and promote family planning AGIR-PF Result 2: Evidence-based service delivery approaches selected, adapted, and implemented Sub-Result 2.1 Efficiency and effectiveness of approaches enhanced through the adaptation and implementation process Sub-Result 2.2 Lessons documented and disseminated on learning from adaption and implementation processes and experiences AGIR-PF Result 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-Result 3.1 Operational policy barriers identified and new/revised policies adopted and implemented Sub-Result 3.2 Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 4 D. Summary Activity/Project/Program to be evaluated On July 5, 2013, the USAID/West Africa Regional Health Office (USAID/WA RHO) awarded to EngenderHealth and two sub-partners, Futures Institute (now Avenir Health) and Camber Collective, a five-year cooperative agreement to implement the AgirPF Project. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. The project works closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP. The approach is to leverage FP momentum, activating the “grassroots” to increase access to, quality of, and demand for FP, and working with the RHO and countries to adapt evidence￾based practices (Result 2); learn about these practices (Sub-Result 2.2); feedback learning to national actors in the form of project/RHO advocacy for adoption and scale-up, grassroots-led advocacy, and information that USAID can use to rationalize policies and contraceptive logistics (Result 3). AgirPF strengthens public, private, and NGO facilities to provide a range of FP services (Result1), including integrated FP/maternal health services and services for youth/men (Sub-Result 1.1). To improve quality, AgirPF uses Client-Oriented, Provider-Efficient Services (COPE®) and facilitative supervision and other tools to improve and promote quality, supporting training, with Centers of Excellence in each capital city (Sub-Result 1.1).To bring FP services to underserved communities, AgirPF supports mobile outreach services; brings health fairs to industries and community sites; and offers “city-based services,” an adaptation of EngenderHealth-managed Community-based Distribution(CBD) in Togo. To lower client cost, AgirPF provides dedicated FP services at low/no cost (special FP days) in each city. To solve logistics issues and estimate commodity needs, AgirPF assists facilities to use COPE for Contraceptive Security and ministries of Health’s Contraceptive Procurement Teams to use Reality Check for contraceptive quantification (Sub-Result 3.2). AgirPF also provides training and refresher trainings to healthcare providers in FP service delivery, including infection prevention (IP), FP counseling (using the rapport building, exploration, decision making, and implementing the decision (REDI) framework), facilitative supervision and gender sensitization. AgirPF supports community leaders with FP advocacy activities. Summary of AgirPF Indicators AgirPF tracks a total of 25 output and outcome indicators with life of the projects targets set and disaggregated annually. Results against indicator targets are reported quarterly and annually, with a few reported biannually. The entire M&E table reporting all 25 indicators is attached in the annex. Furthermore, EngenderHealth has made many documents available to E4D, including the AgirPF baseline assessment and initial projects documents (e.g., project description and SOW, PMP, indicator definitions, etc.). Quarterly and annual reports, PMP and other reports from the AgirPF activity start date to present will be made available to evaluators. IV. EVALUATION QUESTIONS The evaluation team will review and revise the list of the following illustrative evaluation questions to inform data collection tools: STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 5 1. How has AgirPF performed (analysis of monitoring data): a. Against reaching top line indicators of 700,000 method adopters, yielding 1,683,000 Couple years of protection (CYP)? Is the Project on track to reach its targets? [CYP Target]. b. How has AgirPF performed against other selected indicators to date (from the PMP)? (Specifically, AgirPF indicators 11, 16, 17, 19, 20) [Composite indicators targets]. These indicators are: (1) number of HIV positive women who received comprehensive FP services [Indicator 11]; (2) number of Community Health Workers supported and supervised [Indicator 16]; (3) number of youth who participated in educational program on gender, Family Planning (FP) and Sexual and Reproductive Health activities (SRH) [Indicator 17]; (4) number of Best Practices (BPs)/ High Impact Practices (HIPs) for family planning and maternal and child health and/or HIV/AIDS incorporated into local, district or national health protocols or standards [Indicator 19]; and (5) Number of best practices piloted through operations research studies [Indicator 20]. 2. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services]. Consider the following:  What policies, norms, guidelines, protocols, etc. related to the selected HIPs have been advanced?  To what extent have these HIPs been scaled up1 in AgirPF focus countries?  To what extent have these HIPs contributed to AgirPF’s results?  What AgirPF has done to facilitate replication of these HIPs within the country and in other countries?  How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? [Enabling environment]. Proposed indicators include: number of policies, national health standards and guidelines developed or changed, including scale-up [Indicator 21]; and number of policies or guidelines developed or changed with USG assistance to improve access to and use of family planning and reproductive health services [Indicator 22]. 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? [Successes][Challenges] [lessons learned] 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? V. EVALUATION DESIGN AND METHODOLOGY Geographic Focus Data collection will be carried out in the AgirPF implementation cities of Togo (Lome, Sokode, Kara), Côte d’Ivoire (Abidjan1, Grand Ponts, Abidjan 2), Niger (Niamey, Maradi) and Burkina Faso 1 Scaling up is defined as “expanding, replicating, adapting and sustaining successful policies, programs or projects in geographic space and over time to reach a greater number of people.” STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 6 (Ouagadougou, Koudougou, Bobo-Dioulasso). Within those cities, AgirPF has more than 265 intervention and 141 comparison sites from which the evaluators will draw a sample for field work. AgirPF will make available to E4D a comprehensive list of sites and catchment population. The evaluation team will explore if USAID/WA Mission has any particular needs, preferences or priority sites. Technical Requirements Proposed Evaluation Design and Methodology This performance evaluation will use quantitative and qualitative methods, primary and secondary data. The evaluation will meet the criteria of a quality evaluation as defined by USAID. The quantitative data include information from AgirPF baseline, the AgirPF performance management plan, quarterly and annual reports, other programme’s reports and national as well as local surveys such as the Demographic and Health Surveys (DHS). In addition, the evaluation team will use data from Lomé Operations Research on Family Planning E4D conducted in 2016 as well as quantitative data from intervention and comparison sites in the other cities and countries to answer the evaluation questions. These data will allow measuring changes in key outputs and outcomes over the evaluation period (Questions 1a, 1b, 3 and 4). Qualitative data will encompass information from key informant interviews (KII). and focus group discussions (FGD). On average, the team will conduct about 30-40 KIIs and about 10 FGDs per countries. The evaluation team will use information from qualitative data to understand trends in the selected indicators as well as the Project’s implementation success and challenges. The Evaluation team expects to conduct three (3) to four (4) KIIs and two (2) FGDs per day. The team will review programmatic data onsite. The evaluation team is to propose a more detailed and refined evaluation design, as well as complete the design matrix provided below in Table 3. There will be four (4) different evaluation teams for the four countries, one team per country. An overall Evaluation Team Leader will develop the Inception Report, including the methodology and study implementation plan, data collection tools and data processing as well as data quality assurance strategies. The Team Leader will train all the country teams leaders during a workshop, planned in Accra/Ghana or eventually in Lome, Togo, (depending on the discussions with USAID/WA, AgirPF and the RG’s). The expectation is that the training will include a pre-test of the instruments so that the evaluation leaders go back to their countries with a clear sense of what they need to replicate. Data collection tools may be adapted to county-specific contexts, as needed. The country to do the pretest would be Togo and the team leader will actually conduct data collection as part of that exercise. The Team Leader is in charge of the overall management of the evaluation team in each country, the evaluation team will encompass One subject matter expert/Field Coordinator supervisors and data collectors. to conduct E4D will select the country teams from Recipient Groups. Immediately following approval of the SOW by USAID/WA, the proposed Evaluation Team Leader will conduct an initial desk review and develop an overlapping evaluation approach. The desk review will take place over two weeks once project documents become available, provided that the Team Leader is on board. The review will help generate the summary of findings from existing documents (project reports, national policy documents, grey literature), identify gaps in available data and inform the methodology. As USAID reviews the submitted document, the Evaluation Team leader will start working on the inception report and prepare the evaluation materials with support from E4D Research and Evaluation team and Field coordinators in respective countries. Then E4D will organize a workshop in Lome with the entire Evaluation technical team (Team leader, Field coordinators, and E4D research and evaluation team, E4D COP). The main objective of the workshop will be to finalize the Inception Report. The team will review and finalize the evaluation methodology; the general STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 7 evaluation timeline (and road map), the data collection tools and the data analysis plan. Following the development of the first draft of the Inception Report, the entire team will meet with USAID/WA (E4D COR and AgirPF AOR) to present the technical methodology and the proposed tools. The evaluation team will collect USAID/WA initial feedback and general orientation. The Team Leader will integrate the feedback and submit the “final Inception Report”. The final Inception Report (with annexes) will be submitted in English to USAID/WA; and the mission will have one week to review the document. The Team Leader and the Field Coordinators will translate the Inception Report whereas USAID/WA is reviewing the final inception report, (including the data collection tools/instruments) in French. The Country Fields coordinators will organize a training workshop to ensure that the field staff understand the concept of evaluation, methodology and data collection tools. These trainings will happen over not more than five (5) days after the Field Coordinators return home. The Evaluation Team Leader will provide general guidance regarding the format and approach to be adopted for the trainings/meetings. USAID Mission/Embassy in each country will be engaged in these meetings with the USAID/WA prior approval. The E4D COP and the Team Leader will debrief USAID/WA RHO at midpoint and the end of the data collection period. Country evaluation teams will work from their home base on data analysis and report writing. Each country team will do data analysis and prepare the report under the guidance of the Evaluation Team Leader. The Evaluation Team Leader will develop the introduction and synthesis of country results of the final report, and will draw any cross-country comparisons and draft the conclusions section of the evaluation report. The Evaluation Team Leader will be responsible for drafting the methodology section of the report. E4D will submit the first draft of the report for USAID/WA comments. USAID/WA may have up to two weeks for the review of the draft report and will integrate possible inputs provided from different Missions. The Evaluation Team Leader and country teams will have up to one week to make modifications. E4D will have up to one week to review, edit and format the final version of this report. Once the report is finalized, a dissemination meeting among stakeholders at the country level and at the USAID/WA office will be organized to disseminate findings of the evaluation. The Evaluation Team Leader will eventually travel to Accra to present major findings at the dissemination meeting for USAID/WA. Data Analysis Plan Prior to the start of data collection, the evaluation team will present a data analysis plan to USAID/WA for review. The plan should describe how data will be transcribed and analyzed. It should also describe how qualitative and quantitative data will be integrated to reach final conclusions and recommendations. The evaluation team must have its own professional qualitative data analysis software and any other analytical tools required to meet project deliverables. The data analysis procedures must be included in the final report. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 8 Data Disaggregation Data collected should be disaggregated by age and sex, where appropriate. Table 2: Evaluation Design Matrix Questions Indicators Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1a. How has AgirPF performed against reaching top line indicators of 700,000 method adopters, yielding 1,683,000 Couple years of protection (CYP)? Are they on track to reach their targets? [CYP Target] Number of CYP Number and % of new FP users Modern contraception prevalence (MCP) Project documents (including. performance monitoring data, quarterly/annual reports previous evaluations, etc.), HIS data in focus area, national statistics, project staff AgirPF baseline data 2016 E4D Operations research data in Lomé Desk review, Evaluation of project results against approved PMP, key informant interviews Descriptive method, content analysis Disaggregation by country, city, gender, and FP method 1.b. How has AgirPF performed to date on other performance targets in their PMP? (Specifically AgirPF indicators 11, 16, 17, 19, 20.)[Composite indicators targets] Number of Community Health Workers supported and supervised; Number of HIV positive women who received comprehensive FP services; Number of youth who participated in educational program on gender, FP and SRH); Number of Best Practices (BPs)/ High Impact Practices (HIPs) for family planning and maternal and child health and/or HIV/AIDS incorporated into local, district or national health protocols or standards; Number of Project Documents (including. performance monitoring data, HIS data in focus area, national statistics, policies and guidelines) project staff, stakeholders, CHW, youth. Analysis of project results against approved PMP, Key informant interviews, focus group discussions, desk review. Descriptive method, content analysis Disaggregation by country, city, gender if possible, and FP method if applicable. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 9 Questions Indicators Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods best practices piloted through operations research studies. 2. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services]. Consider the following:  What policies, norms, guidelines, protocols, etc. related to the selected HIPs have been advanced  To what extent have these HIPs been scaled up in AgirPF focus countries?  To what extent have these HIPs contributed to AgirPF’s results?  What AgirPF has done to facilitate replication of these HIPs in other countries in the region?  How has AgirPF played a regional role for exchange, learning and dissemination of HIPs?” Number of policies, norms, guidelines, protocols, etc. related to the selected HIPs; Number and list of HIPs, which have been scaled up in AgirPF focus countries. Strategies put into in place for replication Number of workshops (including capacity building) and stakeholders’ meetings conducted for replication and scaling up selected HIPs. Project Documents (including. performance monitoring data, policies norms, and guidelines) project staff, stakeholders Key informant interviews,, desk review Descriptive method, content analysis Disaggregation by country, city. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 10 Questions Indicators Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective [Indicators reported in questions 1a, Ib, 4, 5 and 6]. Number of partners supported to implement evidence-based approaches and deliver quality FP services. Number of Local leaders, civil society, service providers, and municipal government supported to promote family planning activities; Number of formal agreements (MoU) that are signed Number of presentations, publications and workshops to disseminate lessons learned and documented on adaption processes and experiences Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees Projects Documents (including. performance monitoring data) project staff, stakeholders, FP users. Key informant interviews, desk review, Focus Group Discussion. Descriptive method, content analysis Disaggregation by country, city. 4.For AgirPF IR 3, to what extent has AgirPF contributed to removing policy barriers to FP access in the region [Enabling environment] Number of policies, national health standards and guidelines developed or changed, including scale-up ; and Number of policies or guidelines developed or changed with USG assistance to improve access to and use of family planning and reproductive health services Projects Documents (including. Performance monitoring data) project staff, stakeholders, Key informant interviews, questionnaires, desk review STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 11 Questions Indicators Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 5.What are the activity’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? [Successes][Challenges] [lessons learned] Indicators reported in questions Ia, Ib, Ic, 2, 3, 4 and 6. Project Documents (including. performance monitoring data, previous evaluations, etc.), HIS data in focus area, national statistics, project staff, stakeholders, expert knowledge, beneficiaries(women, youth), and CHW Desk review, Key informant interviews, questionnaires or surveys, focus group discussions, Descriptive method, content analysis Disaggregation by country, city, gender 6 How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Number of meeting per month Participation of partners in decision making Reporting timeline Documents (project documents, meetings minutes, emails communications), project staff: including field offices, management staff, consortium partners, stakeholders, Key informant interviews, questionnaires or surveys, focus group discussions, desk review Descriptive method, content analysis Disaggregation by country, city, gender STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 0 Data Quality Data quality must meet USAID’s five standards: Validity, Integrity, Precision, Reliability and Timeliness. The Evaluation team will describe in the Inception Report how they are planning to insure data quality at each step of the evaluation (inception report, data collection tools, data collection, data management and data analysis). The evaluation team will document challenges with collecting quality data in the project countries, how those challenges vary from country to country and how those challenges were mitigated Methodological Strengths and Limitations The evaluation methodology must state all methodological strengths and limitations explicitly in the evaluation Statement of Work (SOW), presentations and draft and final reports. VI. DELIVERABLES AND REPORTING REQUIREMENTS A. Evaluation Implementation Plan: Within one week of onboarding the Evaluation Team Leader a draft work plan that includes all phases and deliverables for the evaluation shall be completed by E4D and presented to the E4D’s Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the competencies of the evaluation team, delineated by roles and responsibilities. B. Within two weeks of obtaining the approval of the Evaluation Work plan the general Integrated Desk Review will be completed and submitted to USAID for approval. The Integrated Desk Review will have a general section of conclusions and sections reflecting the different country reviews. It is assumed that USAID will require one week completing their review of this document. A final version of the Integrated Desk Review will be submitted within seven days of receiving USAID inputs. C. Inception Report: A written report summarizing what is known from routine performance monitoring reports and other project documents is due no later than two weeks after USAID approves the Integrated Desk Review Report. The inception report is also due before finalization of the evaluation design and should include the data collection instruments. The inception report will include the evaluation design (which will become an annex to the evaluation report). The evaluation design will include: 1. A detailed evaluation design matrix that links the evaluation questions in the SOW to data sources, methods, and the data analysis plan; 2. Questionnaires and other data collection instruments or their main features; 3. The list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); 4. Known limitations to the evaluation design; and 5. A dissemination plan. This includes stakeholders’ meeting before the publication of the report, and dissemination workshop to present key findings to relevant stakeholders STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 1 USAID/WA and relevant stakeholders will require 10 business days to review and consolidate comments on the evaluation design through the COR for E4D. Once E4D receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within seven business days, and before field data collection begins (subject to IRB approval and/or MOH waivers of the proposed protocols). D. In-briefing: At the country level, country evaluation team will have an in-briefing with the USAID Health team in each of the country visited for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to make any final adjustments to the evaluation design (if necessary) based on the feedback from the health team. USAID/WA may participate virtually in these presentations, which will be scheduled consecutively. E. Mid-term Briefing: The evaluation team (and the E4D COP) is expected to hold a mid-term briefing with the COR and as applicable with EngenderHealth team on the status of the evaluation at midpoint of data collection to address potential challenges, emerging opportunities and data quality. The team will also provide the evaluation COR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. F. Final Data Collection Briefing: As applicable and subject to USAID country team’s need-to-know, the country evaluation team is expected to hold a briefing once the period of data collection has ended. This presentation will be scheduled as agreed upon during the in-briefing. The country evaluation teams shall prepare and share, at least one day in advance of the exit briefing, a 10 slide (or less) presentation describing the status of data collection and analysis, and any preliminary findings if available. The Evaluation Team Leader will be responsible for organizing and delivering this briefing. G. Draft Evaluation Report: An early draft evaluation report should be submitted within 30 business days after the final data collection briefing, with the main findings. It should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. The country teams will have up to three weeks after data collection has ended in the field to complete their respective country sections, and the Team Leader will have one week to complete the integrated report. Once the initial draft evaluation report is submitted, the COR will have 15 business days in which to review and provide comments on the initial draft, after which point the E4D’s COR will submit the consolidated comments to the evaluation team. The evaluation team will then submit a revised final draft report within 10 business days for review and final comment by USAID. H. Final Evaluation Report: The evaluation team will submit a final report within10 business days of receiving final comments from the health team and E4D’s COR. All project data and records will be STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 2 submitted in full and should be in electronic format in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. I. Dissemination/Learning Event- host dissemination/learning event to discuss and validate the evaluation report with representative stakeholders of USAID/WA, AgirPF, host-country governments, etc. Table 3 - AgirPF EVALUATION timeline Activity Deadline Number of Days Recruitment of Team Leader Oct 30 5 Approval of TL by USAID Nov 12 5 Recruitment of other team members including country level Jan 20-31 9 Approval of other team members by USAID Feb 1-3 5 Desk review and Evaluation implementation plan Jan 31 5 Approval of Desk review Feb 1-3 5 Drafting the inception report Feb 3-12 8 Methodology workshop and Finalization of inception report Feb 13-18 7 Recruitment and training of data collectors in Lome Feb 13-19 7 Recruitment and training of data collectors in the 3 remaining countries Feb 20-26 6 Review of inception report Feb 20-25 5 Finalization of inception report Feb 20-25 3 Data collection and transcription in Togo Feb 20-Mar 7 15 Data collections and transcription the 3 remaining countries Feb 24-Mar 9 15 Midcourse data collection briefing to USAID Mar 13 1 Final data collection briefing to USAID Mar 15 1 Data analysis and draft evaluation report at country level Mar 13- 31 15 Drafting of integrated evaluation report Mar 31-Apr 12 13 Internal E4D review, editing and formatting of the draft report Apr 12-15 3 Review of evaluation report by USAID Apr 15- 22 8 Finalization of evaluation report Apr 29 5 Dissemination workshop TBD VII. EVALUATION TEAM COMPOSITION The evaluation team will consist of: 1. Evaluation Team Leader: an evaluation and/or FP expert with demonstrated experience leading evaluation teams for international public health interventions and who is not employed by USAID. The Evaluation Team Leader must have excellent organization, writing and oral presentation skills, as well as cultural competencies. She should speak French and English. Each country team will consist of: STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 3 2. Senior Local Subject Matter Expert (s)/Field Coordinator (s): a local FP expert with demonstrated experience in developing and managing FP activities in public and/or private sector. One Senior Local Subject Matter Expert will be recruited for each country; 3. Research Assistant: a local researcher with demonstrated experience assisting with surveys, and other methods of research, as well as assisting with data collection, quality control and analysis or data management. The roles, responsibilities and qualifications of the evaluation team are defined below: Evaluation Team Leader Responsibilities The Evaluation Team Leader will be responsible for: • Overall management of the evaluation teams, including coordination of country teams within and across countries; • Preparation of general guidelines for the implementation of the evaluation at the country level, including cross country desk review • Integration of country specific documents into cross-country deliverables • Development of overall Cross Country Inception Report, consisting of draft methodology, detailed work plan; • Coordination of evaluation activities including training of data collectors, data collection, implementation, data management and quality assurance and other related tasks; • Conduct debriefing on the methodology; • Conducting cross country data analysis as required • Conduct debriefing with implementers on evaluation findings; • Conduct field visit to the pilot project site and interviews with stakeholders (Key Informants); • Throughout the evaluation period, exercise strong communication, organizational, team leadership and interpersonal skills; periodically coordinate/update E4D’s Senior Research and Evaluation Advisor and as requested. • Development and submission of the evaluation draft report; • Finalization and submission of the final evaluation report after incorporating feedback received on the draft report; • Disseminate the evaluation findings. Qualifications • A Master’s degree in social sciences, public health, statistics, or a related area from an accredited institution is required; • At least seven years’ experience conducting public health program evaluations with both quantitative and qualitative methods for data collection and analysis; (highly desired) • Previous experience leading evaluation teams is required; • Prior evaluation experience in Sub-Saharan Africa is required; • Excellent oral and written skills in French and English are required; • Previous experience preparing high-quality evaluation reports; STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 4 • Previous experience with USG-funded projects and knowledge of USAID’s ADS2013 policy, standards, guidance and protocols (highly desired). Country Level Senior Local Subject Matter Expert (s) (in-country) Responsibilities The Local Subject Matter Expert Expert's responsibilities include, but are not limited to, the following: • Provide insight and knowledge with respect to the common practices and activities for delivering FP assistance to public and/or private entities; • Assist in developing appropriate evaluation design, methodology, sampling strategy, and data collection instruments for evaluation of a FP assistance intervention; • Assist in coordinating evaluation activities including training of data collectors, data collection, implementation, data management and quality assurance and other related tasks; • Assist in developing data analysis plan and conduct qualitative and /or quantitative data analysis, as required • Actively participate with other team members during data triangulation, presentations and report writing. • Assist the Team Leader in completion of the inception report and the writing of the evaluation report in conformance with the scope of work; Qualifications • A Master’s Degree from an accredited institution in public health, or similar discipline is required. Formal training and experience in family planning is required. • A minimum of seven years of progressive responsibilities in program management for FP programs is required; • Previous experience evaluating international public health programs is highly desired; • Knowledge of West and Central African health institutions as well as familiarity with and sensitivity to socio-cultural factors affecting development in the region is required; • Previous experience with USG-funded projects and knowledge of USAID Evaluation Policy (highly desired); • Strong oral and written communication skills in French and English is required; • Ability to effectively work in teams and embrace participatory approaches; and • Local residents of West Africa required. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 0 VIII. FINAL REPORT FORMAT The final report for the evaluation should include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, recommendations; and lessons learned (if applicable) as described in an outline to be provided. The report should be formatted according to an existing the evaluation report template. The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: ● The SOW for the evaluation; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● Sources of information, properly identified and listed; and ● Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 1 VIII. SAMPLE TABLE: ESTIMATED LOE IN DAYS BY ACTIVITY FOR TEAM MEMBERS IN ALL FOUR countries Activities Evaluation Team Leader Country Field coordinator (One per country) Research Assistants/ Data collectors (2 per country) Logistician 1/ country Desk review and Evaluation implementation plan 8 Desk review at country level and stakeholders’ meeting 5 2 Drafting the inception report 10 0 Methodology workshop and Finalization of inception report 8 8 Finalization of inception report 3 3 Translation of instruments into French 0 2 Recruitment and training of data collection staff 5 5 5 Data collection and transcription 0 15 15 15 Midcourse data collection briefing to USAID 1 Final data collection briefing to USAID 1 Data analysis and draft evaluation report at country level 10 10 Drafting of integrated evaluation report 10 Finalization of evaluation report 5 3 Dissemination workshop 4 1 1 International/regional travel 7 Total LOE by Labor Category 65 56 21 20 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 2 IX. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.2 ● The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. ● Evaluation reports shall address all evaluation questions included in the SOW. ● The evaluation report should include the SOW as an annex. All modifications to the SOW: whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the AOR/COR. ● The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. X. LIST OF ANNEXES (TBC) Annex1: Full AgirPF PMP 2 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. E4D revised SOWAgirPF Performance Evaluation ANNEXES ANNEX 1: Full AgirPF Performance Management Plan Agir Pour La Planification Familiale – AgirPF Performance Monitoring Plan (PMP) July 5, 2013 – July 4, 2018 Agreement No. AID-624-A-13-00004 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 1 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 2 TABLE OF CONTENTS Table of Contents ......................................................................................................................... 2 Acronyms and Abbreviations ...................................................................................................... 3 1. Introduction............................................................................................................................ 4 2. Project Results Framework................................................................................................. 4 3. Summary of Key Activities .................................................................................................. 7 4. Critical Assumptions ............................................................................................................ 9 5. Performance Monitoring and Evaluation ........................................................................ 10 5.1 Indicators .................................................................................................................. 10 5.2 Data collection tools review and training............................................................. 11 5.3 Data collection ......................................................................................................... 11 5.3.1 Routine or monitoring data source ............................................................. 11 5.3.2 Non-routine data source .............................................................................. 12 5.4 Data analysis ........................................................................................................... 13 5.5 Data Quality Assessments (DQA) ........................................................................ 13 5.6 Reporting and use of data ..................................................................................... 13 5.7 Dissemination .......................................................................................................... 14 5.8. Data management and storage ............................................................................ 15 6. Evaluations (baseline, mid-term, end-line) ..................................................................... 15 7. Research and Evaluation ........................................................................................... 17 8. Ethical Considerations ....................................................................................................... 18 9. Roles and Responsibilities of Project Staff in PMP Implementation .......................... 19 Figure 2: AgirPF Data Flow Chart ........................................................................................... 20 Annex A: AgirPF PMP key indicators table ............................................................................ 21 Annex B: AgirPF Research Agenda ........................................................................................ 33 Annex A: AgirPF Research Implementation plan .................................................................. 33 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 3 ACRONYMS AND ABBREVIATIONS 3Is Inform, Inspire, and Involve Approach AgirPF Agir pour la Planification Familiale CHW Community Health Worker COPE® Client-Oriented, Provider-Efficient CPR Contraceptive Prevalence Rate CPT Contraceptive Procurement Table CYP Couple-Years of Protection DHS Demographic and Health Survey DQA Data Quality Assessment FI Futures Institute FP Family planning HIP High Impact Practice IP Implementing Partner IRB Institutional Review Board KAP Knowledge, Attitudes, and Practice M&E Monitoring and Evaluation MICS Multiple Indicator Cluster Survey MOH Ministry of Health MOU Memorandum of Understanding MWRA Married Women of Reproductive Age NGO Non-Governmental Organization OR Operations Research PMP Performance Monitoring Plan PRISM Performance of Routine Information System Management RH Reproductive Health SBCC Social and Behavior Change Communication SNIS Système National d’Information Sanitaire (national health information system) USAID U.S. Agency for International Development WAAF West African Ambassadors’ Fund WHO World Health Organization WRA Women of Reproductive Age STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 4 AgirPF Performance Monitoring Plan July 5, 2013 – July 4, 2018 1. INTRODUCTION The last five decades have seen a revolution in the availability and use of family planning (FP) worldwide. FP saves lives, and it is critical to social and economic development. However, contraceptive use remains low, and unmet need is high in much of West Africa. According to the most recent Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) data, the modern contraceptive prevalence rate (CPR) is only 14–34% in urban areas of Burkina Faso, Côte d’Ivoire, Mauritania, Niger, and Togo, while unmet need ranges from 21 to 35%. On July 5, 2013, the USAID/West Africa Regional Health Office (RHO) awarded a five-year, $29 million cooperative agreement – the Agir Pour la Planification Familiale (AgirPF) Project – to EngenderHealth and two core partners, Futures Institute and EXP Agency Ltd. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in select urban and peri-urban areas of five francophone West African countries: Burkina Faso, Cote d’Ivoire (starting in Year 3), Mauritania, Niger, and Togo. The project will work closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP that followed the February 2011 Francophone West Africa Regional Conference on Population, Development, and Family Planning held in Ouagadougou, Burkina Faso. In the five participating countries, AgirPF will focus on the 10 largest cities (80,000+ population), with the exception of Zinder, Niger, which was not selected due to safety concerns. The focus cities are as follows:  Burkina Faso: Ouagadougou, Bobo-Dioulasso, and Koudougou  Côte d’Ivoire: Abidjan (starting in Year 3)  Mauritania: Nouakchott  Niger: Niamey, Maradi  Togo: Lomé, Sokodé, and Kara 2. PROJECT RESULTS FRAMEWORK The results framework illustrates in a diagram the direct causal relationships between the incremental results of key project activities and the overall objective and goal of the intervention. The results framework of AgirPF project is presented in Figure 1. The overall goal of AgirPF is to enable WRA (15–49) to make, and voluntarily act on, informed decisions about FP, thus saving women’s lives. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 5 The strategic objective consists in increasing access to and use of quality FP services in select urban and peri-urban areas of five francophone West African countries. The three discrete results that are necessary to achieve this strategic objective are:  Result R1: Delivery of quality FP information, products, and services strengthened and expanded  Result R2: Evidence-based service delivery approaches selected, adapted, and implemented  Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Under each of these three results there are two sub-results. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 6 Figure 1. AgirPF Results Framework Sub-result 3.2: Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees. Sub-result 2.2: Lessons documented and disseminated from adaptation and implementation. Sub-result 1.2: Private sector engaged in quality FP information and services provision Sub-result 3.1: Policy barriers identified and new/revised policies adopted and implemented. Sub-result 2.1: Efficiency and effectiveness enhanced through adaptation and implementation. Sub-result 1.1: Public Partners strengthened to implement proven good practices and deliver quality FP services. Result 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated.  HIP1: Task-Shifting of CHWs [under R1]  HIP2: PACFP [under R 1]  HIP enhancement intervention: (i) AY friendly services and (ii) mHealth [under R3]  HIP8: Galvanize FP commitment through policy and advocacy (financing for FP, Supply chain management system, Health communication strategy) [under R3] Result 2: Evidence-based service delivery approaches selected, adapted, implemented and documented.  HIP1: Task shifting/CSW [under R 1]  HIP3: Mobile outreach (through FP Special Days) [under R1]  HIP5: FP integration in immunization [under R2]  HIP7: Integration of HR/Gender in provider technical and counseling curricula [under R2].  HIP8: Quality Improvement (3Is, SWT, COPE for CS.[ under Result 2] Objective: Increase access to and use of quality FP services in select urban and peri-urban areas of five francophone West African countries. Result 1: Delivery of quality FP information, products, and services strengthened and expanded.  HIP1: CHW  HIP2:PACFP [under R1]  HIP3: Mobile outreach [under R1]  HIP4: PPFP [under R1] Goal: Enable WRA (15–49) to make, and voluntarily act on, informed decisions about FP, thus saving women’s lives STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 7 3. SUMMARY OF KEY ACTIVITIES A detailed description of key project activities can be found in AgirPF Year One work plan. These activities are summarized by results below to provide context for the indicators that follow: Result R1: Delivery of quality FP information, products, and services strengthened and expanded Sub-results Key Activities SR1.1: Partners strengthened to implement evidence-based approaches and deliver quality FP services  Develop and validate workplans in coordination with partners  Build the capacity of partners to provide FP counseling and services at the same time and same location where women receive PAC services  Provide a wide range of FP methods through mobile outreach services  Build capacity of partners to provide post-partum family planning (PPFP) counseling and services  Build training systems around Centers of Excellence  Build monitoring and evaluation capacity of public and private sector partners SR1.2: Engage private sector and Civil Society Organizations (CSOs) in quality FP information and services provision  Activity 1.2.1 Build capacity of private sector actors to provide quality FP information and services  Activity 1.2.2 Build capacity of civil society actors to provide FP information and services Result 2: Evidence-based service delivery approaches selected, adapted, implemented and documented. Sub-results Key Activities SR 2.1: Efficiency/effectiveness enhanced through adaptation/implementation process.  Train, equip, and support community health workers to provide a wide range of FP options including injectables  Support FP Special Days  Integrate FP into other health services, including offering FP information and services during routine child immunization contacts  Integrate sexual and reproductive rights and gender into technical and counseling curricula for providers  Build capacity of service providers to offer youth friendly services  Assure service quality using EngenderHealth Site Walk Through (SWT) approach  Implement a systematic and evidence-based social and behavior change communication (SBCC) strategy STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 8 SR 2.2: Lessons learned documented and disseminated.  Identify lessons learned  Develop printed materials  Strengthen SRH technical working groups at the country level  Create SRHR3 communities of practice (COP) at the regional level  Contribute content to internal and external websites and social media (EngenderHealth, FP 2020, IBP, etc.)  Present lessons learned findings at international forums  Create FP communities of practice (COP) at the regional level Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-results Key Activities SR 3.1: Galvanize commitment to FP through advocacy and policy development.  Support implementation of developed advocacy strategies to promote task shifting4  Support advocacy activities for at least three implementations regulations of the RH Law in Burkina Faso, Cote d’Ivoire, Niger  Support the MOH for the revision of “Politique, Normes et Protocoles” (PNP)/RH-FP document in Togo, Cote d’Ivoire, Burkina Faso and Niger  Support implementation of developed advocacy strategies for adoption of RH laws in Cote d’Ivoire and Mauritania  Support implementation of developed advocacy strategies to increase funding for FP promotion, including contraceptive commodity security  Support implementation of developed advocacy strategies to promote FP at intervention cities level  Support advocacy strategies for the integration of FP in other RH programs (FP/immunization, PACFP, PPFP)  Print advocacy and policy strategies documents and brochures in Mauritania and Cote d’Ivoire  Support the development and adoption of new national policies for the involvement of youth in FP promotion  Support advocacy activities for the integration of AYSRH services in national policies  Support advocacy activities for the integration of sexual and reproductive rights/values in national policies 3 SRHR: Sexual and Reproductive Health and Rights 4 Task shifting is an opportunity of addressing FP services accessibility and availability gaps. The Task shifting will allow Health low cadres provide wide range of contraception, including LARCs but not Permanent Methods. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 9  Support development, adoption and implementation of national policies for the promotion of the rallying concept of “responsible childbearing” among the religious leaders in Burkina Faso, Cote d’Ivoire and Togo  Support advocacy activities for the promotion of the rallying concept of responsible parenting among the religious leaders in Niger and Mauritania  Support the organization of a regional workshop to establish coordination of the religious networks on a regional level, in collaboration with WAHO and other partners  Support the organization of a regional workshop to establish coordination of the youth networks in collaboration on a regional level with WAHO and other partners  Support the development of two videos: 1) on the successful advocacy activities conducted by the RCPFAS and 2) on the successful advocacy activities conducted by URCB (Union des Religieux et des Coutumiers du Burkina Faso)  Support the replication of good practices from RCPFAS of intervention countries of AgirPF in Mali, Mauritania and Chad in collaboration with WAHO, the SWEDD project and other partners SR3.2: Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management  Support annual Contraceptive commodities quantification exercises with Reality Check  Support trainings on monitoring and reporting on stock levels  Introduce COPE for Commodity Security (CS) tool at facility level  Introduce improved supply chain management and mHealth technology to continuously report on stocks 4. CRITICAL ASSUMPTIONS The degree to which these results can be achieved depends on a number of assumptions that will need to be supported to achieve the project results. These assumptions are: 1. Social, political and legal environments will remain favorable to AgirPF’s interventions implementation; 2. Governments will not implement new policies, standards and protocols that restrict FP services; 3. Commitment and cooperation from the Ministries of Health and partners to implement the proposed strategy is sustained throughout the life of the project and beyond; STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 10 4. There will be adequate equipment, expendable supplies and contraceptive products in the intervention areas to support the delivery of FP services; 5. There will be timely and continuous availability of funding to support work plan implementation. 5. PERFORMANCE MONITORING AND EVALUATION This Performance Monitoring Plan (PMP) outlines the criteria that will be used to assess the outputs and outcomes of AgirPF Project in five francophone countries of West Africa between July 5, 2013 and July 4, 2018. The use of data in programmatic decision making is an integral component of AgirPF Project. A strong monitoring and evaluation system facilitates the achievement of targets and objectives, tracks the planned use of resources, provides quantitative and qualitative data to assess outcomes, and provides all stakeholders with information on progress and results. This comprehensive PMP will help to ensure that the program is conducted in a systematic and efficient manner and provides essential feedback to ensure that the program is dynamic and responsive to changing conditions. AgirPF has benefited and will continue to benefit from convincing documentation of the effective and appropriate implementation of program activities, as well as from evidence of the effect of those activities. The project uses a range of indicators to document program activities and, when possible, to demonstrate the outcomes of those activities. 5.1 Indicators Indicators are signs or markers that measure one aspect of a program and show how close a program is to its desired path and outcomes. They are used to provide benchmarks for demonstrating the achievements of a program. The AgirPF Project uses a range of indicators to document and monitor implementation of project activities and, when possible, to demonstrate the outcomes of those activities. A number of these indicators initially feed into standard USAID West Africa Regional Health Office Performance Data Tracking indicators, as shown in Annex A which includes a detailed description of each indicator. For PY3, they feed into the newly USAID West Africa Regional Development Cooperation Strategy (RDCS) set of indicators. The table in Annex A includes process and outcome indicators and annual targets for the life of the project for the five focused countries as well as definitions and the source of data for each indicator. The list of indicators included in the indicator matrix table represents recommendations of AgirPF project staff as well as staff from the two core partners, Futures Institute and EXP Agency Ltd. These indicators were initially drafted during the September orientation workshop by those working in these specific areas. AgirPF reviewed the Regional Health Office indicators, PEPFAR indicators, and indicators of similar projects when deciding which indicators to select. The project took into consideration the relevance of each indicator as well as the feasibility of measuring it given the available budget. Indicator targets were set based on the country specific contexts, recent results of similar projects in the region, and the project’s technical approaches. As mentioned above, the list of indicators was updated and consolidated in PY3 to align with the new USAID West Africa Regional Development Cooperation Strategy. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 11 5.2 Data collection tools review and training Given that AgirPF’s approach to monitoring will prioritize data from existing sources, such as service statistics and existing data collection tools, AgirPF has coordinated with partners to conduct a participatory review of processes, flow, and reporting systems for FP data in the Health Management Information System (HMIS) of each country, and develop/adapt monitoring tools and databases. The M&E/R Advisor has lead a participatory review in each country of HMIS processes, flow, and reporting forms, and of NGO/private-sector reporting, to ensure the project uses quality data. In addition, after this review/adaptation has been done, trainings for data collectors on how to use the tools were conducted. These review processes were organized in collaboration with the host country MOH, and with the assistance of the Country Managers and the Regional M&E/R Officer. Upon this review AgirPF decided to establish data focal points and train them on data collection, review and entry into DHIS2. The training for data collectors has been also organized by both the MOH and AgirPF staff who participated in the review process. This training included an overview of the data collection system, data collection techniques, tools, ethics, culturally appropriate interpersonal communication skills and practical experience in collecting data. Retraining has been organized as staff change as has AgirPF data collection process. AgirPF also adapted USAID’s DQA tools and MEASURE’s Data Demand and Use Toolkit—in particular, the proven Performance of Routine Information System Management (PRISM) Framework and tools used and evaluated by MEASURE Evaluation. AgirPF has also strengthened district data systems in Burkina Faso, Niger and Togo. This was planned for Côte d’Ivoire and Mauritania but not yet done. Starting from PY4, AgirPF data will no longer be capture in DHIS2 by the districts data operators but rather by data focal points recruited by the project who will also review the content of the routine data collection forms filled out by the health facilities. 5.3 Data collection AgirPF uses data collected at multiple levels, including the client, activity, service environment, government, and population levels. AgirPF’s approach to monitoring will prioritize data from existing sources, such as service statistics. For each indicator without a pre-existing source, the project develops data collection forms and instructions. Regardless of level, data are commonly divided into two general categories: routine and non-routine data. 5.3.1 Routine or monitoring data source Routine data sources provide data that are collected on a continuous basis. AgirPF’s approach to monitoring will prioritize data from existing sources, such as service statistics and existing data collection tools. AgirPF uses two primary sources of monitoring data:  Information collected from program reports (e.g., training/activity reports, clinical monitoring visits); and  Aggregated service statistics from the national health information system (le Système National d’Information Sanitaire, or SNIS) at the health facility level. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 12 To measure the relative performance of the intervention strategies, service statistics describing the quantity and types of FP information and services provided are abstracted from the service statistics collected as part of the MIS. Service statistics collected describes each of the main FP services being provided, i.e. new and continuing FP clients by type of method; number of contraceptives distributed by type; etc. We then calculate CYPs based on these data as our primary strategy for monitoring changes in FP use. The project uses the most up-to-date CYP conversion factors from USAID and the Impact2 calculator.5 5.3.2 Non-routine data source Non-routine data source serves two purposes: informing activities and assessing the outcomes of AgirPF interventions.  Facility audits o To determine facility readiness (personnel, procedures, infrastructure, health care supplies, contraceptives, infection prevention, medical instruments/equipment, use of information systems) o To provide FP services and integrated FP/MCH services o To evaluate the degree to which the facilities’ systems, processes and physical environment are gender-equitable, and male and youth-friendly o To assess quality and completeness of FP service statistics using proven frameworks, tools, and approaches  Semi-structured interviews with national and local stakeholders such as reproductive health (RH) coordinators, other RH/FP MOH staff, and NGO partners. The purpose of these interviews was to solicit opinions on the project’s scope, suggestions for approaches to adopt/adapt, and attitudes toward FP and gender norms.  FP market segmentation research that has been conducted to tailor the SBCC mesages and materials to the different types of audiences targeted by the project.  Household surveys of men and women aged 15–49 in the urban/peri-urban target areas: Using random sampling, the surveys have collected baseline data in intervention and comparison zones on the reach of SBCC campaigns and knowledge, attitudes, and practices (KAP) related to FP use.  Etc. The FP market segmentation research was conducted in collaboration with Camber Collective while the facility audits, structured interviews and household survey was led by external consultants at baseline and will be replicated at mid-term (Year 4), and end-line (See section 6.1 below for more details). 5 http://transition.usaid.gov/our_work/global_health/pop/techareas/cyp.html STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 13 5.4 Data analysis AgirPF will conduct data analysis and the write-up of reports based on the different data collected or generated by the project: quarterly and annual reports for the donor, baseline study reports, policy briefs, research reports, articles for presentation to scientific/international conferences, etc. Most of the analysis will be descriptive and mainly focused on the examination of relevant indicators in line with the indicators described in the PMP study. All data analysis will be conducted using appropriate statistical software packages for qualitative and quantitative data. When relevant, AgirPF will collect and analyze data so that it can be disaggregated by sex, age and any other social distinctions that inform program decision-making and implementation. 5.5 Data Quality Assessments (DQA) The M&E/R Advisor has lead, in each country, a participatory review of SNIS processes, flow, and reporting forms, and of NGO/private-sector reporting, to ensure the project uses quality data. AgirPF has strengthen the SNIS where needed in order to obtain quality data (e.g. by orienting one provider by AgirPF intervention site in filling out AgirPF’s monthly data routine form). The methodology used to date by AgirPF to ensure timeliness, completeness and quality of its data included a mix of document and record reviews and site visits. AgirPF staff verify the quality of data through:  Working with implementing partners to ensure that they establish sound data collection and maintenance procedures;  Spot checking data submitted by implementing partners;  Providing feedback and mentoring to IPs to improve data quality. When assessing data quality, AgirPF focus on five key standards: validity, reliability, precision, integrity, and timeliness. At least 5 DQAs will be conducted in each AgirPF country quarterly. By late PY3, AgirPF hired data focal points with the explicit mandate of monthly reviewing AgirPF’s filled forms at the facility level and to enter the data in DHIS2. 5.6 Reporting and use of data AgirPF staff communicate information about progress and accomplishments to EngenderHealth headquarters in New York and to USAID/RHO by means of the activity reports cited below. All reports will first be submitted to EngenderHealth/NY for review before submission to USAID/RHO: Quarterly progress reports submitted 45 days after the end of each quarter and Annual reports submitted 45 days after the end of each year. The deadline of 45 days after the end of the quarter/year will allow for both the quarterly and annual reports to include complete and final financial information on all activities from the quarter/year in all five countries. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 14 EngenderHealth works with two categories of partners: international core project partners and local partners. Although only quarterly reports are submitted to USAID, all sub-partners will report their programmatic activities monthly to EngenderHealth. Local sub-partners’ activities are directly captured into AgirPF countries monthly reports. In each country the Senior Program Officer (SPO) and the Country Manager (CM) are responsible for providing quality control and technical review of sub-partner’s reports to ensure their accuracy before their inclusion into the country’s monthly report. This validation is done through regular visits of activities and reported accomplishments, monitoring visits, analysis of data, meeting and questioning beneficiaries and conducting interviews of stakeholders. After validation at the country level, each country’s monthly report is sent to the AgirPF Regional Office for further analysis and use by management. International core project partners also report progress in their activities on a monthly basis, directly to AgirPF at the regional level. The Technical Director and the M&E/R Advisor analyze reports received, look for possible discrepancies, revert to country teams with questions if needed, and when satisfied with the quality of the information received, compile all the data into one AgirPF monthly report. This monthly report goes through final review and approval by the COP and is kept as an internal document. Each quarter, these monthly reports are compiled to produce a quarterly report that is approved and validated by the COP and EngenderHealth Headquarters in New York before its submission to USAID. AgirPF will expand the FP knowledge base in West Africa using the MEASURE Evaluation Framework for Linking Data with Action, which helps stakeholders identify information needed to make informed decisions about BPs, encourages use of information, and monitors use of data in decision-making. We will work with partners to explore innovative non-electronic channels to help providers and others learn about results critical to their knowledge. We produce a final report, a PowerPoint presentation, and a study brief for all models tested, as well as for baseline, mid-term and end-line assessments, highlighting key findings, recommendations, and breakthroughs, and we will capture compelling stories of providers, clients, champions, leaders, and partners via testimonials. Whenever necessary we disseminate evaluation and study findings, as well as training and implementation materials, via the Community of Practice. To facilitate global dissemination of knowledge about BPs, we submit abstracts and manuscripts of results and lessons learned for publication and presentation at conferences. Peer review publications of findings will also be explored. 5.7 Dissemination Collecting data is only meaningful and worthwhile if it is subsequently used for evidence-based decision-making. To be useful, information must be based on quality data, and it also must be communicated effectively to policy makers and other interested stakeholders. Dissemination will inform the community of stakeholders about what the project has achieved and the benefits of using it. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 15 AgirPF conduct dissemination activities to share what the project is doing, to inform and educate the community of stakeholders, to get input or feedback from the community, and to promote the project outputs. The audience targeted by this dissemination activity include the internal stakeholders who are partners in the implementation and the external stakeholders of the West Africa region. A wide range of dissemination methods are used: policy brief, brochures, project meetings, conference presentations, posters, workshops, reports and journal articles, etc. Dissemination activities will be organized during the life time of the project and message will vary with the timeframe. 5.8. Data management and storage Data management refers to the processes and systems for how the project will systematically and reliably store, manage and access M&E data. Data will be recorded and stored in standardized formats to improve the organization and storage of data. Data formats will be physical, such as written forms stored in an office filing cabinet, or electronic, such as a spreadsheet stored in a computer database and audio (recordings of interviews, etc.). AgirPF uses the District Health Information System (DHIS2) to track and analyze SNIS FP service statistics as well as project data on trainings, events, workshops, and so on. For survey data they are stored on the AgirPF server in Lomé and also at EngenderHealth Headquarters in New York. In addition, a copy of the backup will be secured in an External drive stored in the AgirPF Regional Office in Lomé. At all stages of the activity (field data collection, data entry, archiving) AgirPF is responsible for storing the information securely. Once the activity is completed, the questionnaires are presented at AgirPF, who is the owner. Only those individuals directly involved in activity management will have access to the questionnaires and electronic data. AgirPF organized its information into logical, easily understood categories to increase its access and use. Data are organized chronologically (e.g. month, quarter, year), by location, by content or focus area (e.g. different objectives of the project), and by format (e.g. project reports, donor reports, technical documents). Data are easily available to its intended users and secured from unauthorized use (discussed below). Permission are granted and controlled to access data (e.g. shared computer drives, folders, intranet). For security reasons, data will be protected from non-authorized users. This ranges from a lock on a filing cabinet to computer password to access data. Data storage and retrieval also conform to any privacy clauses and regulations for auditing purposes. The M&E/R Advisor has the lead responsibility and accountability of data management. 6. EVALUATIONS (BASELINE, MID-TERM, END-LINE) AgirPF conducted a baseline evaluation in all its 5 interventions countries at the beginning of the project. A mid-term evaluation will be conducted in Year 4 by USAID Evidence for Decision (E4D) project, as well as an end-line evaluation toward the end of Year 5. These evaluations will help assess project effectiveness and changes over time to key services, enabling environment, and demand indicators. The baseline evaluation study is the first source of non-routine data. In Year STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 16 1 of the project in each country, before the start of any intervention, and before beneficiaries and communities learn about the intervention, AgirPF conducted a baseline evaluation study. The purpose of this baseline study was to:  Collect baseline data for the development of baseline indicators against which progress can be measured;  Collect baseline data in intervention and comparison zones on the knowledge, attitudes, and practices (KAP) related to FP use in the target population;  Collect qualitative data on AgirPF scope and approaches to adopt/adapt, and on attitudes toward FP and gender norms;  Determine Health Facilities current readiness of service delivery points to provide quality FP services and integrated FP/MH services to clients in conformity with the existing guides and guidelines;  Determine the degree to which the facilities’ systems, processes and physical environment are gender-equitable, and male and youth-friendly;  Assess the availability of FP information and services for adolescents. This baseline evaluation studies were led by external consultants and have collected both qualitative and quantitative data using the following methodologies:  Facility audits o To determine facility readiness (personnel, procedures, infrastructure, health care supplies, contraceptives, infection prevention, medical instruments/equipment, use of information systems) to provide FP services and integrated FP/MH services; o To evaluate the degree to which the facilities’ systems, processes and physical environment are gender-equitable, and male and youth-friendly; o To assess quality and completeness of FP service statistics using proven frameworks, tools, and approaches.  Semi-structured interviews with national and local stakeholders such as reproductive health (RH) coordinators, other RH/FP MOH staff, and NGO partners. The purpose of these interviews was to solicit opinions on the project’s scope, suggestions for approaches to adopt/adapt, and attitudes toward FP and gender norms. Members of the district management team6 (about 8 to 10 persons) and representatives of the NGO partners were interviewed. An estimated 12 to 15 structured interviews per city were conducted.  Household surveys of men 15-59 and women aged 15–49 in the urban/peri￾urban target areas: Using random sampling, the surveys have collected baseline data in intervention and comparison zones on the reach of SBCC campaigns and knowledge, attitudes, and practices (KAP) related to FP use. Using the Epi Info 6 The district management team termed “Equipe Cadre de District (ECD)” is usually composed of the following persons: le Médecin Chef de District, le Responsable de la Santé de la Reproduction, le Responsable de l’IEC pour de la Communication pour le Changement de Comportement (IEC/CCC), le Responsable du Service d’Hygiène et de la Salubrité; le Président du Comité de Gestion (COGES); le Responsable de la Pharmacie; le Responsable du Contrôle de la Maladie; et le Responsable de la Maternité. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 17 Statcalc program for unmatched case-control study (comparison of ILL and NOT ILL) to estimate sample size to be interviewed and assuming that: (1) CPR increase from 21 percent to 31 percent; (2) Two-sided confidence interval of 95 percent and a power of 80 percent; (3) A non-response rate of 5 percent for Togo and Burkina Faso , and of 10 percent for Niger and Mauritania; and (4) That there will be 0.95 women aged 15-49 years and 0.85 men aged 15-59 years in each household. We estimated a sample size of 1968 women and 1761 men for Togo; 1968 women and 1761 men for Burkina Faso; 1384 women and 1238 men for Niger; and finally 692 women and 619 men for Mauritania. The same sample sizes will be used during mid-term and end-line evaluations. During the fourth year of project implementation, the USAID E4D project will conduct a performance evaluation to assess the effectiveness of the project (how it was implemented; how it is perceived and valued; whether expected results were reached) and to identify strengths and weaknesses, lessons learned, and best practices to better guide future programs. The evaluation will be quantitative and qualitative. The methodology may include a review of activities conducted, a comparison of objectives and results, interviews with AgirPF staff and partners in the Ministry of Health, and mid-term data collection at the facility level. For purposes of comparison, such a data collection exercise would share the same methodology as the baseline data collection conducted at the beginning of the project. The baseline assessments have identified the gaps in clinic functioning that need to be addressed through the clinic strengthening intervention. The mid-term and end-line evaluations will measure the functional capacity of the clinics after the clinic-strengthening interventions and at the time of data collection on the key dependent variables. Such measures are necessary to be able to control for the level of clinic functioning during data analysis. 7. RESEARCH AND EVALUATION Research and evaluation efforts will produce data to inform program decisions. Following USAID WA new Regional Development Cooperation Strategy (RDCS), AgirPF has made strategic shifts in PY3 workplan by focusing most of its interventions on high impact practices (HIP). Since these HIPs are adapted and implemented in new and/or different social and programmatic environments, many evaluation and research activities were proposed focusing on documenting the evidence and effectiveness of these interventions. During PY3, AgirPF has developed a research agenda and its implementation plan that can be found respectively in appendix B and appendix C. The proposed evaluation and research activities will be described in formal protocols. While designing study protocols, the legal requirements around ethical conduct of studies will be ensured, including approval from appropriate authorities. AgirPF will rely on the expertise of experienced research groups, the collaboration with local universities and FP service providers in intervention health facilities as well as the technical assistance and guidance from consultants and EngenderHealth’s HQ Team to conduct relevant studies. The timing of their implementation will depend on the protocol and tools approvals process from EH internal review board, WIRB and local ethical review boards of the host countries. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 18 For PY4, the following gives an overview of these studies. (a) Conduct COPE for Contraceptive Security (CS) baseline evaluation in 25% of sites, and end-line evaluation 6 months after introduction of COPE for CS: The purpose of these evaluations studies is to determine whether and how the COPE® for CS process improves contraceptives security at supported facilities. (b) Document Site Walk-Though (SWT) efficiency through a qualitative and quantitative study: Site Walk-through is a quality improvement tool, a continuous process of identifying issues and providing solutions involving communities. The purpose of this evaluation is to determine the impact of this intervention strategy on FP services in terms of number of additional FP clients, client satisfaction and community engagement. This study will be conducted in Burkina Faso. (c) Document FP Special Days efficiency through a qualitative and quantitative study: A FP Special Days, is defined as a suite of three specific days during which service providers and support staff set aside time to focus on offering FP services (including long￾acting reversible contraception) at no or low cost to clients. The purpose of this evaluation is to determine the impact of this intervention strategy on FP services in terms of number of additional clients and client satisfaction. This study will be conducted in Niger and Togo. (d) Prospective cohort study on Contraceptive use dynamics: Contraceptive continuation is one of the indicators for measuring the effectiveness of increasing FP use. The objective of the study is to measure contraceptive use discontinuation and determine reasons for switching and for discontinuation in project focus countries. (e) Mid-term evaluation of AgirPF Project: A performance evaluation of AgirPF project is planned to take place during PY4Q1 in all AgirPF’s countries but Mauritania where the implementation of the project has just started. The evaluation will be conducted by USAID E4D Project (Evidence for Decision). For each study a protocol describing the methodologies and budget will be fully developed at an appropriate time. As with all special studies, each protocol and tools will undergo rigorous ethical and technical review by EngenderHealth in-house ethical review process, review by USAID Mission, review by local host country and US based Institutional Review Board (IRB). In PY4 a selected number of research studies will be designed and implemented in the Project countries. The M&E/R team will work in collaboration with AgirPF program staff and EngenderHealth HQs to develop or revise concept notes, protocols and tools and provide support to the selected consultants or research firms for the smooth implementation of these activities. Appendix B and Appendix C give more details on the planned research and evaluations. 8. ETHICAL CONSIDERATIONS For all evaluations and research studies, EngenderHealth’s ethical approval process will be followed to ensure that research ethics are respected. Every measure will be taken to ensure respect for the dignity and freedom of each individual invited to participate in the studies. During training of the data collection teams, AgirPF will place special emphasis on the importance of obtaining informed and STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 19 voluntary consent of all participants, respect for confidentiality, and the prohibition of any form of coercion. Each questionnaire will be preceded by an informed consent form guiding the interviewer to present the purpose of data collection, the risks and benefits of participation, and their right to decline participation or to decline to answer any or all questions. Before each interview, the interviewer will sign the form confirming that informed consent has been obtained. To protect participant anonymity, interviewers will write no identifying information on the data collection forms. All studies will undergo ethical review by the EngenderHealth Director of Knowledge Management, Monitoring, Evaluation, and Research who will determine if review by local and/or U.S.-based Institutional Review Board is required. The project will also obtain ethical review and approval at the country level before initiating data collection. 9. ROLES AND RESPONSIBILITIES OF PROJECT STAFF IN PMP IMPLEMENTATION Major sources of data and information for project monitoring and evaluation include secondary data, project output data, evaluation and studies. The people responsible and accountable for the data collection and analysis include community volunteers, field staff, project managers, local partners, and external consultants. Proper management of these M&E/R data requires the involvement of all AgirPF project staff at different steps of the data collection and analysis process at different levels. At the country level, the coordination of the data collection with data collectors is done in collaboration with the Country Manager and the Senior Program Officer involved in the implementation of technical activities. The Regional M&E/R Officer works at the country level to provide technical assistance and supervision and to ensure that quality of data are collected in the field. All data collected receive quality control. After the procedures for checking and cleaning data are implemented, data computerization and analysis is done at the country level. This information is collected from each country and centralized at the regional level. At the regional level, AgirPF key staff including Technical Director, Policy and Advocacy Advisor and the M&E/R Advisor work in team to analyze this information collected from different countries to produce relevant reports for the COP. These reports are then reviewed and approved by the COP and EH Program Managers based in New York before they are sent to USAID. Figure 2 below outlines the flow of data. At step one (Data Collection) the validation is done by the country Senior Program Officer and the Country Manager; at step two (Data Capturing) it is validated by the Regional M&E/R Officer; at step three (Data Review) the validation is done by M&E/R Advisor; at step four (Reporting) the M&E/R Advisor and the Technical Director validate the report; and finally at step five (Report Review) the COP and EngenderHealth New York based backstop approve the final version of the quarterly report before it is submitted to USAID. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 20 FIGURE 2: AGIRPF DATA FLOW CHART Data Collection FP providers, community health workers, supervisors, AgirPF staff and consultants document outputs on an ongoing basis. Data Capturing - AgirPF Senior Program Officers compile data from service delivery points, reports from implementing partners, meetings with peer educators, training sign-in sheets, and activity records. - Data focal points enter data in AgirPF monthly routine data forms - AgirPF ME&R team enters data in regional indicator database. Data Review AgirPF ME&R team cleans the data, assesses its quality, and seeks missing data and clarification as needed from Senior Program Officers. Reporting AgirPF ME&R team prepares quarterly reports and annual performance reports based on data received. Report Reviews AgirPF key personnel and NY-based backstop review the reports prior to submission. Approval and Submission AgirPF Chief-of-Party approves and submits the report to the USAID/West Africa Regional Health Office and EngenderHealth/New York. E4D revised SOWAgirPF Performance Evaluation ANNEX A: AGIRPF PMP KEY INDICATORS TABLE All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP SO: Increase access to and use of quality FP services in select urban and peri-urban areas of five francophone West African countries 1 Number of CYP achieved in AgirPF supported areas (output indicator) (USAID RDCS) The estimated protection by family planning (FP) services during a one-year period, based upon the volume of all contraceptives sold or distributed free of charge to clients during that period. In addition to health facility routine FP data, this information includes the number of commodities distributed during training activities and special FP days. The CYP is calculated by multiplying the quantity of each method distributed to clients by a conversion factor, to yield an estimate of the duration of contraceptive protection provided per unit of that method. The CYPs for each method are then summed over all methods to obtain a total CYP figure. The conversion factor is standardized, and AgirPF uses the USAID standard conversions. MOH supported sites data from DHIS2, FP special days, trainings, Sub￾grantee activity, WAAF, private partnerships (that are not reported in district data) Collect monthly report quarterly M&E/R Advisor Burkina Faso n/a 97,786 107,871 114,269 100,916 420,842 Côte d’Ivoire n/a 312,702 110,896 117,678 102,605 643,882 Mauritania n/a 9,536 20,952 22,257 19,251 71,997 Niger n/a 147,851 67,942 71,947 63,704 351,444 Togo n/a 73,854 40,374 42,753 37,855 194,836 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 22 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP A supported area is activity that AgirPF directly interacts with, i.e. supported sites and activities, not the catchment areas AgirPF works in. To calculate targets, the method mix that was collected during the baseline was used with the project growth rate, leaving the method mix constant. Disaggregated by Method Total n/a 641,728 348,036 368,905 324,332 1,683,000 2 Contraceptive Prevalence Rate (CPR) (outcome indicator) (USAID RDCS) The proportion of women of reproductive age (WRA, age 15-49) who are using (or whose partner is using) a contraceptive method at a given point in time Numerator: number of WRA who self-report using FP Denominator: number of WRA surveyed For setting targets, the projected growth rate is 2% per year Baseline and end-line household Surveys Y1 7& Y5 M&E/R Advisor Burkina Faso 38.5 40.5 42.5 44.5 46.5 n/a Côte d’Ivoire n/a 27.9 29.9 31.9 33.9 n/a Mauritania n/a n/a 19.3 21.3 23.3 n/a Niger 39.2 41.2 43.2 45.2 47.2 n/a Togo 40.3 42.3 44.3 46.3 48.3 n/a Total n/a n/a n/a n/a n/a n/a 3 Total number of FP method users (output indicator) The number of persons during a defined reference period (e.g., one year) who use a modern contraceptive method. These include all users accessing project supported sites/services for re-supply, method changes, and/or new users. Disaggregated by method (method Mix), service delivery type (Fixed routine service, mobile service, Special day, CBD) MOH supported sites data from DHIS2, FP special days, trainings, Sub￾grantee activity, WAAF, private partners Collect monthly report quarterly M&E/R Advisor Burkina Faso n/a 388,734 445,859 493,056 528,454 n/a Côte d’Ivoire n/a 330,718 404,603 495,588 481,548 n/a Mauritania n/a 54,359 76,077 85,270 86,690 n/a Niger n/a 253,431 302,114 331,831 347,833 n/a Togo n/a 150,598 193,878 211,537 213,871 n/a Total n/a 1,177,841 1,422,531 1,617,282 1,658,396 n/a 7 Y1 mCPR are estimates from baseline surveys conducted in the four AgirPF countries (Burkina, Côte d’Ivoire, Niger and Togo). For Mauritania, data presented is an estimation from the MICS conducted in 2011 in this country. AgirPF baseline survey will be conducted in Nouakchott is year 3. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 23 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP 4 Number of additional users of modern methods of contraception (USAID RDCS) Number of new users to a modern method of contraception defined as someone who was not using a modern method of contraception when they received their method –,including people who previously used a method, stopped, and are now starting a method. Disaggregated by service delivery type (Fixed routine service, mobile service, Special day, CBD) MOH supported sites data from DHIS2, FP special days, trainings, Sub￾grantee activity, WAAF, private partnerships Collect monthly report quarterly M&E/R Advisor Burkina Faso n/a 40,389 44,555 47,198 41,682 173,824 USAID DO3: Utilization of Quality Health Services Increased Through West African Partners Côte d’Ivoire n/a 129,158 45,804 48,606 42,380 265,948 Mauritania n/a n/a 3,939 8,654 9,193 21,786 Niger n/a 61,068 28,063 29,717 26,312 145,160 Togo n/a 30,504 16,676 17,659 15,636 80,475 Total n/a 261,119 139,037 151,834 135,203 687,1938 Result 1: Delivery of quality FP information, products, and services strengthened and expanded 5 Percent of FP service providers deemed technically competent based on an assessment according to national international or other defined standards (outcome indicator) (USAID RDCS) Health providers (doctors, nurses, midwives, and community health workers) will be supervised performing the FP services and counseling they were trained in by AgirPF. They will be assessed based on international standards for competency. “Performing up to standards” will be defined as receiving a score of at least 85%. Numerator= number of project-trained FP service providers deemed technically competent Denominator= Total number of FP service providers trained by the project and assessed Disaggregated by sex Facility audits, Facilitative supervision, sub-grantees Quarterly AgirPF Technical Director; Country Managers; SPOs Burkina Faso 80 80 80 80 80 80 USAID Sub-IR 3.1.3: Service provision standards applied (in Côte d’Ivoire pilot sites) n/a 80 80 80 80 80 Mauritania n/a n/a 80 80 80 80 Niger 80 80 80 80 80 80 Togo 80 80 80 80 80 80 Total 80 80 80 80 80 80 Sub Results 1.1: Partners strengthened to implement evidence-based approaches and deliver quality FP services 8 Note: This target is primarily from the MOH health facilities we work with, it is difficult to predict targets from other inputs, such as WAAFs, because what their work will be and their recording mechanism will be defined on a case by case basis as these partnerships are established. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 24 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP 6 Number of local organizations with improved organizational and management capacity as measured by a defined organizational assessment tool Output indicator (USAID RDCS) The number of AgirPF local partners implementing AgirPF activities which are improving organizational and managerial capacity. This improvement will be measured by using the OCAT tool. Disaggregated by type of local organization Routine supervision reports Annually Country Managers Burkina Faso 1 5 4 0 0 10 USAID Sub-IR 3.2.1: Capacity of regional and national institutions and organizations strengthened Cote d’Ivoire n/a 5 4 3 0 12 Mauritania n/a n/a 2 3 0 5 Niger 0 4 3 1 0 8 Togo 1 3 3 0 0 7 Total 2 17 16 7 0 42 7 Number of FP curricula updated to include gender sensitivity, couple counseling, youth and male friendly services (output indicator) FP curriculum integrating gender sensitivity, couple counseling, youth and male friendly services utilized Disaggregated by type of curriculum (Gender sensitivity, Couple counseling, Youth friendly services, Male friendly services, Human Rights in SRH) Activity reports Collect monthly report quarterly Country Managers Burkina Faso 1 0 0 1 0 1 Côte d’Ivoire n/a 0 1 1 0 1 Mauritania 1 0 0 1 0 1 Niger 1 0 0 1 0 1 Togo 1 0 0 1 0 1 Total 4 0 1 5 0 5 8 Number of people trained in family planning and reproductive health with USG funds (output indicator) Number of people (health professionals, primary health care workers, community health workers, volunteers, non-health personnel) trained in FP/RH (including training in service delivery, communication, policy and systems, research, etc.). Disaggregated by sex; type of training Type of training include: New integrated FP Curriculum, Infection prevention, Reality Check, Activity reports Collect monthly report quarterly Country Manager; Regional contraceptive security specialist; Regional policy officer Burkina Faso 99 399 399 399 0 1296 This figure reflects all type of trainings (clinical, Leadership, management, Côte d’Ivoire Advocacy, etc.) 15 659 659 659 0 1992 Mauritania 20 154 154 152 0 480 Niger 160 235 235 234 0 864 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 25 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP COPE, “3Is” Approach, Updated LMIS tools, advocacy Counseling REDI, PPPF PPIUD, Contraceptive Logistics/LMIS, HIV/AIDS, MCH, COPE for Contraceptive Security, Facilitative supervision, Spectrum, TOT in FP on EH approaches, tools, resources & policies, OCAT, Gender, Youth Sexual and Reproductive Health, AgirPF Health Information System, Training on group and individual FP counseling as part of other services, Post Abortion Care (PAC), Integration of Human Rights in SRH. Togo 135 339 339 339 0 1152 Total 429 1786 1786 1783 0 5784 9 Number of HIV positive women who received comprehensive FP services (output indicator) This indicator informs about level of integration of FP services into HIV services. Meaning the providers in these specific services, have received capacity re￾enforcement and are able of providing comprehensive FP services (sensitization, counselling and acceptation of a given method of contraception) Activity reports, WAAF Collect monthly report quarterly Country Managers Burkina Faso n/a n/a n/a n/a n/a n/a Côte d’Ivoire n/a n/a 300 400 500 1200 Mauritania n/a n/a n/a n/a n/a n/a Niger n/a n/a n/a n/a n/a n/a Togo n/a n/a 100 n/a n/a 100 Total n/a n/a 400 400 500 1300 10 Number of special FP days conducted (output indicator) Special FP days are days where the range of FP services are offered free of charge or at low cost by dedicated providers. Typically, this strategy expands the reach of a range of FP services: trained providers travel to remote facilities to expand the range of methods offered. Activity reports, daily consultation registers Collect monthly report quarterly Country Managers Burkina Faso 10 10 200 200 50 520 Côte d’Ivoire n/a 0 200 140 50 500 Mauritania n/a n/a 120 50 40 300 Niger 10 10 150 100 30 395 Togo 10 10 200 120 40 520 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 26 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP Note: a FP Special Day is a 3-day event. Therefore, when we record 5 FP Special Days, FP services were actually provided in this format for 15 days. Total 30 30 870 610 210 2235 Sub Result 1.2: Local leaders, civil society, service providers, municipal government support and promote FP 11 Number of additional USG-assisted community health workers (CHWs) providing family planning information and/or services during the year (output indicator) (USAID RDCS) CHWs supported (trained, equipped with kits) and supervised Disaggregated by sex, type of training (FP, HIV/AIDS, MCH) Activity reports Collect monthly report quarterly Country Managers Burkina Faso 0 0 27 0 0 45 *Starting in Year 3, City Based Services (CBS) will be introduced. CBS implementers called City Based Health Workers (CBHW) will be introduced in Year 3 and will be transferred to MOH in Year 5 Côte d’Ivoire n/a n/a 0 0 0 100 Mauritania n/a n/a 0 0 0 30 Niger 0 0 103 100 100 150 Togo 0 216 180 375 375 180 Total 0 216 310 475 475 505 12 Proportion of women and men reporting increased dialogue with their partner about FP (outcome indicator) A man or a woman is reporting dialoguing with their partners if during the last three months they discussed at least once FP issues including the choice and/or use of a given FP method. Numerator = Number of women and men reporting dialogue with their partner about FP Denominator =Total number of women and men interviewed Disaggregated by sex. Pre and post￾Household KAPB surveys Y1 baseline, Y5 end-line M&E/R Advisor Burkina Faso 52 n/a n/a n/a 65 n/a USAID Sub-IR 3.2.3: Attitudes toward health-seeking behaviors improved Côte d’Ivoire n/a 40 n/a n/a 55 n/a Mauritania n/a n/a n/a9 n/a 35 n/a Niger 39 n/a n/a n/a 50 n/a Togo 44 n/a n/a n/a 55 n/a Total n/a n/a n/a n/a n/a n/a 9 Mauritania’s target will be updated when Baseline data will be available. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 27 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP 13 Percent of men and women with gender￾equitable attitudes (outcome indicator) (Modified USAID gender indicator) (USAID RDCS) (Modified gender indicator) Attitudes of women and men will be assessed to determine attitudes in terms of: their support for women’s sexual and reproductive rights; their support for women’s right to practice a contraceptive method; their support for men’s involvement in the promotion of women’s sexual and reproductive health; their support for joint decision-making about FP; their support for consensual sex in a relationship; their support for women’s involvement in decision-making at the household level; their support for men’s involvement in child care; their resistance to all forms of violence against women; their support for women’s human rights. A composite score of these different attitudes will be compiled and assessed. Disaggregated by sex. Household KAPB survey Y1 baseline, Y5 end-line M&E/R Advisor Burkina Faso 41 n/a n/a n/a 51 n/a Côte d’Ivoire n/a 45 n/a n/a 55 n/a Mauritania n/a n/a n/a n/a 25 n/a Niger 22 n/a n/a n/a 32 n/a Togo 43 n/a n/a n/a 53 n/a Total n/a n/a n/a n/a n/a n/a 14 Percent of women citing lack of information on FP methods as a key barrier to use (outcome indicator) Numerator: number of women citing lack of information on FP methods as a key barrier to use Denominator: number of women surveyed about key barriers to FP method use Disaggregated by sex. Baseline/End￾line survey Y1 & Y5 M&E Advisor Burkina Faso n/a n/a n/a TBD TBD TBD Côte d’Ivoire n/a n/a n/a TBD TBD TBD Mauritania n/a n/a n/a TBD TBD TBD Niger n/a n/a n/a TBD TBD TBD Togo n/a n/a n/a TBD TBD TBD Total n/a n/a n/a TBD TBD TBD 15 Percent of women who have discussed FP with Baseline/Endline survey M&E Advisor Burkina Faso n/a n/a n/a TBD TBD TBD STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 28 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP husbands/partners, friends/family within the last 15 days (outcome indicator) Numerator: number of women who have discussed FP with husband/partner, friends/family within the last 15 days Denominator: number of women surveyed about if they have discussed FP with husband/partners, friends/family within the last 15 days Disaggregated by sex. Y1 & Y5 Côte d’Ivoire n/a n/a n/a TBD TBD TBD Mauritania n/a n/a n/a TBD TBD TBD Niger n/a n/a n/a TBD TBD TBD Togo n/a n/a n/a TBD TBD TBD Total n/a n/a n/a TBD TBD TBD 16 Number of youth who participate in educational program on gender, FP, and SRH (output indicator) AgirPF will adapt EngenderHealth’s teen pregnancy prevention curriculum for use with youth in West Africa. This will be led by peer educators who would lead discussions as moderators with enhanced knowledge on FP. This indicator also includes those reached by group discussion on FP services, WAAF and sub-grantee organizations Disaggregated by sex. Sign-in sheets, activity reports Collect monthly report quarterly Sub-grantees and WAAF Burkina Faso 0 15000 15000 15000 15000 60000 USAID Sub-IR 3.2.2: Evidence and rights￾based policies Côte d’Ivoire adopted n/a n/a 15000 15000 15000 45000 Mauritania n/a n/a 15000 15000 15000 45000 Niger 0 15000 15000 15000 15000 60000 Togo 0 15000 15000 15000 15000 60000 Total 0 45000 75,000 75,000 75,000 270,000 17 Number of Site Walk￾Throughs (SWT) conducted (output indicator) Local community leaders (e.g., women’s group leaders, traditional leaders, youth leaders) will visit the health centers serving their community. Providers will show them the path of a FP client through the facility. Community leaders and providers will then identify barriers to access at the community and facility levels and develop Sign-in sheets, activity reports Collect monthly report quarterly Technical Director Burkina Faso 3 6 30 15 15 84 Côte d’Ivoire n/a n/a 40 20 20 100 Mauritania n/a n/a 5 8 10 20 Niger 2 4 20 10 10 56 Togo 3 6 20 10 10 59 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 29 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP and implement action plans to address those barriers. Total 8 16 115 63 65 329 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented 18 Number of HIPS/BPs for family planning and maternal and child health and/or HIV/AIDS incorporated into national health protocols or standards (outcome indicator) (USAID RDCS) Evidence of HIPs for family planning and maternal and child health and/or HIV/AIDS incorporated into national health protocols or standards Step 1: HIP accepted by MOH Step 2: HIP replicated by MOH Step 3: HIP scaled-up Step 4: HIP integrated into national guidelines We only count a target reached once the HIP is integrated into national guidelines, not for any previous steps. Disaggregate by type of HIP (FP, HIV/AIDS, MCH) National health protocols or standards Collect monthly report quarterly Country Manager; Regional policy advisor Burkina Faso n/a n/a n/a n/a 5 5 USAID IR 3.1: Best practices scaled up Côte d’Ivoire n/a n/a n/a n/a 4 4 Mauritania n/a n/a n/a n/a 3 3 Niger n/a n/a n/a n/a 5 5 Togo n/a n/a n/a n/a 5 5 Total n/a n/a n/a n/a n/a n/a Sub-result 2.1: Efficiency and effectiveness enhanced through adaptation and implementation 19 Number of HIPs piloted through implementation research (output indicator) (USAID RDCS) Implementation research (IR) have been conducted by the project. Implementation research focuses on understanding how programs are implemented, translated, replicated, and disseminated in “real￾world” settings. It expands the focus of traditional research from discovering what Implementation research reports Annually M&E/R Advisor Burkina Faso 0 0 1 2 0 2 USAID Sub-IR 3.1.1: High impact, evidence-based interventions piloted in target countries Côte d’Ivoire n/a 0 1 1 0 2 Mauritania n/a n/a 0 1 0 1 Niger 0 1 1 1 0 3 Togo 0 1 1 2 0 4 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 30 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP works to also discovering how the implementation works in specific contexts. Data Requirements: Evidence, in the form of reports or other outputs of IR and/or evidence of staff involvement in ongoing studies that operations research studies have been conducted Total 0 2 4 5 0 13 Sub-results 2.2: Lessons documented and disseminated from adaptation and implementation 20 Number of regional technical meetings organized and supported by AgirPF and its partners (output indicator) (USAID RDCS) Use USAID Definition once it becomes available Reports Annually M&E/R Advisor Burkina Faso n/a n/a n/a n/a n/a n/a Côte d’Ivoire n/a n/a n/a n/a n/a n/a Mauritania n/a n/a n/a n/a n/a n/a Niger n/a n/a n/a n/a n/a n/a Togo n/a n/a n/a n/a n/a n/a Regional 1 4 5 4 2 17 Total 1 4 5 4 2 17 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-results 3.1: Policy barriers identified and new/revised policies adopted and implemented 21a Number of policies or guidelines developed or changed with USG assistance to improve access to and use of family planning and reproductive health services Number of policies, laws and guidelines introduced or updated as a result of USG￾assistance related to improvement in family planning and reproductive health services Policies and guidelines Annually Policy and Advocacy Advisor Burkina Faso 0 0 2 2 0 3 USAID Sub-IR 3.2.2: Evidence and rights￾based policies adopted. Côte d’Ivoire n/a 0 2 2 0 3 Mauritania n/a n/a 2 2 0 3 Niger 0 0 2 3 0 3 Togo 0 0 2 2 0 3 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 31 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP (output indicator) (USAID RDCS) Total 0 0 10 11 0 15 21b Number of countries with a line item in the national budget for FP (USAID RDCS) Use USAID Definition once it becomes available This indicator is only measuring a line item in the national budget that the AgirPF project has affected. Burkina Faso, Niger, and Mauritania had preexisting line items, but AgirPF had not worked toward those. Activity on this indicator begins in PY3 and is only deemed a success if because of AgirPF activity either 1. A country that had no line item adds a line item, or 2. A country that had a line item increases the budgeted amount. Monitoring Report Collected quarterly reported annually Country Managers; Regional policy advisor; Contraceptive security advisor Burkina Faso n/a n/a 1 n/a n/a 1 USAID IR 3.2: Enabling environment at the national and regional levels strengthened Côte d’Ivoire n/a n/a 1 n/a n/a 1 Mauritania n/a n/a 1 n/a n/a 1 Niger n/a n/a 1 n/a n/a 1 Togo n/a n/a 1 n/a n/a 1 Total n/a n/a 5 n/a n/a 5 22 Number of advocacy presentations created or updated (in collaboration with Avenir Health and HP+) (output indicator) AgirPF will support countries to develop or update country-specific advocacy presentations, including RAPID models Disaggregated by theme of advocacy presentation Reports Quarterly Policy and Advocacy Advisor Burkina Faso 0 1 n/a n/a n/a 1 For Burkina Faso and Togo, the activity will be an update of RAPID models that were done in 2011. For Mauritania and Niger, this will be the first RAPID model done in those countries. Côte d’Ivoire n/a n/a 1 n/a n/a 1 Mauritania 1 n/a n/a n/a n/a 1 Niger 0 1 n/a n/a n/a 1 Togo 1 n/a n/a n/a n/a 1 Total 2 2 1 n/a n/a 5 23 Number of advocacy activities conducted (output indicator) AgirPF will support the initial launch of advocacy activities for the RAPID presentation to policy makers at the country-level. Disaggregated by geographic level (city, national), target population (National policymaker, Religious and Traditional leader, Local government official, Community leaders) Reports Quarterly Policy and Advocacy Advisor Burkina Faso 0 3 1 3 1 6 Côte d’Ivoire n/a n/a 3 3 1 5 Mauritania 3 1 1 3 1 7 Niger 0 3 1 3 1 6 Togo 3 1 1 3 1 7 STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 32 All AgirPF Indicator Indicator Description and Type Indicator Definition (including how measured, disaggregation) [NOTE: All indicators will include disaggregation by country and city where necessary. Additional categories of disaggregation are included in each definition.] Source of information / Data collection method and frequency Person(s) Responsible Target Notes and assumptions /USAID IR Ref. Country Year 1 Year 2 Year 3 Year 4 Year 5 Total LOP Total 6 8 7 15 5 31 24 Number of formal agreements (MoU, policy, declaration, etc.) that are signed at the regional level (annotate direction: regional to national or national to regional) (USAID RDCS) Use USAID Definition once it becomes available Monitoring Report Collected quarterly reported annually Country Managers, Regional Policy Advisor Burkina Faso n/a n/a n/a n/a n/a n/a USAID Sub-IR 3.1.2: Regional and national Côte d’Ivoire n/a n/a n/a n/a n/a n/a priorities harmonized Mauritania n/a n/a n/a n/a n/a n/a Niger n/a n/a n/a n/a n/a n/a Togo n/a n/a n/a n/a n/a n/a Regional n/a n/a 5 3 5 15 Total n/a n/a 5 3 5 15 Sub-results 3.2: Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management 25 Number of SDP reporting stock-outs of contraceptives per quarter (output indicator) SDP reporting stock-outs of contraceptives per quarter. Disaggregated by method. Health facility stock reports, inventories reports Collect monthly report quarterly Country Managers Burkina Faso TBD TBD TBD 0 TBD TBD Total LOP not necessary because is meaningless. Number of SDPs increase over time but stock out rate is presumed to decrease by 5% per year : 30% in Y1 to 10% in Y5 Côte d’Ivoire TBD TBD TBD TBD TBD TBD Mauritania TBD TBD TBD TBD TBD TBD Niger TBD TBD TBD TBD TBD TBD Togo TBD TBD TBD TBD TBD TBD Total TBD TBD TBD TBD TBD TBD10 10 We are working to recalculate the targets around stock-outs. STATEMENT OF WORK Performance Evaluation for Agir Pour La Planification Familiale – AgirPF 33 ANNEX B: AGIRPF RESEARCH AGENDA Microsoft Word 97 - 2003 Document ANNEX C: AGIRPF RESEARCH IMPLEMENTATION PLAN Microsoft Word 97 - 2003 Document EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation Inception Report International Business and Technical Consultants, Inc. February 2017 This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of States Government. EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation Inception Report International Business and Technical Consultants, Inc. Disclaimer This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of States Government. TABLE OF CONTENTS INTRODUCTION ............................................................................................................................................................ 1 Preliminary Findings of the Desk Review ................................................................................................................ 2 EVALUATION QUESTIONS.......................................................................................................................................... 3 DATA COLLECTION METHODS AND INSTRUMENTS..................................................................................... 4 SAMPLING STRATEGY ................................................................................................................................................... 5 Quantitative Data ......................................................................................................................................................... 6 Qualitative Data ............................................................................................................................................................ 6 LIMITATIONS TO THE EVALUATION...................................................................................................................... 7 EVALUATION MATRIX .................................................................................................................................................. 8 IBTCI March 9, 2018 – 1 – INTRODUCTION The West Africa region includes 21 countries with a population of approximately 359 million (Population Reference Bureau (PRB), 2016). The region has the lowest modern contraception use worldwide. Only 12% of married women of reproductive health (WRH) age were using a modern contraceptive method in 2016 compared to the global average of 56% (PRB, 2016). Consequently, the region has the highest total fertility rate (TFR) estimated` at 5.4 children per woman (PRB, 2016). This high level of fertility, among which 26% are unintended (Sedgh, Singh, & Hussain, 2014), coupled with high adolescent fertility (111 births per 1000 women ages 15-19 compared with 52 worldwide) and persistent decrease in child and maternal mortality rates, contribute to high population growth rates. The West Africa population is expected to increase from 359 million in 2016 to 515 million by mid-2030 and 800 million by mid-2050 (PRB, 2016). Such population volumes constitute a threat for the future of the region (available resources, economic growth and population wellbeing). In response to the high level of unmet need in Francophone West Africa, nine governments of Francophone West African countries and their technical partners and financial resources governments launched the Ouagadougou Partnership in February 2011 in Ouagadougou, Burkina Faso. This initiative includes the government of Benin, Burkina Faso, Côte d’Ivoire, Guinea, Mali, Mauritania, Niger, Senegal and Togo. The main objective of the Partnership is to reach at least 2.2 million additional family planning (FP) method users in the nine countries by 2020. The national action plans of the nine countries encompass two objectives: 1) accelerate the achievement of their national goals for modern Contraceptive Prevalence Rate (CPR); and 2) reach at least an additional 1 million women by 2015. These action plans mapped their priority steps for strengthening national FP programs. In West Africa, compared to Anglophone and Lusophone countries, the Francophone countries have the highest rates of maternal and child mortality, the highest fertility rates, and lowest contraceptive prevalence rate. Against this backdrop, USAID/West Africa Regional Health Office (RHO) funded the AgirPF. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. The project works closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP. The approach is to leverage FP momentum, activating the “grassroots” to increase access to, quality of, and demand for FP, and working with the RHO and countries to adapt evidence-based practices (Result 2); learn about these practices (Sub-Result 2.2); feedback learning to national actors in the form of project/RHO advocacy for adoption and scale-up, grassroots-led advocacy, and information that USAID can use to rationalize policies and contraceptive logistics (Result 3). AgirPF strengthens public, private, and NGO facilities to provide a range of FP services (Result1), including integrated FP/maternal health services and services for youth/men (Sub-Result 1.1). To bring FP services to underserved communities, AgirPF supports mobile outreach services; brings health fairs to industries and community sites; and offers “city-based services,” an adaptation of EngenderHealth￾managed Community-based Distribution (CBD) in Togo. To lower client cost, AgirPF provides dedicated FP services at low/no cost (special FP days) in each city. To solve logistics issues and estimate commodity needs, AgirPF assists facilities to use Client Oriented Provider Efficiency (COPE) for Contraceptive Security and Ministries of Health’s Contraceptive Procurement Teams to use Reality Check for contraceptive quantification (Sub-Result 3.2). AgirPF also provides training and refresher trainings to healthcare providers in FP service delivery, including infection prevention (IP), FP counseling (using the rapport building, exploration, decision making, IBTCI March 9, 2018 – 2 – and implementing the decision (REDI) framework), facilitative supervision and gender sensitization. AgirPF supports community leaders with FP advocacy activities. The theory of change for the AgirPF activity is that if a) the delivery of quality FP information, products, and services are strengthened and expanded; b) evidence-based service delivery approaches selected, adapted, and implemented; and c) efforts to remove policy barriers and improve contraceptive commodity security coordinated, then access to and use of FP services will increase in urban and peri￾urban areas in Burkina Faso, Côte d’Ivoire, Niger and Togo (Mauritania is not included). As such, this evaluation uses the results framework below as a guide for the overall assessment Figure 1: AgirPF Results Framework Preliminary Findings of the Desk Review A desk review of AgirPF was conducted, consisting of an in-depth analysis of project quarterly reports and excel data files. These files contained reported project results, and provided an estimate of performance from inception until the third quarter of the third project year. The findings of the desk review were organized according to the given evaluation questions, and the main results included:  Overall, AgirPF has fallen substantially short of its targets in reaching new method and overall FP users, resulting in low CYP achievement. Given the current trajectory of the project, if level of effort remains constant, the project will not be on track to meet its overall goals for CYP.  While training targets for family planning are largely met both at the overall project level and across countries, the number of family planning special days conducted are particularly weak for AGIR-PF Objective Increase access to and use of quality family planning services in select urban and peri-urban areas of four francophone West African countries AGIR-PF Result 1: Delivery of quality family planning information, products, and services strengthened and expanded Sub-Result 1.1 Partners strengthened to implement evidence-based approaches and deliver quality family planning services Sub-Result 1.2 Local leaders, civil society, service providers, municipal government support and promote family planning AGIR-PF Result 2: Evidence-based service delivery approaches selected, adapted, and implemented Sub-Result 2.1 Efficiency and effectiveness of approaches enhanced through the adaptation and implementation process Sub-Result 2.2 Lessons documented and disseminated on learning from adaption and implementation processes and experiences AGIR-PF Result 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-Result 3.1 Operational policy barriers identified and new/revised policies adopted and implemented Sub-Result 3.2 Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees IBTCI March 9, 2018 – 3 – PY3. Poor performance on FP special days in Cote d’Ivoire and Togo contribute to the overall poor performance for the project.  Across its individual countries and through cross-country activities, AgirPF has advanced a number of high-impact practices including integrating family planning into postpartum care, increasing community based distribution of family planning through training of community health workers and mainstreaming youth into family planning services by training providers in youth-friendly services. However, application of these practices is variable and it is unclear how implementation actually takes place.  Project reports indicate efforts at grassroots support and skill around policy advocacy issues, but there is no information on “next steps” after initial engagement takes place. Many activities are left off at the “action plan” stage, but there is little evidence that the project conducts any follow￾up activities or support for implementation of the action plans. This leaves open the question of what actual impact the project has had on removing barriers to policies.  There are indications that there were difficulties in overall project management, staffing and roll￾out at the global level. These findings point to the need for validation of project performance data as well as further investigation into the management of the project. Moreover, given the scope of AgirPF’s work with local partners, stakeholders and policy makers in 4 different countries, in-depth interviews with various individuals is critical to understanding the project’s management capabilities. The desk review, while thorough, highlighted key areas for further investigation, including:  Facility-based records reviews in a sample of AgirPF target facilities, as well as a comparison group of non-intervention sites, will yield important information on the impact of AgirPF in increasing accessibility to and uptake of family planning for new method adopters  Interviews with key informants in the governments, advocacy groups and other key stakeholders will provide detail on the impact of AgirPF’s engagement on supporting policy change and the successes and challenges of the AgirPF approach to policy advocacy. These interviews will also help explore facilitating factors and barriers that either permitted or hindered the achievement of targets. In tandem with a national policy review, their feedback will help determine what specific role AgirPF played in moving family planning policy forward in each country  Interviews with providers and health managers in the AgirPF facilities and districts will provide critical information on the effectiveness and sustainability of AgirPF approaches for implementing high-impact best practices in family planning  Interviews with AgirPF project and partner staff will inform the management successes and challenges of AgirPF in each country and across partner organizations. EVALUATION QUESTIONS The mid-term evaluation aims to answer six specific questions about the process and success of the AgirPF project. These questions are designed not only to answer the question of whether or not AgirPF is meeting its performance objectives, but also how the project is being managed across its country IBTCI March 9, 2018 – 4 – programs, partner organizations and stakeholder relationships. Specifically, this evaluation will aim to answer: 1. How has AgirPF performed (analysis of monitoring data)? 2. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Underlying these evaluation questions are a series of queries related to the process and outputs of the AgirPF project. Namely, the investigation will serve to clarify if activities were carried out as planned, how well they were conducted, and if expected changes occurred at the program level in terms of improved access to family planning and progress on changing policy barriers to family planning in each country. DATA COLLECTION METHODS AND INSTRUMENTS This section describes the specific data collection methods and instruments necessary to answering the evaluation questions. Table 1 summarizes these methods, which are explained in detail below. Facility records review - Health management information system data from a sample of AgirPF facilities in Burkina Faso, Cote d’Ivoire, Niger and Togo will be reviewed to examine the impact of the project’s work on provision of family planning. Monthly family planning data will be collected from 6 months prior to the onset of activities, through the most recent month at time of data collection. Data specific to 1) total number of family planning users, 2) total number of new method acceptors and 3) method mix will be downloaded from HMIS systems and analyzed for changes in trends over time. These data will serve to supplement and validate the project-level reporting that was analyzed as part of the desk review. Appendix 1 provides a database shell into which HMIS data will be entered for analysis. Facility audit Provider interviews – Health care providers - including doctors, nurses and auxiliary nurse midwives- from a sample AgirPF facilities will be interviewed to understand their experience with training and support under the project. They will also be queried on the implementation of high￾impact practices such as postpartum family planning and FP integration into post abortion care, as well as their experience with tools such as COPE for Contraceptive Security and other relevant planning tools. Providers will also be asked about perceived successes, challenges and lessons learned from working with the AgirPF project to advance family planning programming. Only those who were involved in AgirPF trainings or facility-based intervention will be included in the interviews. A minimum of two providers from each sampled facility will be interviewed. Appendix 2 is the interview guide to be used with providers. Local health manager interviews– District and regional health managers (including reproductive health coordinators) from each intervention country, with whom AgirPF IBTCI March 9, 2018 – 5 – coordinated trainings, built capacity and engaged for change, will be interviewed to understand their experience with the program. Specifically, they will be interviewed to understand the effectiveness of AgirPF’s support in regards to training, coordination and implementation of high impact practices at the local level. Utility and sustained use of tools such as COPE, Reality Check, LMIS training and other relevant approaches will be explored from the perspective of the health management teams in each country. Health managers will also be asked about perceived successes, challenges and lessons learned from working with the AgirPF project to advance family planning programming. At least one health manager from each district and region of AgirPF intervention areas in each country will be interviewed. Appendix 3 is the interview guide to be used with health managers. Ministry of Health interviews – Stakeholders from the Ministry of Health will be interviewed to examine the effectiveness of AgirPF’s overall coordination with government partners at the national level. MOH Stakeholders will also be asked about the role that AgirPF has played in removing policy barriers to family planning at the national level. All relevant MOH stakeholders from each country will be interviewed. Appendix 4 is the interview guide to be used with MOH staff. Partner organization interviews–In order to understand AgirPF’s performance in engaging with and managing activities with partner organizations, members of local and international consortium partners will be interviewed. Participants will be asked about their experience working with AgirPF and about success and challenges of coordinating efforts with the project. A particular focus will be put on staff at “centers for excellence”, in which AgirPF attempted to institutionalize training systems, as well as local advocacy organizations, with whom AgirPF engaged to advance policy changes in FP. Appendix 5 is the interview guide to be used with partner organization staff. Appendix 6 is the interview guide to be used with advocacy organization staff. AgirPF country staff interviews–Interviews with AgirPF country staff will be used to understand the performance of headquarters and regional management staff with regards to overall conduct of the program. In particular, issues related to planning, execution and management of implementation will be investigated to understand the adequacy of AgirPF’s approach. Appendix 7 is the interview guide to be used with AGIRPF organization staff. Policy review – All relevant country-level policies related to family planning will be reviewed in the period of intervention to examine any changes or movement in reducing barriers to FP. Review of formal policies as well as reports from MOH stakeholders will inform the policy review Project files review – The project files were previously audited as part of the desk review. However, they will be re-examined in light of the broader evaluation as a means of triangulating data on FP provision and program implementation within each country. SAMPLING STRATEGY IBTCI March 9, 2018 – 6 – Quantitative Data The facility level records review will take place in a subset of intervention facilities in which AgirPF carried out programs. These facilities will be selected on the basis of a simple random sample of facilities in each district of AgirPF intervention. Thirty percent of facilities in each intervention district of each focus country (Burkina Faso, Cote d’Ivoire, Niger and Togo) will be selected for inclusion in the sample. HMIS records from these facilities in each country will be accessed at the district level, with the cooperation and assistance of district health managers. The HMIS records will date from 6 months prior to AgirPF training implementation (variable depending on country), up and through June 2016. Within each facility, a purposive sample of providers will be chosen for an in-depth interview. These providers will be chosen on the basis of their participation in AgirPF-related activities and availability for interview on the selected day. APPENDIX 1: Data Template for HMIS Qualitative Data For all individual interviews, a purposive sample of participants will be chosen from among the following groups:  Health providers: 1-2 participants per sampled facility (please see above for details). Appendix 2: Provider Interview Guide  Local health managers: 1-2 individuals per AgirPF districts in each country will be selected for interview. These will only include individuals with him AgirPF worked in each country, to be verified by in-country AgirPF staff Appendix 3: Health Manager Interview Guide  Ministry of Health stakeholders: Key MOH stakeholders, as identified by AgirPF in-country staff, will be interviewed in each focus country. Appendix 4: MOH Interview Guide  Partner organization interviews: Individuals from partner organizations, defined as both international and local consortium partners and local policy-advocacy groups, will be interviewed. Participants will be selected with the guidance of in-country AgirPF staff, and selected on their basis of partnership and collaboration with AgirPF for implementation activities. For local advocacy groups, individuals who underwent training and planning sessions with AgirPF will be selected. From each organization, a minimum of 2 staff members will be selected for interview. When available and relevant, additional staff from each organization will be interviewed. Appendix 5: Consortium Partners Interview Guide Appendix 6: Advocacy Partners Interview Guide  AgirPF country staff interviews – Key EngenderHealth staff from each AgirPF country program will be selected for interview on the basis of their experience with the management at EngenderHealth regional and headquarters level. This includes heads of each department in focus countries (finance, M&E, operation, programs, etc.), as well as the acting manager in each country. Appendix 7: AgirPF Staff Interview Guide IBTCI March 9, 2018 – 7 – Policy and project records review will rely upon documents gathered from AgirPF and respective MOH in each country. LIMITATIONS TO THE EVALUATION This evaluation will use a variety of methods to answer the question of AgirPF’s success at the midterm point of the project. However, findings will be limited primarily to input, process, output and short-term outcome level indicators. Furthermore, given the limited scope of the evaluation and timeline, we will not be able to assess any comparison sites to examine attribution of AgirPF’s work nor assess the quality of their work directly through provider or client observations or interviews. IBTCI March 9, 2018 – 8 – EVALUATION MATRIX The matrix in Table 2 describes each primary evaluation question with more detail, including secondary questions under each category as well as the 1) measure type, 2) level of intervention and analysis, 3) data courses and 4) proposed data analysis technique. Further devolution of each question is evident in the related data collection tools in Appendices. Evaluation Questions1 Measure type Level of Intervention and analysis Data sources Data analysis techniques Primary Q1: How has AgirPF performed? Sub Q 1.1: How has AgirPF performed against targets for CYP in each country? Quantitative Project level / district level Project records/ HMIS Trend analysis via excel Sub Q 1.2: How has AgirPF performed against targets for reaching new method users in each country? Quantitative Project level/ district level Project records/ HMIS Trend analysis via excel Sub Q 1.3: How has AgirPF performed against targets for reaching total family planning method users in each country? Quantitative Project level/ district level Project records/ HMIS Trend analysis via excel Sub Q 1.4: Did family planning provision increase after AgirPF intervention? Quantitative Facility/ District/ Country level HMIS Trend analysis via excel Primary Q2: Which high impact/best practices (HIPs/BPs) have AgirPF advanced? Sub Q 2.1 What HIPs did AgirPF report implementing across all the countries in their project documents? Qualitative Project level Project records Records review/ summary Sub Q 2.2 What HIPS did health care providers report implementing as a result of AgirPF support? Qualitative Facility level Provider interviews Qualitative analysis Sub Q 2.3 What HIPS did local health managers report implementing as a result of AgirPF support? Qualitative District/ Regional level Health manager interviews Qualitative analysis Sub Q 2.4 Were HIPS promoted with partner organizations as part of AgirPF’s support? Qualitative Country/ Project level Partner interviews Qualitative analysis Sub Q 2.5 Were HIPS promoted with MOH stakeholders as part of AgirPF’s support? Qualitative Country level MOH interviews Qualitative analysis IBTCI March 9, 2018 – 9 – Evaluation Questions1 Measure type Level of Intervention and analysis Data sources Data analysis techniques Primary Q3: To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Sub Q 3.1 Are the elements of IR1 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Qualitative Project level Project records/ key informant interviews Project document review/ qualitative analysis Sub Q 3.2 Are the elements of IR2 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Qualitative Project level Project records/ key informant interviews Project document review/ qualitative analysis Sub Q 3.3 Are the elements of IR3 in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? Qualitative Project level Project records/ key informant interviews Project document review/ qualitative analysis Primary Question 4: For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? Sub Q 4.1 To what extent did AgirPF’s support aid advocacy partners in removing policy barriers to FP? Qualitative Country (policy)/ partner Policy records/ key informant interviews Policy document review/ qualitative analysis Sub Q 4.2 To what extent did AgirPF’s support aid MOH partners in removing policy barriers to FP? Qualitative Country (policy)/ MOH Policy records/ key informant interviews Policy document review/ qualitative analysis Sub Q 4.3 What are any tangible changes in FP policy attributed to AgirPF support? Qualitative Country (policy) Policy records/ key informant interviews Policy document review/ qualitative analysis Primary Question 5: What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? IBTCI March 9, 2018 – 10 – Evaluation Questions1 Measure type Level of Intervention and analysis Data sources Data analysis techniques Sub Q 5.1 What were program successes useful for advancing family planning programming? Qualitative Project Project records/ key informant interviews Qualitative analysis Sub Q 5.1 What were program challenges useful for adjusting family planning programming in the future? Qualitative Project Project records/ key informant interviews Qualitative analysis Primary Question 6: How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Sub Q 6.1 How has AgirPF managed Agir staff in focus countries? (regional AgirPF performance) Qualitative Country/ Project Project records/ key informant interviews Qualitative analysis Sub Q 6.1 How has AgirPF managed Agir staff in focus countries? (headquarter AgirPF performance) Qualitative Country/ Project Project records/ key informant interviews Qualitative analysis Sub Q 6.1 How has AgirPF managed partners in focus countries? Qualitative Project/ Partner Project records/ key informant interviews Qualitative analysis 1. These primary and sub questions are general, and are further devolved through the data collection tools IBTCI March 9, 2018 – 1 – EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation Desk Review Findings Report International Business and Technical Consultants, Inc. January 2017 This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of States Government. EVIDENCE FOR DEVELOPMENT E4D AGIR PF Midterm Evaluation Draft Desk Review Findings Report International Business and Technical Consultants, Inc. Disclaimer This report is made possible by the support of the American people through the United States Agency for International Development (USAID) under AID-624-C-15-00001. The contents of this report are the sole responsibility of States Government. TABLE OF CONTENTS ACRONYMS ....................................................................................................................................................................... 1 INTRODUCTION ............................................................................................................................................................ 2 Purpose of the Evaluation ................................................................................................................................................ 3 Role of Desk Review ........................................................................................................................................................ 3 METHODS .......................................................................................................................................................................... 3 RESULTS .............................................................................................................................................................................. 5 AgirPF Overall Program Results .................................................................................................................................... 5 Result 1: Delivery of quality FP information, products, and services strengthened and expanded ........................... 6 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented .................................. 8 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated ................................................................................................................................................................................................ 9 COUNTRY BY COUNTRY ANALYSIS .................................................................................................................... 10 Togo .................................................................................................................................................................................... 11 Result 1: Delivery of quality FP information, products, and services strengthened and expanded ........................11 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented ...............................12 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated .........13 Burkina Faso ...................................................................................................................................................................... 14 Result 1: Delivery of quality FP information, products, and services strengthened and expanded ........................14 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented ...............................16 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated .........16 Niger ................................................................................................................................................................................... 17 Result 1: Delivery of quality FP information, products, and services strengthened and expanded ........................17 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented ...............................18 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated .........19 Cote d’Ivoire..................................................................................................................................................................... 20 Result 1: Delivery of quality FP information, products, and services strengthened and expanded ........................20 Result R2: Evidence-based service delivery approaches selected, adapted, and implemented ...............................21 Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated .........21 SUMMARY OF ACHIEVEMENT TOWARDS EVALUATION QUESTIONS .................................................. 22 AREAS FOR INVESTIGATION FOR THE BROADER EVALUATION ............................................................. 25 APPENDICES .................................................................................................................................................................... 25 Appendix 1: Bibliography and Project Files used for the Desk Review .............................................................. 25 Appendix 2: In-depth background sections by country .......................................................................................... 27 IBTCI March 9, 2018 – 1 – ACRONYMS ABBEF Association Burkinabè pour le Bien Être Familial CHW Community Health Worker COPE Client Oriented Provider Efficient CPT Contraceptive Procurement Table CYP Couple Years Protection EH EngenderHealth FP Family Planning HPP Health Policy Project LARC Long-acting reversible contraceptive LMIS Logistics management information system MOH Ministry of Health MOU Memorandum of Understanding NRHP/PF National Reproductive Health Program OCAT Organizational Capacity Assessment Tool PAC Post-abortion care PPIUD Postpartum intrauterine device PPFP Postpartum family planning REDI RHO Rapport Building, Exploration, Decision Making and Implementing Decisions Regional Health Office (USAID/West Africa) SBCC Social and Behavior Change Communication SRH Sexual and Reproductive Health SWT TOT UNFPA Site Walk Through Training of trainers United Nations Population Fund URD Demographic Research Unit USAID/WA U.S. Agency for International Development/West Africa WAAF West Africa Ambassador’s Fund WAHO West African Health Organization WHO World Health Organization IBTCI March 9, 2018 – 2 – INTRODUCTION The West Africa region includes 21 countries with a population of approximately 359 million (Population Reference Bureau (PRB), 2016). The region has the lowest modern contraception use worldwide. Only 12% of married women of reproductive health (WRH) age were using a modern contraceptive method in 2016 compared to the global average of 56% (PRB, 2016). Consequently, the region has the highest total fertility rate (TFR) estimated at 5.4 children per woman (PRB, 2016). This high level of fertility, among which 26% are unintended (Sedgh, Singh, &Hussain, 2014), coupled with high adolescent fertility (111 births per 1000 women ages 15-19 compared with 52 worldwide) and persistent decrease in child and maternal mortality rates, contribute to high population growth rates. The West Africa population is expected to increase from 359 million in 2016 to 515 million by mid-2030 and 800 million by mid-2050 (PRB, 2016). Such population volumes constitute a threat for the future of the region (available resources, economic growth and population wellbeing). In response to the high level of unmet need in Francophone West Africa, nine governments of Francophone West African countries and their technical partners and financial resources governments launched the Ouagadougou Partnership in February 2011 in Ouagadougou, Burkina Faso. This initiative includes the government of Benin, Burkina Faso, Côte d’Ivoire, Guinea, Mali, Mauritania, Niger, Senegal and Togo. The main objective of the Partnership is to reach at least 2.2 million additional family planning (FP) method users in the nine countries by 2020. The national action plans of the nine countries encompass two objectives: 1) accelerate the achievement of their national goals for modern Contraceptive Prevalence Rate (CPR); and 2) reach at least an additional 1 million women by 2015. These action plans mapped their priority steps for strengthening national FP programs. In West Africa, compared to Anglophone and Lusophone countries, the Francophone countries have the highest rates of maternal and child mortality, the highest fertility rates, and lowest contraceptive prevalence rate. Against this backdrop, USAID/West Africa Regional Health Office (RHO) funded the AgirPF Project. The goal of AgirPF is to enable women of reproductive age (WRA) (15–49) to make, and voluntarily act on, informed decisions about FP, saving women’s lives in selected urban and peri-urban areas of five francophone West African countries: Burkina Faso, Côte d’Ivoire, Mauritania, Niger and Togo. The project works closely with Ministries of Health (MOHs) and other local partners to support the national action plans for strengthening FP. The approach is to leverage FP momentum, activating the “grassroots” to increase access to, quality of, and demand for FP, and working with the RHO and countries to adapt evidence-based practices (Result 2); learn about these practices (Sub-Result 2.2); feedback learning to national actors in the form of project/RHO advocacy for adoption and scale-up, grassroots-led advocacy, and information that USAID can use to rationalize policies and contraceptive logistics (Result 3). AgirPF strengthens public, private, and NGO facilities to provide a range of FP services (Result 1), including integrated FP/maternal health services and services for youth/men (Sub-Result 1.1). To bring FP services to underserved communities, AgirPF supports mobile outreach services; brings health fairs to industries and community sites; and offers “city-based services,” an adaptation of EngenderHealth￾managed Community-based Distribution (CBD) in Togo. To lower client cost, AgirPF provides dedicated FP services at low/no cost (special FP days) in each city. To solve logistics issues and estimate commodity needs, AgirPF assists facilities to use Client Oriented Provider Efficiency (COPE) for Contraceptive Security and Ministries of Health’s Contraceptive Procurement Teams to use Reality Check for contraceptive quantification (Sub-Result 3.2). AgirPF also provides training and refresher trainings to healthcare providers in FP service delivery, including infection prevention (IP), FP counseling (using the rapport building, exploration, decision making, and implementing the decision (REDI) framework), facilitative supervision and gender sensitization. AgirPF supports community leaders with FP advocacy activities. Purpose of the Evaluation The mid-term evaluation aims to answer six specific questions about the process and success of the AgirPF project. While these aims may not all be specifically addressed in the course of the desk review, they do provide an outline for the analysis of the existing project data, as presented in this report. 1. How has AgirPF performed (analysis of monitoring data)? 2. Which high impact/best practices (HIPs/BPs) has AgirPF advanced? [HIPs: Integrating Family Planning into Post-partum and Post Abortion Care, Community-based Distribution of Family Planning, Mainstreaming Youth into Family Planning services] 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? [Successes][Challenges] [lessons learned] 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Role of Desk Review The desk review is designed to accomplish two main objectives. The primary is to create a summary of AgirPF performance using existing project documents. The review identifies gaps in data and helps to inform the methodology for the formal evaluation. Moreover, the desk review provides background context for each AgirPF country, as relates to family planning service delivery, best practices and policies. To the extent possible, the desk review will attempt to provide a framework for answer the aforementioned evaluation questions. METHODS This desk review consists of several key methodologies, as detailed below.  Literature review: A thorough review of both the peer-reviewed and grey literature was carried out for the general West Africa region as well as for each individual country of Burkina Faso, Cote d’Ivoire, Niger and Togo. The literature review focused on family planning trends in each area, as well as any policies and health systems realities that may affect the delivery of and access to family planning methods.  Program files review: AgirPF program files, consisting of quarterly reports and related data spreadsheets for the up to and including the third quarter of the third project year, were shared with the evaluation team. Data from these program files were analyzed to assess the program’s performance in family planning service delivery against stated targets. Reports were also assessed for specific examples of activities related to policy, advocacy and capacity building. The original AgirPF performance monitoring plan was also shared with evaluators, and serves as a guideline for the overall evaluation activity. Unfortunately, Agir PF did not share any proposals or program plans, so it was not possible in the context of the desk review to independently verify the activities performed each quarter against the original program proposal. All program files that fed into this report are listed in Appendix 1.  Indicator analysis: Individual program indicators were collated and analyzed to determine the projects performance against stated targets. Given the phased approach to roll-out and the delayed project activities in Year 1, AgirPF did not formally commence activities until July 2014. Therefore, the following criteria were used to determine which indicators to analyze: o The indicators must have data for all four countries over the same period of time (namely, PY2 and PY3) o The indicators must represent data that can meaningfully be analyzed numerically. For example, the “number of family planning users reached” is an appropriate numerical indicator. On the other hand, “number of policies changed” may not be meaningful, as one very well-placed policy could be more impactful than 5 lesser policy changes. o The indicators must be clearly defined and valid on their surface.  Country-by-country analysis: The analysis of AgirPF was carried out at two levels, using both literature and project files. First, the overall performance of the entire AgirPF program was assessed, against targets. Then, for each individual country, country-level statistics and quarterly reports were analyzed to glean specific achievements for Burkina Faso, Cote d’Ivoire, Niger and Togo, separately. This review allows evaluators to demonstrate the project’s overall progress, as well as to identify the specific problem areas of performance, at the country level.  AgirPF Results framework as guide: The theory of change for the AgirPF activity is that if a) the delivery of quality FP information, products, and services are strengthened and expanded; b) evidence-based service delivery approaches selected, adapted, and implemented; and c) efforts to remove policy barriers and improve contraceptive commodity security coordinated, then access to and use of FP services will increase in urban and peri-urban areas in Burkina Faso, Côte d’Ivoire, Niger and Togo (Mauritania is not included). As such, this desk review uses the results framework below as a guide for the overall assessment IBTCI March 9, 2018 – 5 – Figure 1: AgirPF Results Framework RESULTS This section presents the overall findings of the desk review, consisting of the literature review, project file analysis and indicator analysis. The structure of this section follows the three project results whereby a summary of findings from Project Years (PY) 2 and 3 are presented by each result. This structure is repeated for tall AgirPF countries and each individual country project. Due to delays in activity roll-out, there are no results as such for Project Year 1. However, the overall activities for PY1 are summarized below. AgirPF Overall Program Results As a preamble to this discussion, it is noteworthy that AgirPF faced critical delays in project roll-out, which shaped both program results and the ensuing discussion about them in this desk review. Due to a number of delays in implementation in PY1, the majority of service delivery activities did not get underway in the first year. Notably, there were significant delays in developing and executing baseline data collection, as well as issues with EngenderHealth’s registration in certain countries (Mauritania, for example). As a consequence, this review will only be presenting service delivery data from PYs 2 and 3, as there were no significant family planning activities in the first year. PY1 consisted mainly of project set-up activities including opening offices, engaging ministries of health (MOH) through memoranda of understanding (MOU), recruiting staff and conducting the baseline survey AGIR-PF Objective Increase access to and use of quality family planning services in select urban and peri￾urban areas of four francophone West African countries AGIR-PF Result 1: Delivery of quality family planning information, products, and services strengthened and expanded Sub-Result 1.1 Partners strengthened to implement evidence-based approaches and deliver quality family planning services Sub-Result 1.2 Local leaders, civil society, service providers, municipal government support and promote family planning AGIR-PF Result 2: Evidence-based service delivery approaches selected, adapted, and implemented Sub-Result 2.1 Efficiency and effectiveness of approaches enhanced through the adaptation and implementation process Sub-Result 2.2 Lessons documented and disseminated on learning from adaption and implementation processes and experiences AGIR-PF Result 3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Sub-Result 3.1 Operational policy barriers identified and new/revised policies adopted and implemented Sub-Result 3.2 Contraceptive commodity needs identified and coordinated among partners and country commodity security and logistics management committees activities. Some key groundwork activities did take place in PY1 including situational analyses of family planning needs in each country, participation in local technical working groups, where appropriate. The largest delay in PY1 was due to the lag in baseline data collection activities. As part of the broader evaluation, it would be important to dive deeper into program management strategies to understand why such delays occurred. Analysis of couple years protection (CYP) is the best measure that this project has to assess its own achievements against targets. Couple years protection represents the culmination and overall reach of project activities and serves as a bellwether for performance. As seen in Figure 2 below, the project fell quite short of its CYP target for PY2, achieving only 30% of its intended goal of 641,729 couple years of protection. Performance in PY3 was improved, but by the end of the third quarter, they had only achieved 58% of their intended target for that year. It is important to note that the results for PY3 represent only the first 3 quarters of the year. Therefore, as a to understand achievement against targets, a general rule of thumb is that if the project has not reached at least 75% of its intended target for PY3 in this and all subsequent graphs, it should be considered to be behind on its performance. Figure 2: Overall AgirPF CYP, results vs targets Result 1: Delivery of quality FP information, products, and services strengthened and expanded The project provided data on both overall FP users served, as well as new method users in each project year (Figure 3), which provides a portrait of the projects services reach. In PY2, the overall AgirPF project failed to achieve its stated targets for total FP users reached (72% of target met) and reached only 44% of its target for new method users. In PY3, performance for total FP users was worse than the year before, meeting only 24% of intended targets. Performance against targets for new method users in PY3 was improved, reaching 63% of targets by the third quarter. Keep in mind, that for PY3, neither indicator reached the 75% threshold thus indicating that the project is not on track to reach its PY3 goals. IBTCI March 9, 2018 – 7 – Figure 3: Overall AgirPF total FP users and new method users, results vs targets Figure 4 shows that the method mix distributed in both PY2 and PY3 is skewed heavily towards long￾acting reversible contraceptives. Implants were the leading method chosen by AgirPF clients in both PY2 (60%) and PY3 (56). In both years, implants were followed by IUD and injectables. Figure 4: Overall AgirPF contraceptive method mix for PY2 and PY3 As a means of assessing the operations and process of the project in service delivery activities, two indicators representing family planning training and family planning outreach (“FP days”) were assessed against targets (Figure 5). In both PY2 the overall AgirPF accomplishments surpassed its targets for the number of people trained in family planning, 2,243 people, and appears to be on track to meet its targets for PY3, having reached 84% of its goal (1,498 people) by the third quarter. In PY2, the project also exceeded its stated targets for family planning outreach activities through special FP days (116 days); however, by the third quarter of PY3 only 36% of the annual target for the year had been achieved by the third quarter. So, results are inconsistent depending on the metric used. Figure 5: Overall AgirPF Training and FP days: Results vs Targets Throughout the conduct of these activities, AgirPF reports engaging local leaders and stakeholders such as MOH staff, regional and district managers and health management teams in family planning activities in order to strengthen their overall capacity to deliver quality FP services. Engaging local stakeholders also contributes to long-term sustainability of activities. Result R2: Evidence-based service delivery approaches selected, adapted, and implemented The project files provide numerical indicators for a number of activities under Result 2. It would not be meaningful to compare these indicators to targets, per se, as one successful approach could address any number of issues that were failed by numerous other approaches. Instead, below is a description of the major evidence-based approaches that were adapted and implemented across AgirPF countries, several of these are designated high-impact practices.  Training, equipping and supporting community health workers to provide a wide range of FP methods  Supporting mobile outreach service delivery to provide a wide range of contraceptives, including long-acting reversible contraceptives  Integrating sexual and reproductive rights and gender into technical and counselling curricula for providers  Integrating FP services into postpartum services by building partner capacity to provide postpartum FP (PPFP) counselling and services  Ensuring service quality using EngenderHealth’s Site Walk-Through (SWT) approach  Developing AgirPF’s social and behavior change communication strategy in collaboration with Camber Collective  Providing technical and financial support for the organization of an SRH good practices selection and documentation workshop aimed at supporting country MOHs team and their SRH partners and local organizations to analyze good practices ongoing in their countries, selecting those that are considered proven, promising or emerging and documenting them. Table 1 provides additional details on the type of best practices and evidence-based trainings that AgirPF delivered across the 4 project countries. Targets and country-level break-downs of data were not available. This table shows that AgirPF was engaged in disseminating a broad range of evidence-based approaches and high-impact practices through training and support in a variety of family planning areas. Table 1: Number of people trained in different FP approaches Type of Training # ofpeople trained in PY2 # of people trained in PY2 Counseling REDI 390 126 PPPF 255 105 PPIUD 271 86 Contraceptive Logistics/Logistics Management Information Systems 0 151 New integrated FP Curriculum 19 161 COPE 277 260 3Is 12 0 HIV/AIDS 51 24 Advocacy 204 0 MCH 0 0 322 36 COPE for Contraceptive Security 27 0 Facilitative supervision 8 54 Spectrum 12 85 TOT in FP on EH approaches, tools, resources & policies. 42 28 OCAT 0 0 Gender 17 50 Youth Sexual and Reproductive Health 60 0 AgirPF Health Information System 73 0 FP counseling and services training in PAC facilities 137 64 Train health service providers on group and individual FP counseling as part of other services 0 50 Post Abortion Care (PAC) adapted to the youth and adolescents’ needs 0 50 Post Abortion Care (PAC) 0 103 Integration of Human Rights in SRH 0 10 During the full-scale evaluation, we will investigate the extent to which these types of activities are being sustained across project countries, and the degree to which they have helped local providers and systems address long-term barriers to providing family planning services. Furthermore, any gaps in implementation and support for these activities will be tracked and noted. Result R3: Efforts to Remove Policy Barriers and Improve Contraceptive Commodity Security Coordinated Just as under Result 2, the AgirPF files provide numerical indicators for a number of activities under Result 3. Rather than describe progress against targets in this arena, which would not be informative, below is a description of the major efforts and implemented across AgirPF countries to support the removal of policy barriers and improve the contraceptive commodity security in AgirPF countries.  Engaging faith-based organizations to advocate for the promotion of the responsible childbearing in Burkina Faso and Togo  Engaging stakeholders from advocacy the Network of Champions in Advocacy for Sustainable Health Funding (RCPFAS) in all countries and building their capacity to advocate for the removal of policy and socio-cultural barriers to FP in those countries through targeted workshops and meetings.  Using the RAPID models specifically developed for the national authorities and for those based in the intervention cities of AgirPF countries. Activities included stakeholders from government offices, WAHO representatives, CSOs and donors, and prompted adoption of the model at the advocacy and grassroots level.  Engaging youth organizations to advocate for youth-friendly sexual and reproductive health services, in specific countries.  Training providers on COPE for Contraceptive Security, a client-oriented provider-efficient method of assessing quality of care with the specific intent of improving family planning commodities and supply chain systems.  Training health managers on the Reality Check tool, a database that provides estimates on commodity needs for desired contraceptive prevalence rates. Use of this system allows health management teams to estimate and budget for the family planning method mix, as well as estimate necessary service expansion to meet targets. The summary of activities under result three provide an outline for potential areas of investigation during the full evaluation. For example, it will be important to speak with stakeholders to understand their perspectives on the advocacy engagement activities and to understand the ways in which their capacity to remove policy barriers was built. With respect to the commodities security activities, the evaluation should examine the extent to which these tools are being utilized across the countries and aiding in securing contraceptive commodities. COUNTRY BY COUNTRY ANALYSIS This section examines the findings for each AgirPF country individually. Table 2 provides a summary of main family planning statistics across Burkina Faso, Cote d’Ivoire, Niger and Togo. Additional details for each country are included in Appendix 2. Table 2: Comparison of major reproductive health variables across countries Burkina Faso DHS 2010 Cote d’Ivoire DHS 2011 Niger DHS 2012 Togo DHS 2011 Average age at first sexual encounter sexual debut (years) 17.5 17.0 15.9 18.2 Median age at first union 17.8 19.7 15.7 20.0 Percent of 19-year￾old women with baby or pregnant 57.4% 49.5% 74.7% 35.8% Ideal family size for all women 5.6 5.2 9.2 4.3 Fertility rate 6.7 5.0 7.6 4.8 Awareness of at least one FP method among women 97% 93% 89% 96% Burkina Faso DHS 2010 Cote d’Ivoire DHS 2011 Niger DHS 2012 Togo DHS 2011 Using contraceptive when survey conducted 14% 18% 14% 19.3% Public sector as source for modern contraceptive supply 74% 26% 94% 53% Unmet FP need (to limit or space children) 29% 27% 16% 34% Togo Result 1: Delivery of quality FP information, products, and services strengthened and expanded Although Togo only achieved 62% targets for CYP in PY2, it surpassed its CYP target in PY3 (Figure 6). In terms of individual numbers of people served with family planning (Figure 7), in PY2, AgirPF Togo exceeded both targets for reaching new method users in both PY2 and PY3, even with incomplete reporting in the third year. In the case of total family planning users, only 76% of the PY2 target was met. However, in PY3, Togo is more or less on track to meet the end of year target, with achievement of 70% of the intended target by the end of the third quarter. AgirPF Togo exceeded training goals for training providers in family planning in PY2 and is on track to meet targets for PY3, with a third quarter cumulative performance of 76% of target achieved (Figure 8). On the other hand, the project in Togo is very behind its target for conducting special FP days in PY3, reaching only 36% of its goal by the end of the third quarter. Targets for FP days were exceeded in PY2, although that may be due to a substantially lower target in PY2 than in PY3. Figure 6: AgirPF Togo CYP, results vs targets Figure 7: AgirPF Togo total FP users and new method users, results vs targets Figure 8: AgirPF Togo Training and FP days: Results vs Targets Result R2: Evidence-based service delivery approaches selected, adapted, and implemented As noted in the above section on overall AgirPF result 2 activities, several evidence-based approaches were implemented in Togo. While there are no detailed cumulative accounts of the types and numbers of trainings by country, below we detail a few of those activities that were designed to create evidence￾based approaches to improving family planning:  Training, equipping and supporting Community Health Workers (CHWs) to provide a wide range of FP options: In Togo, a number of CHWs were trained in FP (as part of the overall FP training efforts) and equipped with IEC materials to use when delivering FP services in the community.  Training in youth-friendly services: AgirPF, along with its implementing partners ATBEF and DSF from the MOH, conducted the training of 17 Togolese services providers to offer youth friendly services, after the baseline study conducted by the project that determined that 0% of health facilities offered youth friendly services. These trainings are designed to help providers make facilities more accessible to youth through: i) youth and adolescent’s separate hours, (ii) special waiting room, (iii) conducive policies/procedures for youth access to FP services in collaboration with regional and district health managers who could support these changes.  Developing AgirPF’s social and behavior change communication strategy in collaboration with Camber Collective: AgirPF worked with the Camber Collective to devise an evidence-based SBCC strategy across countries. In Togo AgirPF organized two workshops with MOH and SRH/FP stakeholders to share the SBCC strategy, validate protocols were validated with MOH, and validate audience segmentation approaches in Lomé urban and peri-urban areas.  Ensuring service quality using EngenderHealth’s Site Walk-Through (SWT) approach. Using SWT engages the community in quality improvement of services by actively contributing in FP use issues identification and solving. In Togo, during several quarters, SWTs were conducted to identify problems with access to and communication about family planning services. These SWTs resulted in identification of issues such as (i) the high cost of FP methods (ii) unavailability of FP providers, (iii) lack of privacy. SWTs resulted in action plans developed by participants to address identified problems.  Piloting the Informed Push Model (IPM): AgirfPF, in partnership with the MOH/Togo, UNFPA Togo Office and CAMEG, employed the Informed Push Model to identify critical areas of contraceptive stock-outs and provide improved contraceptive security. A midterm evaluation of the pilot conducted by AgirPF confirmed the Ministry of Health announcement of zero stock-outs of FP commodities during the pilot period, compared to 80% prior to the pilot. AgirPF documented the approach and its successes for use in other project countries. The activities in Togo span several HIPs and include a variety of approaches to supporting improved access to family planning at the community, facility and individual levels. In the full evaluation, we must assess the degree to which these practices are being used on a continuous basis in supported sites around Togo, and to what extent providers feel they were equipped and supported by AgirPF to implement these practices in the long-term. Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Country-specific activities under Result 3 in Togo closely resemble those already described in the overall AgirPF section above. Additional details about specific activities in Togo are included below:  In Togo, AgirPF’s advocacy efforts led to the stakeholder validation of three new regulations (ministerial orders and decrees) to implement the RH law and the agreement of next steps to have the regulations signed by authorities.  AgirPF in collaboration with the WAHO and HPP, supported 34 religious leaders in Togo the development of their advocacy tools and capacities for the promotion of healthy timing and spacing of births. Activities included identification of socio-political barriers to FP and development of advocacy plans by participants.  AgirPF collaborated with ATBEF’s Youth Action movement to organize in Lomé a workshop to mobilize actors and develop an advocacy document for the promotion of family planning among youth population.  AgirPF Togo held multiple workshops on logistics management and information systems (LMIS) to improve contraceptive security. These workshops were designed to support adequate availability of contraceptives and to train FP service providers to use the national contraceptives logistics management manual.  Engaged youth organizations to advocate toward Togolese parliamentarians for youth-friendly sexual and reproductive health services. Advocacy efforts in Togo mainly involve workshops aimed at increasing the support for and capacity of local actors to advocate for improved family planning policies and funding. The outcomes of these activities are often either training capacity among participants or action plans designed to carry the activities forward after the event. During the full-scale evaluation, it will be important to assess whether or not action plan items were carried out and to what extent participants felt prepared to advocate for FP policy changes following their participation in these events. Burkina Faso Result 1: Delivery of quality FP information, products, and services strengthened and expanded Burkina Faso not only achieved their CYP targets for PY2, but they are well on track to exceed CYP targets for PY3, with 90% target achievement by the end of the third quarter of PY3 (Figure 9). Targets for new method users in PY2 were surpassed, and were on track for full achievement for PY3 (82%). Despite successful attainment of CYP and new method user goals, AgirPF in Burkina Faso fell very short of targets for total FP users in both PY2 (20% achieved) and PY3 (26% achieved). AgirPF in Burkina Faso met 95% of their intended target for number of people trained in FP in PY2; they were also on track to complete their target for training in PY3, having achieved 70% of targets by the end of the third quarter. The project exceeded their target for number of FP special days in PY2, and was at 75% of intended targets for PY3 by the end of the third quarter, putting them on track to accomplish their goal of 200 FP days in PY3. The mismatch between the successes in training, FP days, CYP and new method users contrasted against the poor performance of the indicator for total number of FP users suggests either a miscalculation in the proposed target for total number of FP users, or a hold-over effect from the delayed start-up in PY1. Figure 9: AgirPF Burkina Faso CYP, results vs targets Figure 10: AgirPF Burkina Faso total FP users and new method users, results vs targets Figure 11: AgirPF Burkina Faso Training and FP days: Results vs Targets Result R2: Evidence-based service delivery approaches selected, adapted, and implemented Several evidence-based approaches were implemented in Burkina Faso as part of AgirPF’s activities. While there are no detailed cumulative accounts of the types and numbers of trainings by country, below we detail a sample of those activities that were designed to create evidence-based approaches to improving family planning:  Ensuring service quality using EngenderHealth’s Site Walk Through (SWT) approach. AgirPF organized several In SWTs in Ouagadougou comprised of religious leaders, traditional leaders, members of local youth and women’s organizations, health mangers and providers and educators. Barriers to care were identified and action plans to address the high cost of services, the lack of health provider refresher courses and the need for sensitization at the community level.  Paving the way with Camber to implement a systematic and evidence-based SBCC strategy. Through formative research and cross-country exchanges, AgirPF Burkina Faso created a harmonized strategy for SBCC in the country.  Integrating FP services into postpartum services by building partner capacity to provide postpartum FP (PPFP) counselling and services. As part of their emphasis on HIPs, AgirPF in Burkina Faso undertook several systems strengthening activites to integrate family planning into postpartum care. This included training providers from delivery rooms in postpartum family planning, providing PPFP materials and training supervisors to assist in facilitative supervision.  Strengthening stakeholders from advocacy networks to advocate for the removal of policy and socio￾cultural barriers to FP in those countries. Through a situational analysis in Burkina Faso, AgirPF identified 6 FP areas for strengthening including: (i) Task shifting, (ii) low RH implementation, (iii) Fundraising for FP, (iv) Private sector contribution to FP funding, (v) youth and adolescent access to FP services, (vi) FP policies and commitment follow-up. The participants developed some recommendations to improve these areas such as (i) developing advocacy project to be integrated in the 2016-2020 strategic plan, (ii) reinforcing the RH TWG organizational capacity and its members’ advocacy, fund raising and monitoring/evaluation capacities. However, there was no follow-up to see if any of these tasks were undertaken.  Engaging faith-based organizations to advocate for the promotion of the responsible childbearing in Burkina Faso and Togo. In Burkina Faso, the project engaged 240 traditional and religious leaders in the promotion of the responsible childbearing through various training sessions conducted in Ouagadougou, Bobo Dioulasso and Koudougou based on their Religious RAPID Models Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Country-specific activities under Result 3 in Burkina Faso closely resemble those already described in the overall AgirPF section above. Additional details about specific activities in Burkina Faso are included below:  AgirPF built the capacity of stakeholders from RCPFAS in Burkina Faso to create advocacy strategies fpr adoption of new policies to remove corresponding policy barriers in Burkina Faso  AgirPF in collaboration with the WAHO and HPP, supported religious leaders in Burkina Faso in the development of their advocacy tools and capacities for the promotion of healthy timing and spacing of births. Activities included identification of socio-political barriers to FP and development of advocacy plans by participants.  AgirPF Burkina Faso held multiple workshops on logistics management and information systems (LMIS) to improve contraceptive security. These workshops were designed to support adequate availability of contraceptives and to train FP service providers to use the national contraceptives logistics management manual.  In collaboration with HPP and Deliver and WAHO, AgirPF strengthened numerous stakeholders in Burkina Faso to advocate for the increase of country funding for FP, and task shifting. They developed 4 advocacy strategies whose implementation will increase efforts for obtaining the increase of country funding for FP, and also for the adoption of policy change towards the adoption of task shifting more widely in the region.  Assuring service quality using COPE for Contraceptive Security (COPE for CS). The AgirPF Project audited 15 facilities and identified the need for improved contraceptive security through training facilitators in COPE for contraceptive security in Burkina Faso. The policy advocacy workshops in Burkina Faso constitutes a first step in AgirPF’s work towards removing policy barriers in the country. However, there is no current information on the impact of these workshops following stakeholder participation. The full evaluation should follow-up on the implementation of action plans and examine the progress made towards actual policy change within the country. Niger Result 1: Delivery of quality FP information, products, and services strengthened and expanded CYP performance in Niger is poor when compared to stated AgirPF Niger targets. In PY2, only 24% of the stated PY2 targets were achieved, whereas in PY3 only 42% of the CYP goal was reached by the end of the third quarter. Further, neither targets for new method users nor total FP users were reached in PY2 or PY3. The percent of target achievement for the total number of family planning users was 24 and 27% in PY2 and PY3, respectively. Performance on new method users fared better, but in PY2 only 64% of targets were achieved, while PY3 is on track with 73% of target achieved by the end of the third quarter. In both PY2 and PY3, AgirPF Niger surpassed its goals for training providers in family planning. However, performance for family planning special days fell far short of targets for PY2, reaching only 24% of the intended 150 days. For PY3, the project is on track (80%) to complete its FP special days target, but it should be noted that the total number is substantially lower in PY3 at only 10 days for the entire year. Figure 12: AgirPF Niger CYP, results vs targets Figure 13: AgirPF Niger total FP users and new method users, results vs targets Figure 14: AgirPF Niger Training and FP days: Results vs Targets Result R2: Evidence-based service delivery approaches selected, adapted, and implemented Several evidence-based approaches were implemented in Niger as part of AgirPF’s FP systems strengthening activities. While there are no detailed cumulative accounts of the types and numbers of trainings by country, below we detail a sample of those activities that were designed to create evidence￾based approaches to improving family planning:  A joint advocacy workshop was held for AgirPF Burkina Faso and Niger that joined stakeholders together to discuss policy barriers to FP, understand the importance of data in the policy dialogue to address policy barriers to PF, to allow participants to practice designing advocacy plans to address policy barriers to FP, and allow participants to practice the implementation of advocacy plans by using RAPID Models 2014.  AgirPF Niger supported the national FP action plan (2015-2020) in Niamey and Maradi through planned to scale-up the “Ecole des Maris”1 model and tutorial as a best practice.  Training, equipping and supporting Community Health Workers (CHWs) to provide a wide range of FP options. In Niger, CHWs were trained in Sayana Press2 service provision and equipped with IEC kits to be able to improve dissemination of injectables at the community level.  Paving the way with Camber to implement a systematic and evidence-based SBCC strategy. AgirPF pre￾tested key messages from the FP-SBCC strategy in Niger through focus groups and individual interviews with different segments of women, their spouses and religious leaders in Niamey urban and peri-urban zones. Findings were used to help finalized the regional SBCC strategy.  Ensuring service quality using EngenderHealth’s Site Walk Through (SWT) approach. In Niger, at least one SWT was organized and aimed at sensitizing community members, HIV testing, screening for cervical cancer and STDs, as well as prompting community conversations in local barber shops. Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Country-specific activities under Result 3 in Niger closely resemble those already described in the overall AgirPF section above. Additional details about specific activities in Niger are included below:  In Niger, AgirPF provided technical support to networks of religious leaders to finalize planning of advocacy activities targeting the promotion of healthy spacing and timing of births.  AgirPF staff trained teams at intervention sites on COPE for contraceptive security and assisted them in developing action plans which included (i) MOH technical team members at national, regional and district levels training/orientation, (ii) COPE for contraceptive security baseline facility assessment, (iii) introduction and follow up of COPE for contraceptive security in AgirPF intervention facilities.  An LMIS training in Niger brought together a total of 88 FP logistics managers of the 36 AgirPF/Niger sites (21 in Maradi and 15 in Niamey), helping them to estimate their monthly needs using the Min/Max approach and to learn how to make adequately and timely placed orders. 1 Ecole des Maris is a promising practice in West Africa aiming to promote men participation to family and reproductive health. Peer educators are orientated/trained in MAP and MAP champions are encouraged to introduce discussion on Men participation to their families’ RH issues solving in order to convince them in the importance and benefits of caring the families’ health. It is a kind of promoting community conversations 2 Sayana Press is a new presentation of DMPA (injectable contraceptive)  AgirPF Niger introduced the COPE for contraceptive security process into two centers for excellence, the National Center of Reproductive Health and Regional Hospital in Poudriere, in an attempt to institutionalize the process. As with the policy advocacy and evidence-based work in the other countries, to key to understanding these activities will be in connecting with the stakeholder participants to understand the degree to which the AgirPF activities contributed to improved practices and increased advocacy for family planning. Cote d’Ivoire Of all the individual level county findings, Cote d’Ivoire results in family planning service delivery are the weakest. CYP achievement are a mere 4% in PY2 and only 23% of targets set by the end of the third quarter of PY3. The number of total FP users reached in PY2 and PY3 were negligible, compared to the intended targets. Only 1% and 4% of targets for total number of FP users were reached in each respective year. Achievement of new method user targets are equally dismal, with only 4% of the total goal reached in PY2, and only 26% of targets achieved by the third quarter of PY3. Performance on training in FP and FP special days is slightly better. In PY2, AgirPF reached its modest goal of carrying out 10 FP special days, but reached only 20% of its target as of the third quarter of PY3. As for trainings in family planning, AgirPF Cote d’Ivoire reached 77% of training targets in PY2 and is on track to reach its goal for PY3, with 73% of targeted trainings completed by the third quarter. Result 1: Delivery of quality FP information, products, and services strengthened and expanded Figure 15: AgirPF Cote d’Ivoire, results vs targets Figure 16: AgirPF Cote d’Ivoire total FP users and new method users, results vs targets Figure 17: AgirPF Cote d’Ivoire Training and FP days: Results vs Targets Result R2: Evidence-based service delivery approaches selected, adapted, and implemented In the available project files, there are fewer examples of implementation of evidence-based practices available for Cote d’Ivoire:  Site walk-throughs in Cote d’Ivoire helped community members identify (i) contraceptives stock out, (ii) lack of communication about FP services availability, (iii) lack of awareness on FP methods side effects at the community level. Action plans were created to address these issues. Result R3: Efforts to remove policy barriers and improve contraceptive commodity security coordinated Country-specific activities under Result 3 in Cote d’Ivoire closely resemble those already described in the overall AgirPF section above. Additional details about specific activities in Cote d’Ivoire are included below:  In Cote d’Ivoire, In December 2014, USAID DELIVER held a series of trainings and activities on logistics management and commodity forecasting in which the Côte d’Ivoire SPO and CPM participated. This included: A training of trainers in logistics management of contraceptives; a training of trainers in quantification; and a national contraceptive quantification exercise.  AgirPF built the capacity of several stakeholders from RCPFAS-CI to advocate for 1) the adoption of a national RH law, 2) the increase of country funding for FP at the national level, 3) the increase of funding for FP at the Abidjan city level, 4) task-shifting to community health workers 5) strong commitment of high level decision makers to FP, 6) the integration of FP in other RH services. The team further engaged stakeholders to launch an advocacy campaign for the development and adoption of a new SRH law. In an MOH meeting attended by multiple advocacy groups including AgirPF, resulted in the formal decision by the MOH to revise and reintroduce a draft SRH law, the design of an action plan outlining all the steps leading up to the introduction of a proposed law to the Ivorian Parliament for its adoption, and to the advertisement of the law once it is adopted.  AgirPF collaborated with the Ministry of Health and Aids Control in Cote d’Ivoire to strengthen advocacy capacity of 25 high-level leaders of faith-based organizations (Catholic, Evangelical Protestant, and Muslim), as well as to engage these influental partners in the efforts to reposition FP, through the promotion of the concept of “Responsible Childbearing”. Participants developed advocacy strategies that could lead to the adoption of a national policy for the promotion of “responsible childbearing.”  AgirPF Cote d’Ivoire introduced the COPE for contraceptive security in one center for excellence, the Hospital de Bingerville as a means of building sustained training systems for this approach in the country. Advocacy activities in Cote d’Ivoire, though seemingly less numerous than in other AgirPF countries appear to have resulted in major advances in policy change. The full evaluation should follow up on both the SRH policy and the policy for responsible childbearing to determine if and in what form its final adoption took place. SUMMARY OF ACHIEVEMENT TOWARDS EVALUATION QUESTIONS This section provides a brief analysis of the project’s achievement of stated evaluation goals, based on the findings of the desk review. The questions are answered to the extent possible, given the information available in the project files. 1. How has AgirPF performed (analysis of monitoring data) Performance of the AgirPF project, as assessed through the indicators examined under Result one in this desk review show variable results across indicators and countries. Overall, AgirPF has fallen substantially short of its targets in reaching new method and overall FP users, resulting in low CYP achievement. However, it should be noted that the contraceptive method mix that is provided is skewed towards more effective long-acting reversible contraceptives. Given the current trajectory of the project, if all LOE remains constant, the project will not be on track to meet its overall goals for CYP. On a country-by-country basis, Togo and Burkina Faso are have performed better than Niger and Cote d’Ivoire, with Togo either exceeding or nearly reaching all of its goals for CYP, new method users and total FP users. While training targets for family planning are largely met both at the overall project level and across countries, the number of family planning special days conducted are particularly weak for PY3. Poor performance on FP special days in Cote d’Ivoire and Togo contribute to the overall poor performance for the project. 2. Which high impact/best practices (HIPs/BPs) have AgirPF advanced? Across its individual countries and through cross-country activities, AgirPF has advanced a number of high-impact practices including integrating family planning into postpartum care, increasing community based distribution of family planning through training of community health workers and mainstreaming youth into family planning services by training providers in youth-friendly services. However, application of these practices is variable and it is unclear how implementation actually takes place. As described in the quarterly reports, application of high-impact practices may range, in practice, from a workshop on a topical area to orientation to a tool, to actual training and supervision on a new practice. Furthermore, results data may or may not represent the impact that adoption of these HIPs has on service delivery. Minimum standards and definitions for the implementation of a successful HIP should be clarified by the project, and the evaluation should attempt to understand the added value that these practices have brought to improving the delivery of FP in these settings. 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective The overall results framework does seem to have a cohesive flow from intermediate results to higher level objectives. However, there is substantial overlap in the reporting of activities under results 2 and 3, with similar activities being classified under both or interchangeably between the two. By specifically tracing concrete activities back to specific points in the results framework, the project can further clarify which results are advancing appropriately. There were a number of indicators in the project spreadsheet files for which there were no findings. Furthermore, much of the numerical data in the spreadsheets do not provide adequate information on performance, specifically as relates to Results 2 and 3. In these cases, additional descriptive data, of the kind presented in the quarterly reports is of more value. 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? The quarterly reports provide a lot of information on building grassroots support and skill around policy advocacy issues. Unfortunately, there is less information on the “next steps” after initial engagement takes place. Many activities are left off at the “action plan” stage, but there is little evidence that the project conducts any follow-up activities or support for implementation of the action plans. This leaves open the question of what actual impact the project has had on removing barriers to policies. Full assessment of this questions remains to be done as part of the broader evaluation. However, at least in Cote d’Ivoire, there is evidence that support for advocacy activities on the part of AgirPF and partners have resulted in some movement on an SRH policy. The specific content of this policy, as well as policies in other countries, will be explored as part of the overall evaluation. 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? The quarterly reports do not provide sufficient data to adequately answer this question. The “challenges” section of the reports focuses almost exclusively on management and project logistics, rather than tangible learnings from the ongoing activities. One recommendation on the basis of this desk review would be to encourage project staff to reflect on learnings (positive or negative) during each quarter and document approaches for implementation that have either worked very well (and should be replicated across projects) or have not been successful (and should be revised or abandoned). This full extent of this question remains to be assessed as part of the broader evaluation. 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? Again, there is little clear evidence pertaining to this question in the available project files. However, there are indications that there were difficulties in overall project management, staffing and roll-out at the global level. For example, there are several notations about the delays in registering EngenderHealth in Mauritania, issues with staffing for M&E and poor-performing partners. These issues hint at potential managerial problems, which should be further explored as part of the broader evaluation. AREAS FOR INVESTIGATION FOR THE BROADER EVALUATION The desk review elucidates a number of areas in which deeper investigation would yield critical information in answering the evaluation questions. Below are some preliminary thoughts intended to inform the Inception Report:  Facility-based records reviews in a sample of AgirPF target facilities, as well as a comparison group of non-intervention sites, will yield important information on the impact of AgirPF in increasing accessibility to and uptake of family planning for new method adopters  Interviews with key informants in the governments, advocacy groups and other key stakeholders will provide detail on the impact of AgirPF’s engagement on supporting policy change and the successes and challenges of the AgirPF approach to policy advocacy. These interviews will also help explore facilitating factors and barriers that either permitted or hindered the achievement of targets. In tandem with a national policy review, their feedback will help determine what specific role AgirPF played in moving family planning policy forward in each country  Interviews with providers and health managers in the AgiPF facilities and districts will provide critical information on the effectiveness and sustainability of AgirPF approaches for implementing high-impact best practices in family planning  Interviews with AgirPF project and partner staff will inform the management successes and challenges of AgirPF in each country and across partner organizations. APPENDICES Appendix 1: Bibliography and Project Files used for the Desk Review The following project files were utilized for the desk review. Please note that certain descriptions of project activities, particularly for the findings under Results 2 and 3, were taken directly from this report and may have very similar language. For the data tables in this report, data the latest project results Excel file (Appendix 1- Final Progress Against PMP Indicators_110315) were independently analyzed to create the graphs for this report. 2014_AgirPF Quarterly Report_April to June AgirPF PY2 Q4 AgirPF PY3Q1 For Submission AgirPF PY3Q3 (3) AgirPF Q2 report May 15 Final AgirPF Q3_PY2 AgirPF Quarterly Report 3_January to March_2014 (Q3)_FINAL_15 May 2014 AgirPF Quarterly Report Q2_PY3 AgirPF_PIR results _Evaluation_2016 AgirPF_Quarterly Report_October-December 2013_FINAL Appendix 1- Final Progress Against PMP Indicators_110315 FY15Q1_PY2 - AgirPF_Report Oct-Dec 2015 Institut National de la Statistique et de la Démographie (INSD) et ICF International, 2012. Enquête Démographique et de Santé et à Indicateurs Multiples du Burkina Faso 2010. Calverton, Maryland, USA : INSD et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR256/FR256.pdf. Consulted January 2017. Institut National de la Statistique (INS) et ICF International. 2012. Enquête Démographique et de Santé et à Indicateurs Multiples de Côte d’Ivoire 2011-2012. Calverton, Maryland, USA : INS et ICF International. http://dhsprogram.com/publications/publication-FR272-DHS-Final￾Reports.cfm. Consulted January 2017. Institut National de la Statistique (INS) et ICF International, 2013. Enquête Démographique et de Santé et à Indicateurs Multiples du Niger 2012. Calverton, Maryland, USA : INS et ICF International. Available at http://dhsprogram.com/pubs/pdf/FR277/FR277.pdf. Consulted January 2017. Population Reference Bureau. 2016. World Population Data Sheet with a Special Focus on Human Needs and Sustainable Resources. Available at: http://www.prb.org/pdf16/prb-wpds2016-web￾2016.pdf. Consulted January 2017. Ministère de la Planification, du Développement et de l’Aménagement du Territoire (MPDAT), Ministère de la Santé (MS) et ICF International, 2015. Enquête Démographique et de Santé au Togo 2013-2014. Rockville, Maryland, USA : MPDAT, MS et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR301/FR301.pdf. Consulted January 2017. Sedgh G., S. Singh and R. Hussain 2014. Intended and Unintended Pregnancies Worldwide in 2012 and Recent Trends. Studies in Family Planning. 45: 301–314. doi:10.1111/j.1728-4465.2014.00393.x. Consulted January 2017. Y1 Q5 and Annual Performance Report_Rev121614 Summary Report and Data for PIR. Appendix 2: In-depth background sections by country Burkina Faso Marriage, Fertility and Family Size According to the DHS (2010)3, Burkinabe women will give birth on average to 6.0 children during their fertility life period. There are important differences between rural and urban areas, as the fertility rate is 6.7 children in the rural areas and 3.9 children in urban areas. The fertility rate varies depending on the level of instruction: it is 6.6 among women with no instruction, 5.7 among women with primary education and 3.1 among women with secondary education. Fertility is also high among adolescents. At 17 years of age, 18.4% of girls have already a child or have their first pregnancy. This percentage increases to 57.4% for 19 year old girls. Among interviewed women 25-49 years old, half had initiated their married life at 17.8 years of age. 53% were already in a steady relationship when they turned 18, and 95% were in a steady relationship when they were 25 years old. 42% of women in union live in a polygamous relationship. 12% of women in union have at least two co-wives. Sexual activity is initiated on average when women are 17.5 years old. About one fourth of women (23%) declared not wanting to have any more children. For women, the ideal family size is 5.6. Ideal family size is larger among rural women (5.9) than among urban women (4.2). The median birth spacing period is 35.9 months. The data indicate that in 87% of the cases the birth spacing period is 24 months or higher. Among the rest, 13% of cases, the birth spacing period is lower than 24 months, thus too short. Family Planning Knowledge of at least one modern family planning method is practically universal. However, at the time of the survey only 14% declared using a contraceptive, and 9% a modern contraceptive. Among women not using contraception at the time of the survey, 58% declared to have the intention to use contraception in the future. 18% of those that declared not having the intention of use contraception indicated that they were planning on getting pregnant. 97% of interviewed women declared that they wanted to bear the children to which they gave birth, and 77% declared that births happened when desired. Findings suggested that 29% of interviewed women had unmet family planning needs, either to limit or space children. If these needs were satisfied, contraceptive prevalence could be as high as 43%. 74% of women obtained their contraceptives from a public health facility with 58% reporting to have obtained their contraceptives at a government health center and 13% at a government hospital. Ivory Coast Marriage, Fertility and Family Size4 19% of 25-49 year old women reported having had their first sexual encounters when they were15 years old, with the median age for first sexual encounter for women estimated at 17.0 years of age. 3 Institut National de la Statistique et de la Démographie (INSD) et ICF International, 2012. Enquête Démographique et de Santé et à Indicateurs Multiples du Burkina Faso 2010. Calverton, Maryland, USA : INSD et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR256/FR256.pdf. Consulted January 2017. 4Institut National de la Statistique (INS) et ICF International. 2012. Enquête Démographique et de Santé et à Indicateurs Multiples de Côte d’Ivoire 2011-2012. Calverton, Maryland, USA : INS et ICF International. http://dhsprogram.com/publications/publication-FR272-DHS-Final-Reports.cfm. Consulted January 2017. Median age at first union is 19.7 years old, with differences by age of study participants. Median age at first union for the 25-29 year old is 20.5, and it is 18.8 among 45-49 year old. According to the DHS (2011), an Ivorian women will give birth on average to 5.0 children during their fertility life period. There are important differences between rural and urban areas, as the fertility rate is 6.3 children in the rural areas and 3.7 children in urban areas. The fertility rate varies depending upon the area; the average is 3.1 in Abidjan while it is 6.8 in the North- West region. The comparison of the results of the EDS-MICS with those of the previous surveys reveals that fertility levels have remained virtually unchanged. The ideal family size for women is 5.2 children. Fertility levels vary significantly according to the level of education of women: 2.6 children per women among those with secondary education or more versus 5.8 for those with no instruction. Fertility also varies according to the standard of living of the household in which the woman lives (3.2 children per woman for women in households in the highest quintile versus 6.7 in households in the lowest quintile). In Côte d'Ivoire, adolescent fertility is high. Indeed, 30% of girls aged 15-19 have already begun their fertile life: 23% are already mothers and 7% are currently pregnant for the first time. Early fertility is almost three times higher among uneducated girls (39%) than girls with a secondary level or higher (14%). Almost six out of ten women (63%) were married at the time of the survey took place. 33% of women aged 20-24 were married/engaged by reaching 18 years of age. The median birth spacing period is 37 months. The data indicate that in 85% of the cases the birth spacing period is 24 months or higher. Among the rest, 15% of cases, the birth spacing period is lower than 24 months, thus too short. Family Planning Almost all women (93%) reported knowing at least one modern contraceptive method. The best known methods are the male condom and the pill. However, regardless this high level of knowledge, only 18% of married/engaged women used any contraceptive method and 13% used a modern method at the time of the survey. Women use essentially three modern methods: the pill (7%), the injectable (2%) and the male condom (2%). Comparison with the results of previous surveys shows that the use of modern contraceptive methods has increased significantly since 1994. Among women aged 15-49, 41% would like to delay the next birth of two years or more, while 25% would want another birth within two years. The proportion of women unsatisfied with family planning needs is estimated at 27%. Of these, the vast majority would need to use contraception more to space than to limit (20% vs. 8%). Only 26% of contraceptive users get their method from a public sector facility, and 14% get it from a public health center and 9% from a public hospital. 46% get their method from a private source which in 43% of cases is a pharmacy. Niger Marriage, Fertility and Family Size According to the 2012 Niger DHS,5 a woman has an average of 7.6 children at the end of her fertile life. The average number of children per woman varies from 5.6 in urban to 8.1 in rural areas. The average number of children per woman varies from 5.6 in urban to 8.1 in rural areas. The average number of children per woman also varies significantly, depending on the region, from 5.3 in Niamey to 8.5 in the 5 Institut National de la Statistique (INS) et ICF International, 2013. Enquête Démographique et de Santé et à Indicateurs Multiples du Niger 2012. Calverton, Maryland, USA : INS et ICF International. Available at http://dhsprogram.com/pubs/pdf/FR277/FR277.pdf. Consulted January 2017. Zinder region. The comparison of the results of the EDSN-MICS IV 2012 with those of the previous surveys reveals that the level of fertility has hardly changed. Fertility levels vary significantly according to the level of education of women: 4.9 children per woman in those with secondary education or more compared with 8.0 children per woman in those without education. Adolescent fertility is high. Indeed, 40% of girls aged 15-19 have already begun their fertile life: 33% are already mothers and 8% were pregnant for the first time at the time of the survey. Early fertility is more than three times higher among uneducated girls (50%) than girls with one or more secondary education (15%). The average age at first birth is 18.6 years for women aged 25-49 years. The results of the EDSN-MICS IV 2012 highlight differences by level of education. The average age at first birth for uneducated women is 18.3 years compared with 22.7 years for those with one or more secondary education. Age at first marriage the vast majority of women (89%) were in married/engaged at the time of the survey. Family Planning The vast majority of women (89%) and men (91%) reported knowing at least one modern contraceptive method. Among women, the best-known methods are pill and injectables. Despite this high level of knowledge, only 14% of unionized women used any contraceptive method and 12% used a modern method at the time of the survey. Women use essentially three methods: the pill (6%), the breastfeeding method and amenorrhea (4%) and injectable (2%). The use of modern contraceptive methods among women in union is higher in urban areas (27%) than in rural areas (10%). Nearly one in ten women (8%) said they no longer wanted children, while 86% of women said they wanted more. Among women aged 15-49, 51% would like to delay the next birth of two years or more, while 32% would want another birth within two years. The proportion of women with unmet need for family planning is estimated at 16%. Of these, the vast majority would need to use contraception more to space than to limit (13% versus 3%). Togo Marriage, Fertility and Family Size According to the 2011 Togo DHS,6 A woman has, on average, 4.8 children at the end of her fertile life. The average number of children per woman varies from 3.7 in urban to 5.7 in rural areas. The average number of children per woman also varies greatly by region, from a minimum of 3.5 in Lomé to a maximum of 6.0 in the Savannah region. Fertility levels vary significantly according to the level of education of women: 3.5 children per woman in those with secondary education or more, compared with 6.1 children per woman in those with no education level. In Togo, 32% of women aged 25-49 were in marriage/engage before the age of 18. Half of women (50%) aged 25-49 were married/engaged before the age of 20. Nearly half of women (47%) aged 25-49 began their sex lives before reaching the age of 18. 6 Ministère de la Planification, du Développement et de l’Aménagement du Territoire (MPDAT), Ministère de la Santé (MS) et ICF International, 2015. Enquête Démographique et de Santé au Togo 2013-2014. Rockville, Maryland, USA : MPDAT, MS et ICF International. Available at : http://dhsprogram.com/pubs/pdf/FR301/FR301.pdf. Consulted January 2017. Overall, 17% of women aged 15-19 have already begun their fertile life: 13% are already mothers and 3% are currently pregnant for the first time. The proportion of adolescent girls who started their fertility is four times higher among women with no education (35%) than among those with a secondary level or higher (9%). By the time that women are 19 years old, 35.8% either have already a child or are expecting. The median birth spacing period is 38.0 months. Family Planning Contraceptive prevalence among all women is 19.3% and 19.9% among those in a stable relationship. Among the latter, 17% use a modern method and 3% use a traditional method. The most popular methods are injectable (7%), implants (5%) and male condoms (2%). Among women aged 15-49 who are not in union and sexually active, 38% use a modern contraceptive method. Three out of ten sexually active and sexually active women (30%) use the male condom, 4% the pill and 3% injectable. Nearly one-third of women aged 15-49 in union (32%) say they no longer want children. In addition, 37% say they want to wait two years or more before the next birth. Among these women, those who do not use a contraceptive method are potential candidates for family planning. The proportion of women in union with unmet need for family planning is estimated at 34%. Of these, the vast majority need to use contraception for birth spacing rather than limitation (22% versus 12%). The majority of women (85%) using modern contraception obtained their method from a public-sector source. 61% obtained them from an integrated health center. 10% from health posts, 8% from maternity wards, and 5% from public sector pharmacies. 3/9/2018 1 AGIRPF Midterm Evaluation Presentation to USAID July 6, 2017 E4D 1. How has AgirPF performed (analysis of monitoring data)? 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? 4. For AgirPF IR 3: to what extent has AgirPF contributed to removing policy barriers to FP access in the region? 5. What are the Project’s successes, challenges and lessons learned that the evaluators recommend be disseminated across the region to advance family planning programming? 6. How has AgirPF managed staff in focus countries, consortium partners and environmental compliance? 3/9/2018 2 Evaluation Questions QUANTITATIVE • Project level data (quarterly reports) • Site-level data – 1) Number of FP users (new) – 2) Number of FP users (returning) – 3) By method – Intervention vs control • Environmental compliance • QUALITATIVE • Interviews with district managers • Interviews with Ministry of Health • Interviews with local and regional partners • Interviews with providers (Agir sites) • Interviews with AgirPF staff • Policy review (document review) 3/9/2018 3 Data Collection Methods and Tools INTERVENTION SITES • Random Selection • Burkina Faso – 30 (out of 57) sites • Cote d’Ivoire – 30 (out of 83) sites • Niger – 30 (out of 36) sites • Togo – 30 (out of 48) sites CONTROL SITES • Random selection, or exhaustive • Burkina Faso – 30 (out of 32) sites • Cote d’Ivoire – 30 (out of 36) sites • Niger – 30 (out of 30) sites • Togo – 24 (out of 24) sites 3/9/2018 4 Sampling Strategy – Quantitative 3/9/2018 5 Sampling Strategy – Qualitative Purposive Environmental compliance Provider interviews MOH interviews District manager interviews Partner interviews AgirPF Staff interviews Burkina Faso 10 sites 10 1 3 all 2 Cote d’Ivoire 10 sites 10 1 3 all 2 Niger 10 sites 10 1 3 all 2 Togo 10 sites 10 1 3 all 2 (country) 7 (regional) 3/9/2018 6 Quantitative Analysis • Trend analysis for FP service provision • Analysis for ALL Agir Sites & Comparison of intervention vs control • Global New Users • Global Returning Users • LARC Method • During the period from PY2-PY3 • Environmental compliance – descriptive statistics • Analysis done in Stata 3/9/2018 7 Qualitative Analysis • The analysis will follow the general thematic organization of the interview guides, and answers to specific inquiries of interest will be extracted from each transcribed interview and organized by area of interest/theme. • These findings will be triangulated across different key informants, and analyzed in light of the quantitative data provided. • Qualitative analysis were done by hand in each country, with team leads reviewing recorded transcripts and extracting/analyzing data across different users 3/9/2018 8 1: How has AgirPF performed (analysis of monitoring data)? 813 115,781 88,063 204,657 700,000 373,226 265,058 139,037 Cumulative PY1 Cumulative PY2 Cumulative PY3 * Combined PY1- PY3* End of Project Family Planning Adopters Target vs Actual FP Adopters, Py1-Py3 Actual (Projected) Target Gap of 495,343 FP adopters 3/9/2018 9 1: How has AgirPF performed (analysis of monitoring data)? 3/9/2018 10 1: How has AgirPF performed (analysis of site data)? 0 1000 2000 3000 4000 5000 6000 7000 8000 2015q1 2015q2 2015q3 2015q4 2016q1 2016q2 2016q3 2016q4 Number of new users Number of New Users: AgirPF vs Control Sites AgirPF Comparison *p-value <0.01 3/9/2018 11 1: How has AgirPF performed (analysis of monitoring data)? Non-significant finding 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 Number of returning users 2015q1 2015q2 2015q3 2015q4 2016q1 2016q2 2016q3 2016q4 Number of Returning Users: AgirPF vs Control Sites AgirPF Comparison 3/9/2018 12 1: How has AgirPF performed (analysis of monitoring data)? *p-value <0.01 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2015q1 2015q2 2015q3 2015q4 2016q1 2016q2 2016q3 2016q4 Number of New LARC users Number of New LARC Users: AgirPF vs Control Sites AgirPF Comparison * 3/9/2018 13 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? • Multi-pronged approach to increasing HIPs 1. Quality assurance of services (through training, equipping of health centers and facilitative supervision) 2. Advanced strategies including FP days and community health workers to bring services to underserved areas 3. Emphasis on high-impact practices such as post-abortion care and adolescent SRH 3/9/2018 14 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? • High Impact Practices Promoted: – Community health workers – Integration of FP provision into the postpartum period/PAC – Supply chain management/logistics training – Policy/advocacy support to increase access and financing – Mobile outreach through FP special days – Community engagement through site walk-throughs 3/9/2018 15 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? • Regional Strategies for Exchange, Learning, Dissemination 1. Create a pool of FP trainers (training of trainers) 2. Work with MOH to update FP training guidelines to include things such as youth services, HIV integration, gender, etc. 3. Policy advocacy to include HIPs such as focus on youth SRH 4. Empowering/training CHWs to educate/disseminate FP services 5. Building capacity of local partner institutions on HIPs (through: OCAT, COPE) 6. National and regional workshops on FP service delivery strategies 7. Work on logistics/supply chain management delivery systems 8. Engage community leaders/members for site walk-throughs 9. Strengthen local health managers in leadership/management of FP activities 2. How has AgirPF played a regional role for exchange, learning and dissemination of HIPs? Type of Training # ofpeople trained in PY2 # of people trained in PY3 Counseling REDI 390 126 PPPF 255 105 PPIUD 271 86 Contraceptive Logistics/Logistics Management Information Systems 0 151 New integrated FP Curriculum 19 161 COPE 277 260 3Is 12 0 HIV/AIDS 51 24 Advocacy 204 0 MCH 0 0 Reality Check 322 36 COPE for Contraceptive Security 27 0 Facilitative supervision 8 54 Spectrum 12 85 TOT in FP on EH approaches, tools, resources & policies. 42 28 OCAT 0 0 Gender 17 50 Youth Sexual and Reproductive Health 60 0 AgirPF Health Information System 73 0 FP counseling and services training in PAC facilities 137 64 Train health service providers on group and individual FP counseling as part of other services 0 50 Post Abortion Care (PAC) adapted to the youth and adolescents’ needs 0 50 Post Abortion Care (PAC) 0 103 Integration of Human Rights in SRH 0 10 3/9/2018 17 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? • Pros of Intermediate Results (according to respondents) – Holistic approach effective in addressing overall objectives of the project – Three pillars of demand, supply and the creation of an enabling environment are taken into account – The IRs align with national FP repositioning plans (ex: Togo) – Complements public and private structures in enhanced service delivery • Cons of Intermediate Results – They should include more demand generation at community level – There is need for additional raw resources (human resources, supplies, etc) that cannot be fully addressed by the IR framework – Need more emphasis on systemic capacity building (ex: through training curricula in medical/nursing schools) 3/9/2018 18 3. To what extent are the three intermediate results in AgirPF’s results framework and related activities, necessary and sufficient to achieve AgirPF’s overall objective? “The relevance of the AgirPF project is well-planned because they intervene at different levels: they intervene at the community level, they also intervene at the level of the health units, they even intervene at the level of religious leaders and they intervene at the level of the local authorities” – MOH partner (Togo) 3/9/2018 19 4. To what extent has AgirPF contributed to eliminating political obstacles to access to FP in the region? • AgirPF held Training of Advocacy Partners in 2016 • Developed regional network for policy/advocacy partners including members of MOH, civil society/NGOs, religious leaders • AgirPF work has led to – New financing commitments at MOH level for FP services – Improved attitudes and “language” around FP provision – Draft decrees for improved access to SRH/FP – Draft decrees for inclusion of gender issues in SRH/FP – Draft decrees for promotion of FP among youth – No final laws in place, due to long lead times for policy changes 3/9/2018 20 4. To what extent has AgirPF contributed to eliminating political obstacles to access to FPs in the region? "There is a new change in the language of leaders. There has been a political commitment to FP. With the advocacy, there is a draft decree in progress for the application of the texts of PF. This draft order is inadequate to follow up, and also delays due to administrative movements ” – local partner (Burkina Faso) 3/9/2018 21 5. What are success/challenges to improve FP across the region? SUCCESSES: • Establishment of HIPs like PAC-FP and youth SRH in multiple countries • Enhanced knowledge and skills of FP stakeholders and related areas (rights, legislation, management, etc.); • Material support, logistical support for FP service delivery • Dynamic networking among regional actors in FP • Technical and financial involvement in the planning, implementation and supervision of outreach activities • Strengthening of advocacy in-country and across region • Improvement of contraceptive security system and overall system strengthening 3/9/2018 22 5. What are success/challenges to improve FP across the region? CHALLENGES (contextual): • Disbursement of funds for local partners/activities • Lack of availability of staff at the facility level; difficult to retain trained staff • Barriers to commodities supply coordination at the country level • Socio-cultural and religious barriers within the population (pro-natalist and gender hierarchical attitudes that prevent use of FP) • Lack of consistent/adequate funding for FP • Lack of adequate infrastructure and technical equipment for FP 3/9/2018 23 5. What are success/challenges to improve FP across the region? CHALLENGES (related to project): • Coordination of activities with de-centralized structures cumbersome • Justification of the choice of AgirPF intervention areas not always clear (there are other high-need regions for FP that were not addressed) • Slow start-up due to delays at EngenderHealth headquarters • High turn-over of trained staff at facilities; difficult to adequately replace them • There are limitations in infrastructure at facility level that cannot be addressed with current resources • Financing for implementation of activities was slow (due to internal EH mechanisms) • FP targets may be too ambitious given the level of staffing and number of facilities • Spreading focus between high-level (policy) and direct (outreach) activities may dilute effectiveness for short-term outcomes 3/9/2018 24 6. How has AgirPF managed personnel in the countries of activity, consortium partners? • Mixed reviews on management and performance from AgirPF level • AgirPF staff considered very competent and hard-working • Technical and coordination support from AgirPF appreciated by MOH, local partners • Inadequate staffing levels slowed down progress or even the quality of some activities • In some cases, communication was insufficient between regional/country staff and local partners • In some cases, AGIR behaved in a vertical manner. Interventions are carried out on the ground without the knowledge of the health authorities. 3/9/2018 25 6. How has AgirPF managed environmental compliance? • Environmental compliance is low across countries • Many facilities lacking incinerators • AgirPF staff felt that this was an add-on item that was not funded adequately and outside of the scope of the project • Analysis for this indicator is ongoing 3/9/2018 26 Lessons Learned/ Recommendations • The frequent organization of special days, mobile services and guided tours in communities that are high-impact interventions can boost the use of contraceptive methods. • In order to ensure a promising strategy, it is necessary to define realistic and achievable objectives, at the risk of drowning all the effort made. • Political commitment at the highest level is a determining factor in decision-making for the elimination of barriers to access to FP; As well as the very important individual commitment to the use of contraceptive methods. • Through well-done advocacy, policy can be changed and the influence of sociocultural barriers on the use of contraceptive methods reduced • Greater involvement of community leaders has led to greater community mobilization around the use of FP methods. 3/9/2018 27 Lessons Learned/ Recommendations • Delaying the funds available to implementing partners delays activities and hampers the quality of data. The disbursement method should be revised to enable NGOs active in the field to be effective and not run the risk of interrupting activities due to lack of resources. • Prioritize the rehabilitation of PF clinics as this is the beginning of quality PF services guaranteeing confidentiality. • Provide adequate coordination with all levels of the health systems to ensure buy-in and participation to increase sustainability