IMPROVED REPRODUCTIVE HEALTH IN NIGERIA END OF PROJECT EVALUATION AUGUST 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Barbara J. Spaid, Dr. Olapeju Adenusi and Dr. Muyiwa Oladosu through the Global Health Technical Assistance Project. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA END OF PROJECT EVALUATION DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-573) is available in printed or online versions. Online documents can be located in the GH Tech website library at http://resources.ghtechproject.net. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-0. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION i CONTENTS ACRONYMS ............................................................................................................ i EXECUTIVE SUMMARY ..................................................................................... iii Key Recommendations for Future Programming .............................................................vi I. INTRODUCTION ......................................................................................... 1 Purpose ....................................................................................................................................... 1 Background ................................................................................................................................ 1 Methodology .............................................................................................................................. 2 II. FINDINGS ...................................................................................................... 5 Increased Access to improved RH services ....................................................................... 5 Improved Quality of FP Services ........................................................................................ 10 Strengthened Enabling Environments ................................................................................ 13 Expanded Demand for Improved RH Services ............................................................... 16 Effectiveness of Project Management ................................................................................ 19 Appropriateness of Program Approach ........................................................................... 20 III. CONCLUSIONS .......................................................................................... 23 Challenges and Constraints ................................................................................................. 24 Lessons Learned .................................................................................................................... 25 IV. RECOMMENDATIONS FOR FUTURE DIRECTIONS AND OPPORTUNITIES FOR FP SOCIAL MARKETING ................................ 27 Increased Access to Improved FP Services ..................................................................... 27 Improved Quality of FP Services ........................................................................................ 27 Strengthened Enabling Environments ................................................................................ 27 Expanded Demand for Improved RH Services ............................................................... 28 Effectiveness of Project Management ................................................................................ 28 Appropriateness of Program Approach ........................................................................... 29 ii IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION ANNEXES ANNEX A. SCOPE OF WORK .......................................................................... 31 ANNEX B. DOCUMENTS REVIEWED ............................................................ 43 ANNEX C. LIST OF PERSONS CONTACTED ............................................... 45 ANNEX D. KEY INFORMANT, FGD, AND IN-DEPTH INTERVIEW QUESTIONS.................................................................................... 51 ANNEX E. MAP OF NIGERIA AND IRHIN PROJECT COMPONENTS ..... 55 TABLES Table 1. IRHIN CYP and Commodity Distribution over the LOP ................... 5 Table 2. Selected IRHIN FP Commodities Sold at Recommended Consumer Price ....................................................................................... 8 Table 3. Coverage of IRHIN Social Marketing FP Commodities ..................... 8 Table 4. Availability of IRHIN Social Marketing FP Commodities ................... 9 Table 5. Summary of Quality-related IRHIN Project Targets and Achievements ........................................................................................ 11 Table 6. Platforms Used to Engage Key Decision Makers and Professionals on RH/FP ......................................................................... 14 Table 7. Summary of NARHS Indicators on Attitudes and Knowledge of FP .................................................................................... 17 Table 8. LOP Achievements for Interpersonal Communication Activities .. 18 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION i ACRONYMS AOTR Agreement Officer‘s Technical Representative BCC Behavioral change communication CBD Community-based distributor CHEW Community health extension worker CPR Contraceptive prevalence rate CSO Civil society organization CYP Couple year of protection DFID Department for International Development (U.K.) ECP Emergency contraceptive pill EOP End of project ESMPIN Expanded Social Marketing Project in Nigeria FGD Focus group discussion FMOH Federal Ministry of Health FP Family planning FY Fiscal year GH Tech Global Health Technical Assistance Project GON Government of Nigeria HMO Health maintenance organization IEC Information, education, and communication IPC Interpersonal communication IPCC Interpersonal communication conductor IRHIN Improved Reproductive Health in Nigeria IUCD Intrauterine contraceptive device LOP Life of project MAP Measurement and performance MCFWP Managed care and family wellness program MCH Maternal and child health MDS Manufacturers Delivery Services MIS Management information services MMR Maternal mortality ratio MWRA Married women of reproductive age N Naira (Nigerian currency) NARHS National HIV/AIDS and Reproductive Health Survey ii IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION NASFAT Nasrul-Lahi-L Faith Society NCWS National Council of Women Society NDHS Nigeria Demographic and Health Survey OCP Oral contraceptive pill PCN Pharmacists Council of Nigeria PE Peer educator PPMV Patent and proprietary medicine vendor PI Pathfinder International PMP Project management plan POS Point-of-sale PSI Populations Services International QA Quality assurance RH Reproductive health SFH Society for Family Health SOW Scope of work TPM Team planning meeting TV Television UNFPA United Nations Population Fund USAID United States Agency for International Development WHO World Health Organization IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION iii EXECUTIVE SUMMARY In June 2005 the U.S. Agency for International Development (USAID) awarded the Improved Reproductive Health in Nigeria (IRHIN) Project to the Society for Family Health (SFH) and its partners, Population Services International (PSI) and Pathfinder International (PI), with the strategic goal of contributing to an increase in the contraceptive prevalence rate (CPR) for modern family planning (FP) methods to 16%. Key project components included the following: Nationwide social marketing of FP products Behavior change communication (BCC) activities focused on mass media and inter-personal communication (IPC) for demand creation Strengthened enabling environments for FP through work with community and religious leaders as well as national groups Improved quality of FP service delivery through private reproductive health (RH) clinics The purpose of this end-of-project (EOP) evaluation is to assess achievements under the $16.3M IRHIN Project over the life of project (LOP), June 2005 through June 2011. The evaluation addresses six major areas, the first four in greater depth, as these were major expected project results: Increased access to improved RH services Improved quality of RH services Strengthened enabling environment Expanded demand for improved RH services Effectiveness of project management Appropriateness of program approach KEY FINDINGS AND CONCLUSIONS Increased Access to Improved RH Services The IRHIN Project expanded access to FP commodities, services, and information by promoting and detailing an increased number of quality FP commodities through commercial retail sales outlets, public and private clinics and referral facilities, trained retailers, and RH providers. Over 15.2 million couple years of protection (CYPs) were generated through contraceptive commodity sales over the LOP, significantly exceeding the target of 13.4 million. The 12 FP commodities offered under the project represent an impressive range of choice for spacing or limiting family size. The IRHIN Project built upon SFH‘s established and considerable product distribution infrastructure, expanding the number of accredited wholesalers and detailers that distributed 12 FP commodities to over 29,000 commercial retailers nationwide by the EOP to achieve expanded access to FP commodities. IRHIN fell short of meeting LOP targets for product coverage and availability nationwide, as measured in the Measurement and Performance (MAP) Surveys carried out annually under IRHIN. Even so, over the LOP, the project increasingly targeted more semi-urban and rural areas and focused on moderately poor to middle income clients from the second and third wealth quintiles. iv IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION IRHIN made innovative use of multiple access points, including private clinics and civil society organizations (CSOs). The project tapped public and private sector referral facilities for detailing FP commodities, expanding access to FP products and services. The use of IRHIN detailers to stock 28 private clinics over the LOP resulted in fewer than two stock-outs over the LOP; it also reached additional RH providers – in many cases, doctors and nurses attending professional group conferences – creating product awareness and expanding the universe of providers to whom FP products are sold. An additional 2,992 providers were reached by detailers through IRHIN‘s Storming the Nightingale activity, which identified RH service providers in a geographic area and invited them to attend a four-hour seminar on raising FP awareness and counseling skills. IRHIN broke important ground in expanding access points for FP commodities beyond retail outlets to private CSO clinics, private and public referral facilities, community-based distributors (CBDs), and RH providers. Given its linkages to the commercial sector, SFH‘s distribution network should be able to support a robust expansion of nationwide coverage for a more varied product line in support of maternal and child health (MCH) as well as FP. Improved Quality of FP Services. IRHIN‘s efforts to improve RH service quality focused on providing high-quality FP commodities and training for RH providers and retailers, as well as rehabilitating and equipping private clinics to provide FP counseling and service delivery. The numbers of providers trained and facilities rehabilitated were generally used as proxy measures of service quality across the various project components; data was not collected on provider competency or the actual quality of services delivered. By June 2011, 28 private clinics had been rehabilitated and equipped and 258 RH providers trained, against LOP targets of 26 and 87, respectively. With a few exceptions, client satisfaction was relatively high, as measured by one client exit survey mid-way through the project, suggesting that IRHIN‘s capacity-building interventions had been instrumental. Over 22,000 patent and proprietary medicine vendors (PPMVs) were trained against a LOP target of 20,000. A major mystery client survey carried out in 2010 indicated serious gaps in PPMV practices and responses to first time pill users. The project‘s efforts to detail clinical methods to RH providers were innovative, but there were quality concerns regarding the standards and criteria used to detail the clinical methods as well as concerns on the lack of follow-up with RH providers to ensure the appropriate, safe use of project commodities. Generally speaking, quality assurance (QA) systems were not emphasized under the project. Although IRHIN‘s capacity-building training for RH providers very likely improved FP services, the available collected data was insufficient to conclude that the quality of FP service delivery had improved. Strengthened Enabling Environment. IRHIN used a large number and variety of platforms in an on-going effort to engage key decision makers and professional organizations in improving FP understanding and acceptability among journalists, religious leaders, women‘s groups, and medical organizations. Project staff worked with USAID and other donors engaged with the Federal Ministry of Health (FMOH) on various RH/FP issues that contributed to policy changes. For example, the project‘s work helped win agreement by the Pharmacists Council of Nigeria (PCN) for PPMVs to receive training for and re-supply oral contraceptive pills (OCPs) and the FMOH‘s agreement to support IRHIN distribution of long-term contraceptives such as implants; the project‘s efforts also helped support major Government of Nigeria (GON) funding of FP commodities for the public sector. At the same time, improved attitudes toward FP on the part of key decision makers could not be assessed over the LOP due to the failure to conduct a National HIV/AIDS and Reproductive IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION v Health Survey (NARHS) after 2007. Five of the six LOP-required surveys – that is, MAPs, NARHS, NigerBus surveys, Mystery Client, and Client Exit Interview – were carried out and disseminated through the RH Working Group, but the understanding and use of the resulting wealth of data appeared to be weak. A more conducive environment for FP is emerging in Nigeria today. IRHIN has contributed to this improvement, primarily through its efforts at the national and community levels. To ensure continued improvements in FP acceptance and use in Nigeria, this improved environment will need the sustained, expanded attention of stakeholders at all levels over the next decade. Expanded Demand for Improved RH Services. Mass media activities included TV programs and radio drama series, jingles, and program spots highlighting general FP information and specific FP methods. According to the NigerBus surveys, over 25.7 million respondents had listened to the radio drama programs in 2009 and 26.5 million in 2010, against a LOP target of 26 million. No questions were included in the NigerBus surveys or other mass media surveys to measure behavior change that resulted from IRHIN‘s mass media activities. Early NARHS reflect some improvement in FP attitudes and knowledge occurring between 2003 and 2007, but the 2009 NARHS – which would have more fully measured changes in attitude and knowledge of FP methods during the project – was not carried out due to budget and methodological issues. One-year interpersonal communication (IPC) activities carried out in 22 states via community mobilization carried out by peer educators (PEs) and inter-personal communication conductors (IPCCs) successfully reached 49,267 and 393,138 people, respectively, with information on FP. These efforts resulted in 6,387 women and 635 men visiting referral facilities and accessing FP counseling and/or FP services. Community leaders visited by the evaluation team spoke positively about the benefits that accrued to the community from these short IPC activities. IRHIN‘s combined demand creation strategy using mass media and community IPC almost certainly improved overall FP awareness, knowledge, and intention to use, although this cannot be confirmed until the next round of NARHS results are available. The use of CSOs as an entry into the community with PEs and IPCCs was an innovative, effective strategy that complemented the use of mass media to create demand. Effectiveness of Project Management. The three IRHIN implementing partners – SFH, PSI, and PI – brought considerable strength and experience to the project in terms of their understanding of Nigeria‘s diverse social contexts of Nigeria and qualifications in local and international social marketing, BCC, and health research; they also brought to the table numerous research tools to support program monitoring and decision making, delivery of quality health services, and local and international community mobilization. However, project activities tended to be implemented by individual partners rather than fully benefiting from an integrated management approach drawing on partners‘ complementary skills and expertise. IRHIN‘s lack of a centralized, integrated project management structure resulted in the development of parallel management systems for staff, data collection, and activity implementation by each partner. The end result was that collaboration and synergy among the project‘s various components was less than optimal. IRHIN was rated highly by USAID on several occasions for the project‘s quality of data collection as well as its achievement of project indicators. Monitoring and evaluation was facilitated by the project‘s strong use of numerous national surveys and an effective management information system (MIS) to track commodities and ensure regular collection of service statistics on numbers trained. At the same time, project decision making could have made more regular use of information collected by the surveys; another issue was that project staff and USAID failed to adequately discuss and analyze annual and quarterly commodity distribution trends. The lack of more meaningful project indicators and routine, comprehensive data collection from field vi IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION activity sites on service quality, provider skills, and FP use by method prevented IRHIN from analyzing the relative cost-effectiveness of its various innovative activities and approaches. Frequent changes in USAID‘s Agreement Officer‘s Technical Representative (AOTR) for IRHIN did not facilitate the consistent, rigorous USAID oversight needed for such a complex project. While IRHIN made substantial progress in improving FP access, quality, and demand in Nigeria and strengthening the enabling environment, the project‘s tremendous wealth of knowledge and experience was not fully integrated under a centralized management system to ensure maximum benefit and synergy, nor was the most appropriate data collected and fully monitored to support and document desired achievements under IRHIN. Appropriateness of Program Approach. Three sound program strategies were employed under IRHIN – social marketing, mass media and community BCC using CSOs and CBDs, and improved service delivery. Each strategy contributed to achieving the expected increases in FP access, quality, and demand; the strategies also helped build a friendlier environment for FP and increased acceptance of FP in the communities involved. That said, the three strategies were insufficiently linked to achieve a synergy greater than the sum of their parts. While this was a lost opportunity, social marketing of FP products remains an appropriate, important means of expanding access to FP methods for Nigerians. KEY RECOMMENDATIONS FOR FUTURE PROGRAMMING USAID/Nigeria has already made the decision to separate the social marketing and service delivery components combined under IRHIN into two separate projects going forward. IRHIN‘s social marketing component has already been completed and awarded as the Expanded Social Marketing Project in Nigeria (ESMPIN); the service delivery component has been designed and is currently under competition. The evaluation team has therefore focused its recommendations on programmatic and management areas that could benefit these two new projects as well as future RH projects. Increased Access to Improved FP Services New and innovative approaches to increasing access should be better monitored, measured, and documented so that decisions can be based on the relative effectiveness of each effort in increasing not only access to FP but also its uptake. Approaches should also emphasize follow up to ensure FP continuation. Future social marketing projects should continue to focus on new approaches to expand access points in semi-urban and rural areas for better nationwide coverage. They should also emphasize strategies to get products and services closer to clients‘ homes and innovative approaches for detailing clinical methods to trained providers. Efforts should be made to measure product sales closer to the consumer, ideally from the retail shops but certainly from wholesalers, to achieve more accurate couple year of protection (CYP) measurement and commodity sales reporting. Piloting of several new technologies could facilitate this, including simple cell phones. Further experience is needed with the cross-subsidization taking place at private clinics. This experience should be documented and assessed to inform future efforts to complement public sector approaches to meeting the needs of the poor. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION vii Improved Quality of FP Services Exit interviews and mystery client surveys should be conducted more regularly, augmented by regular data collection against enhanced service quality indicators to guide programmatic decision making. QA plans and monitoring should be built into all future projects for FP service delivery and social marketing of FP commodities. Quality assessment tools with a scoring system based on set criteria that define minimum standards for both facilities and RH providers should be developed to measure quality improvement on a regularly basis. There is a need to establish minimum criteria and follow-up and linkage-to-training protocols for detailers selling clinical methods to RH providers. Strengthened Enabling Environments FP received national-level support in 2010 through the designation of a budget line for FP services and commodities. To maintain momentum and follow up on this positive development, efforts should not only continue at the national level but also should also focus in on the state level by identifying state-specific advocacy platforms and engaging change agents in each state. To increase FP access and use nationwide, future RH projects should support efforts to achieve policy changes that allow community health extension workers (CHEWs) to provide injectables outside the facility setting and to insert IUCDs with appropriate training and supervision. The latter policy change is particularly urgent for increasing access and use of longer-term FP methods. Work with community and religious leaders should be continued, with increased efforts to involve men in FP awareness efforts and supportive decision making. Expanded Demand for Improved RH Services Community interventions should be designed to better meet the needs of each community. Appropriate information, education, and communication (IEC) materials should be developed that take into consideration the local language, dress, and culture of individual communities. Given that mass media communication is expensive, data should be collected on behavior change occurring as a result of mass media, in addition to the information already collected on numbers listening to programs and attitudinal change. Community mobilization activities should collect data to track not only referrals, but also the number of people accepting FP by method and number of people continuing use of the method after 12 months. Adequate budgets should be made available for IEC materials for community outreach, facilities, and point-of-sale (POS) materials for retail shops, so such materials can be more fully used when found to be cost-effective. CSOs, RH providers, and shopkeepers should be encouraged to use the materials regularly. Effectiveness of Project Management Future projects should ensure that a clearly defined, integrated project management structure exists that fully incorporates the strengths of all project partners. Such a structure should provide for integrated implementation of all key project components. Projects should viii IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION have a single integrated MIS instead of separate databases maintained by individual project partners. Project staff at all levels should understand and be able to interpret the data that is collected, and should understand how the data relates to their daily work. All staff responsible for program decision making at the central and field level should be trained in the use of all data collection tools, including surveys; they should be able to interpret and use the data in a timely manner to assess problems and identify solutions to improve project implementation and outcomes. In effect, field staff should not rely on headquarters to analyze and interpret data affecting their field operations. Increased analysis – particularly of national survey findings, but also of other monitoring tools – should be regularly carried out by both project and USAID staff to inform project progress and direction, with findings and implications for the project discussed with USAID on a regular basis. USAID and the staff of future projects should work together to develop indicators that are more meaningful and results-oriented. If the MAP methodology requires revision for a new project, that is a topic that should be considered and discussed. Distribution surveys of outlets could also be piloted. USAID should ensure that regular meetings are held with project staff to discuss a variety of issues, including performance, fluctuations in products distributed, results of project￾generated surveys and assessments, and the use of data to improve performance. These discussions should be documented and passed on to ensure continuity in project oversight. Appropriateness of Program Approach Where multiple program approaches are implemented simultaneously under one project, substantial linkages at the management and implementation levels are needed to ensure optimal outcomes. Sufficient appropriate data should be collected to allow for the comparison of the effectiveness of different approaches. Long-term sustainability should be a consideration in future RH projects, with the understanding that long term may mean decades, not years. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 1 I. INTRODUCTION PURPOSE The purpose of this end-of-project (EOP) evaluation is to assess achievements under the Improved Reproductive Health in Nigeria (IRHIN) Project over the life of project, June 2005 through June 2011, toward four expected results: Increased access to improved reproductive health (RH) services Improved quality of RH services Strengthened enabling environment Expanded demand for improved RH services To accommodate additional concerns and questions incorporated by USAID into the scope of work (SOW), two additional areas have been addressed: Effectiveness of project management Program approach The evaluation identifies areas in which IRHIN exceeded or failed to meet expected results as well as innovations adopted under the project. In separate sections, the evaluation will also address constraints and lessons learned. Life-of-project (LOP) funding for IRHIN was $16.3M. BACKGROUND The status of reproductive health in Nigeria is still poor, a fact reflected in the high maternal mortality ratio (MMR) of 545 maternal deaths per 100,000 live births and high fertility, which has fallen only slightly in the past two decades, from 5.9 births per woman in 1991 to 5.7 in 2008. According to the latest Nigeria Demographic and Health Survey (NDHS), current use of contraception has increased slowly, from 6% in 1990 to 13% in 2003 and 15% in 2008. Use of modern methods has also been slow to increase, rising from 4% in 1990 to 8% in 2003 to 10% in 2008. Current use of contraception varies greatly by locale and wealth quintile. Women in urban areas are almost three times as likely to use contraception as those in rural areas (26% and 7%, respectively). Women in the South West Zone have higher contraceptive use (32%) than those in the North West Zone (3%); similarly, women in the highest wealth quintile are more likely to use contraception (35%) than those in the lowest wealth quintile (3%). Knowledge of family planning (FP) methods is surprisingly high, given the low contraceptive prevalence rate (CPR), with 72% of all women and 90% of men knowing at least one contraceptive method. Unmet need for FP is estimated at 20% for currently married women of reproductive age (MWRA), with an unmet 15% spacing need and 5% limiting need. The IRHIN Project was awarded in June 2005 to the Society for Family Health (SFH) and its partners, Population Services International (PSI) and Pathfinder International (PI), with the objective of contributing toward increasing the CPR for modern FP methods (2007 IRHIN PMP) to 16% and achieving 13.4 million couple years of protection (CYPs) over an initial five-year period. The project was later extended an additional year through June 30, 2011. Key project components included the following: Nationwide social marketing of quality FP commodities through patent and proprietary medical vendors (PPMVs) and pharmacists 2 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Improved quality of service delivery through 28 private RH facilities within the three target states of Abia, Cross River, and Kaduna through collaboration with the Nigerian Medical Association and the Association of General Medical Practitioners in Nigeria Community-based distribution of FP products targeted to hard-to-reach communities in the three target states Community mobilization interventions in 22 IRHIN expanded states using peer educators (PEs), interpersonal communication conductors (IPCCs), and detailing of FP products to referral facilities near the communities Mass media and community-based IPC activities for increased demand A strengthened enabling environment for FP through work with federal and state ministries of health, community and religious leaders, and national associations such as the Pharmacists Council of Nigeria (PCN) and other professional associations METHODOLOGY The evaluation team used a mixture of quantitative and qualitative data review and collection approaches to gain insight on the impact of IRHIN activities and the processes that led to those impacts. A variety of methods and approaches were used to collect and analyze information relevant to the evaluation‘s objectives and questions outlined in the SOW for the evaluation, which is found in Annex A. The following methods and approaches were used: 1. Review of Background Materials. Documents relevant to the IRHIN Project were identified by USAID and assembled for review and analysis. These included Annual and Quarterly Reports and indicators, the Mission‘s 2007 IRHIN project management plan (PMP), the IRHIN Cooperative Agreement, past project assessments, surveys and evaluations, the USAID/Nigeria Strategy, USAID/Nigeria‘s maternal and child health (MCH), FP, and RH strategies, and other public documents related to the project. A complete list of documents consulted is found in Annex B. 2. Team Planning Meeting. The team conducted a two-day team planning meeting (TPM) upon arrival in Nigeria and before starting the in-country portion of the evaluation. During the TPM, the team reviewed and clarified questions on the evaluation SOW, drafted an initial work plan, developed a data collection plan, finalized evaluation questions, developed the evaluation report table of contents, clarified team members‘ roles, and assigned drafting responsibilities for the evaluation report. The TPM outcomes were shared with and approved by USAID/Nigeria; the USAID health team participated in sections of the TPM. 3. Key Informant Interviews and Key Evaluation Questions. The team conducted interviews with key informants from USAID, USAID implementing partners, the Federal Ministry of Health (FMOH) counterparts, UN agencies, donor organizations, and PCN and other private sector organizations. The team used a set of eight key evaluation questions identified during the TPM, based on the evaluation‘s major objectives and illustrative questions included in the SOW: – Has the IRHIN Project increased access to FP and RH services? (coverage, targeting, effectiveness, etc.) – Has the project improved quality of RH service? (capacity of providers, quality standards and protocols, availability of contraceptives, client satisfaction, etc.) – How has IRHIN contributed to strengthening the enabling environment for the provision of FP services? (policies, regulations, linkages, Government of Nigeria [GON] support, attitudinal change among religious and community leaders, etc.) IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 3 – Has IRHIN expanded demand for FP and RH services? (effectiveness, appropriateness of communication strategies, coverage of mass media and interpersonal communications [IPC], information, education and communication [IEC) materials, etc.) – How effectively has IRHIN project management contributed to IRHIN achievements? (management approach, roles and contributions of sub-partners, communication, monitoring and evaluation [M&E] flow of data and information, etc.) – What were the strengths and weaknesses of the programmatic approach adopted under the project? (social marketing‘s appropriateness as a strategy for improving FP use?) – What worked well and what did not under IRHIN? (lessons learned) – What future directions should social marketing take in Nigeria over the next 5-10 years? (linkages with MCH or other health products, appropriateness of social marketing to increase utilization of FP, etc.) A complete list of persons contacted during the assignment is found in Annex C; the full set of interview questions is included in Annex D. Site Visits. Time and security constraints limited the geographic scope of field visits; however, a nationwide perspective was still achieved by visits to the four states of Kaduna, Lagos, Enugu, and Ebonyi, as well as the environs of Abuja. To cover more organizations and project sites, the team split into two groups most days. The evaluation team was joined for the site visits by two USAID project staff as well as by several SFH field staff. The team visited SFH field offices, civil society organizations (CSOs), private clinics and referral facilities, Manufacturers Delivery Services (MDS) offices, numerous PPMVs, and wholesalers. Focus group discussions and in￾depth interviews were carried out with PEs, IPCCs, and non-formal health providers. A map of Nigeria with identification of the states where various IRHIN project components were carried out is provided in Annex E. Data Analysis and Report Writing. The team discussed and reached agreement on findings, conclusions, and recommendations that were presented during three de-brief presentations to USAID‘s Health Team, IRHIN staff and the new Expanded Social Marketing Project in Nigeria (ESMPIN) team. A draft report was prepared and submitted to USAID/Nigeria before the team departed from Abuja. Limitations Associated with the Evaluation Methodology. The team noted several limitations inherent in the evaluation methodology; these limitations were discussed with USAID during the in-country de-brief and are described below in detail at the request of USAID/Nigeria. As noted earlier, the team was limited to five days for site visits outside of the Federal Capital Territory and security concerns further restricted the areas it could visit. Team members were able to visit only one of the three focus states (Kaduna) for Improved Quality of Service Delivery; unfortunately, the Kaduna-based activity visits were limited to one day. Because the team split up, only one team member was able to visit just two of the private clinics under this important IRHIN component. The team was scheduled to visit four of the health facilities, but one could not be assessed due to security reasons and the other was closed due to internal management issues at the time of the visit. In retrospect, the team‘s view was that all three team members should have seen at least five of the private clinics in Kaduna to achieve a better assessment of the service delivery component, given the security situation precluded visits to the other two states (Abia and Cross River) where this component was implemented. As noted above, the team was accompanied in the field and to most stakeholder meetings by two USAID project staff, the Agreement Officer‘s Technical Representative (AOTR), also known as the IRHIN Project Officer or the IRHIN backstop officer, and the Deputy AOTR, as specified in the evaluation SOW. At field sites, team members were accompanied by two to four SFH field staff. While this greatly facilitated visits and introductions – and the HPN/Nigeria team 4 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION routinely provides this level of facilitation to external teams – it did make for a large team, even when split into two groups. Moreover, the presence of both IRHIN and USAID staff may have limited the willingness of those interviewed to discuss negative aspects of the project, including challenges, constraints, and areas needing improvement. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 5 II. FINDINGS Most project indicators set out in 2007 in the IRHIN PMP were met or exceeded, with a few exceptions. Achievements against indicators will be discussed in the appropriate sections below within the context of other achievements and shortcomings not necessarily captured by the PMP indicators. INCREASED ACCESS TO IMPROVED RH SERVICES IRHIN expanded access to FP commodities, services, and information by promoting and detailing an increased number of quality FP commodities through commercial retail sales outlets, public and private clinics and referral facilities, and trained retailers and RH providers. This is clearly reflected in the increased annual sales of individual commodities and in the total couple years of protection (CYPs) generated by contraceptive commodity sales of 15.2 million over the LOP. This indicator significantly exceeded the CYP LOP target of 13.4 million (Table 1). The IRHIN Project‘s strategic goal was to contribute to the attainment of a CPR of 16% for modern methods (2007 PMP). While the 15.2 million CYPs generated by FP commodity sales under IRHIN clearly contributed to the national CPR, it is unlikely that Nigeria‘s CPR for modern methods would have increased to 16% by 2011, given that the CPR for modern methods was calculated at 10% at the time of the last NDHS in 2008. That said, the final, definitive determination on this issue must await the next National HIV/AIDS and Reproductive Health Survey (NARHS). Both CYPs and annual commodity sales are calculated using sales data obtained from the MDS warehouses, which means that a considerable amount of FP commodities reflected in CYP figures is actually sitting with wholesalers, sub-wholesalers and retail stores, also known as PPMVs. Central to the IRHIN Project was the use of SFH‘s established and considerable product distribution infrastructure. This infrastructure includes a central warehouse in Lagos and use of MDS to move commodities from Lagos to 20 MDS warehouses around the country, from which accredited SFH wholesalers (pharmacists) buy FP commodities on a cash basis and sell to sub￾wholesalers and PPMVs at the furthest points in the distribution system. Detailers employed by SFH facilitate distribution at all levels, especially to PPMVs and RH clinic/facility providers. Expansion of Distribution Efforts and Retail Access Points. Under IRHIN, SFH‘s existing distribution network has improved and expanded, beginning with the move to a larger SFH warehouse in Lagos with special climate control areas for FP commodities that need to be stored at a controlled temperature. Table 1. IRHIN CYP and Commodity Distribution over the LOP Product Baseline FY 2006 FY 2007 FY 2008 FY 2009 FY 2010 FY 2011 Qtr 1-3 LOP Target LOP Achievement Gold Circle & Lifestyle Condoms 150M 157.4M 169.7M 187.8M 194.7M 196.8M 169.9M 1,025M 1,076M *CuT IUCD N/A 45,810 53,160 54,300 86,400 72,750 50,460 271,982 362,880 *Depo￾Provera Injectable 571,500 743,200 959,500 1.2 M 1.3 M 609,350 935,100 4.3M 5.8M 6 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Product Baseline FY 2006 FY 2007 FY 2008 FY 2009 FY 2010 FY 2011 Qtr 1-3 LOP Target LOP Achievement Noristerat Injectable 490,700 686,900 896,000 1.1 M 645,703 1.1M 845,900 3.5M 5.3M Norigynon Injectable N/A N/A N/A 44,760 3,101 44,800 17,700 N/A 110,361 *Duofem -- OCPs 4M 4.5M 4.8M 5.6 M 6.3 M 6.1 M 5.6M 29M 33.0M *Locon-F -- OCPs NA NA NA NA 17,100 N/A N/A N/A 17,100 Postinor-2 & Pregnon – ECPs NA 609,600 796,610 828,650 975,155 1.2 M 639,660 4.2M 5.1M *Jadelle – Implant NA N/A N/A 510 6,983 6,660 1,380 N/A 15,533 *Cycle Beads NA 900 5,000 9,400 6,100 9,800 5,000 125,000 36,200 CYPs 1.9M 2.1M 2.5M 2.7M 2.9M 2.8M 2.4M 13.4M 15.4M * USAID/IRHIN-funded. N/A means product not available through IRHIN; in the case of baseline figures and LOP targets, none were established, generally because they were FP products introduced after IRHIN had started. Source: USAID/PMP (baseline and LOP targets) and SFH MIS (annual and LOP achievements). The number of accredited wholesalers who pull commodities from MDS has increased from 40 to 60 and the number of SFH detailers dedicated to IRHIN has risen from 8 to16 over the LOP. In addition, partial support was leveraged from an additional 24 detailers from other SFH￾implemented projects that carry out social marketing for other health products. By the EOP, over 29,000 commercial retailers (PPMVs and pharmacists) nationwide were stocking IRHIN FP commodities. This increase in wholesalers, SFH detailers, and commercial retailers over the course of the project correlates with the overall increase in FP commodities sold over the LOP, as reflected in Table 1. More important, it represents a huge nationwide distribution and sales network. According to the 2008 NDHS, 48.3% of all users of a modern FP method cited PPMVs and pharmacists as their most recent source of FP. Expansion of FP Product Choice. The IRHIN project expanded access and choice by greatly increasing the line of FP commodities distributed over the LOP from the 6 products supported by USAID (Duofem OCP, Copper T 380A IUCD, Depo-Provera injectable, Locon-F OCP, Jadelle implant, and Cycle Beads, a standard days method) to a total of 12 FP products. Four of these additional products (Gold Circle condoms, Noristerat injectables, Norigynon injectables, and Pregnon, an emergency contraceptive pill [ECP]) were leveraged from the British Department for International Development (DFID) just prior to IRHIN start-up through agreement between USAID and DFID. The particular products were chosen to complement those provided by USAID. Two additional products were added on a cost-recovery basis (Life Style condoms and Postinor-2 ECPs). IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 7 Product Targeting. The cooperative agreement for the IRHIN Project specified a target group by age and marital status, primarily for demand creation. These target groups were the following: Women in union 18–30 Men in union 18–40 Unmarried urban men and women 18–30 There were no specific targets set for either rural or urban market penetration. The primary concern was to expand the market reach for improved access nationwide. In fiscal year (FY) 2010 IRHIN began to focus on adding and training more rural and semi-urban PPMVs for improved access in those areas. While specific studies of rural penetration for IRHIN FP commodities were not carried out, according to the 2008 NDHS, 40% of women using socially marketed oral contraceptive pills (OCPs), while 56% of women using socially marketed brands of injectables came from rural areas. However, the 2007 NARHS reported that among rural respondents, only 18% reported that the pill was easily obtained and just 17.7% said injectables were easily obtained. The small amount of data available from national surveys is not conclusive; additional, more project-focused analysis of the rural market would facilitate better targeting of the rural market to achieve increased access. Product Positioning and Pricing. While there was no targeting by wealth quintiles mandated in the IRHIN Cooperative Agreement, the project focused on the 2nd and 3rd quintiles, characterized as middle income to moderately poor, as an appropriate target for socially marketed FP products. No differentiation was made at this point among those living in urban, semi-urban, or rural areas, nor does it appear that USAID and the IRHIN Project staff discussed tracking sales by market characteristics. In an effort to reach these lower-to-middle-income groups, SFH field offices first identified moderately poor areas or communities and the detailers would then identify and work with PPMVs in those areas. Products were priced to appeal to this moderately poor, middle-income group based on an early-willingness-to-pay analysis carried out by SFH in 2006 and a ―basket of goods‖ analysis. The 2008 NDHS shows that the heaviest use of the IRHIN FP products is among the three wealthiest quintiles. Over the LOP, four ―luxury brands‖ were introduced to attract wealthier users to less heavily subsidized brands: Life Style condoms, Locon-F OCPs, Norigynon injectable, and Postinor-2 ECPs. To date, Life Style condoms have captured only a small portion of the total market (3.7%) for the two socially marketed brands, as has Locon-F for OCPs (<1.0%), and Norigynon injectables (1.0%). Sales for Postinor 2, on the other hand, represent 91% of the ECPs marketed under IRHIN, largely because it was introduced by SFH in 2002 as the sole ECP on the market. Although Pregnon, the less expensive ECP, was not introduced until 2007, its sales are increasing rapidly. It is likely to eventually overtake Postinor, as Pregnon is sold for approximately one-third the price of Postinor 2. Because IRHIN commodities are marketed through commercial distributors, suggested prices cannot be enforced and the prices charged by retailers generally exceeded the retail prices suggested by IRHIN. Prices for several of the products had been increased by the project‘s end: Gold Circle condoms were raised from 20 naira (N) to 25 N; Duofem from 30 N to 35 N; and Depo-Provera from 90 N to 130 N. Several of the PPMVs visited were still charging the old price of 30 N for Duofem. The Marketing and Performance (MAP) surveys carried out annually by IRHIN, in addition to tracking changes in product coverage and availability, also track the percentage of retailers selling each product at the recommended price. While the nationwide percentage of retailers selling IRHIN FP products at suggested retail prices varies by year and product, the MAP surveys show a sharp decline in 2010 of the percentage of retailers who sold these products at suggested retail prices (Table 2). The tendency of many retailers to sell slightly 8 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION above the recommended price was confirmed by the team‘s field visits to PPMVs. Several PPMVs said they often buy from other retailers or sub-wholesalers for added convenience, resulting in an extra profit margin that gets figured into the price and eventually charged to the buyer. One PPMV said the price he charges for products depends on where he gets his stock, i.e., how many in-between dealers add their margin to the price he is charged. PPMVs noted that when a product becomes scarce in an area, its price will tend to rise. Table 2. Selected IRHIN FP Commodities Sold at Recommended Consumer Price Year/Product Duofem/Confidence Gold Circle Depo-Provera 2006 24% 70% 3% 2007 26% 60% 37% 2008 10% 29% 17% 2009 31% 65% 3% 2010 0% 9% 4% Source: MAP Surveys, 2006, 2007, 2008, 2009, 2010. Nationwide Coverage and Availability of IRHIN FP Products. The IRHIN PMP set LOP targets for coverage and availability for three key FP products: Duofem, all injectables as a group, and Gold Circle condoms. These were monitored annually through the MAP surveys, which collected information on product coverage and availability as well as prices charged and presence of promotional materials at retail points across the country‘s six geo-political zones. Nationwide coverage of 75% for Duofem was attained in only two years: 2006 and 2009; nationwide coverage of 90% for Gold Circle condoms was never attained during the LOP. Nationwide coverage for injectables as a group is difficult to determine since the MAP surveys measured coverage by individual product but not as a group. However, what is clear is that LOP coverage against targets was not achieved and coverage for all products fell somewhat in 2007 and 2010. While the IRHIN Project has documented stock-outs in 2010 for injectables due to quality control and packaging issues, this does not account for declines in coverage of the other products. Interestingly, the dip in coverage and availability of all FP products in the 2010 MAP correlates with declines in 2010 distribution of the intrauterine contraceptive devices (IUCDs) Depo, Duofem, and Jadelle. IRHIN research staff attribute fluctuations in coverage and availability as measured through MAP to limitations in MAP methodology as it was configured for this project; however, it is not clear to the team that discussions with USAID were ever taken to modify the methodology to make the results more appropriate and usable to IRHIN; in addition, the team was never able to confirm that year-to-year anomalies in coverage and availability were routinely discussed and documented between USAID and IRHIN staff. Table 3. Coverage of IRHIN Social Marketing FP Commodities Product Baseline 2006 2007 2008 2009 2010 LOP Target *Duofem (OCP) 65% 75% 61% 68% 79% 64% 75% Postinor-2 (ECP) 52% 37% 50% 49% 32% Noristerat (Injectable) 47% 45% 41% 55% 43% IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 9 Product Baseline 2006 2007 2008 2009 2010 LOP Target *Depo Provera (Injectable) 45%** 46% 50% 49% 54% 44% 60%** *Locon-F (OCP) N/A N/A N/A 14% 6% Pregnon (ECP) N/A N/A 11% 13% 6% Norigynon (Injectable) N/A N/A 7% 3% 7% Gold Circle (Condom) 85% 77% 63% 68% 76% 61% 90% *USAID-purchased commodities; CuT, Jadelle, and Cycle Beads not included in survey. ** Target was for all injectables but MAP looked at each separately. Source: MAP 2006, 2007, 2008, 2009, 2010. Availability of products in retail shops, also tracked annually by the MAP surveys, shows that almost all products experienced a small amount of stock-out, which IRHIN explains as normal in the commercial sector. Gold Circle shows the least amount of stock-out over the five years (Table 4). Neither the MAP surveys nor the annual project reports highlight or try to account for dips and deficiencies in coverage, access, and prices charged as documented by MAP surveys. IRHIN research staff attribute these annual fluctuations to normal sampling differences in the MAP surveys. Table 4. Availability of IRHIN Social Marketing FP Commodities Product Baseline 2006 2007 2008 2009 2010 LOP Target *Duofem (OCP) 65% 71% 58% 68% 76% 58% 75% Postinor-2 (ECP) N/A 41% 34% 49% 46% 29% N/A Noristerat (Injectable) N/A 45% 42% 39% 49% 40% N/A *Depo Provera (Inj.) N/A 40% 39% 47% 53% 39% N/A *Locon-F (OCP) N/A N/A N/A N/A 4% 5% N/A Pregnon (ECP) N/A N/A N/A 10% 10% 6% N/A Norigynon (Injectable) N/A N/A N/A 6% 3% 3% N/A Gold Circle (Condom) N/A 76% 63% 68% 75% 60% N/A *USAID-purchased commodities; Jadelle and Cycle Beads not included in survey; baseline and target data only available for Duofem. Source: MAP 2006, 2007, 2008, 2009, 2010 10 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Expanded Efforts to Increase Access Points through Community Mobilization, Referral Facilities, and Private Clinics. A notable achievement under IRHIN was the innovative use of multiple access points, including private clinics, CSOs, and public and private sector referral facilities for detailing FP commodities to expand access to FP products and services. In three states, Abia, Cross River, and Kaduna, IRHIN renovated and equipped 28 private clinics and trained providers in counseling and provision for a range of FP products, detailed or sold by the project. A total of 258 providers were trained, comprising nurses, doctors and community health extension workers (CHEWs), greatly exceeding USAID‘s LOP target of 87. This program component confirmed that private clinics can work well as access points once they are equipped and stocked with subsidized FP products with which they could make a profit and staff are trained. Stock-outs were minimized by detailers, for a LOP achievement of only two reported episodes of clinic stock-outs, well under the LOP target of less than seven reported stock-outs. The two clinics visited by the team were still operating and offering FP products after the end of the project. These clinics cited their continued provision of FP services and products as community service that generates good will back to them. With regard to pricing products and services, their guide was not the suggested retail price, but each client‘s appearance of ability to pay. Thus they used a form of clinic-based cross-subsidization, by which they charged those who appeared wealthier higher prices and lower prices for those who appeared unable to afford the suggested price. The community mobilization scheme implemented in 22 states with CSOs, peer PEs, IPCCs, and over 40 referral facilities effectively increased access to services and FP products, although the primary intent was to create demand and improve FP use. In addition to detailing FP products to private clinics and referral facilities, two additional innovative approaches were taken for reaching RH providers, mostly doctors and nurses, through detailer attendance at 30 to 40 professional group conferences over the LOP. The conferences were geared to create product awareness and expand the number of providers to whom FP products were sold. The second activity, Storming the Nightingale, identified RH service providers in a given area and invited them to attend a four-hour seminar presented by an OB/GYN consultant on raising FP awareness and FP counseling skills. IRHIN detailers were also present and sold FP products to the attendees after the seminar. Over a two year period, the Storming the Nightingale activity covered 3,992 providers – 3,250 females and 742 males. While all of these innovative schemes increased access to FP services and products, there is insufficient data to judge how successful they were in generating FP acceptors, the ultimate goal of the project. Certainly the last two approaches raise concerns over the fact that there was no assessment of the providers‘ competency to provide the clinical methods, particularly the IUCD and implant, nor was there a follow-up plan for assessing quality of service delivery and restocking FP products. IMPROVED QUALITY OF FP SERVICES Efforts to improve the quality of RH services under IRHIN focused largely on providing high￾quality FP commodities, training RH providers and retailers, rehabilitating and equipping 28 private clinics, and tracking the number of these private clinics providing FP counseling services. Thus, the number trained was generally used as a proxy measure of service quality across the various project components. Training at 28 Private Clinics. RH training was conducted using a curriculum that was jointly developed with the FMOH in line with the national FP standard of practice guidelines and clinical protocols. The training content, which included FP counseling, contraceptive technology and IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 11 updates, and IUCD insertion and removal, was provided largely to nurses and doctors from the 28 service delivery sites in Abia, Cross River, and Kaduna. These delivery sites were affiliated with eight CSOs already working in RH in these states. The CSO were identified by IRHIN staff, assessed against an organizational assessment tool, and chosen based on those results and their willingness to participate. This project component drew extensively on Pathfinder International‘s extensive expertise in improving RH service quality. Additional training sessions on implant insertion were conducted for doctors at service delivery sites. Where there was a shortage of nurses, CHEWs were trained on FP motivation, counseling, FP methods, and record keeping and documentation. In addition to the training of RH providers, 28 private health facilities were renovated and provided with equipment for FP services as well as documentation tools to capture service statistics. By the end of the project, two facilities were dropped from the project for lack of performance. In all, 258 RH providers were trained against the LOP target of 87 providers. Table 5. Summary of Quality-related IRHIN Project Targets and Achievements Indicator LOP Target LOP Achievement No of health facilities rehabilitated/equipped 28 28 No of counseling visits for FP/RH as a result of USG assistance 81,000 526,977 No of service delivery points providing FP counseling and services 28 26 No of beneficiaries of clinic activities None Established 100,834 No of people trained in FP/RH (total trained includes over 22,000 PPMVs) 20,000 25,656 F=9,371 M=16,285 No of providers trained in SD sites in 3 states 87 258 Source – SFH database. Team visits to the private clinics in the service delivery target area were extremely limited, but providers at the two clinics visited reported they had benefited greatly from the trainings through increased knowledge and skills, particularly in relation to long-term FP methods. Providers credited the training with improving their confidence to counsel and attend to clients who come for FP services. The two clinics observed had separate rooms for FP counseling with well-displayed job aids, IEC materials, and commodity samples to support counseling. The clinics were well stocked with a full range of FP commodities and were secured in locked cupboards. These two facilities reported receiving referrals from surrounding public health facilities particularly for long-term methods even after the project‘s end in June 2011. The preferred method of clients at these two clinics was the injectable. Only one client exit interview survey was carried out at clinics in the service delivery target area toward the end of the project in 2009. These findings revealed the following: 98.4% of respondents were treated politely 99.2% reported providers took time to listen to their worries 93.5% were counseled on child spacing choices and involved in the selection of their contraceptive method 98.4% reported the display of information materials on the wall, but only 60.5% were given information materials 75% of respondents reported waiting an appropriate length of time to receive services 12 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 96.8% of respondents reported their satisfaction with services provided in the clinics Thus client satisfaction, as measured by the client exit survey – an important indicator of service quality – was relatively high with a few exceptions mid-way through the project, suggesting that IRHIN‘s capacity-building interventions were instrumental in this improvement. Other Facility-based RH Provider Training and Quality of Services. In addition to the RH providers trained under the Service Delivery component, 109 nurses and doctors from other states operating within the broader 22 IRHIN focus states received one week of update training on IUCD insertion and removal, and another 869 providers benefited from two-day training sessions on contraceptive updates. Some of the providers who were trained were affiliated with referral facilities within the community mobilization activities, while some had been referred by SFH detailers. The IRHIN Project also leveraged other donor and other USAID-supported projects (e.g., the Women‘s Health Project, COMPASS, PSP One) to carry out these additional trainings. The team did not see evidence that skill improvement was regularly measured for any of the trained providers across the entire IRHIN Project. There was no indication of a clear strategy for training, monitoring, follow up, or measuring client satisfaction at these public and private facilities that served as referral centers for the project‘s community-level interventions in the other states. Service data collected by the three CSOs visited was incomplete and inconsistent across CSOs. There were also varying interpretations of what ―FP services availed‖ meant: Some providers registered only FP acceptors here and others included FP counseling visits as well. Because community mobilization activities lasted only one year, no follow up was done to determine FP continuation rates. Service data was insufficient to determine either the effectiveness of the approach or the quality of services. Based on comments from numerous project sites visited, the IRHIN Project provided good visibility to the important role clinics and referral facilities play in providing quality RH services. Quality of FP Commodities. Based on discussions with the FMOH, other donors, and stakeholders, the quality of FP commodities distributed through IRHIN was considered to be quite high. Observations from the team‘s visit to the SFH warehouse in Lagos and other points in the distribution chain confirm that commodities are generally being stored and handled appropriately. Condoms are tested at the Lagos warehouse by FMOH staff for packaging integrity, pressure, and dimension standardization. Products were properly stored and moved from the warehouse according to expiration date. Two climate-controlled rooms were being used according to product requirements. There were no expired drugs in the warehouse at the time of visit, nor did the team encounter any in shops. Storage conditions were adequate at the MDS sites visited except in Kaduna, where the air conditioning in the cooling room had not worked for several weeks. Products were generally being stored according to guidelines at the wholesalers and PPMV levels, although at one wholesaler‘s facility the team saw a carton of Gold Circle with obvious water damage stains. Most PPMVs tended to store condoms on the topmost shelves, which is the hottest part of the shop; one PPMV displayed Gold Circle condoms in a case with direct sunlight. Improving the Knowledge of Private Patent Medicine Vendors (PPMVs). A major intervention under the IRHIN Project for improving service quality was capacity building of the PPMVs. PPMVs are a significant group in providing FP and other health care products and information. They are often the first contact point for medical concerns, including FP services. As reflected in the 2008 NDHS, 38.6% of users of modern FP methods cited the PPMVs as their most recent source for obtaining a FP method. PPMVs were identified in collaboration with the PCN and enlisted into the project by the detailers. PPMVs tend to be more numerous in urban and semi-urban areas. Although a handful of PPMVs are trained nurses, the vast majority have IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 13 little or no knowledge of FP. For this reason, the IRHIN Project targeted PPMVs for training on the importance of FP to child and maternal health, knowledge of FP methods and their proper use and side effects, and improved dispensing skills. Over the LOP, over 22,000 PPMVs were trained against an initial target of 20,000. One major Mystery Client Survey was conducted in 2010 to assess the quality of information and services provided by the PPMVs. Since no similar survey had been carried out at the start of IRHIN, there was no baseline for measuring improvement. According to findings, only 44.6% of the PPMVs surveyed referred first time pill users for medical check-up, 38.6% prescribed daily oral pills while 16.8% sold pills to the first time pill clients. Thus fewer than 50% of PPMVs took the correct action to first time pill users. Only 15% of the clients who were sold oral pill were given information on possible side effects and only 12% were counseled on how to manage side effects. Of the mystery clients who presented with complications from pill use, 51% were referred to a health facility, while 11% were prescribed treatment and 6% received a changed pill type despite the fact that it is beyond the scope of their practice to do so. These findings show that while the quality of services and information may have improved, serious quality concerns remain and need to be addressed through additional follow-on training. Of particular concern is the low (44.6%) referral of new pill clients to health facilities. PPMVs visited mentioned fear of losing clients as a reason for prescribing without referring for medical checkups. As this survey was conducted toward the end of the project, it is unclear what if any follow-up actions were taken to address these issues. PPMVs visited by the team confirmed that they had received training on the importance of FP, different methods, potential side effects, how to counsel and interact with FP clients, legal/ethical restrictions, and proper storage of FP commodities. The PPMVs also reported having benefited from the trainings, as their knowledge of FP issues had increased as had the recognition of their limitations in providing specific FP methods, e.g., the need to refer a first-time pill user to a clinic for a medical checkup. Several PPMVs confirmed their reluctance to refer first-time pill clients to facilities for fear of losing future trade. In all, over 22,000 PPMVs benefited from orientation and training sessions held over the life of the project. Detailing to RH Providers through ―Storming the Nightingale‖ and Professional Group Conferences. As innovative as these interventions were in increasing product detailing to an expanded number of RH providers, they also raise several serious quality issues. There were no clearly defined standards for provider skills or follow-up activities to determine service quality or the skill level of providers in the clinical FP methods. The nurses, doctors, and pharmacists who attended the ―Storming the Nightingale‖ sessions received only four hours of counseling and general FP information; those providers attending the conferences would have received one-on-one product-specific information from the detailers, but again, with no follow up. There was a serious gap in identifying and using opportunities to monitor providers‘ clinical capacity to provide quality FP services and linking them with other projects focused on building the capacity of providers STRENGTHENED ENABLING ENVIRONMENTS A key USAID indicator of a strengthened enabling environment under the IRHIN Project is the number of reproductive health surveys conducted and disseminated to the Government of Nigeria and the FMOH. Findings suggest that the project conducted five of the six surveys required by USAID during the LOP. Surveys conducted included NARHS, MAP surveys, and mystery client surveys. Findings from the surveys were disseminated through the RH Working Group meetings coordinated by the FMOH and through the sharing of hard copies with stakeholders in the field of RH/FP. Findings suggest that the IRHIN Project performed well on this indicator except for the fact that information from these surveys did not appear 14 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION to have been fully used for on-going decision making either outside or within the IRHIN Project. The connection between dissemination and use appeared to be weak. Another performance indicator that the project was required to address is percentage of all women aged 15 to 49 and men aged 15 to 64 who believed that parents, community leaders, and religious leaders support FP use by couples. This indicator aimed to provide insights on changing attitudes of key decision makers in society toward FP. Statistics from the NARHS showed that the support of parents, community leaders, and religious leaders for family planning declined between 2005 and 2007 (46% vs. 40%, 46% vs. 37%, and 40% vs. 32% respectively). The reasons for the decline are not unclear, though aspects may be due to generally unfavorable government policies (predicated on an unpopular military regime) and social conditions at the time. While it was expected that results of a projected 2009 NARHS would show substantial improvement in the proportion of key decision makers who support FP use by couples between 2007 and 2009, no NARHS have been carried out since 2007 due to funding constraints and a desire to make changes to the methodology. Table 6. Platforms Used to Engage Key Decision Makers and Professionals on RH/FP FY Year Activity Objectives Outputs & Outcomes FY 07 Workshop for electronic media practitioners Meeting with PCN Work with health maintenance organization (HMO) and other stakeholders on managed care and family wellness programs (MCFWP) Correct misconceptions about RH/FP PPMV training on oral contraceptives FP to form part of HMO services 14 participants educated about RH/FP PCN agreed that PPMV receive training OCPs FP included in HMO services FY 08 Review of FP materials Workshop for female religious leaders Participated in HMO providers training Supported NASFAT Women Conference Sensitization workshop for NCWS Participation in World Population Day Religious Leaders Consultative Forum Meeting with FMOH on long term contraceptives Launch of Locon F Need for harmonized materials for FP Sensitization of women leaders on MCH/FP Improvement of method mix in HMO facilities Increased visibility of RH/FP issues Sensitize women leaders on MNCH issues Moderated a youth forum to improve discussion on RH/FP Putting MCH in the front agenda Obtaining policy support for long-term methods Discussions to get acceptance and registration Improved understanding and utilization of FP products & services 41 Christian and Islamic women leaders participated 30 participants received info about FP products 600 participants exposed to RH/FP issues Plans to actualize improved MCH Moderated a youth forum, and participated in a radio program Formal inauguration of interfaith forum to include discussion on MCH FMOH agreed to support IRHIN long-term products Obtained legal backing for the distribution of branded products IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 15 FY Year Activity Objectives Outputs & Outcomes FY 09 Adoption of FP advocacy toolkit Participated in safe motherhood day celebration To gain policy support WHO FP advocacy kit similar to that of IRHIN To increase advocacy on MCH issues FMOH supported use of FP advocacy kit Exposed MCH issues to the National Assembly FY 10 Sensitization workshop in Delta State Training of electronic media Exposure to the reality of RH/FP Strategies for presenting the poor state of RH/FP in the country 34 participants including 18 women leaders, 8 religious leaders, and 8 traditional leaders Set the agenda for bringing RH/FP discussion to the forefront of national discussion Engagement through Various Platforms. The IRHIN Project participated with other stakeholders in creating a reliable platform for continuous dialogue and engagement among donor and international agencies and the Government of Nigeria (GON). USAID, DFID, United Nations Population Fund (UNFPA), World Health Organization (WHO), and others found common ground in the IRHIN Project for engaging in debates and advocacy on RH/FP; these efforts contributed to a major policy change by the GON to fund public sector FP services, including the provision of FP commodities. Additional policy changes to which IRHIN contributed include obtaining agreement by the PCN to use its PPMV register to expand IRHIN retail outlets carrying FP commodities, as well as agreement for PPMVs to receive training for OCPs and to re-supply OCPs. Agreement with FMOH was also facilitated to support IRHIN distribution of long-term contraceptives such as implants. IRHIN was also actively involved in the national RH committee coordinated by FMOH, and participated in the review and harmonization of FP materials for training throughout Nigeria. The FMOH now has a budget line for FP, and key government officials including the Minister of Health and permanent secretaries have openly embraced FP. Table 8 details the multiple and diverse platforms IRHIN used to engender and increase interest in RH/FP in Nigeria, including FMOH working groups, religious forums, professional associations, CSOs, and community leaders and public gatherings. The GON‘s support and appreciation of IRHIN‘s social marketing was noted during the team‘s discussions with FMOH and at the local community level. It is also apparent through past collaborative efforts to ensure the availability of commodities and joint activities to promote RH/FP in Nigeria. IRHIN participated in community contraceptive security meetings where commodity forecasting was jointly conducted with GON and other stakeholders. In 2009 when IRHIN experienced shortages of injectables due to production problems, the FMOH loaned commodities that were paid back once the situation normalized. FMOH staff also acknowledged that IRHIN‘s social marketing has been the only source for sustaining availability of FP commodities during GON stock-outs, providing reliable access to contraceptives over the years. The PCN also confirmed its good working relationship with SFH dating back before IRHIN. IRHIN conducted at least two sensitization workshops for journalists and radio and TV broadcasters, participated in the Religious Leaders Consultative Forum, and was visible at National Council of Women Society (NCWS) meetings, the Nasrul-Lahi-L Faith Society (NASFAT) Women‘s Conference, and workshops for female religious leaders. The objective of engaging at various levels within the country was to sensitize and address RH/FP misconceptions and encourage participants to become FP/RH champions in their respective domains. 16 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Sensitization and advocacy efforts conducted under IRHIN by CSOs were greatly appreciated by the selected communities visited during the evaluation in both the northern and southern regions of the country. Leaders and heads of communities visited showed commitment to RH/FP in their community and openly discussed RH/FP. In most cases community leaders mentioned the benefits that spacing of children provides to the child, mother, and entire family. Thus, much of the IRHIN effort was focused at the national level and the communities where the project operated activities; efforts at the state level were limited. IRHIN‘s advocacy with PCN to enable PPMVs to receive training for the re-supply of OCPs paid off in the form of increased access to and sales of OCPs from retail outlets. In addition, continuous dialogue with PCN provided more acceptance and legitimacy for PPMV to operate in Nigeria. The IRHIN Project also worked with PPMVs and their associations throughout the country to promote continuous tripartite dialogue among FMOH /PCN, the PPMV associations, and IRHIN on common issues such as registration, training, and an improved operating environment for PPMVs in Nigeria. GON policies still prohibit CHEWS from providing FP injections outside facilities and from inserting IUCDs. The team‘s observation in the field confirms that the CHEWS are often the most accessible health provider in the country. EXPANDED DEMAND FOR IMPROVED RH SERVICES The IRHIN Project employed a combined strategy of mass media and interpersonal communication to create demand and increase knowledge of modern FP methods to address common misperceptions and encourage dialogue among couples on FP issues. At the national level, the IRHIN Project collaborated with the Federal Ministry of Health and other key stakeholders to sensitize and create awareness on and acceptance of FP among religious leaders, which resulted in establishment of the Interfaith-Based Forum as well as the first-ever national conference on FP in Nigeria. Mass Media. To reach the greatest number of people possible, a number of different mass media approaches were taken under IRHIN, including TV programs, radio jingles, method￾specific campaigns (IUCDs, pills, and injectables), and drama series, as well as billboards and posters. The use of the mass media, especially the radio, was an appropriate channel for reaching the target audience of men and women of reproductive age group with FP information. This conclusion is supported by the 2008 NDHS, which showed that radio is the most frequent source of FP messages for both women (40%) and men (59%) of reproductive age, and by the 2007 NARHS, which showed that 89.7% of respondents (86.5% women, 92.5% men) viewed radio as the most acceptable mass media method for communicating FP messages. The radio drama series aired under the IRHIN Project were One Thing at a Time in pidgin English, Garin Muna Fata in the Hausa language, Abule Olekemerin in Yoruba, and Odejinjin in Igbo. According to the 2010 NigerBus Survey, 31% of respondents listened to One Thing at a Time, 50% listened to Garin Muna Fata, 21% to Abule Olokemerin, and 12% to Odejinjin in regions where these messages were aired. In all, 40% of male respondents and 37% of female respondents listened to one of the radio drama programs for a total of 25,693,064 in 2010. In 2009, a total of 26,456,723 respondents to the 2009 NigerBus survey listened to the radio drama program, exceeding USAID‘s LOP target of 26 million listeners. No questions were included in the NigerBus surveys or other surveys undertaken to measure behavior change as a result of the mass media efforts under IRHIN. In an effort to measure changes in attitude and correct knowledge of FP and FP methods, the IRHIN Project worked with the NARHS to include appropriate indicators. The last NARHS was carried out in 2007. While the 2009 survey was intended to provide final measurement of these IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 17 changes in FP attitudes and knowledge, the survey has not taken place. This is now proposed for 2012. Table 7 compares earlier NARHS data on these knowledge and attitude indicators. While most indicators reflect improvement in attitude and knowledge from 2003 to 2007, only a few improved between 2005 and 2007. The indicator on intended use shows steady increase across all three years. It is likely that the earlier mass media activities may have contributed to this increase in intention to use FP and the decline in certain misconceptions about FP. The 2010 NARHS is expected to provide more definitive confirmation of improvement in FP attitudes and knowledge. Table 7. Summary of NARHS Indicators on Attitudes and Knowledge of FP Indicators 2003 2005 2007 F M F M F M % of women aged 15-49 and men aged 15-64 who believe family planning/child spacing methods are effective 45.9 54.8 55.1 62.4 47.2 51.5 % of non users (women aged 15-49 and men aged 15-64) who report intending to use a modern FP method within a year 9.1 12.6 16.8 % of women aged 15-49 and men aged 15-64 who report that they believe that modern FP method cause infertility 29.9 29.3 32.6 35.9 21.8 22.5 % of respondents who have discussed FP at least once in the last 12 months with spouse or cohabiting partner 25.9 30.7 30.0 34.5 26.5 34.6 % of female 15-49 years and males 15-64 years who perceive that oral pills, injectables, and IUCDs are affordable: Pill Injectables IUCD 30.1 26.0 13.7 22.6 18.5 7.2 30.3 26.9 13.7 28.0 23.5 9.9 24.3 20.3 8.1 19.5 18.3 6.0 % of female 15-49 years and males 15-64 years who perceive that oral pills, injectables, and IUCDs are easy to obtain: Pill Injectables IUCD 32.9 29.7 15.6 25.9 29.7 8.9 32.9 30.8 15.8 30.4 26.2 11.6 26.5 22.4 10.2 20.9 20.6 6.4 Source—NARHS 2003, 2005 and 2007 Interpersonal Communication The IPC strategy involved the use of community members as PEs and IPCCs to create awareness in the community about FP in the context of child spacing, increase knowledge of FP methods, and address myths and misconceptions about family planning. The IRHIN Project partnered with selected CSOs already working with the communities in 22 focus states. CSOs were identified and selected based on criteria drawn up by IRHIN staff, which included previous experience working in RH and interest in working in FP demand creation. The CSOs facilitated entry into the communities and also conducted trainings for the PEs and IPCCs. The PEs worked with small groups of 20 to 30 community peers on a weekly or bi-weekly basis; the IPCCs reached out to larger groups at markets, mosques, rallies, sports events, etc. At the end of the one-year intervention, a total of 49,267 and 393,138 people had been reached by PEs and IPCCs, respectively. 18 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Based on team visits to communities and discussions with community heads and leaders, PEs and IPCCs, and CBO staff, it was clear that the community leaders had come to understand the importance of FP in improving maternal and child health and that family planning was now being openly discussed. Other community members confirmed that the PEs and IPCCs had benefited the community by improving knowledge and understanding of FP and had corrected myths and misconceptions related to family planning that community members had previously held. PEs cited instances in which the meetings had improved relationships between spouses by encouraging spouses to attend the regular meetings. While the IPCCs were able to reach more community members in larger groups, the PEs could develop on-going relationships with smaller groups. It was felt that the two approaches to IPC complemented each other. Table 8. LOP Achievements for Interpersonal Communication Activities Indicator Male Female Total Number of PE trained 103 147 250 Number of IPCC trained 81 91 172 Number of persons reached by PEs 16,102 33,165 49,267 Number of persons reached by IPCCs 152,422 240,696 393,138 Number of referrals 3,298 13,454 16,752 Number who visited facility through referral 873 6991 7,864 Number who accessed services (FP methods as well as counseling) 635 6387 7,022 Source—SFH MIS. Within the community mobilization activities, there was an established referral system between the PEs, IPCCs, and public or private health facilities. Community members interested in accessing FP services or further FP counseling were provided with a referral slip to a private or public referral facility close to the community. Simple data collection was maintained by the CSOs and included the number of persons referred, the number who actually visited the clinic, and those who accessed services. Of the total population reached by the PEs and IPCCs, 4.9% received a referral slip and 2% accessed services. However, these indicators were not well defined, particularly in the case of accessed services, making it difficult to assess the extent of FP use as a result of this community mobilization effort. Most facilities did not track FP use by method. Although the program had ended prior to the team‘s visit, all communities visited reported that community member still solicit information on FP from the PEs and IPCCs. PEs and IPCCs stated that they continued to provide referral slips, though they no longer held group meetings or conducted group outreach sessions. All were unanimous that the one-year period of implementation was too short to show significant impact at the community level. IEC Materials. Several PEs and IPCCs noted the shortage of IEC materials, especially toward the end of the project. In one community the IEC materials used by the PEs and IPCCs had to be translated to the local language to ensure easy communication and better understanding of the information by clients. Several PEs suggested that IEC materials should be modified to reflect local dress and languages. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 19 The MAP Surveys annually measured the nationwide percentage of retail outlets (PPMVs and pharmacists) with promotional IEC material regarding specific FP methods that are stocked. While never high, the presence of point-of-sale (POS) IEC materials had declined by 2010 to almost nothing. The 2006 MAP found that 17% of shops had POS materials on Duofem and 26% on Gold Circle. By 2010, only 2% of shops surveyed had POS materials for Duofem and 4% for Gold Circle. While it is likely that in certain areas there was sensitivity to displaying FP product information openly, it is also likely that insufficient POS materials were provided through the end of the project. EFFECTIVENESS OF PROJECT MANAGEMENT The three IRHIN implementing partners, SFH, PSI, and PI, brought considerable strength and experience to the project. SFH brought a team experienced in social marketing, behavior change communication (BCC), and health research, along with a firm understanding of the diverse social contexts of Nigeria. PSI brought international expertise and experience in social marketing from over 70 countries and numerous research tools for program monitoring and decision making. PI brought a wealth of experience in the provision of quality health services and in community mobilization, both locally and internationally. While the IRHIN Project made substantial progress in improving FP access, quality, and demand and strengthening the enabling environment, this tremendous wealth of knowledge and experience was not fully integrated under the IRHIN Project to achieve maximum benefit and synergy. IRHIN Management Structure. Project components tended to be segmented by implementing partner rather than fully benefiting from an integrated IRHIN project staff contributing complementary skills and expertise. While it is typical for all project implementing partners to be present and represented in-country, PSI was not represented on the ground. The role of PSI in the project was never satisfactorily explained to the evaluation team beyond ―the provision of occasional technical assistance in financial and contractual management and M&E.‖ Annual Steering Committee meetings with all three partners present appear not to have taken place, thus losing an opportunity and platform for regular discussion with both USAID and the GON of progress and outstanding issues. The lack of shared dedicated office space and the maintenance of separate databases appear to have resulted in a certain fragmentation in the implementation of project components. The team observed that the use of separate administration facilities appeared to contribute to parallel rather than integrated implementation of project components, reducing opportunities for achieving synergy among IRHIN‘ social marketing, community mobilization, and service delivery components. Community mobilization efforts were not co-located to leverage and complement the improved private clinics with newly trained RH providers. In addition, not all the private clinics or CSOs where PEs and IPCs were trying to improve uptake of FP knew where their nearest PPMVs were located that stocked FP products. Management Information System for Commodity Tracking. Technical collaboration between the IRHIN and the DELIVER projects at the project‘s start helped establish a well￾functioning management information system (MIS) for forecasting and tracking distribution, as well as continued, collegial relations between the two projects. Over the life of the project, SFH maintained a reliable and efficient MIS system through its warehouse that adequately tracked commodities to and from all 20 MDS depots used nationally by IRHIN, thereby reducing stock￾outs to a minimum and ensuring more even distribution of FP commodities to and from MDS depots. The evaluation team observed that the MIS system had not developed to the extent of tracking distribution of FP products from the MDS depots to wholesalers and beyond to the PPMVs. Information tracking farther along the retail chain could help reduce artificial or temporary stock-out of commodities and give a more accurate indication of products actually sold. 20 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Project Monitoring and Evaluation. The project design and numerous indicators agreed to by IRHIN and USAID in the 2007 PMP placed considerable emphasis on multiple national surveys to inform program review decision making for improved performance over the LOP. The NigerBus survey was the primary monitoring source of feedback that informed demand creation strategies and implementation; NDHS was used primarily to monitor progress against goals and objectives; the three waves of NARHS (2005, 2007, and 2010, still in process) were used for monitoring key project outcomes in attitudes and knowledge concerning FP; and annual MAP surveys were used to monitor nationwide coverage, quality, equity of access, and efficiency of social marketing systems and service delivery. Additional qualitative monitoring tools used including mystery client surveys and exit client surveys to ascertain service delivery quality. Additional qualitative methods were employed, e.g., key informant interviews and focus group discussions to develop product concepts and pre-test materials and to gain insight into specific behavioral issues, social norms, and values that influence RH/FP use. These qualitative and quantitative data collection mechanisms supplemented routine data collection through the social marketing MIS and data regularly collected at the service delivery and community mobilization activity sites to provide useful program performance information. Much of this data, however, particularly from the regular surveys, were inadequately used for decision making and identification and timely solution of problems. Most surveys were population-based, and may have been unduly influenced by extraneous factors and sampling errors. The project did not have dedicated M&E staff at the field offices to collate and summarize service statistics from the community level to headquarters. It appears that from the design stage of the project, more emphasis was placed on periodic population-based surveys as a source of information for program performance monitoring and evaluation, than on establishing routine data collection structures that would report reliable data on individual field activities back to SFH or PI headquarters. This lack of more comprehensive routine data collection on service quality, provider skills, and FP use by method from field activity sites made it problematic for IRHIN to gauge the relative cost-effectiveness of various activities and approaches. PSI presence on the ground in Nigeria might well have greatly benefited the project in the area of integrating data collection and its use along with survey and qualitative data to better monitor, redirect, and constantly improve project implementation for improved outcomes. USAID Management of the IRHIN Project. Frequent changes in USAID‘s Project Officer or AOTR for the IRHIN project may have contributed to less rigorous management and oversight of the project and its performance. Historical information and written records from project review meetings on discussions and justifications for trends in product distribution and lagging performance was not available to the team, as the former AOTR for IRHIN prior to the project closeout was unavailable at the time of the team‘s visit. APPROPRIATENESS OF PROGRAM APPROACH Social Marketing and BBC. Social marketing along with BCC continues to be an appropriate strategy for increasing access to modern FP methods, increasing awareness of the health benefits of child spacing, improving knowledge of various FP methods, and tracking changing attitudes and behavior that could support increased use of FP in Nigeria. There is already a substantial infrastructure in place for social marketing of FP commodities. Over IRHIN‘s LOP, billions of pieces of FP commodities were distributed, calculated at 15.4 million CYPs. The importance of social marketing is nowhere better documented than by the fact that in 2008, almost half (48.3%) of all current users of modern FP methods obtained their contraceptives from PPMVs and pharmacists (NDHS). Both the NDHS and NARHS confirm that radio and TV are effective means for reaching the Nigerian population with FP messages; of the two, radio has the greater reach, with 40% of women and 59% of IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 21 men exposed to FP messages on the radio, and 25% of women and 33% of men exposed through television. The fact that more men than women heard FP messages on radio and television make these excellent methods for an increased focus of FP information and messages for men. It is also important to note that where the public sector does not have a particularly strong track record in providing a constant supply of FP commodities and services, as in Nigeria, social marketing of FP products has served and can continue to serve over the coming decade as a steady source of complementary contraceptives at affordable prices. Socially marketed FP commodities, largely through SFH‘s resourcefulness over the past several decades, have created a considerable niche for the organization in the FP market, best expressed through a stakeholder interviewed during the evaluation: ―Social marketing still has an important role because it makes commodities available at reasonable cost, and is relevant in the Nigerian situation.‖ Community-based BCC. The IRHIN Project used over 30 CSOs in 22 focus states to take RH/FP advocacy and information to the grass roots level that may not be adequately reached by radio, TV, and retail outlets. The positive reception and responses from the communities visited by the evaluation team suggest that communities can be receptive to attitudes and behavior change once the right change agents in the community are identified and engaged. Clearly, the community-based approach using CBDs, referral facilities, CSOs, PEs, and IPCCs created awareness and attitudinal change about FP. Benefits listed by community leaders included peace among families and couples, reduced frequent pregnancies among women, and healthier mothers and children. IPCs and PEs enumerated the same benefits for the community as well as personal benefits to themselves in the form of their own increased knowledge and confidence about FP, self-fulfillment from having served as a source of help to singles and couples, and unexpected popularity among peers and community members. Likewise, the use of CBDs under the Service Delivery Component increased awareness and access to modern contraceptive methods, as evidenced by referrals observed during team visits to two private clinics in Kaduna. While one year was too short a time to thoroughly field test this approach, there is sufficient qualitative evidence to support increased use of community mobilization efforts over a longer period of time. Improved Service Delivery through Private Clinics. The third major component under the IRHIN Project looked at improving service quality and access to a reliable source of contraceptives through the refurbishment and equipping of 28 private clinics, training of their RH providers, and assurance of a constant supply of FP commodities through linkages with social marketing detailers. This was an interesting linkage of clinics to the social marketing network; in as much that keeping a steady supply of FP commodities available to the community through these clinics was the objective, that goal was well met. While the quality objectives were not sufficiently documented, partially due to a lack of appropriate and regularly collected data, the steady availability of contraceptives was documented. Missing linkages. While socially marketed commodities and BCC were common threads running through all of these approaches, the integration and linkage that could have sparked a synergy among these approaches never really happened. Project components were spread out and few activities operated in close proximity, which would have increased the chances for linkage. The overall number of focus states was large, at 22, and total project resources were not sufficient to cover more than a few CSOs per state. A more focused grouping of activities might also have increased opportunities for linkages. Notwithstanding the lack of adequate linkages among project components, evidence suggests that there were several good examples of linkages between IRHIN and other donor and USAID 22 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION projects. FP commodities were leveraged with other donors and projects to increase FP commodity number and choice. SFH detailers were also leveraged to provide greater monitoring and reach to retail outlets. Linkage and collaboration also occurred with the FMOH and other donors in the area of strengthening enabling environments. Additional opportunities for linkage were lost in not using the IRHIN Steering Committee to strengthen the project and advance FP advocacy efforts. At the programming level within IRHIN, evidence suggests that the three approaches – social marketing, community mobilization, and service delivery – were not adequately linked to maximize synergy and outcomes. Sustainability. With a CPR of only 10.5% for modern methods and 15.4% for all methods, there is still a tremendous unmet need for readily accessible, affordable FP commodities and BCC in Nigeria. Add to that equation a public sector that has been all but moribund over the past decade and is only now coming back to life with a committed budget line item for FP services and commodities. Given these factors, the need for social marketing to continue for decades to come does not sound exaggerated. Sustainability does not appear to have been an issue in the IRHIN Project and was neither mentioned in the cooperative agreement nor in the PMP indicators. Even though sustainability was not mentioned, the IRHIN project wisely took advantage of the existing commercial distribution network rather than attempting to set up an expensive, parallel distribution network for the project. The project has made FP products visible and the public is getting used to seeing FP products in retail outlets. Four ―premium brands‖ were introduced to attract those clients who can afford to pay higher prices closer to market rates; three of these were products brought in and marketed on a cost-recovery basis. Little has been done in the project to track just how quickly these brands could become sustainable. Even so, the project has certainly laid the groundwork for initial sustainability efforts that could benefit future social marketing projects. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 23 III. CONCLUSIONS Increased Access to Improved RH Services. The IRHIN Project broke important ground in expanding access points in a social marketing project beyond retail outlets to private CSO clinics, private and public referral facilities, CBDs, and RH providers. The project achieved 15.2M CYPs by the EOP, comfortably exceeding its LOP target of 13.4M CYPs. While the 15.2 million CYPs generated by FP commodity sales under the project clearly will have contributed to the national CPR, it is unlikely that it has nudged Nigeria‘s CPR to 16% for modern methods, the strategic goal for the IRHIN Project, given that the CPR for modern methods was calculated at just 11% at the time of the last NARHS in 2005 and was 10% in the 2008 NDHS. That said, the 12 FP commodities offered under the project represent an impressive range and choice for those desiring to space or limit their family size. Not all project accomplishments are measured by CYPs or PMP indicators. There is anecdotal evidence that during the IRHIN LOP, the public sector family planning program in Nigeria was stocked out of contraceptives for long periods, especially during the project‘s last few years. At times, the only contraceptives available in-country were through the IRHIN project. The fact is that the IRHIN Project picked up much of the public sector slack and provided access to FP commodities to women and men who otherwise would not have been able to access family planning methods through the private sector. Since PPMV retail outlets cannot stock clinical FP methods (injectables, IUCDs, and implants), IRHIN‘s detailing of these methods to clinics and providers from the private, CSO, and public sectors served to effectively increase access to clinical and longer-term methods through social marketing. Unfortunately, the project failed to collect sufficient data on individual activities to allow for determination of the effectiveness of the various approaches in increasing uptake of FP commodities. Issues of quality assurance are raised by detailing to RH providers on a one-off basis where no assessment of skills or follow-up training takes place. SFH‘s distribution network, linked as it is to the commercial sector, should be able to support a much more robust expansion of nationwide social marketing coverage of a more varied product line in support of MCH as well as FP. Improved Quality of FP Services. While it is extremely likely that capacity-building training for providers (nurses, doctors, and CHEWs) improved the quality of FP services under the IRHIN Project, the data available was insufficient to support the overall conclusion that the quality of FP service delivery improved. Number of RH providers trained is a poor surrogate marker for quality of services, especially without regular follow up of trainees for competency assessment. While mystery client surveys and client satisfaction surveys are useful supplemental data, especially if repeated at regular intervals, they should be supplemented with regular data collection to measure service quality. Qualitative and anecdotal evidence confirms improved client satisfaction occurred as a result of IRHIN interventions as well as improved confidence and knowledge of RH providers to provide FP services. Strengthened Enabling Environments. A more conducive environment for family planning is emerging in Nigeria today. The IRHIN Project has contributed to this, primarily through efforts at the national and local community levels. This more conducive environment needs to be sustained and further strengthened by stakeholders at all levels for increased acceptance and use of FP over the coming decade. 24 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Expanded Demand for Improved RH Services. IRHIN‘s combined demand creation strategy of mass media and community IPC efforts almost certainly improved overall FP awareness, knowledge, and intention to use, although this will not be confirmed until the next round of NARHS results are available. The use of CSOs as an entry into the community with PEs and IPCCs was an appropriate, innovative strategy that complemented the use of mass media for demand creation. Effectiveness of Project Management. Although SFH, PSI, and PI each brought tremendous strength and experience to the IRHIN Project, the lack of an integrated IRHIN Project management structure led to each partner developing parallel management systems for staff, data collection, and project activity implementation, an approach that failed to promote optimal collaboration and synergy among the project‘s various components. Insufficient efforts were made to integrate project activities at the field level for improved outcomes. Frequent changes in USAID‘s AOTR for the IRHIN project did not facilitate the rigorous, consistent USAID oversight needed for a complex project such as IRHIN. Appropriateness of Program Approach. The three program strategies employed in IRHIN– social marketing, mass media and community BCC using CSOs and CBDs, and improved service delivery – are all sound program approaches. These strategies contributed at least partially to the expected increases in FP access, quality, and demand as well as to the development of a friendlier environment for FP and its increased acceptance in the communities involved. However, the three strategies were never adequately linked to achieve a synergy greater than the sum of their parts. What did appear to work well was the use of socially marketed commodities at private clinics, linked by the social marketing detailers. Individual components that were successful and lessons learned can inform future USAID project activities in social marketing, demand creation, and service delivery. CHALLENGES AND CONSTRAINTS The enabling environment for F in Nigeria is still nascent and weak, but is improving. The public sector has recently taken steps to include and fund FP and contraceptive commodities as an important component of improving maternal, neonatal, and child health. At the same time, there is only a minimally functioning public sector infrastructure to provide access to FP services and commodities to those who need free or extremely low-priced services nationwide. This situation places an undue burden on contraceptive social marketing, which is generally intended to serve the moderately poor, not the poorest of the poor. During the IRHIN LOP, the public sector experienced periodic stock-outs. While no records exist to reflect how this affected the IRHIN Project – and stakeholders were reluctant to speculate on the topic – it is likely that increased sales and detailing to the public sector may have distorted annual commodity sales and contributed to dips in nationwide distribution, coverage, and availability of products at retail outlets. FP still meets significant resistance and opposition from community and religious leaders, as well as from husbands and even from the women themselves. Of MWRA currently not using FP surveyed in the 2008 NDHS, 20.8% said they were opposed to FP use, 10% cited their husbands‘ opposition, and another 8% cited religious prohibition as the reason they do not intend to use FP in the future. While increasing, demand for FP is still low. Large families are still the norm nationwide: According to the 2008 NDHS, there is a desired family size of 6.1 children. Of women intending not to use contraception interviewed in the 2008 NDHS, 16.5% stated that they wanted as many children as possible. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 25 LESSONS LEARNED The following two key messages are repeated consistently by donors, FMOH, PPMVs, community groups, and service providers: Approaching any discussion of FP is more acceptable—and therefore more productive—if approached in terms of ―child spacing‖ to benefit the health of children and mothers or when FP is linked with other health issues, such as hygiene, malaria, water and sanitation, etc. Focusing on male involvement and targeting husbands with information and counseling is critical for improving uptake in FP use. At the community level, working through local CSOs brought additional benefits through linkage with other on-going CSO activities in microfinancing and income generation, adult literacy, and other health areas such as hygiene, HIV/AIDS, and MCH. Initiation of community mobilization efforts using community heads and leaders as well as religious leaders as entry points greatly facilitated start-up and implementation. Working with CSOs as entry points into the community and engaging community and religious leaders for community-level interventions facilitated smooth acceptance of the program by the community. The use of community members as FP promoters appears to have contributed to building trust in addition to providing opportunities for building groups of advocates. Private Sector Clinics. The use of socially marketed commodities at the 28 renovated and equipped private clinics worked well. The fact that social marketing detailers were responsible for continuous re-supply ensured almost no stock-outs. This has great potential for increasing availability of longer-term methods through private clinics with trained RH providers. Clients said they were more interested in attending private clinics because they knew they would find FP commodities in stock. Working with the private sector provides better opportunities for increased operational efficiency. The time needed to achieve effective change is greatly shortened when there is decision- making power at the operational point and there are no bureaucratic bottle-necks to overcome. However, it takes effort and patience to work with small private clinics that are primarily focused on maintaining financial viability and may give scant attention to the need for programmatic record keeping, data collection, and training/retraining needs. While at first private clinics showed little interest in providing FP counseling, once they understand counseling‘s importance and how to provide it, the clinics increased their provision of FP counseling. Clinics report that their increased use of counseling has improved clients‘ interest in FP and willingness to accept a FP method. It has also increased their popularity in communities and providers believe that patronage of other services provided at their facilities has increased as a result. 26 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 27 IV. RECOMMENDATIONS FOR FUTURE DIRECTIONS AND OPPORTUNITIES FOR FP SOCIAL MARKETING INCREASED ACCESS TO IMPROVED FP SERVICES New and innovative approaches to increasing access should be better monitored, measured, and documented so that decisions can be based on relative effectiveness of each effort in increasing both FP access and uptake. Follow up to maintain FP quality of care should also be emphasized in the approaches. Future social marketing projects should continue to focus on new approaches to expand access points in semi-urban and rural areas for better nationwide coverage, schemes to get the products and services closer to the clients‘ homes, and innovative approaches for effectively detailing the clinical methods to trained providers. Efforts should be made under future social marketing projects to measure sales from wholesalers or retail shops closer to the consumer to achieve more accurate CYP and commodity sales reporting. Several new technologies would facilitate this, including simple cell phones. Several different technologies could be pilot tested. Further experience and analysis of cross-subsidization taking place at private clinics should be assessed and documented to inform future efforts to complement public sector approaches to meeting the needs of the poor. IMPROVED QUALITY OF FP SERVICES Exit interviews and mystery client surveys should be conducted more regularly, augmented by regular data collection against enhanced service quality indicators to guide programmatic decision making. QA plans and monitoring need to be built into all future projects for both FP service delivery and social marketing of FP commodities. Quality assessment tools with a scoring system based on set criteria that define minimum standards for both facilities and RH providers should be developed to measure quality improvement on a regular basis for future projects. Linkages between referral facilities and PPMVs should be developed in new social marketing projects to better encourage PPMVs to refer new pill clients, confident that they will return to the PPMVs for re-supply. There is a need to establish minimum criteria and follow-up and linkage-to-training protocols for detailers selling clinical methods to RH providers. STRENGTHENED ENABLING ENVIRONMENTS FP received national-level support in 2010 through the designation of a budget line for FP services and commodities. To maintain momentum and follow up on this positive development, efforts should not only continue at the national level but also should also focus in on the state level by identifying state-specific advocacy platforms and engaging change agents in each state. To increase FP access and use nationwide, future RH projects should support efforts to achieve policy changes that allow community health extension workers (CHEWs) to provide injectables outside the facility setting and to insert IUCDs with appropriate training and 28 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION supervision. The latter policy change is particularly urgent for increasing access and use of longer-term FP methods. Work with community and religious leaders should be continued in new service delivery and social market projects, with increased efforts to involve men in FP awareness efforts and supportive decision making. EXPANDED DEMAND FOR IMPROVED RH SERVICES Community interventions should be community-specific in order to better meet the needs of each community. In addition, appropriate IEC materials should be developed taking into consideration the local language, dress and culture of the community. These materials should be available in adequate amounts through the project Given that mass media communication is expensive, data should be collected on behavior change occurring as a result of mass media, in addition to the information already collected on numbers listening to programs and attitudinal change Community mobilization activities should collect data to track not only referrals, but also the number of people accepting FP by method and number of people continuing use of the method after 12 months. Establishing a tracking system for referrals from communities to the health facility could be useful in coming up with accurate data on referral outcomes, which could form a basis for comparison of the two approaches (PE and IPCC) and using such findings in future program decisions. Adequate budgets should be made available for IEC materials for community outreach, facilities, and POS materials for retail shops, so such materials can be more fully used when found to be cost-effective. CSOs, RH providers, and shopkeepers should be encouraged to use the materials regularly. EFFECTIVENESS OF PROJECT MANAGEMENT Future projects should ensure that a clearly defined, integrated project management structure exists that fully incorporates the strengths of all project partners. Such a structure should provide for integrated implementation of all key project components. To improve program management, implementation, and reporting, there should be one integrated data management system instead of separate and multiple data bases being maintained by each project partner. Each project field office should have a designated and trained M&E staff person to facilitate and serve as point person for the efficient flow of all data between the central and field levels, as well as for analysis and use of the data at all levels. Project staff at all levels should understand and be able to interpret the data that is collected, and should understand how the data relates to their daily work. All staff responsible for program decision making at the central and field level should be trained in the use of all data collection tools, including surveys; they should be able to interpret and use the data in a timely manner to assess problems and identify solutions to improve project implementation and outcomes. In effect, field staff should not rely on headquarters to analyze and interpret data affecting their field operations. Increased analysis – particularly of national survey findings, but also of other monitoring tools – should be regularly carried out by both project and USAID staff to inform project progress and direction, with findings and implications for the project discussed with USAID on a regular basis. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 29 USAID and the staff of future projects should work together to develop indicators that are more meaningful and results-oriented. If the MAP methodology requires revision for a new project, that is a topic that should be considered and discussed. Distribution surveys of outlets could also be piloted. USAID project management should ensure that regular meetings are held with project staff to discuss a variety of issues, including performance, fluctuations in products distributed, results of project-generated surveys and assessments, and the use of data to improve performance. These discussions should be documented and passed on to ensure continuity in project oversight Exposure to successful social marketing programs in other countries as well as refresher training in measurement, indicator selection and monitoring of social marketing and FP service delivery projects would be beneficial to both USAID and project staff. APPROPRIATENESS OF PROGRAM APPROACH Where multiple program approaches are implemented simultaneously under one project, substantial linkages at the management and implementation levels are needed to ensure optimal outcomes. Sufficient appropriate data should be collected to allow for the comparison of the effectiveness of different approaches. Long term sustainability should be a consideration in future RH projects, with the understanding that long term may mean decades, not years. No matter how far off sustainability is, every project should view it as an eventual goal and identify strategic measures to be taken over the short as well as long term to achieve it, with at least one indicator included in the PMP that keeps sustainability in sight. In the case of a social marketing project, it could be as simple as an increase in market share for cost-recovery products. That said, cost recovery should not become an end in itself, so that the emphasis on profit overtakes social marketing objectives. 30 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 31 ANNEX A. SCOPE OF WORK Global Health Technical Assistance Project GH Tech Contract No. SCOPE OF WORK (Revised: 07/27/11) I. TITLE Activity: Nigeria: Improved Reproductive Health in Nigeria (IRHIN) Evaluation Contract: Global Health Technical Assistance Project (GH Tech), Task Order No. 01 II. PERFORMANCE PERIOD It is anticipated that the period of performance of this Evaluation will be o/a July 25 to September 15, 2011 depending upon consultant availability, with in-country work taking place from o/a August 1 -o/a August 24. Total approximate time will be four weeks. All in-country work for the Evaluation must be completed by August 25 including presentation of the Evaluation findings and submission of the draft report. III. FUNDING SOURCE Mission will use field support funds through GH Tech. IV. OBJECTIVES AND PURPOSE OF THE ASSIGNMENT USAID/Nigeria plans to conduct an evaluation of the IRHIN Project in Nigeria for the period June 2005 through June 2011. The purpose of the evaluation would be to evaluate achievement towards four expected results: Increased access to improved child survival and RH services Improved quality of reproductive health services, Strengthened enabling environment, and Expanded demand for improved child survival and RH services The last external evaluation of the program was conducted in 2009 in which maternal, child health and reproductive health program across three projects (Access to Clinical and Community Maternal, Neonatal and Women‘s Health (ACCESS/Maternal and Child Health Integrated Program (MCHIP, ACQUIRE/Fistula Care and IRHIN were evaluated. Main issues included- stock outs, high staff turnover, sustainability of community interventions and price hikes among retailers, furthermore rural penetration had not been maximized. In light of the fact that social marketing contributes significantly to the health and well being of Nigerian citizens, particularly those in the lower wealth quintiles, USAID/Nigeria would like to conduct an evaluation of IRHIN Project in Nigeria. The evaluation will focus on the following: 32 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 1. To evaluate IRHIN‘s performance achievements under the current agreement. The evaluation will look at the performance of the project in implementing the social marketing program since 2005, with a special emphasis on contributions to contraceptive prevalence and effective approaches in expanding demand for RH services and quality of services. Identify whether the project did not meet or exceeded expected results and any innovations adopted by the project. V. BACKGROUND IRHIN, the first USAID project directly awarded to a Nigeria organization is a social marketing project that was to contribute to increasing the contraceptive prevalence rate for modern methods in Nigeria from 8.9% to 14% over the six years of the project and achieve 13 million CYP under the project. The key expected results from the project were to increase demand for contraceptive commodities nationwide and increase access to quality FP commodities by working with community-based proprietary medicine vendors and private sector clinics. IRHIN also worked to improve the quality of FP/RH services in 21 clinics in three states and the enabling environment for FP by working with religious leaders, the Pharmacists Council of Nigeria and other national associations. The status of reproductive health in Nigeria is still poor as reflected in the high maternal mortality at 545/per 100,000live births and high under five mortality rate (157/1000), low contraceptive prevalence rate (10% use of modern contraceptive methods) and the total fertility rate remaining high at 5.7 births per woman, which is the same reported in 2003. Fertility peaks at age 25-29 with 265 births per 1000 women. It is also important to note that rural areas have a higher TFR than urban area and there are large urban rural differences as well as between north and south. There is still a wide gap in meeting the family planning needs of women with overall 20% of currently married women having an unmet need for family planning (15 % for spacing and 5% for limiting). ‗If all married women with unmet family planning were to use a contraceptive method, the CPR for any method would increase from 15-35%‘1. There are wide regional and state variations in fertility rates and desires to space or limit births are still grossly unmet. Currently only 23% of modern contraceptive users obtain their contraceptives from the public sector2, while 61% obtain family planning services from the private sector. VI. SCOPE OF WORK The IRHIN evaluation will examine several key issues through qualitative methods and quantitative analysis of national surveys (NDHS, NARHS) and SFH data sources (MAPS, other surveys). Focus will include the following: A. Increased access— Illustrative subject areas/questions for evaluation include: – Describe the extent of the distribution network that IRHIN has established in Nigeria, and assess the effectiveness of the distribution in reaching rural, urban and low income(explore various wealth quintiles) populations, un married young people, men and women in union. Is IRHIN exercising its full potential? Is IRHIN appropriately leveraging its capacity? 1 NDHS 2008 2 NDHS 2008 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 33 – Assess the pricing, positioning, distribution, scope and promotion of socially marketed products. – Describe the effectiveness of community level family planning demand creation program of IRHIN and the possibility of its replication. – IRHIN contracted with NGOs in most states to implement small scale FP promotion programs involving ―peer educators‖ and ―IPC agents‖. What were the costs, coverage and evidence for impact of this approach? – Explore the effectiveness of the marketing and BCC techniques used to create consumer demand. The service delivery component of the project that was implemented through Pathfinder International in 3 focus states (Abia, Kaduna and Cross River) what was this intended to achieve? What lessons were learnt? – What are the significant changes in interest of private sector health care to provide family planning? What factors are responsible for the changes? – What factors did the project encounter in working with for -profit, faith-based health care providers and how did these affect project activities and results? – How has the quality of service delivery changed? How much has this component contributed to the overall results produced under the IRHIN Project. – How has the warehousing and distribution approach worked? – What Policy changes were supported in ensuring increased access to family planning commodities? How effective has this been? B. Targeting, Reaching and Influencing Users – Assess the effectiveness of IRHIN‘s activities in BCC and Mass media on knowledge and attitudes. How were target populations selected for behavior change? What studies were conducted for messaging? What behaviors were targeted ? Was the targeting strategy effective? – What IEC approaches were used? Were they effective? – What method specific and/or product specific approaches were used in targeting and did they work? – Can we estimate the overall exposure of the population to these messages by source and or any associated behavior change? How does this compare with levels of exposure from media programs in other countries and what is the expected impact? Can we measure this impact in Nigeria and how does this compare to the project goals? What are the costs of these efforts? What scale and approach would make sense for Nigeria? – What data exists on the economic level of users/consumers and their ability to pay for condoms and other contraceptives, and other social marketing products? On their ability to pay for commercial brand products? Has there been a substitution effect where users/consumers able to pay higher prices have switched to social marketing products? Is USAID assistance in social marketing therefore appropriately defining and targeting potential users/consumers for behavior changes and sales? What strategy was used for targeting sales? – What has been the behavior change in retailers and PPMVs and their interactions with clients following training? How has quality of services improved? – What information, advertising and behavior change strategies and activities have been adopted by IRHIN and have results been measured appropriately over time so that trends and gaps could be identified? – Is there leakage of social marketing products from social marketing warehouses and sales outlets to public sector facilities? Are any family planning products sold by 34 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION commercial outlets at prices that are less than or exceed those established for the respective products by other social marketing projects? What consequences have ensued if this has happened? – The joint product package of FP commodities with other donors (DFID), how has this worked in terms of availability, efficiency and management? – How did the project change Knowledge, Attitudes, Practice and Perception of clients, vendors and staff? – How well understood is the potential of social marketing in Nigeria and what more can be done to enhance understanding of social marketing contributions to MCH/FP objectives? – What was the relationship with the project sub partners and how did this work out for overall project achievements? C. Sustainability Social marketing and pharmacies/PPMVs: IRHIN provided support for pharmacies and PPMVs to provide FP commodities, including supplies through private sector wholesalers, training and detailing. How did this approach work given the numbers, nature and potentials of pharmacies and PPMVs in Nigeria in comparison to other possible approaches? VII. METHODOLOGY The Evaluation Team will use a mixture of quantitative and qualitative approaches to gain insight on the impact of IRHIN activities (mostly from quantitative) and the processes (mostly qualitative) that lead to those impacts. Sequential and iterative approaches will be used to integrate the mixture of methods and will seek varying degrees of dialogue between quantitative and qualitative traditions at all phases of the evaluation. Background Materials Review Prior to conducting field work, the Team will review background materials such as Annual and Quarterly Reports, Indicators, IRHIN Program Description, past program evaluations and other public documents related to the project. The mission will provide these to the team as soon as possible prior to in-country work. Team Planning Meeting The team will conduct a 2-day team planning meeting (TPM) upon arrival in Nigeria and before starting the in-country portion of the Evaluation. The TPM will review and clarify any questions on the evaluation SOW, draft an initial work plan, develop a data collection plan, finalize the evaluation questions, develop the evaluation report table of contents, clarify team members‘ roles, and assign drafting responsibilities for the Evaluation report. The TPM outcomes will be shared with USAID/Nigeria and the health team will participate in sections of the TPM. Key Evaluation Steps Review program documents, including the technical proposal, annual work plans and annual reports, technical and training materials, and the evaluation reports (list and documents to be provided by the Mission). Engage in a two-day Team Planning Meeting (TPM) to discuss the Evaluation scope of work; agree on team member roles and responsibilities; clarify the Evaluation expectations of USAID; draft an Evaluation work plan; decide on methodology; develop tools/interview guides that will be used by the team for key informant interviews and FGDs; and draft a report outline. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 35 Conduct field visits to service outlets and to see the activities of non-formal health providers. Conduct interviews with key informants from USAID implementing partners, USAID, MOH counterparts, UN agencies, donor organizations, and other private sector providers/entrepreneurs. Conduct FGDs with IRHIN Sales Officers and non-formal health providers. Prepare a presentation and debrief for USAID/Nigeria with main findings and recommendations. Prepare a draft report for the Mission before departure from country. Prepare a final report with an executive summary that includes main findings, conclusions, and recommendations for program improvements. VIII. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT An Illustrative Table of Level of Effort (LOE) LOE Schedule Activity Person Days Per Person Team Leader Team Member Team Member Logistics Preparations and review documents (to be provided by USAID), to occur out of country and prior to beginning the Evaluation. 3 days 3 days 3 days Logistics preparation and assistance through assignment for team‘s in-country visit and field visits/interviews 1 day 20 days Travel to/from Nigeria Up to 4 days 1 day 1 day Team planning meeting (TPM); develop an Evaluation work plan and timeline; develop interview/ FGD questions including list of people to be interviewed, develop report outline. 2 days 2 days 2 days Conduct key informant interviews and meetings. 3 days 3 days 3 days Drafts/tools review 1 day 1 day 1 day In country travel to field sites 1 day 1 day 1 day Field visit for interviews/FGDs 5 days 5 days 5 days Team analysis of findings/consensus on conclusions and recommendations, 3 days 3 days 3 days Conduct debriefings for USAID 2 days 2 days 2 days Finalize outline for the report, prepare draft report --First draft submission prior to team departing country (incorporate comments from briefings) 5 days 5 days 5 days 36 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Activity Person Days Per Person Team Leader Team Member Team Member Logistics USAID comments on draft (10 days) Report finalization (based on Mission‘s comments) - to take place out of country 5 days 3 days 3 days Total LOE in person days 35 days 29 days 29 days 20 days *A six day work week is approved while the team is working in country. Please note that actual travel time will depend upon the consultant‘s home location. The Evaluation Team will consist of up to 4 members including a Team Leader and a logistics assistant depending on availability of funds. The team members should represent a balance of several types of knowledge related to MCH-FP service delivery in Nigeria, health services planning and programming as well as private sector entrepreneurship and commercial marketing. In addition to technical members, the team will have a host country national to provide administrative and logistics support. The technical team members must all have significant national/international health program experience. They should have some Nigeria country or African regional experience, along with comparative experience in social marketing, and MCH-FP service delivery in other countries or regions of the world. At least one member of the team must have Nigeria experience and be familiar with the MCH-FP service delivery structure in the private sector. Some experience in conducting evaluations or assessments is expected of all members, and experience in developing strategies would be useful. Substantial experience in international health is required. All team members must have professional-level English speaking and writing skills. A general idea of the responsibilities and necessary skills/experience of the Team Leader is described below. The Team leader will work with two other national consultants. Each consultant is expected to have an advanced degree in health management, health finance, public health or related field. Demonstrable expertise in marketing, private sector and sales; family planning services, additional /complementary expertise in social communication and media are highly recommended. Team Leader- The Team Leader will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables in consultation with the other members of the team. The team leader will develop tools for the evaluation and share it with USAID/Nigeria. The team leader will develop the outline for the draft report, present the report and after incorporating USAID Nigeria staff comments if necessary, submit the final report to USAID/Nigeria within the prescribed timeline. Skills/Experience: The Team Leader should have: 1. Advanced degree in health management, health finance, public health or related field 2. At least 10 years working experience in the field of international health; 3. Knowledge of health systems and health issues in Nigeria; 4. A good understanding of USAID project administration; 5. Program planning, evaluation, and design experience; IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 37 6. Experience leading a team for international health program evaluations or related assignments; and 7. Excellent writing, communication, and presentation skills The Team Leader will be responsible for overall management of the evaluation; including coordinating and packaging the deliverables in consultation with the other team members. In addition to the technical responsibilities outlined in the scope of work for the assignment, team leader responsibilities include: Preparations 1. Finalize and negotiate with client for the team work plan for the assignment 2. Establish assignment roles, responsibilities, and tasks for each team member 3. Ensure that the logistics arrangements in the field are complete Management 1. Facilitate the Team Planning Meeting to set the agenda and other elements of the TPM 2. Take the lead on preparing, coordinating team member input, submitting, revising and finalizing the assignment report 3. Manage the process of report writing 4. Manage team coordination meetings in the field 5. Coordinate the workflow and tasks and ensure that team members are working to schedule. 6. Ensure that team field logistics are arranged (e.g., administrative/clerical support is engaged, ensuring that payment is made for services, car/driver hire or other travel and transport is arranged, etc.) Communications 1. Handle conflict within the team 2. Serve as primary interface with the client and serve as the spokesperson for the team, as required. 3. Debrief the client as the assignment progresses, and organize a final debriefing 4. Keep the GH Tech HQ staff apprised of progress challenges, work changes, team travel plans in the field, and report preparation via phone conversation or email at least once a week 5. Serve as primary interface with GH Tech in submission of draft and final reports/deliverables to GH Tech. 6. Make decisions about the safety and security of the team in consultation with the client and GH Tech HQ. Direction 1. Assume technical direction lead as required in order to ensure quality and appropriateness of assignment and report content. The expected in-country timeframe for this task is approximately August 1-August 24 2011. Specific start and end dates, travel dates, and due dates for deliverables will be determined in collaboration with USAID and based on the availability of the consultants, and a detailed timeline will be produced during the team planning meeting. 38 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION IX. LOGISTICS A six day work week is authorized for the Evaluation Team while in Nigeria. USAID/Nigeria will provide overall direction to the team, provide key documents and background materials for reading and help arrange the in-briefing and debriefing. GH Tech will provide technical and administrative support including identification and fielding appropriate consultants. In addition, GH Tech will provide all logistical arrangements such as flight reservations, country cable clearance, in-country travel, airport pick-up, lodging and interpreters, as necessary. For business class travel, GH Tech will follow prior agreements with USAID/Washington. The Administrative/Logistics Assistant will be hired locally to arrange field visits, key informant interviews and meetings, local travel, hotel and appointments with stakeholders. X. DELIVERABLES AND PRODUCTS 1. An Evaluation work plan and timeline: prepared during the Team planning Meeting. 2. A detailed report outline: prepared during the Team planning Meeting. 3. Questionnaire/guideline for conducting key informant interview and FGD: prepared during the team planning, submitted to the Mission for review and approval prior to initiating key informant interview and site visits. 4. Debriefings: The full team will debrief USAID/Nigeria on their findings, conclusions and recommendations, before leaving Nigeria. Two power-point presentations for debriefing summarizing findings, conclusions and recommendations will be prepared and distributed during debriefing. USAID will provide feedback during the briefing meeting, and debriefing(s). 5. Draft Evaluation Report: A synthesized draft report will include, at a minimum, the following: scope and methodology used; important findings (empirical facts collected by evaluators); conclusions (evaluators‘ interpretations and judgments based on the findings); recommendations (proposed actions for management based on the conclusions); and lessons learned (implications for future designs and for others to incorporate into similar programs). 6. The Evaluation Team will provide USAID/Nigeria with a draft report that includes all the components of the final Evaluation report prior to their departure from Nigeria. USAID/N will provide written comments on the draft report to the Evaluation Team within 10 working days of receiving the draft report. 7. Final Evaluation: The final report will address the comments provided by USAID/Nigeria on the draft report. The team leader will revise the draft report and deliver an electronic copy of the final revised version to USAID/Nigeria within three weeks of receiving USAID feedback. This report will be a public document. Any procurement sensitive pieces and future recommendations will be separated from the final report and be included in an internal strategy recommendation document. This document will be internal to USAID/Nigeria. Discussions and recommendations related to SOW can be made publicly available. GH Tech will provide the edited and formatted final document approximately 30 days after USAID/NIG provides final approval of the report. The report will not be longer than 40 pages total, excluding Annexes. GH Tech will provide 5 printed and an electronic file. GH Tech will make the results of the evaluations public on the Development Experience Clearinghouse and on its project web. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 39 Proposed OUTLINE for Evaluation REPORT (to be finalized during the TPM) TABLE OF CONTENTS ACKNOWLEDGEMENTS ACRONYMS AND ABBREVIATIONS EXECUTIVE SUMMARY INTRODUCTION Purpose Background Methodology FINDINGS Overall Keys Issues (in the SOW) CONCLUSIONS (as well as any limitations in the inferences) LESSONS LEARNED PRIORITIZED RECOMMENDATIONS FUTURE DIRECTIONS ANNEXES: Evaluation Scope of Work Annotated List of Documents Collected and Reviewed Persons Contacted XI. RELATIONSHIPS AND RESPONSIBILITIES The Evaluation Team will work under the technical direction of USAID/Nigeria. USAID/Nigeria will: Approve country clearances for travel Provide the team with a general list of suggested organizations and contact information, Arrange for initial communication with appropriate government and other organizations at the outset of the process. Client Roles and Responsibilities: Before In-Country Work Documents. Identify and prioritize background materials for the consultants and provide them, preferably in electronic form. Local Consultants. Assist with identification of potential local consultants and provide contact information. 40 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION USAID-Supplied Participants. Provide guidance regarding participation in the assignment by Mission and USAID/W staff (i.e., who will participate, how long, source of funding for their participation). During In-Country Work Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Mission Point of Contact person(s) and provide technical leadership and direction for the team‘s work. Formal and Official Meetings. Arrange key appointments with national and local government officials and accompany the team on these introductory interviews (especially important in high-level meetings). Other Meetings. If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment. Facilitate Contacts with Partners. Introduce the team to project partners, local government officials and other stakeholders, and where applicable and appropriate, prepare and send out an introduction letter for team‘s arrival and/or anticipated meetings. After In-Country Work Timely Reviews. Provide timely review of draft/final reports and approval of the deliverables GH Tech will be responsible for all assignment related expenses for their consultants incurred in carrying out this review including travel, transportation, lodging, and communication costs, etc. The Evaluation Team will be responsible for expanding the list of organizations and persons, and for arranging meetings and appointments. XII. MISSION AND/OR WASHINGTON CONTACT PEOPLE/PERSON 1. Dr. Folake Olayinka Agreement Officer‘s Technical Representative, Improved Reproductive Health In Nigeria (IRHIN) USAID Nigeria Office of Population, Health, Nutrition. folayinka@usaid.gov Cell: 08037129956 2. Kayode Morenikeji Alternate Agreement Officer‘s Technical Representative (IRHIN), kmorenikeji@usaid.gov cell phone-08033177096 Other Nigeria HPN staff that will be on the teams: Joseph Monehin-MCH Program Manager Possible participation from DFID staff member Travel Cost associated with these staff will directly by USAID/Nig XIII. COST ESTIMATE: TBD To be provided by GH Tech IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 41 XIV. REFERENCES (PROJECT AND RELEVANT COUNTRY DOCUMENTS) ANNEX A. List of Pertinent Documents 1. IRHIN/IRHIN Cooperative Agreement 2. Annual work plans 3. Annual reports 4. Monitoring and Evaluation plan/PMP 5. Previous IRHIN Evaluation Reports 6. USAID/Nigeria Maternal, Child and Reproductive Health Program Mid- term Evaluation, Nov 2009 7. Nigerbus Surveys 2005-2011 8. Documenting IRHIN- A story of breaking borders 9. Introducing Standard Days Method-using cycle beads 10. National Strategic Health Development Plan 2015-2016 11. RH Policy 12. Checklist for Assessing USAID Evaluation Reports 1 Ibid Other Illustrative Questions for the Team to Consider Increased access How well understood is the potential of social marketing in Nigeria and what more can be done to enhance understanding of social marketing contributions to MCH/FP and HIV/AIDS objectives? Describe IRHIN‘s collaboration and relationships with the private commercial and NGO sectors as well as the GON. How well has IRHIN utilized MCH/FP and HIV/AIDS data in making other programmatic decisions, for example, data over time on contraceptive prevalence and reasons for continuation/discontinuation of family planning? How well has the social marketing project reflected GON policies and planning with regard to MCH/FP and HIV/AIDS? Storming the Nightingale. IRHIN supports this activity to increase use of FP in large clinical facilities. What are the costs of these sessions and what evidence is there that they make a difference in comparison to baseline or alternative approaches? Targeting, Reaching and Influencing Users Working with available data sources including the most recent Demographic Health Survey and any more recent MCH/FP and HIV/AIDS projections, have current and potential users/consumers been appropriately targeted by USAID-funded social marketing projects for the largest possible impact? If not, how can targeting for impact be maximized? Has social marketing achieved nationwide coverage in terms of targeted segments of the population and respective products and services? Have social marketing strategies been adjusted appropriately for urban and rural areas? What has been the cost of social marketing products and services over time, and should prices for products and services be adjusted in future? 42 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION What links should exist between social marketing communication change strategies and activities and other BCC programs in FP and HIV/AIDS? What kind of data and analyses has IRHIN commissioned over time and how has this information been used by project managers in decision-making? Are there other equally or more appropriate data collection and analyses that USAID should commission under the current project or any future project? Has IRHIN promoted healthy behaviors successfully; which behaviors and with what audiences/groups? Has IRHIN appropriately targeted those groups and developed appropriate strategies and tools to reach and influence those groups? Have the poor, who may be unable to pay market rates for FP commodities, been reached by USAID-funded social marketing projects? Have groups who are unable or unwilling to access other public and private sector services and facilities been served by social marketing behavior change activities, marketing strategies and sales outlets—e.g. youth? Are their efficiencies in the current training model for reaching PPMVs? What else needs to be done? Annex 2 Criteria to Ensure the Quality of the Evaluation Report The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation composition, methodology or timeline need to be agreed upon in writing by the technical officer. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people‘s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 43 ANNEX B. DOCUMENTS REVIEWED Auricle Services, Lagos, Nigeria (2010). Documenting IRHIN: A Story of Breaking Borders. Federal Ministry of Health, Abuja, Nigeria (2007). Integrated Maternal, Newborn and Child Health Strategy. Federal Ministry of Health, Abuja, Nigeria (2010). National Strategic Health Development Plan 2010-2015. National Population Commission (NCP) and ICF Macro, Maryland (2009). 2008 Nigeria Demographic and Health Survey. Population Services International, Social Marketing Research Series (2008). Nigeria (2007): MAP Study on the Availability of Social Marketing Products in Nigeria, Phase 2. Society for Family Health, Research and Evaluation Division, Nigeria (2006). Report on Monitoring, Access and Performance (MAP) Survey. Society for Family Health, Research and Evaluation Division, Nigeria (2009). 2008 Report on Monitoring, Access and Performance (MAP) Survey. Society for Family Health, Research and Evaluation Division, Nigeria (2009). 2009 Report on Monitoring, Access and Performance (MAP) Survey. Society for Family Health, Research and Evaluation Division, Nigeria (2011). 2010 Report on Monitoring, Access and Performance (MAP) Survey. Society for Family Health, Nigeria (2010). Evaluation of Community-level Family Planning Interventions in Nigeria: A Qualitative Study. Society for Family Health, Nigeria (SFH), Population Services International (PSI,) and Pathfinder International (2007). IRHIN Project: FY 2007 Final Report. Society for Family Health, Nigeria (2007). IRHIN Nigeria: FY 2007 Q4 Quarterly Report Matrix. Society for Family Health, Nigeria (2007). IRHIN Nigeria: Results Tables FY 2007. Society for Family Health, Nigeria (2008). Quality of Family Planning Services Among Private Patent Medicine Vendors in Nigeria. Society for Family Health, Nigeria (2010). IRHIN 2005-2010: Prepared for RFA No: 620-05-004 Contraceptive Social Marketing in Nigeria. Society for Family Health, Nigeria (2007). IRHIN Nigeria: Results Tables for Q2 FY 2010. Society for Family Health, Nigeria (2007). IRHIN Nigeria: Results Tables for Q3 FY 2010. Society for Family Health, Nigeria (2007). IRHIN Nigeria: Results Tables for Q4 FY 2010. Society for Family Health, Nigeria (SFH), Population Services International (PSI) and Pathfinder International (2009). IRHIN Project: FY 2009 Final Report. Society for Family Health, Nigeria (SFH), Population Services International (PSI), and Pathfinder International (2010). IRHIN Project: Q1 FY 2010 Report. Society for Family Health, Nigeria (SFH), Population Services International (PSI), and Pathfinder International (2010). IRHIN Project: FY 2010 Final Report. Society for Family Health, Nigeria (2010). Evaluation of Community-level Family Planning Interventions in Nigeria: A Qualitative Study. 44 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Society for Family Health, Nigeria (2010). Exposure to Family Planning Campaign in Nigeria: Analysis of October, 2010 Nigerbus Survey. Society for Family Health, Nigeria (2011). Exposure to Family Planning Campaign in Nigeria: Analysis of October, 2010 Nigerbus Survey. Society for Family Health, Nigeria (2011). Introduction of the Standard Days Method of Family Planning in Two States of Nigeria: Experiences with Use. Society for Family Health, Nigeria (2011). RH Community Based Project: Documenting Successes and Lessons Learnt. United States Agency for International Development (USAID), Nigeria (2007). IRHIN: Performance Management Plan. United States Agency for International Development (USAID), Nigeria (2008). Maternal Child Health, Family Planning and Reproductive Health Strategic Approach. United States Agency for International Development (USAID), Nigeria (2009). Maternal, Child, And Reproductive Health Program Mid-Term Evaluation. United States Agency for International Development (USAID), Nigeria (2010). Mission Strategy: 2010 – 2013. IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 45 ANNEX C. LIST OF PERSONS CONTACTED FEDERAL MINISTRY OF HEALTH – FAMILY HEALTH DEPARTMENT Dr. B. Okoeguale Acting Director Dr. Bose Adeniran Head, Reproductive Health Division Mrs. A.O. Osuntogun Assistant Director, Safe Motherhood Unit Mrs. C.N. Akinsanmi Chief Nursing Officer, Reproductive Health Division Nneka Oteka Chief Health Education Officer, Family Planning Unit SOCIETY FOR FAMILY HEALTH – NATIONAL OFFICE, ABUJA Sir Bright Ekweremadu Managing Director Obi Oluigbo ` Chief Technical Officer Chinazo Ujuju Senior Officer Research Zainab Usman SBC Officer Fatima Muhammad Associate Director, Family Planning/RH Services Ineala Theophilus Senior Officer Family Planning PATHFINDER INTERNATIONAL – ABUJA OFFICE Chinwe Onumonu Director of Programs Fatima Mamman Daura M&E Specialist DFID Ebere Anyachukwu Health Adviser PHARMACISTS COUNCIL OF NIGERIA – ZONAL OFFICE, GWARIMPA Pharm. P.N. Iliya Zonal Officer-in-charge Janet Ugbem Inspection & Monitoring Katherine Eje Education & Training PHC KURUDU Keziah P. Markus Community Health Officer Lami L. Alkali Community Health Extension Worker Patience J. Bawa Community Health Extension Worker SOCIETY FOR FAMILY HEALTH (SFH), FIELD OFFICE – ABUJA Grace Yisa Regional Manager Dennis Aizobu Area Sales Manager 46 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Saman Andrew Health Communication Coordinator Benson Olemu Detailer Jennifer Haruna Peer Educator Juliet Boniface Peer Educator Titilayo Lukman Peer Educator Patience Bawa Peer Educator Lilian Mchivga IPC Conductor Lami Alkali IPC Coordinator CSO—TENDER CARE FOR HUMAN DEVELOPMENT Joy Olagundoye Program Officer COMMUNITY LEADERS, KURUDU Kuyembo Auta Chief of Kurudu Yohaba Zago Ezekiel Bawa John Egude Karshi Adaba Sunday Pada Yusuf Maikasuwa Stephen Bessanyi Samuel Jeshi Shekwozhibo Alkali Maikasuwa Gambo COMMUNITY MEMBERS KURUDU Nbasayi Yako Scala Poyi Sunday Ada Hon. Jeshi Egwe John Kuyembo Robo Shekwozhibo Poyi Karatu Emmanuel Ada Tazamu Yohana Tani Luka IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 47 UN AGENCIES Dr. Bannet Ndyanabangi Deputy Representative, UNFPA Dr. Andrew Mbewe WHO Dr. Taiwo Oyelade WHO KAPITAL RADIO STATION—ABUJA Alice Alkali Manager, Program Sola Odewole Marketing Accountant Abimiku Ibrahim Marketing Department SOCIETY FOR FAMILLY HEALTH—KADUNA FIELD OFFICE Ibrahim Galla Regional Manager Asabe Rose Waida Focal person (Women Health Project) Emmanuel Ede Detailer Bashir A. Bashir HCC Agnes Gata Global Fund Malaria State Coordinator Kayode Oyatoye Motor Vehicle Officer Garba Bako Motor Vehicle Officer Abdullahi Idris Motor Vehicle Officer Isa Ahmed Bello Motor Vehicle Officer/SID OLatunbosun Kolawole Global Fund HIV HCT State Coordinator Temitope Kajola Youth Corp member Abdulhamid Maiwada Youth Corp member RIGASA COMMUNITY Abubakar Tarima Peer Educator Abulhamid Yakubu Peer Educator Nuhu Isa Mohd Secretary District Head Yakubu Bala Member Jibrin Mohd Village Head Hajiya Hadiza Shehu P.A. Village Head Hajiya Saadiya Ishaq Social Welfare Officer Zainab Abdulaziz Member H. Jumar Abdulaziz Member Asabe Abubakar Member Hajiya Tanbari Member 48 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION Amina Hussaini Member Aliyu Bardi Rigasa Peer Educator Shehu Balarabe Member PPMV WHOLESALERS KADUNA Joseph Nwani PPMV Sabon Tasha Elijah Ogbuka Wholesaler WOMEN DEVELOPMENT ORGANIZATION Hajiya Aisha Suleiman Executive Director Habiba Ladan Program Officer KADUNA STATE PPMV ASSOCIATION Clifford Chukwu Member Martins Jimoh Member Senson Udeh Member Inusa Falgore 2nd Vice President Nduka Cosmas Chairman Rashidat Moshood Member MC ROYAL HOSPITAL, KACHIA Dr. R.M. Folajimi Medical Director Mr. John Bernard Community Health Extension Worker ZONKWA MEDICAL CENTER Dr. Bode Odesanya Medical Director Priscilla Gaya Matron SOCIETY FOR FAMILY HEALTH LAGOS FIELD OFFICE Toyin Ogbondeminu Regional Manager Babalola Biyi Sales Manager Christopher Meraiyebu Medical Detailer Ikejide Sebastine Assistant Sales Manager Yinka Goodman Territorial Manager SOCIETY FOR FAMILY HEALTH WAREHOUSE LAGOS Mr, Emetha Okafor Warehouse Manager IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 49 HUMAN DEVELOPMENT INITIATIVE Olubunmi Shonde Program Officer Olufunso Owasanoye Program Advisor Yinka Saka Finance Officer Bolanle Dare Senior Program Officer PHC AKOKA Mrs. Mercy Adekunle Chief Nursing Officer AKOKA COMMUNITY Mrs. K.O. Atanda IPC Conductor Chief (Mrs) Akinwale Peer Educator T.A. Akinbami IPC Conductor L.A. Olukokun IPC Conductor L.O.S. Animashaun Peer Educator A. Oyeniran Peer Educator Agbomola IPC Conductor Ayanwale Peer Educator Ibrahim Peer Educator Busari Peer Educator LAGOS STATE PPMV ASSOCIATION Mr. Austin Ezeh Chairman of Bariga Zone Mr. Stanley Nwakanma Secretary Mr. Vitalis Udagwa Member MDS LAGOS Mrs. D.S. Kantiyok Manager, Lagos Area Warehouse WHOLESALERS LAGOS Mrs. Ugochi Sales Clerk, Hado Pharmacy Mr. Chinedu Pharma Medica Mr. Chide Nwadike Fibica Pharmacy SOCIETY FOR FAMILY HEALTH—ENUGU FIELD OFFICE Ifesi Chilota Detailer Okoro Eucharia Health Communication Coordinator Adizue Jane Regional Manager 50 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION NAPMED ASSOCIATION ENUGU Chief Sylvester Chukwu State President Ike Olisa Jideofor Unit Chairman Kingsley Anyanwu Financial Secretary Ezekiel Okpoko Asst Secretary Nawu Felix Secretary Ume Anayo Member Udeh Davidson Member SAFEMOTHERHOOD LADIES ASSOCIATION (SMLAS) Ugo Uduma Executive Director Okechukwu Chioma FP Accountant Ugwuocha Temple Program Officer Oko Chima Program Officer EZZANGBO COMMUNITY – EBONYI STATE Eze Dr. Chibueze.O. Agbo Community Head Egbe Loveline Peer Educator Igwu Agu Peer Educator Onwe Amaka Augustine Peer Educator Chibueze Chinyere Peer Educator Okwe Solomon Peer Educator Onwe Patrrick IPC Conductor Una Chika IPC Conductor LITTLE SEEDS MATERNITY HOME Rose Okorie Director IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 51 ANNEX D. KEY INFORMANT, FGD, AND IN-DEPTH INTERVIEW QUESTIONS KEY INFORMANT INTERVIEW QUESTIONS—FMOH 1. How has the Integrated Reproductive Health in Nigeria (IRHIN) contributed to the national response for improving RH services in Nigeria? (Probe: increasing access, quality, demand) 2. How important are FP and social marketing to GON MCH/FP objectives? 3. How have you worked with the IRHIN Project over the past six years? (policies, linkages, etc.) 4. What more can be done over the next 5-10 years to enhance understanding and use of social marketing in achieving RH objectives in Nigeria? KEY INFORMANT INTERVIEW QUESTIONS—DONORS (DIFD, UNICEF, WHO, UNFPA) 1. What was your relationship and linkage with the IRHIN Project? (commodities procurement, capacity building, etc) 2. What were the major contributions of the IRHIN Project? (improved access, demand, quality, enabling environment) 3. What were the strengths and weaknesses of the program approach adopted under the IRHIN Project? (Probe: is social marketing an appropriate strategy for improving FP use?) 4. What future directions should social marketing take in this country over the next 5-10 years? KEY INFORMANT INTERVIEW QUESTIONS—SFH, PI, PSI 1. How well has IRHIN increased access to CS & RH services? (Probe: coverage, effectiveness of targeting) 2. How has IRHIN improved quality of RH services? (Probe: capacity of providers, availability of products, quality standards and protocols, client satisfaction) 3. How has IRHIN contributed to strengthening the enabling environment for the provision of FP services? (Probe: policies, linkages, GON support, attitudinal changes among religious and community leaders) 4. How has the IRHIN Project expanded demand for CS & RH services? (mass media coverage and effectiveness of messages in changing behavior; effectiveness and appropriateness of demand creation strategy) 5. How has project management contributed to IRHIN achievements? (describe management approach, roles of sub-partners, communication, M&E, flow of data and information) 6. What were the strengths and weaknesses of the program approach adopted under IRHIN? (Is social marketing of contraceptives still an appropriate strategy for increasing FP use?) 7. Lessons learned—what worked and what didn‘t? (Is there a written report from PI‘s seminar on lessons learned and best practices held in 2010) 8. What future directions should social marketing take over the next 5-10 years in Nigeria? 52 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 9. What has IRHIM done toward sustaining individual program efforts such as social marketing, utilization of private clinics and providers, CSO, community participation? (state and local gov. involvement and buy-in) 10. What has been PSI‘s role and contribution to the IRHIN Project? 11. How has the community benefited from participating in this project? (Probe: specific benefits) 12. Any suggestions on how to improve the project to help you in the future? QUESTIONS FOR FGD WITH COMMUNITY BENEFICIARIES 1. How did you get involved with the IRHIN Project? 2. What do you know about the IRHIN Project? 3. How have you and the community benefitted from the IRHIN Project? 4. Do you have suggestions for improving the project? KEY INFORMANT INTERVIEW QUESTIONS FOR ABUJA RADIO & ESBS MEDIA HOUSE 1. What is your relationship with the IRHIN Project? (Probe: Length of time of relationship; activities) 2. What role do you play in the production of the messages aired by your station? 3. What is the station‘s reach? (Probe: population coverage; any mechanism in place for feedback from listeners) 4. How is the airing of messages funded? (Probes: who funds it; is it at a discounted rate; is it part of your corporate social responsibilities) KEY INFORMANT INTERVIEW QUESTIONS FOR JSI/DELIVER 1. What is your relationship with the IRHIN Project? (Probe: length of time of relationship; activities carried out in this relationship) 2. How has JSI/Deliver contributed to improving/streamlining FP commodities distribution system and logistics management? 3. What constraints remain in the FP commodities logistics system? (forecasting, procurement, clearing and warehousing, distribution and management of expired commodities) 4. What can be done to improve/overcome these existing constraints? KEY INFORMANT INTERVIEW QUESTIONS FOR PHARMACISTS (WHOLESALERS) 1. What is your relationship and role with the IRHIN Project? (Probe: how long, specific activities, role as wholesaler, etc.) 2. What FP/MCH commodities are stocked through SFH? 3. How do you get the commodities? 4. How do you distribute the commodities to PPMVs? 5. How do you ensure compliance with PNC regulations? (Probe: licensure, pricing, storage, expired commodities) 6. Do you stock other non-SFH FP/MCH commodities? ORS? (Probe: How do sales compare? Turnover? Profitability, availability compared to SFH products?) IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 53 KEY INFORMANT INTERVIEW QUESTIONS FOR SFH FIELD STAFF 1. What are the major achievements in Kaduna/Lagos/Enugu under the IRHIN Project in increasing access, demand, quality of FP services and strengthening the enabling environment? 2. How have you established linkages with state and local governments, NGOs/CBOs? 3. Program Management Systems—staff turnover, M&E, info flow of data and communication with HQ 4. Lessons learned—what has worked? What hasn‘t? 5. What constraints and challenges have you faced and how resolved? 6. Future directions—how can the project improve, change for better results? KEY INFORMANT INTERVIEW QUESTIONS FOR CBOS/FBOS/NGOS 1. What is your relationship and role with the IRHIN Project? 2. How have you engaged the PEs and IPCCs in demand creation? 3. How has your organization benefited from the IRHIN Project? (Probe: improved sustainability) 4. Lessons learned. What has worked? What hasn‘t? 5. What challenges and constraints have you faced with the IRHIN Project? 6. What suggestions do you have for future improvements in the project? KEY INFORMANT INTERVIEW QUESTIONS FOR MDS 1. What is your relationship and role with the IRHIN Project? 2. Please describe the distribution process with SFH commodities? 3. What are the main challenges you face with distribution of SFH commodities? 4. How can distribution be improved/streamlined in the future? 54 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION 55 ANNEX E. MAP OF NIGERIA AND IRHIN PROJECT COMPONENTS LOCATION OF IRHIN PROJECT ACTIVITIES: Social Marketing of FP Commodities and Mass Media: Nationwide Improved Quality of Service Delivery through 28 Private Facilities and Community-based Distribution: Abia, Cross River, Kaduna, Community-based IPC for FP and Service Referrals: 1–2 communities located in each of the following states – Abia, Adamawa, Bauchi, Bayelsa, Benue, Borno, Cross River, Delta, Ebonyi, Edo, FTC, Gombe, Imo, Kaduna, Katsina, Kwara, Jigawa/Kano, Lagos, Ogun, Ondo, Rivers, Zamfara 56 IMPROVED REPRODUCTIVE HEALTH IN NIGERIA: END OF PROJECT EVALUATION For more information, please visit http://resources.ghtechproject.net Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com