1 | Page USAID/West Africa Evidence for Development! AID-624-C-15-00001 PACTE-VIH MidteOperatirmPerformanceaso Document Review USAID/West Africa Evidence for Development AID-624-C-15-00001 FINAL EVALUATION PROJECT Leadership Capacity Strengthening (CAPS) Project Performance Evaluation FINAL REPORT January 2020 Disclaimer: This document has been produced with the support of the American people through the United States Agency for International Development (USAID). The contents of this document are the sole responsibility of International Business and Technical Consultants, Inc. (IBTCI), and do not necessarily reflect the views of USAID or the United States Government. 2 | Page USAID/West Africa Evidence for Development AID-624-C-15-00001 FINAL EVALUATION PROJECT Leadership Capacity Strengthening (CAPS) Project Performance Evaluation FINAL REPORT Disclaimer: T Disclaimer: This document has been produced with the support of the American people through the United States Agency for International Development (USAID). The contents of this document are the sole responsibility of International Business and Technical Consultants, Inc. (IBTCI), and do not necessarily reflect the views of USAID or the United States Government. 3 | Page TABLE OF CONTENTS ACRONYMS……………………………………………………...…………………………………….6 EXECUTIVE SUMMARY……………………………………………………………………………….7 EVALUATION PURPOSE AND PERIOD..................................................................................................7 EVALUATION QUESTIONS ...................................................................................................................7 EVALUATION DESIGN AND METHODOLOGY.......................................................................................7 Sampling..........................................................................................................................................8 Data collection ................................................................................................................................8 Data analysis...................................................................................................................................8 CAPS EVALUATION KEY FINDINGS ....................................................................................................10 Overall achievements of CAPS project..........................................................................................10 Question 3: WAHO’s role in advocacy and communication efforts regarding the scaling up of good practices...............................................................................................................................13 WAHO’s leadership in improving the supply chain and commodity security...............................15 WAHO’s role in promoting Public–Private Partnerships (PPP).....................................................15 WAHO’s role in strengthening the HIS and improving reporting .................................................16 Appreciation of WAHO’s leadership in advancing health issues and outcomes in the region......17 LESSONS LEARNED............................................................................................................................20 LESSONS LEARNED FROM THE EVALUATION AND LIMITS................................................................21 KEY RECOMMENDATIONS TO WAHO FOR LEADERSHIP DEVELOPMENT .........................................22 CONCLUSION.....................................................................................................................................24 INTRODUCTION…………………………………………………………………………………….26 Description of the Problem, Development of Hypotheses, and Theory of Change .........................26 Results Framework ...........................................................................................................................27 Figure 1. CAPS Project results framework………………………………………………………………..27 EVALUATION PURPOSE AND PERIOD…………………………………………………………….28 EVALUATION QUESTIONS………………………………………………………………………….28 EVALUATION DESIGN AND METHODOLOGY…………………………………………………...28 Geographic Focus..............................................................................................................................29 4 | Page Evaluation Design..............................................................................................................................29 Sampling............................................................................................................................................30 Data collection ..................................................................................................................................31 Data analysis.....................................................................................................................................32 OVERALL EVALUATION ENVIRONMENT…………………………………………………………36 Key Informants (KIs) Profile ..............................................................................................................36 Awareness of the CAPS project by the resource persons surveyed .................................................36 Organizations Supporting Health Policy Implementation through the Region ................................37 Typology of government support for health system capacity-building............................................39 CAPS EVALUATION KEY FINDINGS………………………………………………………………..40 Question 1: To what extent is WAHO considered a leader in advancing health issues and outcomes in the region?....................................................................................................................................41 Leading organizations in health issues in the ECOWAS region.....................................................41 How is WAHO considered as a leader?.........................................................................................42 Question 2: How is WAHO a leader in promoting and supporting the scaling up of good health practices? ..........................................................................................................................................45 Question 3: What is WAHO’s role in advocacy and communication efforts regarding the scaling up of good practices?.............................................................................................................................48 WAHO’s leadership in improving the supply chain and commodity security...............................50 WAHO’s role in promoting Public-Private Partnerships (PPP)......................................................52 WAHO’s role in strengthening the HIS and improving reporting .................................................54 Question 4: What are areas or domains of WAHO’s capacity that have improved as a result of USAID/WA support? .........................................................................................................................56 Appreciation of WAHO’s leadership in advancing health issues and outcomes in the region.....56 WAHO’s areas of expertise/capacities that have improved with USAID support from 2012 till now .............................................................................................................................................. .58 Current or potential challenges of WAHO leadership in supporting health policies in the countries.......................................................................................................................................60 LESSONS LEARNED…………………………………………………………………………………..64 LESSONS LEARNED FROM EVALUATION AND LIMITS…………………………………………..65 KEY RECOMMENDATIONS TO WAHO FOR LEADERSHIP DEVELOPMENT………………….. 66 5 | Page CONCLUSION…………………………………………………………………..…………….…….68 ANNEXES………………………………………………………………………...….……………….69 6 | Page ACRONYMS AfD Agence Française de Développement AHM Assembly of Health Ministers AIDS Acquired Immuno Deficiency Syndrome BP Best Practice CAPS Capacity Leadership Strengthening CDC Centers for Disease Control and Prevention CERA Cabinet de Recherche et d’Évaluation CM Consultative Meeting DHIS2 District Health Information Software 2 DQA Data Quality Assessment E4D Evidence for Development ECOWAS Economic Community of West African States ECSA-HC East Central and South Africa Health Community EO Expected Outcome EOP End of Project FGD Focus Group Discussion FP Family Planning FSW Female Sex Worker GAVI Global Alliance Vaccine Initiative GIZ German Agency for International Cooperation/Deutsche Gesellschaft für Internationale Zusammenarbeit HDI Human Development Indices and Indicators HIS Health Information System HIV Human Immunodeficiency Virus HMIS Health Management Information System IBP IBTCI Implementing Best Practices International Business & Technical Consultants, Inc. IDSR Integrated Disease Surveillance Response INSP National Institute of Public Health IP Implementing Partner JHPIEGO John Hopkins Program for International Education in Gynecology and Obstetrics JICA Japan International Cooperation Agency KfW German Cooperation-Kreditanstalt für Wiederaufbau (in Deutsche; KfW in French is Reconstruction Credit Institute for German Cooperation) KI Key Informant KII Key Informant Interview LMG Leadership Management in Governance M&E Monitoring and Evaluation MNCH Maternal, Newborn, and Child Health MEL Monitoring and Evaluation Learning MOH Ministry of Health NGO Non-governmental Organization PAGE Large-Scale Passage PPP Public–Private Partnership RG Recipient Group RHO Regional Health Office TFP Technical and Financial Partner UN United Nations UNICEF United Nations International Children’s Emergency Fund UNPFA United Nations Population Fund USAID United States Agency for International Development USAID/WA United States Agency for International Development/West Africa WAHO West African Health Organization WHO World Health Organization 7 | Page EXECUTIVE SUMMARY In 2015, the USAID/West Africa (USAID/WA) Regional Health Office awarded the West African Health Organization (WAHO) a four-year cooperative agreement to implement the Leadership Capacity Strengthening (CAPS) project. The purpose of the project is to strengthen the capacity of WAHO, as a leader in Health in West Africa, to be the lead health institution in the region for policy harmonization and advocacy, promoting public– private partnerships for health and identifying, sharing, and supporting the scale-up of best practices in health. The purpose of the project would contribute directly to USAID/West Africa’s Regional Development Cooperation Strategy’s Intermediate Result 3.2 – Enabling Environment at the Regional and National Level Strengthened. In August 2018 USAID/WA approved the statement of work for Evidence for Development (E4D) to conduct the performance evaluation of CAPS. EVALUATION PURPOSE AND PERIOD The purpose of the performance evaluation for the West African Health Organization (WAHO) Leadership Capacity Strengthening (CAPS) activity is to assess whether it has achieved its purpose. The USAID/WA Regional Health Office (RHO) is interested in knowing whether intermediate results occurred and to what extent selected high￾impact practices are being advanced to document CAPS’ successes and challenges. This evaluation serves as an end￾of-activity performance evaluation. The period of performance to be evaluated is February 2015 to March 2019. The target audiences for the CAPS performance evaluation are USAID/WA Front Office, WAHO, USAID/WA RHO, USAID Bilateral Missions in the region, representatives from the Ministries of Health (MOH), and international partners/ donors [World Health Organization (WHO), United Nations International Children’s Emergency Fund (UNICEF), and United Nations Population Fund (UNFPA)] in the 15 Economic Community of West African States (ECOWAS) countries. EVALUATION QUESTIONS The following are the proposed evaluation questions in order of priority as set by USAID/WA: 1. To what extent is WAHO considered a leader in advancing health issues and outcomes in the region, especially in the area of policy harmonization and advocacy, in promoting Public-private partnerships (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key implementing partners [IPs] in the region)? 2. How has WAHO led and supported the scale-up of best practices in the region (e.g., mapping of current high￾impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale-up of interventions, health information systems integration, and capacity strengthening)? 3. What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale-up, national policy changes for family planning (FP) and HIV/AIDS programs, improved commodity security, public and private partnerships, and health information systems to increase investments in health and improve reporting? 4. What areas or domains of WAHO’s capacity (within institutional and leadership capacity) have improved as a result of USAID/West Africa’s support from February 2015 to March 2019 (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key IPs in the region)? EVALUATION DESIGN AND METHODOLOGY For this performance evaluation, the evaluation team collected data throughout the 15 ECOWAS member countries in West Africa that benefited from CAPS activities, focusing on stakeholders associated with WAHO, including the MOH in all countries and representatives of key partners including ECOWAS, WHO, Global Fund, Palladium, USAID Missions and their implementing partners, and the UN system. 8 | Page This performance evaluation is a cross-sectional study using mixed methods: qualitative, quantitative, primary and secondary data as per the USAID performance evaluation definition. It covers the whole WAHO’s leadership capacity-building activities through the implementation of the CAPS project. The quantitative component consisted of a desk review of CAPS project’s key documents: work plans; quarterly and annual reports; dashboard results; monitoring, evaluation, and learning (MEL) plan; and periodic performance reports. The qualitative component consisted of conducting two different categories of interviews: (1) individual interviews with key informants (KIs) targeting national stakeholders (MOH technical staff/managers) and representatives of IPs working in the field of FP/Maternal, Newborn, and Child Health (MCH); HIV/AIDS; Health Information Systems (HISs) from each of the 15 ECOWAS countries and connected to the CAPS project; and (2) focus group discussions (FGDs)/consultative meetings (CMs) organized with key stakeholders in six countries (Ghana, Nigeria, Niger, Burkina Faso, Togo, and Côte d’Ivoire). Sampling The CAPS performance evaluation covered all 15 countries of the ECOWAS region. This is an exhaustive sampling of the countries. The evaluation included KIs purposely selected from the list of WAHO officials and focal points in the various ministries of health in the ECOWAS countries, specialists and managers of the national health information system (HIS), and IPs involved in strengthening national health policy. The list of the KIs to be interviewed in each country was consolidated in coordination with WAHO management team, USAID/WA Office, and in-country WAHO liaison officers. Selection criteria were based on their involvement in health program management and connection to CAPS project activities. The evaluation team conducted 75 individual interviews with KIs and 12 FGDs/CMs. The rationale guiding the sampling required conducting four KIIs per country distributed as follows: two officials from the MOH directly involved in the CAPS activities at the country level, one in-country WAHO representative, and one IP representative. The structure of the sample was maintained across the 15 ECOWAS countries. In Burkina Faso, where WAHO is based, E4D conducted six interviews targeting WAHO’s leadership and management: the CAPS Project Manager under WAHO, WAHO’s Monitoring and Evaluation Specialist, WAHO’s Capacity-Building Specialist, and WAHO’s Communication Specialist. Data collection The team collected the CAPS routine reports and conducted a full desk review during the period from January to July 2019. The interviews with KIs, conducted from June to August in all 15 countries, were in English, French and Portuguese. Data were collected through in-depth interviews and FGDs/CMs with KIs using three semi-structured interview guides for the different respondent groups— national stakeholders/IPs, the WAHO representatives, and KIs at the global level such as WHO HQ [Implementing Best Practices (IBP)]. E4D’s capacity strengthening recipient groups (RGs) in USAID/WA focus countries, who were members of the evaluation team, collected data face-to-face or remotely, where appropriate, for individual interviews. The FGDs/CMs were conducted face-to-face. Data Quality Assurance Data collection strategy and methods were maintained across the countries to minimize inconsistencies. To ensure consistency and good quality of data collected, E4D’s Senior Research & Evaluation Advisor followed up with the data collectors daily to ensure that the procedures were mastered. In countries where data were collected by RGs, the evaluation team established a mechanism of daily supervision of data collectors by phone calls and systematic feedback. The E4D staff in Accra carried out data quality assurance systematically in coordination with the evaluation team leader. Data analysis The evaluation team analyzed two main sources of data (programmatic data from desk review and qualitative data from KI interviews) for this performance evaluation - quantitative and qualitative. 9 | Page Quantitative data analysis Prior to the start of the data collection, E4D completed a desk review of the CAPS project documents: Proposal/Project description; Project’s routine reports, including annual and quarterly reports; Project’s work plans and associated Performance Monitoring Plans (PMP); and dashboard results. The team used this information/data to learn and describe the background of the project, its main objectives as well as proposed and implemented interventions over the life of the Project. In addition, these programmatic data, mainly the results from the dashboard indicators were used to document main achievements and activities completed or initiated under each Intermediate Result (IR), main challenges encountered, and actions undertaken to overcome these challenges. The team also used these programmatic data to understand the institutional organization and mandate of the CAPS project. Moreover, to have a better understanding of the Project’s performance, the team triangulated this quantitative information with the data drawn from individual Key Informant Interviews (KIIs) and Focus Group Discussions (see qualitative data section below). Qualitative data analysis The evaluation team collected qualitative data utilizing two main sources and methods. The first included individual interviews with Key informants both with national stakeholders and key implementing partner/cooperation agencies supporting the Ministry of Health (MOH) in its health policy implementation. The second method consisted of conducting Focus Group Discussion (FGD) with key informants both from national stakeholders and Implementing Partners (IPs). The qualitative data were analyzed by focusing mostly on the manifest content presupposing that statements made during the individual interviews and the FGDs are complete units from which emerging themes were identified. Data were coded by assigning the same code to the same or similar unit of meaning. The codes were generated by moving through the data and applying a mix of deductive and inductive coding. Principal themes and subthemes were identified and evidence that support or denies each theme and subtheme and links between them were sought. The team used the same method and approach to conduct the data analysis of the information collected through the FGDs without making any distinction between the FGDs conducted with national stakeholders and those conducted with the IPs. The final analysis helped to report the opinions and ideas collected and covered by all the stakeholders including the national or IPs. At the second level and to ensure standardized coding of transcripts, the interviews were analyzed using a thematic grid developed by randomly selecting three interviews and identifying initial themes. The analysis grid was adjusted as additional pertinent topics emerged. Data were analyzed with Nvivo 12 Pro and word clouds were created for the CAPS model of intervention and WAHO’s oversight of CAPS. Content analysis was used to emphasize some key and relevant information. The transcriptions in French and Portuguese were translated into English, and E4D ensured consistency with the original transcription. Specifically, the data analysis was conducted using the following procedures to answer each evaluation question: • Develop the grid (code list) through an initial identification of themes related to evaluation questions in three randomly selected interviews. • Expand the topics in the analysis grid (code list) as additional themes emerge. • Use verbatim sentences as the unit of analysis. • Group results by areas of interest (evaluation question) and generate word cloud for each evaluation question. • Identify different and dominant positions in relation to each evaluation question. • Summarize each position, assessing its strength or degree of importance and propose relevant quote to summarize the dominant opinions,. • Report information from FGD with national stakeholders and implementing partners separately. For example, for evaluation question #2 related to WAHO’s role in Best Practices (BP) scaling up, emerging themes drawn from the interviews and FGDs were: sharing information on good practices, organizing regional BP forum, BP documentation guide, Establishment of BP committees, collaboration for BP promotion. The evaluation team created Verbatim and word cloud to confirm and highlight these themes (for each evaluation question). 10 | Page CAPS EVALUATION KEY FINDINGS Overall achievements of CAPS project The CAPS project was implemented in all 15 ECOWAS countries from February 2015 to September 2019 by WAHO, through funding from the USAID/WA regional office. The current performance evaluation period covers February 2015 through March 2019. In total, 22 main outputs were expected through 29 indicators and 268 activities by the end of project (EOP). At EOP, however, 344 activities had been completed, thus exceeding the target by 76. Basing the evaluation on the number of activities implemented, there is no doubt that the CAPS project entirely achieved the expectations and even went beyond the target. However, considering the level of achievement of indicators as the yardstick to measuring project performance, we see different levels of achievement but still CAPS project’s activities achievements stand out. There are remarkable achievements in the following areas: (1) health policy harmonization and number of resolutions validated by the ECOWAS Assembly of Health Ministers, (2) best practices scaling up, (3) HIS strengthening, (4) engaging private sector in supporting health policy/program implementation. Question 1: To what extent is WAHO considered a leader in advancing health issues and outcomes in the region? The question of leadership organization is not an easy one. According to some respondents, ranking an organization as a leader as far as health is concerned is something that cannot be assumed because they usually operate in different areas of intervention. Therefore, there appear to be many leading organizations, each with a specific area. Nevertheless, there seems to be unanimity in agreeing that the United Nations Agencies, led by WHO, UNICEF, and UNFPA, are leading organizations in the health sector together with some bilateral cooperation organizations. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings and results Comments To what extent is WAHO considered a leader in advancing health issues and outcomes in the region • Harmonization of health policies • Technical assistance • Regional strategies • Health management • Policies • Support • Funding • Programs • Strategies • Training • Partnership • Planning • Capacity building • Data • Information • Scaling-up • Promote and harmonize health policies • Provide technical assistance to support countries’ activities • Develop strategies that can have common application within the region and their advocacy • Get progressively involved in health management and organization It is obvious that WAHO has made a big progress in terms of leadership development in the last years. Its leadership in the sub-region is indisputable as legitimate institution with the official mandate from ECOWAS heads of state to coordinate and harmonize health policies. However, its leadership is still mitigated by issues like financial constraint. 11 | Page Word cloud of organizations recognized by KIs as leaders in advancing health issues and outcomes in the ECOWAS region The quote below summarizes the key findings on to what extent is WAHO considered a leader in advancing health issues and outcomes in the region. “Well, leader! Well, I don’t know. For me, it is in terms of complementarity, structures complement each other. I cannot say that, WAHO is ahead of everyone, but it is in fact a complementarity between the technical and financial partners within the framework of the implementation of activities in the field of health. Well, it's quite true that perhaps in a particular field, WAHO can be up front, that's not a problem, just like UNFPA as far as reproductive health is concerned. That is what I can say about leadership, it is a matter of complementarity” – Burkina Faso FGD _ Implementing Partner KI Majority of KIs interviewed both individually and in the FGDs, however, recognize WAHO as the legitimate leader in health in the ECOWAS region, having its official mandate from the ECOWAS heads of state. KIs stated this through the fact that WAHO have a legitimate mandate from ECOWAS’S Heads of states, support states members to harmonize health policy through governance and capacity building, sharing of updated guidelines and strategic documents with member countries. During the period of implementation of the CAPS project, WAHO played a key role in coordinating and harmonizing health policies among member states. Most of the KIs recognized that WAHO has a proven role in harmonizing health policies by developing, updating, and sharing guidelines and other strategic documents with member states. The KIs recognized that WAHO’s role in the harmonization and integration of ECOWAS countries to address health issues increased during the period of the project. “I think that from a regional point of view it is WAHO, because being regional means that we have to implement the good practices of a country, perhaps on a scale, we have to network the countries, we have to create a little. . . . We have to try to harmonize things; and in that a little I think there is no other institution if it is not WAHO. So clearly, I think WAHO is the current institution that although some efforts are yet to be made, WAHO is the institution that does this.“ – ECOWAS Stakeholder “WAHO is a leading regional institution bringing together member countries through coordination, capacity-building, and governance. Indeed, it adapts and harmonizes its recommendations for Côte d’Ivoire as a member country.” – Côte d’Ivoire National Stakeholder KI WAHO’s leadership, however, is curtailed by certain handicaps including the difficulty to always attract and maintain highly qualified staff to live in Bobo-Dioulasso, the organization’s governance model and the financial constraints. Indeed, WAHO is faced with poor mobilization of resources to support countries to better coordinate partners’ interventions at the country level. “It all boils down to funds. If only they had money. But they don’t have the money and they don’t have the face many challenges mobilizing resources. So, then their role is limited.” – Ghana National Stakeholder KI 12 | Page In all these instances, it is obvious that WAHO is playing an important role and has made some strides in terms of leadership strengthening in the last years. The word cloud generated from the different opinions received from the KIs highlights the following points where addressing WAHO’s leadership: Policies – Support – Funding – Assistance – Programs – Strategies – Meetings – Training – Partnership – Planning – Help – Capacity – Data – Information – Scaling - Best Practices Word cloud summary of different domains highlighting WAHO’s leadership Question 2: WAHO’s leadership in promoting and supporting the scaling up of good health practices Promotion and scaling up of Best Practices (BP) at the country level remains one of the areas where WAHO has achieved key results and this was unanimously recognized by interviewees. KIs stated their opinions through the emerging themes drawn from individual interview and FGD: BP practices sharing, regional BP forum, capacity building on BP documentation, BP documentation guide, BP committee establishment, collaboration for BP promotion, etc. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments WAHO’s leadership in promoting and supporting the scaling up of good health practices • Sharing information on good practices • Regional BP forum • Capacity building on BP documentation • BP documentation guide • BP committee establishment • Collaboration for BP promotion • Technical • Support • Capacity • Develop • Document • Platform • Meetings • Identify • Project • Policies • Help • Finance • Information • Committees • Program • System • Data • Experience • Strengthen • Developing and disseminating methodological tools for documenting good practices and building institutional capacity to use them • Creating a platform where countries meet to exchange their respective experiences on best practices • Providing continuous technical and financial support for countries in implementing their good practices • Harmonizing the scale-up of some good practices in the ECOWAS region, • Support the establishment of Committees PAGE for scaling up best practices Interviewees were unanimous in recognizing the important role WAHO is playing in promoting the scale-up of good practices by ECOWAS countries. It is interesting to note that many other partners have already joined WAHO's efforts to ensure synergy and efficiency. 13 | Page In fact, WAHO plays important and remarkable roles in the following areas: • Identifying and promoting the scale-up of best practices by ECOWAS countries; • Developing and disseminating methodological tools for documenting good practices and building institutional capacity to use them—WAHO supported countries to understand these tools and apply them systematically whenever they want to create or document a best practice; • Creating a platform where countries meet to exchange their respective experiences on best practices; • Providing continuous technical and financial support for countries in implementing their good practices; • Coordinating and working in synergy with other technical and financial partners (TFPs) in different countries for the scaling up of good practices; • Harmonizing the scale-up of some good practices in the ECOWAS region, which has provided a good experience in rationalizing the use of resources; “Yes, WAHO developed this Good Practice strategy WAHO also supervised and coached countries to identify and implement pilot actions, [and] encourages scaling up through technical and financial support (through the drafting of action plans).” – Côte d’Ivoire National Stakeholder KI “It must be noted that at the level WAHO plays a major role, I know that WAHO organizes workshops in relation with good practices, training of actors from different countries on how to develop a good practice. There are many themes that have been the subject of good practices.“ – Burkina Faso FGD National Stakeholder KIs These efforts need to be consolidated, especially by operationalizing the BP committee in each of the ECOWAS countries. The words cloud generated summarize the opinions of KIs opinions. Question 3: WAHO’s role in advocacy and communication efforts regarding the scaling up of good practices WAHO’s advocacy and communication efforts in scaling up good practices are widely recognized by countries. Statements of KIs are highlighted through the following emerging themes summarized here: Communication/ advocacy efforts to scale-up good practices, Advocacy in mobilizing resources; establishment of BP committee. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments WAHO’s role in advocacy and communication efforts regarding the scaling up of good practices • Communications activities • Communications materials • Advocacy for fund rising • Communication/ advocacy efforts to scale-up good practices. • Advocacy in mobilizing resources • Advocacy • Communication • Disseminate • Training • Focal Point • Mobilize • Partners • Website • Resources • Enable • Project • Data • Forum • Practices • WAHO initiated a network of communication managers in each of the 15 ECOWAS countries. • Develop communication and advertorial materials • Operationalize the Advocacy and Communication Plan that was developed as part of USAID’s Leadership Management in Governance (LMG) project • Establishment of focal points in each country’s to strengthen WAHO’s communication and advocacy efforts to scale-up good practices • Organize Advocacy meetings are organized at the national, sub-regional, and regional levels with some including health ministers, prime ministers, and TFPs for good practices. WAHO’s advocacy and communication effort since 2011 has been effective in mobilizing resources for scaling up good practices and supporting countries when responding to health emergencies such as the Ebola epidemic. It is interesting to note that the system is strong to identify inadequacies in communication and provide adequate response. 14 | Page Words cloud summarizing WAHO efforts in advocacy and communication These efforts are the result of various and intensive WAHO advocacy and communication activities carried out in the region. Furthermore, WAHO has an advocacy and communication plan that was developed as part of USAID’s Leadership Management in Governance (LMG) project but had remained dormant. Through the CAPS project, WAHO operationalized it by engaging a consultant to support the process. Thus, within the framework of the CAPS project, a network of communication managers from the 15 countries at the level of ministries of health, in addition to media representatives, was created, supported, and strengthened. Communications materials such as fact sheets, documentaries, and advertorials were developed to ensure that all activities carried out at the regional level were well advertised and disseminated. The advocacy for the establishment of Large-Scale Passage (PAGE) Steering Committees in some countries continues. Advocacy efforts have been made to ensure that committees can operate as institutional tools mechanisms for scaling up of best practices at the country level. The establishment of focal points in each country’s MOH has also helped to strengthen WAHO’s communication and advocacy efforts to scale-up good practices. “For the promotion of good practices, communication, and advocacy, WAHO has done a lot. In each country, they have a focal point for communication. This POC participate in different activities, especially activities at the regional level. Also, these communication representatives are organized in network. They share WAHO’s updates as they are informed of everything that is happening so they can in turn disseminate the information.” – Global KI For health leaders interviewed, WAHO’s advocacy and communication efforts since 2011 have been effective in mobilizing resources for scaling up good practices and supporting countries in responding to health emergencies such as the Ebola epidemic. Even though they cannot attribute the results of resource mobilization and other activities to WAHO’s advocacy efforts alone because other partners are also in the field, they claim to have witnessed or participated in some of these advocacy activities. “On advocacy and communication, I know just before the second-best practice [forum] in 2016, WAHO established the regional communications network and included journalists across the region as well as communication officers in the different ministries of health. And I know that was established, I am not (sure) how much progress has been made but that did happen. And I do know that in some countries WAHO has worked a little bit more closely with their communication officers and the ministry of health to make sure that when WAHO comes in country that their visit and whatever contribution they are bringing is more widely publicized. So those are the things I have on communication and advocacy.” – USAID/WA Some inaccuracies in communication and information dissemination on activities which exist sometime, and which extend to focal points that are supposed to be communication channels between WAHO and the various ministries 15 | Page of health have been identified and addressed. Thus, WAHO has already taken measures to strengthen these communication aspects for a better coordination of its activities. WAHO’s leadership in improving the supply chain and commodity security As part of the CAPS mandate, WAHO has helped strengthen countries’ capacities in commodity safety– security/supply chain management. Emerging themes from KIs interviews that state WAHO positive role in strengthening supply chain and commodity security: countries capacity for commodity security and supply chain management strengthened, technical assistance and harmonization of standard document, integration of health commodities in national HIS, etc. Under CAPS’s project, WAHO has provided support in terms of training, equipment, and technical assistance. They have also helped to harmonize standard documents in the field of pharmaceutical regulation and traditional medicine. Hence WAHO helped countries to develop guidelines and regulations, whether at the pharmacy level, the nursing level, or the level of human resource development in health. As part of supply chain strengthening, WAHO provided financial assistance for the purchase of contraceptive commodities and capacity-building in supply change management and commodity security. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments WAHO’s leadership in improving the supply chain and commodity security • Harmonizing policies • Strengthening pharmaceutical supply chain • Technical assistance • Infrastructure & equipment • Training • Equipment • Capacity building • Needs • Purchase • Buying • Storage • Contraceptive • Efforts were made to improve commodity safety￾security/supply chain management. • Help to harmonize standard documents on pharmaceutical regulation and traditional medicine • Develop guidelines and regulations • Provide support in terms of training, equipment, and technical assistance • Coordinate resources’ mobilization between IPs to carry out group purchases • Integrate health commodities data into DHIS2. WAHO is working in improving the supply chain management and commodity security. However, there is still room for improvement mainly regarding the standardization of legislation at the regional level in terms of procurement/import of health commodities. “Yes, in fact, at the last WAHO review we attended, there were some countries that benefited from capacity-building funds, and some benefited from procurement funds. Niger has opted for the capacity- building fund. I think it is on the basis of sovereignty that WAHO is strongly involved in the purchase of contraceptive products.” – Niger National Stakeholder KI WAHO also played an important role in strengthening supply chain management by integrating health commodities to national health information systems via extensive use of District Health Information Software 2 (DHIS2). Efforts were made to improve the supply chain and commodity security. Some interviewees felt that the efforts must be maintained and even intensified in this area, especially regarding standardization of regional legislation concerning importation of health products. WAHO’s role in promoting Public–Private Partnerships (PPP) WAHO plays a coordinating and advocacy role with donors for the financing of member countries’ activities. It facilitates meetings between donors and actors from member countries to help in getting funding opportunities. As part of the public–private partnership (PPP) program, WAHO organized the first forum on PPP in health in 2017 in Accra with the support of USAID to help countries to implement PPPs and also to set up a PPP unit within the ministries of health. Emerging themes stating KIs opinions on WAHO role in promoting PPP are the following: coordination and advocacy role, strategy guide for PPP promotion, PPF for an organized. 16 | Page Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments WAHO’s role in promoting Public– Private Partnerships (PPP) • Activities on PPP • Coordinating and advocacy role with donors • Training • Meetings between donors and actors • Collaboration with other partners • Resources • Funding • Health • Meetings • Support • Strategy • Information • Activities • System • Policy • Facilitate meetings between donors and member countries to capture funding opportunities • Develop a strategy document to help countries implement PPPs and to set up a PPP unit within the ministries of health. • WAHO organized the first forum on PPP in health • Helps countries to implement PPPs The performance achieved is a result of joined efforts between WAHO and other partners such as World Bank and the European Union. However, coordinating funding of different partners is not always easy as agendas may diverge. “WAHO’s role was to be able to organize the forum, because at that forum there were representatives of governments, private medical sector, private nonmedical sector, USAID partners, and others. So we have started implementing this document and I think we should ensure certain countries . . . identify initiatives that can be developed in the form of public–private partnerships, to also support countries, for example, in setting up units and departments within ministries of health that must deal with the issue of public–private partnerships.“ – Burkina Faso WAHO KI The performance in PPP development has not been a lonely venture of WAHO. Indeed, in the framework of PPPs, WAHO joined forces with other IPs and as such works in collaboration with other partners such as the World Bank, the European Union, etc. to try to mobilize resources. This has contributed to raising awareness that resource mobilization is not an isolated action but should rather be a group effort bringing together multiple stakeholders to fight for a common goal. However, the task of coordinating the funding of the various partners is not always easy because the agendas of IPs sometimes differ. “We have always had a strategy to involve all stakeholders [and] technical and financial partners in the implementation of our activities, but as I said earlier, people often have different agendas and they have approaches different from ours. It can happen that we succeed in doing so [or] it can happen that some want to do it alone, and so it is difficult. There is practically no regional value-added activity that WAHO carries out without involving partners in a coordination mechanism well thought out in advance on the implementation, and this is almost systematic.” – Burkina Faso WAHO KI WAHO’s role in strengthening the HIS and improving reporting To facilitate operationalization of the policy and strategy of the regional HIS, the CAPS project established a regional platform to provide information on health information indicators, including the creation of regional databases. The creation of databases in countries through DHIS2 was well acknowledged by KIs. This activity was made possible through the technical assistance provided by WAHO to countries. Emerging themes stating KIs opinions on WAHO role in strengthening HIS are summarized as following: regional HIS implementation, DHIS2 database, capacity building, improved reporting on surveillance, Data Quality Assessment, etc. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments WAHO’s role in strengthening the HIS and • Capacity building • Harmonize policies • DHIS2 platform • Information • Health • Database • DHIS2 • Operationalization of the policy and strategy of the regional HIS It’s important to mention that WAHO made real efforts in 17 | Page improving reporting • Data • Capacity • Integration • Surveillance • Support • System • Meeting • Building • Training • The CAPS project established a regional platform to provide information on health information indicators, including the creation of regional databases • Improvement of data quality and data analysis • Implementation of harmonized tools for data collection strengthening HIS at countries’ level by leveraging resources for other sources. However, despite the creation of regional databases, challenges remain in terms of country ownership and data completeness and interoperability of the various data systems in use in different countries. Through the CAPS project, the countries’ health information system has seen much improvement, according to some respondents. WAHO’s activities have improved not only data quality but also the implementation of harmonized tools for data collection as well as data analysis. Despite progress made in HIS strengthening, countries continue to face challenges in terms of completeness, timeliness of data reporting, and maintenance of essential components of HIS such as servers. Furthermore, the significant issue of interoperability is not yet fully solved, and the regional database raises concerns over countries’ ownership. Question 4: What are the areas or domains of WAHO’s capacity that have improved as a result of USAID/WA support? Emerging themes stating WAHO’s capacity has been improved through USAID support are summarized as following: leadership in coordinating and harmonizing health policy, convening power to bring together Ministries of Health, collaboration with donors, facilitate coordination at regional level. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings/results Comments What are the areas or domains of WAHO’s capacity that have improved as a result of USAID/WA support? • In health policy capacity implementing • In capacity building • In health action coordination • In harmonize policies • Political organization • Leadership • Management • Resources • Organization • Meeting • Capacity • Information • Partners • Communication • Health • Finding • Practices • Governance, awareness, and visibility of WAHO have improved in the sub-region as well as the recognition of the role of WAHO by TFPs • Leadership in promoting health policy in the region • Visibility and success of some WAHO’s activities increased as a result of establishment of focal points WAHO has achieved an undeniable leadership role in advancing health issues and outcomes in the region. Nevertheless, challenges remain and WAHO is still struggling to make unanimity among international partners and in some countries and legitimate its role as coordinating institution in the region. Appreciation of WAHO’s leadership in advancing health issues and outcomes in the region WAHO’s leadership in advancing health policy and outcomes in the West Africa region is well recognized by the majority of KIs both in the individual interviews and in the FGDs. Respondents appreciate WAHO’s leadership in health. Indeed, according to most of them, in recent years WAHO has positioned itself as a political organization on which partners can rely to advance their agendas. The proof is the increasing number of projects implemented by 18 | Page WAHO. Donors recognize WAHO’s political leadership, which has a comparative advantage in the sub-region that no one has and the legitimacy and credibility to be an ECOWAS body. They therefore collaborate with WAHO to facilitate the coordination of activities at the regional level and find it very useful. “WAHO's role in the region is very important, as it is the main technical and financial partner. It also facilitates the experiences sharing between different countries and intervenes in advocacy.....We have projects with WAHO as technical and financial partner and also does advocacy with partners for resource mobilization for DHIS2 and health mapping.... I think that we cannot evaluate WAHO only for its technical or financial support, but also we must not forget its contribution to the improvement of health systems – Cape Verde National KI Despite its role as a key player in health, there is need for WAHO to maintain the legitimate role of coordinating overall health strengthening activities in the region. The low representation in countries seems to limit WAHO’s effectiveness. WAHO owes this leadership to its ability to collaborate and convince other partners working in the health sector in the West African region in the areas of capacity-building, organization, health care delivery, and research. This collaboration makes WAHO more attractive to win more projects and therefore generate resources for the benefit of the ECOWAS region. WAHO’s areas of expertise/capacities that have improved with USAID support from 2012 till now Emerging themes stating WAHO’s capacity improved are the following: leadership in promoting health, communication and advocacy, HIS strengthening, and portfolio of activities increased. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings/results Comments WAHO’s areas of expertise/capacities that have improved with USAID support from 2012 till now • Leadership in promoting health policy in the region • Coordination of activities at the regional level • Contribution in strengthening the health information system in the region • Projects implementation • Support • System • Capacity • Training • Building • Strengthening • Management • Implementation • Health • Develop • Planning • Policy • Improved governance, awareness, and visibility: WAHO achieved more accountability and more transparency • WAHO’s overall contribution to strengthening the HIS in the region has increased. • Advocacy and communication skills have improved • WAHO has strengthened its resource mobilization capacity by convincing the donors WAHO grew up tremendously in the recent years in term of overall institutional capacity building and KIs (including WAHO staff) are convinced that USAID support played a big role The following are WAHO’s expertise/capacity-building areas that have improved with USAID support: • Governance, awareness, and visibility of WAHO have improved in the sub-region as well as the recognition of the role of WAHO by TFPs. Indeed, WAHO achieved more accountability and more transparency because donors’ procedures and requirements led to more rigorous overall management. • Visibility and success of some WAHO’s activities increased as a result of establishment of focal points in countries serving as liaisons between WAHO and Ministries of Health. • Leadership in promoting health policy in the region is indisputable because it has the convening power to bring together ministries of health in the region. Also, communication, project management, and M&E have improved. • WAHO’s portfolio of activities in terms of the number of programs implemented in the countries of the sub-region has increased. • WAHO’s contribution in strengthening the health information system in the region has tremendously increased. 19 | Page “I think that on the strengthening of the health information system it is still quite visible, since initially it was not easy to get the data from the countries. It took the setting up of this platform for me to think that this is one of the results.” – Burkina Faso WAHO KI • WAHO has strengthened its resource mobilization capacity by convincing the donors. • Advocacy and communication skills have improved communication in a broader sense, taking into account aspects of day-to-day operations, communication platforms, and information sharing up to travel arrangements. “Communication and health information have been strengthened, even at the advocacy level. This is because USAID has still helped WAHO to do a lot of advocacy at the level of other donors.” – Burkina Faso WAHO KI • WAHO’s overall contribution to strengthening the health information system in the region has increased. Current or potential challenges of WAHO leadership in supporting health policies in the countries The challenges mentioned by the respondents are multiple and affect all health sectors but also concern WAHO’s actions in implementing support activities to improve countries’ health systems. The challenges cover multiple areas, including institutional organization, capacity to support the process leading to achieving CAPS quality project outputs. Emerging themes that came out from KIs interviews about main challenges faced by WAHO are the following: insufficient human resources, lack of physical representation as seen in other organizations, insufficient resources mobilization. Evaluation Question Emerging/overarching theme from Thematic analysis Emerging/overarching theme from Word clouds Key findings Comments Current or potential challenges of WAHO leadership in supporting health policies in the countries • Institutional organization • Mobilization of resources • Problem of visibility • Internal capacity • Funding • Capacity • Visibility • Communication • Funding • Project • Policy • Needs • Support • Resources • Partners • Problems • Focal Point • Insufficient mobilization of resources for the implementation of health policies • Insufficient number and capacity of WAHO team members to cover all the 15 countries • Insufficient communication on WAHO activities • The low institutional visibility of WAHO to be able to fully assume its leadership role WAHO should intensify its advocacy activities at the highest political level. It could leverage on the influence of some of our technical and financial partners to mobilise enough financial resources from the Community Levy and also for the implementation of high impact interventions The following challenges were identified: • Insufficient mobilization of resources for the implementation of health policies • Poor performance in carrying out the missions assigned to WAHO due to several factors ranging from capacity to structural organization • Delay in responding to countries in terms of support and in the fight against epidemics such as Ebola • Ensuring the implementation of all the recommendations resulting from the meetings attended by the country • Insufficient number and capacity of WAHO team members to cover all the 15 countries 20 | Page • Insufficient communication on WAHO activities • The low institutional visibility of WAHO to be able to fully assume its leadership role • The lack of physical representation of WAHO in countries, as seen in other organizations with similar missions • The real involvement of WAHO in the scaling up of good practices. LESSONS LEARNED Remarkable achievements have been made under CAPS’s project implementation. During the period of the CAPS project implementation, WAHO played a key role in coordinating, promoting and harmonizing health policies among member states. Main achievements, under CAPS’s project, have been performed in the following areas: − Health policy harmonization: WAHO played an effective role in harmonizing health policies by developing, updating, and sharing guidelines and other strategic documents with member states and its role in the harmonization and integration of ECOWAS countries to address health issues increased during the period of the project with an increased number of resolutions validated by the ECOWAS Assembly of Health Ministers. − Best Practices scaling up: Promotion and scaling up of Best Practices (BP) at the country level remains one of the main areas where WAHO has achieved key results. WAHO played important and remarkable roles in: • Identifying and promoting the scale-up of best practices by ECOWAS countries; • Developing and disseminating methodological tools for documenting good practices and building institutional capacity to use them—WAHO supported countries to understand these tools and apply them systematically whenever they want to create or document a best practice; • Creating a platform where countries meet to exchange their respective experiences on best practices; • Providing continuous technical and financial support for countries in implementing their good practices; • Coordinating and working in synergy with other technical and financial partners (TFPs) in different countries for the scaling up of good practices; • Harmonizing the scale-up of some good practices in the ECOWAS region, which has provided a good experience in rationalizing the use of resources. − Advocacy and communication: Through the CAPS project, WAHO operationalized the advocacy and communication plan that was developed as part of USAID’s Leadership Management in Governance (LMG). Moreover, a network of communication managers from the 15 countries at the level of ministries of health, in addition to media representatives, has been created, supported, and strengthened. Communications materials such as fact sheets, documentaries, and advertorials were developed to ensure that all activities carried out at the regional level were well advertised and disseminated. These efforts contributed to reinforce WAHO visibility within the region and to facilitate collaboration with Technical and Financial Partners at countries and regional level. − Supply chain and commodity security: As part of the CAPS mandate, WAHO has helped strengthen countries’ capacities in commodity safety–security/supply chain management. WAHO has provided support in terms of training, equipment, and technical assistance. They have also helped to harmonize standard documents in the field of pharmaceutical regulation and traditional medicine. Hence WAHO helped countries to develop guidelines and regulations, whether at the pharmacy level, the nursing level, or the level of human resource development in health. As part of supply chain strengthening, WAHO provided financial assistance for the purchase of contraceptive commodities and capacity-building in supply change management and commodity security. In addition, WAHO played an important role in strengthening supply chain management by integrating health commodities to national health information systems via extensive use of District Health Information Software 2 (DHIS2). 21 | Page − Promoting Public–Private Partnerships (PPP): As part of the public–private partnership (PPP) program, WAHO organized the first forum on PPP in health in 2017 in Accra to help countries to implement PPPs and also to set up a PPP unit within the ministries of health. Main actions taken are the following: • Facilitating meetings between donors and member countries to capture funding opportunities • Developing a strategy document to help countries implement PPPs and to set up a PPP unit within the ministries of health. • Organized the first forum on PPP in health − WAHO’s role in strengthening the HIS and improving reporting: through the CAPS project, the countries’ health information system has seen much improvement. A regional platform to provide information on health information indicators, including the creation of regional databases has been established. WAHO’s technical assistance through CAPS project contributed to improve not only data quality but also the implementation of harmonized tools for data collection as well as data analysis. In summary, through the CAPS project, WAHO’s leadership in advancing health policy and outcomes in the West Africa region has been strengthened. WAHO has positioned itself as a political organization on which partners can rely to advance their agendas. Donors recognize this WAHO’s political leadership, which has a comparative advantage in the sub-region and engage effective collaboration with WAHO to facilitate the coordination of activities at the country and regional level and find it very useful. However, despite progress made, efforts need to be maintained to address main challenges encountered during the life of the project. These challenges are, among others: − Insufficiency of human to address timely countries’ needs; − Insufficient visibility and representation and country ‘’a single man representation’’ level limiting its role and reach; − Timely communication between WAHO and countries; − Resources mobilization to better address countries’ needs; − Insufficiency in the follow up and implementation of recommendations from regional meetings. LESSONS LEARNED FROM THE EVALUATION AND LIMITS Conducting a large evaluation such as this one requires dexterity and a good working environment. Lessons learned can be summarized as follows: • Conducting such an evaluation in 15 countries with different data collectors with different experience and in three different languages requires much organization and communication. • Coordinating several teams of data collectors in 15 different countries simultaneously is one of the most challenging aspects, essentially for communication. • Support of some WAHO staff and Liaison Officers in-country in facilitating the data collection process (including setting appointments with KIs as well as logistics for the FGDs) was invaluable. • Willingness and engagement of RGs who were keen and proactive during the data collection in their respective countries was essential for effective data collection. • Leveraging personal and professional networks was a good asset to reach some KIs and organize FGDs. • Initial time allocation for data collection per country was short, because it didn’t take into account the availability of KIs and their absence. More time should be allocated for data collection when planning for similar evaluations in the future. The evaluation has some limitations that could have affected the data collection process and the findings, including the following: • Lack of completeness of programmatic data, often due to delay in reporting project’s activities was a challenge for monitoring indicators of the dashboard and triangulating programmatic data with information collected from individual interviews and FGD; • Stability and availability of WAHO liaison officer: some LOs were new and hesitated to participate in the evaluation or were too busy to facilitate the data collection process; • Difficulty in organizing FGD with KIs in health in some countries: the evaluation team faced some challenges in mobilizing key partners in health for an FGD due to their busy schedules; • Reluctance of some KIs despite the email sent by USAID/WA: they claimed they were waiting for an official 22 | Page introductory letter from WAHO before they could participate in the interview; • KIs were not always well knowledgeable about CAPS technical components; • Difficulty in undertaking data collection activities in Lusophone countries (Cabo Verde and Guinea Bissau) due to language barriers—the evaluation team found it difficult to find someone who spoke Portuguese and could transcribe the interviews into French. Typical other limitations associated with qualitative studies were anticipated and addressed during the sampling strategy. For this study the FGDs were conducted to complement the KIIs (in the sample). The sampling method for KIIs (and participants to FGDs) was designed to ensure the maximum of information is collected from a large variety of beneficiaries (variety of countries, variety of levels of interventions, etc.), taking into consideration the minimum sample size required to reach the “saturation". Once the saturation is obtained with a variety of KIs, the findings are relevant and give a fair indication of the general assessment of the situation. In each country (and for the 15 countries), WAHO focal points were “contacted” as "entry point” as they know more than anybody else the most relevant KIs for interviews. In addition, before data collection, E4D validated the list of KIs with WAHO HO and USAID/WA. The fact that a few WAHO focal points were new did not have any impact on the Study. Finally, all due measures were put in place by E4D to ensure confidentiality and anonymity and therefore to reinforce the good quality of information collected. It is our understanding that participants responded honestly and shared their feelings. Cultural or social implications that frame how the key informants interact with the researcher and the information shared were also addressed by E4D. Indeed, most of the KIs are high-level staffs. E4D ensured that the interviews were conducted by top level professional familiar with the exercise with large experience on how to adjust the tone, the timing and attitude to ensure that the participants are “comfortable”. Therefore, the listed limitations are minor and could not alter the findings in any way. Indeed, the sample size was large enough to overcome the shortcomings identified. Therefore, the results of this evaluation are valid and appropriately meet the set objectives. The listed limitations are minor and could not alter the findings in any way. Indeed, the sample size was large enough to overcome the shortcomings identified. Therefore, the results of this evaluation are valid and appropriately meet the set objectives. KEY RECOMMENDATIONS TO WAHO ON LEADERSHIP DEVELOPMENT Under the CAPS‘s project, WAHO’s role and leadership have been strengthened through the multiplicity and diversity of WAHO’s activities/interventions and partnerships. WAHO has been acknowledged by most of the national stakeholders and partners at the country and regional levels as the legitimate institution in charge of coordinating health policies in the West Africa sub-region. WAHO played a significant role in coordinating, promoting and harmonizing health policies among member states. This leadership has been effective with an increased number of resolutions validated by the ECOWAS Assembly of Health Ministers and growing partnership with the regional body to address health issues such as Ebola outbreak and/or collaborating with regional bodies such as the regional Animal Health Center to develop a regional One Health program. To reinforce this leadership and consolidate the achievements, efforts must, however, be made to overcome some challenges that could mitigate this leadership. 1. To what extent is WAHO considered a leader in advancing health issues and outcomes in the region, especially in the area of policy harmonization and advocacy, in promoting public-private partnerships (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key IPs in the region)? • Carry out institutional strengthening in general through greater country contribution to enable WAHO to be autonomous and be able to play its role. • Strengthen the quantity and quality of WAHO staff in order to: 1) better assist countries in carrying out their activities and respond timely to their requests; 2) establish an effective WAHO representation (bureau) in each country with the necessary and adequate staff resources for the implementation, monitoring and 23 | Page coordination of these activities in the countries. Making use of the in-country professionals will be helpful to carry out activities at the country level. • Strengthen communication and advocacy activities to :1) increase greater visibility of WAHO; and 2) for resources mobilization; • Focus on a few specific and priority areas or problems common to all countries to enable WAHO’s support to be more effective and useful. • Set up a system to follow up on resolutions adopted during various meetings in order to ensure their effective implementation by countries. WAHO should increase advocacy to get countries’ support in the implementation of these resolutions. • Encourage countries to integrate WAHO’s activities into their workplans to promote ownership instead of standing alone as parallel activities. 2. How has WAHO led and supported the scale-up of best practices in the region (e.g., mapping of current high￾impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale-up of interventions, health information systems integration, and capacity strengthening)? Promoting and scaling up Best Practices (BP) remains one of the main areas where WAHO has achieved key results. Among the key achievements are: 1) development and dissemination of methodological tools for documenting good practices and building institutional capacity to use them; 2) creation of regional platforms where countries can meet and exchange their respective experiences on BP; providing technical and financial support for countries to implement their good practices. Over the life of the project, WAHO strengthened the countries’ capacities in commodity safety-security chain management by helping countries to harmonize standard documents in the field of pharmaceutical regulation and by helping countries to integrate health commodities management into national health information system via DHIS2 software. In addition, WAHO helped countries to develop a common strategy for the implementation of Public-Private Partnership (PPP). Moreover, WAHO provided technical assistance through putting in place a regional platform on Health Information System (HIS). In summary, through CAPS project, WAHO made remarkable achievements in the area of BP sharing and scaling up. Therefore, to consolidate these achievements and to enable WAHO to fulfil its role, the following actions need to be taken: • Ensure in-country and wider dissemination of approved, adopted and accepted BP. • Ensure that WAHO’s in-country POC are fully aware of the ongoing activities and ensure local coordination of implementation. • WAHO should continue strengthening its resource mobilization capacities by seeking more financial partners to support countries to implement, document and evaluate their BP; • Ensure the implementation of all the recommendations resulting from the meetings attended by the countries, to ensure that each focal point can have a dashboard to monitor the recommendations and to ensure their implementation from BP forum; • WAHO should help countries to document experiences of scale-up of good practices and share them among countries and also get involved more in scaling up activities on the ground. 3. What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale-up; national policy changes for FP and HIV/AIDS programs; improved commodity security; public and private partnerships; and health information systems to increase investments in health and improve reporting? Through the CAPS project, concrete progress has been made in the area of advocacy and communication. WAHO operationalized the advocacy and communication plan that was developed as part of USAID’s Leadership Management in Governance (LMG). Furthermore, a network of communication managers from the 15 countries at the level of the ministries of health, in addition to media representatives, has been created, supported, and strengthened. Communications materials such as fact sheets, documentaries, and advertorials were developed to ensure that all activities carried out at the regional level were well advertised and disseminated. In spite of this achievement, actions need to be taken to improve WAHO communication strategy for better visibility and more fundraising. The following actions have been recommended: 24 | Page • Strengthen WAHO’s communication system to significantly improve the visibility and leadership development. • Reinforce advocacy and communications activities to increase WAHO’s funding for health services. Like every other organization, WAHO would have to rely on member states or rely on bilateral donors to carry out projects and it has the strength and opportunity to do that. WAHO could also support countries in their resource mobilization effort with external donors to help countries scale-up good practices. • As a regional body, WAHO has to put forward a strong multi-country proposal to funders, one that is more effective than having small contributions. • Strengthen countries’ capacities to develop PPPs. In this regard, WAHO could assist countries in identifying areas of intervention or private individuals with whom they can collaborate and help them set up projects to present to private individuals. • Continue efforts to strengthen communication and advocacy to raise awareness of WAHO’s activities and increase success. • Develop communication activities for a wider dissemination of resolutions and recommendations adopted by Assembly of Health Ministers (AHM); • Pursue advocacy and communications activities to engage countries in implementing resolutions and recommendations adopted by the AHM. CONCLUSION WAHO is considered a legitimate leader in the ECOWAS region as far as health is concerned and it fully deserves that recognition. It is in the best position to fill that role given its proximity to countries and its ability to communicate and work within countries. It is unique, being a regional body that is owned and operated by the region. It has the power to convene and engage Ministers of Health or people around its activities such as the Best Practices Forum and other policy implementations. It has leveraged that power to develop some strong partnerships and works closely with some IPs and private sector actors. WAHO achieved much during the past five years by fostering its own institutional capacity-building (governance, communication and advocacy, PPP) and being able to increase technical assistance and other support to promote more partnerships as well as the harmonization of health policies in ECOWAS countries. This means that other organizations working in the sub-region strongly recognize WAHO’s leadership. This would not have been possible without the CAPS project, which is acknowledged by all to be the driving force of all these achievements. All countries expressed need capacity-building, maintenance of infrastructure and equipment, health care management, and epidemics control. Diversities among countries should also be noted as they have different levels of health system development demanding for adequate support based on thorough needs assessments. Therefore, WAHO must continue to be very active in relation to activities with the 15 countries, which will enable it to improve according to the realities of each country. It must continue to work closely with member states to address their needs and consider the private sector. The good practices identification and scale-up in the regions could be considered as the best output of the CAPS project and should be maintained with continuous support to countries for dissemination of good practices. The performance evaluation showed that at the sub-regional level, WAHO is very effective and well-structured in terms of standards and policies as well as supporting countries’ health systems at various levels going from organization to resource mobilization and governance. WAHO is an important tool, and its leadership has evolved tremendously, which is certainly as a result of USAID funding and other partners’ support. WAHO needs to consolidate its role in helping countries to better coordinate and leverage partners’ interventions at the country level. In addition, efforts need to be made for countries to engage in fundraising activities to support 25 | Page health policy implementation. Moreover, to increase its visibility and better address countries’ needs, WAHO has to reinforce its staffing and develop branding of its projects. 26 | Page INTRODUCTION On February 3, 2015, the USAID/West Africa (USAID/WA) Regional Health Office awarded the West African Health Organization (WAHO) a four-year cooperative agreement to implement the Leadership Capacity Strengthening (CAPS) project. The purpose of the project is to strengthen the capacity of WAHO, as a leader in Health in West Africa, to be the lead health institution in the region for policy harmonization and advocacy; promoting public–private partnerships for health; and identifying, sharing, and supporting the scale-up of best practices in health. The purpose of the project would contribute directly to USAID/West Africa’s Regional Development Cooperation Strategy’s Intermediate Result 3.2 – Enabling Environment at the Regional and National Level Strengthened. The activity has a two-pronged approach: (1) direct funding to leverage existing systems and resources, and (2) providing technical assistance through direct and frequent engagement. Table 1. Brief description of WAHO/CAPS Activity/Project Name Leadership Capacity Strengthening (CAPS) Implementer West African Health Organization Cooperative Agreement/Contract # AID-624-A-15-0000 Total Estimated Award of the Evaluated Project/ Activity (TEA) $ 7,434,000 Life of Project/Activity February 2015 – September 2020 Active Geographic Region 15 Economic Community of West African States Development Objective(s) (DOs) Utilization of Quality Health Services Increased through West African Partners USAID Office USAID/West Africa Regional Health Office Description of the Problem, Development of Hypotheses, and Theory of Change Over 340 million people live in the West Africa region, a landmark larger than the United States and Western Europe combined. Health indicators for this region have lagged behind other African regions and globally. Fertility rate is higher than the global average (5.50 vs. 2.58 children per woman) and close to one in three women in West Africa have an unmet need for family planning, meaning they express a desire to limit or space births but do not use any form of modern contraception. Maternal mortality ratios in the West Africa region are exceedingly high compared to the central Africa region, with 685 and 597 maternal deaths per 100,000 live births respectively, indicating a gap of around 100 points. Child mortality rates are equally staggering with the mortality rate for children under the age of 5 in almost all countries in the region at 100 deaths or higher per 1,000 live births. This means that 10 percent or more of children in West Africa die before the age of 5. HIV/AIDS prevalence in West Africa has remained low compared to other regions at approximately 2 percent or less in nine countries. However, prevalence remains high in some countries such as Côte d’Ivoire (3.7 percent), Nigeria (4.1 percent), and Togo (3.5 percent) and concentrated epidemics among most key populations such as female sex workers (FSWs) and Men who have Sex with Men (MSM). Any attempt to control the trend of the HIV epidemic in this region should include an aggressive program to scale-up access and use of HIV prevention and care services by key populations. In addition, West Africa also has one of the highest urbanization rates in the world with 42 percent of the population living in urban or peri-urban areas. In addition to poor access to health services, urban areas are often overcrowded, leading to poor sanitation and limited access to clean water, further exacerbating health issues. Finally, there are an estimated 104 million young people ages 10–24 in West Africa, making up 32 percent of the region’s total population. Addressing the health and well-being among this population is critical to achieving development goals in West Africa. At the regional level, it is imperative that WAHO be able to act as the lead institution in the health sector in West Africa to address these challenges. USAID/West Africa’s support to WAHO through the CAPS activity enabled 27 | Page WAHO to carry out its political mandate, specifically in the areas of implementation of resolutions and other policy decisions, facilitation of scale-up of high-impact health practices, conducting advocacy work, and harmonizing health information systems at the national level in ECOWAS member countries. The theory of change for the CAPS activity is as follows: 1. If WAHO’s support for ECOWAS member countries to scale-up evidenced-based practices is increased, 2. if WAHOs’ advocacy and communications strategies are operationalized, 3. if WAHO’s role as a regional donor and public–private partnerships are expanded, and 4. if WAHO’s regional health information systems policy is operationalized, 5. then WAHO’s leadership capacity will be strengthened. Results Framework The results framework (schematic of theory of change) for activities carried out by CAPS is shown in the diagram below. WAHO identified the following assumptions during the process of developing the logical framework: • ECOWAS states commit to piloting best practices. • Dissemination platforms continue to exist and accept new materials and products. • Targeted regional partnerships continue to exist, or new ones are formed. • Regional HMIS policy and regional HMIS data warehouse is financed by the Leadership, Management, and Governance (LMG) project (USAID/West Africa), the WARDS project (World Bank), and WAHO/ECOWAS. Figure 1. CAPS Project results framework 28 | Page EVALUATION PURPOSE AND PERIOD The purpose of the performance evaluation for the West African Health Organization (WAHO) Leadership Capacity Strengthening (CAPS) activity is to assess whether it has achieved its objectives. The USAID/WA regional health office is interested in knowing whether expected results occurred, to what extent selected high-impact practices are being advanced, and document CAPS’ successes and challenges. This evaluation serves as an end-of-activity performance evaluation. The evaluation focused on the following achievements of CAPS/WAHO: • WAHO’s support to ECOWAS member countries to scale-up evidenced-based practices is increased. • WAHOs’ advocacy and communications strategies are operationalized. • WAHO’s role as a regional donor and public-private partnerships are expanded. • WAHO’s regional health information systems policy is operationalized. • WAHO’s leadership capacity is strengthened. The period of performance to be evaluated is February 2015 to March 2019. To ensure that data were collected before the activity began its final stage, the team conducted field data collection during the period of June–August 2019. The target audiences for the CAPS performance evaluation are USAID/WA Front Office, WAHO, USAID/WA Regional Health Office (RHO), USAID bilateral missions in the region, representatives from the Ministry of Health, and international partners/donors (WHO, UNICEF, UNFPA) in each of the 15 ECOWAS countries. EVALUATION QUESTIONS The proposed evaluation questions in order of priority by USAID/WA are as follows. As a result of the project: • To what extent is WAHO considered a leader in advancing health issues and outcomes in the region, especially in the area of policy harmonization and advocacy, in promoting public-private partnerships (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key IPs in the region)? • How has WAHO led and supported the scale-up of best practices in the region (e.g., mapping of current high-impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale-up of interventions, health information systems integration, and capacity strengthening)? • What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale￾up; national policy changes for FP and HIV/AIDS programs; improved commodity security; public and private partnerships; and health information systems to increase investments in health and improve reporting? • What areas or domains of WAHO’s capacity (within institutional and leadership capacity) have improved as a result of USAID/West Africa’s support from February 2015 to March 2019 (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key IPs in the region)? EVALUATION DESIGN AND METHODOLOGY The CAPS Evaluation is particularly complex because it involves 15 countries with three different languages (French, English, and Portuguese) and key informants from various levels, positions, and countries (from the governments: policy/decision-makers and technical staff members, partners, and other implementing agencies). Conducting such an evaluation requires multiple languages and technical skills to complete the interviews, facilitate the FGDs, ensure the correct transcriptions and harmonization of responses, and finally to analyze the data and develop the report. Anticipating these challenges, the evaluation team selected appropriate staff, aware of the local context, to 29 | Page coordinate the data collection activities. In countries where E4D has Recipient Groups, members of these institutions, who previously benefited from E4D training in research and evaluation, facilitated the data collection preparation and conducted all the interviews and FGDs in their respective countries. In addition, prior to starting the data collection activities, all the interviewers were oriented to the evaluation purpose and trained to the management of data collection tools by E4D team. To conduct the evaluation, E4D closely worked with USAID/WA, throughout the process, from development of the Scope of Work (SOW) and the Inception Report to the team planning to the final report. Geographic Focus For this evaluation, the E4D team collected data throughout the 15 ECOWAS member countries in West Africa that benefited from CAPS activities, focusing on stakeholders associated with WAHO (including the ministries of health in all countries and representatives of key partners including ECOWAS, WHO, Global Fund, Palladium, USAID missions, and their implementing partners and the UN system). The table below shows the geographical focus of this performance evaluation. Francophone Anglophone Lusophone Benin Burkina Faso Côte d’Ivoire Guinea Mali Niger Senegal Togo Gambia Ghana Liberia Nigeria Sierra Leone Cape Verde Guinea￾Bissau Figure 2. West Africa region map Evaluation Design This performance evaluation is a cross-sectional study using mixed methods: qualitative, quantitative, primary, and secondary data as per the USAID performance evaluation definition. It covers the whole WAHO’s leadership capacity-building activities through the implementation of the CAPS project. The quantitative component consisted of a desk review of the CAPS project’s key documents: work plans, quarterly and annual reports, dashboard results, Monitoring & Evaluation and Learning (MEL) plan and periodic performance reports. The desk review served to document (1) main activities implemented, completed, or initiated; challenges encountered; and actions undertaken to overcome challenges; (2) the methodology for the qualitative component including designing the data collection tools; and (3) triangulation of the results drawn from Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs)/ Consultative Meetings (CMs). The qualitative component consisted of conducting two different categories of interviews: (1) individual interviews with key informants targeting national stakeholders (MOH technical staff/managers) and representatives of IPs working in the field of FP/MCH, HIV/AIDS, HIS from each of the 15 ECOWAS and connected to the CAPS Project, and (2) FGDs/CMs organized with key stakeholders in six countries (Ghana, Nigeria, Niger, Burkina Faso, Togo, and Côte d’Ivoire) where E4D had already developed a network of Recipients Groups (RGs) whose members benefited previously from capacity-building training in research and evaluation from E4D. The individual interviews with Key Informants (KIs) involved the following: 30 | Page • At the country level: key MOH leaders (Maternal and Child Health/Family Planning (MCH/FP), HIV/AIDS programs managers, Health Information Systems (HIS) managers, Best Practices (BP) scale-up committees) and in-country representatives of WAHO, USAID missions, and key donors and IPs such as UNFPA, The Global Fund, and USAID in-country collaborative agencies such as Measure Evaluation, RTI, etc. • Global level: representatives of World Health Organization Headquarters (such as IBP) and other international partner organizations working in the areas connected to CAPS activities) The Focus Group Discussions/Consultative Meetings involved two different profiles: • National stakeholders (MCH/FP program managers, supply chain managers, HIS managers, etc. • Key IPs (in-country representatives of international partner organizations) Sampling The CAPS performance Evaluation covered all 15 ECOWAS countries. This is an exhaustive sampling of the countries. The target participants for the CAPS performance evaluation were representatives from USAID/West Africa Front Office, WAHO, USAID/West Africa Regional Health Office (RHO), USAID bilateral missions in the region, representatives from the Ministry of Health, and international partners/donors (WHO, UNICEF, UNFPA) in each of the 15 ECOWAS countries. The evaluation included key informants purposely selected from the list of WAHO officials and focal points in the various ministries of health of the ECOWAS countries, specialists and managers of the national health information system (HIS) and implementing partners (in-country representatives of international partner organizations) involved in strengthening national health policy. The KIs work in various levels of the ministries of health such as Maternal and Child Health Directorates, Family Planning and HIV/AIDS Program Management units, Health Information System Units, supply chain and pharmaceutical commodities units, and documentation and public relations units. Members of the ECOWAS Commission based in Abuja, Nigeria, and representatives of WHO, other international partner organizations, and WAHO/CAPS staff based in Bobo￾Dioulasso were also included in the sample. In addition, selected key informants from WHO headquarters who directly coordinate with the International Best Practices units in Geneva were also included in the sample. The list of the KIs to be interviewed in each country has been consolidated in coordination with WAHO management team, USAID/WA office, and in-country WAHO liaison officer. Selection criteria were based on their involvement in health program management and connection to CAPS project activities. Table 2 below gives the results of the sampling by country and the number of interviews completed. Table 2. Sampling per country for individual interviews Profile of Respondents Number of Key informants Invited Numbers of Interviews Performed Global Key Informants 9 10 ECOWAS Commission 3 4 Burkina Faso (WAHO and CAPS management/staff) 6 6 ECOWAS Countries (national stakeholders and IP) 57 55 Total 75 75 E4D conducted 75 individual interviews with KIs and 12 FGDs/CMs. The rationale guiding sampling required conducting four KIIs per country distributed as follows: two officials from the MOH directly involved at the country level in the CAPS activities, one in-country WAHO representative, and one IP representative. The structure of the sample was maintained across the 15 ECOWAS countries. In Burkina Faso, where WAHO is based, E4D conducted six interviews targeting WAHO’s leadership and management: the CAPS Project Manager under WAHO, WAHO’s Monitoring and Evaluation Specialist, WAHO’s Capacity-Building Specialist, and WAHO’s Communication Specialist. In addition, E4D conducted 12 FDGs/CMs with national stakeholders from the MOH and with key IPs. These FGDs/CMs were conducted with the MOH staffs in each of the following countries: Burkina Faso, Côte d’Ivoire, Ghana, Guinea, Niger, Nigeria, and Togo and five were completed with IPs in Burkina Faso, Côte d’Ivoire, Ghana, Guinea, and Nigeria. The sampling for the FGD/CM was based on three main criteria: 31 | Page 1. Covering approximately half of the ECOWAS countries (7/15 countries) 2. Maintaining a mix and representation between Anglophone and Francophone countries (five Francophone and two Anglophone) 3. Sampling the countries where E4D has representatives (Recipients Groups or subcontractor) knowing that Burkina Faso (as the host country of WAHO) will be pre-selected Each FGD/CM gathered four to six participants; 54 persons attended the FGDs/CMs. Data collection E4D collected the CAPS routine reports and conducted a full desk review during the period from January to July 2019. The KIIs were conducted from June to August in all 15 countries in English, French, and Portuguese. Data were collected through in-depth interviews with KIs using different semi-structured interview guides for the different respondent groups: • Formative Key Informant Interview Guide – National Stakeholders/Implementing Partners • Formative Key Informant Interview Guide – WAHO Representatives • Focus Group Discussions/Consultative Meeting Guide These guides are mainly structured on the following points: • Background information • Achievements on WAHO’s Leadership Capacity Strengthening • Achievements on WAHO’s capacity to support Best Practices scaling up, communication, and advocacy strategies in the region • Achievements on WAHO’s capacity to improve Public-Private Partnership • Achievements on WAHO’s capacity to address clients’ needs • Challenges encountered during the project implementation • Recommendations for the future Prior to data collection with KIs, information gathered from the desk review served to inform the elaboration of interview guides and the methodology. In Togo, Niger, and Côte d’Ivoire, E4D recruited and trained data collectors from the Recipient Groups (RGs). These RGs included Cabinet de Recherche et d’Évaluation (CERA) in Togo, Centre de Recherche Médical et Sanitaire) in Niger, and Association de Soutien à l’Auto-Promotion Sanitaire et Urbaine and National Institute of Public Health (INSP) in Côte d’Ivoire. Pilot data collection was conducted in Accra, Ghana. The findings from the pilot helped E4D to fine-tune the language used in the Interview Guides including “how to introduce the questions.” Both face-to-face and remote data collection methods were used. Face-to-face interviews were conducted with KIs who were available during the presence of the evaluation team, and remote interviews were conducted when the KIs were not physically available during the presence of the evaluation team in the country. In countries where the evaluation team was not present, remote interviews were systematically done. Interviews were recorded and notes taken during the whole process after obtaining prior verbal consent from each interviewee. Data quality assurance Data collection strategy and methods were maintained across the countries to minimize inconsistencies. To ensure consistency and good quality of data collected, E4D followed up with the data collectors daily to ensure that the procedures were mastered to produce high-quality data. In countries where data were collected by RGs (Togo, Benin, Côte d’Ivoire, and Niger), the evaluation team established a mechanism of daily supervision of data collectors by phone calls and systematic feedback. This daily supervision allowed the evaluation team to monitor the progress made, discuss challenges, propose solutions, and harmonize the approach across countries. In addition, E4D undertook a follow-up of saturation management through periodic reporting. The level of saturation was reached after the completion of about 90 percent of the KI interviews. Therefore, to ensure a full coverage of the 15 ECOWAS countries, the evaluation team decided to complete all the interviews. The transcriptions started shortly after completion of the data collection. Audio recordings and transcriptions were systematically submitted via email, Dropbox, and Google Drive and backups were made at the E4D office in Accra. Before starting the analysis, the E4D 32 | Page team ensured a systematic review of audio recordings and transcriptions to check their conformity. Data quality assurance was systematically done by E4D staff based in Accra in coordination with the evaluation lead. Data analysis Two main sources of data (programmatic data from desk review and qualitative data from KIs interviews) were used for this performance evaluation and analyzed. The evaluation team used quantitative and qualitative methods. Quantitative data analysis Prior to the data collection starting, the evaluation team conducted a desk review. This desk review consisted of collecting data through both quantitative and qualitative measures. Quantitative data consisted to draw information from project documents (protocol of the CAPS project, inception report, project routine reports including annual and quarterly reports, work plans, associated Performance Monitoring Plan (PMP), Plan, dashboard results, etc. from both the country-level and CAPS management project). The information/data were used to describe the background of the project, main objectives as well as proposed and implemented interventions of the CAPS project. In addition, these programmatic data, mainly the results dashboard indicators, have been used to document main achievements and activities completed or initiated under each IR and sub IR, main challenges encountered, and actions taken to overcome these challenges. The evaluation team also used these programmatic data to understand the institutional organization and mandate of the CAPS project. Moreover, to have a better understanding of these programmatic data, the evaluation team triangulated this quantitative information with those drawn from individual Key Informant Interviews (KIIs) and Focus Group Discussions. The evaluation team combined these quantitative and qualitative information to generate tables describing the level of achievements of activities and dashboard indicators defined for the CAPS’s project. This combined approach allowed measuring and documenting changes in key outputs and outcomes over the evaluation period. Qualitative data analysis Two main sources of qualitative data have been used. The first included individual interviews with Key informants (both with national stakeholders and key implementing partner/cooperation agencies supporting the Ministry of Health (MoH) in its health policy implementation) and the second consisted of Focus Group Discussion (FGD) with key informants (both from national stakeholders and Implementing Partners (IPs). The qualitative data were analyzed by focusing mostly on the manifest content presupposing that statements in the individual interviews and in the FGDs are complete units from which emerging themes were identified. Data were coded by assigning the same code to the same or similar unit of meaning. The codes were generated by moving through the data and applying a mix of deductive and inductive coding. Principal themes and subthemes were identified and evidence that support or denies each theme and subtheme and links between them were sought. The FGDs data analysis has been conducted following the same approach without making difference between FGDs for national stakeholders and those conducted with the IPs, but final analysis helped report the opinions and ideas on issues covered according to national stakeholders or IPs. To ensure standardized coding of transcripts, the interviews were analyzed using a thematic grid developed by randomly selecting three interviews and identifying initial themes. The analysis grid was adjusted as additional pertinent topics emerged. Data were analyzed with Nvivo 12 Pro and word clouds were created for the CAPS model of intervention and WAHO’s oversight of CAPS. Content analysis was used to emphasize some key and relevant information. The transcriptions in French and Portuguese were translated into English, and E4D ensured consistency with the original transcription by listening recordings and reading the transcription of a sample. Specifically, the data analysis was conducted using the following procedures for addressing each evaluation question: • Develop the grid (code list) through an initial identification of themes related to evaluation questions in three randomly selected interviews. • Create a word cloud for two topics: the CAPS model of intervention and performance model and WAHO’s oversight of CAPS. • Expand the topics in the analysis grid (code list) as additional themes emerge. • Use verbatim sentences as the unit of analysis. • Group results by areas of interest (evaluation question). 33 | Page • Identify different positions in relation to each important topic. • Summarize each position, assessing its strength or degree of importance. • Report information from FGD with national stakeholders and implementing partners separately. For example, for evaluation question #2 related to WAHO’s role to Best Practices (BP) scaling up, emerging themes drawn were: BP practices sharing, regional BP forum, capacity building on BP documentation, BP documentation guide, BP committee establishment, collaboration for BP promotion, etc, Verbatim and word cloud have been created to address each evaluation question. The table below summarizes the global approach used to analyze and address each evaluation question. 34 | Page Summary the approach used to analyze and address each evaluation question. Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. To what extent is WAHO considered a leader in advancing health issues and outcomes in the region? (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region); PMP, Annual Reports, Quarterly Reports, Previous Evaluation, Interviews with KIIs, FGD Questionnaires, interview guide, Desk review, Interviews and transcription. Descriptive method, Content and thematic analysis with Nvivo12. Analysis done using a thematic grid developed by randomly selecting three interviews and identifying the initial themes. 2. How has WAHO led and supported the scale up of best practices in the region? (i.e. mapping of current high impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale up of interventions; health information systems integration, capacity strengthening); PMP, Annual Reports, Quarterly Reports, Previous Evaluation, Interviews with KIs, FGD Questionnaires, interview guide, Desk review, Interviews and transcription. Descriptive method, Content and Thematic analysis. Analysis done using a thematic grid developed by randomly selecting three interviews and identifying the initial themes. . 3. What has been WAHO’s role in advocacy and communications efforts with respect to best practice scale up; national policy changes for FP and HIV/AIDS programs; improved commodity security; public and private partnerships; and health information systems to increase investments in health and improve reporting? PMP, Annual Reports, Quarterly Reports, Previous Evaluation, Interviews with KIs, FGD , Questionnaires, interview guide Document review, Interviews and transcription. Content and thematic analysis with Nvivo12. Analysis done using a thematic grid developed by randomly selecting three interviews and identifying the initial themes. 4. What areas or domains of WAHO’s capacity (within institutional and leadership capacity) have improved because of USAID/West Africa’s support from 2012 - September 2018? (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region) PMP, Annual Reports, Quarterly Reports, Previous Evaluation, Interviews with KIs, FGD Questionnaires, interview guide Document review, Interviews and transcription. Descriptive method, Content and thematic analysis with Nvivo12. Analysis done using a thematic grid developed by randomly selecting three interviews and identifying the initial themes. 35 | Page Data disaggregation Data analysis was oriented to respond to the main evaluation questions; analysis referred to the results framework developed by WAHO and assessed whether the CAPS project contributed to strengthening the enabling environment at regional and national levels for increased uptake and use of health services by people in West African countries. Data were disaggregated and analyzed across countries and per country. 36 | Page OVERALL EVALUATION ENVIRONMENT Key Informants (KIs) Profile The interviewees were mainly officials of the Ministry of Health at different decision-making levels including WAHO focal points, international project/program managers, technical and financial partners (including WHO, UNFPA, USAID, DAVYCAS), and members of the WAHO CAPS project team (see list of participants in annex 1). There were public health physicians, computer engineers, demographers, sociologists, monitoring and evaluation (M&E) specialists, supply chain managers, data management technicians, and HIS managers. The KIs in different countries operate in similar domains of activity, and the same applies to their responsibilities. These responsibilities include coordination and design of public health activities, health information system activities, sexual and reproductive health of youth and adolescents, neglected tropical diseases, maternal and child health, HIV/AIDS prevention, family planning, communication for change, M&E of project/program activities and health system strengthening activities. Awareness of the CAPS project by the resource persons surveyed To evaluate the awareness of CAPS project, both study participants in individual interviews and FGDs were asked the following question: ‘‘Have you heard about WAHO and the CAPS Project in ECOWAS countries? If yes, what do you know about the CAPS project activities in ECOWAS countries?” As stated below, KIs were, in general well informed or knowledgeable about WAHO. But only a few participants were aware of all the components of the CAPS project. In the 15 countries covered by this evaluation, 56 percent (53/95) of those who responded to the question (Individual KIs = 74 and FGD = 21) are aware of or have heard of the WAHO and the CAPS project. The breakdown of knowledge among different groups shows that 62 percent of the KIs from the individual interviews (46) and 33 percent from the FGD (7) were aware of the CAPS project. Out of these respondents, only the WAHO focal points in the MOH (with the exception of two) and WAHO workers were fully aware of the following four components of the project: • Best practices • Advocacy and communication • Public–private partnership • Health information system Some respondents are aware of the activities of the CAPS project implemented under WAHO but could not describe the 4 components of the CAPS project. “At the level of our institution it is true that we know WAHO because of the technical support it provides us. But concerning this CAPS project, we are not aware of that, but nevertheless we receive a lot of support from WAHO.” – Niger FGD National Stakeholder KI “I know that as part of capacity-building we had to work a lot with WAHO; certainly, it was the CAPS project that was in its active phase.” – Benin National Stakeholder KI The respondents acknowledge that they receive a lot of support from WAHO and even mention certain activities within the components of the CAPS project and clearly attribute them to WAHO without making any distinction. “Yes, I once heard about it, but I don’t know much, except that it was from this project that WAHO set up a sub￾regional health information platform. And each country provides information on the indicators it monitors. We had a team from WAHO last year that came to ask us about our needs in terms of strengthening the health information system. I think it’s still part of this project. I also know that through this project there has been capacity-building of our leaders.” – Benin National Stakeholder KI Among the most well-known CAPS project activities cited by the respondents were the identification and scaling up of best practices, the establishment of a regional health information system platform, and resource mobilization for health. 37 | Page “Yes, we were even beneficiaries of this project. It is a project with four components. There is a component that deals with good practices to scale-up. In this component WAHO organizes every year in countries; for example, in Togo this year, the good practice that has been adopted is the distribution of contraceptives by community health workers called low-community distribution of injectable, where Togo has been a pioneer with a good result. We had received funding of more than 48 million CFA francs. The second component of the CAPS project concerns the health information system. WAHO with the support of USAID has been able to network the different health information systems so that monitoring allows knowing if Togo has introduced all the data up to the last week. The third component is to engage countries in more resource mobilization for health. It is with this in mind that a resource mobilization network is being set up in Togo. Tools have been made available to us for its operation. This network has been able to obtain FP funding from municipalities through advocacy.“ – Togo National Stakeholder KI It is interesting to note that WAHO’s points of contact are also not fully aware of the 4 components. However, the KIs in majority know the activities of the project, which is the most important. Indeed, knowing CAPS is not really relevant as long as KIs know the technical components and therefore can submit requests for support. Overall, the CAPS project could be deemed relatively fair and most countries seem to have benefited from the project activities. However, it is often difficult for interviewees to distinguish between the project and other WAHO activities. Interestingly not all WAHO’s points of contact are fully aware of the 4 components. This is understandable because CAPS belong to WAHO, which also has several other projects that could be confusing. It is important to note that KIs in majority know the activities of the project, which is key. Indeed, knowing CAPS is not really relevant as long as KIs know the technical components and therefore can submit requests for support. Organizations Supporting Health Policy Implementation through the Region When asked ‘‘which organizations are supporting (TA and funding) the government in terms of health system capacity strengthening?’’ many KIs responded that their country benefits from the support of several partners in the area of health system strengthening. All ECOWAS countries receive support for the implementation of their respective health policies. The areas of support provided to countries for the implementation of health policies are almost identical and depend mainly on the area of interest/ intervention of the various TFPs. Thus, IPs have mandates, missions, or objectives that often correspond to the needs of countries and can be ranked by category. Support is provided from a coordination perspective when it comes to providing support to several countries or a joint approach to supporting the health policy of ministries of health in the form of supporting governments in achieving their priorities. Hence, jointly with the government, the priorities established are analyzed to ensure that they really contribute to improving the health of the population. Most of the countries’ priority areas are generally concerned and the most cross-country priorities that stand out are as follows: • Access to the highest levels of quality care delivery • Coordination, to harmonize ECOWAS states’ policies through the pooling of resources and cooperation between countries • Reproductive health, and maternal and child health • Child health with all components going from child diseases prevention to the early management of these diseases and a healthy environment for children to grow up and be capable of contributing to the development of their country as adults • The fight against HIV/AIDS, malaria, neglected tropical diseases, and other epidemics such as Ebola • The One Health initiative • Strengthening the pharmaceutical supply chain for both the procurement of products and the monitoring of the management of these products • The development and deployment of the health information platform (DHIS2) The organizations that collaborate to support governments in the implementation of health policies are, to some extent, the same for different countries. The following are the different names that come up most often at the beginning of this question: 38 | Page • UN Agencies: UNICEF, WHO, United Nations Development Programme (UNDP), UNFPA, UNAIDS (UN Program for AIDS) • German Cooperation Agency • World Bank • African Development Bank • USAID • European Union • French Development Agency • Global Fund • Bill & Melinda Gates Foundation • CIRD, Centers for Disease Control (CDC) • WAHO • MUSKOKA • Johns Hopkins Program for International Education in Genecology and Obstetrics (JHPIEGO) • Expertise France • MEASURE Evaluation • STOP Palu • John Snow Inc. (JSI) • Catholic Relief Service (CRS) • Bilateral cooperation also with some embassies of a number of countries, such as the Netherlands • University of Oslo • ECOWAS • Japan International Cooperation Agency (JICA) In this multitude of partners operating in the different countries, WAHO’s name comes up several times, as illustrated by this KI from Guinea: “There are many of them. Since I’ve been here, I’ve been able to say that we have WAHO, we have WHO, we have USAID, because even yesterday we were in an activity with USAID. In terms of quality assurance, drug quality control assurance, it is an activity that I chaired on behalf of the department. CDC Atlanta also supports us, Georgetown University supports us, expertise in France supports us, so in terms of legislation and regulation, here are many of them, but in the immediate future it is with these institutions that we have exchanged a lot and WAHO is in the forefront.” – Guinea National Stakeholder KI Under the initiative of the MOH, these actors have initiated a dynamic of collaboration to better address the national priorities. Therefore, coordination of these interventions from partners remains a big challenge at country level. There is competition between partners who would like to intervene in the same areas or in the same activities. To address this challenge, there are various attempts of solution ranging from coordination through the lead partner or the creation of a common basket to manage the related funds. “In the department here, when we started thinking about a joint plan, we felt that we should avoid duplication of activities in the same place. Wherever a partner has funding for a particular region and the region is already supported in the same domain, the partner’s activity must be redirected to the place that requires this support. So, in reality, on this side there is an effort, a framework for dialogue and consultation that has been put in place and periodic meetings are held with the partners. At one point we even wanted to have a common basket of funding so that we also know what the amount is, what the financial support is to the Guinea account, and how we can rationalize this support to see how to improve the indicators. It is a reality, it exists, it is not yet controlled, but for the moment that exists.” – Guinea FGD, Implementing Partner KI WAHO’s role in supporting countries to better coordinate partners’ interventions can be very helpful to address this challenge. This needs to be included in its mandate. 39 | Page Typology of government support for health system capacity-building The support provided to governments for health system capacity-building follows the same logic as the one provided for health policy development (i.e., depending on the areas of interest of the TFPs). Organizations involved in providing this support vary enormously from country to country. The diverse areas of support that can be grouped into four broad categories: 1. Organization and strengthening of health care provision: areas under this category are as follows: • Maternal and child health with the World Bank’s Projet d’Appui à la Santé Maternelle, Infantile et la Nutrition (PASMIN) and other partners such as UNFPA, WHO, GIZ, KFW and Plan International under the MUSCOKA funds in Togo • Support for supply chain, pharmaceutical regulation, pharmacovigilance, and traditional medicine strengthening—there is a large training, supervision, and M&E component where several partners are involved, and everything related to strengthening governance and leadership and the PPP • Quality of care—WAHO intervenes at the level of the private health sector on this 2. Strengthening the Health Information System (HIS): through the implementation and use of DHIS2 software, staff training, the provision of necessary resources to produce quality data such as the procurement of computer (tablets and computers) and the data dissemination and archiving 3. Coordination and technical support in tools harmonization (hard or soft) as well as regulation for all 15 ECOWAS countries, mainly supported by WAHO 4. Implementation of good practices and their scaling up and high-impact strategies “There is WAHO, which is like the WHO of Africa, which plays a major role in this. I have participated in at least three WAHO-funded workshops to build our capacity. There are our usual partners, UNFPA, German cooperation (GIZ and KFW), also through the MUSKOKA Fund. Speaking of the HIS, I think WAHO is involved in this and in scaling up the good practices that USAID also supports through the Amplify project.“ – Togo National Stakeholder KI The different kinds of support countries are receiving are either technical (development of health norms and standards and their use, harmonization of curricula, training modules, etc.) or financial (investments or financial support) and most organizations align behind one or another. However, some organizations position themselves as an agency that provides both technical and financial support at the same time, and USAID is one of them. “For technical support, it is through USAID consultants. There is a whole staff: staff who participate in meetings, staff who participate in the development of strategies, workshops and reflections to support the government. It is the technical and financial support that is used to help the government build a stronger health system through the adoption of new practices, guidelines and the formulation of strategies.” – Côte d’Ivoire FGD IP KI Hence, several organizations support ECOWAS countries’ governments in building their health system capacity at varying degrees. While WHO is considered the lead agency for health system strengthening, other names also come up frequently each time the issue is raised. “The objective here is integration, it is the harmonization of health policies at the level of the ECOWAS region. WHO issues guidelines, gives guidance to states and . . . for the . . . organization and management of health systems? So, each organization is based on its missions. That’s it, but, well, WAHO’s mission is really to harmonize, in any case, health policies at the sub-regional level. But it also takes into account the international mechanism, that’s all! . . . There is, of course, WAHO, which is the armed arm, the lonely arm of ECOWAS, and that’s it, so there is naturally WHO, UNICEF, UNFPA, international, regional and health NGOs. So, there are many partners both at the level of what is called sub-regional government, and at the level of international non-governmental organizations.” – Burkina Faso National Stakeholder KI 40 | Page CAPS EVALUATION KEY FINDINGS The CAPS project was implemented from February 2015 to September 2019 by WAHO through funding from the USAID/WA regional office. All 15 ECOWAS countries benefited from the project activities. The current performance evaluation period covers February 2015 through March 2019; 22 main outputs were expected through 29 indicators and 268 activities by the end of project (EOP). At EOP, however, 344 activities had been completed, thus exceeding the target by 76. Table 3 below provides the detail of number of activities planned and achieved by indicator. Table 3. EOP comparison of activities planned vs. achieved by indicator (Source: CAPS)1 N° Indicators EOP Target Achievement Comments 1 Guidebook for Best Practices (BP) completed 1 1 Referring to the number of activities implemented,, CAPS project entirely achieved the expectations and even reached beyond the target (268 activities were planned and 344 were fully executed. Referring to the level of achievement of the 29 indicators, we can also confirm that the CAPS project’s activities achievement is rather good. These achievements are remarkable in the following areas: health policy harmonization and strengthening through WAHIT assistance, number of resolutions validated by the ECOWAS Assembly of Health Ministers, BP scaling up, HIS strengthening, and engaging private sector in supporting health policy/program implementation. Therefore, efforts need to be done for better collaboration with East Central and South Africa Health Community (ECSA-HC) supporting countries to use validation rules. 2 Best Practices fora held 4 3 3 Resolutions validated by Assembly of Health Ministers (AHM) 12 18 4 Meetings between WAHO and East Central and South Africa Health Community (ECSA-HC) Conducted 10 6 5 BP mapping exercises completed 15 10 6 Good Practices requested by countries 14 12 7 Good Practices in scaling up process identified 15 21 8 National scale-up committees established 7 6 9 M&E exercises to track BPs implemented 7 10 10 Plan for advocacy and communications 1 1 11 Communications product types dissemination 55 82 12 Press releases published 12 15 13 Resolutions disseminated 4 24 14 Follow-up visits conducted 3 8 15 Resolutions being implemented 8 8 16 Evaluation activities conducted 1 1 17 Plan for Public–Private Partnership (PPP) policy and guidelines completed? 1 1 18 Private sector firms identified 20 28 19 Private sector firms engaged 0 1 20 PPP regional meetings organized 2 1 21 Meetings with UEMOA, ECSA-HC & AU held 2 5 22 Countries with HIS policy and strategy 15 11 23 Countries reporting weekly Integrated Disease Surveillance & Response (IDSR) data completed 15 15 24 Countries reporting IDSR on time 15 13 25 Countries using validation rules 12 4 26 Epidemiology bulletin on WAHO website uploaded 10 10 27 Countries supported by WAHIT 3 14 28 Document improvement to national HIS platform 3 3 29 Website with One Health approach 1 1 Total 268 344 Basing the evaluation on the number of activities implemented, there is no doubt that the CAPS project entirely achieved the expectations and even reached beyond the target. However, indicator achievement level is the best estimate of the health of a project because the performance is purely based on percent of indicators suffering and percent of indicators achieved even not considering other factors such as amount of budget spent or time delay. In that regard, the desegregated percent of indicators achievement level is as below: 1 Table from document “Copy of CAPS Dashboard_2019_05_20_TA” 41 | Page • 11 indicators achieved above 100 percent target (exceeded the EOP target) • 9 indicators reached the target • 8 indicators achieved between 50 percent and 99 percent of target • 1 indicator achieved below 50 percent of target In summary, referring to these indicators, the CAPS project’s activities achievement is rather good. These achievements are remarkable in the following areas: health policy harmonization, number of resolutions validated by the ECOWAS Assembly of Health Ministers, best practices scaling up, HIS strengthening, and engaging private sector in supporting health policy/program implementation. Table 4 below gives more details of the level of achievement by expected result (ER) of the CAPS project. Table 4: The health of the project in term of performance (Source CAPS)2 Health of Project* ER 1.0 ER 2.0 ER 3.0 ER 4.0 Key Indicators Exceeded Target 33% 57% 50% 13% (>100 %) Indicators Achieved Target 11% 43% 25% 50% (100 %) Indicators Almost Reached Target 56% 0% 25% 25% (25 - 99 %) Indicators Substantially Below Target 0% 0% 0% 13% (<25 %) Indicator Count (29) 9 7 5 8 *The health of project is based purely on indicators (i.e., percent of indicators suffering, and percent of indicators achieved). It does not consider other factors such as amount of budget spent or duration of project already expended. Per the table above, only 13 percent of indicators attached to ER 4.0 (Regional Health Information Systems (HIS) Policy and Strategy Operationalized) were substantially below target. Besides, the ER 2.0 (WAHO’s advocacy and communications strategies operationalized) indicators are either fully executed and or have exceeded target. Finally, targets for ER 1.0 (WAHO’s support to ECOWAS member countries to scale-up evidence-based practices increase) and ER 3.0 (WAHO’s role in regional donor, public-private partnerships expanded) are on track and may be completed by closure of project activities. Question 1: To what extent is WAHO considered a leader in advancing health issues and outcomes in the region? A majority of Key Informants (KIs) interviewed both individually and in the Focus Group Discussions (FGDs) recognize WAHO as the legitimate leader for health in the ECOWAS region as it has this official mandate from the ECOWAS heads of state. KIs stated this through the fact that WAHO have a legitimate mandate from ECOWAS’ Heads of states, support member states to harmonize health policy through governance and capacity building and sharing of updated guidelines and strategic documents with member countries. In fact, during the period of implementation of the CAPS project, WAHO has played a key role in coordinating and harmonizing health policies among state members. Most of the KIs recognized that WAHO has a proven role in harmonizing health policies by developing, updating, and sharing guidelines and other strategic documents with state members. The majority of KIs recognized that WAHO’s role in harmonization and integration of ECOWAS countries to address health issues has increased during the period of the project. However, WAHO faced some constraints related to resource mobilization, supporting countries to better coordinate partners’ interventions at country level. Leading organizations in health issues in the ECOWAS region It is not an easy matter to rank an organization as a leading one as far as health is concerned according to some respondents, because the organizations involved in health usually have different fields of intervention. For example, UN/AIDS could be ranked as a leader in the field of HIV/AIDS, UNICEF as a leader in child health, Global Alliance Vaccine Initiative (GAVI) as a leader in the field of immunization, Roll-Back Malaria in the fight against malaria, etc. Therefore, it appears that there are many leading organizations in specific areas. Nevertheless, there seems to be 2 Table from document “Copy of CAPS Dashboard_2019_05_20_TA”. 42 | Page unanimity in favor of the organizations of the United Nations systems (WHO, UNICEF, and UNFPA) as leading organizations in the health sector. Indeed, their missions are related to development and health issues. Besides, bilateral cooperation organizations are also considered leaders in West Africa, including USAID, the French Development Agency, the Belgian Development Agency, German Agency for International Cooperation (GIZ), Luxembourg Development Cooperation Agency, the Netherlands Development Cooperation agency, WAHO, the World Bank, and the Global Fund. Some NGOs, mostly American and USAID play a big role in global issues related to maternal, newborn, and child health. “Well, leader! Well, I don’t know. For me, it is in terms of complementarity, structures complement each other. I cannot say that, WAHO is ahead of everyone, but it is in fact a complementarity between the technical and financial partners within the framework of the implementation of activities in the field of health. Well, it's quite true that perhaps in a particular field, WAHO can be up front, that's not a problem, just like UNFPA as far as reproductive health is concerned. That is what I can say about leadership, it is a matter of complementarity” Burkina Faso - FGD_ Implementing Partner KI The KIs, therefore, qualify the organizations as leaders according to the area of support in the different departments of the ministries of health. Thus, depending on the areas and nature of support, the leaders also change. The word cloud below graphically represents the organizations mentioned as leaders in advancing health issues and outcomes in the ECOWAS region. Figure 3. Main organizations recognized by KIs as leaders in advancing health issues and outcomes in the ECOWAS region How is WAHO considered as a leader? Almost all the interviewees rank WAHO among the leading organizations in the ECOWAS region, while not overlooking the supremacy of the WHO, whose performance goes beyond the region. They believe that WAHO plays an important role and are aware of WAHO’s growing support over time in terms of activities and especially technical assistance to countries. The strength of WAHO is being the regional focal entity for public health intervention (direction, initiative, innovation? All of the above) and to develop strategies that can have common application within the region and their advocacy. “I think that from a regional point of view it is WAHO, because regional means that we have to implement the good practices of a country, perhaps in scale, we have to network the countries, we have to create a little . . . we have to try to harmonize things. And in that a little I think there is no other institution if it is not WAHO. So clearly I think WAHO is the current institution that although all efforts have yet to be made, WAHO is the institution that does this!“ – ECOWAS Stakeholder Some IPs recognize this leading position of WAHO as they seek that it uses its political mandate to push for an agenda and have governments adopt health policy and programs that should be implemented in the region. 43 | Page “We have seen that there is an opportunity for leadership in working with WAHO. For example, there are political difficulties in being able to influence certain policies that we would like to highlight in the sub-region. For example, the resolution on task-shifting3 . . . so it is this kind of action that WAHO can take at the level of the ministry of health to influence certain policies. That’s a little bit the goal for our collaboration with WAHO.” – Senegal- Global KI (International Partner) WAHO is getting progressively involved in ECOWAS countries’ health management and organization and is expected to do so even with greater emphasis. The figure below is a word cloud that summarizes the domains where WAHO’s leadership is acknowledged by the KIs. Figure 4. Summary of different domains highlighting WAHO’s leadership According to the above word cloud, the better words that qualify WAHO’s leadership are: Policies – Support – Funding – Assistance – Programs – Strategies – Meetings – Training – Partnership – Planning – Help – Capacity – Data – Information – Scaling – Best Practices. WAHO Leadership in Promoting and Harmonizing Health Policies in the ECOWAS Region WAHO contributes to the updating and harmonization of health policies, especially regarding the information systems in different countries. Furthermore, WAHO’s regional health sector activities are organized with the involvement of all stakeholders, including the TFPs. “With regard to the harmonization of health policies, WAHO plays a very important role because WAHO organizes international meetings where all the expertise is brought together. And there are shared experiences. There is WAHO support in terms of tool production. WAHO makes proposals and asks countries to adapt. I always stay in my field of health information. I know that in relation to DHIS2, I took the helm of the management two years ago; in relation to DHIS2, WAHO coached us a lot in terms of implementation. In relation to the revision of the tools, they even provided us with technical assistance first to help us set up while respecting this idea of harmonization at the regional level.” – Niger National Stakeholder KI WAHO’s leadership in promoting and harmonizing health policies, especially in the area of HIS, is recognized by the majority of KIs. “[On the DHIS2 platform], the leadership of WAHO is already an asset to be able to [develop the platform] in all the countries and also annually to monitor and evaluate the level of progress of each country.” – Guinea IP KI In addition, WAHO’s leadership in health policy harmonization has been expanded over the life of the project and covered various areas such adolescent and youth reproductive health, as assessed below: 3 Task shifting is a process of delegation of tasks to less-specialized health workers. 44 | Page “But what I can see [is] that there’s a growing support. . . . Every year I see them more expanding their network and reaching into more areas.” – Ghana National Stakeholder KI “In 2016, we had a consultant from WAHO to assess adolescent health issues in the sub-region, and after that we were invited to a meeting in Dakar to develop an orientation guide. We had another meeting in Bobo—it was there we harmonized it in French, English, and Portuguese, and the assembly of all ministers [of health] of ECOWAS ratified it and sent it back to us.” – Nigeria National Stakeholder KI WAHO plays an important role in harmonizing staff training curricula in health schools and practices. “I think there is a job; already I find very interesting what they have done and what they continue to do. For example, the harmonization of training curricula for basic personnel; whether it is doctors, nurses, or midwives, this is fundamental; it is really one of their main roles. How can we standardize, create the conditions for a common practice in a sub-region, especially when we know the difference between English-speaking and French-speaking countries? I think that this is the basis on which all other actions must also be taken.” – Global KI The role of WAHO in the sub-region is indisputable according to most interviewees. Indeed, according to them, WAHO is the only legitimate leader with the official mandate of ECOWAS heads of state to coordinate and harmonize health policies. “It influences the regional health policy and makes great efforts at the regional level for that. This leadership position is comforted through its presence alongside member countries.” – Côte d’Ivoire National Stakeholder KI “No, you know, the legitimate leader who is in charge of coordinating policies at the sub-regional level, is WAHO. The only leader who has the power to manage the coordination and harmonization of health policies and who amends the official mandate of heads of state is WAHO.” – Burkina Faso WAHO KI “WAHO is a leading regional institution bringing together member countries through coordination, capacity-building, and governance. Indeed, it adapts and harmonizes WHO recommendations for CI-WAHO member countries.” – Côte d’Ivoire National Stakeholder KI Factors limiting WAHO’s role as a leading organization in health issues in the ECOWAS region However, WAHO’s leadership is curtailed by certain handicaps including the inability to attract highly qualified staff to live in Bobo-Dioulasso and the organization’s governance model. “We are still a little weak, we’re not as strong and it’s not because we don’t have qualified staff. It’s just that we don’t have adequate qualified human resources. And part of it is our location here, you know. Many professionals will hesitate a little bit to relocate from Dakar to come and live in Bobo-Dioulasso, you know. It remains a challenge. If it was Ouagadougou perhaps, it’s easy to travel to, but Bobo is still a problem. So, our leadership capacity is still a bit weak because to be able to have a wider reach, you need the human resource capacity. Part of it is also not because of Bobo, it’s also the governing structure. We are more like a secretariat with representation, single man representations in the countries. WHO has offices; and that itself which is equipped can have a better reach than when you are just a liaison officer who reports to a minister.” – Burkina Faso WAHO KI Furthermore, financial constraints are a limiting factor for WAHO as money is key for empowerment/ Elaborate. Funds for what purpose? “It all boils down to funds. If only they have money. They don’t have the money and they don’t have the capacity. So, then their role is limited.” – Ghana National Stakeholder KI On the other hand, there is sometimes competition between organizations supporting countries, particularly during emergencies, as was the case during the Ebola epidemic. In that instance, WAHO’s leadership seemed to have suffered limitations despite the important role it played in the response given the magnitude of the problems to address. “During the Ebola epidemic what role did WAHO play? I know they tried coordinating between countries. However, when the bigger actors come in, where does WAHO fit? If there is a WHO document, a UNICEF document, or UNFPA document, does the WAHO document become superfluous? How does WAHO strategically position itself 45 | Page within this context? You may be surprised; it is probably easier to travel from Geneva or the US to Accra than it is to travel from Bobo-Dioulasso to Accra.” – Ghana FGD Implementing Partner KI In any case, it is obvious that WAHO is playing an important role and has made a big jump in terms of leadership development in the last years. The CAPS project has been instrumental for this achievement to take place because it is a work in progress, but there is already a good momentum that needs to be maintained. Question 2: How is WAHO a leader in promoting and supporting the scaling up of good health practices? Promotion and scaling up of best practices at the country level has been one of the areas where WAHO has made key achievements. WAHO supported knowledge and experience sharing through BP forum organizations. In addition, WAHO provided technical and financial support to help countries identify, document, and scale-up BP. Therefore, these efforts need to be consolidated, especially by operationalizing the BP committee in each of the ECOWAS countries. Among the five expected outputs (EO) regarding WAHO’s leadership in promoting and supporting the scaling up of good health practices, one EO exactly reached the target of 100 percent, two were achieved beyond the target, and two didn’t reach the target but are above 75 percent reached (see the summary in Table 5 below). Table 5. Expected output table of WAHO’s performance in promoting best practices Expected Output Indicator EOP Target Results Comments EO 1.1 Process for identification, documentation, validation, and dissemination of best practices completed Guidebook for identifying, documenting, validating, and scaling up best practices available 1 1 Interviewees were unanimous in pointing out the important role that WAHO plays in identifying and promoting the scale-up of good practices by ECOWAS countries. Emerging themes supporting their opinions are: experiences sharing of BP, technical support, establishment of BP committees, Therefore, referring to the level of achievement of some indicators, efforts need to be done on operationalizing BP committees, collaboration between WAHO and ECSA￾HC for experiences sharing. EO 1.2 Best Practices (BP) Forum held in WA Region Number of BP fora held 4 3 Number of resolutions validated by AHM 12 18 EO 1.3 Increased collaboration between WAHO and ECSA-HC (BP Forums) Number of meetings held between WAHO and ECSA-HC 10 6 EO 1.4 Regional map of best practices implementation in ECOWAS countries (add to the regional data warehouse) Number of best practices mapping exercises completed 15 10 Number of Good Practices requested by countries 14 12 Number of Good Practices in scaling-up process 15 21 EO 1.5 M&E of best practices adoption by ECOWAS countries enhanced Number of National Scale-up Committees established 7 6 Number of M&E exercises to track best practices adoption conducted 7 10 Total 85 87 Interviewees were unanimous in pointing out the important role that WAHO plays in identifying and promoting the scale-up of good practices by ECOWAS countries. The first phase of this process was supported by WAHO through the development and dissemination of methodological tools for documenting good practices and building institutional capacity to use them. Furthermore, WAHO supported countries to understand these tools and apply 46 | Page them systematically whenever they want to create or document a best practice. Lastly WAHO, through the CAPS project, created a platform where countries meet to exchange their respective experience on best practices, as stated below. “CAPS is a very good example, especially if you look at our flagship best practices forum. This is a forum, which provides countries with the opportunity to share best practices in maternal and reproductive health. Previously there was a deficiency in-house: how does one country learn from the other? Now, this project has enabled us to create that platform, which is done annually.” – Burkina Faso, WAHO KI In addition, WAHO gave continuous support to countries in implementing their good practices. “Yes, WAHO developed this Good Practice strategy. It has supervised and coached countries to identify and implement pilot actions [and] encourages scaling up through technical and financial support (through the drafting of action plans).“ – Côte d’Ivoire National Stakeholder KI “It must be said that at this level WAHO plays a major role. I know that it organizes quite a few workshops in relation to these good practices, in relation to the training of actors from different countries on how to really develop [and], the steps to develop a good practice. There are many themes that have been the subject of good practices.“ – Burkina Faso FGD National Stakeholder KI In the process, countries first identify best practices during good practice committee meetings and thereafter make requests to TFPs, including WAHO, to provide support so that the best practices can be scaled up. The good practices promoted by WAHO for scaling up are good practices identified by the countries themselves. Therefore, WAHO relies on actual in-country experience put forward by participating countries. “There are no good practices of WAHO, the expression is very important to grasp....WAHO has to learn (. . .) When it is promoted by WAHO, countries say that these are the good practices of WAHO.“ – Burkina Faso WAHO KI “This is an initiative of the Ministry of Health but strongly supported by WAHO. Since all the requests we send to WAHO to solicit their support, we always get a good feedback. As they say, their mission is to support the 15 ECOWAS countries for the well-being of their populations.” – Benin National Stakeholder KI Given that scenario, WAHO provides guidelines and support for the identification of good practices, creates the platform for good practices fora, and allows countries to meet and exchange best practices. “The forum is the scientific platform or scientific conference and allows very good exchanges between countries so that we can capture the maximum practice shared by both. The post-forum is also support in terms of capacity￾building or everything related to replication or large-up, scale-up and [what] we have used as a strategy is [that] the countries will send a request, for those who want it, those who are interested.” – Burkina Faso WAHO KI WAHO’s contribution to the promotion of good practices is not only technical but also financial. Once a good practice has been identified, countries submit technical or financial support requests to WAHO, which after an analysis accompanies them while monitoring implementation to ensure effectiveness. “We also have Benin that had identified the approaches they wanted to implement, and that asked for the funding and we gave the costing. It’s the same [for] Sayana Press—we gave them half, we can give you the evidence and we’re observing them to see, as soon as it’s going to be conclusive, where convincing, we’ll complete the funding to scale it up.” – Burkina Faso WAHO KI WAHO plays an important role in promoting the good practices identified by countries through human resources capacity-building for this activity. “For scaling up, WAHO has played a key role in building the capacity of agents at the national level and making products available. WAHO has provided funding for some of these activities.“ – Niger National Stakeholder KI For the scaling up of good practices, all TFPs in each country are involved. There is a whole process for scaling up good practices. WAHO supports countries in training and setting up best practice committees. WAHO organizes capacity-building visits for local scaling committees. The scaling committee is composed of local partners and national partners. Other partners accompany WAHO for scaling up. As an example, in Burkina Faso, for the scaling up of some good practices such as postpartum family planning, post-abortion care with the integration of family planning, 47 | Page the birth preparation card, and task shifting are done with the support of WAHO, NGOs (Association des Maires du Burkina Faso and WHO. “We were approached by WAHO in 2014 to assist with a situation analysis. . . . This was the basis for the good practices forum that started in 2015. We had loads of activities with WAHO. . . . We planned for the first forum that took place in Ouagadougou in 2015. We met in Bobo-Dioulasso where we gathered partners in the region through the IBP network. We assisted with trying to document practices with the ministries of health.” – Global KI, WHO Similar examples of synergy of efforts are reported in Niger, where scaling up post-abortion care as well as Sayana Press on large-scale, FP distribution methods have received support from the NGO Human Development Indices and Indicators (HDI). “Yes, especially in the context of the scaling up of post-abortion care in terms of maternal health, we are already on a scale and it is notably the NGO HDI that has invested a lot. We also have the large-scale passage of the Sayana Press concerning large-scale methods in family planning.“ – Niger National Stakeholder KI . Figure 5: Words cloud summarizing WAHO’s role in promoting and supporting the scaling up of good health practices By supporting scaling up, WAHO has even succeeded in harmonizing the scale-up of some good practices in the ECOWAS region, which has provided a good experience in rationalizing the use of resources. “Last year or the year before that, they passed a high-level resolution on task sharing that was very instrumental to harmonizing task sharing policy in the region. I think that has had a big impact in that each country does not have to separately advocate for a policy change. There is sort of an agreed-upon regional resolution for task sharing in family planning and reproductive health. So, that has been very good. At the same time, partners including WHO and others are acting on that now. So you will see, for example, in a lot of action plans, [that] NGOs and partners talk about task-shifting.4 I don’t know if it is harmonized, but a lot of countries have prioritized postpartum family planning, and that has also taken off as a high-impact intervention to improve uptake of contraceptives in the region.” – Global KI 4 Task sharing is the process of enabling lay and mid-level healthcare professionals – such as nurses, midwives, clinical officers, and community health workers – to provide clinical services and procedures that would otherwise be restricted to higher level cadres, safely. 48 | Page In overall, several areas emerged where WAHO’s contribution was substantial in promoting and supporting the scaling up of good health practices in the ECOWAS region. The words cloud below focuses on most of them on most of them. Question 3: What is WAHO’s role in advocacy and communication efforts regarding the scaling up of good practices? WAHO initiated a network of communication managers in each of the 15 ECOWAS countries. In addition, WAHO developed communication and advertorial materials. Moreover, WAHO has developed advocacy and communication activities to help countries mobilize financial resources and has also initiated start-up advocacy activities to get BP accepted by countries health ministries. These efforts are also perceptible in other areas such as supply chain and commodities security and private–public partnership development as stated below. When considering the number of activities planned regarding WAHO’s role in advocacy and communication efforts to support good practice scale-up, the overall level of achievement is beyond the initial target. However, among the six expected outcomes, two reached 100 percent of the target, two exceeded the target, and two did not reach the target but are above 70 percent (see Table 6 below). Table 6: Expected and output table of WAHO’s performance in advocacy and communication for best practices scaling up Expected Output Indicator EOP Target Results Comments EO 2.1 Current WAHO Advocacy and Communication strategy reviewed and implementation plan developed Implementation plan for Advocacy and Communication strategy available 1 1 KIs recognized WAHO’s advocacy and communication efforts in various domains: development of advocacy and communication plan, communication activities in BP scaling up and resources mobilization, establishment of focal point to strengthen WAHO‘s visibility, development of advertorial materials and documentaries. Therefore, KIs suggested that WAHO to reinforce communication activities for resolution dissemination and implementation. EO 2.2 Regional fact sheets, policy briefs, newspaper articles, advocacy videos, topical fact sheets and briefing papers disseminated Number of communication product types disseminated 55 78 EO 2.3 WAHO relations with local media in ECOWAS countries enhanced WAHO relations with local media in ECOWAS countries enhanced 12 15 EO 2.4 Resolutions and recommendations from Assemblies of Health Ministers meetings disseminated Number of resolutions disseminated 4 2 EO 2.5 Follow-up on the implementation of recommendations and resolutions of AHM with member states conducted Number of follow-ups conducted 15 15 Number of resolutions being implemented 8 0 EO 2.6 Advocacy and communication activities evaluated Number of evaluations conducted 1 1 Total 96 112 WAHO’s advocacy and communication efforts in scaling up good practices are widely recognized by countries. These efforts are the result of various and intensive WAHO advocacy and communication activities carried out in the region toward ECOWAS countries. Furthermore, WAHO has an Advocacy and Communication Plan that was developed as part of USAID’s Leadership Management in Governance (LMG) project but had remained sluggish. The CAPS project enabled WAHO to operationalize it by engaging a consultant to support the process. Thus, within the framework of the CAPS project, a network of communication managers from the 15 countries at the level of ministries of health and media representatives has been created and strengthened. Communication factsheets, 49 | Page documentaries, and advertorials are developed to ensure that all activities carried out at the regional level are well advertised and disseminated. Getting ECOWAS health ministers to accept good practices required considerable start-up advocacy, but the effort came to fruition. This accomplishment is partially due to the communications activities implemented as part of the CAPS project. “First, even before leaving on their own initiative, it was the 15 ministers of health who, in 2014 in Monrovia at the MHS (the ECOWAS summit of ministers of health) signed and adopted the resolution that created and established the forum. It means that they have already been convinced of the validity of this forum as a regional platform for sharing and exchanging ideas on things that work, initiatives that are promising, in order to better advance the issue of health in the region. And from the moment the first stakeholders understand, advocacy will easily mobilize the national partners who support the ministry.“ – Global KI The advocacy has continued for the establishment of Large-Scale Passage (PAGE) Steering Committees in some countries. These are multidisciplinary committees composed of resource persons from several institutions. Advocacy efforts have been made to ensure that committees can operate as institutional mechanisms for strengthening communications activities. There are also activities such as national workshops organized in countries to prepare for forum participation and workshops to strengthen countries’ capacities to scale-up good practices. These activities are organized with the participation of the ministries of health. The establishment of focal points in each country’s MOH has also helped strengthen WAHO’s communication and advocacy efforts to scale-up good practices. “For the promotion of good practices, communication, and advocacy, WAHO has done a lot, eh? At the level of each country, they have a focal point in relation to communication, to exchange and communicate and participate in the different activities, especially activities at the regional level. And then also these communication representatives are in the sharing network at the WAHO level where they are informed of everything that is happening to make the diffusion.” – Global KI Advocacy meetings are organized at the national, sub-regional, and regional levels with some including health ministers, prime ministers, and TFPs for good practices. These are relevant moments and opportunities for WAHO to increase its advocacy and communication power. “In any case, the role of WAHO, I think it is decisive enough, at least in advocacy, for countries to apply . . . a certain number of good practices and policies. Because I see consultation workshops organized by WAHO, regional workshops, [and] sub-regional workshops that bring together the ministries of health, and in particular those primarily responsible for health, which often bring together the prime ministers and even the presidents, so to speak about health policy and standards. So I also see consultations at the highest level that are held, often in Burkina Faso and even elsewhere, I see meetings that are held with other institutions such as USAID, at the regional, sub-regional, and international levels, therefore meetings through workshops, but also formal and informal meetings that are organized so that a certain number of policies are put in place, and also applied, thus at the country level. That’s what I can say.“ – Burkina Faso IP KI Per health leaders interviewed, WAHO’s advocacy and communication effort since 2011 has been effective in mobilizing resources for scaling up good practices and supporting countries when responding to health emergencies such as the Ebola epidemic. Even though they cannot attribute the results of resource mobilization and other activities to WAHO’s advocacy efforts alone because other partners are also in the field, they claim to have witnessed or participated in some of these advocacy activities. “On advocacy and communication, I know just before the second-best practice in 2016, WAHO established the regional communications network and included journalists across the region as well as communication officers in the different Ministries of Health. And I know that was established, I am not (sure?) how much progress has been made but that did happen. And I do know that in some countries WAHO has worked a little bit more closely with their communication officers and the Ministry of Health to make sure that when WAHO comes in country that their visit and whatever contribution they are bringing is more widely publicized. So those are the things I have on communication and advocacy.” – USAID/WA 50 | Page “WAHO monitors and advocates to identify Good Practices (GPs) and encourages to solicit funding, communicates on funding opportunities for scaling up identified GPs.” – Côte d’Ivoire National Stakeholder KI Some inadequacies in communication and information dissemination around activities persist sometimes, and even extend to focal points that are supposed to be communication channels between WAHO and the various ministries of health; these have been identified and addressed. Thus, WAHO has already undertaken actions to strengthen these communication aspects for a better coordination of its activities. “There is not a country-level mechanism at the national level for coordination; in fact, today all this work is dedicated to focal points. WAHO is training the focal points and equipping them to become some of the program coordinators at the country level. We are in the process of doing so because we had been moved to Burkina Faso. In fact, during this training as a focal point, we were well briefed on the communications aspect and we are really getting equipped with everything we are doing to be able to coordinate activities at the country level. It is also a weakness of WAHO. Some activities are even financed and we as focal points do not have the information on our interventions.” – Mali National Stakeholder KI The different words that best describe the advocacy and communication effort in scaling up best practices are summarize in the words clouds below: Figure 6: Words cloud summarizing WAHO efforts in advocacy and communication WAHO’s leadership in improving the supply chain and commodity security As part of the CAPS mandate, WAHO has helped strengthen countries’ capacities in commodity safety￾security/supply chain management. As part of these efforts, WAHO has provided support in terms of training, equipment, and technical assistance. They have also helped harmonize standard documents in the field of pharmaceutical regulation and traditional medicine. Hence WAHO helped countries develop guidelines and regulations, whether at the pharmacy level, the nursing level, or the level of human resource development in health. As part of supply chain strengthening, WAHO provides financial assistance for the purchase of contraceptive commodities and capacity-building in supply change management and commodity security. “Yes, in fact, at the last WAHO review we attended, there were some countries that benefit from capacity-building funds, there are some that benefit from procurement funds. Niger has opted for the capacity-building fund. I think for a question of sovereignty that it is WAHO that is strongly involved in the purchase of contraceptive products.” – Niger National Stakeholder KI Regarding procurement, WAHO coordinates interventions and mobilizes resources between IPs to carry out group 51 | Page purchases, thus achieving cost reductions. Indeed, a single country with a low-quantity ordering would be costly while pooling the orders will increase the quantity and lower the cost to save money. Figure 7: WAHO role in improve supply chain and commodity security “WAHO always takes very important steps to coordinate interventions and mobilize resources among key implementing partners; I will give you the case of drug procurement, for example: drugs are expensive when only one country buys them (e.g., in the case of Guinea-Bissau); but what WAHO has done as a solution is to mobilize a gross purchases (placing orders together for several countries), which is cheaper, because if Guinea-Bissau buys drugs in Europe, it is very expensive and it takes time to get there.” – Guinea Bissau National Stakeholder KI WAHO played also an important role in strengthening supply chain management by integrating health commodities to national health information systems via an extended use of DHIS2. “Through the DHIS2 we have a logistics management system, [which] is basically a supply chain management system [that] tracks drugs from the supply center, which is our central medical stores, to the health facilities. This support actually came from WAHO.” – Gambia National Stakeholder KI Despite the efforts made to improve the supply chain and commodity security, some interviewees felt that efforts must be maintained and even intensified in this area, especially the standardization of legislation at the regional level in terms of health product importation. “WAHO should continue working on identifying problems and classifying or categorizing them. It is necessary to standardize legislation at the regional level in terms of health product importation, to see what the needs of each country are and to analyze them and try to have a common framework.“ – Côte d’Ivoire IP KI The effort deployed by WAHO to improve the supply chain and commodity security according to KIs are summarized in the words below: 52 | Page WAHO’s role in promoting Public-Private Partnerships (PPP) The promotion of PPP seems to be one of the priority activities conducted during the CAPS implementation, and three out of four EOs were accomplished above target. However, for EO 3.3, identifying more donor PPP and targeted private sector firms than planned, only one of the donor PPP regional meetings planned was organized—this could negatively affect the impact of this EO. Table 7 shows the EO and EOP targets for PPPs. Table 7: Expected and output table of WAHO’s performance in promoting public–private partnership Expected Output Indicator EOP Target Results Comments EO 3.1 WAHO donor PPP policy, guidelines and implementation plan developed Implementation plan for donor PPP policy and guidelines available 1 4 KIs recognized that WAHO imitated tangible efforts in involving private sector partners and donors to support health policy implementation in ECOWAS countries. WAHO developed a strategy document to help countries to implement PPP, organized forum, imitated regional meeting with donors. Therefore, WAHO needs to help countries to this PPP through a better coordination between partners. EO 3.2 Mapping and engagement of donor PPP and targeted private sector firms conducted Number of donor PPP and targeted private sector firms identified 20 28 Number of targeted private sector firms engaged 6 12 EO 3.3 Two (2) donor PPP regional meetings organized Number of donor PPP regional meetings 2 1 EO 3.4 One regional meeting with UEMOA, ECSA-HC, and the AU for harmonization of health policies for Africa region organized Number of regional meetings with UEMOA, ECSA-HC, and the AU organized 2 5 Total 31 50 WAHO plays a coordinating and advocacy role with donors for the financing of member countries’ activities. It facilitates meetings between donors and actors from member countries to help capture funding opportunities. As part of the PPP program, WAHO organized the first forum on public–private partnerships in health in 2017 in Accra. This meeting brought together governments, private nonmedical sector companies, and TFPs. Furthermore, WAHO, with the support of USAID, hired consultants who are familiar with the PPP field and have developed a strategy document to help countries implement PPPs and to set up a PPP unit within the ministries of health. They then supported the various countries in implementing fundraising strategies. “We managed to have a first public–private partnership forum in Accra, I think about three years ago. It was well patronized and then we developed a framework for the regional public–private partnership initiative, which was adopted by the ECOWAS Assembly of Health Ministers. And then some firms were identified like Nestlé; the problem has been the implementation. So, we’ve not been able to engage these firms as effectively as we should have. And I think part of it is because, you know health professionals are generally not business-minded so there’s a little bit of resistance as to what extent you can engage these partners [and] firms, without them exploiting you to their advantage but we are still working on that.” – Burkina Faso WAHO KI The forum was supported by USAID and undertaken to amend, improve, and validate the regional public–private health partnership strategy document developed by WAHO and made it available to countries to guide them in this activity. “WAHO’s role was to be able to organize the forum, because at that forum, there were the governments that were there, there was the private medical sector that was there, there was the private nonmedical sector that was there, there were the USAID partners and the others that were there. So we have started implementing this document, I think we should ensure certain countries . . . identify even initiatives that can be developed in the form of public– private partnerships, to also support countries, for example, in setting up units and cells within ministries of health that must deal with the issue of public–private partnerships.“ – Burkina Faso WAHO KI 53 | Page The performance in PPP development hasn’t been a lonely venture of WAHO. Indeed, in the framework of PPPs, WAHO joined forces with other IPs and as such works in collaboration with other partners such as the World Bank and the European Union to try to mobilize resources. This has had the merit of raising awareness that resource mobilization is far from being an isolated action and should rather be a group effort bringing together multiple stakeholders working for a common goal. “No, there have been many training courses on the PPP in which WAHO has invited representatives of member states to participate. There are other partners like World Bank that encourage PPPs. So, it is in fact a more or less global strategy, because we now realize that the public sector alone cannot solve health issues, and that at this time, we must encourage the participation of the private sector in the implementation of health programs.” – Burkina Faso National Stakeholder KI As part of the PPP’s activities, a study was carried out to map actors and NGOs working in the health sector. “I don’t know if WAHO supported the activity, but speaking of public–private partnership, there is a mapping that has been done—the mapping of actors and NGOs working in the sector. Before that there was no such thing. At meetings it was mentioned.” – Togo FGD National Stakeholder KI To further involve other partners in PPP, WAHO coordinates interventions and mobilizes resources among key partners. It organizes meetings of partners during the meetings of health ministers of ECOWAS member countries. It plays the technical and financial role, strengthening the capacities of countries by pooling resource mobilization efforts. However, the task of coordinating the funding of the various partners is not always easy because the agendas sometime diverge. “We have had always the strategy of involving all stakeholders [and] technical and financial partners in the implementation of our activities, but as I said earlier, people often have different agendas and they have approaches as different from ours. It can happen that we succeed in doing so, it can happen that some want to go it alone, and it is difficult. There is practically no regional value-added activity that WAHO carries out without involving partners, in a coordination mechanism well thought out in advance of implementation, and this is almost systematic.“ – Burkina Faso WAHO KI Despite this interest in promoting PPP, WAHO coordination and implementation of PPP need to be consolidated to make more effective this partnership. The figure below summarizes the role of WAHO in promoting the Public￾Private Partnerships in the ECOWAS region. Figure 8: WAHO’s role in promoting the Public-Private Partnerships 54 | Page WAHO’s role in strengthening the HIS and improving reporting With 74 activities planned to strengthen the HIS and improve reporting, 71 were fully implemented leading to a 96 percent completion rate. However, among the seven EOs, three reached the target, one went beyond the target, and three did not reach the expected target (only 33 percent of what was planned was implemented). The latter is related to the quality of data improvement, a key element for HIS strengthening. See Table 8 for details on HIS and improving reporting. Table 8. Expected and output table of WAHO’s performance in strengthening HIS and improving reporting Expected Output Indicator EOP Target Results Comments EO 4.1 Strengthen the regional leadership and coordination capacity of WAHO in health information Number of countries that have policy and strategy HIS documents in accordance with regional orientation 15 11 WAHO performed tangible efforts for national HIS strengthening through development of DHIS 2 platform, technical assistance from WAHIT, reporting, surveillance system. Therefore, there are still need of improvement about data quality assessment, adoption, validation and operationalization of HIS policy/strategy. Operationalization of a regional HIS including harmonized tools. EO 4.2 Rate of weekly data reporting on IDSR is increased (by at least 85 percent in all districts) Number of countries reporting weekly IDSR data by at least 85 percent of all districts through regional platform (completeness rate) 15 15 Number of countries reporting weekly IDSR data to WAHO on time through the regional platform (timeliness rate) 15 13 EO 4.3 The quality of data is improved (at least in 80 percent of all the countries using DHIS2) Number of countries using the data quality assessment validation rules (DQA) module 12 4 EO 4.4 Data on IDSR are analyzed and disseminated Number of epidemiological bulletin available on the WAHO website 10 10 EO 4.5 & 4.6 WAHIT team project strategy and deployment Number of countries supported by WAHIT Informatics Team 3 14 Number of documented improvements to national HIS platforms 3 3 EO 4.7 Health information system improved in accordance with One Health approach List of early warning systems for disease surveillance revised in accordance with One Health approach 1 1 Total 74 71 For the operationalization of the policy and strategy of the regional health information system, the CAPS project has enabled the establishment of the regional platform to provide information on health information indicators. The creation of databases in countries through DHIS2 was well acknowledged by KIs. This activity was made possible through the technical assistance provided by WAHO to countries. “In the CAPS project, there is technical assistance that has been provided for strengthening the health information system that I had forgotten. Strengthening of the health information system, particularly in the establishment of the regional platform to make health information available. So, I have a team that has been set up, a team of computer scientists that supports WAHO and countries to strengthen their national health information systems to feed the regional platform.“ – Burkina Faso WAHO KI “Good, the meetings we have had to set up a single data warehouse, including WAHO, it is the only institution that has supported us in any case in this context, just at least for the moment.” – Burkina Faso National Stakeholder KI 55 | Page WAHO has improved the countries’ health information system according to some respondents who believe that much improvement has occurred in the HIS since then. WAHO’s activities have improved not only data quality but also the implementation of harmonized tools for data collection as well as data analysis. “I think that our countries have improved a lot in terms of health information systems; the data are more reliable. Data collection and analysis are reliable; implementation of tools that can be used by everyone, data software that is the same for all countries or accessible to all countries. When we have the results from Mali, Burkina Faso, or Côte d’Ivoire, we know that the database is similar. Data have improved significantly.” – Côte d’Ivoire IP KI ”If there is a partner who is convinced, I say convinced, and who had insisted that the Direction of Statistics can put a tool for collecting, analyzing, and managing health information through DHIS2, it is WAHO that is making things happen. . . . WAHO was a pioneer; it helped the states a lot. If we have adequate funding, we will not even wait for some partners who have kept us waiting a long time. I believe that today it is operational and WAHO has something to do with it. At the Direction¨de la Santé de la Reproduction [DSRE] level also the International Health Regulations have also been well supported by WAHO.” – Niger National Stakeholder KI It’s important to mention that WAHO made real efforts in strengthening HIS at the country level by leveraging resources for other sources. As shown in Table 8, 14 countries benefited from the WAHIT project while the initial goal was to cover only three countries under the CAPS project. However, despite the creation of regional databases, challenges remain in terms of country ownership and completeness of data collected. On top, not all the countries seem to be convinced by the performance of the platform and continue using a different system, which they think is better. Furthermore, some countries are not interested in sending their data on a common platform, so more sensitization is needed. These are the examples from Ghana interviews. “The creation of national databases through DHIS 2 is important. A lot has been done in this regard, even if there are still major challenges, because until then not all countries have joined it, or when they join it, it is not yet systematically and exhaustively integrated within each country and when they also have it is not as systematic as the information that passes from this national level can go up to the regional level to feed our regional database, so there are still many challenges to be met.” – Burkina Faso WAHO KI “We already have a much higher advanced data system than most of the sub-region, so I don’t know whether we have shared experiences. I’m not too sure that we have received any technical assistance on that.” – Ghana National Stakeholder KI Similarly, another significant issue regarding the use of a sub-regional database involving several countries is interoperability. Indeed, how can we ensure that the different systems can talk to each other in order to be able to detect and manage an outbreak in time? This issue has not yet been fully resolved despite the various efforts of WAHO and its partners. “I think that also remains a challenge. How do you make countries’ health information systems interoperable? There’s a debate, which is still ongoing. There is some interoperability but it’s not as sharp as it should be, and we are working on that. I think that’s the next level. So, the idea is that, okay USAID has created this platform, now we have these guys going into the countries to support them. So, how do we get the countries talking to each other, their health information systems talking to each other? How can we get all of you to be reporting to WAHO so that at the regional level for example, if there’s an outbreak of something, we can very quickly inform the Ministers’ of Health to be alert? We haven’t gotten to that stage yet. There is some reporting but it’s not as efficient as it is. But a lot has been done, a lot. A lot of resources have gone into building their DHIS 2 and so on.” – Burkina Faso WAHO KI In addition, despite progress made in HIS strengthening, countries continue to face challenges in terms of completeness, timeliness of data reporting, and maintenance of essential components of HIS such as servers. “Our own challenge now is whenever our server goes down it is a challenge for WAHO to get feedback. But when the server is always on, at the end of every month, or every week when we have our surveillance data, they just sync to theirs.” – Sierra Leone National Stakeholder KI The opinion of KIs on WAHO’s role in strengthening the HIS and improving reporting summarize on the words cloud below. 56 | Page Figure 9: Words cloud highlighted the role of WAHO in strengthening the HIS and improving reporting Question 4: What are areas or domains of WAHO’s capacity that have improved as a result of USAID/WA support? WAHO’s leadership in advancing health policy is well recognized by the majority of KIs both in the individual interviewed individually or the FGDs. Its leadership in promoting health policy in the region is undeniable as it has the convening power to bring together ministries of health in the region. Appreciation of WAHO’s leadership in advancing health issues and outcomes in the region Respondents positively appreciate WAHO’s leadership in health. Indeed, according to most of them, in recent years WAHO has positioned itself as a political organization on which partners can rely to advance their agendas. The proof is the increasing number of projects implemented by WAHO. Donors recognize WAHO’s political leadership, which has a comparative advantage in the sub-region that no one has and the legitimacy and credibility to be an ECOWAS body. They therefore collaborate with WAHO to facilitate the coordination of activities at the regional level and find it very useful. “WAHO's role in the region is very important, as it is the main technical and financial partner. It also facilitates the experiences sharing between different countries and intervenes in advocacy.....We have projects with WAHO as technical and financial partner and also does advocacy with partners for resource mobilization for DHIS2 and health mapping.... I think that we cannot evaluate WAHO only for its technical or financial support, but also we must not forget its contribution to the improvement of health systems – Cape Verde National KI ”Basically, you know, the way you will know they are strong is even the way they attract donor funding. If donors put funds in any organization, you do not use it properly, and there is no clear direction, they won’t go there one more time. When you see something good, that is where people are happy to say ah we need to put in more, they are utilizing this money and we have seen results and all that. That is a good indicator that things are really working with WAHO.” – Nigeria ECOWAS Stakeholder KI 57 | Page Some interviewees linked the strengthening of WAHO’s current leadership to the Executive Director. Indeed, they believe that in recent years, the organization has benefited from enlightened leaders who have been able to breathe new life into it. “For three years now, WAHO’s leadership has been strengthened. Now it is strong leadership. The former CEO who was there was able to put, say, his weight. He put WAHO on, say, at a given level and thanks to him WAHO is known as . . . a lead institution in the region.“ – Burkina Faso WAHO KI “[The new DG] has achieved a lot within a short time. Not only is he changing WAHO to a more proactive organization [and] thinking private sector, thinking in new ways, [but] he has also raised a lot of funds to deal with the most pressing issues in West African health. Like I said, maternal mortality, malnutrition, and, of course, poverty is behind all these. And he has been working with all the departments in ECOWAS as we are trying to find solutions to some of these issues. I am predicting that by the time we finish our term, WAHO will be doing extremely well.” – Nigeria ECOWAS Stakeholder KI Others KIs, however, believe that despite WAHO’s undeniable leadership as a key player in health promotion, it is unable to get all partners to support its cause. So, challenges remain and WAHO still needs to struggle to get unanimity among partners, to be able to hold the legitimate role of coordinating overall health strengthening activities in the region. “For the moment WAHO is looking for itself. In relation to this leadership, it is my responsibility to say that despite the fact that we have been recognized by almost all donors and technical and financial partners for the role we can play and the fact that we are essential today as a major player in health promotion, not all WAHO partners want to do well at the regional level. There is a little bit of this problem, so we are recognized but also not everyone agrees, so it is important to be pragmatic and realistic about this situation.” – Burkina Faso WAHO KI The areas or domains of WAHO’s capacity that have improved as a result of USAID/WA support are summarized in the words cloud below Figure 10: WAHO’s capacity that have improved as a result of USAID/WA support While WAHO’s predominance in the sub-region in health promotion is not disputed in view of its increasing involvement in the implementation of country health programs and capacity-building, some interviewees find that its effectiveness is limited due to its low representation in countries. 58 | Page “I admit that there is a growing sense of WAHO being involved in country programs, organizing workshops, [and] capacity-building. WAHO is on programs and themes to support countries. I don’t want to qualify them too much because I’ve already given my impression on WAHO. I said that WAHO has very little representation in countries. Which means everything. There are aspects on which WAHO cannot be effective due to its level of representativeness. It is on very specific programs and projects; however, organizations that depend on the state are very complicated. They can only encounter difficulties. If the states do not give the money, there is nothing you can do. WAHO is in Bobo-Dioulasso instead of being in a capital city.“ – Côte d’Ivoire IP KI WAHO owes this leadership to its ability to collaborate and convince other partners working in the health sector in the West African region in the areas of capacity-building, organization, health care delivery, and research. This collaboration allows WAHO to be more attractive and to generate more projects and therefore resources for the benefit of the ECOWAS region. WAHO’s areas of expertise/capacities that have improved with USAID support from 2012 till now Below are WAHO’s expertise/capacity areas that have improved with USAID support: • Improved governance, awareness, and visibility of WAHO in the sub-region and recognition of the role of WAHO by TFPs. Indeed, WAHO achieved more accountability and more transparency because donors’ procedures and requirements led to more rigorous overall management. “It can be said that in the area of governance, for example, giving WAHO the opportunity to implement projects such as CAPS is a substantial support from USAID, and this project has brought a lot of progress. Perhaps I would say that without it we would not be here today, equipped enough as a regional institution to lead this kind of initiative and in it fits a little into the capacity-building field of WAHO. Today if you take all the best practices that have been shared at the sub-regional level, which have really been the subject of much appreciation both by countries and by technical and financial partners and here again to raise WAHO’s visibility within the sub-region in relation to the role that people expect WAHO to play in this area. I will talk about both, but I can say notoriety again I will put that on the side of impact and all that again in the coming years, it is much more the visibility of WAHO and also the recognition of WAHO’s role by other stakeholders.“ – Burkina Faso WAHO KI • The establishment of focal points in countries serving as liaison between WAHO and ministries of health has contributed greatly to the visibility and success of some WAHO’s activities, although the performance of the latter is often considered insufficient. Indeed, this performance is person-dependent and closely linked to its dynamism and commitment. Furthermore, the focal points are sometimes deemed to be very busy and not able to devote enough time to WAHO’s activities. “If we already have the support, it means that they are accessible and having a focal point at the ministry level makes it easier, even at the level of coordination. Of course, effectiveness also depends on the person who is there; it requires the leadership of the person who provides the link between WAHO and the actors who are on the ground. Since WAHO is not represented as an institution here in-country and the result are in relation to those who are represented, I think that if we make the balance we find that even if WAHO is not there, they are as effective as those who are there. It is not the presence that is the problem, it is the leadership, the capacity of the institution, of those who represent.” – Guinea FGD National Stakeholder KI • Strengthening the leadership and portfolio of activities in terms of the number of programs implemented in the countries of the sub-region. Leadership, communication, project management, and M&E have improved. With the CAPS project, for the first time in the history of WAHO, a dashboard for monitoring the indicators of the project was elaborated, and that has been extended to all the other projects within the institution, so that’s a big plus with more accountability for people involved in different activities. The support of USAID/WA was acknowledged for this achievement that would not have occurred otherwise. Indeed, WAHO's capacity to implement large projects like CAPS with strong accountability have been strengthened through successive supports of USAID. This opened the door to many other partners to start collaboration with WAHO and this shouldn’t be ignored. “I know that WAHO had excellent leadership from 2014 to last year. We have the mandate of the Nigerian (I forgot his name), which had excellent leadership that allowed WAHO to be more visible, to raise many resources and to 59 | Page create programs that are available to countries. That is very strong leadership. WAHO was only visible during this period.” – Niger IP KI • Identification and scaling up BP. This is a major achievement as part of the support of USAID that support the organization of BP forum. For scaling up of BP, there is a system established to accompany countries to identify and promote their best practices. • WAHO has strengthened its resource mobilization capacity by convincing the donors to release funds. “We have a study that was conducted last year, and that showed that there is a lot of practice in the health domain, a lot of things have been improved, including visibility [and] funding. . . . We have had in less than two years, almost 20 projects funded by partners.” – Burkina Faso WAHO KI • Advocacy and communication skills have been improved. Communication is in the broadest sense, taking into account aspects of day-to-day operations, communication platform, information sharing up to travel arrangements. “Yes, I know communication has improved significantly. Although as liaison officer I get too much email from them. . . . But definitely they have been providing a bit of support to accommodate that, internet services, phone calls, etc. They have improved in many ways by actually following up—they send in monitoring and evaluation teams, they send in auditors, and also the focal point will follow up on those activities and make sure technical and financial reports are prepared for onward transmission to WAHO. They are always in touch. The program officers who are in charge of some of the programs write through the bilingual secretary, so you always know whom you are dealing with. They respond to emails on a daily basis. The travels are well organized. We are not complaining in that area. Information sharing, WhatsApp platform created for liaison officers, and also WAHO’s Website is also informative to the extent that some people are able to apply for some of the vacancies.” – Gambia National Stakeholder KI “Communication and health information have been strengthened. Even at the advocacy level, it has been strengthened. Because USAID has still helped WAHO to make a lot of advocacy at the level of other donors.” – Burkina Faso WAHO KI • The contribution in strengthening the health information system in the region. “I think that for the strengthening of the health information system it is still quite visible, eh! Since initially to get the data from the countries it was not easy. It took the setting up of this platform for me to think that this is one of the results.” – Burkina Faso WAHO KI The summary of WAHO’s areas of expertise and capacities that have improved with USAID support from 2012 through the end of CAPS project is contained in the world cloud below. 60 | Page Figure 11. Expertise and capacities of WAHO that have improved with USAID support since 2012 Current or potential challenges of WAHO’s leadership in supporting health policies in the countries The challenges mentioned by the respondents are multiple and affect all health sectors but also concern WAHO’s actions in implementing support activities to improve countries’ health systems. The challenges cover multiple areas, including institutional organization, capacity to support the process leading to reach CAPS project outputs, and the quality of the outputs. The following challenges have been identified by the different respondents and should be considered as potential future areas of improvement for WAHO to assume full leadership role in the health sector in the ECOWAS region. • Insufficient mobilization of resources for the implementation of health policies. KIs suggest that WAHO should continue strengthening its resource mobilization capacities by seeking more financial partners to support countries because this weak financial capacity influences its visibility and leadership. “Financial too, they are very weak. Maybe that’s why their presence is not felt. Financially, they are very weak, because you won’t get anything from them.” – Ghana National Stakeholder KI • Poor performance in carrying out the missions assigned to WAHO due to several factors ranging from capacity to structural organization. “Our role is to ensure that we support the countries to implement all these strategies, policies, [and] resolutions that they would have signed up to at the World Health Assembly. The question then arises, are we really playing that role effectively? To some extent we are but I think there is limited human resource capacity and that limits our role. We are not as strong as WHO is for example, within the region. That seems a bit contradictory but that is the reality on the ground but WHO has country offices, we don’t. We are only represented by liaison officers, so to what extent can [the liaison officer] look at all the activities we are implementing in the countries? So generally, we are supposed to be the strongest but for now, no [we are not].” – Burkina Faso WAHO KI • Weak promptness in responding to countries in terms of support and in the fight against epidemics such as Ebola. Indeed, the KIs (even WAHO representatives) believe that WAHO should be much more responsive and prompter in the event of a health emergency and even for any other support requested by countries. This efficiency, however, implies simplifying procedures to be taken to provide a quicker response when 61 | Page needed. “But we have many challenges, many organizational challenges, to be able to respond when countries need it; effectiveness in responding to countries. Whether it is for epidemics, whether it is for endemics, whether it is even for other supports, the response speed must really be improved. So, we need to have the mechanism that allows us to respond as soon as the need is expressed, and that is important. It is very important.” – Global KI • The challenge is to ensure the implementation of all the recommendations resulting from the meetings attended by the country, to ensure that each focal point can have a dashboard to monitor the recommendations and to ensure their implementation. • The insufficient number and capacity of WAHO team member to not only cover the 15 countries but also to cope with the increasing number of different projects. “Challenges are resources—human resources—because I think that today WAHO is “under-staffed.” There are 15 countries to cover and programs often suffer in the implementation on the ground. Also, the financial resources that must accompany the deployment of these human resources. Because even if we want to go to the countries, WAHO does not have the human and financial resources to deploy these teams, and that is a problem.” – Niger National Stakeholder KI • Insufficient communication on WAHO activities. A lot of improvements have been made but it is still weak. Indeed, there is little communication on WAHO activities, which means that some key activities contributing to institutional visibility are not well known. Indeed, the latter are often unknown to beneficiaries and even to focal points who could attribute ownership to a third-party institution. WAHO should well spread information in terms of good practices as well as other activities conducted with countries. “The first challenge would be for WAHO to ensure that the focal point has all the information regarding all WAHO interventions. There will need to be one or more interlocutors who will transmit all WAHO information to the focal points. . . . I was saying that the focal point should first have information on WAHO’s activities in the countries. There is a plan that has always communicated to us, but not all the information is at the level of this plan. We need to gather all this information and be well informed so that we can report back.” – Mali National Stakeholder KI • The low institutional visibility of WAHO to be able to fully assume its leadership. This requires representativeness in countries and the establishment of national offices. The lack of a country office, as in the case of some organizations such as WHO, affects WAHO’s visibility in countries. This visibility also requires an intensification of communication. KIs suggest that WAHO should have a real representation (bureau) of in each country with the consequent resources. “The problem of WAHO’s representativeness must be solved first. I don’t see much WAHO in the country. WAHO must be present in all health sectors.” – Côte d’Ivoire IP KI “The other challenge is to reposition ourselves better—we need to work a little bit on the visibility of the institution, so that we can be a little more visible. We are focal points, who are not WAHO officials, but representatives of the Ministry of Health who have limits. So, I think that’s the two things I can bring to your study.” – Global KI • The lack of physical representation of WAHO in countries, as do other organizations with similar missions. “The fact that WAHO does not have physical boots on the ground is a constraint in itself. So, they have had to rely on a liaison person who is often not a health person himself to report and coordinate for them. I think rectifying that will go a long way. Even if it is a local person who is a WAHO focal person will go a long way. But I cannot deem WAHO’s presence as being leading in the country. Because clearly there are multiple stakeholder’s collaborative meetings that they will not be a part of it. So that itself should say otherwise.” – Liberia National Stakeholder KI • The real involvement of WAHO in the scaling up of good practices. Beyond all the support currently being promoted, support alongside countries is desirable for the success of these activities. WAHO could also document experiences of scale-up of good practices and share among countries and get more involved in 62 | Page scaling up activities on the ground. “Well I think one challenge that they face, which is very similar to WHO in many ways, is that they are a regional organization, so their role is really for advocacy, streamlining policies, setting priorities, etc. But WAHO staff themselves are not on the ground implementing. So, one challenge has been getting that advocacy and prioritization down to their partners who are on the ground implementing. They do that through the Good Practice Forum and possibly other meetings that they can convene. But that is always going to be a challenge when you’re dealing with a regional body like WAHO. They themselves aren’t actually on the ground implementing or scaling up.” – Global KI • The challenge of using the evidence to convince country governments. For this purpose, WAHO must collaborate with research institutions to produce more evidence to guide and inform countries’ health systems. “In the area of health challenges, what WAHO can do, and as I said earlier, WAHO can work with research institutions to have, in any case, something that is evidence-based. So, if it works with research institutions, to be able to do research and find evidence, therefore, on good practices, it will be able to convince governments to put them into practice.“ – Burkina Faso IP KI “WAHO should really train to create a research culture. How can the result be used at the country level? I have reviewed [country-level research results], they have done many things just like that that interest countries, with technical support sheets. But then he has to use this for planning. I can see the national statistics that are described; for example, you won’t see a single WAHO report in it as a reference, [and] that’s a problem. While we go to get the WHO reports [and] the UNFPA reports, we ourselves (our reports) are in them. Their institutional diagnosis, diagnosis of medical practice, [and] of health practice in the country must be well known.” – Global KI • Challenges related to data validity and veracity, poor data quality, data analysis and the use of biometric data. “We still have challenges with the validity and veracity of the data. We need more help from WAHO regarding the validity of this data. We collect the data, but WAHO needs to have more eyes on it so that we really have complete and quality data. In this case, WAHO must be involved in the validation of the data produced. Speaking of challenges, we also need to achieve the bio metrification of data, which will enable us to carry out advanced analyses of this data. It is true that the actors have been trained for the collection, but the use of paper does not guarantee the quality of the data. So, if the patient record is biometric it will really move the health sector forward.” – Togo National Stakeholder KI • Develop strategies to capture real needs in different areas (advocacy, communication) of the countries. There is a problem related to the focal point. This has to do with the profile, the anchoring and the powers (headquarters, communication, locomotion, having a secretariat) of the latter. Indeed, some focal points are not health technicians or are often located at a more political than operational level, which explains the lack of performance. In addition, there is a problem of stability of their mandate with a quick staff turnover leading to a perpetual restart. “There is a focal point, but this focal point is attached to the level of the Minister’s office and the office is so unstable that we change the focal point almost every two years. So, this is a focal point that has had a duration of one year and after that it’s a new one coming up. . . . The focal point position has been put on a purely political level where things change depending on the ministers, so that’s what’s really problematic at the moment.” – Senegal National Stakeholder KI • WAHO’s procedures are deemed to be heavy and not allowing quick response when a request is issued no matter it being urgent or not. • The challenge of national ownership of WAHO actions. WAHO must succeed in getting the actions to be carried out by the governments, by the ministers of health of the ECOWAS countries. This requires strong 63 | Page political commitment and support from opinion leaders. “[My first suggestion] is to hold countries more accountable for implementing the policies they adopt and the practice and declarations that they signed on to. So WAHO should hold countries accountable and encourage countries to be more transparent in the health governance. Those are the two things I would suggest. That’s what I would say.” – Ghana IP KI • Challenges in developing standardized regional policies and strategies. “The first challenge we see in Benin is how to have standardized regional policies and strategies since these problems are cross-cutting with these countries. We see that we have good things happening just in our neighbours and we have no information, whereas if we had a unifying framework we would manage better and avoid wasting time and resources while good practice is on the side.” – Benin National Stakeholder KI • The challenge of the availability of quality medicines in the ECOWAS region. “The other challenge, the Minister of Health has made it a priority, how to make quality medicines available in the ECOWAS region by group orders. It will be cheaper for us to facilitate later, [and] it will facilitate the traceability of these products in the ECOWAS region. It was even proposed at the last ECOWAS Health Ministers’ Retreat held in Accra on March 9, and they entrusted WAHO with the responsibility of exploring what could be set up to order certain products on behalf of the sub-region.” – Benin National Stakeholder KI • Challenges related to data availability at the community and private sector levels “The first challenge for me is to strongly support obtaining data completeness at the community level and obtaining data from the private sector and to develop more modules [such as] the hospital management module.” – Benin National Stakeholder KI The Current or potential challenges of WAHO leadership in supporting health policies in the countries Figure 12: Summary of WAHO’s current and potential challenges in supporting health policies in the countries The above words cloud WAHO’s highlight the current and potential challenges through the various wording provided by the KIs and that is consistent with the text above. 64 | Page LESSONS LEARNED Remarkable achievements have been made under CAPS’s project implementation. During the period of the CAPS project implementation, WAHO played a key role in coordinating, promoting and harmonizing health policies among member states. Main achievements, under CAPS’s project, have been performed in the following areas: − Health policy harmonization: WAHO played an effective role in harmonizing health policies by developing, updating, and sharing guidelines and other strategic documents with member states and its role in the harmonization and integration of ECOWAS countries to address health issues increased during the period of the project with an increased number of resolutions validated by the ECOWAS Assembly of Health Ministers. − Best Practices scaling up: Promotion and scaling up of Best Practices (BP) at the country level remains one of the main areas where WAHO has achieved key results. WAHO played important and remarkable roles in: • Identifying and promoting the scale-up of best practices by ECOWAS countries; • Developing and disseminating methodological tools for documenting good practices and building institutional capacity to use them—WAHO supported countries to understand these tools and apply them systematically whenever they want to create or document a best practice; • Creating a platform where countries meet to exchange their respective experiences on best practices; • Providing continuous technical and financial support for countries in implementing their good practices; • Coordinating and working in synergy with other technical and financial partners (TFPs) in different countries for the scaling up of good practices; • Harmonizing the scale-up of some good practices in the ECOWAS region, which has provided a good experience in rationalizing the use of resources. − Advocacy and communication: Through the CAPS project, WAHO operationalized the advocacy and communication plan that was developed as part of USAID’s Leadership Management in Governance (LMG). Moreover, a network of communication managers from the 15 countries at the level of ministries of health, in addition to media representatives, has been created, supported, and strengthened. Communications materials such as fact sheets, documentaries, and advertorials were developed to ensure that all activities carried out at the regional level were well advertised and disseminated. These efforts contributed to reinforce WAHO visibility within the region and to facilitate collaboration with Technical and Financial Partners at countries and regional level. − Supply chain and commodity security: As part of the CAPS mandate, WAHO has helped strengthen countries’ capacities in commodity safety–security/supply chain management. WAHO has provided support in terms of training, equipment, and technical assistance. They have also helped to harmonize standard documents in the field of pharmaceutical regulation and traditional medicine. Hence WAHO helped countries to develop guidelines and regulations, whether at the pharmacy level, the nursing level, or the level of human resource development in health. As part of supply chain strengthening, WAHO provided financial assistance for the purchase of contraceptive products and capacity-building in supply chain management and commodity security. In addition, WAHO played an important role in strengthening supply chain management by integrating health commodities to national health information systems via extensive use of District Health Information Software 2 (DHIS2). − Promoting Public–Private Partnerships (PPP): As part of the public–private partnership (PPP) program, WAHO organized the first forum on PPP in health in 2017 in Accra to help countries to implement PPPs and also to set up a PPP unit within the ministries of health. Main actions taken are the following: • Facilitating meetings between donors and member countries to capture funding opportunities • Developing a strategy document to help countries implement PPPs and to set up a PPP unit within the ministries of health. • Organized the first forum on PPP in health 65 | Page − WAHO’s role in strengthening the HIS and improving reporting: through the CAPS project, the countries’ health information system has seen much improvement. A regional platform to provide information on health information indicators, including the creation of regional databases has been established. WAHO’s technical assistance through CAPS project contributed to improve not only data quality but also the implementation of harmonized tools for data collection as well as data analysis. In summary, through the CAPS project, WAHO’s leadership in advancing health policy and outcomes in the West Africa region has been strengthened. WAHO has positioned itself as a political organization on which partners can rely to advance their agendas. Donors recognize this WAHO’s political leadership, which has a comparative advantage in the sub-region and engage effective collaboration with WAHO to facilitate the coordination of activities at the country and regional level and find it very useful. However, despite progress made, efforts need to be maintained to address main challenges encountered during the life of the project. These challenges are, among others: − Insufficiency of human to address timely countries’ needs; − Insufficient visibility and representation and country ‘’a single man representation’’ level limiting its role and reach; − Timely communication between WAHO and countries; − Resources mobilization to better address countries’ needs; − Insufficiency in the follow up and implementation of recommendations from regional meetings. LESSONS LEARNED FROM EVALUATION AND LIMITS Conducting a large evaluation such as this one requires dexterity and a good working environment. Lessons learned can be summarized as follows: • Conducting such an evaluation in 15 countries with different data collectors with different experience and in three different languages requires much organization and communication. • Coordinating several teams of data collectors in 15 different countries simultaneously is one of the most challenging aspects, essentially for communication. • Support of some WAHO staff and Liaison Officers in-country in facilitating the data collection process (including setting appointments with KIs as well as logistics for the FGDs) was invaluable. • Willingness and engagement of RGs who were keen and proactive during the data collection in their respective countries was essential for effective data collection. • Leveraging personal and professional networks was a good asset to reach some KIs and organize FGDs. • Initial time allocation for data collection per country was short, because it didn’t take into account the availability of KIs and their absence. More time should be allocated for data collection when planning for similar evaluations in the future. The evaluation has some limitations that could have affected the data collection process and the findings, including the following: • Lack of completeness of programmatic data, often due to delay in reporting project’s activities was a challenge for monitoring indicators of the dashboard and triangulating programmatic data with information collected from individual interviews and FGD; • Stability and availability of WAHO liaison officer: some LOs were new and hesitated to participate in the evaluation or were too busy to facilitate the data collection process; • Difficulty in organizing FGD with KIs in health in some countries: the evaluation team faced some challenges in mobilizing key partners in health for an FGD due to their busy schedules; • Reluctance of some KIs despite the email sent by USAID/WA: they claimed they were waiting for an official introductory letter from WAHO before they could participate in the interview; • KIs were not always well knowledgeable about CAPS technical components; • Difficulty in undertaking data collection activities in Lusophone countries (Cabo Verde and Guinea Bissau) due to language barriers—the evaluation team found it difficult to find someone who spoke Portuguese and could transcribe the interviews into French. 66 | Page The listed limitations are minor and could not alter the findings in any way. Indeed, the sample size was large enough to overcome the shortcomings identified. Therefore, the results of this evaluation are valid and appropriately meet the set objectives. KEY RECOMMENDATIONS TO WAHO FOR LEADERSHIP DEVELOPMENT Under the CAPS‘s project, WAHO’s role and leadership have been strengthened through the multiplicity and diversity of WAHO’s activities/interventions and partnerships. WAHO has been acknowledged by most of the national stakeholders and partners at the country and regional levels as the legitimate institution in charge of coordinating health policies in the West Africa sub-region. WAHO played a significant role in coordinating, promoting and harmonizing health policies among member states. This leadership has been effective with an increased number of resolutions validated by the ECOWAS Assembly of Health Ministers and growing partnership with the regional body to address health issues such as Ebola outbreak and/or collaborating with regional bodies such as the regional Animal Health Center to develop a regional One Health program. To reinforce this leadership and consolidate the achievements, efforts must, however, be made to overcome some challenges that could mitigate this leadership. 1. To what extent is WAHO considered a leader in advancing health issues and outcomes in the region, especially in the area of policy harmonization and advocacy, in promoting public-private partnerships (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, and key IPs in the region)? • Carry out institutional strengthening in general through greater country contribution to enable WAHO to be autonomous and be able to play its role. • Strengthen the quantity and quality of WAHO staff in order to: 1) better assist countries in carrying out their activities and respond timely to their requests; 2) establish an effective WAHO representation (bureau) in each country with the necessary and adequate staff resources for the implementation, monitoring and coordination of these activities in the countries. Making use of the in-country professionals will be helpful to carry out activities at the country level. • Strengthen communication and advocacy activities to :1) increase greater visibility of WAHO; and 2) for resources mobilization; • Focus on a few specific and priority areas or problems common to all countries to enable WAHO’s support to be more effective and useful. • Set up a system to follow up on resolutions adopted during various meetings in order to ensure their effective implementation by countries. WAHO should increase advocacy to get countries’ support in the implementation of these resolutions. • Encourage countries to integrate WAHO’s activities into their workplans to promote ownership instead of standing alone as parallel activities. 2. How has WAHO led and supported the scale-up of best practices in the region (e.g., mapping of current high-impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale-up of interventions, health information systems integration, and capacity strengthening)? Promoting and scaling up Best Practices (BP) remains one of the main areas where WAHO has achieved key results. Among the key achievements are: 1) development and dissemination of methodological tools for documenting good practices and building institutional capacity to use them; 2) creation of regional platforms where countries can meet and exchange their respective experiences on BP; providing technical and financial support for countries to implement their good practices. Over the life of the project, WAHO strengthened the countries’ capacities in commodity safety-security chain management by helping countries to harmonize standard documents in the field of pharmaceutical regulation and by helping countries to integrate health commodities management into national health information system via DHIS2 software. In addition, WAHO helped countries to develop a common strategy for the implementation of Public-Private Partnership (PPP). Moreover, WAHO provided technical assistance through putting in place a regional platform on Health Information System (HIS). In summary, through CAPS project, WAHO made 67 | Page remarkable achievements in the area of BP sharing and scaling up. Therefore, to consolidate these achievements and to enable WAHO to fulfil its role, the following actions need to be taken: • Ensure in-country and wider dissemination of approved, adopted and accepted BP. • Ensure that WAHO’s in-country POC are fully aware of the ongoing activities and ensure local coordination of implementation. • WAHO should continue strengthening its resource mobilization capacities by seeking more financial partners to support countries to implement, document and evaluate their BP; • Ensure the implementation of all the recommendations resulting from the meetings attended by the countries, to ensure that each focal point can have a dashboard to monitor the recommendations and to ensure their implementation from BP forum; • WAHO should help countries to document experiences of scale-up of good practices and share them among countries and also get involved more in scaling up activities on the ground. 3. What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale￾up; national policy changes for FP and HIV/AIDS programs; improved commodity security; public and private partnerships; and health information systems to increase investments in health and improve reporting? Through the CAPS project, concrete progress has been made in the area of advocacy and communication. WAHO operationalized the advocacy and communication plan that was developed as part of USAID’s Leadership Management in Governance (LMG). Furthermore, a network of communication managers from the 15 countries at the level of the ministries of health, in addition to media representatives, has been created, supported, and strengthened. Communications materials such as fact sheets, documentaries, and advertorials were developed to ensure that all activities carried out at the regional level were well advertised and disseminated. In spite of this achievement, actions need to be taken to improve WAHO communication strategy for better visibility and more fundraising. The following actions have been recommended: • Strengthen WAHO’s communication system to significantly improve the visibility and leadership development. • Reinforce advocacy and communications activities to increase WAHO’s funding for health services. Like every other organization, WAHO would have to rely on member states or rely on bilateral donors to carry out projects and it has the strength and opportunity to do that. WAHO could also support countries in their resource mobilization effort with external donors to help countries scale-up good practices. • As a regional body, WAHO has to put forward a strong multi-country proposal to funders, one that is more effective than having small contributions. • Strengthen countries’ capacities to develop PPPs. In this regard, WAHO could assist countries in identifying areas of intervention or private individuals with whom they can collaborate and help them set up projects to present to private individuals. • Continue efforts to strengthen communication and advocacy to raise awareness of WAHO’s activities and increase success. • Develop communication activities for a wider dissemination of resolutions and recommendations adopted by Assembly of Health Ministers (AHM); • Pursue advocacy and communications activities to engage countries in implementing resolutions and recommendations adopted by the AHM. 68 | Page CONCLUSION WAHO is considered as a natural and legitimate leader in the ECOWAS region as far as health is concerned and it fully deserves this recognition. It is in the best position to fill that role given its proximity to countries and its ability to communicate and work within countries. It is unique by being a regional body that is owned and operated by the region. It has the power to convene and engage Ministers of Health or people around its activities like Best Practice Forum and other policies implementation. It has leveraged that to develop some strong partnerships and work very closely with some implementing partners and private sector actors. WAHO achieved a lot during the last years by fostering its own institutional capacity-building (governance, communication and advocacy, PPPs, etc.) and being able to increase technical assistance and other support and promote more partnership and the harmonization of health policies in ECOWAS countries. This gives WAHO strong recognition of its leadership by the other organization operating in the sub-region. This would have not been possible without the CAPS project, which is recognized by all to be the driving force of all these achievements. However, the field of health is where all countries have problems going from capacity-building, maintenance of infrastructure and equipment, health care management, and epidemics control. Diversities between the countries should also be noted because they have different levels of health system development demanding adequate support based on thorough needs assessments. Therefore, WAHO must continue to be very active in relation to activities in the 15 countries, which will allow it to improve according to the realities of each country. It must remain close to these countries to address their needs and should also consider the private sector. The good practices identification and scale-up in the regions could be considered as the best output of CAPS project and should be maintained with continuous support to countries for dissemination of good practices. The performance evaluation showed that at the sub-regional level, WAHO is very effective and well-structured in terms of standards and policies as well as supporting countries’ health systems at various level going from organization to resource mobilization and governance. WAHO is an important tool, which has seen its leadership evolving tremendously in the recent years, certainly as a consequence of USAID funding and another partners’ support. However, WAHO needs to consolidate its role in helping countries to better coordinate and leverage partners’ interventions at countries level. In addition, efforts need to be made for helping countries to engage fundraising activities for supporting health policy implementation, Moreover, for increasing its visibility and to better address countries’ needs, WAHO has to reinforce its staffing and develop branding of its projects. 69 | Page ANNEXES Annex 1: questionnaires. WAHO Performance Evaluation FORMATIVE KEY INFORMANT INTERVIEWS - NATIONAL STAKEHOLDERS BACKGROUND & INFORMED CONSENT Date: Country: Location: Start Time: Name of Interviewer: _____________________________________________ Respondent Profile: Name Title/Organization Contact information (email/telephone) Gender 70 | Page 1: Background Information 1.1 Can you introduce yourself? (Name, position/role and institutional affiliation) 1.2 In which way is your institution working in implementing the national health policy? Probes: Capacity building? Advocacy and Communication? Health Information System management? Best Practices scaling up? 1.3 Does your organization collaborate with other institutions (at national or regional level) to support the government in implementing its health policy? If so, what are these institutions? and in which ways are you collaborating? 1.4 Have you heard about WAHO and the CAPS Project in ECOWAS countries? If yes, what do you know about the CAPS project activities in ECOWAS countries? II: Achievement: WAHO’s Leadership Capacity Strengthening 2.1 Which organizations are supporting (TA and funding) the government in terms of health system capacity strengthening? 71 | Page 2.2 In which areas do these institutions support the government in implementing its health policy? Probe: Health policy harmonization (regarding legal and regulatory obstacles, promoting regional strategies)? Best practices scale up/sharing? Health Information Strengthening-HIS? Advocacy and Communications? Public-Private Partnership-PPP? Supply Chain strengthening? 2.3 Has WAHO played a role in strengthening the implementation of the best practises? (if WAHO not listed in 2.1). Which role and how? Probe: Health policy harmonization (regarding legal and regulatory obstacles)? Best practices scale up? Health Information System (HIS) Strengthening? Advocacy and Communications? Public-Private Partnership (PPP)? Supply Chain strengthening/commodities security? 2.4 Has WAHO initiated actions for coordinating interventions/leveraging resources between key Implementing Partners at national/regional levels? Which role and how? 2.5 Which organizations would you say are leaders in advancing health issues and outcomes in the West African region? Probe: Is WAHO a leader? If yes, how would you rank WAHO among others and why? III: Achievement: Scaling Up of Best Practices, Advocacy and Communications Strategies in the Region 3.1 What best practices in health is your country currently scaling up? Which organizations have assisted that effort? Probe: What high impact practices have you heard of? In what ways have these best practices been scaled up? 72 | Page 3.2 Has WAHO played a role (if not listed)? If so, in what ways has WAHO influenced the scaling up of best practices? (specify) 3.3 What has been WAHO’s role in advocacy and communications efforts, with respect to best practices scale up? Probe: Concerning domains like: • National policy changes and harmonization for FP, MCH and HIV/AIDS programs? • Improved commodity security? • Public-Private partnerships? • Health information systems to increase investments in health and improve reporting? • Data utilisation to support advocacy and communication activities and promoting measurement and accountability in health? • Domestic fundraising to ensure sustainability? IV: Achievement: WAHO’s Capacity Improvement and Role in PPP and HIS? 4.1 What has WAHO’s role been in promoting/strengthening regional donor, public and private partnerships? Probe: Has it expanded from 2012 – till now? How? 4.2 How successful is the operationalization of the West Africa Regional Health Information System (HIS) Policy and strategy? Probe: What has been achieved? What are the challenges? (As far as M&E, capacity and advocacy for integrated HIS strengthening are concerned) 73 | Page 4.3 How do you rate WAHO’s leadership in advancing health issues and outcomes in the region? Probe: Does WAHO play a strong, moderate or weak role? V: Client Needs Met: WAHO’s CAPS 5.1. To what extent does the TA and funding provided by WAHO address the issues that the government is facing regarding best practices implementation? Probe: What can be done by WAHO to be more effective in identifying needs? And in satisfying those needs? 5.2 What do you think WAHO could do to better address the scaling up/sharing of best practices in the region? 5.3. What do you think WAHO could do to better address the implementation of best practices in health information and data sharing the region? 5.4 What do you think WAHO could do to better address the implementation of best practices for advocacy and communications in the region? 5.5 What do you think WAHO could do to better address the best practices implementation of PPP in the region? 74 | Page 5.6 What do you think WAHO could do to better address the implementation of best practices for commodities security/supply chain strengthening in the region? VI: WAHO: Operational Effectiveness 6.1 What are the challenges or potential challenges in WAHO’s leadership in supporting health policy implementation in country? 6.2. What can be done to improve WAHO’s leadership in capacity building and make it the lead institution in the health sector in West Africa? 75 | Page CAPS WAHO Performance Evaluation FOCUS GROUP DISCUSSION/CONSULTATIVE MEETING GUIDE-IMPLEMENTING PARTNERS BACKGROUND & INFORMED CONSENT Date: Country: Location: Start Time: Name of Facilitator: _____________________________________________ Participants’ Profile: N Name Title/Organization Contact information (email/telephone) Gender 1. 2. 3. 4. 5. 6. 7. 8. 76 | Page I: Background Information: Introduction and Overview 1.1 Can you introduce yourself? (Name, position/role and institutional affiliation) 1.2 In which ways do your respective institutions support the government in implementing its health policy? Probes: Capacity building? Advocacy and communication? Health Information System management? Best practices scaling up? Commodities Security? 1.3 Have you heard about WAHO and the CAPS Project in ECOWAS countries? If yes, what do you know about the CAPS project activities in ECOWAS countries? Are your institutions collaborating with WAHO? II: Achievement: WAHO’s Leadership Capacity Strengthening 2.1 Which organizations are supporting (TA and funding) the government in terms of health systems capacity strengthening? In which ways do these organizations support the government to implement its health policy? Probe: Policy changes/harmonization, Best practices scale up? HIS? Advocacy and Communications? Public￾Private Partnerships (PPP)? 2.2 Has WAHO played any role? (if WAHO not listed in 2.1). If so, which role and how? 77 | Page 2.3 Which organizations would you say are leaders in advancing health issues and outcomes in the region? Probe: Is WAHO a leader? If yes, how would you rank WAHO among others and why? III: Achievement: Scaling Up of Best Practices in the Region 3.1 What best practices in health is your country currently scaling up? Which organizations have assisted that effort? Probe: What high impact practices have you heard of? In what ways have the best practices been scaled up? Did WAHO play a role (if not listed)? 3.2 What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale up? Probe: Concerning domains like: • National policy changes and harmonization for FP, MCH and HIV/AIDS programs? • Improved commodity security? • Public and private partnerships? • Health information systems to increase investments in health and improve reporting? • Domestic fundraising to ensure sustainability? IV: Achievement: WAHO’s Capacity Improvement and Role in PPP 4.1 How has WAHO’s capacity improved as a result of USAID support from 2012 – till now? Probe: WAHO institutional capacity and leadership capacity improvement (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region) 4.2 What has been WAHO’s role in regional donor, public and private partnerships? 78 | Page Probe: Has it expanded from 2012 – till now? How? 4.3 How successful is the operationalization of the West Africa Regional Health Information System (HIS) Policy and strategy? (WAHO + Country level + HIS professionals + HIS managers) Probe: What has been achieved? What are the challenges? (As far as M&E, capacity and advocacy for integrated HIS strengthening are concerned) 4.4 How do you rate the WAHO’s leadership in advancing health issues and outcomes in the region? Probe: Does WAHO play a strong, moderate or weak role? V: Client Needs Met: WAHO’s CAPS 5.1 To what extent does the TA and funding provided by WAHO address the issues that the government is facing regarding best practices implementation? Probe: What can be done by WAHO to be more effective in identifying needs? And in satisfying those needs? 5.2 What do you think WAHO could do to better address the implementation of best practices in the region? 5.3 What do you think WAHO could do to better address the implementation of best practices in health information and data sharing the region? 79 | Page 5.4 What do you think WAHO could do to better address the implementation of best practices in advocacy and communications in the region? 5.5 What do you think WAHO could do to better address the implementation of best practices for Public-Private Partnerships (PPP) in the region? 5.6. What do you think WAHO could do to better address the implementation of best practices for commodities security/supply chain strengthening in the region? VI: WAHO: Operational Effectiveness 6.1 What are the challenges or potential challenges in WAHO’s leadership in supporting health policy implementation in the country? 6.2 How have WAHO’s leadership, management and governance practices improved? 6.3 What are the challenges or potential challenges in WAHO’s leadership for capacity building? 80 | Page 6.4. What can be done to improve WAHO’s leadership in capacity building and make it the lead institution in the health sector in West Africa? 81 | Page WAHO Performance Evaluation FORMATIVE KEY INFORMANT INTERVIEWS-WAHO REPRESENTATIVE BACKGROUND & INFORMED CONSENT Date: Country: Location: Start Time: Name of Interviewer: _____________________________________________ Respondent Profile: Name Title/Organization Contact information (email/telephone) Gender 1: BACKGROUND INFORMATION: 1.1 Can you introduce yourself? (Name, position/role and institutional affiliation) 1.2 In which way does your institution support countries in implementing their health policy? 1.3 Does your organization collaborate with other institutions (at national or regional level) to support governments in implementing their health policy? If so, what are these institutions and in which areas? 82 | Page 1.4 Have you ever heard about the CAPS Project in ECOWAS countries? If yes, what do you know about the CAPS project activities in ECOWAS countries? II: Achievement: WAHO’s Leadership Capacity strengthening 2.1 Which organizations are supporting (TA and funding) governments in terms of health systems capacity strengthening? 2.2 In which areas do these institutions support governments in implementing their health policy? Probe: Health policy harmonization (regarding legal and regulatory obstacles, promoting regional strategies)? Best practices scale up/sharing? Health Information System (HIS) strengthening? Advocacy and Communications? Public-Private Partnership (PPP)? Supply Chain strengthening? 2.3 Has WAHO played a role in strengthening the implementation of the best practices? (if WAHO not listed in 2.1). Which role and how? Probe: Health policy harmonization (regarding legal and regulatory obstacles; Best practices scale up? Health Information System Strengthening-HIS? Advocacy and Communications? Public-Private Partnership￾PPP? Supply Chain strengthening/commodities security? 2.4 Has WAHO initiated actions for coordinating interventions/leveraging resources between key Implementing Partners at national/regional levels? Which role and how? 83 | Page 2.5 Which organizations would you say are leaders in advancing health issues and outcomes in the region? Probe: Is WAHO a leader? If yes, how does WAHO rank among others and why? III: Achievement: Scaling Up of Best Practices, Advocacy And Communications Strategies in the Region 3.1 What best practices in health are WAHO currently promoting for scaling up? Which organizations have assisted in that effort? Probe: What high impact practices have you heard of? In what ways have these best practices been scaled up? 3.2. What has been WAHO’s role in advocacy and communications efforts with respect to best practices scale up? Probe: Concerning domains like: • National policy changes and harmonization for FP, MCH and HIV/AIDS programs? • Improved commodity security? • Public and private partnerships? • Health information systems to increase investments in health and improve reporting? • Data utilisation to support advocacy and communication activities and promoting measurement and accountability in health? • Domestic fundraising to ensure sustainability? 84 | Page IV: Achievement: WAHO’s capacity improvement and role in PPP and HIS? 4.1 What areas of WAHO’s capacity have improved as a result of USAID support from 2012 – till now? Probe: WAHO’s institutional capacity and leadership capacity improvement (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region) 4.2 What has been WAHO’s role in regional donor, public and private partnerships? Probe: Has it expanded from 2012 – till now? How? 4.3 How successful is the operationalization of the West Africa Regional Health Information System (HIS) Policy and strategy? (WAHO + Country level + HIS professionals + HIS managers) Probe: What has been achieved? What are the challenges? (As far as M&E, capacity and advocacy for integrated HIS strengthening are concerned) 4.4 How do you rate WAHO’s leadership in advancing health issues and outcomes in the region? Probe: Does WAHO play a strong, moderate or weak role? V: Client Needs Met: WAHO CAPS 5.1. To what extent does the TA and funding provided by WAHO address the issues that governments face regarding best practices implementation? 85 | Page Probe: What can be done by WAHO to be more effective in identifying needs? And in satisfying those needs? 5.2 What do you think WAHO could do to better address the scaling up/sharing of best practices in the region? 5.3. What do you think WAHO could do to better address the implementation of best practices in health information and data sharing the region? 5.4 What do you think WAHO could do to better address the implementation of best practices in advocacy and communications in the region? 5.5 What do you think WAHO could do to better address the implementation of best practices for PPP in the region? 5.6 What do you think WAHO could do to better address the implementation of best practices for commodities security/supply chain strengthening in the region? VI: WAHO: Operational Effectiveness 86 | Page 6.1 What are the challenges or potential challenges in WAHO’s leadership in supporting health policy implementation in countries? 6.2 How have WAHO’s leadership, management and governance practices improved? 6.3 What are the challenges or potential challenges in WAHO’s leadership for capacity building? 6.4. What can be done to improve WAHO’s leadership in capacity building and make it the lead institution in the health sector in West Africa? 87 | Page Annex 2: Scope of Work for the consultant USAID/WEST AFRICA EVIDENCE FOR DEVELOPMENT SCOPE OF WORK FOR THE EVALUATION SPECIALIST PERFORMANCE EVALUATION FOR THE WEST AFRICAN HEALTH ORGANIZATION (WAHO) LEADERSHIP CAPACITY STRENGTHENING – CAPS SEPTEMBER, 2018, REVISED NOVEMBER 2018 Background On February 3, 2015, the USAID/West Africa Regional Health Office awarded the West African Health Organization, a four-year cooperative agreement to implement Leadership Capacity Strengthening (CAPS). The purpose of the project is to strengthen the capacity of WAHO as a leader in Health in West Africa, to be the lead health institution in the region for policy harmonization and advocacy, promoting public-private partnerships for health, and identifying, sharing, and supporting the scale-up of best practices in health. USAID/West Africa will monitor and assess the change in WAHO’s leadership capacity over the life of the award. The activity has a two￾pronged approach, 1) direct funding to leverage existing systems and resources, and 2) providing technical assistance through direct and frequent engagement. Objectives The purpose of the Performance Evaluation for the West African Health Organization (WAHO) Leadership Capacity Strengthening – CAPS activity is to assess whether CAPS’ purpose has been achieved. It serves as an end-of-activity performance evaluation. As per the USAID performance evaluation definition, using mixed qualitative and quantitative methods, the USAID/WA regional health office is interested to know whether expected results are occurring, to what extent selected high impact practices are being advanced, and document CAPS’ successes and challenges. The proposed evaluation questions are as follows and in order of priority: 1. To what extent is WAHO considered a leader in advancing health issues and outcomes in the region? (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region); 2. How has WAHO led and supported the scale up of best practices in the region? (i.e. mapping of current high impact practices, regional consensus and prioritization of best practices, best practices forum, capacity to support scale up of interventions; health information systems integration, capacity strengthening); 3. What has been WAHO’s role in advocacy and communications efforts with respect to best practice scale up; national policy changes for FP and HIV/AIDS programs; improved commodity security; public and private partnerships; and health information systems to increase investments in health and improve reporting? 4. What areas or domains of WAHO’s capacity (within institutional and leadership capacity) have improved as a result of USAID/West Africa’s support from 2012 - September 2018? (based on external feedback from WAHO’s clients, peer organizations, ECOWAS, key IPs in the region) The evaluation team will collect data from throughout West Africa, focusing on stakeholders associated with WAHO; Ministries of Health in ECOWAS member states; WHO AFRO; the World Bank, KfW; Foreign Affairs, Trade and Development Canada; International Development Research Centre. This performance evaluation will use quantitative and qualitative methods, primary and secondary data. The quantitative data include information from CAPS performance management plan and the quarterly and annual reports. These data will allow measuring changes in key outputs and outcomes over the evaluation period. Qualitative data will encompass information from key informant interviews (KII) and focus group discussions (FGD). On average, the team will conduct about 50 KIIs; at a minimum 3 KIIs per ECOWAS Country (the WAHO POC, one MOH official and one partner). In Burkina Faso 88 | Page (were CAPS is based), the Evaluation team will conduct at a minimum 10 KIIs from the MOH, USAID/Burkina Faso, World Bank, AfD, etc. and the WAHO staff and 2 FGDs. The evaluation team will use information from qualitative data to understand the Project’s implementation success and challenges. The Evaluation team expects to conduct three (3) to four (4) KIIs per day. The team based in Burkina Faso will review programmatic data (from WAHO) and validate the information onsite. This performance evaluation will use multi-level analysis using quantitative methods at some levels and qualitative at others. The present SOW describes the roles and responsibilities of the Evaluation Specialist. Specifically, the Evaluation Specialist will assist in developing the questionnaires, data collection and analysis and review of the final evaluation report. Total LOE will be 40 days and the period of performance in December 01 through February 28, 2019. 1. Developing of the questionnaires: 5 days 2. Data collection 10 days 3. Data analysis: 8 days 4. Developing draft Report: 7 5. Review report in French/English: 5 days 6. Travel: 5 days Specific Tasks In close collaboration with the E4D Team, the Evaluation Specialist will: 1. Develop the evaluation design, methodology, sampling strategy, and data collection instruments; 2. In coordination with the E4D Sr R&E and Research Assistant, assist in the development of the Evaluation Inception Report; 3. Conduct remote and in-country interviews in 6 countries within ECOWAS; 4. Coordinate with the E4D Sr Research and Evaluation and Research Assistant the Evaluation activities including data collection, implementation, data management and quality assurance and other related tasks; 5. Develop data analysis plan and conduct qualitative and /or quantitative data analysis, as required; 6. Actively participate with other team members during data triangulation, presentations and report writing; 7. In coordination with the E4D Sr R&E and Research Assistant, develop the evaluation report in conformance with the scope of work; 8. Develop final evaluation report with quality assurance and timeliness of all deliverables to USAID and be responsive to all comments. Deliverables/Due Dates The following deliverables are due to E4D/USAID: 1. Questionnaires; 2. Final Inception report, including timeline for data collection, analysis and reporting; 3. Data collection Report (for 7 countries) including Copies of the transcripts of the interviews conducted in the 7 countries; 4. Draft Evaluation Report; 5. Final Evaluation Report. Timeline for key deliverables Ref Deliverables Due dates 1 Questionnaires December 15, 2018 2 Inception Report December 15, 2018 89 | Page Ref Deliverables Due dates 3 Data collection Report in English with annexes including copies of the interviews and transcripts; January 31, 2019 4 Draft Study Report in French and English February 10, 2019 5 Final Evaluation Report in French and English February 28, 2019 Payment Schedule The Evaluation Specialist will receive payment incrementally throughout the course of the study, contingent upon timely submission and USAID approval of all deliverables. Each monthly payment will be processed at the IBTCI Home Office upon receipt of the Evaluation Specialist’s monthly invoice. The payment schedule is delineated in the following table: Table 1: Payment Schedule Deliverable(s) Payment Invoice Date 1 20% January 15, 2019 2 – 3 30% February 01, 2019 4 30% February 28, 2019 5 20% March 15, 2019