END OF PROJECT EVALUATION OF THE HEALTH PROGRAM OF THE COMMUNITY LIVELIHOODS PROJECT (CLP) FINAL REPORT March 5, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared under the Yemen Monitoring & Evaluation Project by International Business and Technical Consultants, Inc. (IBTCI) YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) ii FINAL EVALUATION OF THE HEALTH PROGRAM OF THE COMMUNITY LIVELIHOODS PROJECT (CLP) MARCH 5, 2014 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016, Order Number 1 Monitoring and Evaluation Project DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Authors: Circey Trevant and Bushra Abbasi YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) iii ACKNOWLEDGEMENTS The evaluators would like to thank the Yemen Monitoring and Evaluation Project (YMEP) for providing full technical and logistical support, and the CLP staff (Health Team, Grant Unit, M&E Officer) for sharing various documents, double checking the data, their patience during numerous interviews to clarify many points, setting up meetings, and providing additional documentation. Thanks also to the Yemeni researchers and team members Dr. Amatullatief and Dr. Abduljabbar Ali Abdullah. The YMEP Chief of Party and the Senior Monitoring and Evaluation Expert were most accommodating to the evaluation team by working through the Eid holidays and making possible the data collection through the focus group discussions held in Sana’a. The evaluators would like to also thank the reviewers for their patience and candid review of the drafts and their invaluable input and expertise to finalize the document to its current state. Special recognition is also due to the IBTCI headquarters team for their logistical and technical support. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) iv Table of Contents ACKNOWLEDGEMENTS............................................................................................................................iii Map of Project Areas ....................................................................................................................................vi Acronyms..................................................................................................................................................... vii Executive Summary ................................................................................................................................... viii 1. 1ntroduction…………………………………………………………………………………………….. 1 1.1 The Health Context of the Program............................................................................................ 1 1.2 Political Context........................................................................................................................... 2 1.3 Goal and Objectives of the CLP Health Component .................................................................. 2 1.4 Evaluation Purpose and Methodology .......................................................................................... 2 1.5 Limitations..................................................................................................................................... 3 2. Historical Narrative of the CLP Health Program...................................................................... 3 2.1 Introduction.................................................................................................................................... 3 2.2 Program Timelines ......................................................................................................................... 3 3. Program Design and CLP’s Strategic and Operational Approach ...........................................4 3.1 Introduction..................................................................................................................................... 4 3.2 Adequacy of Program Design.......................................................................................................... 4 3.3 Adequacy of Logical Framework and the PMP ............................................................................... 5 3.4 Reflecting on the Strategic Approach.............................................................................................. 5 3.5 Adequacy of the CLP Operational Approach ................................................................................. 7 3.6 Experiences with Implementation Mechanisms .............................................................................. 8 3.7 The CLP’s Approach to Sustainability of Health Activities.............................................................. 9 3.8 Conclusions....................................................................................................................................10 3.9 Recommendations..........................................................................................................................11 4. Planning, Monitoring, Data Quality and Reporting .........................................................................12 4.1 Introduction....................................................................................................................................12 4.2 Adequacy of Planning......................................................................................................................12 4.3 Adequacy of the CLP Monitoring System ......................................................................................13 4.4 Data Quality ...................................................................................................................................14 4.5 Reporting ........................................................................................................................................15 4.6 Conclusions....................................................................................................................................16 4.7 Recommendations..........................................................................................................................16 5. Efficiency, Effectiveness and Resilience .............................................................................................17 5.1 Introduction....................................................................................................................................17 5.2 Efficiency was Reduced Because of the Slow Roll Out..................................................................17 5.3 Efficiency Through Resilience.........................................................................................................18 5.4 Achievements viewed corresponding to plans...............................................................................18 5.5 Cost Effectiveness ..........................................................................................................................20 5.6 Local Perceptions about Effectiveness ...........................................................................................21 5.7 Cost Effectiveness of the Mobile Medical Teams...........................................................................21 5.8 Effectiveness of PPMs .....................................................................................................................24 5.9 Conclusions....................................................................................................................................26 5.10 Recommendations.........................................................................................................................27 YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) v 6. Overall Conclusions and Recommendations ....................................................................... 28 6.1 Project Design and Operational Approach...................................................................................28 6.2 Planning, Monitoring, Data Quality and Reporting .......................................................................29 6.3 Efficiency, Effectiveness and Resilience..........................................................................................31 Annexes Included in Separate Document YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) vi YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) vii Map of Project Areas YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) viii Acronyms ANC Antenatal Care AO Assistance Objectives AOTR Agreement Officer’s Technical Representative BHS Basic Health Services Project CLP Community Livelihoods Project DG Director General EmOC Emergency Obstetric Care FP Family Planning FY Fiscal Year GHO Yemen Governorate Health Office GIZ Gesellschaft für Internationale Zusammenarbeit IDP Internally Displaced Person IEC Information, Education and Communication IRs Intermediate Results LoP Life of Project M&E Monitoring and Evaluation MMT Mobile Medical Team MNCH Maternal, Newborn and Child Health MoE Ministry of Education MoPHP Ministry of Public Health and Population MoPIC Ministry of Planning and International Cooperation NGO Non-governmental organization OTI USAID Office of Transition Initiatives PNC Prenatal Care PMP Performance Management Plan PPM Private Provider Midwife RGP Responsive Government Project RH Reproductive Health RMR Routine Monitoring Report ROYG Republic of Yemen Government SFD Social Fund for Development SOW Scope of Work ToT Training of Trainers USAID United States Agency for International Development YFCA Yemen Family Care Association YMA National Yemeni Midwives Association YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) ix Executive Summary This document presents the findings, conclusions and recommendations of the Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) in Yemen. The CLP is an integrated, multi￾sectoral, assistance project designed to mitigate the instability in some of Yemen’s most difficult regions through the facilitation and implementation of quality government service delivery, job creation, responsive local governance, and active civic participation. Within the CLP the health program has focused largely on the extension of reproductive health care to populations that did not have access to health services. The USAID/Yemen Mission signed a Cooperative Agreement, number 279-A-00-10-00032-00, with Creative Associates International, at an estimated commitment of $123,534,771. The initial budget of the Health Program was $8,000,000. The original project dates were July 2010 to June 2013. Later, a no-cost extension was approved by USAID, resulting in an effective program end date of September 30, 2013. Pathfinder International worked under a 2012 CLP sub-agreement to train more than 500 medical personnel in pediatric, family planning and maternal health skills. The Program Context Yemen has among the worst levels of maternal and child health in the Middle East, largely due to preventable diseases (e.g., neonatal tetanus, measles, polio) and malnutrition, especially in rural areas. Health facilities and manpower are concentrated in cities and in more populous and economically developed governorates where 75% of the population has access to health care, compared to only 25% of the population in rural areas. Violence and civil unrest made it difficult and dangerous to travel to the areas of greatest need in Yemen. Events such as the Arab Spring (January/February 2011) and the growth of an Al Qaeda presence posed unpredictable challenges in implementing local health and development activities. Rural health facilities are prone to stock-outs of essential drugs and fight to maintain funding for operations. Low salaries of health staff leads to low morale and high absenteeism. Goal and Objectives of the CLP Health Component The health project of the CLP aimed to improve maternal, newborn and child health (MNCH), ultimately intending to reduce maternal, neonatal and child mortality. This goal is aligned with the health goals of the Ministry of Public Health and Population (MoPHP), as well as USAID’s MNCH and Reproductive Health (RH), Family Planning (FP), and Nutrition priorities. In March 2012, in order to address the uneven distribution of health services in populations with limited access to health care, in particular to female health services, USAID/Yemen re-focused project efforts on: Promoting Mobile Medical Teams (MMT) to bring essential services to vulnerable, marginalized, and internally displaced people, especially in hard-to-reach, deprived and remote areas. At the same time, furnishing and better equipping Private Provider Midwife (PPM) Clinics in rural communities. Improving the quality of health services: Focus on improving the capacity of service providers through a) training courses; b) use of updated guidelines, educational materials and messages, and health beneficiaries’ cards; and c) rehabilitation of some health facilities. Influencing behavior change through outreach and raising of awareness: Focus on: i) early marriage and pregnancy, ii) malnutrition, iii) utilization of FP methods, iv) improving the reach and quality of MNCH/FP/RH services, and v) strengthening linkages among communities, health facilities and offices. Evaluation Purpose and Methodology This evaluation of the CLP Health Program aims to provide USAID with recommendations to inform the design of future health programs in Yemen and similar fragile states. To this end, the evaluation assessed: a) the quality of the program design and the strategic and operational approach; b) the quality planning, monitoring, data quality and reporting; and c) the efficiency and effectiveness of the Program. The evaluation team consisted of two international health-sector experts, two local health-sector consultants and received technical support from the M&E Unit of the Yemen Monitoring and Evaluation Project (YMEP). YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) x Methodology: Evaluation field activities took place in Yemen from September 29, 2013 to November 5, 2013. The methodology for the evaluation included: desk review of documents and secondary data, key informant interviews (n=32), focus group discussions (n=101 discussants), and site visits to health facilities rehabilitated by CLP. The evaluation team developed all required tools. Quantitative data were processed by the YMEP M&E Unit using SPSS statistical software. Conclusions and Recommendations Project Design and Operational Approach: There were improvements seen in access to and the quality of health services during the life of the project. Health-sector activities related to PPM and MMTs were particularly effective in contributing to improved health access in rural areas. From 2011-2013, 9 health centers were rehabilitated. The numbers of assisted births in specific areas increased from 206 in 2011 to 894 in 2013. The project made key investments in training. The project can claim credit for 352 midwives being trained between 2011 and 2013. During that same period, 213 other staff were trained in maternal-child care. Recommendation: More linkages could have been made to a broader group of health workers. Although the CLP did not take advantage of community health workers (CHWs), future activities should leverage mobilize all allied health personnel (paid and unpaid) at the grassroots level. In any next phase of the health systems strengthening in Yemen, community health workers will play a vital role in the referral system either to clinics of PPMs or public health centers. USAID Partners should map “community health workers” as part of the project inception to identify those likely to collaborate with government health workers, and invest in an incentive scheme that strengthens their capacity to perform appropriate/legitimate roles in the community. All this will be based on their complementary and strategic position in target communities. When mobile teams are visiting a given district, the CHWs can do the outreach to let people know to come to that community. The contribution of the project to observable policy change or to nutritional promotion was minimal, consistent with planned activities as the project unfolded (policy and nutrition were both components of the original scope of work). Both topics received diminished priority after the original design, although some advocacy was achieved by raising awareness about the minimal age of marriage, building on work started under the BHS. The Mobile Model: Mobile Medical Teams effectively extended the coverage of reproductive health services to remote and displaced populations at a reasonable cost. 47 mobile medical teams ran from 2011-2013. 8,859 reproductive health counseling visits were accomplished. The MMTs were affordable within general budget ranges and within what is expected when trying to extend services to sub￾populations lacking access for reasons of geographic remoteness and social exclusion. Nevertheless, the MMTs did not achieve enough ownership by Ministry budget planners, in part because they did not fit within government budgeting practices or traditional administrative structures. Operational Approach: Much was learned about health projects procedures with regard to overall program implementation: Based on stakeholder feedback, there was not enough coordination, participation, and ownership in the process, especially with the authorities at the governorate levels. The project responded well to the priority reproductive and child health gaps in Yemen. However, the CLP’s health component may have benefited from a more structured approach than possible given security concerns and absorptive capacity of partners. The lack of requisite systems and resources, as well as less-than-optimal management, minimized the benefits of core processes in the health project. There were isolated attempts at establishing rudimentary monitoring systems, which would have benefited management, as well improving how they capitalized on synergies with other programs or sectors, with some success. Although “youth” was intended to be a major theme of USAID’s Yemen programs, it was not a topic per se in the planning or monitoring in CLP’s health work, though it was in fact both in the nature of the targeting of young women of reproductive age (giving them education in life options), and in the CLP training of 669 “health friendly” team members on youth-health issues at schools. Based on feedback from stakeholders, there was not enough coordination, participation, and ownership in the process, especially with the authorities at the governorate levels. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) xi Recommendation: To ensure the government’s engagement and effectiveness, USAID should keep the government counterparts abreast of projects and progress, including operations at the Governorate level. Despite the poor security environment, USAID and its partners should share newsletters, briefs, and field data with the government, particularly when it changes. USAID’s implementing partners need to be more creative and aggressive in working with ROYG to ensure their engagement and to enhance ROYG effectiveness. Wherever possible, a project may “second” staff to Governorate Health Offices or District Health Offices to align operational strategies. Recommendation: Projects should seek regular meetings with the Deputy Minister for Population of the Ministry of Public Health and Population, and GHO Directors where project is being implemented should help to update on any modifications in project work plans, budget, and changing realities on the ground. Apart from providing “real-time” briefings on project progress and challenges, the meetings will also allow for opportunities to discuss emerging promising practices and lessons learned, as well as jointly solve operational problems. Health projects should support ministry leadership in standardizing health protocols, guidelines and training curricula. Recommendation about the selection of advocacy and other sub-grant recipients: Health projects should pursue active involvement of the GHOs and Yemeni Midwife Association in PPM candidate selection to ensure proper selection according to ministry criteria and to create a sense of governorate responsibility to the midwives who need support, monitoring, and basic resources. Working with national associations also can support the decentralization strategy where they engage their local chapters. USAID projects should ramp up efforts with the MoPHP to apply more rigorous selection criteria and involvement of the GHOs in the selection, training and field placement of the PPMs. This will ensure that the GHO/RH have a greater sense of responsibility towards the midwives who need support and follow-up monitoring and basic resources (i.e. registration books, and contraceptive commodities). USAID projects should confront ministry counterparts with choices regarding site selection and present training candidates to the MoPHP to create an environment of transparency in the planning process, as well as ensure coherence between the project approach and priorities of the MoPHP. Health projects should offer technical support to the MoPHP to design and implement training management systems to sustain training functions and emerging training needs. Planning, Monitoring, Data Quality and Reporting: The original hypothesis of the project’s logic model, that health service delivery would contribute to national political stability and peace remains undetermined. The lack of counterfactuals and the existence of other numerous extraneous driving factors make it difficult to measure any attribution. The possibility of USAID’s strategy shift was not accounted for in the original program design. The original strategic design and approach were not adequate and not based on accurate assessments of the NGO sector and the evolving security profiles in the country. The project employed innovative modalities (e.g., MMT, PPM) that, if tested and documented properly, could have served as local, scalable best practices for delivering health interventions in Yemen. However, both the project design and the lack of reliable data on program effectiveness limit the ability to draw in￾depth conclusions about these intervention modalities for achieving desired health outcomes and impacts. Data about the extent to which the project contributed to USAID’s goals is limited. While the projects achieved outputs and outcomes seem to have contributed to improving the quality of health services, as per the feedback from stakeholders, there was minimal measurement within the CLP of the quality or effectiveness of the CLP training outputs among health workers. Such measurement would have aided the evaluators to interpret change resulting from the CLP quantitatively. Recommendation: USAID partners need to commit to the establishment of evidence-based planning and M&E from the inception of a program, investing in the necessary staff, systems, mechanisms, and tools. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) xii Recommendation: When designing a project, if impact is mentioned in the results framework, then this should be reflected in the PMP with outcome results at minimum; a tight research based baseline study at the beginning of the project implemented, and that the project is usually at least 5 years duration. USAID should give greater, early clarity in new projects about ensuring that ongoing data is captured sufficient to support impact, cost-benefit and value-for-money analyses, and in a manner that is standardized enough to allow for the broadest comparisons with other projects, in integrated and multiple sectors, and in other countries. Inadequate baseline or appraisal information was generated at the CLP launch. Recommendation: USAID should direct future health projects to conduct a comprehensive needs assessment that leads to a detailed plan to build on the legacy of previous projects (e.g., Basic Health Services Project (BHS), Responsive Governance Project (RGP) and CLP notably the PPMs). An evaluation of the PPMs trained by both projects would establish a baseline for future interventions with them. Recommendation: The health sector merits periodic, mini-situation analyses as part of annual reviews to re-assess existing implementation conditions of the Yemeni Health NGOs, better understand their capabilities and limitations, and ensure adequate capacity building of the grantees. Recommendation: In planning new trainings for private sector midwives, USAID should engage the National Yemeni Midwife Association to conduct a study that would assess the needs for midwives for each governorate based on their population of women of reproductive age. Such a study would yield data on the existing number of midwives (both public and private) already trained , gaps in their training including knowledge, and project what each governorate needs in order to meet the needs of these women especially in hard to reach areas and/or marginalized groups There is evidence that the project made some efforts to perform basic data verification to improve the quality of data on the number of beneficiaries reached, as well as on process data related to training. Recommendation: USAID should support implementing partners to conduct a comprehensive baseline needs assessment at the outset of any health program which can buttress a detailed plan of options to build upon previous projects (e.g., BHS, RGP and CLP, notably the PPMs). Lack of M&E technical leadership in the form of a full-time M&E officer early on in the project led to missed opportunities to put in place a robust monitoring system that could be implemented throughout the life of the project, even in the event of staff changes. Because M&E was not institutionalized from the onset of the project, this contributed to ineffective flow and use of information between the field and the central office and therefore hindered monitoring of progress toward targets. Recommendation: Even in fragile and transitional societies, particularly in the health sector, a sound work plan based on realistic targets with a strong M&E system requires leadership in place in all key positions (M&E especially) to design good plans and monitoring systems that will address gaps in a timely fashion and make the necessary adjustments. Work plans should have matching PMPs and be regularly updated and approved by USAID annually. Recommendation: Future USAID initiatives should work with the GIZ (the author of the model) to conduct a satisfaction mini-study to evaluate how the youth appreciated the information/ messages, how the information was used, and the outcomes if any in terms of key behavior changes in youth on reproductive health and other health issues of interest. This would allow for fine tuning messages ‘lost in translation’ and realize the most impact of this MoPHP best practice model. In theory, the GMS provides comprehensive and accessible data on status on activities, issues, completion certificates on closed projects. However, because this system was not in place from the beginning of the project, the full benefits of the GMS could not be exploited and had a major bearing on the ability to evaluate the project’s results. Recommendation: All project staff should be familiar with the Performance Management Plan (PMP) and be kept updated regularly on the modifications as they occur. Community health workers should be YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) xiii involved in those updates. By involving the field staff in the feedback loop, useful information could be obtained quickly for planning and reporting purposes. Efficiency, Effectiveness and Resilience: The CLP health project demonstrated reasonable resilience in re-inventing itself in response to changing conditions in Yemen and evolving analyses and priorities. There was a significant amount of under-spending on the project, with only 66% of money allocated for activities actually spent through August 2013. The CLP successfully re-focused its program design and reoriented implementation to geographical areas that were more stable to respond to the political situation. It was able to produce some results as best as its coping mechanism would allow. Adjustments made by USAID and CLP (e.g., shifting to direct implementation; shifting focus from a rural to peri-urban settings in May 2011) were key. The Cooperative Agreement and the grants under it were designed in response to stabilization goals, but instability, in turn, hindered the implementation of health sector outreach in Yemen. Efficiency calculations were hampered by standardized data collected. The CLP was not set up to link activity costs to specific results and outcomes. Financial cost data were insufficiently disaggregated by activity (i.e. training, provision of equipment/materials to allow for cost effectiveness analysis. Recommendation: USAID, in tandem with other donors and with the MoPHP should support more inclusive value-for-money studies on models including the MMT and PPM to draw out potential best practices. Partners should routinely document project expenditure data in so that cost effectiveness and cost efficiency analyses can be conducted in a standard manner in future projects. The sub-grant mechanism managed by Creative Associates for local nonprofits did not achieve sufficient control of the technical direction of activities and their quality standards. Grant recipients were often unfamiliar with what was expected of them, including their cost-share requirements. Given the inability of CLP staff to travel to program locations, and given the poor expertise in reporting back by the nonprofits, it would have required more time to train grantees to make the system work. The short-time frame of the health program militated against a full test of this grant mechanism. Based on available evidence, this small￾grants model produced too few results in too little time. Recommendation: Future USAID programs should ally with local partners already trained under the CLP, BHS and RGP, and who are familiar with USAID procedures to minimize the learning curve and sprint into action in the first year of implementation. This approach would allow more time for health initiatives to build capacity of new partners as part of their resilience and sustainability strategy. For new partners, assessments should verify their experience/positive track records implementing health projects, as well as measure absorptive capacities for new funding, ability to extend coverage of services beyond their current catchment populations, and their prospects to provide timely, consistent and accurate reporting on health outcomes besides metrics of services delivered. At the same time, USAID health projects should pursue more strategic branding in order to gain acceptance at the local level and leverage the project visibility toward expanded collaboration with the community, the MoPHP and the RoYG government. The renovation of and provision of medical equipment to health facilities were worthwhile investments when viewed in the long-term as the equipment will contribute to improving the functioning of the medical environment and quality of services. The MMTs extended the coverage of reproductive health services at a modest cost to USAID to otherwise neglected areas and remote and displaced populations, achieving the objective set. The PPM and MMT intervention strategies appear to have contributed toward increasing access to critical and life-saving health care and knowledge, particularly in rural areas. Recommendation: Future health projects should pay attention to high impact interventions with minimal cost. Working more on skill development of service providers to improve quality of health services, especially for women and children, and focus on life saving programs, such as Help Babies Breath, Post-Partum Care, Misoprostol use, and Family Planning. The MMTs achieved high visibility, and high impact while demonstrating RoYG commitment to serving YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) xiv vulnerable populations. They attracted a high volume of clients during their visits to communities, health facilities, and schools in those targeted governorates. MMT personnel used these opportunities to disseminate health messages about MNCH/FP/RH while clients waited to be seen by service providers: 82,100 clients received health education via MMTs, 2,385 referrals were made. Recommendation: USAID should continue to adopt and support MMTs and midwives working in the private sector, with the provision of health education and the furnishing and equipping of their facilities. The MMTs were within the range of expense expected to reach the most difficult-to-reach subpopulations. USAID can contribute to sustainability of mobile health systems service through a Fixed Amount Reimbursement Agreement (FARA), to improve access to services for the vulnerable groups, mothers and children in the remote areas. Recommendation: USAID should target the ROYG at national and governorate levels with technical and modest budget support to be able to take a more direct hand in planning for PPM and MMT activities. While the PPMs and MMT made noteworthy contributions to improving health access in rural areas, their resilience in the future will hinge on government interest, resources and management capacity. Recommendation: In future MMT and PPM strategies, partners and USAID should collaborate with other donors that can provide essential medicines and FP commodities (e.g. WHO, UNFPA, UNICEF) that the MMTs and PPMs need to dispense in the field. This planning will ensure lasting impact on the health of the clients. The CLP Health Sector achieved some level of synergy by building on the familiarity of the governorates with USAID through the BHS, and RGP and the fact that the mechanisms adopted – the PPM and MMT -- were already accepted by the MoPHP as best practices coming into the CLP project. Had the health sector component achieved greater synergy with other USAID projects (i.e., RGP) and integration with other CLP components (education, agriculture, water), there would have been an opportunity to explore a) efficiencies and/or cost savings achieved through pooling of resources and b) more-favorable results given project inputs. Creative Associate’s organizational strength in the education sector bode well for education-related activities of the CLP, but the organization may not have brought sufficient expertise to the health sector to achieve among the sectors. Recommendation: Future projects should expend greater effort early on and throughout to explore layering and integration of project components, within and across USAID’s flagship projects. The health project could have gone considerably further to integrate with other flagship projects in Yemen even to achieve the overall goals of the USAID Global Health. Partners should invest in opportunities for cost￾sharing/joint programming in areas of overlap (e.g., in terms of geographic focus, sector of work, target beneficiaries, etc.) with other projects, sectors and donors. To promote decentralized health access, implementing partners should direct aid toward a system of master trainers to be developed and supported at the governorates level so as to reduce dependency on centralized trainers in., They can support local skill development rollout and contribute to a more effective and sustainable decentralization. Recommendation: The GHOs should explore means to obligate or incentivize the selected candidates for the PPM program to remain active in providing the services to their local community for some period of years. If the PPM intends to move to another location, she should work along a pre￾established pathway with the GHO who can help to identify new locations. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 1 1. Introduction This report presents the findings, conclusions and recommendations of the Final Evaluation of the Health Program of the Community Livelihoods Project (CLP), an integrated, flexible, multi-sectoral, development project funded by the United States Agency for International Development (USAID)/Yemen Mission to complement its Responsive Governance Project (RGP) project, which ran from 2010-2012. 1 It was intended for the CLP to mitigate instability as per USAID/Yemen’s Fiscal Year (FY) 2010 – FY 2012 strategic aim which was: Yemen’s Stability Increased through Targeted Interventions in Highly Vulnerable Areas.2 Within the CLP the health program has focused largely on the extension of reproductive health care to populations that did not have access to health services. In July 2010, the USAID Mission in Yemen signed the Cooperative Agreement 279-A-00-10-00032-00 with Creative Associates International for the period of July 01, 2010 to June 30, 2015, at an estimated commitment of $123,534,771. The initial budget of the Health Program was $8,000,000. Later, a no-cost extension was approved by USAID, resulting in a program end date of September 30, 2013. Also as part of the CLP, Pathfinder International worked under a 2012 CLP sub-agreement, partnering with Creative Associates to train more than 500 medical personnel in pediatric, family planning and maternal health skills. Creative Associates proposed three types of activities through in-kind and fixed obligation grant instruments: Rapid Response Projects, Stabilization Projects, and Cluster Committee Projects. These were meant to be relatively simple, low-cost yet high-impact activities to fill immediate gaps in community development that could be completed within a few months. Ultimately, the CLP used both grants and direct implementation mechanisms to execute these activities. Under its original development hypothesis, the CLP was intended to test the assumption that targeted development interventions could increase trust and stability while reducing systematic physical violence in Yemen. However, the CLP’s mid-term evaluation found that the CLP mechanism was not an effective tool to accomplish this. This evaluation supports the finding that there is insufficient evidence to conclude the extent to which government fragility and violence are determined by health interventions. 1.1 The Health Context of the Program Prior to the CLP, USAID/Yemen had implemented the Catalyst Project (2004-2005) and the Basic Health Services (BHS) Project (2006 – 2010) with more conventional modalities for health and development programs.3 The RFA for the CLP called for building on the legacy of BHS and continue it through the Private Provider Midwives (PPM), Mobile Medical Team (MMT) and reproductive health interventions through “quick impact projects” for stabilization. Table 1 gives key country-wise metrics about Yemen’s maternal and child health outcomes, as stated earlier, the worst in the region and due largely to poor nutrition and preventable diseases (e.g., neonatal tetanus, measles), especially in rural areas. Table 1. Key, selected reproductive and child health indicators in Yemen Indicators4 Maternal mortality ratio 2010, Adjusted 200 deaths per 100,000 Total Fertility 2011 (births per female over their lifetime) 5.2 Under-5 mortality global rank 36 Child Mortality Rate (child under-five) 2011 77 deaths per 1,000 live births Infant mortality rate (under 1) 2011 57 deaths per 1,000 live births Neonatal mortality rate, 2011 32 per 1,000 live births Antenatal care—at least one visit: percentage 2007-2012 47 Skilled birth attendance: Percentage 2007-2012 36 Contraceptive prevalence (%) 2007-2012* 28 1 An integrated, multisectoral government program designed to complement the CLP. 2 USAID, 2010 RFA 279-10-006 Community Livelihood Project 3 Health Systems 20/20 also worked with the government on its Reproductive Health Management Information System. 4 Source of data: http://www.unicef.org/infobycountry/yemen_statistics.html YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 2 Indicators4 Percent of newborn with low birth weight (<2500 mgs) 32 Percent of Unmet Demand for contraception 51 Percent of Young Women Giving birth by the age of 18 25 Health facilities and manpower are disproportionately concentrated in urban and in more populous and economically developed governorates where 75% of the sub-population has access to health care whereas only 25% of the rural populations can access minimal health services. Security issues make it even more difficult and dangerous to travel to areas of greatest need. Rural health facilities are prone to stock-outs of essential drugs and fight to maintain funding for operations. Low salaries among health staff leads to low morale and high absenteeism. 1.2 Political Context Yemen ranks 22nd out of 75 countries along the fragile state spectrum. Distrust, political uncertainty and factional tensions impaired the implementation of the CLP. For example, the Mid-term evaluation noted delayed start up, with three phases during the first 18 months of the project — Change in Implementation Strategy, New Management, and Building Up — that were attributed to uncontrollable events such as the Arab Spring (January and February 2011) and the expansion of Al Qaeda presence in the country. 1.3 Goal and Objectives of the CLP Health Component The CLP’s health component aimed to improve maternal, newborn and child health, ultimately contributing to reduced maternal, neonatal and child mortality, aligned with the health goals of the Ministry of Public Health and Population (MoPHP) and USAID health priorities (namely, Maternal and Child Health [MCH] and Reproductive Health, Family Planning and Nutrition [RH/FP/N]). The objectives were:  To Improve access to services: To address the uneven distribution of health services in populations with limited access to health care, and to female health providers, with a focus on: Mobile Medical Teams (MMT) that can bring essential services to vulnerable, marginalized, and internally displaced people, especially in deprived and remote areas. Private Provider Midwife (PPM) support through furnishing and equipping clinics that can reach in rural communities.  Improve quality of health services: Focus on improving the capacity of service providers through a) training courses b) use of updated guidelines, educational materials and messages, health beneficiaries’ cards, as well as c) rehabilitation of some health facilities. The Behavior Change outreach and Raising of Awareness activities addressed: i) early marriage and pregnancy, ii) malnutrition, iii) utilization of family planning methods and iv) improving the reach and quality of MNCH/FP/RH services, as well as v) strengthening linkages among communities, health facilities and health offices.5 1.4 Evaluation Purpose and Methodology Purpose: This evaluation of the CLP Health Program aims to provide USAID with recommendations that will help to inform the design of future health programs in Yemen. To this end, the evaluation assessed: a) The quality of the program design with the strategic and operational approach; b) The quality of planning, monitoring, data integrity and reporting; the efficiency and effectiveness of the Program. Annex 1 presents the Evaluation Scope of Work. 5 Specific activities included: 42 CLP-sponsored educational radio spots on family planning; Direct health education through the MMTs; Training of Trainers (TOT) in selected secondary schools on youth health reproductive, safe age of marriage and FP topics who later trained “Health Friendly ; Teams”; Distribution of booklets on youth reproductive health related issues; Publishing and dissemination of the results/findings of the “Reasons and Risks of Early Marriage among Young Yemeni Women” conducted under BHS project and development and dissemination of IEC materials to convey the findings to the general population. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 3 Methodology: The evaluation was conducted by a team comprised of two international health sector evaluators (one team leader and one health expert) and two local health sector consultants. The consulting team received technical and administrative support from the M&E Unit of the Yemen Monitoring and Evaluation Project (YMEP). The evaluation research conducted within Yemen took place from September 29, 2013 to November 5, 2013. Prior to arriving in Yemen, the IBTCI team proposed an evaluation methodology that was later finalized with the support of the YMEP M&E Unit in Yemen and submitted to the USAID/Yemen Mission. The methodology for the evaluation included: desktop review of documents and secondary data, key informant interviews (n=32), focus group discussions (n=101 discussants), and site visits to health facilities rehabilitated under the CLP program. The evaluation team developed all required tools. The evaluation team also visited two health facilities rehabilitated by the CLP: Ma’ain Health Center and Al Rhawdha Hospital. The evaluation team developed all required tools. Annex 2 presents the Detailed Evaluation Methodology. 1.5 Limitations These limitations are noted with regard to the ability to draw conclusions from the available evidence: 1. Local insecurity in the country did not allow the evaluation team to travel far beyond from Sana’a. This limited the evaluation methodology to be primarily qualitative, with limited quantitative analysis through semi-structured interviews of 100 participants. 2. For all of the results, the CLP health project only had one outcome indicator that could be drawn upon in this evaluation: the number of deliveries assisted by skilled birth attendants. 3. No baseline data were available for any of the health indicators. 4. Lack of documentation about changes in the project from those proposed, including budget cuts (as noted anecdotally by CLP staff) and shifts in focus limited the evaluators’ ability to reconstruct what was done in a complete manner. 5. The evaluation team acknowledges the USAID question about assessing the impact of the CLP health program on maternal and child mortality in Yemen and advises USAID to use the DHS survey currently in progress in country to obtain the data to measure the contribution to attributable impact of all of the projects instead of each project in isolation of the others. 2. Historical Narrative of the CLP Health Program 2.1 Introduction The purpose of the historical narrative is to give a roadmap of the implementation of CLP Health Sector activities, including the challenges that the CLP faced as well as the corrective actions taken to overcome them, from the signing of the award in July 2010 to September 30, 2013. 2.2 Program Timelines Figure 1 provides an overview of key events including challenges during the life of the project. One key issue that had a significant bearing on the project, including its strategy, geographic orientation, and implementation modalities, was the socio-political unrest that grew worse in 2011. The bearing that this and other unpredictable factors had on the ability of the project to achieve intended outcomes are discussed later in this report. The following challenges are further analyzed in Annex 22. The first main challenge for the CLP was its late national registration, 10 months into its first year. The lengthy delay in the organization’s registration caused a slower than expected roll-out of the CLP into the governorates, and correspondingly a lower profile among partners of the CLP in the governorates. (USAID and CLP’s management agreed to sequentially roll out activities in Marib, Al Jawf, Amran, and Shabwah in the 2nd Quarter (October to December 2010) where there was existing experience with USAID MMTs and the PPM Program via its grants mechanism to local implementing partners.) YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 4 Figure 1. Project Milestones Starting with the CLP Cooperative Agreement In the second year, the CLP navigated through several waves of political unrest which limited of health operations of the MoPHP and the governorate health offices, as well as road closures and the evacuation of both CLP and USAID staff. In response, USAID directed the CLP to shift direction from the original 8 governorates to include the more populous urban governorates of Sana’a, Amanat Alasimah, Taizz, Ibb, and Hudaydah; and implement long term, integrated sustainable development and livelihood programming. In its third year, the CLP addressed the gap of technical leadership in health by hiring a highly qualified Health Advisor who dramatically improved the implementation of the health activities. 3. Program Design and CLP’s Strategic and Operational Approach 3.1 Introduction This section presents an analysis of the adequacy/appropriateness of the CLP’s program design for implementing a health program in a traditional assistance context. Elements of the design and implementation which are discussed include: the logical framework in relationship to the PMP, the strategic and operational approaches (grant and direct implementation), and the type of USAID award (cooperative agreement).The section concludes with a discussion on the sustainability of the health program. 3.2 Adequacy of Program Design The design of the CLP in 2010 coincided with an emerging political instability in Yemen, and in response USAID chose to focus on mitigating the drivers of instability as a development objective. Thus, the CLP project was designed to directly support USAID’s recent stabilization strategy by contributing to the achievement of four strategic, intermediate results, which were related to the two Assistance Objectives (AO)s: 1. livelihoods in vulnerable communities improved; and 2. governance capacities improved to mitigate drivers of instability6. The April 2012 mid-term evaluation of the CLP found and concluded that “the original development theory, 6 RFA Jan 19, 2010 - Community Livelihood Project. At the time that Request for Applications was prepared, USAID’s theory was based on a social understanding of drivers of instability and ignored the main instability in the country, considering social movements and the struggle between the central and regional governments and tribal leaders. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 5 and the accompanying design and the Cooperative Agreement were not appropriate for the changing context of Yemen.” This evaluation concurs with those findings and conclusions in the evaluation of the health program for the main reason that the health sector requires an integrated technical assistance rather than ad hoc activities to stabilize communities. WHO defines integrated health service delivery as the “management and delivery of health services so that beneficiaries receive a continuum of preventive and curative services, according to their needs over time and across different levels of the health system.” Therefore within a livelihood initiative, realizing this continuum would be difficult when health programming is confined to one component out of several instead of being part of an integrated process. The hypotheses in the original Cooperative Agreement posited that health would improve social stability. The evaluation findings suggest that perhaps livelihood interventions may more directly reduce the causes for extremism and conflict. The requisite skill development to ensure high quality healthcare in many countries is a lengthy process which demands patience, persistence, and determination to achieve the desired outcomes, and ascertain the expected impact on improved health profiles of the relevant communities. To sustain skill development, it is essential to concurrently build efficient and effective health systems and relevant institutional structures to maintain the responsiveness to emerging health challenges, including capacity building and training management systems, to create cadres of well qualified medical and administrative personnel capable of effectively serving the emerging needs of the population, particularly in underserved and vulnerable communities. The Yemeni health sector needs require medium- to long-term programming to develop effective and sustainable systems, to elevate healthcare service delivery to higher plateaus of performance. 3.3 Adequacy of Logical Framework and the PMP The CLP Results Framework7 in the April 2011 version of their PMP officially approved by USAID, reflected USAID/Yemen’s FY 2010 – FY 2012 Strategic goal: “Yemen’s Stability Increased through Targeted Interventions in Highly Vulnerable Areas’’. The strategy was organized under two AOs and five Intermediate Results (IRs). AO1 Livelihoods in vulnerable communities improved: focused on service provision via 2 IRs: 1.1: Employment opportunities increased and; 1.2: Access to and delivery of quality services improved. AO2 Governance capacities to mitigate drivers of instability improved: emphasized capacity building with 3 IRSs: 2.1: Public policies and institutions facilitate more equitable socio-economic development; 2.2: Local governance and basic service provisions addressing community-level needs improved; 2.3: Community-based institutions and mechanisms to ensure active participation in governance and locally-driven solutions strengthened. USAID aimed for this project to contribute to stability strategy in Yemen, though the exact logical pathways were not explicit. The Mid-term Evaluation found an incomplete logical framework, with the links between the IRs and the long term results missing. The stabilization hypothesis was supposed to build on a community based methodology and include appropriate indicators with which to measure impact. In the case of the health program, the closest link was achieved for the two IRs (increasing employment opportunities and improving access to and delivery of quality services) through the Number of Private Provider Midwives trained directly related to improving employment opportunities as a results of USG assistance: 205 PPMs were trained and equipped with private clinics to serve in remote areas. This was confirmed via the stakeholder’s survey since more than 75% agreed that the livelihood of the PPMs increased/improved as a result of the CLP training. 3.4 Reflecting on the Strategic Approach In order to provide a clearer and better defined strategy with regard to the health program, Creative 7 CLP Final PMP April 11, 2011 Version YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 6 Associates prepared a Sector Vision and Strategy for Health document in April 2011, which detailed the operational and methodological framework for the health technical sector over the life of the CLP project. Within this framework, specific long-term goals were designed to map the health sector vision and provide a conceptual blue print on how CLP will achieve its immediate and long-term goals8. The CLP defined three subsector strategies: (i) Maternal and Child Health, (ii) Family Planning, and (iii) Nutrition. Each strategy clearly defined the objectives, scope, and proposed interventions that fed into the larger vision of the health sector. The CLP health interventions also had local effects through increased employment, through the rehabilitation of six health facilities and the Al-Rawda Hospital, the promotion of community participation among school teachers and students, and support to local governorates. The CLP Strategy for Maternal and Child Health Subsector focused on interventions directly aimed at addressing the issues of access to health largely MNCH, and assisting local health capacity and IEC for pregnant and postpartum women, newborns and children less than five years of age. The activities included the training of new and pre-existing midwives in neonatal, anti natal care, deliveries, reproductive health, and family planning. Indicators that were used to monitor achievements included numbers of skilled birth attendants at deliveries, antenatal care visits, and post-partum care visits. The CLP Strategy concerning Family Planning (FP)/Reproductive Health (RH) focused on issues of access, awareness, and technical capacity of health workers with particular attention to increasing utilization of modern methods of family planning among postpartum women and married adolescents, and community and private midwives were trained to provide FP services. This training improved the clinical and communication skills in counseling for FP, which was meant to increase grassroots demand for family planning, and ultimately improve maternal and child health outcomes. Indicators that were used to monitor the FP results were: the numbers of health workers trained in FP related topics; modern methods of contraception distributed from service delivery points to beneficiaries, numbers of family planning targets, and stock-outs of key contraceptives. At the same time, the CLP health strategy paper emphasized the importance of the participation and involvement of the Governorate and District Health Offices in the geographic selection for CLP's health interventions, as these government offices are largely responsible for the provision of the resources for the functioning, maintenance and supply of the facilities, and for the increase in midwives’ capacity and credibility with the communities to obtain the necessary material, supervisory, and referral support. In addition, the CLP coordinators were supposed to play a greater role in providing technical at the governorates’ levels. The several changes that occurred after the project started reflect how the original strategic design and approach were insufficiently thought through. Even though the CLP Health Strategy Paper had good strategic value, it was not translated to full participatory implementation at all geographical and governmental levels in the country, as revealed through interviews with various health officials. The health interventions then shifted with a peri-urban focus on improving access to health services, turned away from the livelihood approach and emphasized sector driven approaches, but without the full requisite coordination with the relevant Yemeni health entities to gauge their concerns and needs in the fast￾changing environment, thus causing delays and reducing the full impact of the envisioned health sector strategic objectives. Had the strategy design been based on accurate assessments of the NGO sector and the evolving security profiles in the country, the implementation process could have been very different and with possibly more visible and sustainable impact. Overall, the CLP health interventions, especially those built on the accomplishments of the previous BHS program, were responsive to the needs, were well targeted and were linked to the USAID’s development objectives of improving maternal-child health. However, according to the perspectives of the Director Generals (DGs) informants met with at the Governorate Health Offices, and also the local-hire CLP Field Coordinators during two Focus Group Discussions, project coordination with the DGs was very poor, despite good potential envisioned in the 8 Sector Visions and Strategies – Health, April, 2011 YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 7 CLP health strategy paper which emphasized the importance of full participation and commitment of governorates and district health offices. This failed to follow the intention expressed in the Cooperative Agreement document: “As local interventions are developed, CLP will work hand in hand with local and district governments”. According to interviews with the Field Coordinators, they ended up providing logistical support, which consumed the preponderance of their time and effort, in dealing with the inadequate general financial and logistical support systems. Table 2 below shows the percentages of respondents to the structured questionnaire when they were asked about the CLP activities that addressed important health needs. (5) Shows the highest degree, (1) the lowest, and (0) signifies that the respondent did not know or that the question was not applicable. Table 2. Stakeholder responses about activity success, as a percentage per row, 5=high, 1=low Statement 5 4 3 2 1 0 Average Score “CLP activities successfully addressed important health needs in my geographic area” Mobile Medical Teams 29% 9% 12% 3% 11% 31% 3.8 Private Provider Midwives 39% 18% 7% 4% 6% 23% 4.3 Improvements to Health Clinics 22% 22% 8% 3% 12% 25% 3.9 Health Awareness campaigns 27% 17% 12% 7% 12% 24% 3.8 3.5 Adequacy of the CLP Operational Approach The mid-term evaluation report, observed that many of the factors contributing to the larger social and political instability in Yemen were beyond the capacity of USAID or its Implementing Partners to control. Meanwhile, onerous operational constraints imposed by Yemen’s instability seriously impaired CLP’s effectiveness and sustainability potential. To ensure effective sustainable livelihoods approaches, there was a need to develop an approach that required the proper combination of inputs (infrastructure, human resources, budgets, etc.) and activities, to enable and empower people to deal with shifting Yemeni trends, such as the changing security profiles. At the same time, the CLP sought to develop the requisite strategies to achieve the desired livelihood outputs and outcomes. For security purposes, the central CLP office was located in a secure compound, where they could invite and meet with their counterparts and stakeholders. A trusted local security transportation company was contracted to facilitate CLP expats transport. These arrangements were convenient but limited to Sana’a location only. CLP expatriate travel in country were very limited, and CLP international experts could not revisit the local communities and the locations where activities took place thereby seriously eliminated follow-up and supportive supervision of their national employees. The CLP had a regional office in Aden, but insecurity and lack of staff limited its functions and hence was forced to close. Evidence from both the Key Informant Interviews (KIIs) and Focus Group Discussions (FGD) revealed how the CLP faced severe challenges with its basic operations, particularly its logistical support and financial systems. The CLP delayed transportation and per diems reimbursements for lengthy periods of times, which put the YFCA, YMA9 and the CLP field coordinators in difficult situations with the trainees, trainers, MMT and PPM teams implementing the activities. This exacerbated the already negative image of CLP, USAID and its projects, acquired earlier from not understanding how CLP works with USAID10 A number of discussants in Focus Groups, from the GHOs, teachers/students, and each of the several health groups felt that the CLP project did not share their yearly plans with them, and only asked them to participate in work related to the selection of trainees, targets, PPMs and MMTs. 9 Several Correspondences between CLP and Yamani Family Care Association (YFCA) – Nov and Dec 2012 through April 2013. 10 Due to the late registration of CLP in country and subsequently in the governorates, CLP branding came late and many people did not understand that the NGOs were funded by CLP/USAID funds. Therefore, both CLP and USAID lost a branding opportunity which watered their visibility and credibility in those governorates where CLP was implemented. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 8 The CLP coordinators also confirmed that there was inadequate communication or coordination with the head office, and all decisions were made at the central level in Sana’a and later sent to them for implementation. Decentralization of skills and decision-making capabilities was not taking place and the potential for affecting sustainable outcomes at the governorates levels was lost. Another aspect of the implementation of the health activities were inaccurate estimation of operational costs for the interventions. Government expenditures for health were not enough to meet the operational needs, in particular at the governorate and community levels. As one example, the MMTs suffered crippling fuel shortages and electricity disruptions that limited their outreach, for example in Lahj, while vehicle maintenance was at times under-budgeted. Frustratingly, the CLP could not provide the solutions due to both limits to their mandate and to security; the CLP only provided per diem for transport and fuel, but did not manage the vehicles, which were transferred to the government from the BHS project. As reported by CLP, besides the security issues, operational obstacles included: (i) lack of safe domestic shipping and transport options; (ii) delays in delivering materials and training supplies to the governorates; (iii) unavailability of quality goods due to declining imports; (iv) weak market and lack of vendors and business partners; (v) inflation spirals and related erosion of local currency purchasing power; and (vi) disruption of work in agencies and ministries. In addition, the scarcity of highly qualified and skilled employees in Yemen, and lacking industry ethical standards, made it difficult to recruit qualified and reliable staff to maintain regular work flow, especially at the governorates level. By the time USAID and CLP refocused the program, put in place a new grants management system, and agreed to restructure CLP management and expand their technical staff, the CLP was half way into its implementation. 3.6 Experiences with Implementation Mechanisms The CLP cooperative agreement design was built on a small grants program to improve the livelihoods in the politically unstable communities and to increase the population’s trust in the Yemeni government. However, these expectations were hindered by unanticipated political turbulence, the inability of the CLP to fully manage these grants in a timely manner, and lacking the capacity of the local civil society organizations to adequately implement and monitor them. Consequently the small grants program did not seem to USAID to be an effective enough approach for health sector development under the circumstances of limited travel to the project sites. The CLP was authorized by USAID to shift to “direct” implementation in June 2011, away from the small grants approach that was specified in the initial Cooperative Agreement. While $72 million in grant activities had been planned for all sectors (not only health) under the larger CLP, the cumulative spending on the CLP’s health project through the original grants mechanism was less than $2,632,207. The grant system was not abandoned only because of capacity gaps among the counterpart Yemeni organizations. The abilities of the CLP project office were inadequate to get out and work with these partners, so that they could quickly gain the organizational skills to manage new USAID grants. One unrealized goal had been to build some of the local health groups up to become direct grantees of USAID, an approach successfully demonstrated in Yemen by another USAID project, the RGP. Among the few promising institutions that CLP could have worked with in the health field, two were involved early in the project, but were then dropped. These were the Yemen Family Care Association (YFCA) and Yemeni Midwives Association (YMA) which expressed their concerns about the limited quantity and quality of the CLP’s interaction with them while they were receiving grant support. Other prospective Yemeni institutions may not have been given sufficient consideration due to a lack of a proper assessment or lack of effort at reaching out and assessing more options and investing in bringing up to par those that needed support. Examples include the Charitable Society for Welfare (Islah), the Red Crescent Association, the Yemeni Women’s Union, the Yemeni Public Health Association, and the Yemeni Safe Motherhood Alliance, all of whom had experience implementing grants from international sources, in addition to many other registered local associations and charities in the governorates that provided health services successfully. Reporting on its grants, after the award in June 2010 through the evaluation period in Nov 2013, the CLP switched from using the database from the USAID Office of Transition Initiatives (OTI) for grant monitoring to using a simpler spreadsheet approach until the end of 2011. Early in the project, the CLP YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 9 submitted to USAID incomplete reports on grants until August of the same year. In response to the midterm evaluation recommendation to establish in place a Grants Management System (GMS) that tracks the outputs against the indicators over time, the CLP introduced a new GMS in February 2012. This system was designed to improve reporting and analyses on grants, but the CLP did not install it immediately; they continued using the less powerful (Microsoft Excel) spreadsheet-based grants tracking system11 . Furthermore, staffing the grants unit was difficult: two expatriate consultants were hired for this but then left the project shortly thereafter, until an experienced grants manager took over in June 2012. In its internal evaluation, the CLP reported it’s lessons-learned that the use of the small grants mechanism was demanding for many CLP health activities. The limited number of local partners adept at good reporting back, and the administrative burdens of executing and tracking grants combined to overwhelm the CLP’s staff time in monitoring and evaluation as well as the project’s close-out12. In general, the grants briefly given did not achieve the desired long term capacities or service delivery. Another lesson was that the MoPHP was more familiar with direct implementation, as it had demonstrated under the previous BHS project, but was similarly less familiar with implementation under grants. The CLP concluded that consultations with the RoYG at all levels and with external stakeholders should have been convened earlier, during the design phase of the program to encourage better ownership and commitment. Table 3 below summarizes the pros and cons of sub grants and direct implementation. Table 3. Summary of Pros and Cons of Sub Grants vs. Direct Implementation Type of agreement Pros Cons Cooperative Agreement with Sub- Grants Minimal involvement of the CLP seen with the recipient/grantee during performance and implementation Increases employment opportunities and supports decentralization efforts Promotes community participation and empowerment in center and periphery Improves local governance and basic service provision addressing community￾level needs Small grants mechanism require staff and supervision, and effort to select good NGOs Grants procedures are time and staff intensive The MoPHP was unfamiliar small grants Nonprofits were not well nor experienced to manage and track grants The CLP design did not ensure sustainability mechanisms for the grants implementation Direct Implementation Better control of activities, documentation of changes, and reporting including technical quality control Monitoring and evaluation improved to capture the change in the access for health services, and more data reported more accurately and timely manner Though similar to a grant, sponsor’s staff may be actively be involved in proposal preparation, and anticipates having substantial involvement in activities once awarded Misplaced vehicle of implementation causes problems; a contract instead of cooperative agreement would be more feasible, given USAID’s involvement 3.7 The CLP’s Approach to Sustainability of Health Activities Attention to long sustainability was inadequately planned out in the original program design. USAID’s original request for applications (RFA) solicited an approach to provide the support to the governorate, district, facility, and community levels by facilitating engagement with both the RoYG and community stakeholders via their involvement in the selection, development, implementation, and monitoring of grant activities in the sub-sectors described. Had this approach been more fully realized, it may have encouraged the RoYG to feel greater ownership and to continue to strengthen its health systems.13 11 CLP Mid Term Evaluation, April 2012, page 31 12 CLP Lessons Learned document, Oct 10, 2013 13 RFA Jan 19, 2010 - Community Livelihood Project YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 10 The original design of CLP did not adequately allow for the steps to ensure the sustainability of its interventions, including aspects related to the financial, technical and institutional sustainability. In lieu of planning for sustainability, the CLP health program pursued a more ad hoc approach to sustainability: 1. The health project pursued small infrastructure projects by rehabilitating and equipping seven health facilities, including a hospital. The CLP also went about building capacity of health service providers, such as via training community and PPMs, who were selected from the targeted communities with the most needs. Health activities strategically built upon prior-initiated interventions by the USAID BHS project14. 2. The CLP used local partners for implementation. The CLP's partners were the Governorate and District Level Health Offices together with other important institutions such as the YMA and YFCA since they had to provide the necessary inputs: staffing, facilities, maintenance, consumable supplies, drugs, vaccines, and health education/community awareness-raising. With their information systems and institutional history, they were also best positioned to help target geographic areas with greatest potential for impact. Cost-sharing with these local partners was a key ingredient; for example partners paid for salaries while the CLP paid for the cost of activities. However, while these local institutions received small grants, they faced logistical (such as scaling up) and financial (tracking and reporting) challenges whose solutions did not come easily during the implementation process. 3. Decentralization of health activities to governorate and district levels improved implementation on logistic and programmatic fronts, and mitigated security issues since mobility was restricted. However, the decentralization was limited. Greater decentralization could have enhanced sustainability. Utilizing local authorities for needs-based program planning would have helped to ensure better utilization of local inputs and would have resulted in more accurate and informed decision-making. Moreover, preparing local trainers would have built more capacity and promoted the constructive sense of ownership and commitment, with an expected positive impact on sustainable outcomes. With proper analyses of and support to strengthen existing capacities of government entities, NGOs and CSOs, a decentralized approach can catalyze replication and scaling up of programs in several governorates or districts, thus promoting better chances of sustainability through cost-efficient partnership mechanisms and instruments. Unfortunately, the CLP failed to capitalize upon the high level of mutual commitment of both USAID and RoYG to decentralize. A well defined decentralized approach could have strongly enhanced the chances of sustainability of CLP interventions. 3.8 Conclusions There are 4 versions of PMP, April 11, 2011 officially approved by USAID, and 3 updated versions in May 2012, March 2013, and June 2013; although they were utilized for monitoring and reporting, there are no records of official USAID approvals of these updates. PMP indicators were changed from one to another version, and EOPS and base line data were identified arbitrarily. CLP data collection tools, the quality and reliability of the data collected were not ideal. There were important improvements observed in access to and the quality of health services during the life of the project. Health-sector activities related to PPM and MMTs were particularly effective in contributing to improved health access in rural areas. From 2011-2013, 9 health centers were rehabilitated. The numbers of assisted births in specific areas increased from 206 in 2011 to 894 in 2013. The technical approach was appropriate for the reproductive, maternal and child health priorities in Yemen. CLP managed to improve the access and quality of health services, thus contributing to improving MCH and population health, which also contributed to USAID’s global health goals. The project made key investments in training. The project can claim credit for 352 midwives being trained between 2011 and 2013. During that same period, 213 other staff were trained in maternal-child care. The CLP program design lacked the inputs, short term results, milestones and long-term impacts in its 14 CLP health vision paper April 2011 YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 11 LogFrame. The health project of the CLP required a more structured approach than worked under the livelihood approach. Creative Associates did not manage the health grants productively and lacked the requisite systems and resources. The CLP design was rendered ineffective after the first year of implementation. The small grants program did not prove to be effective for health sector development in Yemen. USAID and CLP modified the approaches and directly implemented their interventions, and the program’s focus shifted from a rural to peri-rural and urban settings in May 2011. The contribution of the project to observable policy change or to nutritional promotion was minimal. Both topics received diminished priority after the original design, although some advocacy was achieved by raising awareness about the minimal age of marriage, building on work started under the BHS. Mobile Medical Teams effectively extended the coverage of reproductive health services to remote and displaced populations at a reasonable cost. Forty-seven MMTs ran from 2011-2013. 8,859 reproductive health counseling visits were accomplished. The MMTs were affordable within general budget ranges and within what is expected when trying to extend services to sub-populations lacking access for reasons of geographic remoteness and social exclusion. Nevertheless, the MMTs did not achieve enough ownership by Ministry budget planners, in part because they did not fit within government budgeting practices or traditional administrative structures. Much was learned about health project procedures with regard to overall program implementation: Based on stakeholder feedback, there was not enough coordination, participation, and ownership in the process, especially with the authorities at the governorate levels. The project responded well to the priority reproductive and child health gaps in Yemen. However, the health component of the CLP would have benefited from a more structured approach than was possible given the local dynamics and absorptive capacity of partners. Although “youth” was intended to be a major theme of USAID’s Yemen programs, it was not a topic per se in the planning or monitoring in CLP’s health work, but it was in fact both in the nature of the targeting of young women of reproductive age (giving them education in life options), and in the CLP training of 669 “health friendly” team members on youth-related health issues at schools. 3.9 Recommendations 1. Conduct accurate situation analyses to define existing conditions of the Yemeni Health NGOs, to better understand their capabilities and limitations, and ensure adequate capacity building of the grantees, if the approach is to work with national NGOs and concurrently contribute to NGOs and civil society development in the country. 2. Maintain a high level of communication, coordination and transparency with stakeholders, with vision of ownership and partnership. 3. Actively involve stakeholders, including government at all levels and at the appropriate stages in design and implementation of the program. 4. Health projects should pursue active involvement of the GHOs and Yemeni Midwife Association in PPM candidate selection to ensure proper selection according to ministry criteria and to create a sense of governorate responsibility to the midwives who need support, monitoring, and basic resources. Working with national associations also can support the decentralization strategy where they engage their local chapters. According to YMA informants, the YMA was not involved at all levels in the final selections. The Ministry had guidelines for selection of PPMs in as far as the PPMs should be selected from the remote areas to increase access to health services for women living in those areas. YMA have asked and recommended that they be included in the selection process so that they can provide the donors and the MoH better data to help a lot in the decision making and situation analyses. 5. USAID projects should ramp up efforts with the MoPHP to apply more rigorous selection criteria and involvement of the GHOs in the selection, training and field placement of the PPMs. This will ensure that the GHO/RH have a greater sense of responsibility towards the midwives who need support and YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 12 follow-up monitoring and basic resources (i.e. registration books, and contraceptive commodities). USAID projects should confront ministry counterparts with choices regarding site selection and present training candidates to the MoPHP to create an environment of transparency in the planning process, as well as ensure coherence between the project approach and priorities of the MoPHP. 6. Put in place functioning systems that visibly and effectively demonstrate the linkages among the LogFrame, PMP, and indicators; with clear inputs, outputs and outcomes, and develop the proper tools for the M&E and data quality. 7. Introduce cost-sharing as a tool for activity resilience and sustainability, ownership and participation. Confirm that cost-sharing and resilience plans are included in the program design, strategies, and plans. 8. Expand similar programs in the Maternal, Newborn and Child Health initiatives, including training, community midwives and health workers to contribute to improving MCH mortality rates 4. Planning, Monitoring, Data Quality and Reporting 4.1 Introduction The April 2012 CLP mid-term evaluation found that the overall CLP M&E system was adequate in staffing, data collection forms and designated software. However, it made some recommendations to fill a few gaps such as the lack of reconstructed baseline data, and poor definitions of the input and output indicators to measure accurately the CLP deliverables. 4.2 Adequacy of Planning The Performance Management Plan: The first PMP that Creative Associates submitted to USAID for the CLP was approved relatively late, 10 months into the project’s first year of running. By then, the CLP health project had begun, having already disbursed several grants, and its results were provided in quarterly reports to USAID. With respect to indicators being documented, there was no baseline identified for any of the health project indicators. Annual targets were defined consistently with gender, geographical and age specifications plus links to intermediate results. How the project would lead to results was not conceptualized. Neither inputs, nor milestones, nor results were included in the logical framework. The final version of the PMP (June 13, 2013) included 13 health indicators of which only one was an outcome indicator. Few had End-of-Project (EOP) targets, nor corresponding PIRS. The original PMP from April 2011 only had seven health indicators (but no PIRS), with some EOP targets thereby indicating that the PMP evolved over time as the health sector defined and planned its activities. In all, the PMP was revised four different times. It is unclear how the CLP health team used the PMP to set quarterly/annual targets, for planning its budget to allocate funds to meet targets, and monitor progress on its achievements by analyzing the data collected for input into decision making. Annual Work Plans: The findings indicate a mixed approach to planning even though the CLP developed three annual work plans. The work plans were set up by sector with their respective activities, outputs, and budgets. For the health sector, work plans become more comprehensive over time and by the third year, the activities were well defined with respective budgets and outputs. Unlike the first year, the work plan was more general with few activities, and most outputs were not yet determined. Also, for the health sector, the annual targets were not consistent as such. One could not follow a logical progression of changes in the CLPs’ annual targets over time. Linkages between PMPs and Work Plans: Based on the desk review for the final evaluation, the CLP’s PMP (including the Performance Indicator Reference Sheets) and the first-year annual work plan did not match since the PMP was developed long after the start of implementation. The subsequent annual work plans for Year Two and Year Three were not effectively linked with revised versions of the PMPs. The quarterly report appeared to be useful in light of the dynamic, changing environment, yet there was a YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 13 lack of alignment with annual work plans. However, Table 4 below does indicate a modest progression in the level of planning during the life of the project. Adequacy of Stakeholder Participation in Planning: As part of the Final Evaluation, the adequacy of stakeholder participation in planning was assessed via FGDs and a Stakeholder Questionnaire. These two mechanisms complemented one another, revealing different things. While the FGD participants reported that planning was not participatory, 60.4 % of the stakeholders who were surveyed felt that CLP involved the right stakeholders at the right times during all stages of planning and implementation of health activities. It is possible that stakeholders did not understand the question in the questionnaire therefore causing this discrepancy in the result.15 Table 4. Linking the CLP Annual Work Plans with the PMPs Work Plans PMP Major finding Work Plan Year 1 (July 2010-June 2011) First USAID approved PMP April 2011 Health activities did not match the first PMP. There was no budget and defined deliverables for the health activities. Work Plan Year 2 (July 2011 – June 2012) PMP was revised twice in: Dec 2011 and May 2012, For the first time, the PMP established quarterly and Year 2 targets for the indicators in Q1 (July to September 2011). Also for the first time, a budget was included for health activities. Activities in the work plan did not always link to the PMP (i.e. distribution of solar powered refrigerator with UNICEF as per the work plan was not reflected in the PMP. Work Plan Year 3 (Oct 2012 – June 2013) PMP was revised twice in March 2013 and June 2013. This work plan shows significant improvement in that health activities were clearly identified with a budget in the PMP. Adequacy of CLP’s Operational Planning: For the first two years, the CLP did not have an effective tool to inform operational planning. The CLP Mid-term evaluation found that the OTI database used by the project was not appropriate for the complex operation of issuing grants, monitoring performance of grantees to measure their progress. During the Health Sector Final Evaluation, the evaluation team found that the CLP took some corrective action by investing into the development of a formal GMS that enhanced the CLP’s monitoring capacity by managing information on the physical and financial progress of grants, and allowed better fiduciary control. The GMS also has the ability to produce reports by program and activity and also keep track of any type of modification made during a grant period such as (modification of budget; end dates) and key information such as the amount disbursed and total approved budget per grant. [Annex XV provides a report of the health sector grants]. The grant unit is now able to provide weekly updates on ongoing activities and a completion certificate for grants that have closed. 4.3 Adequacy of the CLP Monitoring System The loss of leadership in the M&E position early on in the project led to a lost opportunity to put in place a robust monitoring system that could be implemented throughout the life of the project, even during change in project staff. This further led to an ineffective flow of information between the field and the central office and therefore hindered efficient monitoring of measuring progress against targets. During this final evaluation of the health sector, the evaluation team found a project monitoring organizational structure that consisted of an M&E officer based in Sana’a with a health program officer who supervised seven CLP Health Coordinators based at the governorate level, who in turn worked with four community mobilizers gathering information at the community level. The field staff collected data from the sub grantees and the PPMs and MMTs, and they followed the progress of all health activities to report monthly to Sana’a using data collection forms and written reports. CLP heath staff from the Sana’a office confirmed that they relied heavily on the field staff for monitoring the 15 It is possible that appropriate stakeholders came to participate at planning workshops but did not express themselves as they not feel they were invited to make inputs into decision making but felt that they were expected to endorse already formulated plans. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 14 work of the grantees since insecurity issues did not allow them to travel out to the governorates. The communication was mostly via telephone (voice calls and SMS) and email. Grants Management System: In June 2012, CLP put in place a GMS that provided better information status of grants, issues affecting implementation, and completion certificates on closed projects. Limitations of the data tracking by the CLP 1. Despite the rudimentary reporting and tracking mechanism put in place for the project, it was difficult to track accomplishments over time. This is due in part to the delay in establishing a PMP, having various versions of the PMP tool, modifying the original project targets (several targets changed over time), and cancelations of some activities in the work-plan. The data from the PMP were not detailed enough to establish causal links to the IRs and targets by number, quarter, year, geographical locations, gender and age. Some definitions of deliverables hindered understanding and tracking what was accomplished. 2. Although a generic PMP was laid out in the cooperative agreement, and the Mid-term evaluation team confirmed that there were data collection forms, monitoring was flawed on multiple fronts. The plan was not applied since it lacked several critical components such as indicators (baseline data and targets). Secondly, there were no annual targets set for the health indicators; consequently, even though the community mobilizers and CLP coordinators were available in the field, there was neither explicit guidance nor plan provided for them to follow in the field. These coordinators were not aware that they should be following CLP performance against any type of targets (either quarterly or annual). 3. Until the GMS was put in place with reporting tools for the sub-grantees at the end of the second year (June 2012), monitoring of CLP activities seemed to be ad hoc. The CLP employees reported that they were not able to travel to rural areas on a regular basis to check on their sub-grantees. Once the GMS was established, the CLP was able to adequately document the status of various activities of its sub￾grantees and create a flow of information between the field and the central office in Sana’a. The CLP found that that weekly reports produced by the GMS helped for monitoring the status of the various health activities and provide adequate information for decision making. The GMS provides status on activities, issues and completion certificates on closed projects. Had the GMS been in place from the beginning of the project, CLP would have had an effective means by which to monitor the progress of grants and identify issues affecting implementation in a timely fashion. 4. Even the most recent CLP M&E officer, who only has a six-month tenure with the project, acknowledged that the ad hoc short term technical assistance (STTA) provided by Creative Associates from HQ did not provide the consistency that CLP needed to build an efficient M&E system with a feedback mechanism for planning and reporting. By the time the M&E officer was onboard with the Health Technical Advisor and her team in the third and last year of implementation, activities had already taken place without an approved PMP and matching outcome indicators. As a result, the health sector lost the opportunity to measure outcomes of its capacity building investment among others. 5. Most of the CLP’s health coordinators expressed dissatisfaction with their roles in the field feeling that they were used mainly as data collectors rather than participating actively in the planning and actual implementation of activities in the field. The coordinators also reported not knowing the indicators and the targets of the project. They also felt that CLP should have consulted them more for their input instead of being under constant pressure to provide numbers to CLP office in Sana’a, or addressing logistical issues for the grantees. In addition, they felt that there was no feedback mechanism whereby the central CLP office in Sana’a communicated back to them about the implications of the data collected and how the future plans for health activities in the field were evolving as a result. While CLP health staff in Sana’a and the M&E officer confirmed that the CLP health coordinators were engaged in confirming data collected for clarification and correction purposes, the evaluation team found it difficult to triangulate this data to confirm how the actual monitoring took place during the life of the project. 4.4 Data Quality The YMEP Data Quality Assessment (DQA) of the health sector, conducted in July 2012, revealed that, for YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 15 the most part, the health indicators were appropriate, with the exception of three indicators (out of ten), which had serious data quality issues. Six indicators had no quality issue and one only had a minor issue. The final evaluation confirmed that the project pursued two of the recommendations but other corrective actions were not possible for two other indices that did not match sufficiently national definitions for these indicators: a) disaggregate the Antenatal Care (ANC) and tetanus toxoid (TT) vaccination of women, tracking pregnant and non-pregnant women separately; b) the CLP should not combine maternal-child health (MCH) data (which is mostly primary care) with other, curative medical services data. The evaluation team also confirmed that the CLP had corrected the number of direct beneficiaries for the number of people with improved access to potable water through the receipt of silver filters to clean water supply at their homes. The total population benefiting from this water quality intervention was calculated by applying an estimate of 7.116 persons per household. Similarly, the CLP also developed a database in Microsoft Access including all the service providers trained; this allowed CLP to identify and rectify double/triple counting of the number of service providers trained. The CLP only collected partial service data from PPMs and MMTs for a specific period of time using data collection forms for the various services. For example, CLP only collected PPM data for the first group of PPMs trained in the first two years (2011 to 2012). The CLP health coordinators and community mobilizers were responsible for this data collection. Similarly, the CLP only collected service data from the MMTs only during their grant period; the last MMT operated in March 2013. The CLP health team did not collect service data on the six health facilities and the hospital that were rehabilitated. However, service statistics collected by the M&E team consultants during the final evaluation show that following the rehabilitation of the two facilities selected (Mai’ain and Shahid Al Lukia)17 there was an increase in most categories of services provided by these facilities, including a tripling of FP acceptors. There was no assessment of whether the improvements led to improved quality but it can be assumed that beneficiaries coming in greater numbers do so knowing that services were available and acceptable.18 These positive outcomes from rehabilitation of services were not reported. 4.5 Reporting The CLP missed its opportunity to report on many important health outcomes including the outcomes of key capacity building efforts such as the multiple trainings for the service health providers. There was only one health outcome-level indicator; too few. All other indicators were input and outputs, such as “number of people trained in various technical topics,” without relating the outcomes of those trainings. The CLP health sector contribution to improving access to and quality of services was not captured in the PMP. CLP met the reporting requirements in terms of the production of all the required quarterly (12 in total) and three annual reports as per the cooperative agreement. In addition, there were weekly reports to the mission. The reports detailed information on health activities with cumulative data by quarter and annually. During the first three quarters of Year 1, results were reported in quarterly reports by activity and sector. In the last quarter of that year for the first time, CLP began reporting cumulative data by sector using the indicators in the PMP approved in April 2011. Even though the PMP had established longer-term EOP target for most indicators, CLP did not link these EOP targets to the quarterly nor the annual progress reports. This suggests that CLP management did not keep an eye on final targets when looking at quarterly achievements or making plans, and neither did USAID brought their attention to this lack of connection or for planning. The team confirmed that CLP health team used the quarterly reports primarily to establish priorities for each subsequent quarter as part of their planning. 16 Poorer families on average have larger households in Yemen (above eight), as elsewhere. Estimates vary about the average household size among target populations. The Alliance for Clean Cookstoves estimates the mean household size in Yemen as seven. United Nations agencies have used seven in their planning assumptions since the last nationwide family health survey. 17 These facilities were selected because they were easily accessible. 18 See Table 6.5 in Section 6 for the full results of this assessment. The data quality of service statistics from the MOPHP has been average at best for many years but this differs by health facility depending on the facility’s management and supervision. Whatever that quality may be, the implementer has the responsibility of securing primary data on its achievements if the available data from the MOPHP cannot be trusted or improved. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 16 4.6 Conclusions The original hypothesis of the project that health service delivery would contribute to national political stability and peace remains unknown. The lack of counterfactuals and the existence of other driving factors make it difficult to measure attribution. The project employed innovative modalities (e.g., MMT, PPM) that, if tested and documented properly, could have served as local, scalable best practices for delivering health interventions in Yemen. However, both the project design and the lack of reliable data on program effectiveness limit the ability to draw strong conclusions about the appropriateness of the intervention modalities for achieving desired health outcomes and impacts. Data about the extent to which the project contributed to those goals is sparse. While the project’s achieved outputs and outcomes seem to have contributed to improving the quality of health services, as per the feedback from stakeholders, there was minimal measurement within the CLP of the quality or effectiveness of the CLP training outputs among health workers. Such measurement would have aided the evaluators to interpret change resulting from the CLP quantitatively. Nevertheless, there is evidence that the project made efforts to perform basic data verification to improve the quality of data on the number of beneficiaries reached, as well as on process data related to training. Lack of M&E technical leadership in the form of a full-time M&E officer early on in the project led to missed opportunities to put in place a robust monitoring system that could be implemented throughout the life of the project, even in the event of staff changes. Because M&E was not institutionalized from the onset of the project, this contributed to ineffective flow and use of information between the field and the central office and therefore hindered monitoring of progress toward targets. 4.7 Recommendations 1. In planning new trainings for private sector midwives, USAID should engage the National Yemeni Midwife Association to conduct a study that would assess the needs for midwives for each governorate based on their population of women of reproductive age. Such a study would yield data on the existing number of midwives (both public and private) already trained, gaps in their training including knowledge, and project what each governorate needs in order to meet the needs of these women especially in hard to reach areas and/or marginalized groups. 2. Even in fragile and transitional societies, particularly in the health sector, a sound work plan based on realistic targets with a strong M&E system requires leadership in place in all key positions (M&E especially) to design good plans and monitoring systems that will address gaps in a timely fashion and make the necessary adjustments. Work plans should have matching PMPs and be regularly updated and approved by USAID annually. 3. When designing a project, if impact is mentioned in the results framework, this should be reflected in the PMP with outcome results at minimum; a tight research based baseline study at the beginning of the project implementation, and the project is usually at least 5 years duration. 4. All project staff should be familiar with the Performance Management Plan (PMP) and be kept updated regularly on the modifications as they occur. Community health workers should be involved in those updates. By involving the field staff in the feedback loop, useful information could be obtained quickly for planning and reporting purposes. 5. USAID should give greater, early clarity in new projects about ensuring that ongoing data is captured sufficient to support impact, cost-benefit and value-for-money analyses, and in a manner that is standardized enough to allow for the broadest comparisons with other projects, in integrated and multiple sectors, and in other countries. 6. USAID should direct partners to conduct periodic, mini-situation analyses as part of annual reviews to re-assess existing implementation conditions of the Yemeni Health NGOs, better understand their capabilities and limitations, and ensure adequate capacity building of the grantees. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 17 5. Efficiency, Effectiveness and Resilience 5.1 Introduction The effectiveness, efficiency and resilience of CLP’s Health project were examined via several viewpoints.19 5.2 Efficiency was Reduced Because of the Slow Roll Out Given all the setup costs and efforts by USAID, partners and Creative Associates, the slow rollout of the health program led to low effectiveness and lower efficiency in the first year. At the time of project launch, there was a six-month delay in the company registration, which was beyond USAID’s and CLP’s control, and was a result of the Ministry of Local Administration rules and regulations. This led to delays in opening offices and hiring staff, signing a Memorandum of Understanding (MOU) with the MoPHP, and thus in issuing grants and implementing interventions. Few health grants were issued during the first year of the project.20 Further delays occurred when USAID’s strategy changed, and implementation approaches were re-aligned.21 with a shift from the use of the small grants mechanism to more direct implementation that created required additional national and expatriate staff to implement the project, which in turn required time and effort to set up and train, especially at the governorates, plus updated systems. The CLP staff reported cancellation of some health activities (i.e. computer training for health staff in the GHOs) due to lack of funds, but there was no documentation made available to the evaluation team to validate these claims. In the absence of documentary evidence, it is likely that these were adjustments made by USAID in the process of approving work plans and budgets, in light of obligated funds, contractor’s burning rates, pipelines, accrual reports and other factors.22 “Cuts” discussed in CLP reports appear to refer to activities that were not approved in their annual/quarterly work plans though these activities had been previously proposed and discussed with partners or USAID prior to USAID approval.23 Efficiency Through Leveraging: The CLP built on the legacy of the BHS program interventions (MMT, PPM, FP program) and did not have to pilot these interventions. On a positive note, CLP reactivated five MMTs set up under BHS in addition to funding additional MMTs in its target governorates. CLP trained 205 PPMs and furnished their clinics, deployed 28 MMTs over the life of the project in 10 governorates, and restaged the early marriage campaign initiated under BHS by bringing together the coalition of Yemeni NGOs and leaders in the women, health, and social justice sectors to influence parliament to reconsider signing the law to raise the age of marriage. This has resulted in a macro-level achievement of national decision makers now considering raising the age of marriage. Furthermore, CLP also continued training service providers in FP counseling techniques and services. The CLP achieved operational efficiency through partnering with the RGP in order to roll out the EmOC guidelines which the RGP had updated. 19 Effectiveness refers to the ability of an intervention to achieve its accomplishments. Efficiency combines the effectiveness with consideration of the resources required per unit outcome. 20 The CLP Health Program seems to have been more efficient in the last year of the project in comparison to the first two years of the program., In the first two years (July 2010 to March 2012), the project only trained and equipped 58 PPMs in four governorates through small grants valued at $478,644 versus 147 in nine more governorates through direct implementation in a year (April 2012 to June 2013) through direct implementation for $1,623,951 - over four times the amount spent for tripling the number of PPMs in a year. CLP was more efficient using direct implementation than it was when using the grant approach 21 CLP Mid Term Evaluation, April 2012 22 Activities that the CLP specified in its 5 year plan were not final and are subject to final approval within each annual plan (in CLP’s case, the quarterly plans were used instead). It is understandable that activities are often planned with partners who feel more often than not that these are commitments as soon as they agree to them. Yet these partners should be well briefed by the implementer about the USAID approval process and made to understand that the agreements are final only when approval by USAID is granted. In practice, activity managers (AM) if not the contract or agreement officer representatives (AORs and CORs) on USAID side are often involved with the implementers in planning negotiations with governmental partners especially at the national and governorate levels. This minimizes the cases where USAID, the implementer and the partners are not on the same page. 23 Partners then should have been briefed by the implementer about the USAID approval process and made to understand that the agreements are final when USAID approves. The implementer bears the responsibility to manage the expectations of partners and beneficiary communities and not promise things which, if not kept, may harm the image of US assistance and the USG. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 18 As well, there was leveraging of USAID resources with UNICEF and WHO on the National Polio and Measles Campaign in which CLP trained 7,633 medical staff (from MMTs and district supervisors) to immunize children (0 to 10 years of age) in five governorates. 5.3 Efficiency Through Resilience The health program achieved efficiency by additional layering and integration with other donor initiatives. In the education sector, CLP applied the (GIZ) - Yemeni-German Reproductive Health Program (YGRHP) – “Health Friendly Team” approach and trained 659 “Healthy Teams” in 65 schools. For School Health, CLP trained 58 Health Educators in nine schools but no outcomes were measured. In the Water sector, CLP distributed silver filters24 and reached 17,118 people and also 25 schools in Amran governorate. The CLP may have missed opportunities to measure outcomes of these synergies including, for example how improving access to potable water has improved health of children. Table 4 Integration of Health with Education and Water Sectors (2011 to 2013) Total Grants Given in the Health Sector Water and Health $318,780 Awareness raising in School $410,902 Total $729,682 5.4 Achievements viewed corresponding to plans The CLP pursued several approaches to achieve health goals and implemented a range of interventions to address the needs of vulnerable groups: mothers and children in the urban and rural areas. Output indicators were developed annually and new indicators were added subsequent to project start-up. Over 83 health sector activities were implemented by September 24, 2013.25 The following table provides general information about how the health program dedicated its resources over the life of the project. Table 5. Types of Sub-Awards for CLP Health Activities (2010-2013)26 Intervention Small Grants ($) Direct Implementation ($) Total ($) MMT 566,749 308,724 875,473 PPM 478,644 1,623,951 2,102,595 Water 0 318,780 318,780 Awareness Raising 135,131 461,804 596,935 Capacity Building 439,040 46,482 485,522 GHO Equipment 218,918 125,107 344,025 Health Facility Rehabilitation 31,233 659,233 690,466 Family Planning Mass Media 103,458 0 103,458 School Health 97,311 83,989 181,300 IDP Support 367,780 0 367,780 Materials Printing 193,942 637,556 831,498 Campaigns 0 584,984 584,984 Pathfinder Sub-contract 0 700,000 700,000 Total $2,632,207 $5,550,611 $8,182,818 Planned versus Achieved outputs: The following table shows the planned versus achieved outputs for the years 2011, 2012 and 2013. It also shows the percentages of the achievements for each indicator. The planned outputs for each year were extracted from the different versions of the revised PMPs, where they 24 These are filters where exposure to silver molecules is used to kill waterborne pathogens. 25 CLP Health Sector Activities Excel Sheet generated by YMEP Sept 24, 2013 26 Drawn from the grant activity disbursement spreadsheet made available to the evaluation team. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 19 have been modified for each respective year. Higher percentages of achievements might be related to the significant decrease in the planned targets that were not achieved in the previous year.27 The targets for capacity building and training indicators for 2012 were underachieved. The number of trained personnel in specific subject matters did not exceed 25 percent for any indicator, except in the case of the number of PPMs Trained, for which the ratio of achieved to target was 77 percent. The achieved outputs for 2013 exceeded the planned targets for some outputs, and were underachieved in others, but in total the performance of the last year of the project was more effective. As seen in Table 6 above, these activities with their indicators for 2011, 2012, and 2013 did not in most instances have annual or LoP targets, and, then, in some of these cases the targets were still not achieved. Several activities were conducted and numbers reported without prior planning. The CLP did not report on its indicators in a regular manner, which makes it difficult to make comparisons or draw conclusions. Table 6. Planned Outputs Contrasted with Achieved Outputs Source: document review of CLP PMP yearly, quarterly, weekly reports; PPM and MMT records; interviews and FGDs Some of the output targets were exceeded in the second year, and then made a noticeable improvement in the third year, even as the geographic range of the program expanded. In the original design of the CLP, the geographic focus was limited to eight governorates (Marib, Al Jawf, Amran, Shabwah, Aden, Abyan, Lahj and Dhale’e). With the refocus of the program in June 2011, the program shifted to more populated governorates of Sana’a, Amanat Alasimah, Ibb, Taizz, Hodaydah, and Dhamar for a total of 15 governorates. Targets were surpassed for those output indicators that are related to the outcome indicator of increased access to health services. The target for the number of women and children with access to an improved MCH care was surpassed by 27 percent; the target for the number of antenatal care (ANC) visits by skilled 27 For example, the plan for 2011 was to train 1,600 practitioners in reproductive health and family planning (RH/FP), yet only 56 practitioners were trained (4%). For the next year 2012, the planning target dropped to 120, while 29 practitioners were trained, which means the project achieved 24% of its target. For 2013, 594 practitioners completed the training, and the project achieved 495% of the target. This random direction of effort reflects ineffective planning and misleading targets that were not based on the feasibility or interest.. As another example, Creative Associates set a target of 50 health facilities to be rehabilitated in 2011, but none were achieved. In 2012, CLP planned to rehabilitate 16 HF, but only two were rehabilitated (12.5% of the target). In 2013, the CLP achieved 44% of their target by renovation seven out 16 planned health facilities, including Al-Rawda Hospital. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 20 providers was surpassed by 53 percent; while the target for the number of Reproductive health and family planning counseling visits was surpassed by 45 percent. This indicates that the planned targets were either over-estimated for the first year, or underestimated for the second and third years. It also indicates that the data collection tools and the M&E system were not advanced enough to provide the proper supervision and reliable data collection, which leads to underreporting on the number of the beneficiaries receiving services with the project support IBTCI’s evaluation team collected data from two health facilities that were renovated and equipped with CLP assistance: Ma’ain Heath Center in Amanat Alasimah, and the Al-Shahid Al-Lukia Health Center in Taizz. Health Clinic Monthly Record Sheets were collected and compared for the same month before and after interventions were completed. Table below 7 shows the percentages of improvement for each service provided at each health facility (HF) before and after the interventions. The percentages increased for most of the services provided at these HFs. Table 7. Comparison of Reproductive Health indicators, per health facility, June 2012 vs 2013 # Health indicator Ma’ain health facility Shahid Al-Lukia HF June 2012 June 2013 Percent change June 2012 June 2013 Percent change 1 Numbers of family planning visits (new) 135 589 336 % 2 5 150 % 2 Numbers of family planning visits (old) 120 402 235 % 42 52 24 % 3 # follow up visits for family planning 255 991 289 % 44 57 30 % 4 Numbers of visits or family planning session 201 214 7 % 2 5 150 % 5 Numbers IUDs insertion 5 17 240 % 2 0 -100 % 6 Numbers IUDs removal 9 8 -11 % 1 0 -100 % 7 Numbers IUDs follow up 14 22 57 % 1 1 0 % 8 Numbers ANC (first time) 100 64 -36 % 0 0 0 % 9 Numbers ANC (2nd time) 101 150 49 % 0 0 0 % 10 Numbers ANC (3rd time) 23 13 -44 % 0 0 0 % 11 Numbers ANC (4th time) 22 15 -32 % 0 0 0 % 12 Numbers of delivers inside H.C 0 2 n/a 0 0 0 % 13 Numbers of visits to Gynecological OPD 1005 1116 11 % 48 20 -58 % 14 Vaccination of tuberculosis: # children 122 151 24 % 0 4 n/a 15 Numbers of children receiving penta-vaccine 336 453 35 % 39 23 -41 % 16 Numbers of children measles vaccine 255 318 25 % 14 28 100 % 17 Numbers of children get polio vaccine 336 453 35 % 53 41 -23 % 18 Numbers of beneficiaries received lab tests 4122 3824 -7 % 46 149 224 % This impact of training health providers at these health facilities is expected to multiply. Al-Rhawdha Hospital, for example, that was rehabilitated towards the end of the project, provides Emergency Obstetrical and Neonatal Care Services to an estimated 1,000,0000 person population in the surrounding areas where subsidized services are not available. 5.5 Cost Effectiveness Following are estimates of the relative cost effectiveness of key delivery models. The cost of services provided by MMTs is less per beneficiary ($3.05),28 compared to PPM’s cost at $16.42 per beneficiary. The 28 The total disbursed in grants to MMT is: $566,749 and CLP gathered data on the 25 MMTs with a service coverage of 185,470 beneficiaries. It is not clear whether data was collected on those MMTs funded under the direct implementation YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 21 MMT cost is the total in grants given to 25 MMTs but capital investments in this program are not included. The reasoning is described in the sections below. Limitations: The CLP office did not collect the records necessary to fully analyze the relationship between costs and health outcomes. Often the cost data obtained was not clearly disaggregated. So it was not feasible to make clear links of health outcomes and interventions with their true costs. There are also no industry standards against which to analyze and compare the MMT data since the MMT grants only covered transportation costs and per diem for team members during their operation in the field excluding all other costs such as MoPHP personnel salaries, supplies, medicines and initial capital investments. 5.6 Local Perceptions about Effectiveness The overall feedback from the interviews with key informants, group discussions and stakeholder questionnaire indicated that stakeholders considered that the CLP health program succeeded in contributing to improving access, quality of health services, and raising health awareness. All stakeholders were pleased with the 205 PPMs who were trained and equipped with their own clinics. They further emphasized the need to train many more midwives in the public and private sectors to make the services available in remote areas, especially those with no access to health services. In the structured questionnaire, significant portions of the respondents strongly agreed that the CLP activities successfully addressed important health needs in their geographical areas: 28.7 percent answered MMTs, 38.6% regarding the PPM program, 28.7 percent Improvements to Health Clinics, and 26.7 percent for Health Awareness campaigns. Only 10% of the respondents strongly agreed that CLP did NOT have any positive impact in their communities, and 25.7% strongly disagreed with the statement. The percentages of respondents, who strongly agreed that the CLP’s health activities contributed to improving the livelihood of the midwives was 46 percent, access to health services 45%, the quality of health services provided by Mobile Teams 23 percent, the quality of health services provided by Midwives 42%, the health awareness at the community level 37 percent, and the quality of health services provided by Health Facilities that were rehabilitated and received equipment 30 percent. 5.7 Cost Effectiveness of the Mobile Medical Teams The operational costs seen for the MMT model were higher per beneficiary visit under CLP than they were under the predecessor project BHS.29 The MMT performed for 15.7 months out of the 48 months of the life of the project ceased activities when the CLP stopped sharing. While the local health authorities attest to the benefits of this program, and recognize its value to their communities and to the ability of health facilities and agencies to provide services, the costs of taking over these activities were never included within the GHOs budgets and no discussions were either initiated or successful in building these costs into the annual budget of the GHOs by the MoPHP.30 The MMT approach was not expected to be a long-term alternative to having a fixed health facility. Instead the MMTs were an attempt to reach populations in areas where fixed clinics were not equipped or staffed, or in those areas where vulnerable populations had no access to fixed clinics. The MMTs offered an alternate approach for dealing with the shortage of medical staff as well as a better use of staff.31 mechanism. 29 It is difficult to draw comparisons when mobile health teams had different durations and different numbers of concurrent MMTs in operation between the BHS and CLP versions. 30 Not only was there a considerable amount of time lost during the program when no services were provided but opportunities were lost because no plans were drawn or implemented to have the MOPHP or the GHOs and local councils assume the running costs of the MMTs in a gradual phased-in approach. It was confirmed by the local staff representing CLP at the governorate level and interviews with the GHOs, DGs, and FGDs that once the CLP program started closing out, all MMTs stopped functioning due to lack of pipeline funding. 31 A doctor sitting in a health facility will not be nearly as well utilized as when he/she works in a MMT that serves a number of health facilities together and collects clients in one session who otherwise would be scattered over a four weeks period (at least those who health needs can wait to be addressed for up to four weeks until the MMT arrives). Daily numbers for health facilities average 10 - 25 at best when there is a doctor and much less when there is none. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 22 The MMT interventions targeted vulnerable, marginalized, and Internally Displaced Persons (IDPs) groups, especially women and children lacking health services in the remote areas. These services included maternal and family health services and education, and school health interventions. The CLP activated 25 MMTs in eight Governorates as shown in Table 8 below. Through the MMTs, the CLP project supported the MoPHP and the NGOs to provide 185,478 medical examinations for beneficiaries 32 in eight governorates where the MMTs functioned. The MMTs covered 63 districts, and provided MCH, RH/FP and other basic health services to the remote communities. Table 8. MMT by province and dates of functioning # Governorate # of MMTs Start and End Dates of MMT Interventions by Year Duration of activity Currently functioning 2010 2011 2012 2013 Total Months 1 Amran 5 Dec June 5*19=95 No Amran 3 July April 3*10=30 No 2 Marib 5 Dec April 5*17=85 No 3 Shabwah 4 Dec June 4*19=76 No Shabwah 1 June April 1*23=23 No 4 Lahj 1 Jul June 1*12=12 No Lahj 1 July July 1*13=13 No 5 Sana'a 1 Jan Jan 1*13=13 No 6 Hajjah 2 Jan Jan 2*13=26 No 7 Aden 1 July July 1*13=13 No 8 Abyan 1 Sept March 1*7=7 No Total 8 25 393/25=15.7 No The MMT’s ceased operation when the CLP stopped supporting them, as confirmed in interviews led by the evaluators. This is sustainability issue: that the CLP failed to include sustainable strategies in their original design. Figure 2 shows the number of beneficiaries served through MMTs, broken down by governorates. Figure 2. Number of Medical Services Provided Through MMTs, by Governorate The CLP Coordinators who were embedded out in the GHOs managed the MMT outreach, including the logistical and organizational planning, collected data services and submitted reports to CLP headquarters. 32 Number of medical services were calculated because the CLP data collection system does not permit the calculation of the number of beneficiaries, which would have required that each patient be assigned a unique identifying number and a database of clients constructed (fed by the requisite flow of information from MMT registration books) YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 23 These MMTs were able to serve large numbers of marginalized groups and refugees because they were located in densely populated areas. The CLP eventually supported an MMT in Abyan Governorate as part of CLP’s participation in the Southern Governorate Response Plan rebuilding that governorate. CLP collaborated with the World Health Organization (WHO) to make medications available to MMTs operating in Aden and Abyan because supply shortages at the MoPHP. To improve the MMT service providers’ skills, the CLP trained doctors and midwives in EmOC. The MMTs provided MNCH/RH/FP services to 10,161 beneficiaries as shown in Table 9 below. Gynecological, ante-natal exams dominated the benefits and coverage, followed by health education and family planning messaging and counseling. Larger MMTs worked at schools where they were delivering services, often with services to peoples displaced by fighting, who crossed from one governorate to another. These IDPs did not otherwise have access to services. In the areas of fighting, the MMTs faced greater risk of car-jacking. Table 9. Mobile Medical Team Statistical Data for Health Services by Type and Governorate Cost-effectiveness calculations: The total expenditure for the 25 MMTs as reported by Creative Associates reached $875,473 through a combination of grants and direct implementation mechanism. The incremental cost per beneficiary-visit or medical examination (total visits of 101,691) is roughly $8.60 per medical examinations.33 The comparison of cost per visit under CLP and its predecessor project, running the same activities, shows that the CLP cost is roughly four times of what it was for BHS (394.9%). This difference in the cost per beneficiary visit is likely due to the fact that under the CLP, the MMT operated out in communities (school, hall, or some other venue provided by the community) and rarely at any health center. 34 In addition, no data was obtained specific to the number of patients seen by either a physician or a nurse. The costs calculated as part of the former BHS were for a full MMT, i.e. including expense for a new four wheel drive vehicle equipped with a value of $27,000 worth of medical equipment and working out of health facilities 33 The expenditures included the operational costs, including repairs of the vehicles, air conditioners, staff salaries per diems and transportation reimbursements for the service providers. MMTs’ vehicle repairs were frequent and necessary during the Creative Associates program because the MMTs traveled to remote areas that had difficult access and therefore increased the wear and tear on the already aged vehicles which required more fuel and more maintenance. 34 The CLP did not adhere strictly to the previous BHS model in which MMTs operated from designated health centers and the clients came to obtain services at those centers; under CLP, the MMTs went to the remote areas and operated from schools, community designated areas, some were fully self-contained mobile clinics operated by the YFCA and initially started by UNFPA many years ago. Mobile Medical Team, working from van, in Shabwah YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 24 exclusively. If there were modifications, such as using other community venues, which made the program less costly and provided the same result it would be to the credit of the CLP approach. The MMT visits provided a broader array of services: access to a physician and to services covering a large scope including all preventive specialized RH/MNCH services including ultrasound if needed, whereas the health facility’s regular services are visits to a community midwife at best and no access to most of the services offered by the MMT. One theoretical alternative to MMTs would be to set up a team and equip each of the health centers that otherwise would be served by the MMT. Under the BHS project, the ten teams operating in 2008 served an average of 144 health centers, out of a total of 603 health facilities available, or nearly a quarter of health facilities in five governorates. It is unlikely that the MoPHP can allocate 144 physicians and 300 midwives to staff and equip rural once a month Another way in which USAID can contribute to sustainability of this mobile reproductive health service, is through a Fixed Amount Reimbursement Agreement (FARA), a mechanism in which the Ministry of Health pays for the service on a quarterly basis, once a program of support is analyzed and agreed upon by USAID and the MoPHP In this mechanism, the MoPHP would be reimbursed by USAID upon verification of effective service delivery.35 Using the figure of $2.18 as a starting figure it is possible to plan on setting aside a budget based on expected demand that would be used to defray the costs of setting up and running as many mobile teams as needed or can be financed. To achieve a number of visits of 100,000 per year would require an investment of $218,000 per year over a 5 year period and where the initial year would see a higher budget to pay for the initial capital investments. Using these CLP figures for cost per beneficiary-visit, the budgeted amount would be roughly four times higher, while an average in between is likely to be more realistic considering that the situation has changed since 2008 when the BHS seemed to have had less operational difficulties and logistics costs. 5.8 Effectiveness of PPMs The PPM program, besides increasing women’s access to RH/FP and increasing the percentage of deliveries by Skilled Birth Attendants, aimed at creating work opportunities for the trained yet unemployed midwives by supporting them to establish home based midwifery clinics. The PPM intervention was the most expensive activity out of the entire CLP health component. There is insufficient data to adequately evaluate the PPMs since little outcome data has been obtained for the work of the 147 PPMs who were trained in the second wave of training, i.e. under direct implementation. It lacked measurable outcomes; total costs associated with the PPM activities and a benchmark to compare typical results for the PPM interventions. The PPM intervention for training and equipping their clinics cost 35 The initial payment by the MoPHP might require a ministerial agreement with the Ministry of Finance which approves fund on quarterly basis to the MoPHP thus enabling it to fulfill its the responsibility. This has potential benefits for Yemen: ultimately it will be paid for by USAID, use existing MoPHP systems, develop the discipline to service to marginalized populations, report on ways to affect health system reform, and increase the potential for sustaining the service once they become old hands at delivering it. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 25 $2,102,595 for an average of $10,256/PPM. There is no itemized cost information available for this program; the total is a summary of both grants to local NGOs and direct implementation disbursements by CLP. It was not possible to estimate a total amount for training per se since it was not tracked separately by Creative Associates; instead training was part of the package for building the capacity of PPMs. This average cost per PPM is roughly 5 times the cost spent under the BHS project for its assistance to set up PPMs. During its first operational year, the CLP used its small grants mechanism to implement through the National YMA, in collaboration with GHOs in four governorates. Toward the end of the project, the CLP expanded PPM activities to all eight governorates as planned, in addition to Sana’a, Amanat Alasimah, Taizz, Ibb, and Dhamar. Creative Associates and YMA used a decentralized approach to enhance operational and program implementation. The training model which was applied included components for mapping training to enable PPMs to draw maps of their communities, portray potential beneficiaries and focal points in their catchment areas, and identify the nearest appropriate referral points and contacts to physicians.; The model also provided business management training to build the capacity of PPMs to run and manage small business/private clinics. Lastly, it provided Emergency Obstetric Care (EmOC) practical training (antenatal care, delivery including active management of third stage of labor, postpartum care, newborn care and resuscitation, diagnoses of high risk cases, and medical management of eclampsia). Table 10. Trained 208 PPMs by Governorate PPMs were provided with equipment and furnishings with which to establish their own clinics and to provide MCH/RH services and earn more income by operating from their own homes. The equipment included autoclaves to fulfill standards set by the MoPHP’s standards to obtain a license for new PPM clinics. The evaluation team was able to collect PPM service delivery data from Creative Associates’ reports from and about four governorates. The data from the other governorates, which started at later stages, was not available. PPMs provided Family Planning services to 4,330 new and continuous beneficiaries as presented in Table 11 below. Table 11: The PPMs’ Beneficiaries and Service, by Governorate Governorate Family Planning Beneficiaries FP Repeat Clients Total Beneficiaries Al Jawf 114 1,907 2,021 Amran 306 456 762 Marib 323 476 799 Shabwah 281 467 748 Total 1,024 3,306 4,330 The PPMs provided a total of 23,793 health services, including counseling and health education services in the four Governorates under the CLP project as presented in Table 12 below. These services contributed to improving access and quality of services provided to mothers and children and the vulnerable groups. Hypothetically, this number could reach 75,000 services if the service data could be collected from the 12 governorates where PPMs functioned and these interventions being implemented. When the YMA’s grant ended, Creative Associates worked directly with the GHOs and the YMA branches at governorate levels, which circumventing the central YMA office in Sana’a. This blocked the flow of support and monitoring going to the PPMs and impaired the data collection and the ability of the PPMs to continue to generate periodic statistical reporting. The reason is that branches of the YMA are independent with little capacity to provide the kind of service the YMA was providing. The evaluation team was unable to verify the figures provided by YMA36 as they don’t receive regular reports from PPMs from all governorates. 36 Additional data on PPMs was collected by the evaluation team from the YMA. According to the YMA, who provided supervision Governorate Trained PPMs Governorate Trained PPMs Al-Jawf 15 Taizz 20 Marib 14 Dhamar 20 Amran 15 Al-Dhale 20 Shabwah 14 Aden 17 Sana'a 20 Lahj 10 Ibb 20 Abyan 20 YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 26 Table 12: PPMs’ Client Service Utilization Statistics by Governorate Governora te Family Planning Counseli ng FP Follow￾up ANC Assisted Deliveri es PNC Home Visit Genera l Home Visit TT Vaccine Child Vaccin e Health Educ. Total Service s Al Jawf 320 328 2,161 775 1,080 1,919 44 13 1,235 7,875 Amran 170 87 848 434 517 657 19 42 631 3,405 Marib 211 69 960 290 395 1,354 88 186 588 4,141 Shabwah 883 282 1,230 967 1,263 1,387 96 70 2,194 8,372 Total 1,584 766 5,199 2,466 3,255 5,317 247 311 4,648 23,793 5.9 Conclusions Consistent with both the MoPHP and USAID health priorities for MCH and RH/FP, CLP implemented a range of activities to improve MNCH with the ultimate goal of reducing maternal, neonatal and child mortalities. During the project life, CLP facilitated and coordinated efforts among government, private service providers, and communities to address the pressing challenges related to awareness, access and the capacity to provide care. The CLP built on previous USAID experiences and mobilized a number of approaches, such as improving access to health services by supporting the community and private midwives in rural areas; improving the clinical and awareness-raising skills of existing providers through in-service training and; and engaging communities through health advocacy and education. The main goal of the health program was to put in place a sustainable system of increased access and improved quality of MCH/RH/FP services targeting under-served women and children. The aim was to increase the number of antenatal care visits, the numbers and quality of skilled birth attendants, the provision of post-partum care for mother and newborn, the use of FP modern methods. The CLP met some of their targets during the second year of the health program, during which CLP implemented most of its activities in the year. Measuring the performance of the health program was not an easy task in light of the fact that there was no base line data for the indicators, the outcome indicators changed from one year to another, and were built on vague assumptions. CLP M&E system, including program and financial planning and reporting were not designed properly and did not provide the appropriate data for sound decision making processes. The CLP processes did not apply supportive supervision for its Coordinators in the field. There were gaps in communication between the headquarters and governorates. Similarly, Communication between USAID/Yemen and CLP were not always smooth, especially at the beginning of the project. The constant change in Yemen’s security situation and the need to promptly respond to the evolving needs exacerbated the process of implementing the planned interventions. After the midterm evaluation, the communication between USAID and CLP seemed to improve. The CLP relationship with the MoPHP at the central and the GHO and DHO levels in the governorates was not built on partnership and ownership perspectives. These authorities were not involved at early stages in the project design, and many of the plans were not developed in a joint manner. The CLP-supported MMTs made a key contribution to increased access to services, but governmental authorities did not sustain mobile teams beyond the period of donor assistance. The MMTs were affordable within general budget ranges and within what is expected when trying to extend services to subpopulations lacking access for reasons of geographic remoteness and social exclusion. Nevertheless, the of the PPMs, all 205 trained midwives received equipment, had a practicing license and their clinics are functioning. YMA reported that PPMs during 2012 year provided a total of 96,900 prenatal, antenatal, and postnatal care services, including normal deliveries, family planning and health education services. Creative Associate supported YMA through a small grant mechanism at the beginning of the project, to establish a data base for all midwives (public and private) in Yemen. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 27 MMTs did not achieve enough ownership by Ministry budget planners, in part because they did not fit within traditional administrative structures. The renovation and equipment of HF and Al-Rhawdha Hospital were worthwhile activities which will contribute to improving the efficiency of the medical environment and quality of services. Since these interventions were completed towards the end of the project, there was not enough time or periodic reporting to enable meaningful assessment or accurate measurement of effectiveness. 5.10 Recommendations 1. Future projects should support the MoPHP to design and implement training management systems, including developing training database, systems to regularly develop/update clinical and management standards, and sustain training functions and emerging training needs. 2. USAID initiatives in Yemen should ally with local partners already trained under the CLP, BHS and RGP, and who are familiar with USAID procedures to minimize the learning curve and sprint into action in the first year of implementation. This approach would allow more time for health initiatives to build the capacity of new partners as part of their resilience and sustainability strategy. For new partners, assessments should verify their experience/positive track records implementing health projects, as well as measure absorptive capacities for new funding, ability to extend coverage of services beyond their current catchment populations, and their prospects to provide timely, consistent and accurate reporting on health outcomes besides metrics of services delivered. 3. Future health projects should pay attention to high impact interventions with minimal cost. Working more on skill development of service providers to improve quality of health services, especially for women and children, and focus on life saving programs, such as Help Babies Breath, Post-Partum Care and Misoprostol, Family Planning, with emphasis and scale up on MCH and RH/FP programs. 4. Implementers should ensure that Master Trainers are created and supported at the governorates level to reduce dependency on centralized trainers in all critical areas and subject matters, to support local skill development rollout and contribute to a more effective and sustainable decentralization. 5. A key strategy to continue is on improving the quality of care and health systems at the Health Centers and Hospitals of the MoPHP, utilizing decentralized approaches to reach district and community levels. 6. USAID should continue to adopt and support MMTs and midwives working in the private sector, with the provision of health education and the furnishing and equipping of their facilities. The MMTs were within the range of expense expected to reach the most difficult-to-reach subpopulations. USAID can contribute to sustainability of mobile health systems service through a FARA, to improve access to services for the vulnerable groups, mothers and children in the remote areas. 7. Health programs should continue to develop and update IEC materials, clinical and management standards, according to the needs of the ministry rather than reprinting them, and enhance ownership and partnership approaches in this process. 8. Future projects should expend greater effort early on and throughout to explore layering and integration of project components, within and across USAID’s flagship projects. The health project could have gone considerably further to integrate with other flagship projects in Yemen even to achieve the overall goals of the USAID Global Health. Partners should invest in opportunities for cost￾sharing/joint programming in areas of overlap (e.g., in terms of geographic focus, sector of work, target beneficiaries, etc.) with other projects, sectors and donors. 9. Develop a research agenda and improves periodic statistical reporting to strengthen evidence-based health policy development and affect improvement in quality of health services. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 28 6. Overall Conclusions and Recommendations 6.1 Project Design and Operational Approach There were improvements seen in access to and the quality of health services during the life of the project. Health-sector activities related to PPM and MMTs were particularly effective in contributing to improved health access in rural areas. From 2011-2013, 9 health centers were rehabilitated. The numbers of assisted births in specific areas increased from 206 in 2011 to 894 in 2013. The project made key investments in training. The project can claim credit for 352 midwives being trained between 2011 and 2013. During that same period, 213 other staff were trained in maternal-child care. Recommendation: More linkages could have been made to a broader group of health workers. Although the CLP did not take advantage of community health workers (CHWs), future activities should leverage mobilize all allied health personnel (paid and unpaid) at the grassroots level. In any next phase of the health systems strengthening in Yemen, community health workers will play a vital role in the referral system either to clinics of PPMs or public health centers. USAID Partners should map “community health workers” as part of the project inception to identify those likely to collaborate with government health workers, and invest in an incentive scheme that strengthens their capacity to perform appropriate/legitimate roles in the community. All this will be based on their complementary and strategic position in target communities. When mobile teams are visiting a given district, the CHWs can do the outreach to let people know to come to that community. The contribution of the project to observable policy change or to nutritional promotion was minimal. Both topics received diminished priority after the original design, although some advocacy was achieved by raising awareness about the minimal age of marriage, building on work started under the BHS. The Mobile Model: MMTs extended the coverage of reproductive health services to remote and displaced populations at a reasonable cost, with 47 mobile medical teams ran from 2011-2013 and with 8,859 reproductive health counseling visits. The MMTs were affordable, within general budget ranges and within what is expected when extending services to sub-populations who lack access for reasons of geographic remoteness and social exclusion. Nevertheless, the MMTs did not achieve enough ownership by Ministry budget planners; in part because due to government budgeting habits. Operational Approach Much was learned about health projects procedures with regard to overall program implementation: Based on stakeholder feedback, there was not enough coordination, participation, and ownership in the process, especially with the authorities at the governorate levels. The project responded well to the priority reproductive and child health gaps in Yemen. The health component of the CLP would have benefited from a more structured approach than feasible given the local dynamics and absorptive capacity of partners. The lack of requisite systems and resources, as well as less-than-optimal management, minimized the benefits of core processes in the health project. There were isolated attempts at establishing rudimentary monitoring systems, which would have benefited management and improved the synergies with other programs or sectors, with some success. Although “youth” was intended to be a major theme of USAID’s Yemen programs, it was not a topic per se in the planning or monitoring in CLP’s health work, but it was in fact both in the nature of the targeting of young women of reproductive age (giving them education in life options), and in the CLP training of 669 “health friendly” team members on youth-related health issues at schools. Based on all feedback from stakeholders, there was not enough coordination, participation, and ownership in the process, especially with the authorities at the governorate levels. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 29 Recommendation: to ensure the government’s engagement and effectiveness, USAID should keep the government counterparts abreast of projects and progress, including operations at the Governorate level. Despite the poor security environment, USAID and its partners should share newsletters, briefs, and field data with the government, particularly when it changes. USAID’s implementing partners need to be more creative and aggressive in working with ROYG to ensure their engagement and to enhance ROYG effectiveness. Wherever possible, a project may “second” staff to Governorate Health Offices or District Health Offices to align operational strategies. Recommendation: Projects should seek regular meetings with the Deputy Minister for Population of the Ministry of Public Health and Population, and GHO Directors where project is being implemented should help to update on any modifications in project work plans, budget, and changing realities on the ground. Apart from providing “real-time” briefings on project progress and challenges, the meetings will also allow for opportunities to discuss emerging promising practices and lessons learned, as well as jointly solve operational problems. Health projects should support ministry leadership in standardizing health protocols, guidelines and curricula. Recommendation about the selection of advocacy and other sub-grant recipients: Health projects should pursue active involvement of the GHOs and Yemeni Midwife Association in PPM candidate selection to ensure proper selection according to ministry criteria and to create a sense of governorate responsibility to the midwives who need support, monitoring, and basic resources. Working with national associations also can support the decentralization strategy where they engage their local chapters. USAID projects should ramp up efforts with the MoPHP to apply more rigorous selection criteria and involvement of the GHOs in the selection, training and field placement of the PPMs. This will ensure that the GHO/RH have a greater sense of responsibility towards the midwives who need support and follow-up monitoring and basic resources (i.e. registration books, and contraceptive commodities). USAID projects should confront ministry counterparts with choices regarding site selection and present training candidates to the MoPHP to create an environment of transparency in the planning process, as well as ensure coherence between the project approach and priorities of the MoPHP. Health projects should offer technical support to the MoPHP to design and implement training management systems to sustain training functions and emerging training needs. 6.2 Planning, Monitoring, Data Quality and Reporting The original hypothesis of the project’s logic model- that health service delivery would contribute to national political stability and peace remains undetermined. The lack of counterfactuals and the existence of other numerous extraneous driving factors make it difficult to measure any attribution. The possibility of USAID’s strategy shift was not accounted for in the original program design. The original strategic design and approach were not adequate and not based on accurate assessments of the NGO sector and the evolving security profiles in the country. The project employed innovative modalities (e.g., MMT, PPM) that, if tested and documented properly, could have served as local, scalable best practices for delivering health interventions in Yemen. However, both the project design and the lack of reliable data on program effectiveness limit the ability to draw in-depth conclusions about these intervention modalities for achieving desired health outcomes and impacts. Data about the extent to which the project contributed to USAID’s goals is limited. While the project’s achieved outputs and outcomes seem to have contributed to improving the quality of health services, as per the feedback from stakeholders, there was minimal measurement within the CLP of the quality or effectiveness of the CLP training outputs among health workers. Such measurement would have aided the evaluators to interpret change resulting from the CLP quantitatively. Recommendation: USAID partners need to commit to the establishment of evidence-based planning and M&E from the inception of a program, investing in the necessary staff, systems, mechanisms, and tools. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 30 Recommendation: When designing a project, if impact is mentioned in the results framework, then this should be reflected in the PMP with outcome results at minimum; a tight research based baseline study at the beginning of the project implemented, and that the project is usually at least 5 years duration. USAID should give greater, early clarity in new projects about ensuring that ongoing data is captured sufficient to support impact, cost-benefit and value-for-money analyses, and in a manner that is standardized enough to allow for the broadest comparisons with other projects, in integrated and multiple sectors, and in other countries. Inadequate baseline or appraisal information was generated at the CLP launch. Recommendation: USAID should direct future health projects to conduct a comprehensive needs assessment that leads to a detailed plan to build on the legacy of previous projects (e.g., Basic Health Services Project (BHS), Responsive Governance Project (RGP) and CLP notably the PPMs). An evaluation of the PPMs trained by both projects would establish a baseline for future interventions with them. Recommendation: The health sector merits periodic, mini-situation analyses as part of annual reviews to re-assess existing implementation conditions of the Yemeni Health NGOs, better understand their capabilities and limitations, and ensure adequate capacity building of the grantees. Recommendation: In planning new trainings for private sector midwives, USAID should engage the National Yemeni Midwife Association to conduct a study that would assess the needs for midwives for each governorate based on their population of women of reproductive age. Such a study would yield data on the existing number of midwives (both public and private) already trained , gaps in their training including knowledge, and project what each governorate needs in order to meet the needs of these women especially in hard to reach areas and/or marginalized groups There is evidence that the project made some efforts to perform basic data verification to improve the quality of data on the number of beneficiaries reached, as well as on process data related to training. Recommendation: USAID should support implementing partners to conduct a comprehensive baseline needs assessment at the outset of any health program which can buttress a detailed plan of options to build upon previous projects (e.g., BHS, RGP and CLP, notably the PPMs). Lack of M&E technical leadership in the form of a full-time M&E officer early on in the project led to missed opportunities to put in place a robust monitoring system that could be implemented throughout the life of the project, even in the event of staff changes. Because M&E was not institutionalized from the onset of the project, this contributed to ineffective flow and use of information between the field and the central office and therefore hindered monitoring of progress toward targets. Recommendation: even in fragile and transitional societies, particularly in the health sector, a sound work plan based on realistic targets with a strong M&E system requires leadership in place in all key positions (M&E especially) to design good plans and monitoring systems that will address gaps in a timely fashion and make the necessary adjustments. Work plans should have matching PMPs and be regularly updated and approved by USAID annually. Recommendation: Future USAID initiatives should work with the GIZ (the author of the model) to conduct a satisfaction mini-study to evaluate how the youth appreciated the information/ messages, how the information was used, and the outcomes if any in terms of key behavior changes in youth on reproductive health and other health issues of interest. This would allow for fine tuning messages ‘lost in translation’ and realize the most impact of this MoPHP best practice model. In theory, the GMS provides comprehensive and accessible data on status on activities, issues, completion certificates on closed projects. However, because this system was not in place from the beginning of the project, the full benefits of the GMS could not be exploited and had a major bearing on the ability to evaluate the project’s results. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 31 Recommendation: All project staff should be familiar with the PMP and be kept updated regularly on the modifications as they occur. Community health workers should be involved in those updates. By involving the field staff in the feedback loop, useful information could be obtained quickly for planning and reporting purposes. 6.3 Efficiency, Effectiveness and Resilience The CLP health project demonstrated reasonable resilience in re-inventing itself in response to changing conditions in Yemen and evolving analyses and priorities. There was a significant amount of under￾spending on the project, with only 66% of money allocated for activities actually spent through August 2013. The CLP successfully re-focused its program design and reoriented implementation to geographical areas that were more stable to respond to the political situation. It was able to produce some results as best as its coping mechanism would allow. Adjustments made by USAID and CLP (e.g., shifting to direct implementation; shifting focus from a rural to peri-urban settings in May 2011) were key. The Cooperative Agreement and the grants under it were designed in response to stabilization goals, but instability, in turn, hindered the implementation of health sector outreach in Yemen. Efficiency calculations were hampered by standardized data collected. The CLP was not set up to link activity costs to specific results and outcomes. Financial cost data were insufficiently disaggregated by activity (i.e. training, provision of equipment/materials to allow for cost effectiveness analysis. Recommendation: USAID, in tandem with other donors and with the MoPHP should support more inclusive value-for-money studies on models including the MMT and PPM to draw out potential best practices. Partners should routinely document project expenditure data in so that cost effectiveness and cost efficiency analyses can be conducted in a standard manner in future projects. The sub-grant mechanism managed by Creative Associates for local nonprofits did not achieve sufficient control of the technical direction of activities and their quality standards. Grant recipients were often unfamiliar with what was expected of them, including their cost-share requirements. Given the inability of CLP staff to travel to program locations, and given the poor expertise in reporting back by the nonprofits, it would have required more time to train grantees to make the system work. The short-time frame of the health program militated against a full test of this grant mechanism. Based on available evidence, this small￾grants model produced too few results in too little time. Recommendation: Future USAID programs should ally with local partners already trained under the CLP, BHS and RGP, and who are familiar with USAID procedures to minimize the learning curve and sprint into action in the first year of implementation. This approach would allow more time for health initiatives to build capacity of new partners as part of their resilience and sustainability strategy. For new partners, assessments should verify their experience/positive track records implementing health projects, as well as measure absorptive capacities for new funding, ability to extend coverage of services beyond their current catchment populations, and their prospects to provide timely, consistent and accurate reporting on health outcomes besides metrics of services delivered. At the same time, USAID health projects should pursue more strategic branding in order to gain acceptance at the local level and leverage the project visibility toward expanded collaboration with the community, the MoPHP and the RoYG. The renovation of and provision of medical equipment to health facilities were worthwhile investments when viewed in the long-term as the equipment will contribute to improving the functioning of the medical environment and quality of services. The MMTs extended the coverage of reproductive health services at a modest cost to USAID to otherwise neglected areas and remote and displaced populations, achieving intended objectives. The PPM and MMT intervention strategies appear to have contributed toward increasing access to critical and life-saving health care and knowledge, particularly in rural areas. YMEP: Final Evaluation of the Health Program of the Community Livelihoods Project (CLP) 32 Recommendation: Future health projects should pay attention to high impact interventions with minimal cost. Working more on skill development of service providers to improve quality of health services, especially for women and children, and focus on life saving programs, such as Help Babies Breath, Post-Partum Care, Misoprostol use, and Family Planning. The MMTs achieved high visibility, and high impact while demonstrating RoYG commitment to serving vulnerable populations. They attracted a high volume of clients during their visits to communities, health facilities, and schools in those targeted governorates. MMT personnel used these opportunities to disseminate health messages about MNCH/FP/RH while clients waited to be seen by service providers: 82,100 clients received health education via MMTs, 2385 referrals were made. Recommendation: USAID should continue to adopt and support MMTs and midwives working in the private sector, with the provision of health education and the furnishing and equipping of their facilities. USAID can contribute to sustainability of mobile health systems service through a FARA, to improve access to services for the vulnerable groups, mothers and children in the remote areas. Recommendation: USAID should target the RGOY at national and governorate levels with technical and modest budget support to be able to take a more direct hand in planning for PPM and MMT activities. While the PPMs and MMT made noteworthy contributions to improving health access in rural areas, their resilience in the future will hinge on government interest, resources and management capacity. Recommendation: In future MMT and PPM strategies, partners and USAID should collaborate with other donors that can provide essential medicines and FP commodities (e.g. WHO, UNFPA, UNICEF) that the MMTs and PPMs need to dispense in the field. This planning will ensure lasting impact on the health of the clients. The CLP Health Sector achieved some level of synergy by building on the familiarity of the governorates with USAID through the BHS, and RGP and the fact that the mechanisms adopted – the PPM and MMT -- were already accepted by the MoPHP as best practices coming into the CLP project. Had the health sector component achieved greater synergy with other USAID projects (i.e., RGP) and integration with other CLP components (education, agriculture, water), there would have been an opportunity to explore a) efficiencies and/or cost savings achieved through pooling of resources and b) more-favorable results given project inputs. Creative Associate’s organizational strength in the education sector bode well for education-related activities of the CLP, but the organization may not have brought sufficient expertise to the health sector to achieve among the sectors. Recommendation: future projects should expend greater effort early on and throughout to explore layering and integration of project components, within and across USAID’s flagship projects. The health project could have gone considerably further to integrate with other flagship projects in Yemen even to achieve the overall goals of the USAID Global Health. Partners should invest in opportunities for cost-sharing/joint programming in areas of overlap (e.g., in terms of geographic focus, sector of work, target beneficiaries, etc.) with other projects, sectors and donors. To promote decentralized health access, implementing partners should direct aid toward a system of master trainers to be developed and supported at the governorates level so as to reduce dependency on centralized trainers in., They can support local skill development rollout and contribute to a more effective and sustainable decentralization. Recommendation: The GHOs should explore means to obligate or incentivize the selected candidates for the PPM program to remain active in providing the services to their local community for some period of years. If the PPM intends to move to another location, she should work along a pre￾established pathway with the GHO who can help to identify new locations.