This publication was produced at the request of the United States Agency for International Development. It was prepared by Social Impact, Inc. (SI) through Dr. Donald Lauro, Ms. Beverly Tucker, Dr. Joselito Vital, Ms. Carole Bandahala and Ms. Erica Holzaepfel. USAID/Philippines Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM2) Project Final Evaluation Report Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM2) Project Final Evaluation Report June 2013 USAID Evaluation Services IQC No: AID – RAN – I- 00-09-00019 Task Order No: AID-492-TO-13-00004 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The evaluation team would like to express its appreciation to USAID/Philippines and the PRISM2 Project for their support throughout this evaluation. In particular, Mission Director Gloria Steele and Deputy Mission Director Reed Aeschliman provided helpful guidance at the outset of the team’s visit to the Philippines and during debriefing before departure. The team was ably supported by USAID’s Monitoring and Evaluation Unit, consisting of John Callanta, Monitoring and Evaluation Specialist, Fatima Verzosa, Project Development Specialist and Gender and Development Specialist, and Gerald Britan, Senior Strategy and Evaluation Advisor for USAID/Philippines. This unit briefed the team on this assignment and encouraged the team to maintain both flexibility and an unbiased (outsider) perspective while carrying out the evaluation. Mr. Callanta was also readily available by email and phone throughout the assignment to provide timely assistance on a variety of matters. The staff of PRISM2 gave unstintingly of their time to help ensure that the team received the data it needed; staff interviewed as key informants were candid, forthright, and patient. Finally, the team greatly appreciated the availability and willingness of many partners from both public and private sectors as well as several others not directly connected to PRISM2 to be interviewed. We hope and trust that the findings, conclusions, and recommendations contained in this report will help PRISM2, USAID/Philippines, and service providers throughout the Philippines improve and increase much needed Family Planning and Mother and Child Health services. Finally, the team could not have conducted this assignment without the very skilled assistance of Jenet Minanga, contracted by Social Impact to provide logistical support for the evaluation team. TABLE OF CONTENTS ACRONYMS ........................................................................................................................................................................ I EXECUTIVE SUMMARY ................................................................................................................................................... I 1. INTRODUCTION ........................................................................................................................................................ 1 2. COUNTRY CONTEXT .............................................................................................................................................. 1 2.1 FP/MCH in the Philippines .................................................................................................................................... 1 2.2 USAID/Philippines Support for the Private Sector ......................................................................................... 2 2.3 PRISM1 Project ....................................................................................................................................................... 2 3. PROJECT DESCRIPTION ........................................................................................................................................... 3 3.1 Project Design ......................................................................................................................................................... 3 3.2 PRISM2 Project Implementation ......................................................................................................................... 3 4. EVALUATION FRAMEWORK .................................................................................................................................. 4 4.1 Evaluation Purpose ................................................................................................................................................. 4 4.2 Evaluation Methodology ........................................................................................................................................ 5 4.3 Evaluation Directives and Questions ................................................................................................................. 7 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS .......................................................................... 9 5.1 Prism2 Achievements ............................................................................................................................................ 9 Question 1 ...................................................................................................................................................................... 9 Question 2 .................................................................................................................................................................... 12 Question 3 .................................................................................................................................................................... 14 Question 4 .................................................................................................................................................................... 16 5.2 Project Implementation and Management ...................................................................................................... 19 Question 6 .................................................................................................................................................................... 19 Question 7 .................................................................................................................................................................... 21 Question 5 .................................................................................................................................................................... 24 5.3 Future Prospects for Public-Private Partnerships ......................................................................................... 26 Question 8 .................................................................................................................................................................... 26 Question 9 .................................................................................................................................................................... 29 Question 10: ................................................................................................................................................................. 30 6. PROGRESS TOWARDS THE THREE PROJECT COMPONENTS ............................................................... 33 7. LESSONS LEARNED .................................................................................................................................................. 37 8. LIST OF ANNEXES .................................................................................................................................................... 39 ANNEX A: STATEMENT OF WORK (*Abridged) ................................................................................................ 40 ANNEX B: EVALUATION METHODOLOGY ....................................................................................................... 54 ANNEX C: SITE SELECTION ..................................................................................................................................... 57 ANNEX D: CATEGORIZATION OF EVALUATION QUESTIONS BY DIRECTIVE ................................... 59 ANNEX E: EVALUATION TEAM ROLES AND RESPONSIBILITIES ................................................................ 61 ANNEX F: LIST OF INITIAL DOCUMENTS REVIEWED .................................................................................... 65 ANNEX G: ANNOTATED BIBLIOGRAPHY OF SELECTED DOCUMENTS REVIEWED ......................... 66 ANNEX H: OVERARCHING QUESTIONS AND GENERIC QUESTIONNAIRE ....................................... 82 ANNEX I: COST ANALYSIS ....................................................................................................................................... 91 ANNEX J: KEY INFORMANT INTERVIEWS ......................................................................................................... 93 ANNEX K: EXTRACTS FROM KEY INFORMANT INTERVIEWS.................................................................... 98 ANNEX L: ANALYSIS OF THE PRISM2 PERFORMANCE MONITORING PLAN (PMP) ...................... 107 ANNEX M: PRISM2 ORGANIZATION CHART .............................................................................................. 110 ANNEX N: TABLE 1 FROM PRISM2 Q14 REPORT (05-16-13) .................................................................... 111 ANNEX O. RESOLUTION OF STATEMENT OF DIFFERENCES (SOD) ................................................... 130 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report ACRONYMS ADP Alternative Distribution Points ANC Antenatal Care AO Administrative Order BCC Behavior Change Communication BEmONC Basic Emergency Obstetric and Newborn Care BEST Best Practices at Scale in the Home, Community and Facilities BTL Bilateral Tubal Ligation CA Cooperating Agency CB Capacity Building CEmONC Comprehensive Emergency Obstetric and Newborn Care CHD Center for Health Development CHED Commission on Higher Education CHO City Health Office/Officers CMS Commercial Markets Strategies CO Central Office COP Chief of Party COR Contracting Officer’s Representative CPR Contraceptive Prevalence Rate CYP Couple-Year of Protection DCOP Deputy Chief of Party DOH Department of Health DOLE Department of Labor and Employment DOTS Directly Observed Treatment Short Course EOP End of Project FHS Family Health Survey FHSIS Field Health Service Information System FP Family Planning FP-CBT1 Family Planning-Community Based Training Level 1 FY Fiscal Year GBV Gender-Based Violence GoP Government of Philippines GS Gender Strategy HealthGov Strengthening Local Governance for Health HMIS Health Management Information System HMO Health Maintenance Organization HPDP Health Policy Development Project ICV Informed Choice and Voluntarism ILHZ Inter-Local Health Zone IMAP Integrated Midwives Association of the Philippines IR Intermediate Result ISO International Organization for Standardization s IUD Intra-Uterine Device IWG Informal Working Groups JHPIEGO Johns Hopkins Program for International Education in Gynecology & Obstetrics LA Long-Acting LGU Local Government Unit LMA Local Market Area LMAM Local Market Area Manager MCH Maternal and Child Health Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report MCP Maternal Care Package MDG Millennium Development Goals MHO Municipal Health Office MMR Maternal Mortality Ratio MNCHN Maternal, Neonatal, and Child Health and Nutrition MOA Memorandum of Agreement NDHS National Demographic Health Survey NGO Non-Government Organization OC Oral Contraceptive OH Office of Health, USAID/Philippines ORS Oral Rehydration Solution PAC Post-Abortion Care PBSP Philippines Business for Social Progress PHO Provincial Health Office/Officers PhP Philippine pesos PhRO PhilHealth Regional Office PIRS Performance Indicator Reference Sheets PM Permanent Methods PMP Performance Monitoring Plan POPCOM Commission on Population PPM Private Practicing Midwives PPP Public-Private Partnership PPR Performance Plan and Report PRISM Private Sector Mobilization for Family Health Phase I PRISM2 Private Sector Mobilization for Family Health Phase II RH Reproductive Health RTI Research Triangle Institute SAM Senior Area Managers SBA Skilled Birth Attendant SDN Service Delivery Network SHIELD Sustainable Health Improvements through Empowerment and Local Development SHOPS Strengthening Health Outcomes through the Private Sector SI Social Impact, Inc. SIFI Sugar Industry Foundation, Inc. SO Strategic Objective TB LINC Linking Initiatives and Networking to Control Tuberculosis TIPPPs Technical Initiatives for Public-Private Partnership TSP Technical Support Package TWG Technical Working Group UHA University Health Alliance UHC Universal Health Care USAID United States Agency for International Development USG United States Government WHO World Health Organization Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report i EXECUTIVE SUMMARY EVALUATION PURPOSE USAID/Philippines contracted Social Impact, Inc. (SI) to conduct a final performance evaluation to determine the Private Sector Mobilization for Family Health Phase II (PRISM2) Project’s progress through the first three years of implementation. Following four directives in the evaluation Scope of Work (SOW), the evaluation team provided the USAID/Philippines Mission with answers to questions regarding specific aspects of the project’s performance: (1) determine the effectiveness of the project as measured by its outcomes and outputs compared with baselines and targets; (2) assess the effectiveness of project design, implementation, and management, including the extent to which the project achieved contract deliverables and established productive relationships with key clients from the government and the private sector; (3) identify PRISM2’s most significant contributions to the establishment of a private sector market for Family Planning (FP) and Mother and Child Health (MCH); and (4) assess the success of the collaboration between PRISM2 and other projects in USAID/Philippine’s Health portfolio and how well PRISM2 complemented the priorities of the United States Government (USG) and the Government of Philippines Department of Health (GoP/DOH). The four-person evaluation team, accompanied by a logistician and a central-level DOH Health Officer, conducted this evaluation using the directives above to summarize findings, conclusions, and recommendations. These recommendations are intended to help the Mission and its implementing partners improve PRISM2 implementation in its final year and identify lessons that can be considered for USAID’s other regional health projects. CONTEXT AND PROJECT BACKGROUND Due to various constraints, women’s health outcomes in the Philippines are not progressing as planned to achieve the targets set in the Millennium Development Goals (MDGs). National-level support for FP in the Philippines has been inconsistent since the 1970s. In the 1990s, responsibility for health program implementation devolved from the national to the sub-national level, leading to several shortcomings and challenges. Currently, the disbursement of funds received from the local Internal Revenue Allotments for health programs, including FP, is determined by the local government units (LGUs) and other subnational administrative units. The DOH has long recognized the challenge of widening coverage of health products and services for its citizens, particularly among the poor. The DOH has also acknowledged that private sector contributions are crucial to increasing the scope and scale of available health services. The Philippines Health Insurance Program (PHIC or PhilHealth), funded through premiums paid by LGUs on behalf of indigent families, is the primary source of financial risk protection for the poor. However, PhilHealth coverage of FP and MCH remains limited and inconsistently administered. Since the 1990s, USAID/Philippines has engaged the private sector as part of its assistance to expand FP/MCH programs, including investments in social marketing, workplace health programs, and support for midwives and their birthing homes. The PRISM Project (2004–09) marked significant investment to consolidate efforts and advance FP/MCH through the private sector. PRISM2, with its expanded vision of public-private partnership (PPP), envisioned a broader approach for applying private sector capabilities to advance public sector FP/MCH goals. The PRISM2 Project operates under a 5-year (October 2009–14), nearly $35 million contract with Chemonics International. The PRISM2 contract objectives are: 1. Increase and sustain private sector provision of quality family planning and maternal and child health services and products. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report ii 2. Increase utilization of quality family planning and maternal and child health services and products in and from the private sector. 3. Improve the policy environment for private sector provision of services and products. PRISM2 was designed to produce nationwide impact. From 2009 to 2011, the project covered seventy￾seven provinces and cities, reaching an estimated 73.6 million people. Throughout its first two years, PRISM2 approached the development of PPPs by conducting consultations with government partners and identifying eight critical areas of technical need. In 2012, USAID and PRISM2 leadership recognized the need to capitalize on achievements to date and ensure depth of engagement and thus streamlined project efforts to target only thirty-six sites covering an estimated 46.7 million people. Additionally, leadership narrowed the project’s focus from eight technical need areas to four (private practice midwives [PPM], hospitals, long-acting and permanent methods [LA/PM], and securing contraceptive supplies), with the remaining four Technical Initiatives for Public-Private Partnership (TIPPPs) becoming four cross-cutting areas (National Health Insurance Program, training, behavior change communication [BCC], and monitoring and evaluation [M&E] for FP/MCH). USAID provided guidance to focus LA/PM on itinerant service delivery and PPMs on accreditation. EVALUATION SCOPE AND METHODS This performance evaluation focuses on the effectiveness and sustainability of activities implemented since the inception of PRISM2 in October 2009 through March 2013, nineteen months before the project’s end. Adhering to USAID Evaluation Policy guidelines and standard performance evaluation practices, the evaluation team utilized a combination of complementary qualitative methods to collect and analyze data. These methods included: reviewing over eighty-five documents, interviewing fifty-nine key informants, conducting seven focus group discussions, visiting six PRISM2 local management areas (LMAs), observing five project activities, and visiting several sites not directly supported by PRISM2 (birthing homes, clinics, hospitals, and one university) but within project LMAs. Abiding by the four directives presented above, the team used ten evaluation questions provided in the SOW as the framework for this evaluation. SUMMARY OF EVALUATION RESULTS BY DIRECTIVE The following is a summary of findings, conclusions, and recommendations for each of the four evaluation SOW directives. Ten evaluation questions, also included in the SOW, address these four directives in greater detail; the full report answers each of these questions. 1. Determine Project Effectiveness: Investigate national-level achievements in terms of two outcomes and two outputs. The PRISM2 project SOW specified the following expected increases in outcome and output indicators:  Increased private sector contribution to the contraceptive prevalence rate (CPR) by 8.0 percent “from a baseline of 40.8 in 2009 to 48.8 in 2014”1  Increased proportion of deliveries assisted by skilled birth attendants (SBAs) by 8.0 percentage points “from a baseline of 62 percent in 2009 to 70 percent in 2014”  “Increased contraceptive market for pills by 2 percent annually”  “Increased contraceptive market for injectables by 6 percent annually” 1 From Section C of post award debrief notes and subsequent contract modification, correcting an error in the original CPR baseline and expected outcome figures. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report iii Findings PRISM2’s indicator for CPR is based on national-level data from the 2008 National Demographic and Health Survey (NDHS); however, CPR measured at the national level does not provide a reliable measure for assessing PRISM2’s performance. Determining PRISM2’s contributions to changes in CPR at the national level would require tracking project-level contributions to CPR by monitoring project outputs, for instance, the number of additional FP users due to project inputs such as trained and accredited PPMs and private hospitals providing FP services. Not only is CPR not measured by PRISM2 at the project level, but also national level surveys with which project level data would need to be compared are only conducted once every five years. Furthermore, PRISM2 was delayed in implementing training for LA/PM and direct support for marketing contraceptives, though the latter began in May 2011, soon after USAID approved the PRISM2 grants manual. While these delays have thus far limited the potential for PRISM2 to reach the large number of FP clients needed to have an effect on national￾level CPR, in line with the expected increase in private-sector FP users included in the project SOW, it is likely that the last eighteen months of the project will see a rise in the provision of services for LA/PM. However, to increase CPR by 8 percent in five years would require adding over 400,000 FP clients per year, a number far beyond PRISM2 achievements.2 The indicator for deliveries assisted by doctors, nurses, and midwives or skilled birth attendants (SBAs) also relies on secondary data from NDHS or the Family Health Survey (FHS). This indicator is also probably neither an appropriate nor a reliable measure of PRISM2’s project-level performance. The expected increase over five years of 8% would require 40,000 additional SBA-assisted births a year.3 This is largely beyond the reach and timeframe of PRISM2. Though PRISM2 may well be adding to the number of SBA-assisted births by increasing the number of private providers accredited for maternal care package (MCP), it is not at a sufficient scale to affect this national-level outcome. The estimated number of government midwives alone is at least 20,000, as reported by the President of the Integrated Midwives Association of the Philippines (IMAP); private midwives probably also number in the tens of thousands.4 Unlike with CPR, however, PRISM2 does monitor an output indicator tracking the number of deliveries assisted by SBA, which can provide a limited understanding of the project’s contribution to the national￾level outcome indicator of the percentage of SBA-assisted births. PRISM2 collects monthly data on the number of deliveries by PRISM2-assisted private practicing midwives (PPMs). Performance-to-date against the five-year, cumulative target of 43,596 is 130% or 56,679 SBA-assisted births. Though many of these PPM-assisted births would have occurred without PRISM2’s presence, these data indicate that the project is making some positive contributions towards this outcome. PRISM2 did not set a baseline for this indicator or provide a rationale about how annual targets were set, undermining a more complete analysis of how the project connects its activities to contributions to this outcome measure. 2 Eight percent over five years is 1.6 percent per year. The Population Reference Bureau estimate for 2012 midyear female population is 26.2 million. Therefore, adding 1.6 percent per year to CPR requires an estimated 419,200 additional users per year. 3 Eight percent over five years is 1.6 percent per year. The current population of the Philippines is approximately 100 million, and an estimate of the crude birth rate provided by the Population Reference Bureau is 25 births per 1,000 population. This yields an estimated 2.5 million births per year. 1.6% of 2.5 million is 40,000. 4 The team did not find a document referencing the number of midwives. However, USAID and the president of IMAP both provided estimates in this range. This estimate is consistent with the large numbers of births annually in the Philippines. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report iv For the two output measures on contraceptive sales, PRISM2 cumulates the volume of oral and injectable contraceptive sales using secondary data obtained from IMS Drug Distribution Data (IMS￾DDD) on an annual basis. According to its Performance Monitoring Plan (PMP), PRISM2 committed to a 2 percent annual increase in oral contraceptive sales at the national level. The 2009 baseline for this indicator was 8,575,401 cycles. However, the 2010 target was set at 12.22 million, approximately a 12 percent increase. The five-year cumulative target is 13.23 million, with a year-to-date accomplishment rate of 63 percent. The last period for which data was available was October 2012. Though largely unable to significantly contribute to broad national-level measures, during the past year, PRISM2 has focused considerable resources towards achieving outputs that relate to these outcomes. For example, the number of people trained in MCH and FP/reproductive health (RH) is 518 and 439, respectively. Performance-to-date against the project’s five-year targets for these indicators is 86 percent and 73 percent, respectively. Other key outputs of PRISM2 include the number of national policies supporting private sector provision of FP/MCH information, products, and services; the project has achieved 85 percent (29 policies) of its five-year cumulative target of 34 policies. This indicator refers to governance, regulatory, and financing policies emanating from DOH/CHDs and PhilHealth/PhilHealth Regional Offices (PhROs). Another indicator measuring policy development at the LGU level reflects the number of provinces and independent cities that submitted or issued new local policies in support of private sector provision of FP/MCH services and products. PRISM2 has accomplished 56 percent (20 out of 36) of its five-year cumulative target for this indicator. Conclusions For all but one development objective level indicator – tracking the number of births assisted by PRISM2-assisted PPMs – it is not possible to accurately assess PRISM2’s contributions towards these expected outcomes (CPR and SBA) and outputs (contraceptive sales) because data are not available at the appropriate level of disaggregation. Significantly contributing to CPR, SBA-assisted births, and contraceptive sales at the national level is beyond what the current project can achieve and perhaps beyond the manageable interests of PRISM2 even at its outset. Indicators based on national survey data may provide useful measures for entire programs or investment strategies. However, as the 2012 evaluation of USAID’s MCH/FP portfolio substantiates, this may not produce measurable results. For example, for CPR this portfolio evaluation revealed no statistically significant province-level differences between 2006 and 2011 where USAID provided support to FP/MCH projects. Though SBA-assisted deliveries showed statistically significant positive changes had occurred during the same period, nearly the same level of improvement also occurred in provinces where USAID had no presence. With these results in mind, caution should be exercised about using national-level or even province-level indicators to measure effects of a single project. However, where changes in CPR and SBA-assisted births are considered to be important outcomes and a particular project is anticipated to operate at sufficient scale, project-level surveys could be commissioned. The Project has advanced PPP, established service delivery networks, and further involved PPMs in achieving public sector FP/MCH goals. However, the gap between some of the project’s indicators that rely on national-level data and the project’s activities, which are implemented in select areas of the country, make it difficult and in some cases impossible to interpret the project’s performance. The outcomes and outputs discussed above do not fully portray PRISM2’s achievements. A theory of change or logic model that connected inputs to output targets to eventual outcomes may have led to improved measures for tracking project progress. Early on, when the PMP was being developed by the project and approved by the Mission, developing a theory of change or logic model to connect what PRISM2 does to national-level outcomes such as CPR and percentage of SBA, could have proven very useful. Recommendations Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report v 1. If the new health projects are expected to contribute to regional outcomes and outputs, USAID should ensure that each project elaborate a theory of change or logic model that connects planned activities and indicators to these higher level measures. 2. Indicator data collected on a routine basis at the project level will provide a much richer source of learning about the project’s actual performance than relying on national-level statistics that are reported every three to five years. An example of a high-level indicator that captures achievements of other activities would be the establishing and/or strengthening of service delivery networks (SDNs). 3. USAID to ensure the inclusion of an evaluation design in M&E plans of new projects that will allow for more rigorous comparison of populations or geographic areas receiving and not receiving USAID assistance. 2. Assess Effectiveness of PRISM2: Determine effectiveness of design and implementation in relation to achievement of contract deliverables and relationships with key clients from the government and private sectors. Findings Building on previous private sector investments in FP/MCH, USAID/Philippines proposed a bold design for PRISM2. The first Project Chief of Party (COP), Mario Taguiwalo, elaborated a vision toward sustaining private sector participation in achieving public health goals: DOH stewardship to foster cooperation and connection between the public and private sectors. During initial implementation, PRISM2 involved the DOH regional Centers for Health Development (CHDs) as primary focal points from which stewardship would emanate. The development of eight Technical Initiatives for Public￾Private Partnership (TIPPPs) resulted. Unfortunately, Mr. Taguiwalo’s sudden illness during much of 2011 and subsequent passing in April 2012 contributed to leadership loss at a critical moment in carrying this vision forward. The situation was followed by the concurrent departure of the Deputy COP, the passing of the Mindanao Senior Area Manager (SAM), and two subsequent short-term assignments of acting COPs. PRISM2 did not have a permanent COP again until May 2012. From a management perspective, perhaps little could have been done to effect a smoother transition. However, as reported to the evaluation team during interviews, the vision and energy with which PRISM2 started was also largely lost in the process. The Chemonics staff member sent from Washington, D.C. in September 2011, Shaun O’Neil, extended his stay, became acting COP, and helped rejuvenate Project efforts. A facilitated all-staff meeting to develop the Year 3 work plan was an important part of this. As one high-level PRISM2 staff member noted, Year 3 became the “Year of the LMAM [Local Market Area Manager].” Fully engaging field as well as technical staff helped to re-launch momentum for promoting PPP; subcontractors also gained fuller understanding of their roles in the project. With project implementation lagging, USAID/Philippines subsequently stepped in to make substantial reductions both geographically (from seventy-seven LMAs to thirty-six) and substantively (from eight to four TIPPPs, with the remaining 4 TIPPPs becoming secondary and/or cross-cutting). Although this and other midcourse adjustments, such as shifting project focus from the formal workplace to informal working groups (IWGs), changed the scope of the project, no corresponding changes were made in high-level expected outcomes or outputs. Nevertheless, despite slow start-up, delayed recovery from leadership loss, and consequent implementation shortfalls, in some settings the PRISM2 design itself has markedly changed the landscape by increasing DOH interest in, and involvement with, the private sector. As one City Health Officer (CHO) reported, despite only sporadic contact with the project, she credited PRISM2 with catalyzing her conversion from Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report vi rejecting private practitioners to embracing them as partners. A number of other sub-national DOH staff similarly expressed openness to private partners, though sometimes attributing this evolution more to the Linking Initiatives and Networking to Control Tuberculosis Project5 (TB LINC) than to PRISM2: … The grey area of PPP is how to do it. We had patterned it on the TB program – important to helping us. Started by WHO [World Health Organization] and the Global Fund a number of years ago, the TB DOTS was the first one to have this PPP. When we are trying to reap the harvest, along comes the resistant strains. …most of Well Family clinics are housed in hospital based; providing feedback on types of referrals; there was a very good TB LINC Project … this was definitely reaching out through the private sector to provide DOTS, for example. …we had an initial [PPP] partnership after devolution. We had the local health board and inter-local – invite stakeholders from NGO [Non-Governmental Organization], civil society … so we had partnerships already from the 1990s – initially we already had partnership there was already a strong link between the private – especially in TB because we had the PPP coordinating council with regards to AO [Administrative Order] or memo we had that already and with the PhilHealth administration with universal health care we have already that since devolution so now it’s been strengthened. The PRISM was already in – I was also a resource speaker for PRISM1 orientation – there was a series of activities with the government and private sector. We have had PRISM1 and PRISM2. To promote linkages between the public and private sectors, PRISM2 initially only engaged the DOH at the Central Office (CO), but routinely connected with the regional CHDs. At the provincial level, the DOH participated in the training of PPMs and, when accreditation criteria were met, supported PPMs and their private birthing homes. In return, PPMs affiliated with PRISM2 were more likely to provide service statistics to the DOH than those not affiliated with the project. Though PRISM2 was active at the sub-national level, the indicators used to track achievements did not capture the complexities the first few years of the project that were dedicated to capacity building (CB)6 and stewardship within the public sector to continue or expand PPP. PMP indicators P21 and P22, which respectively seek to measure the number of local organizations participating in PPPs for FP/MCH and the percentage of DOH-CHDs with improved capacities to sustain program initiatives beyond project life, were not monitored until Year 3 of the project. Moreover, the definition of P21 falls short of fully reflecting an effective PPP, as it does not require evidence of a two-way, reciprocal, functioning referral system. CB data can be qualitative and quantitative, gathered via focus groups, individual interviews (with both closed- and open-ended questions), surveys, and document reviews. Possible indicators that help capture the complexities of CB include: number of joint activities with other organizations; frequency, type, and depth of contact with national and sub-national organizations within public and private sectors; 5 Philippines Business for Social Progress (PBSP) was the first non-governmental organization (NGO) grantee in the Philippines to manage TB LINC, working closely and assisting the DOH and LGUs deliver quality services for TB control and prevention. PBSP was able to gain the support, cooperation and engagement of the private sector, especially medical practitioners, with the DOH’s strategy of Directly-Observed Treatment Short Course (DOTS). DOH appreciated this strategy since private practitioners identify, treat and/or refer TB clients for treatment. Treatment compliance has been the perennial problem of the DOH with regards to the TB program. With the private sector coming into the picture, more patients are advised and supervised with respect to their treatment compliance. 6 Capacity building resources include: A Guide to Monitoring and Evaluation of Capacity-Building Interventions in the Health Sector in Developing Countries (http://www.cpc.unc.edu/measure/publications/ms-03-07) and the Capacity Development Results Framework (http://siteresources.worldbank.org/EXTCDRC/Resources/CDRF_Paper.pdf). Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report vii and types and frequency of outcomes from links with public and private organizations. The lack of continuous DOH CO involvement throughout the project led to little subsequent high level DOH buy￾in for continuing the vision, tools or activities developed by the Project. Though the Secretary of Health sets overall directions for national health efforts, closer, consistent connection of PRISM2 with central DOH would also help advance PPP at the sub-national levels. Conclusions With Service Delivery Networks (SDN) embedded in the 2008 DOH strategy for Maternal, Neonatal, and Child Health and Nutrition (MNCHN) and TB LINC providing a first positive experience to connect DOH with the private sector, PRISM2 was well-timed, designed, and positioned to build PPP. Unfortunate and unforeseen circumstances led to delays that placed the Project in catch-up mode to reach previously established targets and outcomes. For example, one unforeseen circumstance was the lack of established SDNs to ‘strengthen’. Thus, a significant effort was made to establish SDNs for subsequent strengthening, both activities that require considerable effort to develop a common vision among the many stakeholders in the public and private sectors. Having political support at the local level for a public-private collaborative effort was the most commonly cited factor for ease in establishing a SDN: It’s not that easy to engage activities at the local level – the minds of the different Local Chief Executives – this year there is a new batch of incumbent mayors – so changing their minds assuming they have been already exposed to this – the newly elected mayors who have no head of how local governance is done – this will be a challenge for us and for PRISM. The problem with devolution there are many Local Chief Executives who are not supportive – the unavailability of commodities – provide to the local factories with their clinics. There was a forum given by PRISM [for Municipal Health Offices (MHOs)] on operations of the private lying in clinics as to how it would be operated so they would be convinced to put up private lying in clinics in their area. … some MHOs are supportive and others are not. Those that are not can block it. They don’t want competition with their government-sponsored facilities. The project also focused more on achieving outputs than building toward sustainable change. For example, perhaps partly because the project is far behind, training has become its major activity, yielding output data on numbers trained but little by way of follow-up to ensure that trainings lead to improved practices and strengthened SDNs.7 USAID well recognizes this deficiency. As one current USAID staff member stated: “if (PRISM2) train(s) a health provider, they are just stuck on training and they don’t monitor after the training to see if they are applying the training – same with midwives, they gave a lot of trainings to conduct usapan series – but to know how many of those trained have actually applied the knowledge or skills is very difficult to see.” Given implementation delays, low output levels, and little direct connection with central DOH, many PRISM2 activities will only be sustained if the CHDs and USAID regional health projects build on the initial gains. Recommendations 1. To reflect the Year 3 reduction to thirty-six LMAs and four TIPPPs, and the shift from workplace to IWGs, PRISM2 should also correspondingly adjust, with USAID concurrence, targets for major outputs and outcomes. 2. Even at this late date, PRISM2 could be encouraged to focus data collection on particular 7 A strengthened SDN, as defined by PRISM2, is a “6-1-1,” that is, six Alternative Distribution Points (ADPs), one private sector provider, and 1 private facility. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report viii provinces, LMAs, or even SDNs to showcase what has occurred and not occurred as a basis for passing lessons learned along to other projects. 3. Identify PRISM2’s Key Contributions: Identify unique achievements and methods/approaches of the project and compare with other effective private sector approaches for FP/MCH. Findings The focus of PRISM2 on PPMs builds on a long history of working with private midwives, in the Philippines and elsewhere, to improve FP/MCH services and, ultimately, outcomes. The unique aspect of PRISM2 has been in forging public-private referral networks between PPMs and public and private hospitals and fostering PhilHealth accreditation of the PPMs. While the project counts and reports on numbers of PPMs both accredited and accreditable (at present 23 and 150, respectively), it is not clear when a public-private referral network is in place and functional. Establishment of a strengthened SDN, for which currently three qualify out of the thirty-six targeted by the end of Project, may be a poor proxy measure. A functional, two-way, referral system that encompasses private and public sector partners is what PRISM2 should be aiming for within an SDN. Currently, referrals within SDNs are neither well documented nor well developed, in part because the public sector component was often not in place. Building on previous social marketing investments made by USAID and other donors, PRISM2 continued a relationship started in the first PRISM Project with Alphamed. This led to new contraceptive products being added and marketed, including contraceptives new to the Philippines as well as other MCH products and supplies. Through a one-year grant (extended to 15 months), Alphamed has contributed just over 200,000 CYP nationwide. Alphamed has also been a major participant supporting alternative distribution points (ADP) as well as other alternative distribution outlets.8 This builds on prior success with sari-sari stores, barangay boutiques, and PopShops. Particularly among the ADPs established with LGUs and PPMs, Alphamed expresses confidence that these are sustainable. The recent end of the Alphamed grant will provide a timely test. In its winning proposal, Chemonics included Philippines Business for Social Progress (PBSP) as a subcontractor. Under the prior PRISM Project, PBSP had connected with some 500 industries to increase awareness of FP/MCH. PRISM2 continued and initiated engagement in some formal workplace settings, including an impressive partnership wherein grantee Sugar Industry Foundation, Inc. (SIFI), linked with the Sugar Mill and Refinery in Talisay City in Negros Occidental. However, activating PBSP’s broader platform was delayed until Year 3, by which time USAID/Philippines had concerns about providing FP services and products within formal workplaces. With the subsequent shift to IWGs, now, well into Year 3, PBSP is gaining momentum with such informal workforces as pedicab drivers, market venders, and cooperatives. Had PBSP been part of PRISM2 from the outset, as their proposed subcontractor status indicated, progress within workplaces as well as with IWGs would likely have been greater by this time. Other ongoing efforts within PRISM2, such as the study of oxytocin in a Uniject delivery system, and work with youth, could bring needed innovations to the Philippines. However, these Mission-promoted initiatives, as useful as they could prove, are peripheral to achieving core PPP results. More germane to PPP are experiences and experiments occurring in other countries with vouchers, outsourcing, and other arrangements for involving private practitioners in public health provision. Unfortunately, neither 8 An ADP is defined as a private-sector distributor (local or individual) who requests restocking of a FP/MCH commodity within a 6-month period for two consecutive periods. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report ix the Mission nor Chemonics ensured that such knowledge was brought into PRISM2: …in terms of the CHD outsourcing expenses, the government has no capacity to contract directly with the private sector groups to do the training … They can't move their pipeline …The trainers are hired individually as consultants as opposed to through their organization. The outsourcing was not happening…and there was little hope that it would happen any time soon. Conclusion The design itself, especially as rolled out in the initial years, is PRISM2’s most unique, innovative feature. Otherwise, both broadly to advance FP/MCH and specifically to further PPP, what the Philippines needs is the application of practices that have worked well, both within the Philippines and elsewhere. The shift from the formal workplace to IWGs was a USAID decision. The result, however, was a missed opportunity for PRISM2 to serve readily-accessible populations, likely with high unmet need for FP/MCH, in as many as 500 industrial settings, nearly 300 of which had previously expressed interest in participating. Specifically, the project missed opportunities to reach the “captured” audiences of young women and youth in formal workplace settings. Unlike their IWG counterparts, who are very mobile and non-permanent, and thereby hard to access for additional services and follow-up visits, these prospective clients in formal work settings could easily be accessed for services and counseling, as they are directly linked to their respective companies or factories. Another loss was in PRISM2’s lack of connection to comparable, contemporary experiences with PPP in other countries. Recommendations 1. PRISM2 to use data, practitioner, and client flows within an SDN to measure whether and how well referrals are functioning within the SDN. 2. PRISM2 and USAID to connect with, and learn from, practices and experiences with PPP ongoing elsewhere. USAID’s global Strengthening Health Outcomes through the Private Sector (SHOPS) project provides useful connections to ongoing efforts. 3. USAID to encourage the new, regional health teams to incorporate and build on lessons and practices from a range of countries into their projects. This would involve, in addition to useful approaches for working with the private sector (such as those featured on the SHOPS website), considered exploration of practices such as outsourcing and vouchering that are occurring in a number of countries, including populous Indian states such as Gujarat and Bihar. 4. PRISM2 to provide the DOH with technical assistance to establish a system or mechanism to contract directly with institutions which have certified private trainers to promote FP/MCH. 5. In the time remaining, PRISM2 should fully document and share field-level practices where PPP has significantly advanced. This would both demonstrate the project’s positive accomplishments and help the new regional FP/MCH projects. 6. PRISM2 and the new regional health projects should promote PPP by supporting the DOH to outsource FP- and MNCHN-related services. 4. Assess PRISM2’s Collaboration: Measure how the project collaborated with other USAID/Philippines Health Portfolio projects and complemented strategies and priorities of the USG and the host country DOH. Findings Collaboration with other USAID projects has been neither notable nor strong. Beyond initial orientation and periodic coordination meetings, collaboration did not appear to result in significant efforts to work together on shared challenges. PRISM2’s design and focus on PPP did, however, complement the USAID Best Practices at Scale in the Home, Community and Facilities (BEST) strategy and projects implemented within the USAID portfolio. Recently, there has been some sharing of expertise, particularly with the new regional health projects overlapping PRISM2 priority areas in Luzon, Mindanao, and the Visayas. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report x PRISM2 has lost several staff to these new projects – not a favorable situation for PRISM2, but an advantage for USAID overall in terms of shared learning and experience across projects in the health portfolio. With the DOH, there is clear complementary between PRISM2 and the 2009 Administrative Order (AO) for the MNCHN strategy. In addition, especially with FP receiving higher-level attention in recent years, what PRISM2 is able to provide via private sector strengthening complements what the DOH would like to achieve overall. The team found that DOH at various levels often expressed having need for data from private sector providers. PRISM2 has had some positive, though still fledgling, effects on reporting from private sector facilities in hospitals and birthing clinics. Though DOH CO did report being involved and oriented to PRISM2 early in the project, little direct contact has occurred since that time. With regional CHDs, as well as CHOs and LGUs, the amount of contact and coordination ranges from considerable to peripheral, largely depending on joint progress being made on establishing and strengthening SDNs. The team found the LMAMs to possess a solid core of expertise that PRISM2 has built and should continue to build upon. Though some LMAMs already brought experience and connections with them to these positions, many have learned valuable lessons for working with private and public partners during their time with PRISM2: One problem in this region is that the regional staff is few. The LMAM does everything, and is currently traveling but still being held responsible for providing quarterly data. LMAMs manage the provision of technical assistance into the LMA. It’s more of a technical position than a management position. With gender receiving considerable attention from USAID globally as well as locally in the Philippines, new opportunities have emerged for collaboration and connection across projects. PRISM2’s Gender Strategy lays out an impressive plan for integrating gender across five critical features of FP/MCH service delivery, which could, in theory, be applied consistently across USAID’s Health Portfolio. The USAID Philippines Health Strategy also provides a strong basis for integration of stronger gender initiatives across the Health Portfolio. In the same way that general collaboration is lack lacking between PRISM2 and the other USAID projects, the team also found a dearth of collaboration or synergies in support of gender integration in FP/MCH between PRISM2 and other USAID health projects. PRISM2’s work plans reveal that inclusion of a gender-focus in the project has changed from a cross￾cutting initiative to more of a separate, stand-alone activity, as an add-on to specialized trainings, workshops, and discussions. PRISM2 may not have planned to address gender in FP/MCH beyond the project landscape or build partnerships with other USAID projects around this initiative. Nonetheless, at least in terms of proposed objectives, activities, deliverables, and outcomes, gender is being implemented largely as a standalone activity. Visits to several field sites, observations of numerous project activities, and interviews with many staff and partners did not produce evidence of widespread gender considerations. The team’s only engagement with gender-focused activities was observing the delivery of the gender-based violence module in the usapan facilitators’ training. Conclusions Collaboration with other USAID projects has not been a notably strong component of PRISM2. In contrast, PRISM2’s initial immersion to identify what PPP was and how it could effectively be implemented involved extensive connections with regional CHDs. Leadership loss, implementation delays, and consequent pressure to perform pushed the project to rely more on what was in its own control than on investing in time-consuming relationship-building at various levels. Recently, however, the project appears to be engaging with the new regional-based USAID-funded health projects. DOH stewardship to partner with the private sector remains essential to the long term success of what PRISM2 has started and would likely have advanced further with continuous PRISM2 staff engagement with DOH CO and regularized relations with regional CHDs, including basing LMAMs at CHDs. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report xi Recommendations 1. PRISM2 to continue and create opportunities for collaboration with and handover to the new USAID projects, including regional and provincial-level coordination. 2. To capitalize on achievement and identify shortfalls, USAID should encourage PRISM2 and the new, regional health projects to hold a series of best-practices/lessons-learned workshops focusing on those technical and geographic areas where PRISM2 may have learned the most and have the most to showcase. 3. Gender assessments and specialized tools and training modules developed in PRISM2, such as the usapan module on gender and gender-based violence (GBV) should be shared by USAID with staff specialists from the new, regional projects via a series of facilitator trainings. CONCLUDING STATEMENT PRISM2 was implemented in the context of a CPR that had stabilized, a maternal mortality rate that was rising, growing concern about reaching MDG targets, and increasing amounts of DOH financial resources. This project marked a departure from previous USAID/Philippines investments for FP/MCH, this being the first to promote PPP. The PRISM2 design built on a long, largely-successful history of efforts to engage the private sector. These efforts made particularly important contributions during periods when public sector attention to, and funding for, FP/RH waned. Though PRISM2 was boldly designed and courageously launched under a less-than-optimal public sector environment for advancing FP/MCH, seeking to forge partnerships between the public and private sectors was forward-thinking. That itself, and the vision further articulated by the first COP, have contributed to improving the landscape for advancing PPP. Though the number of SDNs established and strengthened remains small, if PRISM2 is able to make sufficient progress in the coming 19 months, a solid, sustainable foundation of continuing public-private collaboration could become a permanent and progressive feature for serving health care needs in the Philippines. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 1 1. INTRODUCTION This report begins with a review of the context around Family Planning (FP) and Mother and Child Health (MCH) in the Philippines. This is followed by a brief review of USAID/Philippines activities around the promotion of private sector involvement in addressing FP/MCH challenges through two phases of the Private Sector Mobilization for Family Health Phase (PRISM) Project. The report progresses into a discussion of the evaluation methods, limitations, and recommendations for future evaluations. This is followed by a thorough discussion of the findings, conclusions, and recommendations for each evaluation question. This focus on the specific questions provides opportunity for full reference to and citations of data collected during the evaluation. The presentation of answers to these questions is organized within three topics: PRISM2 Achievements (questions 1–4), Project Implementation and Management (questions 5–7), and Future Prospects for Private Public Partnership (questions 8– 10). The next section reviews progress made by PRISM2 within the three project components, and the final section looks to future prospects for public-private partnership by reviewing and revisiting the two SOW hypotheses in light of evaluation results. 2. COUNTRY CONTEXT 2.1 FP/MCH IN THE PHILIPPINES National level support for family planning in the Philippines has been inconsistent since the 1970s. The creation of the Commission on Population (POPCOM) for policy formulation and implementation of the national FP program signaled importance at the national level. Twenty years later, devolution of health policy application from the national to sub-national levels began. To increase responsibility and buy-in at the sub-national levels, the Local Government Code (Republic Act No. 7160) tasked the Local Government Units (LGUs) to deliver various basic services and manage facilities. This included maintenance and supervision of health facilities and personnel, while continuing to adhere to national health policies and standards established by the DOH. Unfortunately, few LGUs were prepared to accept a role in policy formulation or program implementation. Approximately 1,700 municipalities currently determine the disbursement of funds received from the local Internal Revenue Allotments for health programs, including FP. Because elected officials at the LGU and provincial levels serve three-year terms, local financing for FP is not predictable. The DOH has long recognized the challenge of widening coverage of health products and services for its citizens, particularly among the poor. DOH AO No. 2012-009 (“National Strategy Towards Reducing Unmet Need for Family Planning as a Means to Achieving MGDs on Maternal Health”) states that POPCOM “shall have a pivotal role of increasing demand for FP goods and services and providing an enabling environment for capacity building and advocacy.” The DOH also acknowledged that private sector contributions are crucial to increase the scope and scale of available health services. Also recognizing capacity gaps within the public sector, the DOH issued supportive policies such as the 2006 AO on Public-Private Collaboration in Delivery of Health Services for Women of Reproductive Age Group, the 2008 AO on Implementing Health Reforms for Rapid Reduction of Maternal and Neonatal Mortality, the 2011 MNCHN Manual of Operations, and the Responsible Parenthood and Reproductive Health Act of 2012, which specifically mandates private sector engagement. However, PhilHealth, financed through premiums paid by the LGUs on behalf of indigent families, is the primary source of financial risk protection for the poor. PhilHealth coverage of FP remains limited and inconsistently administered. Within the above constraints, the health outcomes for women in the Philippines are not progressing as planned to achieve the targets set in the Millennium Development Goals (MDGs). Data showed minimal improvement in service coverage between the 2008 National Demographic and Health Survey (NDHS) and the 2011 Family Health Survey (FHS): In both surveys, 78 percent of pregnant women reported having at least four antenatal care (ANC) visits, while those with first trimester visits improved slightly Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 2 from 54 percent to 58 percent. These same surveys show that deliveries by skilled birth attendants (SBA) increased from 62 percent to 73 percent. Comparing FHS data, CPR decreased slightly from 50.6 in 2006 to 48.9 in 2011, even as modern method use modestly increased from 35.9 percent to 36.9 percent. Such slow progress likely contributed to the maternal mortality ratio increasing from 162 per 100,000 live births in 2006 to an estimated 221 in 2011. It is not likely that the MDG target of 52 for the Maternal Mortality Ratio will be attained by 2015. More positive for MDGs are the indicators on infant and under-five mortality; 2011 data demonstrated a steady decrease when compared with previous years. 2.2 USAID/PHILIPPINES SUPPORT FOR THE PRIVATE SECTOR Since the 1990s, USAID/Philippines has included private sector involvement as part of its assistance to expand FP/MCH programs, including social marketing, workplace health programs, and provision of support to midwives and their birthing homes. The Social Marketing for Change project was a pioneer in supporting three oral contraceptive pill (OC) manufacturers to market and sell their products under a common brand name “Couples Choice”. The Mission-funded DKT/Philippines to market affordable contraceptives, specifically low-cost OCs and injectables. The Mission also expanded access to FP/MCH by supporting the establishment of high quality, affordable Friendly Care clinics and a network of peri￾urban midwife-owned Well Family Clinics. USAID/Philippines also funded initiatives to address policy and financial issues for private sector involvement through the Commercial Markets Strategies (CMS) program. 2.3 PRISM1 PROJECT In 2004, USAID/Philippines launched the five-year Private Sector Mobilization Project for Family Health Project (PRISM). The timing coincided with the Mission’s plans to cease donating contraceptives to the DOH, support contraceptive self-reliance, and develop the private sector as a major provider of FP services and commodities. The main objective of PRISM was to expand the provision of quality FP/MCH services by the private sector. This objective had three components: (a) establishing workplace family health programs, (b) developing the Philippine contraceptive market, with focus on OCs and injectables, and (c) promoting the business value of FP/MCH among private practicing midwives. During the term of the contract, PRISM:  Facilitated accreditation for 200 midwife-owned birthing homes,  Established 500 workplace-based family health programs reaching over 300,000 employees and developed models for workplace health programs to provide FP and MCH services,  Introduced seven new OCs and injectables along with one re-priced OC (through Organon), and  Made a new Intra-Uterine Device (IUD) brand and a new zinc tablet available in the Philippine market. Accompanying the above achievements, the PRISM End-Of-Project Evaluation provided some critical findings on the project’s operation and implementation, some of which include:  The interventions implemented under the workplace component did not greatly contribute to increasing FP and health services provision. Facilities for providing MCH services, such as a dedicated area for lactating women, were seen in only a few workplaces. Most interventions focused on information dissemination and counseling, which did, however, raise awareness.  Working with local pharmaceutical companies was more beneficial and sustainable than working with multi-national companies. The LGU market for contraceptive procurement was not maximized. There was limited project-wide effort to generate demand. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 3  The assistance to private midwives generated the most service delivery outcomes. These interventions should be scaled-up, with additional attention on accreditation and reimbursement; identifying funds for investing in clinic construction, renovation and major equipment; and improving the midwife business through other revenue streams.  Not all component activities were implemented in the entire project sites; activities would best be implemented together, as an integrated package.  A common issue across components was the need to address policy requirements to support all private sector initiatives.  There was little coordination with the public sector. For the program to fully succeed, closer coordination and collaboration with the public sector is required. PRISM set the groundwork for developing and implementing models for private sector service and product delivery. However, scaling up of these initiatives to achieve critical mass was a major challenge, as was obtaining commitment to public-private partnership (PPP) from the DOH, the Department of Labor and Employment (DOLE), and other national and local partners. These issues were included in the design of the follow-on PRISM2 Project. 3. PROJECT DESCRIPTION 3.1 PROJECT DESIGN As cited in the original contract with Chemonics in 2009, the purpose of PRISM2 was to assist the DOH, DOLE, and LGUs to engage and mobilize private sector resources in the delivery of FP and MCH services and products. PRISM2’s objectives are to (1) increase and sustain private sector provision of quality FP and MCH services and products, (2) increase utilization of quality FP and MCH services and products in and from the private sector, and (3) improve the policy environment for private sector provision of services and products. PRISM2 was to develop and strengthen local markets9 for FP and MCH services and products, and to strengthen policies and systems to encourage the public sector to engage the private sector. It was thought that if PRISM2 efforts were designed and implemented within the existing resources and priorities of the DOH, the private sector FP/MCH initiatives would be better accepted, implemented, and sustained. Within this framework, PRISM2 included various MCH interventions, such as prenatal care, promotion of breast feeding, and postnatal care. Despite efforts to probe several key informants on clandestine abortion as a contributor to high maternal mortality, the team found little recognition that PPMs in particular could play a critical role in helping alleviate worse consequences. 3.2 PRISM2 PROJECT IMPLEMENTATION Based on how key informants within USAID and PRISM2 described evolution within the project, PRISM2 may be summarized as having proceeded through the following phases: Inception: This initial period lasted through the first two years of PRISM2. During this period, the project sought to move from a vision of PPP to practical approaches for implementation. Through much of this time, the first Chief of Party (COP) of PRISM2, Mario Taguiwalo, imparted his unique and 9 In this report, as stated in the Evaluation SOW, a local market is an interplay of suppliers (or providers) and consumers (or clients) of services and products and the regulatory environment where the local market operates. It encompasses both public and private providers for a fully-integrated approach to expanding and improving FP and MCH. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 4 innovative vision of public-private stewardship to project staff and partners alike. Numerous inception and consultative workshops with CHDs and LGUs, with provincial health officers (PHOs) and city health officers (CHOs) sometimes participating, were conducted to promote public-sector stewardship for engaging the private sector. These efforts resulted in the creation of a series of eight Technical Initiatives for PPP (TIPPPs) to serve as blueprints for engaging the private sector in aspects of project implementation such as involving private midwives, working with private hospitals, and providing long￾acting and permanent contraceptive methods. Unfortunately, Mr. Taguiwalo became ill and passed away at the end of Year 2, as did one of his key staff, the senior area manager responsible for leading PRISM2 efforts in Mindanao. Re-Focus: With the loss of its COP and ensuing uncertainty about how to actualize the stewardship vision, the project seemed adrift as it approached Year 3. The Deputy COP (DCOP) had also resigned and Chemonics headquarters was not able to effectively address this situation until a D.C. staff member extended his short-term stay through six months, to March 2012. To reinvigorate staff, he organized an all staff meeting to develop the Year 3 Work Plan. Similarly seeking to put the project back on track, USAID/Philippines also intervened to reduce by half PRISM2’s substantive focus (from 8 to 4 TIPPPs) as well as geographic scope (from 77 to 36 LMAs). No corresponding adjustments were made in expected outcome achievements. Catch-Up: With approval of the amended Year 3 Work Plan and a new COP on board in May 2012, the project entered an ambitious period to implement activities and gain momentum for PPP. Behind on targets for many outputs, the project renewed efforts to implement related field activities. Reflecting new leadership, the management structure was also redesigned with technical staff reporting to the COP and operational staff to the DCOP. During Year 3, USAID also raised concerns about promoting and providing FP in work places, resulting in a project shift to informal work groups (IWGs). PRISM2 also issued a number of sub-grants to support various aspects of implementation. Other USAID/Philippines projects with which PRISM2 was to collaborate, such as Strengthening Local Governance for Health (HealthGov), Sustainable Health Improvements through Empowerment and Local Development (SHIELD), TB LINC, and Health Policy Development Project (HPDP), recently reached their ends-of-project. At present, with its own end-of-Project nineteen months away (in October 2014), PRISM2 faces the added challenge of many staff leaving to join one of several new USAID projects just starting up. Three of these are regional health projects, awarded to Research Triangle Institute (RTI), Johns Hopkins Program for International Education in Gynecology & Obstetrics (JHPIEGO), and EngenderHealth, to increase FP/MCH services in some of the same LGUs and with many of the same partners engaged by PRISM2. With nineteen months to go, PRISM2 will likely enter a final close-out phase perhaps a year from now. 4. EVALUATION FRAMEWORK 4.1 EVALUATION PURPOSE This is a final performance evaluation of the PRISM2 Project. The purpose is to review and assess this project and what it has accomplished to this point, nineteen months before it ends in October 2014. As requested by USAID/Philippines and encouraged by PRISM2 leadership, this evaluation covers implementation from project start-up in 2009 through March 2013. The evaluation team has considered all aspects of project implementation: from planning and initial implementation to midterm adjustments and ultimately progress towards achieving outcomes and contributing to targets. To do so, the evaluation team has developed and applied qualitative and quantitative instruments and used these to collect data. Ten evaluation questions from the evaluation SOW, together with four directives in the problem statement, provide the basic framework within which the team conducted this evaluation. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 5 Towards making improvements between now and project’s end in 2014, the Evaluation team suggests some adjustments for consideration by PRISM2 and USAID/Philippines. Beyond PRISM2, the team also suggests a number of additional and/or alternative approaches for the Mission to consider, possibly develop, and potentially invest in to promote further advances in FP/MCH in the Philippines. 4.2 EVALUATION METHODOLOGY The SOW for the PRISM2 Evaluation (Annex A) indicated three methodologies suggested for this evaluation: document review, key informant interviews, and focus groups. This section briefly describes how the team applied these three methodologies as well as some additional approaches to collect needed data, as time and field visits allowed. Additional information on the methodologies developed and applied during this evaluation is contained in Annex B. Project Document and Literature Review USAID/Philippines released to the Evaluation team a set of some 40 internal documents on PRISM2 (such as annual work plans, quarterly reports, and so on). Initial review of these documents gave the team a basic shared understanding of project operations and evolution. As the evaluation proceeded, the team added other relevant documents, often requested of and provided by USAID/Philippines and PRISM2. In addition, team members identified and reviewed other relevant documents from other sources within and outside of the Philippines. The team developed an annotated document database to facilitate sharing and retrieval of materials useful for deeper understanding of particular topics. In all, the team reviewed a much larger number of documents than that which was initially provided. A list of documents reviewed by the team prior to field work is provided in Annex F, and an annotated bibliography of additional selected key documents is in Annex G. Key Informant Interviews To collect additional information and insights on PRISM2, evaluation team members conducted in-depth, open-ended interviews with key informants. Based on initial suggestions from USAID/Philippines and PRISM2, the team compiled a list of potential key informants and their organizational affiliations. These identified individuals had relevant knowledge of PRISM2 operations and future options. As evaluation proceeded, new information needs emerged. The team consequently added other key informants to the list based on their particular expertise or experience with particular areas of the project. The final list of key informants (Annex J) contains only those actually interviewed. Focus Groups Opportunities for conducting focus groups proved limited. Though group interviews were often conducted when several key informants were interviewed at one time, on only one occasion was the team able to conduct a focus group session. This was with a group of PPMs who had been participating in an ADP training in Cagayan de Oro. On another occasion, at a usapan training in Pampanga, a group of participating PPMs gave written responses to a team-developed questionnaire10. Site Visits 10 Language requiring a mini-survey was included in the SOW, though, the Mission approved SI’s work plan which did not include a mini-survey. Members of the team discussed the limitation of time for a more rigorous approach for collection and analysis of quantitative data derived from a mini-survey. However, a mini-survey was developed for a group of midwives since it was not possible to hold a focus group within a training course on usapan in Pampanga. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 6 On several occasions, team members were able to observe activities being conducted by PRISM2 directly, including an usapan training in Pampanga, an alternative distribution point (ADP) training in CDO, and a service delivery network (SDN) launch in Bacolod. This provided a rich source of data on PRISM2 engagement with partners and enabled team members to interview additional key informants who were attending these events. The team also actively sought opportunities to visit private sector birthing homes, clinics, and hospitals which had already partnered with PRISM2 as well as some that had not. A Site Selection Matrix can be found in Annex C. Cost-Effectiveness The evaluation team was unable to analyze cost effectiveness as part of this evaluation due to two facts: PRISM2 had insufficient achievements to subject to such analysis, and cost data had not been collected by the project in a manner amenable to such finely-focused analysis. Both PRISM2 and Chemonics stated that disaggregating cost data already collected would not be possible and that PRISM2 had insufficient achievements to warrant such study. For instance, results from LA/PM training are too small to subject to such analysis. Similarly, usapan trainings have not been tracked to determine if numbers of new FP clients have resulted. If SDNs had been fully in place by this time, estimates of what these cost would be interesting. However, it would also be difficult to identify one quantifiable measure or outcome of a strengthened SDN upon which to focus cost-effective analysis. More details are provided in Annex I. Comparison Sites Prior to finalizing site selection, USAID asked the team to consider visiting sites that are no longer receiving support directly from PRISM2. As one USAID official expressed it, “we would be interested/like to look at some of the sites of the original 77 to see why they may have been underperforming.11” Given that PRISM2 intentionally retained those LMAs most likely to benefit from PRISM2 assistance, the team maintained focus on current implementation within PRISM2. However, to accord with USAID’s interest in non-USG sites, the team also identified an LMA in Mindinao that had been dropped, when the Project cut back to 36 LMAs, and made plans to visit the site. Unfortunately, a credible security alert received from USAID the day before the scheduled visit caused the team to cancel the site visit. Though information continued to be gathered during interviews with LMAMs about work in previous LMAs, it was not possible to re-schedule a visit to this or other prior sites. The team was, however, assiduous in visiting specific service-providing sites such as private birthing homes, clinics, hospitals and universities, along with a sari-sari store selling contraceptives within the BALANCED Project area. These sites varied from those having had no or minimal contact with PRISM2 to others, most notably some owned, managed or staffed by the PPMs, which were more fully engaged with PRISM2. Visits to these sites provided the team with ground-level perspectives on the context within which PRISM2 implementation was occurring, but not a basis for before-after comparisons. Variations among sites were too great (e.g., franchised networks versus stand-alone sites), levels of PRISM2 inputs too different (one-off training versus ongoing support towards accreditation), and PRISM2 inputs were too recent (i.e. from Year3 onward) to permit such comparative analysis. Though not strictly comparative for PRISM2 and non-PRISM2 sites, it is possible to offer some general observations based on these site visits. In general, the PPMs seemed impressively committed to serving clients. At least for the birthing homes visited, PRISM2 was not, nor did it aim to be, a visibly dominant force, such as had previously occurred with establishment of the Well Family franchised network. Thus while birthing home midwives would express appreciation for a PRISM2 training opportunity they may 11 The team talked to LMAMs that were part of the original 77 sites but did not continue when the scope was narrowed, and was unable to visit non-PRISM sites in Bukidnon after being advised not to proceed because of security concerns related to bomb attacks in the region during that period of field work. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 7 have had, in most instances they could not readily connect the PRISM2-supported training to subsequent service improvements or client increases. Similarly, neither of the two universities visited, one which had participated in a PRISM2 peer-educator training about a year ago and the other which was not offered such an opportunity, were implementing peer-education programs. Achievements within PRISM2 that could rise to the level of considering comparative analysis would be accredited birthing homes and/or strengthened SDNs. One could envision, perhaps by Project’s end, sufficient data and timelines to warrant a two- or four-celled comparative analysis. Such an analysis would depend however upon PRISM2’s internal data collection and potential to disaggregate data to measure results achieved by sites within the different cells, for example, accredited versus non￾accredited, and strengthened SDN versus non-strengthened SDN. Such a quasi-experimental design, though far beyond the SOW for this Evaluation, could be embedded within the designs of future projects. 4.3 EVALUATION DIRECTIVES AND QUESTIONS USAID/Philippines presented four directives to guide this evaluation. In addition, a list of ten evaluation questions informed the development of data collection tools. 1. Determine the effectiveness of the project, by investigating its achievement (in comparison with baselines and targets) of the following:  Outcomes – (a) increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources, (b) increased share of deliveries attended by skilled birth attendants, and (c) increased contraceptive market for pills, injectables, and LA/PM  Outputs – various in the areas of (a) service delivery expansion and utilization, (b) behavior change communication, (c) policy and systems strengthening, and (d) capacity strengthening of GPH national and regional health agencies (and related agencies), and local governments to engage the private sector in FP/MCH service delivery 2. Assess the effectiveness of PRISM2 design as well as implementation (management, operation, and monitoring systems), in relation to achievement of contract deliverables and relationships with key clients from the government and private sectors. 3. Identify PRISM2’s key contributions (e.g., unique achievements, innovative methods/approaches), and compare with other effective approaches of private sector involvement in FP and MCH. 4. Assess how PRISM2 collaborated with other projects in the USAID/Philippines Health portfolio, as well as how PRISM2 complemented the strategy and priorities of the United States Government (USG) and GPH DOH. Compare PRISM2 collaboration efforts with lessons learned in this aspect from PRISM. The directives of the evaluation were fulfilled by responding directly to a list of ten questions presented to the team in the evaluation SOW. 1. What is the project’s contribution to improvement in national FP and MCH indicators? Outcome indicators: a. Increase in CPR (modern methods from private sector sources) b. Increased share of deliveries attended by skilled birth attendants c. Increased contraceptive market for pills, injectables, and LA/PM 2. What are the key outputs and/or outcomes in the PRISM components, and which ones made the most contribution to project success? Three Project Components describe the PRISM2 Project objectives: Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 8 a. Increase and sustain private sector provision of FP/MCH services and products through the workplace, private midwives, and other appropriate health providers. b. Increase utilization of quality FP/MCH services and products in and from the private sector by providing health information to communities and marketing products and services. c. Improve the policy environment for the private sector by providing technical assistance to strengthen the government’s capacity for engaging the private sector in providing health services and products. 3. How effectively has PRISM2 strengthened the capacities to engage the private sector in FP and MCH service delivery? 4. How did PRISM2 specifically address the following? a. Development/strengthening of local markets for FP/ Maternal, Neonatal and Child Health and Nutrition (MNCHN) products and services; b. Responsiveness (inclusive of beneficiary targeting) to FP/MNCHN needs of government (DOH, DOLE, PhilHealth, CHDs, provincial/city/municipal health offices, provincial/municipal LGUs) as well as private sector partners and specific client groups. 5. What are the respective plans of the DOH and local governments in order to sustain the project-developed systems and interventions? Which initiatives are likely to continue and which ones will not be sustained once PRISM2 ends? 6. Describe briefly PRISM2’s management mechanisms (i.e., headquarters oversight and involvement, organizational structure, field level operational set-up, personnel complement and their skill set, and the short-term technical assistance), and operational and monitoring mechanisms. How appropriate and effective were they, and how did they influence project performance and client satisfaction? 7. How did PRISM2 complement (a) other USAID/Philippines Health projects, (b) USG’s “Best Practices at Scale in the Home, Community and Facilities (BEST): An Action Plan for Neonatal, Maternal , Child Health, Nutrition and Family Planning”, and (c) DOH’s MNCHN strategy and Universal Health Care agenda? 8. What PRISM2 interventions can be considered good practices (e.g., unique achievements, innovative methods/approaches), and which may be recommended for scaling-up in the future? How cost effective are these? 9. Are there more effective approaches with the private sector which could have been explored to achieve FP/MNCHN outcomes? 10. How have gender considerations been integrated in USAID’s PRISM2 project? What are the effects of the project on male and female beneficiaries? The SOW also asks the evaluation to revisit two developmental hypotheses during the course of the evaluation: 1. Public-private sector collaboration in health services provision, demand generation and policy formulation and enforcement will significantly contribute to FP and MCH outcomes such as contraceptive prevalence rate (CPR) for modern methods and skilled birth attendance (SBA) in USG-assisted areas 2. There are promising public-private partnerships (PPPs) from PRISM2 that contributed to improved FP and MCH outcomes which can be replicated or scaled up. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 9 Given the lengthy list of questions and directives to consider throughout the evaluation, the team presented a data collection instrument (Annex D) that aligned each of the ten questions with one of the four directives. This alignment provided a useful structure for the Executive Summary. In the following section, each of the ten SOW questions are answered in detail within the following three subject areas: PRISM2 Achievements (questions 1–4), Project Implementation and Management (questions 5–7), and Future Prospects for Private Public Partnership (questions 8– 10). 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 5.1 PRISM2 ACHIEVEMENTS QUESTION 1: What is the project’s contribution to improvement in the national FP and MCH indicators listed below?  Increase in CPR (modern methods from private sector sources)  Increased share of deliveries attended by skilled birth attendants  Increased contraceptive market for pills, injectables, and LA/PM Findings Based on prior FP/MCH success within the private sector, PRISM2 was accorded the following targets for measured outcome achievement (from the PRISM2 Contract SOW, p.13): a) Increased CPR for modern methods obtained from private sector sources from a baseline of 40.8 percent in 2009 to 48.8 percent in 2014 b) Increased share of deliveries attended by skilled birth attendants from a baseline of 62 percent in 2009 to 70 percent in 2014 c) Increased contraceptive market for pills by 2 percent annually d) Increased contraceptive market for injectables by 6 percent annually Towards these ambitious achievements, PRISM2 implemented the following: support for private midwives, marketing FP/MCH products, increasing access to LA/PM through private hospitals and clinics, and strengthening service delivery networks to meet comprehensive FP/MCH service needs. However, as a result of a long start-up period, implementation has been slow, producing relatively low numbers reported for related measures. To date, only twenty-three midwives have been accredited, with another 150 eligible for accreditation(‘accreditable’), that is, having already completed and submitted application requirements to PhilHealth and awaiting assessment and approval. For IUDs, eighty-two practitioners received training as have thirty for Bilateral Tubal Ligation (BTL). Only three SDNs have been strengthened (following the PMP definition). To increase CPR in the Philippines by just one percentage point per year would require an estimated 419,200 additional contraceptive users annually.12 The reported outputs above have not made a measurable contribution to CPR at the national level, and it is not reasonable to expect that they would, especially given changes in the project’s scope and pace of implementation. Midwives and other private providers that PRISM2 has directly supported are just too few in number to have had such an impact. One way to dramatically increase CPR is to introduce and mainstream additional contraceptive 12 Eight percent over five years is 1.6 percent per year. The Population Reference Bureau estimate for 2012 midyear female population is 26.2 million. Therefore, adding 1.6 percent per year to CPR requires an estimated 419,200 additional users per year. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 10 methods. Grantee Alphamed’s introduction of a one-month injectable is promising and, together with increasing use of other injectables, will positively contribute to CPR. Unfortunately, PRISM2 efforts to expand access to other methods, such as IUDs, tubal ligation, and vasectomy, have been negligible. While the cadre of new providers trained to insert IUDs and provide BTLs is a positive development, follow-up is needed to track how many procedures are subsequently performed. There could well be obstacles beyond PRISM2’s control, such as PhilHealth reimbursement levels or processing complications, that reduce results. Within the purview of PRISM2, it would be possible as part of quality assurance to follow up with trainees to collect data on the numbers as well as difficulties encountered in subsequently serving clients. Only Alphamed, and the contraceptives it distributed, have operated at sufficient scale to produce a measurable effect on CPR. Within its 15-month grant, Alphamed tallies 202,321 couple-years of protection (CYP)13 for all contraceptive products distributed (including information that cycle beads at 1.5 CYP yielded 61,020 CYP14). CYP, particularly based on product point of sales figures, is a poor proxy for contraceptive use: Not only does it measure products distributed, rather than those actually used, but also, CYP as an estimate of users within a given time period is fraught with conversion complications. For example, would these products eventually serve previous or new users? Alphamed also reports that fully 59 percent of products distributed were through LGU purchases for use within public sector facilities. Taking all of this into account, Alphamed’s CYP contribution would likely amount to far less than the hundreds of thousands of new users needed to impact private sector increases in CPR. Though contributions to national-level CPR are likely very small, PRISM2 and Alphamed have usefully opened the way for expanding access to contraceptives by establishing Alternate Distribution Points (ADPs), notably in the private sector and among midwives. Of 622 FP ADPs as reported by Alphamed (with slight definitional differences from PRISM2 ADPs), 271 are in private birthing homes. Missing from PRISM2, however, are previously promising interventions to improve FP/MCH access within formal workplace settings. Some workplace activities continued under PRISM2, though without support from the proposed subcontractor, Philippines Business for Social Progress (PBSP). A long delay in formalizing the PBSP subcontract agreement, followed by change of focus from the formal workplace to the informal workforce, was significant, particularly in terms of the possible missed opportunity in contributing to CPR. The shift was recommended by USAID Mission to ensure that potential ICV vulnerability between employers and employees will not happen. Fortuitously, the shift also addresses equity issues. 2008 NDHS reported that greater unmet needs can be found among those who are in the very poor and near poor economic quintiles, less educated and less access to information and services. Since the DOH is focusing on these population, shifting PBSP's focus to informal working group will also address unmet need in these population. In addition to involving midwives to expand contraceptive choice and access, PRISM2 support for PPMs makes a positive contribution towards increasing the numbers of births attended by SBAs. Key to this has been PhilHealth and its potential for attracting providers and clients to deliver in accredited facilities. To date, PRISM2 has been instrumental in encouraging and assisting fewer than two hundred midwives towards receiving such accreditation, and is reported to be assisting 300 more PPMs, out of an estimated one thousand who have had some project engagement, for example, by participating in a training (to put this in perspective, the president of IMAP estimated the total number of all midwives in the Philippines to be 40,000 to 45,000, of whom half are in 13 Based on sales/distribution by Alphamed and associated organizations: 1,101,960 for oral contraceptives, 174,148 3-month injectables, 74,3531 month injectables, 52,025 condoms, 40,680 beads, and 3,945 IUDs. 14 See MEASURE Evaluation Population and Reproductive Health Project website for conversion factors of contraceptive methods to CYP: http://www.cpc.unc.edu/measure/prh/rh_indicators/specific/fp/cyp. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 11 the public sector). However, PRISM2’s contribution to increasing the share of births attended by SBAs is unlikely to reach the outcome target. To increase SBA-attended deliveries by eight percent in five years would require adding approximately 40,000 SBA-assisted births per year.15 To achieve this, 200 midwives would each deliver 200 additional babies per year. While accreditation is a likely pathway to increase SBA deliveries, PRISM2 efforts to accredit midwives, as well as to help them attract more clients, did not get underway until well into Year 3. Additional outcomes specified for PRISM2 were to increase the market share of oral contraceptives by 2 percent annually and injectables by 6 percent annually. However, such national-level market share increases are both beyond what could be completely affected by the project and require a broader data set. To this end, PRISM2 routinely reports IMS data on countrywide contraceptive sales to USAID/Philippines. Thus, the Mission has up-to-date data on USG-supported commodities within the Philippines marketplace. However, except for products distributed by grantee Alphamed, PRISM2 contributes little to these direct sales. For its own product lines, Alphamed sales did significantly increase during the grant period: For oral contraceptives, by 665%, to over one million cycles; for 1- month injectables, by nearly 300%, to some 75,000 units; and for 3-month injectables, to nearly 175,000 units. Conclusions The team was asked to assess project contributions to increases in CPR, SBA-assisted deliveries, and market share for some contraceptive methods. Some expansion of method mix availability has occurred as a result of PRISM2, likely resulting in some increase within the private sector’s share of CPR. However, slow start-up in fielding activities, together with the relatively small numbers of providers reached, resulted in little contribution to regional or national levels of CPR. That a recent evaluation of the entire USAID MCH/FP portfolio could not show any statistical significance between 2006 and 2011 CPRs in provinces where USAID had invested is a cautionary note for positioning such high level outcomes to gauge achievements within a single project. The larger point is that PRISM2, particularly after its start-up, has not gone far enough in fielding activities or operating at a scale sufficient to have material impact on national or provincial FP/MCH statistics as measured by indicators such as SBA￾assisted deliveries; further, the comparative data will not be available until 2014. Contraceptive method availability has expanded as a result of PRISM2 support, notably through Alphamed marketing of oral and injectable contraceptives and some training for LA/PM. Again, however, this is not enough to impact national or regional indicators. Recommendations 1. Use appropriate measures and set realistic targets. To have had the impact on CPR set in the PRISM2 SOW, the project would need to serve well over one million new FP clients during its five years. When specifying numerical targets for high-level outcomes, it is important to relate these to the resources, scope, and time available. 2. PRISM2 to conduct follow-up to measure uptake and results of PPMs using usapan to increase clients served. 3. USAID to continue USAID investments to expand modern method availability through public as well as private sectors, particularly through the recently launched FP/MCH projects focused on Visayas, Mindanao, and Luzon. 15 The computation is as follows: 8% over 5 years = 1.6%/yr. Estimating the current population of the Philippines at 100 million, a crude birth rate estimated by PRB of 25/1000, yields an estimated 2.5 million births per year. 1.6% of 2.5 million is 40,000. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 12 4. PRISM2 to expand the contraceptive method mix and increase access to this more complete range by continuing to invest in sustained or sustainable institutions already in the private sector, such as IMAP, Alphamed, and established clinic franchises, like Mother Bless, Blue Star, and Well Family. QUESTION 2: What are the key outputs and/or outcomes in the PRISM components, and which ones made the most contributions to project success? Findings PRISM2’s components, predominantly referred to as objectives within the project documents, are to (1) increase and sustain private sector provision of quality FP and MCH services and products, (2) increase utilization of quality FP and MCH services and products in and from the private sector, and (3) improve the policy environment for private sector provision of services and products. The team conducted a thorough analysis of the PMP and quarterly reporting data for the relevant performance indicators that underpin each of these intermediate results. The results of this analysis are in Annex L. The PMP is fundamental to this, or any, project’s ability to track and measure progress. That the Project’s PMP was not approved until 3 years into the project, as noted in the RIG performance audit, was itself a hindrance to monitoring and measuring progress. In addition, weaknesses in the PRISM2 PMP constrain the project from fully capturing and reporting on indicators fundamental to documenting successes as well as shortcomings.16 The response to Question 3 below illustrates that PRISM2 recorded and reported data contains some deficiencies due to PMP weaknesses and weaknesses in how some measures of actual project achievement are collected and reported. In its early years, the project focused on cultivating stewardship among the CHDs. In the PRISM2 document “CHDs as Stewards of FP-MCH Outcomes” stewardship is loosely defined as the long-term institutional capacity of CHDs to exert a positive and substantial influence on local markets by increasingly asserting their region-wide leadership and direction to achieve better FP/MCH outcomes in their regions, such as increased CPR, SBA, and consumption of pills and injectables. A core feature embraced by PRISM2 is the establishment of a partnership between PRISM2 and CHDs that leaves a substantial and enduring effect on the CHD’s capacity to improve FP/MCH outcomes in their regions. Unfortunately, the PRISM2 PMP does not fully capture the short-term accomplishments that mark progress towards this long-term goal. Among PRISM2’s most critical indicators is “the number of service delivery networks strengthened.” The Performance Indicator Reference Sheet (PIRS) states that this indicator was selected to support a new initiative in line with the 2008 DOH MNCHN strategy, and therefore no established baseline data was available. The team was unable to determine how the annual targets were set. The 2013 fiscal year (FY) target at 20 percent (a 100 percent increase from FY 2012’s unmet target) is unlikely given the 0 percent cumulative accomplishment rate for Years 1 to 3 against the target of 10 percent. The 14th Quarterly Report shows a 15 percent year-to-date accomplishment rate against this target. Through discussions with PRISM2, the team learned that the definition of SDN is a significantly limiting factor in terms of the project being able to accurately demonstrate its accomplishments. SDNs are only considered strengthened when one provider, one facility, and six ADPs have been integrated into the “network.” The inclusion of six ADPs presents the most significant challenge due to the definition of this indicator, which requires purchases in at least two of the last four quarters, and is, reportedly, 16 See the Family Planning and Reproductive Heath Indicators Database for further guidance on indicators on topics including Repositioning of FP, LA/PM, FP/MCH, PAC, Adolescent and Youth Sexual and RH, Sexual and GBV, and Male Engagement in RH programs: http://www.cpc.unc.edu/measure/prh/rh_indicators. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 13 unrealistic. Consequently, the project has proposed that USAID use private distribution outlets instead of ADPs in the definition of SDN strengthening, which will better-reflect the project’s efforts to integrate private providers and FP/MCH supply outlets in the local SDNs. The team identified several overarching incongruities between the project’s intermediate results (IRs) and the indicators upon which progress towards these results is based:  Example 1: Sub IR 1.4 “Effectiveness of private providers’ FP-MCH service provision increased through training.” Three of the four indicators tracking project performance under Sub IR 1.4 are output indicators counting the number of individuals trained: the number of people trained in FP/RH, the number of people receiving trainer’s training, and the number of people trained in maternal/newborn health. Increased effectiveness in service provision could be achieved by providers’ completion of training. However, in order to determine private provider’s effectiveness, the project would need to monitor outcomes such as clients’ satisfaction with services provided pre- and post-training and long-term performance of services provided, such as whether patients experience any complications.  Example 2: Sub IR 2.1 “Health provider’s capabilities and practices in generating demand for FP/MCH among clients and potential clients improved,” is not adequately captured by the one indicator responsible for measuring progress against this result, “the number of USG-assisted service delivery points providing FP counseling services.” The unit of measure for this indicator is the number of SDPs, which are defined as participating private and DOH-retained hospitals, private birthing homes/lying-in clinics, campus-based clinics, IWGs and NGO clinics. The central feature of Sub IR 2.1 is demand generation: “Health providers’ capabilities and practices in generating demand for FP MCH among clients improved.” Capabilities and practices are not reflected in the existing indicator, which is therefore, not a reliable measure of Project performance against this result.  Example 3: Sub IR 3.3 “Public-private partnerships in FP-MCH through effective implementation of central DOH policies strengthened.” The indicator to track achievement of this result is “number of national policies supporting private sector provision of services and products developed.” However, the drafting of a policy and the effective implementation of a policy, as above in IR 3.3, are not one in the same. Also important to consider is decentralization; although DOH formulates new policies, the LGUs do what they think is good for their constituents. Implementation of policies issued at the national level is not automatic, particularly for controversial programs involving family planning. Conclusions As cited in Annex L, PRISM2’s indicators should provide the foundational reference point for understanding the project’s performance on outputs and outcomes with respect to achievement of success in each of the project components or IRs. The PMP of this project contains indicators that do not reliably or fully measure the project’s outcomes as well as targets that are over- or under￾ambitious. Because the PMP is a fundamental document to determine achievements, this undermines the ability of project managers, donors, and evaluators to identify the real gains and weaknesses of the project. The absence of baseline values, the inconsistencies between the indicator reporting format of the PMP and quarterly reports (where the former includes annual targets and actuals, and the latter presents cumulative accomplishment rates) limit the utility of the data and make it difficult to monitor trends in indicator performance. While the team found that a great deal of effort has gone into the development and revision of the PMP, as well as the tracking of indicators and the setting of appropriate targets, a clear articulation and discussion about the logical connections between project activities, expected outputs and outcomes, milestones and targets is lacking. Such an explanation of the overarching theory Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 14 behind project design and implementation would serve to clarify the largely implicit connections between various levels of the project. Recommendations 1. PRISM2 to retool project indicators to capture the breadth and depth of the project activities, including BCC-related and demand generation activities. For example, number of counseling visits is captured for usapan, but data on results in terms of subsequent FP acceptance was not similarly captured until the most recent quarterly report. 2. For projects with activities that need years to be accomplished, such as capacity building, a sufficient number and variety of indicators should be used to track yearly progress towards a long-term result. PRISM2 could have tracked progress against the three essential changes outlined in their stewardship paper referenced earlier in this section.17 3. Particularly for formulating indicators at project outset, USAID to ensure that indicators should be within the scope of a project’s manageable interests. This may require budgeting for project￾level M&E in survey design, survey execution and data analysis. For example, if CPR rate is to be tracked by the project, data should be obtained from the project level rather than from a population-based, national-level survey. A local survey firm with sound experience in data collection could be hired to collect this data either at the household-level among a representative sample of households or within the project’s target LMAs. LMA baseline assessments should be established for CPR within each LMA prior to measuring the project’s contribution to CPR with routine follow-up surveys on an annual basis. 4. Indicators that are excessively exceeded or missed, as reported in the quarterly or annual reports, should be recalibrated by PRISM2 (with USAID approval) and accompanied by a strong, comprehensive rationale. QUESTION 3: How effectively has PRISM2 strengthened the capacities to engage the private sector in FP & MCH service delivery? What factors and conditions significantly contributed to or hindered the achievements of outputs and outcomes in questions 2 & 3? The World Bank Institute defines capacity building (CB) as a locally driven process of learning by change agents that affects change in sociopolitical, policy-related, and organizational factors to enhance local ownership to carry out stated objectives.18 CB, a key component of PRISM2, was to result in individuals and groups having increased motivation and skills to engage the private sector to increase demand for and supply of FP/MCH services and supplies. CB is a complex, multi-year effort: CB in the health sector can be described and measured in terms of four levels: health system, organization, health personnel and community. Capacity at one level can be influenced by actions at other levels. Capacity development goes beyond a simple technical intervention, focusing on behavior change and organizations. Thus, capacity-building M&E must capture conditions and concepts such as motivation, culture and 17 The document “CHDs as Stewards of FP-MCH Outcomes” suggested three changes that are essential for all CHDs to complete to become effective stewards of public health outcomes in their regions: (1) CHD internalizes organizational commitment to improve region-wide public health outcomes, (2) CHDs build outcomes-based institutional partnership and alliance with PHOs and CHOs, and (3) CHD operations use public-private partnerships to attain public health objectives. 18 World Bank Institute. The Capacity Development Results Framework – A strategic and results-oriented approach to learning for capacity development. http://siteresources.worldbank.org/EXTCDRC/Resources/CDRF_Paper.pdf Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 15 commitment as well as resource availability, skill levels and management structure. 19 It was not possible for the team to quantify how effectively PRISM2 strengthened the capacities to engage the private sector in FP/MCH service delivery as PRISM2 CB efforts are not fully captured in the PMP. The reporting of PRISM2 CB activities as results, such as “trainings held” and “policies developed” does not fully capture the breadth of its CB activities which, in addition to enhancing skills and formulating policies, included processes such as building alliances, mobilizing and creating networks, and supporting the decentralization of planning and training. However, qualitative data collected from interviews (see Annex K) combined with observations of the team members, provide insights into some of the factors and conditions that influenced key outputs and outcomes in the PRISM components, which components made the most contributions to project success, and how effectively PRISM2 strengthened capacities to engage the private sector in FP and MCH service delivery. Among the people interviewed, there was widespread appreciation for PRISM2, broad acceptance for the public sector to work with the private sector to increase the supply of and demand for FP commodities and services, and there was interest from both PhilHealth representatives and private providers to have a health system that was efficient, equitable, and responsive. These data indicate positive short-term project outcomes. However, as indicated earlier, CB is to enhance local ownership to carry out stated objectives. As one PRISM2 grantee stated: Coming from the private sector we are not very exposed to public sector ways … PRISM is the one bridging this. They have a very big role in helping us move forward, giving us ideas, facilitating, helping us … they have a real and clear understanding of how government works and they are able to use this in helping the private sector deal with them. The concern voiced by some central and regional DOH staff about “being able to continue to promote and expand PPP when PRISM2 ends” reinforces the need to fully engage the DOH at all levels to take responsibility for developing, sustaining, and expanding the PPPs. The PHO and CHD often referenced the PRISM2 grantee as their point of contact with the project. Several sub-national DOH officials as well as LGU health officials referenced PRISM2’s participation in and/or convening of the intra-agency technical working group. One PHO commented that he was more likely to get service statistics from PPMs within the PRISM2 area than the PPM outside the Project – a statement confirmed by others. Conclusions PRISM2 is tracking and reporting only a few aspects of their CB efforts such as enhancing skills and expanding networks through numbers of practitioners accredited and policies formulated. The PMP does not capture core components of their work, such as raising awareness and improving teamwork among the private and public sector providers. It does not capture the quality of training events, appropriateness of methodology, learning objectives attained, or knowledge retained. Working within the established structures such as the CHD and the Inter-Local Health Zone (ILHZ) was the correct path, but, as reported by DOH key informants, developing tools and service packages outside the DOH limited buy-in (with one DOH informant stating she was waiting for the DOH version of the TIPPP), and the project’s implementation delays undermined building momentum as well as capacity. Efforts to establish and nurture the sense of DOH ownership and empowerment to promote PPPs are warranted. PRISM2 grantees are a vital component to reaching providers, but the numbers reached are low in total, and the number of trained providers reported do not distinguish between those trained in FP and MCH 19 A Guild to Monitoring and Evaluation of Capacity-Building Interventions in the Health Sector in Developing Countries. http://www.cpc.unc.edu/measure/publications/ms-03-07 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 16 and do not have targets by which they can track and report target attainment. The numbers of people trained (per the 14th Quarterly Report) are low for Year 4 accomplishments. There is little time and opportunity left within PRISM2 to conduct follow-up with trainees to further build capacities. With the exception of UHA (discussed in question 4), PRISM2-supported training appears focused on training individuals and not building a cadre of trainers, which would increase the number of private providers to provide FP/MCH services. Delays in awarding the subcontracts and sub-grants contributed to delays in reaching project targets and potential. PRISM2 has had an impact, albeit relatively late and not as robust as it could have been. Recommendations 1. USAID to ensure that the End of Project (EOP) Report should have detailed case studies on PPP activities, such as the ILHZ in Negros Occidental, the private sector accessing public-sector funds as with UHA, or managing relationships with local officials and stakeholders in order to accelerate PPPs in lagging LGUs 2. PRISM2 (with USAID approval) to modify the indicators to better capture PRISM2 capacity building efforts, perhaps by focusing on a key component such as SDNs, conceptualized and supported through a Logic Model, which would be useful to capture appropriate EOP outcomes based on viable inputs, activities, outputs, and outcomes. 3. PRISM2 to consider expanding the role of training grantees to include other elements for accreditation of midwives and facilities. QUESTION 4: How did PRISM2 specifically address the development/strengthening of local markets for FP/ Maternal, Neonatal and Child Health and Nutrition (MNCHN) products and services, and was it responsive to FP/MNCHN needs of government, sector partners, and specific client groups? Findings Service Delivery Networks (SDNs) As an outcome of the Year 3 Work Plan, PRISM2 tasked itself with strengthening SDNs as the FP/MCH service provision and referral system to ensure client access to primary and secondary basic emergency obstetric and newborn care (BEmONC) and tertiary comprehensive emergency obstetric and newborn care (CEmONC) facilities as needed. Not until late 2011 did PRISM2 discover that only a few SDNs were already in existence. PRISM2 used LMAs20 to help manage the task of identifying and strengthening SDNs as delineated areas of responsibility for the LMA manager (LMAM). SDNs are a key component of the DOH MNCHN strategy. PRISM2 integrated some private service providers operating in the same LMA into SDNs. Private health facilities, such as birthing homes of PPMs, clinics, and hospitals would be able to join the SDNs and become part of the referral system. All private providers within the SDN are to provide service statistics to the respective LGU health offices for inclusion in the Field Health Service Information System (FHSIS). However, much remains to be done: Until recently, the project had reached only 10 percent of its target SDNs. PRISM2 key informants attribute this shortfall to the lack of established SDNs at the onset of the project. Thus PRISM2 had to establish, rather than simply strengthen, SDNs by integrating private 20 Each LMA (which may include multiple SDNs) corresponds to an existing government tier at the sub-national level such as a province, city, or municipality. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 17 facilities and practitioners. PRISM2 did provide some technical support for drafting CHD Administrative Orders to guide the LGUs in engaging PPMs within FP/MNCHN programs. In some of the regions, many LGUs (20 to date) have issued corresponding resolutions in reference to the CHD guidelines. Accreditation of Private Practicing Midwives (PPMs) PRISM2 assists PPMs and their birthing homes to become accredited with PhilHealth to increase access for enrolled members to FP/MNCHN services. To facilitate accreditation, PRISM2, along with the DOH and PhilHealth, supports PPMs in the fulfillment of requirements, such as providing specialized training and ensuring that facilities meet requirements. As one example, a PRISM2 grantee provided 30,000 PhP ($700) per birthing home to obtain instruments and equipment needed for accreditation. PRISM2 issued three grants to directly support accreditation: Kilusang Manggawang Pilipino, Inc., Integrated Midwives Association of the Philippines, and Conrado L. Alcantara Foundation, Inc. So far, 57 percent of the PPMs targeted are accredited or accreditable and 65 percent of their birthing homes are Maternal Care Package (MCP) capable. As with other targets, denominators are based more on expectations than what is needed for impact. Another PRISM2 sub-grantee, the UHA Caregivers School in Cebu City, provides training for PPMs using qualified, certified private health providers as Family Planning-Community Based Training Level 1 (FP-CBT1) trainers alongside CHD/LGU trainers. This model ensures future training courses include the private sector and, upon completion of the training-of-trainers, the PPM trainers are certified by DOH as FP-CBT1 Certified Trainers. The recent batch of PPM trainers were from eight private-sector academic and midwife organizations representing Cebu, Bohol and Negros Oriental. Currently, 13 trainers have completed training requirements, and three are recommended to become certified trainers. For a project eighteen months from completion, this modest accomplishment is both humbling and something to build on. Though this illustrates public and private sector collaboration, there are some challenges. Foremost the CHD gives these private trainers considerable leeway to run FP-CBT1 trainings. However, since the CHD does not have a mechanism to directly transfer funds to a private institution, only the CHD can disburse funds during trainings. Mechanisms that would enable CHDs to fully outsource to the private sector would advance PPP. Another PRISM2 training activity is the usapan, a modified interpersonal communication approach to generate clients for FP and provide a platform for FP counseling and interaction. At present, usapan trainings are being conducted in all PRISM2 LMAs. It is too soon to have results or impacts of usapan trainings in terms of informing, motivating, or serving clients. Alternative Distribution Points (ADP) ADPs are venues from which products can be distributed. In PRISM2, ADPs can include birthing homes, company clinics, cooperatives, mobile pharmacies, NGOs, HMO clinics, RHU pharmacies, health providers-entrepreneurs, community-based hospitals, ILHZ or cooperative operated Botika ng Barangay, and social franchise outlets. ADPs are not new, but PRISM2 coined the term and expanded the number of FP/MNCHN products. With an ADP, a PPM can also generate additional revenue by dispensing commodities, such as contraceptive pills, condoms, and zinc formulations. The Project has several sub-grantees to scale up current ADPs managed by PPMs or to identify potential ADPs. To enhance the entrepreneurial skills of the PPMs, Alphamed used its grant to conduct ADP trainings for PRISM2-supported PPMs. The Project has reached 80 percent of its target to introduce new FP/MNCHN products such as oral rehydration salts, zinc tablets, and injectables. With 31 percent of the ADPs distributing FP commodities in USG-assisted areas, there remains room to expand the contraceptive market. Alphamed, whose grant has ended, and other pharmaceutical distributors participated in the recent ADP training in Cagayan de Oro. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 18 Informal Work Groups (IWGs) In the original PRISM Project, the Philippine Business for Social Progress (PBSP) worked with formal workplaces of medium to large companies and corporations. In PRISM2, based on concerns raised by USAID/Manila about promoting FP in work places, PBSP shifted focus to IWGs such as vendor associations, labor unions, and transport groups to generate demand for FP/MCH services, become ADPs, and/or be part of public-private referral networks. The PBSP area coordinators reported difficulties engaging the IWGs and having them commit to an FP/MCH program. Although aware of the importance of such programs, in one illustrative example, the IWGs were reported by one key informant working on this at ground level to be more interested in livelihood training. Some IWGs expressed concern about the time and personnel commitment required of the core team. One particular issue that was raised was the 4 four-day live-in usapan training. To address this concern, PRISM2 recently agreed that the 4-day usapan training design may be split into two consecutive weekends. A number of IWGs have limited capacity to conduct usapan, and particularly to link this to the provision of services. IWGs will need to establish partnerships with the CHOs and other LGU units willing to provide assistance by assigning public and/or private midwives as health service providers during the usapan sessions. Some IWGs who are service providers, such as hospital-based cooperatives, also agreed to adopt the IWGs near their hospital to handle the usapan series and make their hospital their referral center. The hospital-based cooperatives saw an opportunity to increase the clientele for their hospital by handling the demand generation activities of the IWGs near their hospital. Conclusions The main thrust of PRISM2 is towards stewardship for the public sector to engage the private sector. PRISM2 was able to break the gap between the private and public health providers through its approach and efforts in the SDNs. The Project made it possible to identify more private sector health providers and institutions to provide FP/MNCHN services in the different LMAs. It furthered its support through efforts towards accreditation of PPMs and establishment of ADPs. On paper, the design is good: The private sector gets integrated into the SDNs within functional referral and reporting systems. However, many LMAs still do not have functional or operational SDNs into which the private sector can integrate. This systemic shortcoming may not be manageable within the limited time remaining for PRISM2. With functional SDNs not yet in place into which private practitioners may integrate, momentum can erode and private sector participants may lose interest. Though PRISM2 has provided some support to CHDs to develop regional administrative orders and guidelines to engage the PPMs and other sub-national policies to support ADPs, for sustainability it needs to develop deeper capacities within the public sector to continue working with the private sector. Recommendations 1. PRISM2 should live up to its vision of fostering stewardship. Though PRISM2 looks to the public sector to engage the private, it is as if the first two years of intensive consultation did not happen. Rather than spreading resources to meet targets within thirty-six LMAs, USAID and PRISM2 should consider further concentrating resources within SDNs, where traction has already been gained and proof of concept is possible. 2. One of PRISM2’s most innovative approaches was with the UHA training approach that enabled private sector and public sector trainers to work together. However, the grant encompassed only one training (Family Planning-Community Based Training Level 1 [FP-CBT1]). Based on this positive experience, the UHA grant should be scaled up to include the other three courses PPMs need for accreditation. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 19 3. PRISM2, through PBSP, should continue to be flexible in its engagement with IWGs. Usapan may not be the only approach to apply; empower the PBSB coordinators to identify, devise, and try approaches appropriate for this unique group. 4. Given the key role of PhilHealth, USAID should consider how to continue and/or provide needed additional support (e.g., for costing studies, to facilitate interactions with DOH, etc,) within existing or new projects to revise policies and practices for accreditation and reimbursement. 5.2 PROJECT IMPLEMENTATION AND MANAGEMENT21 QUESTION 6: Describe briefly, PRISM2’s management, operational, and monitoring mechanisms. How appropriate and effective were they? How did they influence project performance and client satisfaction? Leadership is critical within any project. Unfortunately, leadership loss occurred just as PRISM2 was prepared and poised for full implementation. The tragic illnesses and deaths of both the founding COP and the Senior Area Manager (SAM) for Mindanao led to disrupted operations, further delays in implementation, and changed management. The subsequent departure of the Deputy COP deepened the void, as did delays in replacing these and other key positions, such as the private midwife specialist position that was unfilled for more than two years. Compounding the impact has been a relatively high staff turnover, whether from non-renewed contracts or staff departures. The latter recently accelerated due to several staff members taking new positions within USAID’s recently-awarded policy and regional health projects. PRISM2 deployed mid-level health professionals, or LMA managers (LMAMs) to Local Market Areas (LMA), who are each supervised by a SAM based in one of three “regional” offices, in Mindanao, Visayas, and Luzon. LMAM’s are based in proximity to the one or two LMAs they support. While regional offices are maintained in Davao and Cebu, for Mindanao and Visayas, the regional office for Luzon province is housed within PRISM2 in Manila. These staff and this structure serves the country’s 14 regions. As shown in the PRISM2 organogram, each SAM reports to the Manila-based DCOP. Manila-based technical staff members, who provide technical input and oversight for field activities, report to the COP. Throughout key informant interviews, field staff reported that the management structure is often challenging, particularly for negotiating priorities and accessing technical resources. Extracts from LMAM and other field-level key informants highlight such difficulties: TIPPPs were like straightjackets – needed flexibility in the field – regional staff became clerks and the (technical) specialists needed arbitrators. A venue for active cross communication of operations and technical [would be useful]… [Management from Manila] is more top-down. They think that theoretical things can be applied a-to-z in the field. But (every region) is different. Some things need to be adjusted. The relationship between LMAMs, the support is there, whenever we need assistance without having to ask our technical specialists. As other key informants observed, Luzon’s unique Manila base affords both advantages and disadvantages that highlight some of these difficulties. On the one hand, they have better access to technical specialists; on the other they have more project oversight and less autonomy than the other regions. 21 To facilitate logical flow, Question 5 is purposefully presented out of sequential order. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 20 At the field level, operational structures and systems are in place and functioning reasonably well. However, especially with the elimination of regionally-based M&E and communications positions and the loss of field staff to new projects, stress is high and morale low. As one key informant stated, “(We are) spread too thin …with loss of communications and M&E positions … At this point [in the Project], these are the positions they should have retained.” Related to this, little effort is made to learn from LMAM experiences and share lessons across LMAMs and regions. Though the three SAMs and project leaders now convene quarterly, this may not bring about full learning from and across staff. The team’s review of the PRISM2 organogram with the DCOP identified a dozen physicians in the central office with DOH backgrounds. This seems disproportionate to the breadth of technical skills needed to implement a project which also encompasses the private sector. The oversight provided by Chemonics headquarters appears straightforward; financial oversight is a principal responsibility. The Chemonics-based Project Director provides input on the annual Work Plan, reviews quarterly reports, and occasionally, with Mission pre-approval, visits the project. Chemonics does not seem well or deeply connected with the field of FP/MCH; management from a distance, and PRISM2 being a contract rather than a cooperative agreement, may cause dissonance in relation to the pace, priorities, and practicalities of implementing the innovations and creativity needed for this project to fully thrive and meet its potential. The Office of Health (OH) of USAID/Philippines also exercised management control. At critical periods, when PRISM2 appeared to be falling short in reaching expected goals, OH staff brought about significant positive change, such as the reduction in target LMAs and the focus on four TIPPPs instead of eight. Recently, however, the Mission may have over-reached its role in operations: The elimination of the field-based M&E and Communications positions and the new organizational structure, have exacerbated cross-regional and field-technical communication challenges. As one Manila staff member suggested, “personally I thought the M&E people at the regional offices would have stayed until the end of the project, it’s too late to clean data when it reaches the central level.” Some historical perspective is also provided later in the same interview: “The reason we had twenty-eight sites at the start of the contract is because that’s where the other projects were working as well and USAID wanted continuity, PRISM was given room to select additional sites, so we increased to seventy-seven.” Project monitoring mechanisms are in place. However, the team found it challenging to track accomplishments over time (see question 2 on outputs and outcomes). This is due in part to the delay in establishing a PMP, having various versions of the tool, retaining the original project targets after reducing the number of LMAs, and shifting away from key activities such as the workplace. In addition, some definitions of deliverables hindered understanding and tracking what was actually accomplished, for example, “midwives touched,” “hospitals engaged,” and “MOA [Memorandum of Agreement] signed.” Conclusions Leadership loss early on, combined with long delays in finding permanent replacements for key positions, contributed significantly to the long, slow start-up during the project’s first three years. Project implementation has been too slow, with the strong likelihood that in the end it will have counted thirty￾six SDNs as strengthened (100 percent of its target) with most barely meeting PRISM2’s own minimal criteria; engaged PPMs significantly and meaningfully so as to improve some particular practices, but for perhaps only some 300 PPMs out of a universe of thousands. While LMAMs, with guidance from the SAMs, were building the necessary relationship at the field level during the early years, technical direction, primarily in the form of TIPPPs, was slow in coming. Based on feedback from key informant interviews, the field staff members were actively discouraged from adopting the technical packages in response to field level realities. PRISM2 appears to have become increasingly centralized, imparting directives on how and what to do. As the project enters its last year, emphasis seems to be on ensuring that targets are attained. Inability Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 21 or unwillingness to listen to and learn from the field led to considerable staff disaffection and turnover. The USAID-initiated restart and revision of the Year 3 Work Plan was needed, but was not sufficient to overcome delays in outcome attainment or shift the project to supporting field efforts. According to USAID, working with youth and mainstreaming gender are USAID priorities which could have been integrated in all TIPPPs from the beginning of the Project. Recognizing this, additional specialists were added to work on these without delaying the implementation of four TIPPPs in the field; however, evidence still supports that youth, and gender mainstreaming, as well as Uniject oxytocin, are not priorities for a project focused on promoting PPP. These elements allowed the project to further drift from re-focused attention on the thirty-six LMAs and four technical areas. Finally, the sudden shift from work-based services caused some further loss of potential for achievements in services provided and partnerships strengthened. Recommendations 1. When forwarding a bold, creative initiative such as PRISM2, USAID must carefully consider if a contract is the appropriate mechanism for a project in which adaptability to field conditions is required. 2. A directive approach to management is inimical to the partnership that PRISM2 is trying to foster and develop. In the time remaining, it would be useful for PRISM2 to mine the learning that is occurring at field level and share lessons learned within as well as outside the project. 3. PRISM2 to reinvigorate staff morale, renew energy, and fully prepare for a final push towards project completion. Repeating the well-facilitated Year 3 all-staff meeting could be helpful. 4. PRISM2 to continue fostering relationships with local counterparts and hand-off, perhaps during regional consultations, promising developments to the new USAID regional projects. QUESTION 7: How did PRISM2 complement other USAID/Philippines Health projects, the USG “Best Practices at Scale in the Home, Community and Facilities (BEST): An Action Plan for N/MCH/FP”, and DOH’s MNCHN strategy and Universal Health Care agenda? PRISM2 is one of several USAID projects within Strategic Objective 3 (SO3): “Improved Family Health Sustainably Achieved.” PRISM2 was awarded in 2009, which was mid-cycle with the other SO3 projects, including the Health Policy Development Project (HPDP), Strengthening Local Governance for Health (HealthGov), Sustainable Health Improvements through Empowerment and Local Development (SHIELD), and the Health Promotion Project (HealthPro). Unique within the USAID portfolio, PRISM2 has the special feature of engaging and mobilizing private sector resources for health, through and with government stewardship, to increase FP/MCH products and services. Considering that other USAID projects were directed towards enhancing the capacity of the public health sector, PRISM2 had or should have had working partnerships with HPDP at the national level, as well as with HealthGov and SHIELD at the local level. Particularly in the waning years of these other USAID projects, there was minimal collaboration. USAID established coordination mechanisms such as an Inter-Cooperating Agency (CA) Meeting among COPs/DCOPs and Technical Working Groups, composed of key project staff. However, the same coordination structure was not apparent at the operational level. Projects often discuss common issues but meet separately with the same CHD or LGU personnel. Many DOH stakeholders interviewed clearly observed this non-collaboration among projects under the USAID umbrella. An example is the introduction of varying forecasting tools to LGUs by USAID CAs, thus generating confusion among local partners. As one project COP mentioned, it is not that they went out to the field because they heard about PRISM2’s work. Rather, it was USAID that ordered them to coordinate as they were both Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 22 working in the same area. A more positive and productive example of coordination was the use within usapan by PRISM2 of a tool developed by HealthPro. The intent of PRISM2 complements the FP/MCH/N Action Plan of the Philippines. The Project was designed to expand access to integrated services and strengthen the capacity of LGUs and the private sector to plan, implement, and monitor those services. The IRs of the action BEST strategy (to improve supply of services, to strengthen demand for services, and to remove policy and system barriers) are parallel to the mandate of PRISM2. The project is working with PhilHealth to support the Universal Health Care Initiative and facilitate the accreditation of private midwives and private facilities to enable services rendered by the private sector to be reimbursed by PhilHealth. However, the small number of private providers and facilities touched by the project seems out of proportion to a $35 million budget of a project that is in the fourth year of a 5-year contract. Likewise, the scale of influence that the project has had on PhilHealth in affecting changes in the level and type of coverage of FP/MCH services is negligible. PRISM2 is undertaking an “informal” study of the actual costs for BTL by asking some private facilities about actual costs incurred. The project intends to provide PhilHealth with the study results in hopes of persuading them to increase reimbursement from the current 4,000 PhP ($93) to one that covers the cost of the procedure, which is closer to 6,000 PhP ($140). This type of effort of PRISM2 to influence policy should be expanded; private providers reported to the team that the cost of the newborn kits exceeds the reimbursement ceiling of PhilHealth. A grant or even a new project for a group experienced in incorporating FP/MCH into national insurance reimbursement packages would be beneficial, which may involve small, but well￾designed studies on relevant topics such as cost, comprehension, and implementation of PhilHealth policies and procedures. Likewise, a national- or provincial-level effort to oversee accreditation and uniform compliance of PhilHealth policies at the provincial level or already established national associations such as the Commission on Higher Education (CHED) or the Philippine Ob/Gyn Society may have been a better use of resources than helping individual midwives or clinics. PRISM2 is consistent and compatible with BEST. It supports key FP interventions and aims to strengthen the capacity of local government units and the private sector to plan, carry out, and monitor the availability and quality of integrated FP/MCH/N products and services. The project strove to enhance the FP/MCH communication skills of the PPM, included other MCH and nutrition products such as zinc supplements and oral rehydration solution among products distributed through ADPs, and attempted to strengthen the health management information system (HMIS) by supporting inclusion of data from the private sector. The team noted that technical support is needed at the national, subnational and the local level to improve the HMIS; information gathered by team members during site visits to various PRISM2 LMAs showed that improvements in data from the private sector are largely still not in place. Systemic change will be needed, such as data reporting being part of accreditation, before there is likely to be large scale improvement. Under BEST, the engagement of the private sector specifies expanding access to FP/MCH among working men and women. The gains from prior collaboration with DOLE offered a good opportunity for increasing FP/MCH within Family Welfare Clinics. However, USAID’s shift from workplace to work force put that on hold. Collaboration with other development partners, as stipulated in BEST, was not explored as an opportunity to continue momentum in work places. The key components of the PRISM project are aligned with the government’s Universal Health Care (UHC) Initiatives and the MNCHN Strategy. Albeit on a small scale, the project is: improving financial risk protection by assisting PPMs and private facilities to become PhilHealth accredited; enhancing health facilities to improve access to quality health by assisting midwives upgrade their facilities; and attaining the health-related MDGs by engaging the private sector to include FP/MCH services and products. However, PRISM2, through its participation in the USAID CA meetings and in other in-country venues, Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 23 could have advocated for a broader range and realistic pricing of FP/MCH services to help address the reluctance of the private sector to engage with PhilHealth. The project is working to ensure that PhilHealth reimbursement and accreditation policies at the LGU are understood, while also providing support to the MNCHN Strategy by helping to establish, when needed, and strengthen SDNs. For its part, PRISM2 leadership expressed that slow progress to accord with the MNCHN AO of setting up SDNs among public sector facilities constrained PRISM2 efforts to encompass private sector facilities within strengthened SDNs. Though the project mobilized private providers to engage within the SDNs, there are indications and field feedback that SDN launchings appear to be “ribbon cutting” activities more than a reflection of partnerships among service providers and local chief executives. … [regarding] creation of the SDN, the Manual of Operations says we should do this and that – then were told just to follow the 1-1-6. Difficult to create that SDN because we need to follow steps, need to map it out, identify hospitals, etc. but they just laughed at us. Why were we making it hard for ourselves? We just followed the MNCHN – we are creating a network. 1-1-6 does not include a CEmONC necessarily. BEmONC capable facilities; BEmONC capable facilities. Why did we have to get all these hospitals when just needed 1-1-6. What is behind that directive is just to check the box and reach the target of 12 by September. For me, I targeted one SDN to be launched in two LGUs. They already had the mapping, and those written in the Manual of Operations. Why would we launch only one hospital – one facility cannot compose a network. At present, PRISM2 has only strengthened three SDNs that fully met its own criteria; it hopes to have thirty-six SDNs by the project’s end. Conclusions PRISM2 has a pivotal role in providing significant inputs to mobilize the private sector. This complements the other USAID health projects, which were mainly focused on the public sector. However, there are no visible efforts of learning and sharing significant breakthroughs among projects. The establishment of Inter-CA Coordination Meetings did not permeate or create functioning working partnerships at the field operational level to facilitate either broad understanding or well-coordinated processes for implementing innovative strategies among CHD staff and local stakeholders. Opportunities for synergy were missed. For example HPDP is engaging the CHDs and LGUs to develop memoranda and policies to engage private health providers. Similarly, PRISM2’s stewardship efforts could have been supported by working alongside HealthGov, which was tasked with supporting local governance, or the usapan effort could have generated more local buy-in had there been closer collaboration with HealthPro. Regarding the BEST Action Plan for Neonatal, Maternal, Child Health, Nutrition and Family Planning, there are relatively simple matters and some significant findings that should be considered such as (1) the delayed development and installation of the enhanced HMIS that incorporates private sector data, (2) consideration of what should be done in the aftermath of dropping the viable strategy of improving FP/MCH services within the workplace, and (3) using the ADPs to maximize distribution of MCH commodities, such as DIAZINK. The assistance in accrediting private birthing clinics is appreciated, but there are critical policy and financing support that warrant review. The current thrust to fast-track establishing SDNs as a major project output is countered by key informants’ reporting that the process is hurriedly done, forsaking some core principles and steps needed to ensure enduring partnerships for FP/MCH. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 24 Recommendations 1. USAID to promote and support a productive coordinating structure among USAID-funded Cooperating Agencies at sub-national levels to ensure participation in the planning and decision making needed to develop and maintain PPPs. 2. USAID to provide and cultivate partnerships with other donor resources to promote and expand work-based efforts to increase CPR and reduce the maternal mortality ratio (MMR). 3. PRISM2 and USAID to support the pilot testing of the general guidelines in the MNCHN Manual of Operations about establishing SDNs; foster and nurture this effort to establish a functioning referral system. QUESTION 5: What are the respective plans of the DOH and local governments in order to sustain the project-developed systems and interventions? Which initiatives are likely to continue and which ones will not be sustained once PRISM2 ends? The spirit of stewardships that PRISM2 espouses is not evident at the national level. The DOH Central Office staff members, particularly in the Family Health Office, were initially consulted in the development of the technical packages on the usapan sessions and the usapan barkadahan, which is a specific peer education approach for the adolescent youth. However, the participation of DOH was no longer evident in the ensuing finalization and implementation processes of the said packages. Central Office has no knowledge of the PRISM initiative with CHD Central Visayas regarding building the capacity of UHA as a local institutional provider for Family Planning Competency Based Training. In interviews, the team learned that the limited engagement of DOH staff in PRISM2 activities hindered their interest to accept, adopt, or sustain PRISM2 activities or products. The Year 1 Work Plan detailed how PRISM2 would work with their government counterparts to develop the packages, but it appears that the effort to coordinate planned activities was not fully pursued in all regions, and less so at the national level. The DOH CO staff interviewed told the evaluators they were unaware of the status and success of PRISM2 and had no plans of sustaining the project initiatives. While responses to questions posed in a key informant interview cannot be taken as an official stance, this perspective was also echoed at the regional level. In response to a question on the sustainability of PRISM2 activities, one regional DOH official stated: Upon crafting our province wide investment plan for health – the PRISM people attended and they charged some activities in the PIPH… For the procurement of commodities – it’s all stated in our PIPH but the release is not very high…. PRISM’s involvement in policy making – most of them inputs tie up with the regional office to craft policies for instance, the reporting system, the led our regional director to issue an MOU regarding that, upon issuance of the MOU they also introduced at the provincial level – the activities – so the policies are being crafted at the regional level before they are cascaded down. Here at the provincial level we are not prepared to take on the PRISM activities – no sustainability plan. At the sub-national level, some PRISM2 staff members interact routinely with regional counterparts. One CHD Director provided office space to staff of the different cooperating projects. However, some LMAMs who had joined PRISM2 from other projects (such as HealthGov and UNFPA) reported that it was their personal relationships with colleagues in the public sector that facilitated their work rather than the reputation of PRISM2. A Regional Inter-agency Coordinating Team was established by one CHD to discuss program activities with other partners, including DOH-retained hospitals. PRISM2 used this venue to provide information and updates but was, at times, seen by the CHD as the meeting organizer rather than a catalyst for PPPs. However, most regional DOH staff interviewed perceived PRISM2 as the main technical and resource support for private sector participation in health development. CHD participation was often limited to including private midwives in DOH-organized Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 25 FP/MCH training activities or participating in PRISM2 trainings such as the Quality Assurance Training for midwives. Although the project has not been successful in transferring ownership of some of its activities, as indicated in previous questions about sustainability of PRISM2 efforts, it did promote a sense of ownership for the concept and design of PRISM2 at some of the local levels visited. One health officer admitted that she was reluctant to work with the private providers during the first PRISM Project because project staff had directly approached PPM without coordinating with her office. Under that project, she felt that the PPM and the public providers could have their respective activities and that these two were likely to compete with the public sector. This perspective changed when PRISM2 facilitated partnership building in which the PHO led the process and explained the contribution of each sector for better public health outcomes. This same health officer reported that there were regular partnership meetings, commodity support to private health facilities, integration of private data to the FSHIS, and perceived improved health outcomes. She also said that given this new understanding and appreciation of private sector contributions, she will continue working with private providers, even when PRISM2 ends. The relationships between the LMAM and sub-national staff are delicate; most CHDs interviewed were appreciative of private sector contributions but were uncertain how to actualize public-private partnerships. At the provincial level, one LMAM reported that when we did not have a COP we got lost. … [our] SAM taught us to do what we can – to do these things on our own and with the help of the public sector. When most of us did that, the people in Manila called us “autonomous.” …The SAM had been from the field. For the technical initiatives, we know that Year 3 would be scale up. Per technical initiative, he assigned different TIPPPs to implement and then learn from each other. Some TIPPPs they did not provide “how to”; we tried to adapt. But they wanted us to replicate. As in question 3 about building capacity, awareness, and buy-in, it is critical to engage the CHD at all phases on the project. In an interview with CHD FP Coordinators, the team was told that even if there were an existing DOH policy to include private providers in their training program, there is not a conscious effort to include them in their 2013 and 2014 DOH-CHD training plans; available funds were sufficient only for public health workers. When queried if the CHD would ever provide support to PPM, one FP Coordinator replied that “it would be difficult” because she would not know the midwives who would need training nor how to organize their participation. She added that “the PRISM2 staffs are doing these functions and CHDs are already burdened in coordinating the LGUs and that approval for public health providers to participate in any trainings or events is difficult, given the devolved set up.” As mentioned earlier in this report, the need to revisit stewardship of the project is important, as most CHDs interviewed were appreciative of private sector contributions but uncertain how to actualize PPP. There are potential and increasing resources for FP/MNCHN programs at the DOH, national, and sub￾national levels. Efforts and funding can be harnessed to support PPP efforts. In the context of promoting stewardship and capacity building of the public sector, the project overlooked the needs of the central￾level DOH to efficiently and effectively manage and support the health service delivery of local level partners. Given the current government thrust to rationalize its systems and structures, the role and readiness of both the public and the private sectors to establish mutually-beneficial working partnerships should be in place. The DOH has numerous procedural requirements to outsource and issue contracts to private entities, such as for training and technical assistance. As a result, it contracts other public sector institutions which are often overburdened or may lack appropriate technical expertise. Conclusions As a result of PRISM2, more private sector providers and institutions are providing FP/MNCHN services and are linked with public health services. The project changed the perception and attitudes among Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 26 some local health authorities towards relating to and working with private groups; the public sector found advantages in engaging the private sector in improving public health outcomes. With nineteen months remaining, PRISM2 staff turnover is untimely and could hinder the project’s momentum gained in the partnership-building process. Sustainability of PRISM2 efforts would be more likely if the DOH felt ownership of PPP products and activities. Indications of ownership could include well-defined characteristics of organizational or individual commitment, documentation of a functioning private-public referral system, or percentage of local budget to train or support accreditation of providers. A study or pilot LGU to test and document the indicators, design, and level and type of technical assistance might have been useful. Likewise, seconding or insourcing a staff person to DOH CO or CHDs could have provided consistent mentorship in working with the private sector. Some PRISM2 staff members were discouraged from modifying directives or adapting tools to reflect sub-national variances and challenges. It appears PRISM2 did not recognize or encourage local initiatives that could have facilitated the stewardship process at the sub-national level. As discussed in question 3, there are existing gaps in the government’s ability to contract out services to the private sector given its available resources. PRISM2 has been unable to respond and build the needed capabilities for both sectors to establish a functional mechanism of technical and management cooperation among private and public sector parties. Recommendations 1. PRISM2 to revisit, reinvigorate, and empower PRISM2 staff to support capacity-building and stewardship activities with national and sub-national DOH staff. 2. PRISM2 to work alongside the DOH CO and CHD to ensure actualization of support to private providers by including specific activities in their annual work and financial plans. 3. PRISM2 to provide the DOH with technical assistance in strengthening and expanding their monitoring of quality of private and public facilities, building on ISO (International Organization for Standardization) trainings and practices to embrace principles and practices of quality improvement. 5.3 FUTURE PROSPECTS FOR PUBLIC-PRIVATE PARTNERSHIPS QUESTION 8: What PRISM2 interventions can be considered good practices, and which may be recommended for scaling-up in the future? How cost effective were these? For the purposes of this evaluation, a good practice is a measurably effective intervention to increase access to FP/MCH that has gained traction for improving services and service availability, and is highly replicable from a time and cost perspective. By this definition, while PRISM2 has implemented a number of promising practices, none as yet have sufficient data to be considered a good practice at this time. Among the promising practices of PRISM2 are the usapan for convening and counseling groups of prospective clients, the ADPs as means to efficiently provide point-of-contact access to a range of FP methods and related MCH products, and integrating PPMs within SDNs to improve the quality of and expand the range of services offered. PRISM2 has successfully adapted good practices from others’ experiences in the Philippines and elsewhere, such as taking the requisite time to generate initial buy-in from local stakeholders, distributing and marketing competitively priced contraceptives and other products, and contracting for services that others can provide efficiently. As presented in the 7th Quarter, PRISM2 was challenged to optimize connections and synergies with other USAID projects that were midway through their life of project as PRISM2 began. The round of new USAID projects, including the regional health projects, offers an opportunity to reverse this trend. Considerable effort and attention to handing over PRISM2 achievements by end-of-project could leave a legacy of strengthened and integrated SDNs on which these new projects can build. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 27 PRISM2 invested two years working with sub-national stakeholders to develop a common vision for empowering them to become stewards of the FP/MCH outcomes in their respective regions and to jointly identify priorities to achieve FP/MNCHN targets. This investment in time and resources directed at the sub-national level helped to advance a common vision for FP/MCH priorities within PPP. Among many key informants interviewed, some local health authorities reported that they intend to actualize and sustain the partnerships. Beyond what PRISM2 has been able to do during the brief time it has implemented activities, the government of the Philippines now appears more receptive to private collaborations: It has engaged the private sector through mechanisms including new policies, as documented in various AOs, and PhilHealth has expanded its coverage to include some FP/MCH services. Building and nurturing capacity within the DOH to actually outsource by contracting private sector entities and rationalizing policies and procedures related to FP/MCH within PhilHealth are next logical steps for either PRISM2 or successor projects. Though attribution to the project is difficult, in Bohol, PPMs who are affiliated with PRISM2 are providing services in a public clinic. Likewise, in key informant interviews, it was reported that the PPMs affiliated with PRISM2 were more likely to submit their service statistics for inclusion in the FHSIS. These are small but significant steps forward. The team encouraged PRISM2 to be more forthright about identifying and sharing such positive results. USAID has supported private-sector FP/MCH programs for more than thirty years and public￾sector FP/MCH programs for decades more; it is thus challenging to identify innovative methods and approaches. PRISM2 did promote and encourage various approaches to engage the private sector. For example, the PRISM2 subcontract with Alphamed promoted ADPs. Though it is not clear that these will survive without the support that PRISM2 provides, the participation of Alphamed and other commodity distributors in an ADP training in Cagayan de Oro is a promising indication of their commercial viability. As reported to the team, the project is also conducting an informal cost study to effect a change at PhilHealth in the pricing structure for BTLs to more fully cover the costs associated with service provision in the private sector. Similarly, PRISM2 engaged its sub-contractor PBSP to promote worker programs, although this effort was defined with only nineteen months remaining in the project, severely limiting potential for impact, though there is time for testing viable approaches. PRISM2 is promoting corporate social responsibility through its sub-grant to SIFI in Negros Orientale; the team observed a launch event at which representatives of the community, public, and private sectors made a commitment to collaborate in improving the health of women, children, and families working on the sugar cane plantation. Another PRISM2 sub-grantee, UHA in Cebu, is providing training to public and private midwives, an effort worthy of replication. Many of the tools developed by the sub-grantees to track accreditation status of providers and facilities warrant being shared across project sites. Some key informants said that the usapan was a repackaging of demand-generating efforts used previously in the Philippines, but acknowledged that targeting men through this midwife-led effort was unique. The team found that it was not feasible to provide cost-effectiveness measures for selected outputs or outcomes. PRISM2 was not contractually obligated to nor has it collected or reported costs by intervention or activity at any phase of the project, nor does it require its subcontractors or grantees to report disaggregated expenditures. Furthermore, since most activities did not get underway until well into Year 3, outcome measures of any note are unavailable. For example, investments in the usapan discussion groups can only reasonably measure at this point the numbers who have been trained in this BCC approach, not the numbers of clients reached, let alone the numbers who consequently accepted a contraceptive method. Chemonics made no efforts to connect this project with experiences in PPP and FP/MCH occurring outside of the Philippines. As one high level Chemonics staff member stated, PRISM2 would have been Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 28 better off being directed only to the private sector. With decades of USAID-funded effort to promote private-public involvement, including in neighboring countries such as India and Bangladesh, opportunities to learn from other projects should be explored. Likewise, additional steps can be taken to scale up some interventions, for example providing technical assistance to the DOH to issue and manage contracts to private institutions, such as UHA, for training additional service providers. PRISM2 could pilot the use of vouchers for FP/MCH services to supplement the PhilHealth reimbursement program and/or include credible private mid- and senior-level providers in the CHO/PHO TWGs. Conclusions DOH’s MNCHN strategy and efforts to promote universal health care through PhilHealth are solid building blocks for future healthcare and expanded coverage. Could such anomalies in operationalizing these be usefully assisted through additional development investments? For example, providing technical assistance to build upon ongoing discussions between DOH and PhilHealth could help both by mediating and further developing and refining policies and practices. If not within the manageable interests of PRISM2, this is certainly a worthwhile investment for USAID/Philippines to consider. PRISM2 has a modest track record of working closely with other USAID projects. The slow start-up, unique vision of stewardship between the public and private sectors, a corporate sense of top-down management, and insufficient empowerment of field-based staff, particularly evident in current operations, contributed to the disconnection from other projects. PRISM2 will be able to leave a stronger legacy if it reverses this pattern during the remaining months of the project to identify and cultivate opportunities for synergies with USAID projects just now being fielded in Luzon, Mindanao, and Visayas. Recommendations Regarding key contributions and best practices, it is important not to underestimate the innovativeness of PRISM2. Given findings and conclusions discussed above, the evaluation team recommends that: 1. USAID/Philippines and PRISM2 should create and maximize opportunities to effectively bridge the ongoing and new health projects in Luzon, Mindanao and Visayas, including the transfer of knowledge and experiences from PRISM2 SAMs and LMAMs. 2. USAID/Philippines and PRISM2 should support efforts at the central level through the provision of technical assistance in training a cadre of trainers within the DOH to maximize capacity development, inclusion of FP/MCH, and post-abortion care (PAC) in midwife and nurses classes (discussed in more detail in question 9), issuing RFPs and the subsequent management of contracts for training, monitoring, and contraceptive distribution.22 3. USAID/Philippines, through its connections with USAID Global Health and other investment donors, should identify experiences ongoing within other countries in public-private partnerships and invest to share these with implementing partners in the Philippines. 4. USAID/Philippines should inform contractors when cost-effectiveness measures of particular 22 Information on PAC and USAID support of PAC can be found on the following sites: http://www.postabortioncare.org/index.shtml http://www.shopsproject.org/sites/default/files/resources/2609_file_Formatted_9_PAC_Kenya_Case_Study_by_M ary.pdf http://transition.usaid.gov/our_work/global_health/pop/techareas/pac/index.html, http://www.medscape.com/viewarticle/560965, http://www.popcouncil.org/pdfs/frontiers/presentations/PACmeeting2008/Curtis1.pdf http://www.respond-project.org/pages/pac/2012-summer-newsletter.html Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 29 outcomes are important and require contractors to collect cost data by activity or purpose and possibly by region from the beginning of a project. It is not recommended to ask PRISM2 staff to disaggregate cost data on expenses already incurred during the first three years of the project or at the end of the project. QUESTION 9: Are there more effective approaches with the private sector which could have been explored to achieve FP/MNCHN outcomes? PRISM2 built on a number of proven or promising private sector approaches in the Philippines. For example, PRISM2’s grant to Alphamed to market FP/MCH products built upon a long history of successful experiences with social marketing in the Philippines, including experiences within the original PRISM Project. PRISM2 also continued to engage PPMs for FP/MCH and to work through IMAP as a grantee, both of which had previously produced positive results within the previous PRISM Project. However, based on what the team observed during site visits, the project could have done more in some other related approaches. For example, in a country where some 40 percent of deliveries still occur at home, PRISM2 could have encouraged PPMs to extend coverage by partnering with traditional birth attendants (TBAs) or hilots. Extending the reach of PPMs could have proven useful, particularly in rural areas. As previously indicated, PRISM2 could have capitalized more fully, especially early in the project, on groundwork previously laid by PBSP within some 500 industries. Extending FP/MCH services within workplace settings has proven effective in a number of countries, including Mexico and Egypt, whose experiences could have provided a solid base for avoiding concerns about coercion which subsequently came up in the Philippines. Before turning more fully to other international experiences with the private sector, there are other contemporary private sector approaches within the Philippines also worth considering. For example, Blue Star, Mother Bless, and Well Family Clinic franchises have been partnering with the government in a number of regions. Similarly, within PRISM2 itself there are ongoing experiences worthy of replication. For example, PRISM2’s work with a sugar plantation in Bocolod is a model to emulate where appropriate within the country. Similarly, grantee UHA’s training of private providers to become qualified trainers for FP-CBT1 could be extended to other accredited trainings. In addition, what PRISM2 understands as keys to progress in some provinces, such as Cebu, should be shared for others to learn from. PRISM2 has not yet sufficiently identified and documented such positive project experiences. More broadly, the team found no evidence that PRISM2 had ever connected with or attempted to learn from PPP experiences in other countries. In addition, though the team tapped into the website of USAID’s SHOPS Project (http://www.shopsproject.org/) for some documents on contemporary international experiences with PPP, this resource never came up within key informant responses to questions about PPP experiences with potential relevance for the Philippines. Rather, some held quite provincial points of view, such as the local DOH official who stated: “We had worked in PPP in other areas – TB, but not FP. We don’t need to know about other country examples of PPP in FP, what we have is sufficient.” In addition to information that SHOPS is collecting, there are other contemporary international practices offering parallel and relevant approaches in PPP. For example, in India, initially in Gujarat and subsequently in other impoverished states, the public sector contracted out to the private sector to dramatically increase institutional deliveries, with corresponding reductions in maternal mortality. Also, experiences in a number of countries with national health insurance schemes, such as in India with the Rashtriya Swaasth Bima Yojna, provide ongoing experiences of potential value for the Philippines. Given Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 30 the breadth of experience that a contractor like Chemonics brings, this omission of international experiences is unexpected and unacceptable. Finally, one approach that has gained considerable traction within the private sector in many countries, often with USAID support for training and advocacy, has been post-abortion care (PAC).23 While it is important to ensure that PAC is an integral part of maternal care within the public sector, often private practitioners are those consulted when a woman is dealing with the complications of a spontaneous or a clandestine, illegal abortion. Meeting women’s needs for such emergency care has been an effective part of national efforts in countries such as India and Nigeria to reduce maternal mortality. PAC recipients are among those who have the greatest need for effective family planning services. Conclusions PRISM2’s approach to support private health providers is based on private sector approaches that have been effectively implemented in previous projects in the Philippines. However, PRISM2 did not fully build upon what had previously worked well in the Philippines nor has it fully shared within or outside the project some of its own positive experiences and results. While lessons being learned within PRISM2 can be extracted within the time remaining, failure to connect with public-private partnership experiences in other countries is a missed opportunity not so easily remedied at this point. Within PRISM2 there are practices and approaches that can still be shared across the project and with new USAID projects coming on line. From abroad, many other countries are similarly grappling with the same complexities that confront the DOH and PhilHealth in engaging the private sector. Learning about and building on both internal and international experiences has not to date been a hallmark of PRISM2. Recommendations 1. In the time remaining, PRISM2 should fully document and share field-level practices where PPP has significantly advanced; this would both demonstrate the project’s positive accomplishments and help the new regional FP/MCH projects. 2. PRISM2 and the new regional health projects should promote PPP by supporting the DOH to outsource or contract out FP/MNCHN related services. 3. Solicit and elicit PPP experiences from other countries. SHOPS is a resource to be tapped, collaborative cross-country visits of government as well as private sector participants could be supported, and/or experts could be invited to share experiences from Bangladesh, India, Egypt, or other countries (perhaps as keynote speakers for a PRISM2 all-staff or USAID project coordination meeting). QUESTION 10: How have gender considerations been integrated in USAID’s PRISM2 project? What are the effects of the project on male and female beneficiaries? Findings As stated in the PRISM2 SOW, all project activities were required to strengthen men’s and women’s decision-making in determining the number and spacing of children, ensure that gender is considered in the delivery of health services and products to both men and women, improve the capacity of health facilities to respond to the special needs of men, women, boys, and girls in providing FP and MCH, and guarantee the participation of men and women in project activities, with men and women treated as 23 See USAID’s PAC website: http://transition.usaid.gov/our_work/global_health/pop/techareas/pac/. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 31 equal partners in their roles as clients, providers of services, and potential implementers of technical assistance under the project. The USAID Philippines Health Strategy 2011-2016 echoes these requirements, which encourage the creation of more male-targeted activities in FP programs. This is based on the finding that Filipino men consider FP important and are interested in being involved in FP decision-making. Specifically, the Strategy calls for demand generation interventions to promote male participation in FP with information on male FP methods widely distributed and satisfied users of vasectomy utilized as champions to increase men’s involvement in FP. The Strategy points out that FP programs in the Philippines are almost exclusively focused on women. Building on the USAID Health Strategy, PRISM2 developed its own Gender Strategy (GS), which purports that gender equality should be reflected in the overall project design and within each of the TIPPPS, technical support packages (TSP), and activities. The GS includes a table that summarizes the features of gender equality within FP/MCH divided across five areas of gender equality. The GS states that all of the features will be used as indicators in assessing the extent of contributions of the TIPPPs to gender awareness, mainstreaming, transformation and synchronization. However, the team has been unable to locate any evidence of these indicators. The GS also presents a comprehensive table disaggregated across Project Development Cycle, PRISM2 Gender Actions, and Activities/Procedures, Tools, and Materials of TIPPPs where Gender Actions will be integrated. The team was unable to locate a record of the project’s progress towards completion of the listed activities and procedures. While the PRISM2 GS highlights that almost 20 percent of unmet need for FP among Filipino couples can be attributed to husband’s negative perceptions of contraceptives, the usapan discussion sessions by PPMs lack a thoughtful plan to increase male participation. The team did not find evidence of how PPMs use the PRISM2 usapan module specifically designed for men, Usapan Bagong Maginoo, to target and recruit men. The only data available at the time of writing this report were for usapan sessions conducted in Visayas during the 14th Quarter, which did not indicate that any Usapan Bagong Maginoo sessions had been conducted. However, of the thirty-three total, standard usapan sessions conducted project-wide by PPMs for their usual clientele receiving services, male participation was reported at 11.5 percent. The plan to engage IWGs as participants of Usapan Bagong Maginoo through grantee PBSP was discussed earlier in this report, but, in short, this model to reach men began only last year and is experiencing challenges in recruiting volunteer facilitators. In compliance with these strategies and guidelines, PRISM2 identified critical steps for integrating gender across all areas of the project, starting with the Year 1 Work Plan. However, a thorough review of PRISM2’s subsequent annual work plans reveals the project’s movement away from a rich and ambitious set of activities and objectives underpinned by gender analyses, assessments, and issue papers, to a re￾focused pursuit of gender almost exclusively within the delivery of a one-day gender and GBV-oriented module. The Year 3 Work Plan states that project milestones and expected outputs include a guide to promoting gender transformative FP/MCH services, the development of a gender strategy paper, the drafting of a gender audit tool and application of the tool with CHDs in pilot LMAs in the Visayas with action plans for integration of gender into FP/MCH based on the results, and the orientation of the project grantee, SIFI, on integrating gender into FP/MCH accompanied by a baseline gender assessment. Among these proposed and approved milestones and activities, the team was only able to verify completion of the GS Paper. As of the 14th Quarter, the single gender-related indicator tracked by the project had accomplished 14 percent of its five-year cumulative target. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 32 The team found the Gender Specialist to be a highly valuable staff member of PRISM2’s team. She was exceptionally well-versed in the gender dimensions of both FP/MCH and the Philippines. The initiatives she is responsible for designing and facilitating are dynamic and thoughtful. Conclusions PRISM2’s work plans reveal that inclusion of a gender focus in the project has changed from a crosscutting initiative to more of a separate, stand-alone activity, an add-on to specialized trainings, workshops, and discussions. PRISM2 may not have planned to address gender in FP/MCH beyond the project landscape or build partnerships with other USAID projects around this initiative. Nonetheless, at least in terms of proposed objectives, activities, deliverables, and outcomes, gender is largely being implemented as a one-off activity with little promise for widespread outcomes. It is unfortunate that the current Gender Specialist did not join the project earlier, as she likely would have been able to keep the project’s original aspirations for gender integration on track with her notable expertise and obvious passion for her work. The project lacks data to suggest that it has reached out to populations outside those already using the services of PPMs, that is, women who are already seeking FP/MCH services. The inclusion of gender-focused information in the usapan discussion series did not provide the midwives with methods for outreach to obtain a diversified clientele, such as would be affected by gender-based violence emphasized in the training. Tracking gender achievements within PRISM2 is made difficult by the lack of indicators that could be used to track the activities proposed in the annual work plans and to examine the progress against those activities reported in the quarterly reports; the lack of explanation about the connection between the activities described in the work plans and those presented in the GS; and lack of discussion about the connection between milestones and achievements reported in the quarterly reports, planned activities described in the work plans and the one, gender-specific project indicator. For example, the gender section of the 14th Quarterly Report includes Year 4 Milestones about the number of women and men receiving GBV services, which is reported at 9,700. At the same time, the 14th Quarter indicator tracking table only reports 1,239 people reached by GBV services for the 14th Quarter and a year-to￾date total of 2,036. Though gender is tracked only in the usapan sessions, there are many activities included in the quarterly reports that are likely gender related but are not captured because the single indicator for all of the gender-related activities is “the number of people reached by a USG-funded intervention providing GBV services.” Recommendations 1. PRISM2 should compose definitions with accompanying indicators for GS objectives.24 2. PRISM2 should carry out its statement in the GS regarding the development of indicators to measure the up-take of gender integration into the 4 prioritized TIPPPS, such as the repeated objective within the TIPPP on PPMs that states “ensure gender equity and equality in policies, systems, and procedures in engaging the PPM”. 3. PRISM2 should seek to determine the effectiveness of usapan as a demand generation tool by collecting data on the following: 24 Possible useful resources for future discussion about gender and GBV include Gender and Health Data and Statistics (https://www.cpc.unc.edu/measure/publications/ms-12-52) and A Summary report of New Evidence that Gender Perspectives Improve Reproductive Health Outcomes (http://www.prb.org/igwg_media/summary-report-gender￾perspectives.pdf) Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 33 a. The number of clients (both new and existing) who are new users of contraceptives as a result of participating in an usapan session, b. The number of new clients seeking FP-MCH services, and c. The diversity of new clients, e.g., age, sex, marital status, and number of children. 4. USAID should encourage PRISM2 to test new approaches to integrating gender across the project, such as identifying champions of male contraception and involvement in FP/MCH. 5. The level of effort for the gender specialist should either be increased or resources should be provided by PRISM2 so that he/she can train others to lead the gender-focused modules, because she is spread too thin and unable to respond to all the requests for the training. 6. PROGRESS TOWARDS THE THREE PROJECT COMPONENTS PRISM2’s Project Components (or objectives) outline the broad areas within which this Project has operated and towards which it has made some progress. However, due in part to a number of unforeseen circumstances as previously discussed – loss of leadership as the project readied for take-off, periodic changes in direction and scope, project management variations – progress has been neither steady nor always noteworthy. In this section, we summarize results of the evaluation, based largely on answers provided to the 10 SOW questions in the preceding section, but also incorporating the latest available data from USAID and PRISM2. We do this in broad strokes by returning to review the three Project Components and assessing PRISM’s overall accomplishments within these. Findings 1. Increase and sustain private sector provision of FP/MCH services and products through the workplace, private midwives, and other appropriate health providers. Due to the USAID mandate to shift project focus to workers, accomplishments within workplaces were limited to the first half of project implementation. Building on previous the PRISM Project and over 500 work-based industries and other similar venues where FP/MCH awareness had been increased, PRISM2 started off with momentum for high FP/MCH achievement in such settings. However, PBSP, the prime implementer for these efforts, did not have a signed contract until late in the third year. Though PRISM2 had developed some similar activities within industrial and commercial settings, these were on a smaller scale than would have otherwise have occurred. When USAID surfaced strong concerns about FP in such settings, PBSP shifted to worker groups. However, with this late start and little traction, this is unlikely to make major contributions to FP/MCH by the project’s end. Similarly, in building on PRISM’s prior accomplishments with 200 PPMs, PRISM2 has to date fallen short. Accreditation by PhilHealth has been PRISM2’s primary tactic towards expanding services provided by PPMs. Of some 1,000 PPMs that PRISM2 has engaged to date, 150 are accreditable and twenty-three have actually been accredited. With accreditation, PPMs and other private practitioners have received totals of 637 FP and 14,327 MCH reimbursements from PhilHealth. Some 250 birthing homes and private doctors also operate ADPs, and a number of PPMs have been trained in usapan discussion groups. However these activities did not start until Year 3, and are unlikely to generate large numbers by end of Project. With only thirty-four private organizations tabulated as participating in local PPPs and only three SDNs counted as strengthened midway through Year 4, PRISM2 achievements seem again slow and small. PRISM2 is also committed to improving access to LA/PM by training qualified health providers. Through December 2013, PRISM2 trained sixty-two midwives and other private providers in IUD insertion, including postpartum insertion. For BTL, thirty-six qualified physicians, both public and private, have Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 34 been trained. Had such newly trained providers been well-integrated into strengthened SDNs, data would be routinely available for tracking procedures subsequently performed and clients served. Linking private hospitals and multi-purpose clinics to SDNs and referral networks does not appear to have received as much attention as it could have. For one thing, the reimbursement levels for FP/MCH do not appear to cover hospital-based costs. For another, and more directly to PRISM2’s perception of PPP, more attention is given to PPMs than to private hospitals and other non-governmental clinics. Existing franchises such as Friendly Care and Blue Star operate largely outside of PRISM2 efforts to strengthen SDNs. Both the SDN definition of strengthened and the criteria for dropping LMAs prioritized PPMs. 2. Increase utilization of quality FP/MCH services and products in and from the private sector by providing health information to communities and marketing products and services. With only three SDNs counted as strengthened, prospects for increasing quality FP/MCH services are low. Though field implementation directed at SDNs only began in Year 3, over the years LMAMs have been in place to support DOH evolution towards partnership arrangements. Instead of robust SDNs throughout the country that combine public and private practitioners into one system, what currently exists are a few strengthened SDNs offering more promise for the future than actual delivery of services in the present. Marketing contraceptives and introducing some new products has been more productive. The network of private sector outlets called ADPs may continue to grow and may become self-sustaining through sales, increasing ready availability of low cost FP/MCH products. Alphamed, by the end of its grant in August 2012, counted 275 functioning ADPs, of which 126 were in birthing homes. As of December 2013, the PRISM2 count for ADPs was 209 offering FP products and twenty-five with MCH products. Grantee Alphamed also introduced some new products, such as the one-month injectable, as an added choice for clients. In addition, some already available Alphamed products gained substantially larger market shares, including oral contraceptives and the three-month injectable. Through ADPs and other activities Alphamed also did some demand generation. Within PRISM2, however, “usapan is BCC”, as one project leader stated. In general, the project has been “more supply than demand. A little of demand, but we are predominantly supply.” Alphamed tallied just over 200,000 CYP from product distribution and sales during its fifteen-month grant. PRISM2 also monitors and reports on the contraceptive share of USG products within the country. The result of all this is a project with insufficient accomplishments to have made marked or perhaps even noticeable contributions to FP/MCH services. As reported to the team by PRISM2 project leadership: “Our definition of CYP has changed fifteen times during the course of the Project, …(including) three times (during the last year) in terms of what we count and what we report on …This is what USAID needs to report. They want to know how much of USAID products are selling across the country …They have products from 25 years ago that they need to report. The CYP does not have any relevance in terms of what we report. [Interviewer: We only need to report what PRISM2 has contributed to CPR via CYP?] (This has) nothing to do with us. If you find it out , let me know. I am serious.” Such outcome measures as CPR and percentage of SBA-assisted births are much more likely to increase because of the changed position of the government on MCH/FP than as a result of what this project has done. And that is often the case – projects do not often impact countrywide markers. However, such broad measures can and should be tracked and assessed over the long term. This was done as part of the 2010 Assessment of the USAID’s FP/MCH portfolio to determine if USAID funding at regional levels made a difference in CPR; it did not. 3. Improve the policy environment for the private sector by providing technical assistance to strengthen the government’s capacity for engaging the private sector in providing health services and products. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 35 The project’s design and scope of field level activities, initially in seventy-seven and currently in thirty-six LMAs, have made a difference on government practices and to some extent policies for PPP. On policies promulgated within the LMAs, PRISM2 counts twenty LGUs as having two policies in place to work with the private sector. This promising start may be built upon in the coming nineteen months. Perhaps more policy and procedure-shaping efforts and buy-in from Manila-based public-sector leaders could have led to greater field level accomplishment. The Philippines’ devolved system and differential applications of policies and procedures clearly argue for less attention centrally than to field level. By deployment of staff to the field, PRISM2 has done that. But neither should central DOH nor PhilHealth be ignored; these both have important roles and significant influence over the range of efforts that can be pushed at field level. Central level is also a conduit to support how funds, of which there are much more now than in the past, may be allocated to support FP/MNCHN services. Conclusions and Recommendations: Hypotheses Reviewed and Revisited In this final section of the Report, we consider through the prism of our results the two developmental hypotheses in the Evaluation SOW. As USAID’s Senior Evaluation Advisor opined during a meeting on preliminary findings from this evaluation, properly speaking these are not actually hypotheses: Neither offers a description or posits a theory on how change is to occur. Rather, these are more aspirational statements about anticipated and hoped for results from the investments made in PPP and PRISM2. Nevertheless, the team has been asked and is pleased to revisit these statements in light of evaluation results. Public-private sector collaboration in health services provision, demand generation, and policy formulation and enforcement will significantly contribute to FP and MCH outcomes such as contraceptive prevalence rate (CPR) for modern methods and skilled birth attendance (SBA) in USG￾assisted areas Collaboration between public and private sectors could impact outcomes such as increasing CPR and percentages of SBA-assisted deliveries. As in many countries, more women deliver in birthing homes or at home than in public sector facilities. Collaboration between the public and private sectors should bring more attention to safe delivery as well as contraceptives to avoid unwanted or unplanned pregnancies. However, though collaboration covers a broad range of possible activities, developing a connected, cohesive system is likely to lead to greater impact. During this evaluation, the team received many indications, particularly from public sector officials, of openness to working with the private sector. It is not clear however that differences between the two sectors are fully understood or appreciated. Looking out for public health interests is the prerogative of the public sector. This is far different from the private sector’s focus on satisfying the needs of individual clients and gaining income in the process. That some of the public health staff interviewed also themselves engaged in private practice may help to foster mutual understanding. Many public health officials expressed interest in partnering with the private sector. This was often expressed more in terms of what could be gained for the public sector, such as data about services provided, than what could result for the entire system of healthcare as a whole. Although there is wide expression of interest in improving the quality of services, for public officials this could easily mean controlling quality within the private sector rather than both sectors together improving the overall quality of service provided. Preparing both sectors to fix problems rather than affixing or avoiding blame will be a long process. That ISO training is already fully underway in many public sector institutions should help in this regard. PhilHealth is the bridge between the two sectors. Through accreditation, PhilHealth provides the means to ensure basic standards of quality while also providing access to revenues and more clients. DOH licensing of and need for data from private providers could be linked to PhilHealth accreditation. Reimbursements for providing much needed services, such as timely referrals for complicated deliveries Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 36 or LA/PM, would be helpful. Perhaps PhilHealth could recognize and give benefits to those participating in an SDN. Dedicated project assistance in developing more efficient accreditation and reimbursement systems would be useful to consider. Commitment to quality health services for the poor and near-poor makes PhilHealth accreditation and enrollment critical components for increasing private provider participation. It may also be that the connections and working relationships between public and private sectors may more quickly build within curative rather than preventive care. As several key informants from within both the DOH and private sector attested, the TB LINC Project provided a first positive experience for engaging the private sector to meet a public health need, with case detection, standardized treatment and patient support, and an available drug supply. As shown, for example, by the number of reimbursements for FP (637) compared to MCH (14,327), maternal and child care improvements may well occur more quickly than increases in family planning services. Given the changed climate for FP in the Philippines, the openness within the MNCHN strategy to SDNs, and previous private sector accomplishments in FP, PRISM2 was a logical though bold next step. Despite mixed results, one hopes that USAID will continue to build on momentum thus far gained. Though the private sector has served FP well in the past in the Philippines, forging public-private partnerships may require other more expansive approaches. There are promising public-private partnerships from PRISM2 that contributed to improved FP and MCH outcomes that can be replicated or scaled up. Those SDNs that have been viably if not quite visibly strengthened as a result of PRISM2 offer the best opportunities for replication and scale up. It may be useful to have PRISM2 not just count which SDNs qualify as strengthened (three to date), but which of all SDNs offer the most robust possibilities for building upon and sustaining initiated partnerships. This may be somewhat different than having direct impact on FP/MCH outcomes; what may be more important here are process indications that technical working groups and referral networks are actually in place and functioning. Where traction within particular SDNs can be demonstrated, these should be identified as models to build on and replicate. The best that PRISM2 may offer may not at this point be in terms of FP/MCH outcomes, but rather in SDN possibilities. Along these lines, it may not be what is in the PMP database that is of most use. Rather, it may more be the unmeasured and undocumented processes occurring to build partnerships that will provide the most valuable insights for sustainability. This is not to negate promising investments in ADPs for making supplies more readily available or usapan as means to provide counseling or generate demand. PRISM2 should continue to track results from such investments to determine FP/MCH outcomes as best it can in the time remaining. So should initial forays with IWGs and youth be mined in order to pass on promising approaches to others interested in these groups, both of which are likely to receive considerable attention from donors’ future investments. Similarly, the Uniject-oxytocin study underway could well provide an added tool towards improved maternal care practices. However, to fully understand what has transpired towards building effective partnerships requires a finer focus than collected data on its own can provide. What remains largely within the experiences of the LMAMs, SAMs, and the partners with which they have most closely connected may still provide lasting value for what has transpired within and as a result of PRISM2. The soul imparted as stewardship through this project now needs to be explored at ground level to uncover the heart of what has actually occurred. On this basis, PRISM2 may impart a legacy of SDN and partnership experiences on which others can build and replicate. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 37 7. LESSONS LEARNED In this section, the team suggests alternate approaches to monitoring and measuring progress within PRISM2. Based on this evaluation, these lessons learned are offered more as considerations for future development investments than correctives for the current project. We hope and trust that these may prove useful for subsequent projects as ambitious and bold as PRISM2.  Ensuring evaluability. To evaluate the effectiveness of a project like PRISM2, in terms of its contribution to national FP and MCH indicators like CPRs, private SBA-assisted births, or sales of contraceptive products and services, it is necessary to attribute changes in the indicators to the project. The changes cannot, by themselves, demonstrate effectiveness – it is necessary to collect evidence on how the changes would differ had there been no project. It is especially important to collect such evidence in the case of regional interventions likely to have at most small impacts on national averages. Alternatives for building this evidence are included below. o Compare changes in regional FP/MCH indicators for regions where the project is active with changes for regions where the project is not active but which are comparable in other relevant ways (difference-in-differences and matching approach). This avoids attributing changes to the project that are occurring in comparable regions without it. The main risk is that project and comparison regions may differ in relevant but unobserved ways. o Reflect a specific theory of change (TOC) in targets for outputs due to the project, assumptions about risk factors, and targets for regional or national FP/MCH outcomes. Compare actual results with the targets and assumptions, and then revise them for the next performance period. While this approach does not fully solve attribution problems, it tests the relations between outputs, risk factors, and outcomes in the TOC. The TOC should improve with revisions over the course of the project. o Measure outputs such as private SBAs mobilized by the project or contraceptive product, and services sales increases due to the project and then relate those outputs to the FP/MCH outcome indicators in question. This is valid, however, only in the very limited circumstance where there are strong reasons to believe the outputs reflecting project activities in no way displace changes that would occur without the project. This approach can also understate effectiveness when projects have indirect effects like the partnership-building component of PRISM2. Comparison of outputs reflecting project activities with overall changes in national indicators for the output provides little support because it does not address the displacement and indirect-effect issues. Similarly, comparison of outputs reflecting project activities with overall changes in the output indicator for regions where the project is active provides little support without a further comparison with regions where the project is not active.  Measuring capacity building. To do justice to the importance of capacity building in projects like PRISM2, M&E plans can measure the effectiveness of training through testing or, better, follow￾up surveys rather than just attendance and time allotted. Similarly, they can measure the effectiveness of demand generation activities like PRISM2’s usapan groups by measuring FP uptake rather than just counseling. More broadly, indicators focused on trainer – rather than individual – training would probably energize the efforts of similar projects to build sustainable cadres ready to carry out capacity building. Regarding PPP capacity building, it would be possible to assess the effectiveness of public-private referral networks like those within PRISM2 SDNs by measuring flows of information, clients, and practitioners. Other measures could include joint Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 38 public-private organization activities, the frequency and depth of public-private national and sub￾national organization contacts, and instances of specified outcomes of public-private organization links.  Capturing the value of SDNs. Indicators for the establishment and strengthening of SDNs would be useful output measures for gauging the effectiveness of PRISM II over the rest of its life and of future projects like it. Both efforts appear to be highly relevant to the project’s distinctive PPP approach to FP/MCN. And their strong dependence on key PRISM II activities – including support for private providers, capacity building for government engagement of private providers, and partnership support – makes them robust measures of effectiveness in realizing contract deliverables.  Scaling innovations. It is important not to underestimate the innovativeness of projects like PRISM II, especially its design of helping government engage the private sector within SDNs providing frames for effective public-private referral networks and ADPs to improve access to new FP/MCH products and services. For one thing, it is important simply to celebrate this kind of innovation. It is also important to proliferate them, and that may mean scoping projects reflecting similar levels of innovation as pilots to be scaled up to national level only after the inevitable period of trial and error leads to replicable successes.  Supporting decentralization. Even where partner-government agencies have decentralized certain services or operations, as in the case of the establishment of regional CHDs in the Philippines, it may be necessary to build capacity to outsource or delegate related activities retained at the center in order to obtain all of the benefits expected from decentralization.  Measuring cost-effectiveness. Cost-effectiveness assessments require implementers to disaggregate costs by activity or purpose and, if possible, by region or implementing office. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 39 8. LIST OF ANNEXES A. Abridged Statement of Work: USAID/Philippines Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM 2) Project B. Evaluation Methodology C. Site Selection Matrix D. Categorization of Evaluation Questions by Directive E. Evaluation Team Roles and Responsibilities F. List of Initial Documents Reviewed G. Overarching Questions and Generic Questionnaire H. Cost Analysis I. Annotated Bibliography of Selected Documents Reviewed J. Key Informants K. Excerpts from Key Informant Interviews L. Analysis of the PRISM2 Performance Monitoring Plan (PMP) M. PRISM2 Organizational Chart N. Table 1 from PRISM2 Q14 Report (05-16-13) O. Resolutation of Statement of Differences (SOD) Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 40 ANNEX A: STATEMENT OF WORK (*Abridged) STATEMENT OF WORK USAID/Philippines Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM 2) Project I. SUMMARY A. Introduction The United States Agency for International Development (USAID) in the Philippines seeks to conduct a final performance evaluation of its Private Sector Mobilization for Family Health Phase II (PRISM2) Project. The evaluation will determine achievement of PRISM2’s overarching objectives of contributing to a) increased share of deliveries attended by private skilled birth attendants, and, b) increased market for pills, injectables, and long-acting and permanent methods (LA/PM) of family planning. The evaluation will also identify lessons which can be considered for new health projects as well as provide recommendations to improve PRISM2 implementation in its last year. USAID is seeking the services of a third party evaluation team (with participation of a local evaluation specialist) over a period of six weeks from February 11 to April 12, 2013 (or 48 work days), to implement the requirements defined in this statement of work. B. Background USAID Philippines’ Engagement of the Private Sector for Family Planning Traditionally, health services have been provided through the public sector. The long￾standing dominance of a public-sector-only model of service and product delivery has created market conditions that are unfavorable to a more vigorous commercial response among private providers capable of meeting the population’s need for family planning (FP) and maternal and child health (MCH). The National Objectives for Health (NOH) 2005-2010 defines the contribution of private sector from both for-profit and non-profit providers, focusing on those which are market-oriented and where health care is paid through user fees at the point of service. USAID’s support to increasing private sector participation has, in the past years, generated positive results. Under the “Well Family Midwife Clinic” and “PRISM” projects, models and technical assistance packages have been developed to establish viable community private midwife clinics and birthing homes that offer quality and low-priced FP and MCH services. The 2004, 2005 and 2006 Family Planning Survey results showed a significantly increasing trend of modern contraceptives obtained from private sector sources (32.8 percent, 35.6 percent and 40.7 percent, respectively). It should be noted though that the gradual phase-out of US contraceptive donations from 2003 to 2008 could have contributed to the shift to private sector sources. While the increases are significant, they are not enough to affect the national figures for contraceptive use. There is a need for scaling up interventions developed and implemented with USAID support to significantly contribute to the achievement of national goals for family planning and maternal and child health as means to reducing maternal, infant and child mortality. USAID Philippines Health Strategy for 2006-2012 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 41 The USAID program for 2006 to 2012, within which context PRISM2 has been designed and set up, was governed by a Strategic Objective Agreement (SOAG) signed with the Government of the Philippines (GPH) on 27 September 2006, with a Strategic Objective (SO) of “Improved Family Health Sustainably Achieved” (see Figure 1). To achieve the SO, the strategy focuses on four intermediate results (IRs). These are: IR 1: LGU provision and management of health services strengthened; IR 2: Provision of quality services by private and commercial providers expanded; IR 3: Appropriate healthy behaviors and practices increased; and IR 4: Policy environment and financing for provision of services improved. Similar to PRISM 1, the PRISM2 project is tasked to focus on IR2 and is expected to work closely with the other health projects working on the other IRs. However, its facilitative cross￾cutting activities should contribute to and/or capitalize on what have been developed in the other IRs, particularly IRs 3 and 4. In terms of its project components, PRISM 1 was almost solely focused on IR2, distinguished only by key target service/product delivery groups, i.e, workplace, private midwives and the pharmaceutical sector. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 42 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 43 II. DESCRIPTION OF THE PROJECT TO BE EVALUATED - USAID/Philippines PRISM2 Project The PRISM2 project is a 5-year contract (2009 – 2014) with a total budget of $34,852,261,which was awarded in October 2009 to Chemonics International and subcontractors Philippine Business for Social Progress (PBSP), Program for Appropriate Technology in Health (PATH) and The Manoff Group. PRISM2 supports USAID Health Program SO of “Improved family health sustainably achieved”, by providing assistance to the Government of the Philippines (GPH) to engage and mobilize private sector resources in the delivery of FP and MCH services and products. The main objectives of the assistance are: (a) increase the use of modern contraception obtained from private sector sources; (b) increase the share of deliveries attended by skilled birth attendants; and (c) increase the sales for pills and injectables. Designed to have nationwide impact, PRISM2 coverage was, from 2009 to 2011, 77 provinces and cities covering an estimated population of 73.6 million people, but has been, in 2012, streamlined to 36 sites covering an estimated population of 46.7 million people. Coverage was streamlined to ensure depth of engagement in these areas. PRISM2 seeks to assist the GPH Department of Health (DOH), the Department of Labor and Employment (DOLE) and Local Government Units (LGUs) in the engagement and mobilization of private sector resources in the delivery of FP and MCH services and products. PRISM2 focuses on three components, namely:  Component 1: Increasing and sustaining private sector provision of quality FP and MCH services and products through the workplace, private midwives and other health providers as appropriate;  Component 2: Increasing utilization of quality FP and MCH services and products in and from the private sector by providing health information to communities and marketing products and services; and  Component 3: Improving the policy environment for the private sector by providing technical assistance and strengthening the government’s capacity for engaging the private sector in the provision of health services and products. The focus of PRISM2 is to develop local markets for FP and MCH services and products, where there is interplay of suppliers (or providers) and consumers (or clients) of services and products and the regulatory environment where the market operates. Because it encompasses both public and private providers, the markets are venues for a fully-integrated approach to expansion and improvement of FP and MCH. PRISM2 will also undertake national level activities related to policy, systems and structural development that strengthen these local markets. PRISM2 will work on the gains of PRISM 1:  Around 200 midwife-owned birthing homes assisted in obtaining accreditation and improving their business  Various models of workplace health programs developed, including in the Autonomous Region of Muslim Mindanao (ARMM), which can be used as vehicles for provision of FP and MCH services in the locality  Over 500 workplace-based family health programs established, reaching over 300,000 employees in the formal sector and large cooperatives.  In partnership with DKT (the leading contraceptive social marketing firm in the Philippines) and other pharmaceutical distribution companies, introduced seven (7) new oral and Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 44 injectable contraceptives and one re-priced oral contraceptive, leading to an increase in the total market for pills and injectables PRISM2 intends to scale-up its reach to achieve critical mass. While PRISM 1 and 2 have similar objectives, they differ in the way they engage partners: PRISM 1 assisted the private sector directly to contribute to health outcomes; while PRISM2 assists the government partners and works with them in engaging the private sector, in line with USAID’s goal of strengthening the local capacity of the host country government to operationalize and manage its programs in the long run. As one of the projects implemented to support the USAID/Philippines Health SO, PRISM2 was expected to coordinate and work closely with other interventions namely:  Strengthening Local Governance for Health (HealthGov)  Sustainable Health Improvements through Empowerment and Local Development (SHIELD)  Micronutrient and Child Blindness Project (A2Z) 2011  Linking Initiatives and Networking to Control Tuberculosis (TBLINC)  Health Policy Development Project (HPDP)  Health Promotions Project (HealthPRO)  Building Actors and Leaders for Advancing Community Excellence in Development BALANCED) III. PROBLEM STATEMENT Looking at the first three years of PRISM2 implementation (October 2009 – September 2012), this performance evaluation of PRISM2 aims to: 1. Determine the effectiveness of the project, by investigating its achievement (in comparison with baselines and targets) of the following: Outcomes – (a) increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources; (b) increased share of deliveries attended by skilled birth attendants; and (c) increased contraceptive market for pills, injectables and LA/PM Outputs – various in the areas of (a) service delivery expansion and utilization; (b) behavior change communication; (c) policy and systems strengthening; and (d) capacity strengthening of GPH national and regional health agencies (and related ones), and local governments to engage the private sector in FP & MCH service delivery 2. Assess the effectiveness of PRISM2 design as well as implementation (management, operation and monitoring systems), in relation to achievement of contract deliverables and relationships with key clients from the government and private sectors 3. Identify PRISM2’s key contributions (e.g., unique achievements, innovative methods/approaches), and compare with other effective approaches of private sector involvement in FP and MCH. 4. Assess how PRISM2 collaborated with other projects in the USAID/Philippines Health portfolio, as well as complemented the strategy and priorities of the United States Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 45 Government (USG) and GPH DOH. Compare PRISM2 collaboration efforts with lessons learned in this aspect from PRISM 1. In the short-term, evaluation findings will be used to improve the last year of PRISM2 implementation. In the medium term, the good practices, lessons learned and recommendations will inform future USAID interventions involving the private sector, replicated either in the form of stand-alone activities, components of regional FP and MCH projects, or components of other USAID/Philippines-supported projects. Results of this evaluation will be discussed and disseminated with the GPH DOH, implementing partners, other development partners (such EU, JICA, UNFPA), and other institutions working to improve FP and MCH outcomes in the country. A. Development Hypotheses In the course of the evaluation, the hypotheses acting as underlying premises of PRISM2 design will be revisited:  Public-private sector collaboration in health services provision, demand generation and policy formulation and enforcement will significantly contribute to FP and MCH outcomes such as contraceptive prevalence rate (CPR) for modern methods and skilled birth attendance (SBA) in USG-assisted areas  There are promising public private partnerships (PPPs) from PRISM2 that contributed to improved FP and MCH outcomes which can be replicated or scaled up. B. Illustrative Evaluation Questions The list of illustrative evaluation questions will be reviewed and revised (if needed) by the evaluation team to inform a reasonable data collection tool: Effectiveness 1. What is the project’s contribution to improvement in national FP and MCH indicators? Outcome Indicators Baseline Achievement Target Actual Increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources Increased share of deliveries attended by skilled birth attendants Increased contraceptive market for pills, injectables and LA/PM 2. What are the key outputs and/or outcomes in the PRISM components, and which ones made the most contribution to project success? PRISM2 Components Achievement: Outputs and/or Outcomes Local Level/PRISM sites National Level Target Actual Target Actual Service delivery expansion and utilization Social marketing and behavior change communication Policy and systems strengthening 3. How effectively has PRISM2 strengthened the capacities to engage the private sector in FP & MCH service delivery? (Note: The evaluation team may refer to specific indicators identified in Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 46 several PRISM 2 documents related to “Technical Initiatives for Public-Private Partnerships” [TIPPs] to measure enhanced capacity in the said TA packages): Technical Assistance (TA) Packages PRISM2 Partners – Indications of Enhanced Capacity (cite outputs or indicators) DOH DOLE CHDs LGUs Ensuring contraceptive supply Information provision Engaging private midwives Expanding FP services in hospitals Increasing availability and accessibility of LA/PM services Tapping and financing resources for health Monitoring and evaluation of FP/Maternal, Neonatal and Child Health and N ii Institutionalizing FP/MNCHN Other TA What factors and conditions significantly contributed or hindered the achievements of outputs and outcomes in questions 2 and 3? 4. How did PRISM2 specifically address the following? • Development/strengthening of local markets for FP/ Maternal, Neonatal and Child Health and Nutrition (MNCHN) products and services • Responsiveness (inclusive of beneficiary targeting) to FP/MNCHN needs of government (DOH, DOLE, PhilHealth, CHDs, provincial/city/municipal health offices, provincial/municipal LGUs) as well as private sector partners and specific client groups Sustainability 5. What are the respective plans of the DOH and local governments in order to sustain the project-developed systems and interventions? Which initiatives are likely to continue and which ones will not be sustained once PRISM2 ends? (Cite reason/s.) Other Essentials Contractor Performance 6. Describe briefly PRISM2’s management mechanisms (i.e., headquarter oversight and involvement, organizational structure, field level operational set-up, personnel complement and their skill set, and the short-term technical assistance), and operational and monitoring mechanisms. How appropriate and effective were they, and how did they influence project performance and client satisfaction? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 47 7. How did PRISM2 complement (a) other USAID/Philippines Health projects; (b) United States Government’s “Best Practices at Scale in the Home, Community and Facilities (BEST): An Action Plan for Neonatal, Maternal , Child Health, Nutrition and Family Planning”; and (c) DOH’s MNCHN strategy and Universal Health Care agenda? Good Practice 8. What PRISM2 interventions can be considered good practices (e.g., unique achievements, innovative methods/approaches), and which may be recommended for scaling-up in the future? How cost-effective are these? (Note: The Evaluation Team may, subject to discussion with and approval of USAID, select PRISM2 interventions other than above enumerated for the cost-effectiveness analysis) 9. Are there more effective approaches with the private sector which could have been explored to achieve FP/MNCHN outcomes? Gender 10. How have gender considerations been integrated in USAID’s PRISM2 project? What are the effects of the project on male and female beneficiaries? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 48 IV. WORK PERFORMANCE REQUIREMENTS The Evaluation Team is expected to utilize qualitative and, as practicable, quantitative methods to obtain information necessary to meet the requirements of this SOW. This methodology is not prescriptive and other forms of information collection may be deemed necessary by the consultants. USAID will assist the consultant as much as possible to ensure that all the appropriate and necessary input is obtained to maximize the results of this engagement. 1. Desk Review and development of data collection tools Prior to in-country work, the Team will carry out a desk review of relevant documents and reports compiled by USAID/ Philippines and the PRISM2 contractor. This review will help team members familiarize themselves with the project and other matters relevant to the evaluation task, and will assist them as they document Mission efforts and the results and lessons learned from private sector involvement in health. The Team can request from the USAID Mission additional reports and data on the PRISM2 and other Mission health projects, as well as relevant government data. The desk review will also help the Team organize the materials and develop the evaluation tools. The team will be expected to extract relevant components of the reports for the purposes of the evaluation. Initial analysis and confirmation of project-specific data will be done at this point. Upon USAID approval of the Team-formulated list of outcome/output indicators and corresponding interview questions, the Team will develop the following: (a) quantitative and qualitative data collection instrument/s (i.e., before and after scenarios in USAID-assisted sites and unassisted sites), (b) initial list of sites to visit and accompanying selection criteria, and (c) initial list of interventions to assess for cost-effectiveness and accompanying methodology for the said analysis. During the desk review time, the Team is expected to participate in planning conference calls with USAID/Philippines, to review the goals and objectives of the assignment, discuss the evaluation design framework proposed by the Team, clarify team members’ roles and responsibilities, and draft the evaluation workplan. The work plan includes itinerary and interview schedule, and gender￾sensitive data collection methods and instruments (i.e., interview guides for key informant interviews and focus group discussions). Except for the itinerary and interview schedules, these will be finalized before the country visit. The list of documents to be provided to the Team is found in Annex C. 2. Country visit The evaluation team will spend approximately 4 weeks or 23 working days in the Philippines, in order to interact with USAID Philippines, key PRISM2 officers and staff, and project partners and other key stakeholders from the public and private sector. Activities in-country may include key informant interviews, focused group discussions, mini-surveys, and sample field validation of project data. In addition to assessing the before-and-after scenarios in project-assisted sites, visits and interviews in non-project sites (i.e., sites dropped from project coverage in Year 3) will be conducted to compare performance. The country visit will start with a Team Planning Meeting held at USAID/Philippines on the first day to review the goals and objectives of the assignment, review the status of the work plan, clarify any issues on the background material, finalize team members’ roles and responsibilities, review and make last revisions to the data collection methods and instruments, review and finalize the country itinerary and schedules. The Team shall develop evaluation tools that are consistent with USAID’s Evaluation Policy and Gender Policy, and will consider the project’s outcome/output indicators (Annex D) and expanded illustrative list of questions (Annex E). Evaluation tools discussed with USAID will be validated and improved, as needed, based on the pilot application of these tools. Within the first five days of in-country visit, the Team will finalize with USAID the quantitative and Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 49 qualitative data collection instrument/s, sites to visit, and at least three (3) PRISM2 interventions to assess for cost-effectiveness. The following are possible collaborations for this undertaking. During the entire Country Visit, the Evaluation team is expected to cooperate with the following, to be identified by the USAID/Philippines Mission: A representative from the GPH Department of Health and/or Center for Health Development, with extensive experience in FP and MCH program management, operations, policy, and monitoring and evaluation. A representative from USAID/Washington Global Health Bureau familiar with FP and MCH program design, monitoring and evaluation and is familiar with the workings of the private sector will assume an advisory role in the team. The following general development skills will be covered by one or both representatives of GPH-DOH and Washington GH Bureau: policy, health system strengthening, quality assurance, advocacy and behavior change communication, social marketing and gender, and health program evaluation. Similarly, an illustrative list of resource persons/organizations for the evaluation is in Annex F. 3. Debrief/Report The Evaluation Team will provide an internal preview/presentation for specific Mission staff, followed by a presentation to DOH and other relevant stakeholders prior to the end of their in￾country visit. (Based on discussion with the Mission, the Team may be asked to provide a mid- term update during their in-country visit.) The Team is expected to provide the first draft evaluation report within 12 days after completion of in-country work (six days LOE). USAID will provide comments within seven days from receipt of report. The Team is expected to submit final draft report within seven days (four days LOE), and USAID may send further comments within seven days. The Team shall then submit the final evaluation report within seven days (four days LOE), and USAID shall provide final comments seven days later, inclusive of the content of the publishable report and a draft statement of differences (if any). The Team shall submit the publishable report not later than 30 April 2013 (three days LOE). V. TEAM COMPOSITION AND QUALIFICATIONS To reduce bias, service of a third party evaluation team is being sought, particularly with the participation of a local evaluation specialist, who has not in any way been involved in the implementation of PRISM2. The following is an illustrative team composition for this Evaluation service. However, the Offeror is not limited to propose a set of Key Personnel with corresponding levels of effort they deem more appropriate for this undertaking. Senior Health Sector/Evaluation Expert (Team Leader). S/He must have excellent understanding of the challenges and opportunities in the health sector particularly FP and MCH. S/He is well versed on health systems, policy, regulation, service delivery and financing issues, as well as functional/operational arrangements that define current health services delivery in both the public and private commercial and civil society sectors. S/He must have at least ten years of experience in evaluating health programs, especially in Asia and should be able to provide samples of evaluation work done in the past three years. S/He must have knowledge of various program approaches in various countries as well as USAID’s Population and Health development framework. S/He must be familiar with USAID Evaluation Policy and contracting procedures. As Team Leader, s/he shall exercise overall supervision of the team and synthesize the various findings and recommendations from the consultations facilitated by the team. S/He will lead the preparation of the evaluation Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 50 workplan and will subsequently finalize it; and coordinate and lead the consultations with central and local government agencies and private sector groups. Excellent writing, analytical and team management skills are required. (Expatriate/Local) Institutional Capacity-Building Expert. S/He must have experience in designing, managing and evaluating organizational and program strategy change programs in the health sector, particularly in the area of FP and MCH in the Philippines or similar context. S/He is knowledgeable about management and partnership mechanisms that will improve delivery of public health services, at both the central and local levels, and involving both public and private sectors. S/He must have experience in design, coordination and execution of governance and capacity-building programs. S/He shall have experience in facilitating and participating in multi- agency and public-private partnerships (PPPs). S/He has extensive understanding of the workings of health policy formulation and enforcement and service delivery in a decentralized/ devolved setting. She has been engaged in institutional capacity-building work as manager, coordinator or evaluator in the last five years. S/He must have 7-10 years work experience in various aspects of institutional capacity-building. An advanced degree or formal training in management, human resource management, economics, organizational development, governance, and related courses is required. S/He has been engaged in institutional capacity- building work as manager, coordinator or evaluator in the last five years. (Local/Expatriate) Monitoring and Evaluation Specialist: A college degree or formal training in economics, statistics, demography, public administration or related course is required. S/he must have experience in the application of various analytical/statistical methodologies and tools used in health evaluation designs including meta-analysis. Familiarity with sampling designs and methodologies used in demographic and health surveys including analysis and interpretation of survey data. S/he should have experience in conducting cost-effectiveness assessment of health interventions. S/He should have excellent understanding of basic health indicators particularly indicators for FP and MCH and has at least 7-10 years of experience in a USAID project M&E. S/He must be familiar with the USAID evaluation policy. S/He must be familiar with health data quality-related issues in the Philippines and/or other developing countries. S/he must have prior documented experience (5-7 years) in project monitoring and evaluation, applied health statistics, estimation procedures of survey data, as well as procedures for data quality analysis. Operation of basic statistical software is a must. (Local/Expatriate) Business and Market Development Specialist. S/He must have prior documented experience (at least ten years) in business development, or franchising, and social enterprise development involving public and private sector partners. An advanced degree or formal training in Business Administration, Marketing, Economics or related course is required. S/He must have been involved in programs to institutionalize development cooperation interventions; keen in analyzing market forces and trends; and experienced in pharmaceutical marketing and distribution. S/He has been involved in social marketing of FP/MCH products, and is familiar with social behavior change communication as a marketing tool. S/He must have prior documented experience in the last five years in the above-mentioned areas. S/He is at least familiar with the Philippine market or has experienced working in similar countries, preferably in Asia. (Expatriate/Local) VI. PERIOD OF PERFORMANCE The assignment will be conducted from February 11 to April 12, 2013 (48 days). A six-day work week is approved. Illustrative Table of Level of Effort (LOE, in person/days) ACTIVITIES/TASK LOE Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 51 Background Preparation ○ Desk review of relevant materials ○ Develop evaluation design framework and methodology, data/information collection instruments and interview guides, criteria for selection of sites and non-project sites to visit, work plan and draft itinerary 5 Travel day(s) to/from Philippines 3 In-country work ○ Team Planning Meeting ○ Initial meeting/briefing with Mission ○ Field visits/ interviews/ information collection, focus group discussion, report drafting (possible mid-term update with Mission staff managing the evaluation) ○ Submission to USAID of draft findings at least two (2) days before debriefing 23 VII. DELIVERABLES 1.Detailed evaluation design and work plan; methodology (including data collection instruments and method of general evaluation analysis and cost-effectiveness analysis); evaluation report outline; and draft itinerary. The design should include recommendation on criteria for selecting sites to be visited. These are due on the last day of the desk review and will be finalized with USAID on Day 2 of the in-country evaluation period. 2. Accomplished interview guides containing information from key informant interviews and focus group discussions, as well as complete data sets from mini￾surveys (electronic). 3. Summary of draft findings to USAID two days prior to the debriefings. 4. A Powerpoint presentation containing findings with conclusions and recommendations, for the internal debriefing to USAID (Mission Director, OH and PRM) as well as the external debriefing for the DOH and other relevant stakeholders (e.g., representative/s from PRISM2, PhilHealth, PopCom, CHD, and LGUs). 5. Detailed first draft of the evaluation report, to be provided to USAID/Philippines no later than 12 days after completion of in-country work. The report should not exceed 30 pages with an executive summary of no more than three (3) pages, excluding annexes. The report shall (a) follow the USAID general guidance on Preparing Evaluation Reports (Annex A), (b) satisfy the detailed USAID criteria for Evaluation Reports (see Annex B), and (c) contain all sections listed in the approved report outline (see Sample Evaluation Report Template as reference). 6. Final draft of the evaluation report, to be provided to USAID/Philippines within 14 days from receipt of USAID comments on the first draft report. 7. Final publishable evaluation report, to be provided to USAID/Philippines no later than June 30, 2013. Submission is inclusive of the following:  Three (3) hard copies  Electronic copy in PDF and MS Word formats Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 52  Electronic copy of the PowerPoint presentation on the highlights of the Evaluation Report VIII. PERFORMANCE STANDARDS Contractor The Contractor will coordinate with relevant USAID staff (PRM and OH), and manage the evaluation team. It will undertake the following specific responsibilities throughout the assignment: ● Recruit and hire the evaluation team. ● Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. ● Send in advance, for discussion with and approval of PRM and OH, the evaluation design framework, evaluation tools, work plan, and other documents relevant to the evaluation. ● Conduct consultation calls with relevant USAID staff (PRM and OH) will be arranged to review the objectives of the assignment, discuss the evaluation design framework, draft the work plan (including a country itinerary and schedule of interviews), clarify team members’ roles and responsibilities, and develop gender-sensitive data collection methods and instruments (finalized with USAID staff). ● Prepare itinerary and schedules of meetings and interviews with priority government agencies, private sector partners, grantees, experts/consultants and donors supporting other FP and MCH interventions will be set, with OH inputs, on the first few days of the evaluation period. USAID/Philippines The Mission will provide general oversight of the evaluation contractor and the process throughout the assignment, and will assist in the following tasks: Before In-Country Work ● SOW. Respond to queries about the SOW and/or the assignment at large. ● Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. ● Documents. Compile and send electronic copies of most relevant project documents and background materials for the consultants through the Contractor, at least one week prior to the inception of the assignment.  Other Members of the Team. Identify USAID/GH and GPH DOH representatives the Evaluation Team will possibly collaborate with. ● Site Visit Preparations. Prepare initial list of USG and non-USG sites for field visits based on criteria proposed by the Team, key contact persons and their contact numbers. Provide the said list to the Evaluation Team. ● Lodging and Travel. Provide information on recommended secure hotels and modes of travel (i.e., car rental companies and other means). During In-Country Work Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 53 ● Meetings with Team. Throughout the in-country work, ensure constant availability of the PRM M&E specialist and the PRISM2 COR, and provide technical leadership and direction for the team’s work. Brief the team at the start of the evaluation to address questions and clarifications about the assessment, and any modifications in design, evaluation tools and itinerary. ● Meeting Space. Provide guidance with the team’s logistics coordinator on the team’s selection of a meeting space for interviews and/or discussions (i.e. USAID space if available, or other secure spaces). ● Facilitate Contact with Implementing Partners. Assist the team in arranging and coordinating meetings with stakeholders. Introduce the evaluation team through letters to implementing partners and other stakeholders prior to the team’s arrival and/or anticipated meetings. After In-Country work USAID will provide timely review of draft and final versions of the evaluation report, as well as approval of deliverables. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 54 ANNEX B: EVALUATION METHODOLOGY The starting point for this Evaluation was an extensive review of documents, initially what USAID provided, but subsequently what the team requested or located on its own as supplementary, e.g., USAIDs RIG Audit, the Portfolio Assessment, or articles on private sector participation in FP/MCH. When in country, the team conducted more than 50 Key Informant Interviews. Early in the Evaluation, the team began making site visits to a birthing home and clinic (in Manila) and also had the opportunity to observe a PRISM2 activity (in Pampanga). The team conducted interviews with multiple key informants, similar to a focus group, and then also met again with select key informants to confirm or challenge interpretations of data collected. The team reviewed additional documents obtained and often revisited some documents reviewed at the beginning of the assignment with the broader lens obtained from the weeks in the field and hours of interviews. In accordance with the Evaluation SOW and the team’s Work Plan, the methodology for the PRISM2 Evaluation featured document review and key informant Interviews. In all, team members initially read and subsequently incorporated into their analysis some 40 documents. The team added to these a substantial number of documents, articles, and information collected from research conducted on the internet. In addition, team members interviewed as key informants, 21 PRISM2 staff and 9 representatives of partner grantees and subcontractors. The team used other approaches, such as focus group discussion; in all conducting 7 group discussions, each with 3 to 7 participants. The team also observed PRISM2 activities on a number of occasions, such as training programs and the launch of an SDN and visited birthing homes, clinics and hospitals to collect complementary information from partner participants and organizations. These methods are detailed as follows: • Document Review: Initially during the week of April 1-5, but also continuing throughout the entire period of this evaluation, the team reviewed a comprehensive set of PRISM2 and related Project documents. As indicated in the SOW, USAID/Philippines provided a comprehensive set of documents for the team to review. In addition, team members identified a number of other documents, from both internet searches and some key informants. Documents reviewed include the USAID Best Strategy (2011-2017); PRISM2 Evaluation SOW; USAID’s Monitoring and Evaluation Guidelines; PRISM2 Quarterly Reports; TIPPPs; as well as documents related to major subjects covered during this Review, such as private sector experiences with FP/MCH and capacity building. Key documents reviewed for this evaluation are listed in Annexes G and J of this Report. • Interview Key Informants: For purposes of this evaluation, a key informant was defined as one with breadth as well as depth of knowledge about PRISM2 and/or the broader fields of FP/MCH and private practitioner involvement in providing related information and services. Though the SOW did not specify a specific number of key informants to be interviewed, the team estimated at the outset that it would interview between 25 and 40 key informants. In all, team members surpassed this by interviewing 57 key informants from 15 organizations and 9 private sector facilities. As particularly useful for this methodology, most of these interviews were done in-person. Some key informants were interviewed by phone, particularly those based in the US. A few key informants were not directly connected with Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 55 PRISM2 but offered broad perspectives on private-public partnerships, FP/MCH, and/or related developments in the Philippines. Working from a preliminary list provided by USAID/Philippines, key informants interviewed included a number of staff working within PRISM2, current and previous staff of partner organizations, and representatives of other organizations who have current, previous, or potentially future involvement with PRISM. The evaluators made particular effort to interview field-based PRISM2 staff as well as direct beneficiaries such as private midwives. The list of all those interviewed is in Annex _ of this Report. To interview key informants, the evaluators developed semi-structured questionnaires using open-ended questions. Building upon the ten questions included in the SOW and four directives formulated within the SOW Problem Statement, the team generated a generic questionnaire. This questionnaire was framed within and coded to the ten questions and four directives. In advance of each key informant interview, team members developed a tailored questionnaire. The generic questionnaire, which provided the base from which questions were selected for these interviews, is appended to this Report as Annex. Most in-person interviews were conducted between April 11th and 26th, when all team members were together in the Philippines. During this extended period, team members had face-to-face interviews with a number of key informants in Manila as well as a number based in locations to which team members traveled, in NCR, Luzon, Mindinao, and Visayas. Some selected key informants were interviewed more than once. This was particularly the case with the PRISM2 leadership who made themselves available for multiple interviews. In addition to USAID Philippines Office of Health staff being interviewed as individual key informants, the evaluation unit of USAID, notably John Callanta, Fatima Verzosa, and Jerry Britin also had, opportunities to guide this evaluation during the meetings of April 10th, 16th, May 2nd, and May 3rd. The list from which the evaluators identified key informants and conducted interviews is in Annex J. • Service Delivery Site Visits: As time and proximity allowed, team members also visited private birthing homes, clinic and hospital sites providing FP/MCH services. This gave perspective on how PRISM2 partners may have benefitted before and after this Project as well as insight on parallel developments occurring within the private sector. • Activity Observation: On a limited basis and as opportunity and time allowed, team members observed ongoing activities of PRISM2. Though limited by time and opportunity constraints, such opportunities enabled team members to collect a different level of data than that yielded by interviews or document review. These opportunities also enabled the team to interview other key informants, who were participating in these activities, as well as collect some information from activity participants, such as through a focus group discussion with midwives participating in an ADP training in Cagayan de Oro. Data Analysis: Documents reviewed and key informants responses to interview questions are the primary data sources for this evaluation. These data are complemented by responses received from other approaches used to collect data as well as direct observations that team members were able to make. To access and organize information from the literature review, many of the documents reviewed were categorized by relevance to the ten questions included in the SOW. This approach enabled interviewers to collate documents as appropriate to specific SOW questions. As a result, team members were able to quickly identify and extract from documents salient information for formulating answers to the questions. A similar process of codifying and layering questions was used to structure key informant interviews and analysis of responses. Almost all interviews were digitally recorded (voice recordings) Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 56 and saved as electronic files. The recordings were then transcribed. To analyze key informant responses, the evaluators also reviewed responses by listening to the digital files, a useful exercise that provided deeper insight into what key informants were reporting. The written transcriptions of interview responses were coded in relation to the four SOW directives in the Problem Statement and associated with the 10 evaluation questions in the SOW. This feature of the interview data allowed for collation of responses at these different levels. This facilitated analysis of some responses on specific topics. However, within the time constraints of this Evaluation, it was not possible to comprehensively analyze all responses at these different levels. Stored as electronic files, some selected content areas within these files were also searched for particular themes and issues relevant to the evaluation. Analysis of data proceeded at two levels: the 10 evaluation questions in the SOW and the four higher level overarching directives provided in the SOW Problem Statement. Information collected through other means, notably the focus group discussion and direct observations, though less comprehensive than the key informant responses, were also reviewed in relation to the ten evaluation questions. Preliminary summations by answering these questions were checked against information from project documents, member survey responses, and opportunities to observe activities or service delivery sites. Triangulation occurred in terms of what documents indicate PRISM2 intended to and/or did accomplish, what key informants reported about PRISM2 implementation and operations, and what team members learned from this rich combination of document review, key informant interviews, additional data sources, and occasions to observe PRISM2 in action. As encouraged by USAID/Philippines, special attention was paid to future prospects and potential for public-private partnership to advance FP/MCH, especially within mid and low income families in the Philippines. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 57 ANNEX C: SITE SELECTION As shown in this Figure, of the 36 PRISM2 LMAs, the Evaluation team collected data from the 10 sites (in purple) in five regions. That the SOW included opportunity and time for the team to visit such sites added a critical and useful dimension to this Evaluation. To cover more ground, the team divided into two for some of these visits. All went to Cagayan de Oro; with one team then traveling to Davao and the other to Bohol, Bacolod, and Cebu in Visayas. When the team reconvened in Manila, they conducted additional and some follow-up interviews undertook preliminary analysis of data collected to that point. Initial findings were presented to USAID/Philippines on May 2nd and May 3rd. Analysis and Report writing were completed in the following weeks. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 58 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 59 ANNEX D: CATEGORIZATION OF EVALUATION QUESTIONS BY DIRECTIVE D I R E C T I V E S 1. Determine Project Effectiveness – achievements in terms of outcomes and outputs 2. Assess Effectiveness of PRISM2 – design as well as implementation 3. Identify PRISM2’s Key Contributions – including comparison with other effective private sector approaches for FP/MCH 4. Assess PRISM2’s Collaboration – with other USAID/Philippines projects and as a complement to strategies and priorities of USAID/Philippines and host country DOH EVALUATION QUESTIONS by Directive Category 1. Investigate Project achievement (in comparison with baselines and targets): • Increased Outcomes – CPR; SBA deliveries; and market for pills, injectables and LA/PM • Outputs – (a) service delivery expansion and utilization; (b)BCC; (c) policy and systems strengthening; and (d) capacity strengthening of GPH national and regional health agencies (and related ones), and LGUs to engage the private sector in FP & MCH service delivery 2. Which key outputs and/or outcomes contributed most to project success? 3. How effectively has PRISM2 strengthened capacities to engage the private sector in FP/MCH service delivery? 4. How were local markets for FP/MNCHN products and services strengthened and how responsive was the Project to government as well as private sector partners and specific client groups? 5. Sustainability – Which initiatives are likely to continue and which ones will not be sustained once PRISM2 ends? 6. Management, operations, monitoring: Appropriate/effective 8. What PRISM2 interventions can be considered good practices (e.g., unique achievements, innovative methods/approaches), and which may be recommended for scaling-up in the future? How cost-effective are these? 9. Are there more effective approaches with the private sector which could have been explored to achieve FP/MNCHN outcomes? 7. How did PRISM2 complement (a) other USAID/Philippines Health projects; (b) United States Government’s “Best Practices at Scale in the Home, Community and Facilities (BEST): An Action Plan for Neonatal, Maternal , Child Health, Nutrition and Family Planning”; and (c) DOH’s MNCHN strategy and Universal Health Care agenda? 10. How have gender considerations been integrated in USAID’s PRISM2 project? What are the effects of the project on male and female beneficiaries? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 60 for performance and client satisfaction? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 61 ANNEX E: EVALUATION TEAM ROLES AND RESPONSIBILITIES Don Lauro/Team Leader:  Manage the SI team throughout the evaluation –before departure to Philippines, during the fieldwork in Philippines, and following conclusion of the fieldwork during the report-writing and report revisions phase  Ensure the overall quality of the work done by the SI team throughout the duration of the evaluation; make sure that SI quality assurance protocol are instituted and followed throughout the project  Prior to departure to the field, take the lead on drafting the evaluation work plan, methodology, and data collection tools – help to organize the contributions of other team members and ensure overall quality of the deliverables (including timeliness in submission to the USAID/Vietnam Mission)  During the field-work portion of the evaluation, function as lead point-of-contact with USAID/Philippines on behalf of the whole SI team according to the agreed upon evaluation work-plan and schedule;  Respond to any concerns that USAID/Philippines might have with respect to any of the activities included in the evaluation work plan developed by the SI team and approved by USAID/Philippines;  Inform the finalization of all data collection instruments assessing PRISM II implementation, performance and achievement of results;  Take the lead on data analysis as well as managing the contributions of other team members to this process  Contribute directly to the successful completion of the data collection activities  Lead responsibility for drafting the evaluation report outline  Lead responsibility for managing the writing process of the draft evaluation report  Ensure that final evaluation recommendations are relevant to USAID/Philippines management needs;  Lead responsibility for preparing and delivering the oral presentation to USAID/Philippines;  Lead responsibility for managing the writing and editing and review process for the final evaluation report Beverly Tucker/Capacity Building Expert:  Support the SI team in the field with expertise on capacity building in FP and MCH as it relates to the evaluation and context in the Philippines;  Provide technical input to the development and finalization of the evaluation work plan, strategy and methodology, as well as the data collection tools;  Help to identify relevant sources of information available on capacity building in FP and MCH as it relates to the evaluation of the PRISM II project and the Philippines for triangulation purposes;  Contribute to the development of all technical components of the project including all deliverables; Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 62  Contribute to the successful completion of the data collection activities, including key informant interviews, focus group discussions, conducting site visits, and implementation of the mini survey;  Ensure that all the evaluation recommendations are appropriate and useful for the needs of USAID/Philippines;  Assist with data analysis, draft report writing, preparation of the oral debrief and PowerPoint presentation and writing of the final evaluation report as needed. Dr. Joselito Vital/Business and Marketing Development Expert:  Support the SI team in the field with technical evaluation and business and marketing development expertise contextualized to the Philippines ;  Provide support as needed for communications in Tagalog during field work and any minor document/conversation translations when necessary;  Provide technical input to the development and finalization of the evaluation work plan, strategy and methodology, as well as the data collection tools  Identify all relevant sources of locally and culturally-specific information available on either the PRISM II Project or past evaluations of similar Programs in the Philippines for triangulation purposes;  Assist with the Identification of, and communication with, key informants;  Contribute to the development of all technical components of the project including all deliverables while ensuring that they are specially tailored to the Philippines context;  Provide the whole SI team with insights on the political, cultural and social milieu where the evaluation will take place;  Contribute to the successful completion of the data collection activities, including key informant interviews, focus group discussions, and implementation on the survey;  Ensure that all the evaluation recommendations are appropriate and useful for the needs of USAID/Philippines;  Assist with data analysis, draft report writing, preparation of the oral debrief and PowerPoint presentation and writing of the final evaluation report as needed. Erica Holzaepfel/Evaluation Expert:  Support and work closely with the TL regarding the management of the evaluation team from an SI quality-assurance standpoint  Support and work closely with the TL on all interactions with the Mission to ensure the evaluation team’s compliance with the contractual obligations  Ensure the overall quality of the work done by the SI team throughout the duration of the evaluation; make sure that SI quality assurance protocol are instituted and followed throughout the project  Ensure the rigor of the data collection instruments developed to answer the evaluation key questions on strategic design, management, effectiveness, and sustainability  Working closely with the TL, serve as the lead evaluation technical input to the development and finalization of the proposed evaluation strategy, methodology, and data collection tools Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 63  Respond to any concerns that USAID/Philippines might have with respect to any of the activities included in the evaluation work plan developed by the SI team and approved by USAID/Philippines;  Contribute directly to the successful completion of the data collection activities including key informant interviews, focus group discussions, conducting site visits, and implementation of the mini survey;  Ensure that all the evaluation recommendations are appropriate and useful for the needs of USAID/Philippines;  Assist with data analysis, draft report writing, preparation of the oral debrief and PowerPoint presentation and writing of the final evaluation report as needed. Carol Bandahala/Department of Health, Family Health Office  To the extent her position and travel costs allow, work as a full team member during meetings with relevant organizations, interviews with key informants, and site visits;  Support the team and substantiate findings based on her unique experience and expertise as a long-time staff member of the DOH;  Serve as a special liaison between the team and DOH as well as other government officials at various levels  Assist and deepen team understanding of the DOH vis a vis devolution and partnering with the private sector  Provide support for approaching barangay community health teams Jenet Minanga/Logistics Coordinator:  Support the SI team in the field with technical evaluation expertise contextualized to Philippines and provide logistical support during field work as needed;  Help to identify all relevant sources of locally and culturally-specific information available on the PRISM II Project or past evaluations of similar projects in the Philippines for triangulation purposes;  Assist with the Identification of and scheduling of meetings with key informants and site visits;  Support the SI team with scheduling plane flights, hotel accommodations, local transport, translation services, printing, the set up and delivery of the final presentation, procurement of cell phones, cell credit, materials and support services for the evaluation as needed. David Apgar/Senior Technical Advisor:  Support the development of deliverables submitted to the Mission as necessary  Provide cost effective analysis on PRISM2 activities as it relates to key evaluation questions  Provide quality assurance for all deliverables submitted to Mission Rajwantie Sahai/Program Manager:  Support the field team from Social Impact headquarters  Provide technical and administrative backstopping support Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 64  Manage correspondence with Mission  Maintain fieldwork schedule and ensure that deadlines are met Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 65 ANNEX F: LIST OF INITIAL DOCUMENTS REVIEWED (Pre-Field Work)  RFTOP 492-13-000003 - Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM2) Project  Best Practices at Scale in the Home, Community and Facilities: An Action Plan for Maternal, Neonatal, Child Health and Nutrition October 2011 to September 2016  Social Impact’s Technical Proposal in response to RFTOP 492-13-000003  PRISM2 Work Plan (Yr 1- Yr4)  PRISM2 Quarterly Reports  PRISM2 Results Framework  PRISM2 Final Strategy Assessment Report  Philippines Best Action Plan  TIPPPS (8)  USAID Action Memorandum  Health Assistance Strategy - Family Health Improved Meeting Women’s Contraceptive Needs in the Philippines  Fight for Reproductive Health Bill grows in the Philippines (article)  Statistics on Filipino women and men's health and family planning  Fertility and Family Planning: 2011 Family Health Survey  PRISM2 website  Reproductive Law implementing rules clarifies gender sensitive provision of health services  Bangladesh Family Planning: Private Health Sector Assessment  Bangladesh Family Planning: LAPM  Private Sector Mobilization for Family Planning Project Assessment  Approaches for Expanding Choice and Access to Long-Acting Reversible Contraceptives and Permanent Methods of Family Planning  USAID’s Pioneering Work with the Private Sector  Partnerships with the Private Sector in Health: What the International Community Can Do to Strengthen Health Systems in Developing Countries  Improving the Quality of Private Sector Delivery of Public Health Services: Challenges and Strategies  USAID/Philippines: Performance Evaluation of the Family Planning and Maternal Child Health Portfolio Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 66 ANNEX G: ANNOTATED BIBLIOGRAPHY OF SELECTED DOCUMENTS REVIEWED DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS RFTOP 492-13-000003 Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II (PRISM 2) Project This request for proposals to conduct the evaluation of PRISM2 is a seminal document. Among other specifics, the document contains four overall directives for conducting the Evaluation as well as a list of 10 Illustrative Questions to be answered during the course of the Evaluation. Social Impact’s Technical Proposal This well written proposal in response to RFTOP 492-13-000003 clearly outlines Social Impact’s approach to the PRISM2 Evaluation. Best Practices at Scale in the Home, Community and Facilities: An Action Plan for Maternal, Neonatal, Child Health and Nutrition October 2011 to September 2016 In relatively concise form, this document outlines the focus of USAID’s current health strategy. In doing so, it provides a great deal of useful background information and data as well as baseline data and overall targets to achieve by 2016. It also focuses in on areas for programmatic improvement and investment, such as BCC, strengthening public sector capacity, particularly within LGUs, supporting the equity in providing health care, private sector, coastal resource areas, and Autonomous Region of Muslim Mindanao (ARMM). Within budget limitations, estimated total investment budget for this year estimated at $21 million/year, more attention is given to specific areas than to broader health concerns. USAID Action Memorandum Health Assistance Strategy - Family Health Improved Health Assistance Strategy - Family Health Improved DAAD This internal document outlines the bilateral program between USAID and the Philippines and includes the Development Activity Approval Document (DAAD), which generally reiterates the 2011-16 BEST Strategy, USAID’s Action Plan for Maternal, Neonatal, Child Health and Nutrition. Updates are provided to reflect current conditions, e.g., administration change in the Philippines, shortfalls for meeting MDG 2015 targets in MCH, etc. While IR2 focus on the private sector remains central to the strategy, with the exception of implementation partners (DOLE, LGUs) and broad directions (increase and improve midwifery services), detailed direction for project-level implementation is not provided. Rather this gives a broad sense of USAID investments. Section C. -- Opportunities for the New USAID Program – gives considerable attention to the new government’s focus on the private sector as a “public-private partnerships as a cornerstone of its economic and social policy. This high-level view of the overall program envisioned for USAID is a useful starting place within which to situate PRISM 2. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 67 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS RESULTS FRAMEWORK Development Objective 1 – Intermediate Result 1.4 This update of indicators and targets under Intermediate Result 1.4 lays out the current parameters within which PRISM2 operates and will contribute towards overall USAID investments. There are some notable differences between some targets for some indicators (e.g., CPR, percentages of births attended; percentages of adolescent pregnancies) compared to official Philippine Development Plan, 2011-2016. This is the most recent articulation of the Results Framework and its major measures. Meeting Women’s Contraceptive Needs in the Philippines http://www.guttmacher.or g/pubs/2009/04/15/IB_M WCNP.pdf This review article summarizes relevant data from 2008 on contraceptive use and access to safe delivery in the Philippines. It provides compelling evidence of the need for improving FP services and analyzes some of costs and benefits to providing access to a full range of modern contraceptives. The article highlights the health benefits to women and families as well as the financial benefits to the country. Likhaan and the University of the Philippines worked with Guttmacher to produce this brief. Unintended Pregnancy and Induced Abortion in the Philippines http://www.guttmacher.or g/pubs/2006/08/08/Philippi nesUPIA.pdf This study, based on data collected between 2002 and 2005, makes a strong case both for the importance of increasing the use of effective methods of contraception and placing concerted effort on providing high quality post abortion care. Among some of the principal findings from this study are: “An estimated 473,000 abortions occur annually. One-third of women who experience an unintended pregnancy end it in abortion… Because the cost of relatively safe procedures performed by trained providers in hygienic settings (4,000–15,000 pesos, or US$73– 273) is often many times higher than that of unsafe and less effective methods (costing as little as US$1), poor women tend to use unsafe methods… Increased use of effective contraceptives would help women achieve their desired family size, and thus prevent unintended pregnancies, which, in turn, would reduce the need for abortion and the grave health consequences and costs of unsafe abortion… increased resources should be directed at improving the quality of postabortion care.” Fight for Reproductive Health Bill grows in the Philippines 10.1016/S0140- 6736(12)61162-3 This brief article, in the July 14 2012 issue of Lancet, updates progress on passage of the comprehensive and controversial RH Bill in the Philippines. This decade-long debate (somewhat resolved with recent approval of the Bill) will result in greater funding and more equitable access to modern contraceptive methods. Relatively high maternal mortality (MMR of 221 or 11 deaths per day) and high population growth rate (1.9%) provide compelling arguments in face of continuing religious-based opposition. Falling short of MDG 2015 targets is also giving impetus to passing this long-awaited Bill. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 68 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Statistics on Filipino women and men's health and family planning http://pcw.gov.ph/statistic s/201210/statistics￾filipino-women-and-mens￾health-and-family-planning This posting by the Philippine Commission on Women summarizes a decade of statistics on CPR and maternal mortality, including timeline graphics on MMR and CPR. It includes some data from the 2011 FHS and a striking graphic comparing past MMRs to the MDG 2015 target of 53. Fertility and Family Planning: 2011 Family Health Survey http://www.scribd.com/do c/98937655/Fertility-and￾Family-Planning-2011- Family-Health-Survey-for￾2011 This set of PowerPoint-type slides the 2011 Family Health Survey its major findings. Some graphics usefully place these latest data in the context of previous national surveys and other graphics illustrate regional and urban-rural variations. The data reveal some progress in recent years: TFR from 4.1 in 1991 to 3.1 in 2009 and CPR from 47.0 in 2001 to 48.9 in 2011. The report carries the logos of both DOH and USAID. PRISM2 website http://www.prism2.ph/we b/ The website seems useful for announcing and soliciting proposals for a wide variety of activities related to promoting public-private efforts in the Philippines. It provides some overview of activities in the various Regions. Except for the RFP announcements, it does not appear to be a particularly dynamic mechanism to inform about Project activities, developments and accomplishments. RH law implementing rules clarifies gender sensitive provision of health services http://pcw.gov.ph/article/r h-law-implementing-rules￾clarifies-gender-sensitive￾provision-health-services This web posting by the Philippine Commission on Women (tagline: The National Machinery for Gender Equality and Women's Empowerment) celebrates successful passage of the RH Law in December 2012. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 69 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Bangladesh Family Planning: Private Health Sector Assessment http://www.shopsproject. org/sites/default/files/reso urces/Bangladesh%20Fami ly%20Planning%20Private %20Health%20Sector%20 Assessment%20Brief.pdf This assessment present a synthesis, by contraceptive, of prevalence, drivers of demand and the risks and challenges to increased acceptability based on informant interviews and desk research Bangladesh Family Planning: LAPM http://www.shopsproject. org/sites/default/files/reso urces/Bangladesh_Private _Sector_Assessment_LA PM_FINALapvd%2020111 1143.pdf This assessment identifies challenges and opportunities on the demand and supply sides that could accelerate the reduction in TFR through a reprioritization of LAPMs, including injectables, and a refocusing of private sector engagement – including NGOs and especially for-profit health providers. Private Sector Mobilization for Family Planning Project Assessment This is the assessment of first PRISM Project, conducted 3 years before the EOP. Approaches for Expanding Choice and Access to Long-Acting Reversible Contraceptives and Permanent Methods of Family Planning http://transition.usaid.gov/ our_work/global_health/p op/techareas/repositionin g/repfp_ebulletin/index.ht ml This recent (March 2013) Action E-Bulletin from USAID’s Repositioning Family Planning website makes a strong case for expanding access to long-acting reversible contraceptives (LARCs) and permanent methods (PMs). This brief also provides cost data by method as compelling evidence for shifting method use in these directions: “LARCs are also highly cost-effective for programs, ranging from about $0.05 per year of use for the CuT 380A IUD, to $1.80 per year of use for Jadelle, and around $5.40 per year of use for Implanon, when used for their full number of years of effectiveness.” Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 70 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Corporate Social Responsibility http://www.scribd.com/do c/56147184/Corporate￾Social-Responsibility￾Adam-Lindgreen-and￾Valerie-Swaen This introduction to a 2010 a special issue of the International Journal of Management Reviews on corporate social responsibility (CSR) serves as both a useful overview and an update on theoretical explorations and practical applications. This introduction focuses on reviewing CSR literature within five areas: communications (or branding); implementation; stakeholder engagement; measurement; and the business case. It then introduces the articles that have been included in this issue: ‘Maximizing business returns to corporate social responsibility: the role of CSR communication’; ‘Organizational stages and cultural phases: a critical review and a consolidative model of CSR development’; ‘Stakeholder engagement, discourse ethics and strategic management’; ‘Measuring corporate social performance: a review’; and ‘The business case for CSR: a review of concepts, research and practice’. The introduction concludes by noting areas of interest to academics and practitioners for further research. USAID Philippine Health Strategy 2012-2016 This is what USAID prepared as the basis for a new cooperation agreement with the Government of the Philippines. The strategy presented is to be anchored on 3 main thrusts:  Improving supply of services - by helping to eliminate availability and quality gaps in the public sector and seeking opportunities to draw the private sector into an expanded primary care role.  Strengthening demand by encouraging adoption of appropriate healthy behaviors by addressing individuals through IPC/C and more mass media, community mobilization around FH, and more advocacy for FH by influential citizens.  Eliminating policy and systems barriers - particularly in the area of financing. The discussion is focused on how they will operationalize these thrusts into the different health elements like FP, MCNH, Tb and HIV/AIDS and includes targets, approaches and activities to be done. USAID’s Pioneering Work with the Private Sector S_Radloff_Opening_Rem arks_Final_Version This series of PowerPoint slides, from Scott Radloff, traces the history of USAID’s involvement in and support for FP in the private sector. This presentation and summary dates from PSP1, which preceded by some years the current private sector project -- Strengthening Health Outcomes through the Private Sector (SHOPS) run by Abt Associates. These slides usefully illustrate the various types of support that USAID has provided over the years since the 1980s in such areas as different generations of social marketing models; work-based programs; social franchise models; public-private partnerships; pharmaceutical partnerships; health financing; and more recently corporate social responsibility and base of the pyramid models. Short on details, this presentation shows that international investment by USAID/DC has been paralleled in the Philippines by a similarly long and broad history of Mission investment in private sector approaches. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 71 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Partnerships with the Private Sector in Health What the International Community Can Do to Strengthen Health Systems in Developing Countries April Harding, Chair November 2009 This Final Report of the Private Sector Advisory Facility Working Group of the Center for Global Development contains among other things a useful table summarizing the various approaches that have been used in a number of settings by governments engaged in promoting private￾public partnerships. These approaches range from contracting out and accreditation to vouchers and insurance. This table (pp. 4-5) provides a useful backdrop for looking at the various PRISM2 supported efforts and other opportunities underway in the Philippines. To address the critical question of “whether there was need and demand for Support”, the Working Group conducted structured interviews (36), including some with potential clients. This appears to be a formational document to seek funding for additional work, with a focus on Africa. Nevertheless, the overview it provides is useful. In addition, it has an extensive bibliography which includes a number of articles on private sector involvement to achieve public health outcomes. International Finance Institutions and Development through the Private Sector www.edfi.be/component/ downloads/.../54.htm This broad overview of various ways in which international finance institutions, such as the Asia Development Bank and many others, have invested in the private sector for positive development outcomes is notable for the omission of health from among its many case studies. Extending Service Delivery – A Global Reproductive Health and Family Planning Project http://www.esdproj.org/si te/PageServer?pagename= Homepag This centrally funded project, managed by Pathfinder International, involves collecting and sharing information and knowledge drawn from FP/RH in countries across the world. This may be a model for subsequent efforts more directly focused on public-private partnerships. The current project gives some attention to working within the area of Corporate Social Responsibility. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 72 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Improving the Quality of Private Sector Delivery of Public Health Services: Challenges and Strategies Documents/Added Documents/ Improving the quality of private sector delivery 1998 This article from Health Policy and Planning, though 15 years old, provides a quite useful and balanced perspective on practices and prospects of private practitioners providing services to meet public health needs. Using evidence drawn from a number of studies it illuminates various factors that operate at different levels to affect private practitioners performances. Recognizing the need to improve service quality, it goes beyond providing information and training to suggest that interventions must be more fully cognizant of the situation within which private providers work, how macro-factors intersect with their own knowledge and attitudes as well as with the needs and expectations of their clients. Figure 2 on p. 114 provides a useful quality improvement framework relating policy/regulatory improvements in relation to provider as well as client community interventions. Interventions should be perceived and undertaken within this broader multi-faceted context – as the authors state “dissemination of best practices is necessary but not sufficient”. This article provides a concise overview of the challenges faced in improving the capacities of front-line private sector providers. TIPPP on increasing FP & MCH contributions for the PPM Designed to enable DOH to support local health officials manage professional midwives practice a members of the PHO/CHO/MHO-led service delivery network for FP=MCH info, products and services. Provides overview of rational for using these service providers to reach MDG and recommendations to all levels to embrace PPM into the public sector This TIPPP has 3 modules with 8 technical assistance packages (TAPS) or packages. Modules: 1) Stewardship of professional practice of midwives (PPM), 2: Quality assurance of PPM; 3) Improving the business of PPM Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 73 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS TIPPP on improving local M&E for FP & MCH Need to merge public & private sector data described and role of PRISM2 in the national FHSIS standards and guidelines. Various recommendations for CHD cited in which they formulate, assist and oversee M&E improvement plan in public and private sectors and 1. National epidemiology center (NEC) should mandate DOH oversight of statistic reporting and monitoring 2. NEC should establish ground rules for universal service delivery reporting, including private sector 3. NEC should develop a feedback mechanism so the monitoring data is useful for local implementers PHO/CHO or MLO should adapt locally relevant measures to improve accuracy of data collection. 3 modules with 6 packages: 1) Stewardship of M&E on MCH; 2) Public sector compliance with standards in M&E for FP/MCH; 3) Private sector compliance with standards on M&E for FP/MCH TIPPP of training service providers for FP/MCH Overview of local need for trained SBA and the DOH-sponsored courses 1. Provides recommendation to DOH, including Provide continuums availability of courses, Include private sector, Maximize benefits of raining and Enable CHD and PHO to cope with needs for training 2. Recommendation to CHD to work with PRISM to build PPPs for training, including develop plan, maintain training data base 3. Identify private-sector partners, conduct TOT, provide support to PHO and CHO in conduct of courses 4. Recommendations on what PHO, CHO or MHO should do for implementing the training system, including develop plan, support attendance to ToT, conduct FP training & provide follow-up 5. Recommendations for local PPP, including organize a body to coordinate public & private participation in the FP training. 3 modules developed with 7 technical assistance packages (TAP) available: 1) Stewardship; 2) Training trainers; 3) Training service providers Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 74 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS TIPPP on BCC to improve utilization of FP & MCH products & services This TIPPP supports & strengthens DOH, DOLE and LGU to mobilize the private sector to improve the quality, availability and usefulness of FP & MCH IEC approaches for various target populations, including policy makers, health care providers and the public. 4 modules 1. Stewardship of FP-MCH information Provision 2. Creation of awareness, positive attitudes & opinion climate about FP/MCH 3. Improve interpersonal communication through FP/MCH messages and more accessible styles of communication 4. Improve FP-MCH counseling (use of buddy system) TIPPP on securing contraceptive supply Opens with good intro of why securing local access to contraceptives is critical: total potential market for modern FP could be >73% of MWRA; CPR 34% OC & injectables each contributing 15.7 and 2.6% in mix. [DHS 2008]. CPR not changed in 5 years so is critical CHD 1. Generate demand for contraceptives 2. Facilitate supply in public sector 3. Facilitate commercial sales 4. Increase availability, accessibility and affordability of FP products Operational details of the TIPPP on LA/PM This draft document details how PRISM2 is to assist the CHD/LGU Health Offices to identify and develop LA/PM Accessibility Facilitators. Three modules: 1) Increase support to make LA/PM services accessible & affordable; 2) Increase the number & improve deployment of LA/PM providers ay more local market areas; 3) Improve case loads of the providers of LA/PM. Proposes a sustainability spectrum. TIPPP on expanding hospital based FP-MCH services Of the 1,795 hospitals in 2009, public = 721 and 1,074 were private, both sectors, lost opportunities in service provision and FP acceptance. PRISM2 recommended Region-wide training and the establishment of regional technical teams (RTT) for hospitals who then implement the regional program in other participating hospitals. Hospitals see their role as a clinical (treatment) rather than preventative or public health and some CHD staff acknowledged a lost opportunity by not including them in the PHC effort. Recommendation include that PHO/CHO/ MHO enroll their hospital in the program and at the PPP level to create an environment that welcomes & integrates the hospitals into the service delivery network. Multi-region meetings were in CHD IX, CHD XI, CHD XII & CHD Caraga and resulted in TASP for 1) RTT for hospitals; 2) Capacity building for hospital-based FH-MCH services (stage 1) ; 3) Regular hospital FP-MCH services (stage 2) and hospitals as Hubs of local service delivery (stage 3) Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 75 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS USAID/Philippines: Performance Evaluation of the Family Planning and Maternal Child Health Portfolio This overall evaluation of the USAID/P portfolio was completed in December 2012. It compiles, pp.4-7, recent trend data in FP/MCH and the challenging context over the last 40 years. This evaluation included interventions throughout the country undertaken within 5 major projects. The Report however has little in any depth to report on PRISM2, which is mentioned only two or three times. Annex F contains a summary of field trips and is an interesting rich data source. Again, however there is no or little attribution to PRISM2, though some other projects (SHIELD, HealthPro, HealthGov) do receive some mention. The Executive Summary shows scant difference for CPR between provinces that received substantial USAID support and those that did not. There were however some gains in terms of deliveries at facilities and by skilled birth attendants. Audit of USAID/Philippines’ Private Sector Mobilization for Family Health Project, Phase II This recent audit of USAID/P and PRISM2 (completed March 2013) summarizes findings as follow: “engages the private sector through the public sector rather than working with them directly. As of January 6, 2013, the mission had obligated $23.3 million and disbursed $18.9 million for the project…. policy support contributed to an increase in the use of modern contraceptive methods1 provided by the private sector from 40.8 percent in 2006 to 53.8 percent in 2011 (Philippine National Statistics Office, 2006 and 2011 Family Planning Surveys)… the net effect on modern contraceptive use has been negligible. The proportion of married women using modern contraceptives, whether from a public or private sector source, increased only 1 percentage point—from 35.9 percent in 2006 to 36.9 percent in 2011. Moreover, the unmet need for family planning in the same group of women increased from 15.7 percent to 19.3 percent during the same period. The audit disclosed the following problems:  The monitoring and evaluation (M&E) plan and data were weak…  Gaps in service delivery indicate clusters of stakeholders without the support needed to develop the sustainable family planning and maternal and child health services envisioned by the project…  The mission’s slow response hindered progress. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 76 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS ACCESS Final Report of Alphamed The Report contains results of the 3-year sub-grant of Alphamed with PRISM2. It has CYP totals for methods sold, but does not calculate cost per CYP. Eliminating Bead sales, a NFP method, and its high CYP factor of 1.5 years, the data for other methods can easily be converted to cost/CYP. This will show a comparatively strong cost-effectiveness for this component of the Project On CYP: “The efforts of its hardworking and dedicated sales force allowed ACCESS to contribute a total CYP of 202,321 to the intended beneficiaries in the project areas. The top three commodities with the highest CYP equivalent were Famila pills (CYP=73,464), followed by SDM Cycle Beads (CYP=61,020), and Depogestin 3-month injectable (CYP=43,537). Overall, these CYP averted unplanned pregnancies and helped couples attain their reproductive intentions. Table 21 next page shows the complete CYP calculations for the reported sales volume during the grant period.” For more on CYP see Table 21 on p. 25. For ADP data see Table 24 page 32. MEASURE Evaluation Population and Reproductive Health Project http://www.cpc.unc.edu/m easure/prh/rh_indicators/ specific/fp/cyp This website definition for Couple-years of protection (CYP) contains conversion factors for all methods including natural family planning, here named Standard Days Method. It provides the value of 1.5 CYP per trained adopter of this method. This conversion is based on research; notable however is the use of “trained adopter” in the definition. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 77 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Health and Family Planning Indicators: A Tool for Results Frameworks Volume I http://sara.aed.org/publica tions/cross_cutting/indica tors/html/indicators1.htm This is a 1999 publication from AED and supports work of the Africa Bureau. “Discussion: CYP may serve as a lower-level proxy indicator to track progress when CPR is not available. Missions are cautioned not to convert CYP data … Estimates of couple years of protection (CYP) based on family planning commodities distributed and/or services provided can typically be calculated on an annual basis at low cost, providing useful trend information for the years between population￾based surveys. However, CYP data are less reliable than contraceptive prevalence rates obtained through surveys because the amount of contraceptives distributed in a given time period does not necessarily correspond to the quantity actually used by clients during the same time period. Often contraceptives are distributed nationwide or through sales networks well in advance of their actual use by consumers. Furthermore, supplies may be damaged or destroyed in transit or storage or may be diverted to markets outside of the area where the target population resides. CYP figures derived from service delivery or consumer sales data are more timely and relevant to current use by the target population than are figures based on national importation or distribution of contraceptives. In either case, CYP cannot substitute for CPR as an overall measure of program performance and should not be converted to or reported as CPR. Missions considering the use of CYP data should review the source, quality, and completeness of the program data used to calculate.” What is the Impact of Contraceptive Methods and Mixes of Contraceptive Methods on Contraceptive Prevalence, Unmet Need for FP and Unwanted and Unintended Pregnancies? An Overview of Systematic Reviews. http://r4d.dfid.gov.uk/PDF /Outputs/SystematicRevie ws/FINAL-Q35- Contraceptive-Mix￾Overview-of-Reviews￾Protocol-DFID￾Portsmouth.pdf This review article, commissioned by DFID, summarizes the previous literature on what and how a mix of contraceptive methods contributes to CPR increases. Among other studies, it cites a 1989 study by Anrud Jain (Fertility reduction and the quality of family planning services. Studies in Family Planning 20(1), 1–16) of the Population Council and summarizes its conclusion as having “estimated that the widespread addition of one method to options available in a country would be associated with an increase of 12% in contraceptive prevalence.” The review contains a useful conceptual framework relating various factors, such as policies and provider bias, to the contraceptives ultimately available and used by consumers. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 78 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS CHDs as Stewards of FP￾MCH Outcomes in their Regions: Notes on PRISM2 Support (hard copy only) This is a seminal part of the history of PRISM2 and what it had hoped to accomplish, particularly with the CHDs. Dated May 15, 2011, this “note” apparently never became an official document of PRISM2. This is unfortunate. It contains the thinking and hopes that were at the heart of the Project. It provides a very interesting roadmap for what could have been. It may well be that such a devolved, decentralized approach to the public sector as steward for private sector involvement was fundamentally inimical within the constraints of a contract. Talisay Event This .ppt was presented at the launch event in Baranguay Matab-and in Talisay City, Negro Oriental. It provides some health statistics on the hacienda population collected by the FWATS. It summarizes the training undertaken by the FWATS in basic FP, client referral, CHT training, and IEC distribution and orientation and presents the health resources available to community members including PhilHealth and clinics. USAID Inter-CA Meeting: Discussion on SDN and Project Updates This .ppt is from CDH meeting in Tagbilaran City, Bohol on October 18, 2012 it was provided to the team by a SAM and illustrates the type of sub-national capacity building P2 undertook to establish SDNs. It presented updates from USAID-funded projects (BALANCED, Healthgov.HPDP2 and P2) and the handover from USAID efforts with HealthGov to P2. USAID Approach to Monitoring Capacity Building: Experiences, lessons learned, and best practices In this .ppt, Duane Muller of USAID presents the agency’s thoughts on monitoring CB. The .ppt was shown at a UNFCCC Experts Meeting on Capacity Building in 2007 in St John’s, Antigua. It is not health specific but provides an overview of the Paris declaration in light of the differences between monitoring and evaluation, characteristics of indicators and performance evaluations. Presentation by P2 to the Evaluation team This .ppt presented to the Evaluation team served as the orientation to the project. it presented P2 framework, objectives, focal areas and statistics to date Excerpts of Successful Attainment of Objectives from ILCI report This .ppt presents the accomplishments of the sub-grantee ILCI in CHD-7. The number of MOAs, sustainability plans created, certificates of recognition for birthing clinics as members of the SDN and presented success stories of PPM working with the public sector. PPM & Birthing House Monitoring / Tracking Chart – ILCI This file was provided to the team by ILCI and illustrates the tools developed and used by a P2 sub-grantee to identify and track accreditation Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 79 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Technical Initiative for PPP on Increasing Family Planning and Maternal and Child Health Contribution of the Professional Practice of Midwives – unpublished This document illustrates the internal review process within P2 for TIPPPs and the identification of potential TAPs FP CBT I participant rosters one from 23 – 27 July in Angeles City and one from 25 Feb – 1 March, possibly also Angeles City This document illustrates the variety of midwives who attended this P2 training events, that is midwife, Nurse, Midwife/Owner and Nurse/Owner as that there were participants from the public and private settings. FP CBT II participant rosters one from 25 – 27 March, in Angeles City This document illustrates the variety of midwives who attended this P2 training event, that is midwife and nurse – all from private birthing homes. USAID’s Pioneering Work with the Private Health Sector This undated .ppt of Scott Radloff presents an overview of USAID’s long involvement with support of the private sector in FP – including the agency’s rationale, approaches, private-sector financial contribution to health (Africa), access of population to private for profit health resources (Africa) Capacity Development: Practice Note - UNDP This document is to provide UNDP staff and other development practitioners to UNDP’s approach to supporting capacity development. It distinguishes between technical and functional capacities UNDP’s processes to support local effort to improve the capacity to help achieve the MDGs. It does not detail M&E of building capacity Section C of the Task order No 04 for PRISM2 This document is the SOW for PRISM2 Reproductive Health Law No 10354 On 23 July 2012, the GOP approved Act Number 10354 which provides a national policy of responsible parenthood and reproductive health. PRISM2 presentation to USAID - undated This .ppt from P2’s initial presentation to the USAID Mission reflects some health statistics, perhaps to serve as a baseline, and lists P2 program rational, e.g., ‘PRISM2 work at LMAs should focus on increasing frequency and quality of provider-client contacts around FP’ PRISM2 presentation to USAID on 14 January 2010 Ann provided the Evaluation team with this .ppt from P2’s initial presentation to the Mission on the day of the debrief. P2 provided USAID with an overview of the project and rational for program direction. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 80 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Notes of P2 support on CHDs as Stewards of FP￾MCH Outcomes in their regions on 15 May 2011 This document presents “the rationale, approach and operational directions guiding PRISM2 support to 16 CHDs and DOH-ARMM in performing their roles as stewards of FP-MCH outcomes in their respective regions.” A Guide to Monitoring and valuation of Capacity￾Building Interventions in the Health Sector in Developing Countries. MEASURE Evaluation Manual Series No 7, March 2003 Although already 10 years old, this is a good resource for program managers developing a CB component to a health program. It presents variables, indicators, methodological approaches, sources of data. The bibliography is also useful for further reading. This is one of the documents to be shared with the Mission as they strive to establish a viable M&E approach for CB. Improving the quality of private sector delivery of public health services: challenges and strategies. Health Policy and Planning, 13(2) 107-120, Oxford Press 1998 This piece discusses the need for involvement of policy makers and private practioners in regulating the quality of services rendered in the private sector – private practioners may provide inappropriate treatments and their patients may need education on appropriate treatment-seeking and treatment-taking behavior. DOH Order No. 2011 – 0188 – Kalusugan Pangkalahatan Execution Plan and Implementation Arrangements, Aug 02, 2011 This document outlines the CHD responsibilities, resources and targets in implementing activities to attain MDG targets. Partnership with the Private Sector in Health, Center for Global Development. Dec 2009 The Private Sector Advisory Facility Working Group recommended the creation of a global advisory group to address the need for technical assistance on how to engage the private health sector and that it be based within the World Bank– International Finance Corporation (IFC). IFC is now the world’s largest multilateral investor in the private health care and education sectors in emerging markets and collaborates with the Johns Hopkins Medicine International. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 81 DOCUMENT TITLE AND LOCUS SYNTHESIS/KEY POINTS Organizational Capacity Building Framework: a Foundation for Stronger, more Sustainable HIV/AIDS Programs, Organizations & Networks. AIDSTAR Technical Brief, No 2 Jan 2011 Although the original audience are managers of AIDS programs, the framework presented is useful for those working in FP. It presents definitions and indicators. Capacity Development Results Framework. World Bank 2009 http://siteresources.world bank.org/EXTCDRC/Res ources/CDRF_Paper.pdf The Framework articulates the process to the planning, implementation, and evaluation of projects and programs designed to build capacity for development at a national or sub-national level. It discusses monitoring and assessing achievements. DOH Administrative Order No 2012-0009 This DOH document authorized the “National Strategy towards Reducing Unmet Need for Modern FP as a means to Achieving MGDs on Maternal Health” Regional Inspector General AUDIT OF USAID/PHILIPPINES’ PRIVATE SECTOR MOBILIZATION FOR FAMILY HEALTH PROJECT, PHASE II. AUDIT REPORT NO. 5- 492-13-005-P MARCH 25, 2013 This report contains nine recommendations to help the mission improve the efficiency and effectiveness of PRISM2. Key RIG findings:  Monitoring and Evaluation Plan and Data Were Weak  Insufficient Training Outreach Limited the Project’s Impact  Project Disbursements Exceeded Approved Limit Delayed USAID Response Hindered Project’s Progress Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 82 ANNEX H: OVERARCHING QUESTIONS AND GENERIC QUESTIONNAIRE LEGEND: SOW DIRECTIVES ITALICIZED SOW QUESTIONS in BOLD Questions for respondents plain BACKGROUND QUESTIONS: (select one or two of the Background Questions to start most interviews) BG-1 In general, how do you think that the private sector can be more fully engaged to meet FP/MCH service needs? BG-2 Please describe a typical work day for you. a. What kinds of things do you do on a fairly regular basis, including those apart from PRISM2? b. What kinds of things do you do on a fairly regular basis in relation to PRISM2? BG-3 What in your previous professional experiences most prepared you for implementing, supporting, or working with this Project? BG-4 How did you come to this point in your career, significant responsibilities over/within … (e.g., a project that seeks to advance PPP)? BG-5 Please describe your position within or your relationship to the PRISM 2 Project? BG-6 What readings or documents have you come across which have best informed and stimulated your thinking about how population, health and the private sector may be interlinked or interactive? BG-7 Please provide an overview of the PRISM2 Project and your experiences, both positive and negative, working with this Project Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 83 I. Determine the effectiveness of the project, by investigating its achievement (in comparison with baselines and targets) of the following:  Outcomes – (a) increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources; (b) increased share of deliveries attended by skilled birth attendants; and (c) increased contraceptive market for pills, injectables and LA/PM  Outputs – various in the areas of (a) service delivery expansion and utilization; (b)behavior change communication; (c) policy and systems strengthening; and (d) capacity strengthening of GPH national and regional health agencies (and related ones), and local governments to engage the private sector in FP & MCH service delivery Effectiveness I-1 What is the project’s contribution to improvement in national FP and MCH indicators?  Increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources  Increased share of deliveries attended by skilled birth attendants  Increased contraceptive market for pills, injectables and LA/PM I-1-1 What progress has PRISM2 made toward expanding quality services through private and commercial providers to increase the following IR 2 indicators?  Increased Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources  Increased share of deliveries attended by skilled birth attendants  Increased contraceptive market for pills, injectables and LA/PM I-1-2 In your opinion is PRISM 2 quantitatively on target or has it fallen short in making contributions to the following: private sector CPR; percentage of deliveries attended by skilled birth attendants; increasing the market for such contraceptives as pills, injectables, and long-acting/permanent methods? I-1-3 From now until the end of Prism2 (2014), what, where and how should this Project continue to make progress for advancing FP/MCH in the private sector and/or where/how can it make up ground in the time remaining? I-1-4 Has the impact that PRISM2 has had on FP/MNCHN outcomes in the Philippines been high, moderate, or minimal? Please explain and provide evidence for your answer. I-1-5 What have been the factors (facilitating and/or hindering factors) that provided the PRISM2 data results? I-1-6 What can be done by PRISM2 to improve FP/MCH performance? How can PRISM2 activities be implemented more effectively? I-1-7 How can interactions among the PPP stakeholders at the sub-national level (be specific) be improved? I-2 What are the key outputs and/or outcomes in the PRISM components, and which ones made the most contribution to project success?  Service delivery expansion and utilization  Social marketing and behavior change communication  Policy and systems strengthening Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 84 I-2-1 Broadly speaking, what do you hope and expect PRISM2 to achieve through private sector service delivery, social marketing, behavior change communication, and/or policy/systems change? I-2-2 Within private sector service delivery, social marketing, behavior change communication, and/or policy/systems change, where has PRISM2 contributed the most? In which areas have PRISM2 contributions to these components been less successful or less than anticipated? I-2-3 Please provide such evidence, quantitative or qualitative, as are available for tracking or documenting PRISM2 achievements in the following outputs and outcomes:  Service delivery expansion and utilization  Social marketing and behavior change communication  Policy and systems strengthening I-2-4 In your opinion, in which of the following areas has PRISM2 been the most and least successful? Do you have data, qualitative or quantitative, to support this?  Service delivery expansion and utilization  Social marketing and behavior change communication  Policy and systems strengthening I-2-5 What do you consider to be the most significant gains in the key project components: service delivery expansion; social marketing and behavior change communication to increase utilization of FP and MNCHN services; and policy and systems support? In which areas does PRISM2 need to achieve more? I-2-6 Please describe your role and/or your understanding of how the PPM indicators were developed? a. How would you describe the indicators that are now tracked: input, output, outcome, or other? b. What other indicators could one envision as important to track over the long term to measure achievements of a project like PRISM2? I-2-7 What was the one best practice among the PPP interventions that has produced successful results? I-2-8 What are the remaining gaps in PPP strengthening? II Assess the effectiveness of PRISM2 design as well as implementation (management, operation and monitoring systems), in relation to achievement of contract deliverables and relationships with key clients from the government and private sectors II-3 How effectively has PRISM2 strengthened the capacities to engage the private sector in FP & MCH service delivery? (Note: The evaluation team may refer to specific indicators identified in several PRISM 2 documents related to “Technical Initiatives for Public-Private Partnerships” [TIPPPs] to measure enhanced capacity in the TA packages (Ensuring contraceptive supply; Information provision; Engaging private midwives; Expanding FP services in hospitals; Increasing availability and accessibility of LA/PM services; Tapping and financing resources for health; Monitoring and evaluation of FP/Maternal, Neonatal and Child Health and Nutrition (MNCHN) programs; Institutionalizing FP/MNCHN training; and Other TA II-3-1 PRISM2 seeks to strengthen the capacities of organizations such as DOH, DOLE, CHDs, and LGUs to promote and make progress for FP/MCH in the private sector. a. How does it do this? b. What has its impact been to date? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 85 c. What challenges has it faced? II-3-2 Describe and weight the mix of activities (training, technical assistance, operations research, etc.) through which PRISM2 seeks to strengthen and institutionalize LGU capacities for private-public partnership. a. What has its impact been to date? b. What challenges has it faced? II-3-3 Have there been changes in PPP capacity within provinces, regions and particular LGUs, since 2009 when PRISM2 got under way? a. To what extent can any change be directly attributed to PRISM2 b. What was PRISM2’s contribution and how did this come about?? II-3-4 From your perspective, which strategy approaches and activities of PRISM2 have been the most effective? And which PRISM2 strategy, approaches, and activities have been less effective? II-3-5 Part of PRISM2 places emphasis on developing and supporting policy for PPP. a. To what extent can the emergence of recent favorable policy changes for PPP be directly attributed to PRISM2? b. What were PRISM2’s specific contributions? c. Should/could PRISM2have done more to create and support PPP policy? II-3-a What factors and conditions significantly contributed or hindered the achievements of outputs and outcomes in questions I-2 and I-3? II-3-a-1 In which areas has PRISM2 achieved the most and where have there been significant shortcomings? To what do you attribute such success or lack of success? II-3-a-2 During the last twelve months, how often and for what purposes have you as a PRISM2 staff member or consultant provided technical assistance or support of some kind? a. Focusing on what you would consider the most successful occasion when you worked to develop capacity of a partner organization, please provide details. b. Concentrating on a capacity building assignment that was not optimally successful, please analyze what in retrospect could have been done differently? II-3-a-3 Have you personally and/or your organization as a partner of PRISM2 received support from this Project to strengthen the abilities of your organization to engage with and activate private sector channels for FP/MCH? Please describe this relationship and interactions with PRISM2. What positive and/or less than optimal; outcomes resulted? II-4 How did PRISM2 specifically address the following?  Development/strengthening of local markets for FP/ Maternal, Neonatal and Child Health and Nutrition (MNCHN) products and services  Responsiveness (inclusive of beneficiary targeting) to FP/MNCHN needs of government (DOH, DOLE, PhilHealth, CHDs, provincial/city/municipal health offices, provincial/municipal LGUs) as well as private sector partners and specific client groups II-4-1 Please describe how decisions are/were made for allocating PRISM2 resources, financial and human, to develop/strengthen local markets for FP/MNCHN products and services? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 86 II-4-2 In terms of developing and strengthening local markets for FP/MNCHN, where and when has PRISM2 achieved significant success? Where and when did PRISM2 not produce the results anticipated? II-4-3 Please describe how decisions are/were made for allocating PRISM2 resources, financial and human, to identify and meet capacity development needs of government partner organizations, such as (DOH, DOLE, PhilHealth, CHDs, provincial/city/municipal health offices, provincial/municipal LGUs, other partners in the private sector, and client groups. II-4-4 Among potential government partner organizations, which have been the most and least responsive to working with PRISM2 and the support it could provide? Why? II-4-5 What resources has PRISM2 leveraged for FP/MNCHN from in-country public and private institutions? Will these continue after the Project ends? II-4-5-a What resources has your institution (public or private) provided for FP/MNCHN in conjunction with working with PRISM2? Would this have happened if PRISM2 were not involved? Will these funds continue after the Project ends? II-4-6 What opportunities are there to advocate for and leverage funding from different sources and at different levels? SERVICE PROVIDERS at facilities: II-4-7 a. where do you refer your clients interested in methods not available in this facility? How do you ensure that the client accesses services? Is there a functional referral system? b. How do you promote FP and recruit clients? c. Did you receive any follow-up after you were trained? II-5 What are the respective plans of the DOH and local governments in order to sustain the project-developed systems and interventions? Which initiatives are likely to continue and which ones will not be sustained once PRISM2 ends? (Cite reason/s.) II-5-1 Which PRISM2 approaches and activities promote country ownership and long-term sustainability of development efforts? Please provide specific examples of where and how this has worked well. Have there also been occasions where such sustainable development efforts were less successful? II-5-2 In what ways and with which partners has PRISM2 worked to generate, sustain, and increase local funding for funding for FP/MNCHN? Please illustrate with specific examples. II-5-3 What PRISM2 processes or approaches have been used successfully with which partners to ensure sustainability of systems and/or interventions after the project life? What approaches and with which partners have been less successful? II-5-4 Which systems or services that were developed in conjunction with PRISM2 does the DOH intend to continue and sustain after the Project ends? What plans has the DOH put in place to do so? II-5-5 Which systems or services developed in conjunction with PRISM2 does this LGU intend to continue and sustain after the Project ends? What plans has the LGU put in place to do so? II-5-6 Does this LGU have increased financing for FP/MNCHN as a result of support received from PRISM2? Does this LGU now engage private sector as institutional long-term partners for health? II-5-7 Has PRISM2 worked in concert with PhilHealth and to what concrete results? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 87 II-5-8 What is the practice and potential of PhilHealth for making access to FP/MCH more equitable and affordable? II-6 Describe briefly PRISM2’s management mechanisms (i.e., headquarter oversight and involvement, organizational structure, field level operational set-up, personnel complement and their skill set, and the short-term technical assistance), and operational and monitoring mechanisms. How appropriate and effective were they, and how did they influence project performance and client satisfaction? II-6-1 Please describe the overall PRISM2 strategy and design. Is it sound? What, if any, are its deficiencies? What changes would you suggest for the future? II-6-2 In your opinion has PRISM2 program management effectively carried out its tasks? How appropriate is PRISM’s organizational structure and its set-up for field level operations? II-6-3 How appropriate and effective is the overall PRISM2 program design, management, operational and monitoring mechanisms? How have these or other factors affected PRISM2 performance? II-6-4 In your opinion did the PRISM2 personnel or consultants with whom you have worked have sufficient and appropriate skills to do the task needed? Please provide examples of when this has been useful and complementary and when it has not. II-6-5 How often and with what usefulness does PRISM2 headquarter provide oversight and/or get involved with management and operations. Please illustrate with specific examples of when this has been useful and not useful. II-6-6 How appropriate and useful is the monitoring system, and related indicators, for the wide range of inputs and outputs that PRISM2 provides? For instance, how does PRISM2 measure progress within such areas as capacity-building of government partners, improving the quality of services? II-6-7 What IT products has PRISM2 developed and how has it facilitated achievement of project deliverables? Will any of these products be useful and used after PRISM2 ends? II-6-8 From your perspective, has PRISM2 utilized its grants, subcontracts and other mechanisms effectively to achieve project objectives and how has it done so? II-6-8-a What contributions/expectations does your part of the management structure or project have to make to supporting PRISM2 and how do you go about doing this? II-6-9 What are the high and low points of working with or within PRISM2 since this Project started, in 2010? Please provide concrete examples of each? Were lessons learned from these experiences, and (if so) how were they shared with others? II-6-10 What during the last year or over the course of PRISM2are you personally most proud of having done with or within PRISM2? What do you have regrets about having done or not done that would have contributed to PRISM2? II-6-11 What technical and managerial support does PRISM2 receive on a regular basis from USAID? Does PRISM2 need more or less of this support? a. Please give specific examples of where support provided by USAID has been most helpful to achieving the PRISM2project objective and IRs. b. Where has USAID’s assistance and support fallen short or worked against PRISM2 being effective? II-6-12 Did your organization join the project in YR1? If not why? Is the role of your organization on PRISM2 clear? Do you have the resources to be successful? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 88 II-6-13 For Phil Health. sub-national DOH or PPM a. What could PRISM2 do to help achieve increased CPR? b. How could PRISM2 interventions and activities be more effectively implemented? c. What are your needs to increase CPR? III Identify PRISM2’s key contributions (e.g., unique achievements, innovative methods/approaches), and compare with other effective approaches of private sector involvement in FP and MCH. III-8. What PRISM2 interventions can be considered good practices (e.g., unique achievements, innovative methods/approaches), and which may be recommended for scaling-up in the future? How cost-effective are these? III-8-1 Cite specific examples of good practices, useful approaches, and/or tools that PRISM2 has adapted or used: a. Which do you identify as the most useful and likely to be most widely used? Where have these been described? b. How often and where have these actually been applied within field activities directly supported by PRISM2? c. What other best practices is PRISM2 learning about that it intends to apply? III-8-2 Cite specific examples of practices, useful approaches, and/or tools that PRISM2: has adapted or used: a. Has used but without great success to date b. Has used and seems to have great potential c. Has not used but should be considered for use in the future III-8-3 How does PRISM2 currently build capacity within organizations – through subcontracts, technical assistance, training, staff in residence, and other approaches? Please provide examples where this has proceeded well and not so well. III-8-4 What are PRISM2’s unique contributions or special capabilities for advancing PPP? III-9. Are there more effective approaches with the private sector which could have been explored to achieve FP/MNCHN outcomes? III-9-1 Within the Philippines what promising practices, networks, or models that may have been missed opportunities for PRISM? III-9-2 What are other countries doing to effectively engage the private sector that could be applied in the Philippines towards contributing to FP/MNCHN outcomes? III-9-2 Beyond PRISM2, what other promising approaches are underway or have potential in the Philippines to engage the private sector and involve the public sector in promoting FP/MCH? III-9-3 What is the potential and promise of PhilHealth for increasing access and improving equity of FP/MNCHN services? III-9-4 What has been the experience with franchised clinic networks in the Philippines and does this have potential for the future? IV Assess how PRISM2 collaborated with other projects in the USAID/Philippines Health portfolio, as well as complemented the strategy and priorities of the United States Government (USG) and GPH DOH. Compare PRISM2 collaboration efforts with lessons learned in this aspect from PRISM1. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 89 IV-7. How did PRISM2 complement (a) other USAID/Philippines Health projects; (b) United States Government’s “Best Practices at Scale in the Home, Community and Facilities (BEST): An Action Plan for Neonatal, Maternal , Child Health, Nutrition and Family Planning”; and (c) DOH’s MNCHN strategy and Universal Health Care agenda? IV-7-1 What do you expect PRISM2 to provide in terms of leadership, innovation, and integration in its efforts to collaborate with other projects in the USAID/Philippines Health portfolio? a. What are the collaborative strengths to date? b. What collaboration weaknesses or shortcomings are occurring and how may these be overcome? IV-7-2 At the national, regional, and/or local levels to what extent and how has PRISM2 collaborated with other USAID partners? a. With which organizations has collaboration been most effective and/or groundbreaking? b. Have there been missed opportunities for collaboration or areas where more needs to be done? c. In what ways has PRISM2 been pro-active in helping USAID global partners strengthen their own PPP approaches and programs? IV-7-3 What types of technical and other inputs does PRISM2provide to other USAID projects and organizations with which it collaborates? a. Please provide examples of specific inputs provided by PRISM2 and assess their relevance and technical soundness. b. Are there also some examples where PRISM2 assistance has fallen short of increasing the capacity of an organization? c. Are there examples of collaborations that did not develop or continue because what PRISM2 could provide did not or was not perceived to match what was needed? Please elaborate. IV-7-4 With which localities has PRISM2 enjoyed the most success during the last three years? a. Which in-country partners have achieved the greatest successes or have the most potential? b. Have some anticipated or unanticipated country-level partners experienced an expanded or diminished role during the course of PRISM2? c. Where and with which organizations do you wish that PRISM2had been able to do more? IV-7-5 What were the viable contributions made by your project to help PRISM in its implementation and achieve its goals? IV-10 How have gender considerations been integrated in USAID’s PRISM2 project? What are the effects of the project on male and female beneficiaries? IV-10-1 What gender issues were most pronounced in PRISM2 areas of operations? How have these been addressed/integrated in PRISM2 activities? IV-10-2 How was the PRISM2 Gender Action Plan developed and how recently has it been updated? IV-10-3 To what extent has the PRISM2 Gender Action Plan been implemented? Are there gaps in the project design and/or implementation that would have improved gender considerations? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 90 IV-10-4 What gender-transformative interventions are being implemented within PRISM2 and can these be sustained in FP/MNCHN programs? Summary Questions: S-1 In the remaining two years, what would you like to change either about the project or about your own work? S-2 At this midpoint in implementing PRISM2, are there any significant adjustments that you would like to consider? Are there adjustments within the current management structure or PRISM2approaches that would be helpful? S-3 What are the most useful and fulfilling parts of your work? What makes you frustrated or unsatisfied? Do you have suggestions for improving the Project, what are these? S-4 If the project were starting anew, what would you change, structurally as well as substantively? S-5 If you were the director of PRISM2, or of USAID, what would you change about this project, USAID’s approach to public-private partnership, or USAID itself? S-6 How often and for what purposes do you yourself use the PRISM2website? Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 91 ANNEX I: COST ANALYSIS RFTOP No. 492-13-000003, awarded to Social Impact, referenced a cost effectiveness analysis in the proposed data collection methods and in examples of questions to be considered for the informant interviews and FGDs. It asked “if the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness” and “if the report would include financial data that permits computation of unit cost structure and scalability of the intervention as well as its effectiveness”. To provide cost effectiveness analysis for individual interventions/activities or even groupings of interventions/activities the Evaluation team will need to be able to attribute impact to specific intervention/activities. In addition, for each intervention where specific impact can be attributed, the Evaluation team will need to be able to calculate the total cost of the interventions with attributable impact. The cost per unit of impact would then be calculated for the intervention and then compared with other interventions for which the calculation can be made. PRISM2 was neither contractually obligated to nor are they themselves disaggregating cost expenses. Neither are they asking their grantees or subcontractors to disaggregate expenses. The lack of disaggregated expenses throughout the project period precludes the ability to capture or review meaningful data to conduct a cost analysis or to make assumptions on cost effectiveness at this part of the project cycle. Since the award to CHEMONICS, PRISM2 has captured and reported expenses to USAID with the following 10 line items. I. Salaries II. Fringe III. Overhead IV. Travel & transportation V. Allowance VI. Other direct costs VII. Strategic partnership fund VIII. Grants (paid through Letter of Credit) IX. G&A X. Fixed fee As PRISM2 has not collected or reported costs by intervention or activity, the Evaluation team recommends that a cost monitoring system be designed that will capture this data as it is generated. It is not recommended to ask PRISM2 staff to disentangle cost data on expenses already incurred during the first three years of the project or at the end of the project. Unfortunately, with the level and type of data collected by PRISM2, it is also not feasible to prepare useful estimates of cost per CYP. In order to calculate CYP, the team would have to rely heavily on UNFPA benchmarks resulting in relatively little distinctive content in the numbers or the ability to attribute any changes in CPR to PRISM2 activities. Other means to measure cost elements, such as those described below, were considered and rejected by the team because the cost data were not collected in a disaggregated form to provide the needed figures. 1. If there were any data on Strategic Partnership Fund investments or letter-of-credit Grants, the team may have been able to compare the cost of those investments or grants with deviations in the outcomes (contraceptive prevalence rates or CPRs, skilled births attended or SBAs, and local contraceptives market volume) in the regions where particular partners were active. 2. Even though not contractually obligated, had Chemonics rolled up field-related costs from its 36 sites and allocated costs across the three PRISM2 components in its sites, the team could have tried Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 92 to correlate those individual site costs with outcomes and perhaps have been able to estimate the average cost-effectiveness (across CPRs, SBA, and market growth) of spending on service delivery, communications and technical assistance to governments. 3. If estimates of effectiveness ratios were of interest, the team would need combinations of local outputs and outcomes which would be used to estimate a standard cost for the relevant outputs which would be backed into estimates of cost-effectiveness. In order to provide effectiveness ratios, the team would have needed the following elements for at least several sites or regions: Local CPRs, SBAs, or contraceptives market volumes, and any one of the following: a) Number of providers assisted by type (i.e. contraceptive service or midwife group or contraceptive distributor) b) Number of alternative distribution points by type c) Scale of behavior-change-communication activity d) Technical assistance packages by type. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 93 ANNEX J: KEY INFORMANT INTERVIEWS USAID Philippines Ms. Ann Hirschey, Chief, Office of Health Dr. Yolanda Oliveros, Development Assistance Specialist, Office of Health Ms. Helen Hipolito, Project Development Specialist, Office of Health PRISM 2 Ms. Michelle Mary P. Gardner, Chief of Party Dr. Lemuel Marasigan, Deputy Chief of Party Mr. Jed Sevilla, Senior Area Manager - Mindanao Mr. Mike Lucero, Senior Area Manager - Visayas Dr. Glenn Paraso, Senior Area Manager - Luzon Ms. Nicolas Catindig, Monitoring and Evaluation Specialist Dr. Jonathan Flavier, Itinerant LA/PM Specialist DR. Gloria Viola, Results Tracking and Reporting Director Mr. Richard Jaquez, Administration and Finance Director Mr. E. Alejo, Product Specialist, PATH Mr. Diosdado Chantengco, Senior Marketing Specialist Dr. Myrna Hernandez, Youth Specialist Ms. Rubylenne De Paula, LMAM for Negros Oriental Ms. Maharlika Cossid, LMAM Coordinator for Davao Dr. Lady Castillo, LMAM for Davao City and Davao del Norte Ms. Jennifer S. Nandu, LMAM for Lanao del Sur Ms. Antonieto G. Alaban, LMAM for Misamis Oriental Ms. Renee Fajarmo, LMAM for Pampanga Ms. Elaine Teleron, LMAM Cebu, Mandaue, Lapu-Lapu City Ms. LucinaTapere, LMAM Cebu Province Ma. Solia Virtudazo, LMAM for Bohol and Negros Oriental Department of Health National Office Dr. Florence Apale, Family Planning Coordinator Dr. Minerva Vinluan, Adolescent Health and Youth Coordinator Dr. Rosalie Paje, Women’s Health Division Chief Dr. Diego Danila, Maternal Care In-Charge Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 94 Regional and Provincial Offices Cagayan de Oro City Cheryl S. Balane- CHD 10 LHAD Dr. Rachel Dilla, CHO Cebu City Dr. Lakshmi I. Legaspi, RN and MAN Director III- Assistant Regional Director Dr. Alma Corpin, Field Health Operations Division Chief Dr. Fe A. Cubuao, Assistant City Health Officer Dr. Stella Minoza Ygona, City Health Officer Davao City Dr. Ma. Socorro D. De Gracia, Assistant Regional Director Ms. Nelia Gumela, RN- Regional Program Manager for Family Planning Dr. Cemille Bernadette Sabay, Regional Maternal and Child Health Care Coordinator Dr. Annabelle P. Yumang- Officer in Charge Local Health Maria Teresa C. Requillo, RN- Regional Youth Development Program Coordinator Pampanga Dr. Maria Imelda Ignacio – Provincial Health Officer I Evelyn Mariano City, RN- Public Health Nurse Teresita de la Cruz, Family Planning Coordinator Dr. Eloisa Pineda: Chief of Health Operations Tagbilaran City, Bohol Dr. Reymoses Cabagnot, MPH- Provincial Health Officer II Dr. Portia Reyes – DOH Rep Team Leader Marisa Samante – Assistant PHTL Bohol Province Abilar Anyanga – DOH representative in 3 municipalities Ruth – DoH Rep Giferrepner – Doh rep in 3 municipalities Lorena Conado – Botika ng Barangay (Barangay Pharmacy) Subcontractor/Philippines Mr. Eric E. Camacho, Director, Health PlaCE Management Office, Philippine Business for Social Progress (and three Program Coordinators for IWG Grant) Grantees Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 95 Dr. Erwin Sison, Corporate Accounts Manager, ALPHAMED Pharma Inc. Mr. Jerrywel Jaafar, Manager for Mindanao, ALPAHAMED Pharma Inc & J Health Marketing Ms. Mia Zamyra Baguio – Vice President, UHA Caregiver Training –ZaniViv Ms. June Gambe, Program Manager, UHA Caregiver Training –ZaniViv Ms. Edith Y. Villanueva, President & COO, Sugar Industry Foundation, Inc Mr. Jeremy Agsawa – KMPI Ms. Carla Rodriguez- KMPI Ms. Dionisia Gacayan- IMAP Bohol Partner Schools Mrs. Jessica Sidow, Guidance Coordinator and Counselor, Mindanao University of Science and Technology Ms. Juliet Torres, Vice President for Institutional Affairs, University of Mindanao, Davao City Philippine Health Insurance Inc. (PhilHealth) Dr. Mary Antoinette Raymonte, Head, MDG Products Team Dr. Leilanie Asprer , Head, Accreditation Dept, Phil health Main Office Mr. Andrew Ebio, Social Insurance Officer for Hospitals, Region X Mr. Gerion Cabang, Social Insurance Officer for Midwives, Region X Dr. Jose Rostrada, Head of Benefits Section, PhilHealth Region III Office, Pampanga Dr. Danilo Reynes, Head of Healthcare Delivery Division, PhilHealth Region III Mr. Vicel R. Bracamonte, Chief Social Insurance Office, PhilHealth, Bacolod City Private Hospitals and Clinics Dr. Victor Tiglao, Tiglao Hospital, Mabalacat, Pampanga Orlila Agsalud, Head Nurse of Buda Community Health Center (German Doctors’ Hospital) Ms. Myrna Danuco, RN, Sacred Heart Hospital Ms. Maricel Nacalaban – Staff Nurse, Cagayan de Oro Medical Center Ms. Maria Lourdes Manuel- Chief Operations Officer, Friendly Care Mandaluyong City Private Practicing Midwives/Birthing Clinics Ms. Gertrude, Owner, Well Family Clinic Pasig City Ms. Grace Lagnacio, Owner, Private Birthing Home, Pampanga Ms. Sheila Mae Paquibot, Lapu Lapu City Birthing Home Ms. Marissa Datoon, IMAP President for Cagayan de Oro City and Owner, Fayean Family Planning and Birthing Home Clinic Ms. Cadidia Tabao, Owner, Saduc Midwifery Clinic, Lanao del Sur Ms. Sheila Bagas, Staff, Living Waters Clinic, Misamis Oriental Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 96 Ms. Diesybel A. Justo, RM, Owner Happy Family Midwife Clinic, Misamis Oriental Ms. Merlyn M. Raiz, Owner Raiz-Rich Family Planning and Birthing Home Clinic Ms. Marivel Diango Triambulo, Owner, Well Family/ Dan’s Fatima Birthing Clinics, Iligan City Ms. Grace S. Lacquian, Owner, Divine Grace Birthing Home Clinic, Angeles City Other USAID Projects Dr. Dolores Castillo, Chief of Party, Mindanao Health Dr. Jose Rodriguez Chief of Party for Visayas Health Dr. Alexander Herrin, Chief of Party, Luzon Health Dr Susana Madarieta, Deputy Chief of Party, Visayas Health and former CHD VI Regional Director Dr. Orville Solon, Chief of Party, Health Policy Development Program (HPDP 2) Dr. Carlo Panelo, Deputy Chief of Party, Health Policy Development Program (HPDP 2) Dr. Joan Castro, BALANCED Project Dr. Ricky Hernandez, BALANCED Project Dr. Rebecca Ramos, FP/MCH Advisor, Health Policy Development Program Dr. Cesar Magdalyon, Mindaneo Health Ms. Merlyn Rodriguez- Engenderhealth Visayas Health Provincial Partners Bacolod, Negros Occidental Trino S. Montinola III, General Manager, Sugar Mill & Refinery, First Farmers Holding Corporation Edith Y. Villanueva, President & COO, Sugar Industry Foundation, Inc Mr. Trino S. Montinola II, CEO, Sugar Mill & Refinery, First Farmers Holding Corporation, Bacolod, Negros Occidental Others – US Based Mr. Shaun O’Neil, PRISM2 Director, Chemonics International Dr. Janet Vail, PATH/Seattle Dr. Sadaf Khan, PATH/Seattle Contacted, but not formally interviewed Ms. Gloria Steele, USAID Mission Director Mr. Reed Aeschliman, Deputy USAID Mission Director Ms. Nora Pinzon, Acting Office Chief for Program Resources Management, USAID/Philippines Ms. Judy Chen, Deputy Chief, Office of Health Mr. Jerry Britin, Senior Strategy and Evaluation Advisor, USAID/Philippines Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 97 Mr. John Callanta, Monitoring and Evaluation Specialist, USAID/Philippines Ms. Fatima Verzosa, Project Development Specialist, Gender and Development Officer, USAID/P Ms. Tricia Oriel, Project Associate, PRISM2 Dr. Arturo Tolentino, President, CEO, ALPHAMED Pharma Corp. Maria Teresa Carpio, Project Development Specialist (Private Sector), Office of Health Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 98 ANNEX K: EXTRACTS FROM KEY INFORMANT INTERVIEWS The following are extracted from recordings and transcripts of selected key informant interviews. These are grouped by category of respondent. Each paragraph break indicates that the information is taken from a different interview. Except for parentheses or italics, these are direct quotes from key informants. USAID/Philippines Rated in terms of contract deliverables they have reached their CYP targets – they brought in (Alphamed) pharma partner who introduced the 1 month injectable in the market, they brought in oral contraceptive pills, IUD, condoms, etc. without the direction of PRISM, they took the initiative on their own. They are also responsible for building the alternative distribution points. PRISM has a lot of ideas, but the implementation has not been seen as well yet….Need to do more in terms of capacity building the private providers – this is one of their deliverables. There has been some lag in project implementation because they spent two years doing inception workshops etc. In PRISM1 their focus was to go directly to the private sector and engage them, but in PRISM2 the DoH was telling PRISM to transfer the technology to the public leaders in the health sectors, provincial, regional so that DoH will be able to engage the private sector more effectively… if they train a health provider, they are just stuck on training and they don’t monitor after the training to see if they are applying the training…what will it take for the private sector to work with the public sector? There needs to be a compelling reason for them to be engaged, it’s been encouraging them, but its’ not clear (as the private sector) what they will get in return to engage with the public sector – this needs to be very clear to get their participation. Even PRISM cannot define their PPPS – they have different arrangements in terms of PPPS, it’s not that clear. It’s not one size fits all – need to have different arrangements based on culture and intake PRISM2/Manila I think really from my perspective is that the focus is on number 1 – primarily a service delivery project – USAID had other projects focused on BCC, and the policy piece was to support the integration of the service delivery piece… project had a slow start – not that that’s bad, PRISM 2 had a very different strategy than PRISM … Those processes took time – but in terms of direct intervention piece only got going in year 3. There is still a lot to do across the board. Some of the things that have moved fastest are the ADPs – first grant we managed to get off the ground, Alphamed created a big number of ADPs - big challenge to sustain them because the grant is finished… PPM accreditation is a really important. Area and I think it has been successful –t he 3 grantees that are on board are really focused on that – it changes the requirements are likely to change at the end of our project… The stewardship piece is almost done – in the sites we’re in we don’t need to talk about this anymore - it’s there – we need to talk about the systems and the policies to make it real, but in terms of encouraging stewardship that’s been a big achievement… The communication is pretty good, there is the two way communication and a lot of personal relationships in the field staff. We have not done everyone meeting together, I don’t think there would be significant value added… they spend a lot of money in the first two years, they must have been doing something –people were traveling around doing this stewardship thing – he was only really planning to do 2 years on the project to get the stewardship piece done, but there was not much thought about what came next. There was a pause – then lots of management changes, Mario get sick, Wes (the DCOP) left soon after, so there was no DCOP no COP for about a year at a point when the project needed (middle of year 2 to year 3) to be moving forward… a lot was achieved in the first two years for stewardship but it was not measured or tracked… USAID is extremely hands on, particularly given that we are a contract, should have shifted the technical implementation to the project, but since 2009 they have been extremely hands on – too much yes probably – I don’t think they have given us the space to develop the project the way we want – we’re doing their project, but they’re managing us more like a cooperative agreement rather than a contract…the work place thing has huge Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 99 potential, USAID cannot work on this, but the team in general thinks that the requirement for any employer with 200 or more employees has to have clinic, so you have quite a bit audience… All of the training materials have been reviewed to make sure they’re gender-sensitive and that its integrated into all pieces. The usapan is divided between men and women – trouble is that our partners are primarily the PPMs not many male partners… We’re trying to make midwives into women’s health practitioners – holistic. It has to be profitable… PRISM facilitated the paradigm shift of government engaging the private sector for health services which was envisioned by the former COP, Mr. (Mario) Taguiwalo – whom we all looked up to. So far, there are 2 CHDs that have supported the private sector using their own resources- CHD 7 and 6… More could have been done on policy aspects…More concretely there was not much being done to build capacity at the hospitals in terms of framing policies…There is dire need for capacity development in policy formulation… Ideally there should be M&E people in the field – personally I thought the M&E people at the regional offices would have stayed until the end of the project, it’s too late to clean data when it reaches the central level… the reason we had 28 sites at the start of the contract because that’s where the other projects were working as well and USAID wanted continuity, PRISM was given room to select additional sites, so we increased to 77 – the former COP wanted to achieve scale… Communications and M&E are separate but very related – you can’t be effective at communicating success if you don’t have the right numbers… we developed data quality checklist for all indicators but that was during version 3 of the PMP, we have not been able to update it yet. Because the PMP was only approved less than a month ago… We revisit the targets every now and them based on current circumstances – there are certain indicators they really want – usually USAID asks for upward adjustments – they do not dictate, as long as the assumptions are clear with them, they approve the adjustments… We hope that the current set of indicators will be the same set until the end of the project – once USAID changes the indicators it’s a chain reaction. The process to approve grants is due, in part, to the legacy of mismanagement of funds in P1 awards. The grants program can take 3 months to approve. A more ‘rapid response’ to approve grants would be welcome… If a cost analysis was envisioned by USAID, this should have been part of the initial design. Not realistic to track expenses by objective or activity at this stage in the LOP as each expenditure from YR1 would need to be coded… Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 100 PRISM2 Field Staff Spread too thin in Davao with loss of Communications and M&E positions; we heard about it through rumors months before, but announced officially at the end of the Year 3 when we saw the organizational chart… At this point these are the positions they should have retained. This decision was made last year. A USAID decision we were told. It’s all PRISM, every day – no personal life. It has been my attitude, even in my previous job I am not complacent I am strict with my way of doing things I am not happy if things in PRISM are just that I cannot sleep if there is some work or people that I need to talk to. I am sleepless. It’s like weaving you have to be multifaceted, patient, sometimes you ask questions if I don’t do this and then I have more work tomorrow. Meet with the midwives, the provincial health officers, the program health officers, the chief of hospital, OB-Gyn, drugstores, distributors of commodities, training, logistics, commodities, meetings, leveraging of resources... PRISM 1 was laying the ground, PRISM 2 was cementing the ground. PRISM 2 is harder it takes a lot of your time… There is a big difference between the grantees in PRISM 1 and 2 –before (one) could readily contact the grantee, now (it has) to go through the central office. The usapan is a good training, but it is too filled with information. Originally was three days – one day practicum. After integrating gender became too heavy. If specialists who handles the usapan is from Manila it is usually rigid, need to follow all the rules. But here we adapt, depending on who the participants are, in Manila they give rules to do this and do that. We can learn from the participants… If developed by Project, central thinks that it should be done like this and like that. But hello, some of the people in the City Health Office are more experienced than we are… It is more of top-down. They think that theoretical things can be applied a to z in the field. But (every region) is different. Some things need to be adjusted… TIPPPs – it should have been helpful, it took too long. On how to operationalize that was lacking. If we try to implement what was written or make comments, we were told what to do. Only changed with fast tracking things during the last year – when some of the things we had been saying since the beginning were adapted… then were told just to follow the 1-1-6: one facility; one private provider; and six ADPs. Difficult to create that SDN because we need to follow steps, need to map it out, identify hospitals, etc. but they just laughed at us. Why were we making it hard for ourselves? We just followed the MNMCHN – we are creating a network. 1-1-6 does not include a CEmONC necessarily, or BEmONC capable facilities. Why did we have to get all these hospitals when just needed 1-1-6. We were creating a network in one inter-local health zone. What is behind that directive is just to check the box and reach the target of 12 by September. One of the high points was multi-region meetings where we presented the TIPPS because that actually fast-tracked the involvement of our government. partners for them to also be active partners of PRISM2. It encourages me to do more (because of) the relationship we have established with the government partners – provincial health office and the LGU – I have seen that they are committed to the project, they value it – this motivates me to do more. Also the relationship between LMAMs, the support is there, whenever we need assistance without having to ask our technical specialists… When our sites were dropped I was assigned two LMAs and we were just surprised that (others) was dropped and we left; we had started a very good working relationship with the local LGUs and the private partners and were told that it would be discontinued, just like that. We had a long-term plan and then these areas were just dropped. Follow-up after training seems to be missing. One must have a post-implementation plan to see how successful we are… PRISM has set the standard on how things should be done to establish SDNs and capacity development. Things will be a lot easier for (the new projects) to achieve success. But they should get a wider focus. Make one comprehensive project for health, even the partners get confused by many projects at the same time. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 101 I asked for administrative support during the training so I do my paperwork in the evenings. I work from home but I can float to the PHO or other [DOH] offices because of my prior relationships with them on other projects… IMAP has been productive and helping the public sector … it is a good practice to include dual practice midwives … private sector has fewer protocols than public sector. In PRISM1, Negros Oriental was unsuccessful. IMAP organized an exchange trip of PPM to Bohol to see how it worked … need to document PRISM2 products in simple terms to successfully share efforts … PRISM2 had an organizational development (OD) person who did training. The ICLI grant and IMAP also had an OD survey, strategic planning and review… TIPPPs were like a straightjacket – needed flexibility in the field – regional staff became clerks and the specialists needed arbitrators. a venue for active cross communication of operations and technical [would be useful]… DOH Central is not briefed on what PRISM has accomplished. Policy changes are generated at the region level… To speak a common language – USAID terms and frameworks can seem so foreign ... When ICLI award was granted in 2011, no tools yet approved. As grant technical supervisor, I repackaged tools from PRISM1... End-of-project report for PRISM1 was prepared in the last quarter of the project. The end￾of-project report for PRISM2 should be a priority to capture the richness of the experience Central DOH How do you think that the private sector can be more fully engaged to meet FP/MCH service needs? Data sharing, records, especially deaths…we don’t get that now. In NCR, 60%-70% are private providers. We have a hard time meeting our indicators. The private is not required by law to submit this information. We license all hospitals, but it is not a requirement that they submit all data… Accreditation in PhilHealth might be able to compel them to provide this data; we are talking to PhilHealth about this. Like the carrot and the stick, same organization providing data and reimbursements… For maternal health, we have been involved with private midwives training. The number is actually now very high with midwives who are now accredited. A team approach has been used for the training. This started with PRISM1… We also have to be in the work place; a great number of females are there… PRISM2 is a work in progress, just realizing that private sector is the missing link… Essential new born care and BEmONC are where they have made contributions in service delivery. They could be of help in training. Outsource training – I have talked to PRISM about this. Pretty soon all our government training centers will be overwhelmed… Maybe they could help with the pre-service training in BEmONC – colleges of medicine, etc. So that we will not do too much training post-graduate... Evidence of building capacity – or is it the other way around. BIMOC and their formulation of the supervisory list for midwives was based on what we had in DOH. They used our BIMOC manual. Not just clinical skills capacity…they also developed the entrepreneurial capacity of the midwives. They started the part of getting some buy-in for the private sector, this is food for thought for us, engagement with the private sector…PRISM2, they are functioning like a central office, producing manuals. We haven’t seen that impact yet in the private providers. With PRISM2 although we were engaged initially with (participated in) some of their primary activities, and even in the IEC part, training, during their orientation, back in 2010-2011. Then we were also involved in finalizing their IEC instrument -- usapan. After implementation, we did not hear much back. This approach could also be useful for the public sector. Also we heard from others that they were involved initially, but then did not hear back… Usapan – is not really a new approach for demand generation – has been used by others before. PRISM2 used this approach among other groups, youth, males, etc…. A good effort is being made with supplies such as through Alphamed… Alphamed making products available to the departments has been helpful… Gaps in funding can be filled by GOP budget, which is rather extensive… coordinating with PhilHealth in the packages to be issued. PRISM2 is discussing the need to revisit the IUD scheme… Regional and Local DOH Officials Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 102 We have an MOU with the birthing centers- we could train them with capability trainers – they can organize the trainers and we can be the resource for the training… How did you determine the increase in CPR? Based on our report data and the target set by the DOH...In the past we were not able to get the reports of the private sectors, but due to PRISM we got the private sector report… PRISM has been more of technical support and training – bringing stakeholders together in meetings for trainings…Mostly on capacity building to the private and public midwives and putting us together in a meeting so that we can come up with strategies for FP and MCH – they have coordinated meetings, and the trainings of the midwives… I really have a hard time – with my staff being utilized as a trainer – they have plenty of work, so I verbalized that to PRISM – how can we undertake the trainings without too much load on my staff – how we can contract out to CSOs… It’s all recommendations, still it has not been resolved, but we will work out how to maximize trainings because we need to do this… Another area where PRISM has helped us is to understand the prices of the PhilHealth accreditation. Understand the process – until now we do not understand it… accreditation is part of sustainability… Even without prism we would have done this – it took us a long while before we were doing this…We can do it without PRISM – maybe PRISM made us aware of the system and for us to increase CPR to incorporate the private sector, but we just have to maintain and sustain. It was our plan all along to engage the private sector… not yet addressed adolescent health. Prism opened our eyes – awareness – that this is important, so now we are including the adolescent health and youth development in our plans…before there was no relationship between the private and government birthing centers but through prism we are able to see how to maximize services for maternal and child care through the birthing centers by really coordinating… We had a meeting with prism where they called on the big business establishments in 2011 and they talked about having family planning clinics in big companies, but it’s not implemented so this is a missed opportunity – there was no follow up. But this is a big opportunity…we would very much welcome a person seconded to our office from PRISM I am always looking ahead to both PRISM and even UNFPA, for an extension, a little over year from now, it’s not that easy to engage activities at the local level – the minds of the different LGUs’ local chief executives – this year there is a new batch of incumbent mayors – so changing their minds assuming they have been already exposed to this – the newly elected mayors who have no head of how local governance is done – this will be a challenge for us and for PRISM – it will be very difficult if PRISM ends next September. We will be losing a partner to help convince the local partners – I don’t think we can do this alone… PRISM’s role in this training…They provide the technical and logistics – funds for the training… through PRISM we were able to come up with our policy on maternal death review – the only one in the region and in the Philippines – other provinces will replicate – it’s a provincial policy. It was already in place when I arrived in 2010 or early 2011 – early part of the project… I think Usapan is unique – I have been working in FP for 20 years, I talked about the Chat session under AED, the concept is the same, the word is just new. What concrete results have you seen from something that PRISM2 has done? – It’s too early to see results, this is the initial phase of orientation and training, maybe after some months the midwives will be able to use it in their field and the private sector. The one big problem is submitting reports so we can hope this will come. In our place we are only seeing limited data from the midwives, but eventually we hope they will be reporting… I think there is already a partnership but it’s not yet very strengthened…(in) a strong partnership the private would really carry out our project and give data, attend meetings, referring their patients for hospital exam, the pharmacies would refer to the health center. So there is already a strong partnership with TB cases. The PRISM people help to mobilize the private sector, especially the lying-ins; then they tie up the private sector to the govt. and they provide assistance for the reporting system to be integrated into the FSHIS – PRISM is spearheading this. They are also involving private hospitals… So far I think they are Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 103 very effective, esp. the implementation of the lying-in centers – they have a quota for the whole province, I think they already have reached the quota for setting up birthing centers, 15 in the entire province. And they are already MCP accredited by PhilHealth… there is not an equal distribution of lying in clinics – some MHOs are supportive and others are not. Those that are not can block it. They don’t want competition with their govt. sponsored facilities. PRISM has not provided any support to try to convince the municipal health officers to support the lying in clinics… the mobile clinic for FP acceptors – it’s been planned but not installed yet. Outreach mobile clinic. They can perform vasectomy, itinerant teams – we don’t have facility for this. We have the knowledge to do this but not the facility… Here at the provincial level we are not prepared to take on the prism activities – no sustainability plan… The SDN – that it would really work, after the launching it seems that nothing is happening any more. The referral network – would want this to really work – it’s not two way… Related to the referral system – we have some cases like the incidence of maternal death the IMAP lying in has been holding the patient for almost 3 days before referring to the hospital – this is not a good practice, there should be a referral mechanism from the IMAP to the RHU, especially for the availability of the ambulance – it should be strengthened – from the IMAP to the RHU…. Yes we are positive about the PPP initiative. Is it possible if PRISM is not there? We always say that DOH cannot do it alone. We are the coordinator of this, we have DOH representatives, but then their hands are tied only with the public, we have so many public sectors, going into the private sector we cannot do it alone… Since they have started the PPP by when they finish we hope the linkage will be kept and sustained but we know and it’s our experience that when a project ends everything ends… We hope that those who are reporting and have been linked with the RHU person in their catchment area we hope that will be continued to be partners even without PRISM – we can only hope. But if the system is there then the hope becomes more of a reality. Because I have witnessed some systems that are there with the project but when it ends, then so does the system… We had worked in PPP in other areas – TB, but not FP. We don’t need to know about other country examples of PPP in FP, what we have is sufficient. Most effective is the capacity building of the private sector midwives; PRISM2 is shouldering the bulk of expense for training the private sector. Organization of SDN is receiving technical assistance. SDN is included in the MNCHN operation manual 2010, but we have not really organized it. A SDN does not only focus on public sector but also includes private sector, within a referral system. Quality training which was organized by PRISM is a sort of tool – can be also useful for the public sector; we are going to adopt that same tool to monitor the public sector midwives. We are adopting the check-list into the ISO; this was our initiative, not that of PRISM… The system back in 2008-09, we did not really care about the private sector. Then slowly this has been changing. The best experience I have had is with the TB program – the partnership with the private sector is going well… the DOH cannot have it all; need a group to look into the private sector which cannot be handled by us. Our work and budget are more focused on the public sector. It is good to have an organization like PRISM to help the private sector… Especially that we now have a point person for PPP in our office, a nurse. Some of the activities can be sustained. We decided in the CHD to have such a point person. I think there is a national directive to have such a point person. We do not know if it is functioning in other regions – but we still need to inform our point person about the activities with PRISM… The grey area of PPP is how to do it. We had patterned it on the TB program – important to helping us… We do not see abortions in our maternal death reviews, it is not reported. Manual vacuum aspiration was piloted at Davao Hospital about six years ago, but it did not prosper. If properly handled it could be re-introduced – it is good. As far as I know I have not heard of prism doing anything in policy… PRISM2’s target is for the midwives to be an accredited health care provider – during the DOH trainings. IMAP conducts the trainings, they invite trainers from DOH. We want quality service which is why we want an accredited health care provider… PRISM provides us with TA in policy development – it was last year or the other year. There was one time that we issued a regional memo to urge our local health personnel to follow the Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 104 guidelines on FP/MCH and they provided us with guidance on how to formulate the memo… Prism drafted the memo. And it was adopted by the region later and we cascaded it down to the LGUs… HealthGov also conducts meetings to improve the service delivery network of FP/MCH. They ended the project last November. HealthGov had an office inside the CHD – only PRISM has a separate office… I always observe – there are times they don’t coordinate with the region they go directly to the LGU, perhaps they lost their interest to deliver the goods, because they have their own timelines and we cannot adopt their schedule – we’re not only focused on MCH, we have other programs, so it they cannot wait for us they go directly to the LGU. We would prefer for them to stay with us like the other NGOs, physically with us. There has to be a complementary relationship with us; there are times when they are very demanding – I am ordered to go to the central office to attend meetings and sometimes it becomes irritating, very demanding. There are text messages, please block 3 days and consider it a priority to attend BTL training for example. They are like my secretary. I am friends with (the LMAM)…the (SAM) is very good… We had one champion he was the mayor of a town in the south – he submitted himself for a vasectomy as a model...PRISM has not done anything to train CHD on gender. I did not coordinate with PRISM1 maybe there was a little bit of competition with the private sector, that was my feeling,…later I made myself available and realized that there was no competition…before they (PRISM1) went to the private (directly), maybe they were informing me, but they were not really interested because they were already doing something with the private…it was a very good idea that they were coordinating with the private companies…(which) had their own clinics…PRISM2 is helping the private midwives with accreditation. Private Providers I have one administrator in each clinic to do the paperwork for PhilHealth – but I am the only provider in my clinics. We are allowed to only have 3 clinics under PhilHealth accreditation – as a midwife… Some of my friends are under PRISM project – doing the same like us, but only Well Family is different because before you join you have to follow lots of instructions, there are standards, but for the PRISM as long as you have a clinic you can join the PRISM. They assist, they help with the trainings. And the trainings seem similar – I think so. I heard the same. I have not joined any of the PRISM activities. How many well family clinics? – Hundreds all over the Philippines. It was originally the JSI project. JSI is no longer involved. All the clinics that are here now are sustainable on their own. After the project we formed a foundation – the network of the well family. We are also franchise after the project, through the foundation – where does the funding come from for the foundation? USAID gave seed money in the foundation, and we pay monthly dues, royalty fee, franchise fee, and 10% goes to the foundation and 90% of the franchise and royalty goes to the NGO – they have quarterly meetings, updates, they are doing the visiting us once in a while to make sure the standard is still there. PRISM 1 started with me (PPM) as one of the first clinics – we didn’t know what to do, we had our maternity clinic. Family planning, skills training for the DMPA (injectables) and then they helped us also to go back to the training for suturing, iv insertion, and internal examination. These were doing PRISM 1. They started in 2004, I was one of the pioneers. All of the seminars that PRISM has done, we have been to everyone. ..They told us about PhilHealth accreditation – this we got through PRISM2. It took us about one year to become accredited because we did a lot of trainings… We are very sure now about what we are doing because of the trainings. We are also now involved/connected with the barangay, and the referral system is now ok – the people around us recognize the birthing home, before they didn’t know what a birthing home was – before they would go to the hospital… in the barangay, I work with them and we also educate them what to do and we ask them if the patient wants to deliver at the house (if the patient will deliver in the house and they are a PhilHealth member, rather than having the traditional midwife Helot to do it, we ask them to bring the patient here and then we pay the Helot)… If a patient comes in and was already bleeding an…abortion – I refer to them to the Ob/Gyn – I ask Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 105 questions, what they are doing, send for ultra sound at referral hospital, when they come back I call the Ob/Gyn who will do the D&C. I go with her to the hospital. Does this happen often? In a month, maybe two or so. Abortion is in the slum areas… I belong to IMAP – they also do trainings, we go to other places – the convention is in Davao, but training is here. (As a PPM, I was trained as a trainer by) UHA – a combination project between UHA and PRISM and we were the participants. We were about 20 who graduated from the training. The rest of the participants were teachers from different schools – they are working at the schools; they are Clinical Instructors RNs RMs – registered nurse and registered midwife. I have the 2 year midwife course. Was the govt. involved in the training? Yes – they were there during the monitoring especially – like the FP-CBT1 we had a series of activities that we need to implement so by doing that we need to go from private to public set up – so we visited some of the private midwives, we’re taught in the training we need to do it actual in the field. When we interview the client for the FP, we try to use the gather approach and introduce the commodities. Like discussing the advantages and disadvantages of different FP commodities. Have you trained anyone else now that you finished your training? The training that ICMCI conducted was April 1-5 - conducting a training was part of the practicum. Are you slated to be part of the trainers on the next FP CBT1 training? Yes, we have the schedules… In my personal opinion, I think they (PRISM2) are more on to helping those midwives the private like me to improve and enhance my skills in terms of the seminars and training they are giving us and in our facilities they are helping us to improve our services, the skills acquired and the knowledge and they are giving us, technical assistance… I have done the second clinic on my own and would have done it even without prism – my husband is there, so if prism does not pay, it’s ok. Most of the people are sponsored by PhilHealth, but they are still delivering at home because there is not a birthing home there where they can avail the services…(For usapan) we gather the participants who of reproductive age and are willing to have the FP – the acceptors and those who are acceptors already, and those who are post-partum delivered – we encouraged them to have the group discussion. We had posters put in the health centers – we coordinate baranguay health workers, who know we are having these activities – we give free services pap smear free IUD insertion and removal and also for the discussion of in every family – the Baranguay Health workers (BHW) go house to house to inform the community for the activities – to every house each BHW has a designated area. We have 3 participants with their husbands who came along, as we notice, the husbands are so difficult to be invited – they still need more counselling to better understand. We have cue cards – to help the participants envision their family in the future – we distribute the forms. Our president was here yesterday together with my colleagues they were the ones discussing each FP method. The goal is to help them realize what they want for their lives in the future and then to encourage them to use FP to limit children and space children… Only 30-40% (of my patients are PhilHealth covered) so I recommend my patients who are not covered – I even have the forms and ask them to fill it out and then send it to PhilHealth – because I will benefit from this. They ask of me statistics, how many BTL done, but I don’t know what they are doing with the data. I do not submit data to CHO. Before when we had supplies from the city health office like condoms, pills, but now we don’t have any supply from city health from USAID), so we purchase our own supplies. Other USAID Projects For the private sector, our project would take over from what PRISM has started and take over from there… historically the reason we have a large private sector is because the public sector was not up to par; but they have improved and demand has increased, and they need to link up with private sector; for example midwives, who have been supported in the private sector even prior to PRISM1. Now that they are there and public has improved, desire to link up – how they are licensed, how they are linked; continuum of service from birth to child… PRISM talks about SDNs, selectively involving private sector to increase services… most significant achievement for the private sector has been the dramatic increase in pills (OCs); no dip in overall CPR Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 106 because private sector became the source – commercial as well as social marketing (mainly DKT, the major player there)… To me the numbers; we were involved in the previous evaluation – the numbers were very small: no. of midwives involved was very small compared to the need. Also there is a design issue that has not been considered – FP and MCH as a package, but in reality birthing clinics are mainly for deliveries, not much FP, and even internal contradictions. Midwives not the ultimate solution, not reaching enough people, women…Introducing new products; one of the most effective ways of increasing new products, offering a wider choice to clients… a lot of grey areas in terms of accreditation, quality of care; some would say need more regulations to ensure quality; others that we need a more encouraging environment… Too many deliveries handled in big hospitals, with all equipment and services, rather than handling normal deliveries outside by midwives…financing – to establish a birthing center for example. I am not sure that that is something that PRISM2 has done. TA without accompanying financial package will have less impact. PhilHealth is the driver – but reimbursement rates do not provide much incentive…(Usapan is a) tool for the community health teams, usapan – modules to organize sessions with clients to communicate in an interactive way. My project has recognized the value of this and is working with PRISM on this. They have developed the materials farther than before – focus on limiters, or some other groups… ADPs is another one. We have seen the need in many areas the need for access to contraceptives. There is already a lot of effort that has gone before – PopShops; national pharmaceutical federation; boutique baranguay; a lot of work has gone on that they picked up on features here and there. A gap is the sustainability of the initiatives. CHDs – most of the initiatives, like in PRISM1 before, lack of follow up, sharing best practices among midwives, started good work in working places, but nothing now. Provisional headquarters are not aware of private midwives. Also with PhilHealth, many public sector midwives have shifted to the private sector – an unintended consequence… During the time of the late Mario T, really recognizing the SDN as the driver. But the technology that has been provided should be scaled up. But by the time he left…feeling on part of LGUs that USAID has left us, they have not delivered. Subcontractors and Grantees: The govt. system is so rigid that they don’t even know how to manage it with the new training system. They say this is only for this year and they will find ways on how this can be done better/differently next year. From what I have observed from the leadership of CHD you can really see the full support, but going down the line, that’s where there might be some need for more clarification for the new role of CHD – but the support of the director for the project. You can really see the effort of the director, the support and understanding of the project. We hope that CHD will look to PRISM to assist them on how to go about this. When we finalized the costing for the training it was with PRISM and CHD... Coming from the private sector we are not very exposed to public sector ways – so I would think that PRISM has developed a deep understanding of how to deal with govt. and that understanding and skill they are transferring to us so we will be able to have that win-win agreement and be able to be aligned in our thinking and mindset. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 107 ANNEX L: ANALYSIS OF THE PRISM2 PERFORMANCE MONITORING PLAN (PMP) The PMP is a project’s foundational tool for tracking project implementation, measuring progress against targets, reporting on key outputs and outcomes to analyze advancement toward achievement of results, and for informing decision-making, resource allocation, and learning. Thus, the preponderance of project successes and shortcomings should be captured by the Project’s standard indicators housed in the PMP. Where that is not the case, and evaluation findings from primary data collection are predominantly inconsistent with indicator performance, a significant gap between Project activities and the PMP is revealed. Accordingly, the team necessarily dissected the PRISM2 PMP and its associated indicator data as presented in the Quarterly Reports (Q) to both inform the answers to evaluation question 2 and to serve as the basis for understanding all aspects of Project performance. PRISM2’s PMP underwent several rounds of modifications, revisions, and adaptations during the first three years of the Project. The latest version, dated March 2013, was revised in response to RIG Audit findings and developed through a series of consultative meetings among PRISM2 staff. The PMP comprises 34 indicators that span three overarching categories including: 1) USG BEST Action Plan and non-BEST Action Plan 2) Results Area a. USAID Development Objectives b. PRISM2 Development Objectives c. PRISM2 Intermediate Results 3) Indicator Level a. Process b. Outcome Many of the indicators in this version of the PMP were established to measure new Project initiatives, such as “the number of maternity care package accredited health facilities as a result of USG assistance,” which was added at the start of Year 3 and is an indicator drawn from the USG Best Action Plan. Numerous indicators like the aforementioned, lack targets as well as performance data for the first 2-3 years of the Project, with measurable outputs only available beginning in 2011 and 2012. Likewise, several pre-existing Project indicators, such as “percent of DOH-CHDs with improved capacities to sustain program initiatives beyond program life,” lack measurements for the first 3 years of Project implementation. In addition to recommending extensive revisions to the Project indicators, the RIG Audit advised the Project to reconstruct baseline values where they were nonexistent and to establish ambitious, yet realistic, targets for each of the indicators. A thorough review of the PMP reveals that baseline measurements have not, in fact, been set for many indicators. Several performance indicator reference sheets (PIRS) cite that the indicators’ definition during PRISM was not consistent with the DOH-FHSIS definition and, therefore, not comparable to PRISM2, leading to a baseline of zero. For select indicators, such as “the number of pregnant women with at least four antenatal care (ANC) visits by skilled providers from USG-assisted facilities,” a thoughtful and detailed rationale is provided for how the targets were established despite the absence of a baseline. Conversely, in the case of “number of deliveries assisted by skilled birth attendants (SBAs) in USG-assisted programs,” the PIRS states that the baseline source is the 2009 PRISM Year 5 accomplishments from the 136 private practicing midwives (PPMs) across the 10 assisted local market areas (LMAs) but the baseline provided is zero. A detailed explanation about the basis for establishment of targets was provided to the team in late June along with the evolution of the indicators and would have been useful as part of the briefing documents. That said, according to the target and actual figures included in the PIRS, the FY 2011 and 2012 targets for this indicator were exceeded by 20% and 37% respectively. The PIRS states that the targets for FY 2011 and Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 108 2012 were set at 12,862 and 15,215 respectively, which should have produced a cumulative target for years 1 to 3 of 28,077 and thus an actual accomplishment rate of 129%. However, PRISM2’s Q14 Progress Report presented 15,215 as the Year 1 to Year 3 cumulative target for this indicator and 238% as its cumulative accomplishment rate, highlighting a significant discrepancy between the PIRS figures and the Q14 figures. Difficulties that result when indicators are not an appropriate measure of Project-level performance and lack reliable baselines and targets are the following illustrations drawn from the most recent draft quarterly report of 16 May 2013 to the USAID Mission. PRISM2’s draft 14th Quarterly Report, covering the period from January to March 2013, provides values for 25 Project indicators for the first two quarters of FY13 as well as the annual percentage accomplishment rate. The remaining 8 of the Project’s 33 total indicators either had values of 0 (3 indicators) or were recorded as N.A. (5 indicators that rely on secondary data sources for which data is not yet available). The report also provides summary statistics on the Project’s indicators for year 1 to 3 accomplishment rates as well as year-to-date progress against 5-year Project targets. Five of the indicators reported under this section are based on moving rather than cumulative targets, which means that the numbers presented are not, in fact, indicative of year 1 to 3 cumulative accomplishment rates, but rather, are reflective of only year 3 performance. Ninety percent of the 20 cumulative indicators with percent accomplishment measurements from Year 1 to Year 3 achieved more than 100% of their targets, while 30% reached greater than 200% accomplishment rates. Eighty percent (4 out of 5) of the indicators with moving targets reported in the draft Q14 report under year 1 to 3 accomplishment rates, achieved more than 125% of their targets. These accomplishment rates indicate that target-setting was a challenging exercise for these indicators and in the future, should be carefully revised to be more ambitious. With the Project 70% completed, indicator accomplishment rates for the 5-year, cumulative targets ranged from 0 to 234%. Thirty-one percent of the 32 indicators reporting year-to-date accomplishment rates for the 5- year targets have achieved less than 50% of their targets. While starting with the Q12 report, indicator shortfalls were accompanied with a thorough and helpful explanation, the team was unable to locate this same information in earlier quarterly reports or any explanation of indicator targets that had been significantly exceeded. The draft Q14 report also reveals that more than one fifth (21.4%) of indicators reported to date for Year 4 achieved more than 100% of their annual target with two reporting periods remaining. This emphasizes the finding that targets for these indicators were not well established. For example, the indicator “number of current users of modern FP methods from participating service delivery points” exceeded the FY 2010 - FY 2012 target by 204% With two quarters remaining in FY 2013, the annual target of 11,169 has already been exceeded by 45%. While the PMP reports that year 4 and 5 targets were adjusted based on the assumption that 30% of the targeted number of people counseled would accept an FP method, the data confirms that this assumption was considerably off-base. Fifty-four percent of reported Q14 indicators achieved less than 50% of their Year 4, annual target, such as the “number of people receiving FP trainer’s training with USG support”, which accomplished 8% of its target, and the “number of informal working groups implementing FP-MCH activities”, which achieved 0% of its annual target. Of the remaining indicators, 25% appear to be on track to meet their targets with between 50-100% year-to-date accomplishment rates. Indicators that fall into the latter category with well-established targets include an indicator measuring a new, 2012 PRISM2 initiative: “number of USG-supported educational institutions and youth-oriented NGOs implementing AYRH/FP￾MCH activities” reported a 64% annual accomplishment rate, as well as “the number of USG-assisted service delivery points providing FP counseling services” has achieved 83% of its target for the year. In conclusion, PRISM2’s indicators should provide the foundational reference point for understanding the Project’s performance on outputs and outcomes with regard to achievement of success in each of the Project components or IRs. With a PMP containing indicators that are not a reliable measure of the Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 109 Project’s outcomes, with targets that have been consistently and grossly exceeded or missed by more than 5% and lack a comprehensive rationale, with outcome-level results that are measured in terms of output-producing indicators, and with the first 2-3 years of project resources spent on producing outcomes, such as stewardship, in the absence of robust indicators to monitor progress towards this goal, it is exceptionally important to provide an exhaustive critique of the PMP. The weaknesses that permeate this cornerstone document undermine the ability of anyone, let alone an evaluation team, to identify the real gains and weaknesses of the Project. The pervasive absence of baseline values, the inconsistencies between the indicator reporting format of the PMP and quarterly reports (where the former includes annual targets and actuals, and the latter present cumulative accomplishment rates) undermine the data’s utility and make it difficult to monitor trends in indicator performance. Moreover, in light of the weaknesses of many PRISM2 indicators, which do not accurately correlate to the results that are intended to track progress towards results, it is difficult to draw conclusions about which outputs and outcomes have contributed the most to project successes. The utility and reliability of conclusions that could be drawn from many of the Project’s indicators is questionable. USAID’s FACT info system, as captured by the performance plan and report (PPR), requires projects to provide explanations any time that an indicator exceeds or falls short of its target by 10%. PMPs that do not contain the 10% rule of thumb are subject to increased scrutiny by performance auditors when targets are not met. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 110 ANNEX M: PRISM2 ORGANIZATION CHART Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 111 ANNEX N: TABLE 1 FROM PRISM2 Q14 REPORT (05-16-13) Table 1. Year 4 status of PRISM2 targets and accomplishments as of Q14 (January-March 2013) * no targets set for the year but actual accomplishments will be reported; n.a. - not applicable Indicator Year 1 to 3 Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct-Dec Actual Q14 Jan￾Mar Total Year 4 % Five Year Target Actual % USAID Development Objectives Couple-Years of Protection (CYP) in USG-supported programs 1,005,801 1,377,170 137% 700,316 IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 809,965 2,516,082 1,377,170 55% Amount of in-country public and private financial resources leveraged by USG programs for FP/RH (in US $) (CUMULATIVE) 198,327 623,634 314% 186,165 160,998 174,235 335,233 180% 25,155 409,647 958,867 234% Number of current users of modern FP methods from participating service delivery points (MOVING) 1,835 5,574 304% 11,169 6,361 16,181 16,181 145% 17,379 16,440 16,181 98% Number of pregnant women with at least 4 antenatal care (ANC) visits by skilled providers from USG-assisted facilities (CUMULATIVE) 3,822 5,398 141% 5,117 3,449 2,795 6,244 122% 1,934 10,873 11,642 107% Number of deliveries assisted by skilled birth attendants (SBAs) in USG- assisted programs (CUMULATIVE) 15,215 34,977 230% 19,689 9,250 11,019 20,269 103% 8,692 43,596 55,246 127% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 112 Number of deliveries in USG￾assisted health facilities (CUMULATIVE) 28,077 33,937 121% 19,689 9,221 10,943 20,164 102% 8,692 56,458 54,101 96% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 113 Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct-Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % Number of postpartum women who initiated breastfeeding within one (1) hour of delivery (CUMULATIVE) (NEW) 0 0 N.A. 13,782 N/A 5,878 5,878 43% 6,084 19,866 5,878 30% Number of people reached by a USG-funded intervention providing gender-based violence (GBV) services (CUMULATIVE) 800 2,377 297% 9,733 797 1,239 2,036 21% 6,833 17,366 4,413 25% PRISM2DevelopmentObjectives Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources 54.40% 53.80% 99% 56.00% National survey data not available National survey data not available National survey data not available N.A. 57.60% 57.60% 53.80% 93% Percent of deliveries with a skilled birth attendant (SBA) 65.40% 69.30% 106% 67.00% National survey data not available National survey data not available National survey data not available N.A. 68.60% 68.60% 69.30% 101% Sales volume of oral contraceptives 12.71M 39.56M 311% 12.97M IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 13.23M 64M 40M 63% Sales volume of injectable contraceptives 1,071,334 1,379,357 129% 400,729 IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 424,773 1,896,836 1,379,357 73% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 114 Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct-Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % Intermediate Result 1 - FP/MCH Services Number of service delivery networks strengthened (CUMULATIVE) 10 0 0% 20 1 2 3 15% 16 36 4 11% Number of informal working groups implementing FP-MCH activities increased 0 0 N.A. 200 N/A N/A 0 0% 25 225 0 0% Number of USG-supported educational institutions and youth-oriented NGOs implementing AYRH/FP- MCH activities 0 0 N.A. 22 8 6 14 64% 26 26 14 54% Number of people trained in FP/RH with USG funds (CUMULATIVE) 150 295 197% 350 35 109 144 41% 100 600 439 73% FPCBT1 * 252 N.A. * 35 95 130 N.A. * * 382 N.A. FPCBT2-IUD * 56 N.A. * 6 20 26 N.A. * * 82 N.A. FPCBT2-BTL * 26 N.A. * 1 3 4 N.A. * * 30 N.A. Number of people receiving FP Trainer’s Training with USG support * 4 N.A. 40 0 3 3 8% 20 60 7 12% Established functional DOH training system on FP/RH N.A. N.A. 3 0 0 0 2 5 0 0% Number of people trained in MCH with USG funds (CUMULATIVE) 150 198 132% 350 34 286 320 91% 100 600 518 86% EINC * 198 N.A. * 37 221 258 N.A. * * 460 N.A. QAP for Midwives * 0 N.A. * 0 38 38 N.A. * * 38 N.A. BEMONC * 0 N.A. * 0 0 0 N.A. * * 0 N.A. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 115 Newborn Screening * 0 N.A. * 0 75 75 N.A. * * 75 N.A. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 116 Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct-Dec Actual Q14 Jan￾Mar Total Year 4 % Five Year Target Actual % Intermediate Result 1 - FP/MCH Products Number of sustainable alternative distribution points (ADPs) for contraceptives products supported with USG￾assistance (MOVING) ** 200 258 129% 400 209 143 143 36% 600 600 143 24% Number of LGUs implementing their Contraceptive Self Reliance plans (MOVING) 130 282 217% 210 167 1 168 N.A. 210 210 (168) N.A. Number of distribution outlets for USG-assisted MCH products (MOVING) 30 74 247% 60 25 47 47 93% 70 70 56 80% Sales volume of USG- assisted MCH products *** 303,380 596,496 197% 333,718 150,500 51,400 51,400 15% 367,090 1,004,188 647,896 65% Intermediate Result 2 - Demand generation for FP/MCH Services Number of new FP-MCH products introduced in the local market 3 4 133% Oxytocin in Uniject 0 0 0 N.A. 1 5 4 80% Number of USG-assisted service delivery points providing FP counseling services (MOVING) 100 54 54% 250 149 208 208 83% 350 350 208 59% Number of counseling visits for FP/RH as a result of USG assistance (CUMULATIVE) 5,100 12,700 249% 29,200 7,526 9,369 16,895 58% 20,500 54,800 29,595 54% Intermediate Result 3 - Policy and financing for sustainability of private sector FP/MCH service provision and use improved Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 117 Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct-Dec Actual Q14 Jan-Mar 2013 ToTal Year 4 % Five Year Target Actual % Number of private practice midwives (PPMs) becoming PhilHealth accredited and accreditable as a result of USG￾assistance (CUMULATIVE) 60 73 122% 375 25 75 100 27% 70 505 173 34% Accreditable 0 58 N.A. * 18 74 92 N.A. * * 150 N.A. Accredited 0 15 N.A. * 7 1 8 N.A. * * 23 N.A. Number of Maternity-Care Package Accredited Health Facilities as a result of USG￾i t (CUMULATIVE) 15 41 273% 250 24 53 77 31% 85 350 118 34% Accreditable 0 25 N.A. * 19 48 67 N.A. * * 92 N.A. Accredited 0 16 N.A. * 5 5 10 N.A. * * 26 N.A. Number of FP-MCH claims filed for PhilHealth reimbursement (CUMULATIVE) 0 16,290 N.A. 6,000 4,646 5,697 10,343 172% * * 26,633 N.A. FP 0 637 N.A. * 307 321 628 N.A. * * 1,265 N.A. MCH 0 15,653 N.A. * 4,339 5,376 9,715 N.A. * * 25,368 N.A. Number of LGUs that submitted/issued new local polices in support of private sector provision for FP- MCH services and products (CUMULATIVE) 10 18 180% 0 2 0 2 10% 6 36 20 56% Number of national policies supporting private sector provision of FP-MCH information, products, and services developed (CUMULATIVE) 14 28 200% 15 0 1 1 7% 5 34 29 85% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 118 Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target ActualQ 13 Oct-Dec 2012 Actual Q14 Jan-Mar Total Year 4 % Five Year Target Actual % Number of private local organizations participating in local PPPs for FP-MCH (CUMULATIVE) 20 22 110% 30 12 0 12 40% 10 60 34 57% Percent of DOH-CHDs with improved capacities to sustain program initiatives beyond project life (CUMULATIVE) 25% 25% 100% 75% 46% 46% 46% 61% 100% 100% 46% 46% ** revised definition *** data from Alphamed only (old definition) Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 119 REVISED Table 1 from PRISM2 Q14 Report (05-16-13) * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % USAID Development Objectives O1 Couple-Years of Protection (CYP) in USG￾supported programs 1,005,801 1,377,170 137% 700,316 IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 809,965 2,516,082 1,377,170 55% O2 Amount of in￾country public and private financial resources leveraged by USG programs for FP/RH (in US $) (CUMULATIVE) 198,327 623,634 314% 186,165 160,998 174,235 335,233 180% 25,155 409,647 958,867 234% O3 Number of current users of modern FP methods from participating service delivery points (MOVING) 1,835 5,574 304% 11,169 6,361 16,181 16,181 145% 17,379 16,440 16,181 98% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 120 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % O4 Number of pregnant women with at least 4 antenatal care (ANC) visits by skilled providers from USG-assisted facilities (CUMULATIVE) 8,128 5,398 66% 5,700 3,449 2,795 6,244 110% 2,516 16,344 11,642 71% O5 Number of deliveries assisted by skilled birth attendants (SBAs) in USG￾assisted programs (CUMULATIVE) 28,077 34,977 125% 19,689 9,250 11,019 20,269 103% 8,692 56,458 55,246 98% O6 Number of deliveries in USG-assisted health facilities (CUMULATIVE) 28,077 34,977 125% 19,689 9,250 11,019 20,269 103% 8,692 56,458 55,246 98% O7 Number of postpartum women who initiated breastfeeding within one (1) hour of delivery (CUMULATIVE) 0 0 N.A. 13,782 N/A 5,878 5,878 43% 6,084 19,866 5,878 30% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 121 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % O8 Number of people reached by a USG￾funded intervention providing gender-based violence (GBV) services (CUMULATIVE) 800 2,377 297% 9,733 797 1,239 2,036 21% 6,833 17,366 4,413 25% PRISM2 Development Objectives O9 Contraceptive Prevalence Rate (CPR) for modern methods obtained from private sector sources 54.40% 53.80% 99% 56.00% National survey data not available National survey data not available National survey data not available N.A. 57.60% 57.60% 53.80% 93% O1 0 Percent of deliveries with a skilled birth attendant (SBA) 65.40% 69.30% 106% 67.00% National survey data not available National survey data not available National survey data not available N.A. 68.60% 68.60% 69.30% 101% O1 1 Sales volume of oral contraceptives 12.71M 39.56M 311% 12.97M IMS Data will be available on October IMS Data will be available on October IMS Data will be available on October N.A. 13.23M 64M 40M 63% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 122 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % 2013 2013 2013 O1 2 Sales volume of injectable contraceptives 1,071,334 1,379,357 129% 400,729 IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 424,773 1,896,836 1,379,357 73% Intermediate Result 1 - FP/MCH Services P1 Number of service delivery networks strengthened (CUMULATIVE) 10 1 10% 20 1 2 3 15% 6 36 4 11% P2 Number of informal working groups implementing FP-MCH activities increased (NEW) 0 0 N.A. 200 N/A N/A 0 0% 25 225 0 0% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 123 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % P3 Number of USG-supported educational institutions and youth-oriented NGOs implementing AYRH/FP-MCH activities (NEW) 0 0 N.A. 22 8 6 14 64% 26 26 14 54% P4 Number of people trained in FP/RH with USG funds (CUMULATIVE) 150 295 197% 350 35 109 144 41% 100 600 439 73% FPCBT1 * 252 N.A. * 35 95 130 N.A. * * 382 N.A. FPCBT2-IUD * 56 N.A. * 6 20 26 N.A. * * 82 N.A. FPCBT2-BTL * 26 N.A. * 1 3 4 N.A. * * 30 N.A. P5 Number of people receiving FP Trainer’s Training with USG support * 4 N.A. 40 0 3 3 8% 20 60 7 12% P6 Established functional DOH training system on FP/RH * N.A. N.A. 3 0 0 0 0 2 5 0 0% P7 Number of people trained in MCH with USG funds 150 198 132% 350 34 286 320 91% 100 600 518 86% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 124 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % (CUMULATIVE) EINC * 198 N.A. * 37 221 258 N.A. * * 460 N.A. QAP for Midwives * 0 N.A. * 0 38 38 N.A. * * 38 N.A. BEMONC * 0 N.A. * 0 0 0 N.A. * * 0 N.A. Newborn Screening * 0 N.A. * 0 75 75 N.A. * * 75 N.A. Intermediate Result 1 - FP/MCH Products P8 Number of sustainable alternative distribution points (ADPs) for contraceptives products supported with USG-assistance (MOVING) ** 200 258 129% 400 209 143 143 36% 600 600 143 24% P9 Sales volume of USG-assisted contraceptive products (CUMULATIVE) 10,166,286 17,084,132 168% 5,258,466 IMS Data will be available on October 2013 IMS Data will be available on October 2013 IMS Data will be available on October 2013 N.A. 5,363,636 25,702,402 IMS Data will be available on October 2013 N.A. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 125 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % P10 Number of LGUs implementing their Contraceptive Self Reliance plans (MOVING) 130 282 217% 210 167 1 168 80% 210 210 168 80% P11 Number of distribution outlets for USG-assisted MCH products (MOVING) *** 30 74 247% 60 25 47 47 78% 70 70 47 67% P12 Sales volume of USG-assisted MCH products *** 303,380 596,496 197% 333,718 150,500 51,400 201,900 61% 367,090 1,004,188 647,896 65% P13 Number of new FP-MCH products introduced in the local market 3 4 133% Oxytocin in Uniject 0 0 0 0% 1 5 4 80% Intermediate Result 2 - Demand generation for FP/MCH Services Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 126 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % P14 Number of USG-assisted service delivery points providing FP counselling services (MOVING) 100 54 54% 250 149 208 208 83% 350 350 208 59% P15 Number of counselling visits for FP/RH as a result of USG assistance (CUMULATIVE) 5,100 12,700 249% 29,200 7,526 9,369 16,895 58% 20,500 54,800 29,595 54% Intermediate Result 3 - Policy and financing for sustainability of private sector FP/MCH service provision and use improved P16 Number of private practice midwives (PPMs) becoming PhilHealth accredited and accreditable as a result of USG￾60 73 122% 375 25 75 100 27% 70 505 173 34% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 127 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % assistance (CUMULATIVE) Accreditable 0 58 N.A. * 18 74 92 N.A. * * 150 N.A. Accredited 0 15 N.A. * 7 1 8 N.A. * * 23 N.A. P17 Number of Maternity-Care Package Accredited Health Facilities as a result of USG-assistance (CUMULATIVE) 15 41 273% 250 24 53 77 31% 85 350 118 34% Accreditable 0 25 N.A. * 19 48 67 N.A. * * 92 N.A. Accredited 0 16 N.A. * 5 5 10 N.A. * * 26 N.A. P18 Number of FP￾MCH claims filed for PhilHealth reimbursement (CUMULATIVE) 0 16,290 N.A. 6,000 4,646 5,697 10,343 172% * * 26,633 N.A. FP 0 637 N.A. * 307 321 628 N.A. * * 1,265 N.A. MCH 0 15,653 N.A. * 4,339 5,376 9,715 N.A. * * 25,368 N.A. Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 128 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % P19 Number of LGUs that submitted/issue d new local polices in support of private sector provision for FP-MCH services and products (CUMULATIVE) 10 18 180% 20 2 0 2 10% 6 36 20 56% P20 Number of national policies supporting private sector provision of FP￾MCH information, products, and services developed (CUMULATIVE) 14 28 200% 15 0 1 1 7% 5 34 29 85% P21 Number of private local organizations participating in local PPPs for FP-MCH (CUMULATIVE) 20 22 110% 30 12 0 12 40% 10 60 34 57% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 129 * no targets set for the year but actual accomplishments will be reported; N.A. - not applicable Indicator Year 1 to 3 Accomplishments Year 4 (FY 2013) Year 5 (FY 2014) TARGET CURRENT Accomplishments Target Actual % Target Actual Q13 Oct￾Dec 2012 Actual Q14 Jan￾Mar 2013 Total Year 4 % Five Year Target Actual % P22 Percent of DOH-CHDs with improved capacities to sustain program initiatives beyond project life (CUMULATIVE) 25% 25% 100% 75% 46% 46% 46% 61% 100% 100% 46% 46% Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 130 ANNEX O. RESOLUTION OF STATEMENT OF DIFFERENCES (SOD) Mission Statement of Differences SI Response to USAID SoD Mission’s Validation of SI Changes in Report Body 1. Recommendations, number 4, pp. ix “PRISM2 to provide the DOH with technical assistance to establish a system or mechanism to contract directly with institutions which have certified private trainers to promote FP/MCH” This suggestion has already been implemented under PRISM 2. The Centers for Health Development (regional offices of the DOH) in Region 7 and Region 5 were recipients of technical assistance from PRISM2, which resulted in DOH contracting/ outsourcing the training services to private institutions (UHA Zani Viv in Region 7 and Naga Foundation in Region 5). Comment 1: As DOH cannot contract with institutions but only with individuals, SI feels strongly about keeping this clause in the publishable report. A Ministry of Health respondent confirmed that a standard mechanism did not exist for DOH to outsource training services to private training institutions, and this was verified by interviewees at UHA and University of Bohol. No change (kept intact in report) 2. Findings, last paragraph, pp. x “Nonetheless, at least in terms of proposed objectives, activities, deliverables, and outcomes, gender is largely being implemented as a one-off activity entirely subsumed within the usapan discussion sessions.” Gender is not implemented as a one-off activity, since usapan sessions are continuously implemented by the project and a key part of the communication approach under PRISM2. Comment 2: SI has revised the statement as “…gender is being implemented largely as a standalone activity.” Change made in report (see p.x) “Nonetheless, at least in terms of proposed objectives, activities, deliverables, and outcomes, gender is being implemented largely as a standalone activity.” 3. Evaluation purpose, 1st paragraph, pp. 4 “To do so, the evaluation team has developed and applied qualitative and quantitative instruments and used these to collect data.” Comment 3: Though this language requiring a mini-survey was included in the SOW, the Mission approved SI’s work plan which did not include a mini-survey. Members of the team discussed the limitation of time for a more rigorous approach for collection and analysis Change made in report (see p.5, footnote 10) “Language requiring a mini-survey was Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 131 Mission Statement of Differences SI Response to USAID SoD Mission’s Validation of SI Changes in Report Body The quantitative instrument was not developed by the evaluation team. This was requested in the SOW. A planned mini-survey did not occur because the team felt there was limited time to conduct field work. of quantitative data derived from a mini-survey with the COR when the in-country scope was approved. However, a mini-survey was developed for a group of midwives since it was not possible to hold a focus group within a training course on usapan in Pampanga. This approach is now included as a footnote in the report. included in the SOW, though, the Mission approved SI’s work plan which did not include a mini-survey. Members of the team discussed the limitation of time for a more rigorous approach for collection and analysis of quantitative data derived from a mini-survey. However, a mini-survey was developed for a group of midwives since it was not possible to hold a focus group within a training course on usapan in Pampanga.” 4. Comparison Sites, 1st paragraph, pp. 6 “Prior to finalizing the site selection, USAID asked the team to consider visiting sites that are no longer receiving support directly from PRISM2. As one USAID official expressed it, “we would be interested/like to look at some of the sites of the original 77 to see why they may have been underperforming”.” The evaluation team did not visit non-PRISM sites as required in the SOW which explicitly stated that the team “visits some comparison sites not directly or currently serviced by PRISM 2” and that “visits and interviews in non-project sites (i.e., sites dropped from project coverage Comment 4: The team interviewed several LMAMs who were part of PRISM II’s original 77 sites. The team also planned to visit non￾PRISM II sites, but was unable to visit the selected sites in Bukidnon after being advised not to proceed due to security concerns related to bomb attacks in the region during that period of field work. SI can include this explanation as a footnote in the publishable report. Change made in report (see p.6, footnote 11) “The team talked to LMAMs that were part of the original 77 sites but did not continue when the scope was narrowed, and was unable to visit non￾PRISM sites in Bukidnon after being advised not to proceed because of Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 132 Mission Statement of Differences SI Response to USAID SoD Mission’s Validation of SI Changes in Report Body in year 3) will be conducted to compare performance”. security concerns related to bomb attacks in the region during that period of field work.” 5. Question 6, paragraph 8, pp.20 “Recently, however, the Mission may have over-reached its role in operations: The elimination of the field-based M & E and Communications positions and the new organizational structure appear to have exacerbated cross-regional and filed-technical communication challenges.” The change in the organizational structure was not a unilateral decision made by USAID, but was based on recommendations of PRISM2 to respond to developments in the project. For example, the completion of the Project Tracking and Results Information System (PTRIS) program allows easy uploading of data forms from the provinces up to the central office in Manila, which eliminated the need for M&E persons at the regional offices. However, it did require additional M&E personnel at the central office for a more thorough and acceptable review and analysis (not evident before) of the raw data coming from the field. Comment 5: SI has revised the first clause to read, “The elimination of the field-based M&E and Communications positions and the new organizational structure, have exacerbated cross-regional and field-technical communication challenges.” Change made in report (see p.20) “The elimination of the field-based M&E and Communications positions and the new organizational structure, have exacerbated cross￾regional and field￾technical communication challenges.” 6. Recommendations, number 1 to 4, pp. 28 The team provided general recommendations that do not respond to the SOW question about “what PRISM2 interventions can be considered good practices, which may be recommended for scaling-up in the future?”. USAID expected more discussions of interventions/ approaches that have potential to be good practices, as well as steps/suggested actions to integrate them into PRISM 2 Comment 6: SI’s recommendations are substantiated by findings, which portray usapan as having some promise (though data is lacking as to whether it should be replicated) and UHA as a program that should be replicated for other training courses. SI has added the following summative sentence, “Regarding key contributions and best practices, it is Change made in report (see p.28) “Regarding key contributions and best practices, it is important not to underestimate the innovativeness of Final Performance Evaluation of the Private Sector Mobilization for Family Health Phase II Project Final Report 133 Mission Statement of Differences SI Response to USAID SoD Mission’s Validation of SI Changes in Report Body and other projects. important not to underestimate the innovativeness of PRISM2.” PRISM2.” 7. Scope of Work, Work Performance Requirements, pp. 48 On page 48 of the report, under Work Performance Requirements, it was mentioned that “Within the first five days of in-country visit, the Team will finalize with USAID the quantitative and qualitative data collection instrument/s, sites to visit, and at least three (3) PRISM2 interventions to assess for cost-effectiveness.” PRISM2 interventions include the grants provided to some local organizations, where cost-effectiveness can be assessed. PRISM2 has detailed information and data on said grants’ activities and expenses but the cost effectiveness analysis was not performed by the evaluation team. Comment 7: SI does not believe it is meaningful to assess the cost-effectiveness of individual grants as contrasted with activities. It was not possible under the approved work plan to include data from all the grants comprising an activity, or alternatively to ensure that a sample of grants would be statistically representative of the whole group. No change (kept intact in report)