March 23, 2012 This publication was produced for review by the United States Agency for International Development (USAID) by Jean Meyer Capps, A. F. Qureshi, S. Israr and Mehboob Sultan. It was prepared by Management Systems International (MSI) under the Independent Monitoring and Evaluation Contract (IMEC). EVALUATION Family Advancement For Life and Health (FALAH) Evaluation Report USAID FAMILY ADVANCEMENT FOR LIFE AND HEALTH (FALAH) EVALUATION REPORT MARCH 23, 2012 This publication was produced for review by the United States Agency for International Development (USAID) by Jean Meyer Capps, A. F. Qureshi, S. Israr and Mehboob Sultan. It was prepared by Management Systems International (MSI) under the Independent Monitoring and Evaluation Contract (IMEC). FAMILY ADVANCEMENT FOR LIFE AND HEALTH (FALAH) EVALUATION REPORT Contracted Under No. GS-23F-8012H and Order No. AID-391-M-11-00001 Independent Monitoring and Evaluation Contract (IMEC) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Family Advancement for Life and Health – Evaluation Report i TABLE OF CONTENTS Introduction .................................................................................................................................... 1 The Development Problem and USAID’s Intervention in Response ......................................... 2 USAID’s Intervention in Response .................................................................................... 3 Changes in the FALAH Project ............................................................................. 5 Purpose of the Evaluation ............................................................................................................. 7 Evaluation Methodology ............................................................................................................... 7 Limitations of the Final Evaluation Methodology .............................................................. 9 Findings and Conclusions............................................................................................................ 10 Findings and Conclusions on Evaluation Question #1 ..................................................... 10 Findings and Conclusions on Evaluation Question #2 ..................................................... 12 Findings and Conclusions on Evaluation Question #3 ..................................................... 14 Findings and Conclusions on Evaluation Question #4 ..................................................... 15 Findings and Conclusions on Evaluation Question #5 ..................................................... 32 Findings and Conclusions on Evaluation Question #6 ..................................................... 34 Findings and Conclusions on Evaluation Question #7 ..................................................... 36 Recommendations ........................................................................................................................ 37 References ..................................................................................................................................... 39 Annexes ......................................................................................................................................... 40 Annex 1: The Impact of FALAH on Contraceptive Use and on Unmet Need for Family Planning ................................................................................................................ 40 Annex 2: Evolution of FALAH Implementation Districts, 2007-2011 ............................ 44 Annex 3: Initial Getting to Answers Table ....................................................................... 45 Annex 4: Evaluation Schedule and Itinerary of Evaluation Team Visits to FALAH Districts ............................................................................................................... 48 Annex 5: Documents Reviewed........................................................................................ 52 Annex 6: List of Participants in Interviews and Focus Group Discussions ...................... 54 Annex 7: Questionnaires and Guidelines .......................................................................... 63 Guidelines for Group Discussion With MWRA .................................................. 63 Interview Guidelines - Leadership Fellows ......................................................... 66 Interview Guidelines - Health Care Providers ..................................................... 68 Discussion Guidelines for Currently Married Men.............................................. 70 Discussion Guide for Religious Leaders.............................................................. 72 Discussion Guides for Lady Health Supervisors ................................................. 73 Interview Guide for Family Welfare Workers ..................................................... 76 Discussion Guide for Lady Health Workers’ Supervisors (LHS) ........................ 78 Interview Guides for Provincial and District Managers ...................................... 81 Annex 8: Result – D “Social Marketing of Family Planning Commodities” – Year wise Activities (provided by Population Council) ................................................... 84 Family Advancement for Life and Health – Evaluation Report ii LIST OF TABLES AND FIGURES Table 1: Project Summary ........................................................................................................... vi Table 2: Targets Versus Achievments on Project Indicators ......................................................... 11 Table 3: Availability of Information on Key Project Indicators by Source ................................... 15 Table 4: Brands of Contraceptives Provided Through FALAH .................................................... 30 Figure 1: Map of Project Area ....................................................................................................... vii Figure 2: Trends in Contracrptive Use in Pakistan, 1984-2007 ....................................................... 2 Figure 3: Change in Contraceptive Prevalence Rate ..................................................................... 13 Family Advancement for Life and Health – Evaluation Report iii ACKNOWLEDGEMENTS The evaluation team would like to thank Population Council and its partners in the Family Advancement for Life and Health project for sharing their time so generously with us to answer our numerous questions about the experiences and lessons learned implementing the project in an extremely challenging environment. The team would also like to thank the Government of Pakistan officials at the national, provincial and district levels, plus the health care providers in clinics and communities, especially the Lady Health Workers, Lady Health Supervisors, and mobilizers as well as the project beneficiaries who opened up to our team members and shared their perspectives on how Healthy Timing and Spacing of Pregnancies has had a positive impact on mothers, babies and families in Pakistan. Any discussion of government and social systems in Pakistan must acknowledge the instability in some parts of the country and the significant security challenges that are in the background at all times. The enthusiastic response of the highly qualified and experienced Pakistani health and population professionals in the country to embrace new and innovative approaches to long-standing public health challenges should be acknowledged and commended. USAID should also be commended for using this opportunity to help Pakistan as they move into the new phase of population and health planning and programs and address significant challenges posed by rapid devolution. Family Advancement for Life and Health – Evaluation Report iv ACRONYMS AIP Annual Implementation Plan BCC Behavior Change Communication BHU Basic Health Units CAM Communications, Advocacy and Mobilization CBO Community Based Organization CBV Community Based Volunteer CCFPS Client Centered Family Planning Services CE Chief Executive CF Constella Futures CMO Community Mobilization Officer CMP Compliance Monitoring Plan COP Chief of Party COTR Contracting Officer’s Technical Representative CPR Contraceptive Prevalence Rate CTO Cognizant Technical Officer CYP Couple Years of Protection DCNP District Coordinator, National Program (for PHC and Family Planning) DCLHW District Coordinator, Lady Health Worker (Program) DDPWO Deputy District Population Welfare Officer DOH District Officer, Health DOH Department of Health DDOH Deputy District Officer, Health DFID UK Department for International Development DFP District Focal Person DHDC District Health Development Centre DHMS District Health Management System DHIS District Health Information System DHQ District Headquarters (Hospital) DPWO District Population Welfare Officer DTC District Technical Committee EC/P Emergency Contraception/ Emergency Contraception Pill EDO Executive District Officer ESD Extended Service Delivery (Project) FALAH Family Advancement for Life and Health FGD Focus Group Discussion FP Family Planning FWC Family Welfare Center FWW Family Welfare Workers GIS Geographic Imaging System GOP Government of Pakistan GSM Greenstar Social Marketing HANDS Health and Nutrition Development Society HMIS Health Management Information System HPN Health, Population, and Nutrition HTSP Healthy Timing and Spacing of Pregnancy IEC Information Education Communication IMEC Independent Monitoring and Evaluation Contract IUCD Intra-Uterine Contraceptive Device Family Advancement for Life and Health – Evaluation Report v KP Khyber Pakhtunkhwa (Province) KPK Khyber Pakhtun Khuwa LAM Lactational Amenorrhea Method LHS Lady Health Supervisor LHV Lady Health Visitor LHW Lady Health Worker M&E Monitoring and Evaluation MDG Millennium Development Goals MICS Multi-Indicator Cluster Survey MNCH Maternal and Newborn and Child Health MOH Ministry of Health MOPW Ministry of Population Welfare NGO Nongovernmental Organization NIPS National Institute of Population Studies NP National Program NPFP & PHC National Program for Family Planning and Primary Health Care NRSP National Rural Support Programme NWFP Northwest Frontier Province OC/OCP Oral contraceptive/Oral Contraceptive Pills PAIMAN Pakistan Initiative for Mothers and Newborns PC Population Council PDHS Pakistan Demographic and Health Survey PHC Primary Health Care PHDC Provincial Health Development Centre PHS, MNCH Public Health Specialist of Maternal, Newborn and Child Health (Program) PMP Performance Monitoring Plan PPHI People’s Primary Health Care Initiative PSLM Pakistan Social and Living Standard Measurement Survey PWD Population and Welfare Department RHC Rural Health Center RHS(A) Reproductive Health Service (A) RSP Rural Support Program RSPN Rural Support Programmes Network RTI Regional Training Institution SDM Standard Days Method SM Social Mobilizers SOW Statement of Work STC Save the Children TOT Training of Trainers TPM Team Planning Meeting UDL United Distributers Pakistan Limited UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government VHC Village Health Committee WHO World Health Organization Family Advancement for Life and Health – Evaluation Report vi PROJECT SUMMARY The purpose of Family Advancement for Life and Health (FALAH) program was to reduce the unmet need for family planning and increase contraceptive use.1 Healthy Timing and Spacing of Pregnancy (HTSP) is an important component of the United States Agency for International Development’s (USAID’s) global population policy and was the basis of the approach in Pakistan. HTSP does not emphasize reduction of family size for controlling population growth, but repositions family planning as a health intervention integrated with maternal, newborn, and child health programs. FALAH’s major capacity building activities supported HTSP as a means to decrease unmet needs for family planning and reduce mortality in support of Millennium Development Goals (MDGs) four and five2 consistent with current international evidence-based guidelines.3 TABLE 1: PROJECT SUMMARY Implementing partners Population Council in partnership with Greenstar Social Marketing (GSM), Health and Nutrition Development Society (HANDS), Jhpiego, Mercy Corps, Rural Support Programmes Network (RSPN), and Save the Children-US. Cooperative agreement (or other term as applicable) Cooperative Agreement No. 391-A-00-07-01092-00 Project dates June 1, 2007 to May 30, 2012 (original). Original end date revised to March 31, 2012. Project budget Original $59, 999, 2374 and revised $48,424,5665 Project Location6 Punjab Province: Dera Ghazi Khan and Jhelum districts Sindh Province: Dadu, Ghotki, Larkana, Sanghar, Sukkur and Thatta districts Khyber Pakhtunkhwa (KP) Province: Charsadda, Swat, Upper Dir, Buner, Battagram, Lakki Marwat districts Balochistan Province: Gwadar, Jaffarabad, Khuzdar, Lasbella, Turbat, Zhob districts The map of 1 USAID Pakistan FALAH Evaluation Statement of Work (SOW) June 2011. 2 MDG 4: Reduction of under-5 mortality by two-thirds from 1990 to 2015; MDG5: Reduction in maternal mortality ratio by three-fourths between 1990 and 2015. Source: UN Millennium Project 2002-2006. 3 The Partnerships for Maternal, Newborn and Child Health, A Global Review of the Key Interventions in Reproductive, Maternal and Child Health (RMNCH), Geneva, Switzerland 2011. 4 FALAH Project Cooperative Agreement, Modification 1 5 FALAH project Final Evaluation SOW, 2011. 6 A table illustrating changes in the project districts over the life of the project can be found in Annex 2. Family Advancement for Life and Health – Evaluation Report vii Figure 1 shows the districts in which Population Council implemented the project. Family Advancement for Life and Health – Evaluation Report viii FIGURE 1: MAP OF PROJECT AREA Family Advancement for Life and Health – Evaluation Report ix EXECUTIVE SUMMARY Pakistan’s large and rapidly growing population challenges the country’s ability to provide essential services, economic opportunities, and a good quality of life for its citizens. Furthermore, high total fertility rates put children and women at greater risk for higher mortality and poor overall health. Most non-first births in Pakistan fall into the high-risk category, and infant and maternal mortality rates remain high. Evidence shows that longer intervals between births can reduce the incidence of both neonatal and maternal mortality. Pakistan instituted a comprehensive and detailed family planning program in the mid￾1960s to address the impact of poor birth spacing and high fertility and has continued to explore different approaches to expand family planning services, improve quality, and reach underserved areas. Family planning services were delivered primarily through the separate Population and Welfare Ministry and not connected to the Ministry of Health services. Decades of policy and donor support (primarily the United States Government [USG] through USAID) for family planning services increased contraceptive use from 9 percent of the population in 1984-85 to 30 percent in 2006-07.7 However, the 2006-07 Pakistan Demographic and Health Survey (PDHS) suggests that contraceptive prevalence has stalled at around 30 percent among Married Women of Reproductive Age (MWRA) with lower rates in impoverished areas of the country. One-fourth of married women have an unmet need for family planning services, 11 percent for birth spacing, and 14 percent for limiting births. Furthermore, the gap between ever use and current use of contraception (49 percent versus 30 percent) has never been as wide as reported in the 2006-07 PDHS. One of the major reasons for the dropout is the fact that only 33 percent of modern method users were informed about the possible side effects of contraceptives or problems with the method they were using, and only 29 percent were informed about what to do if they experienced side effects. Similarly, only 38 percent were informed about other methods available. These data imply that dropout could be reduced considerably if sufficient information about family planning methods was provided to contraceptive users.8 Although there is support for family planning at the policy level, and the Government of Pakistan (GOP) has an extensive network of public sector delivery facilities, these facilities reach only about a third of the country’s population.9 Approximately 70 percent of people are served by the private sector. Public sector facilities are plagued by a range of operational health system issues including, but not limited to, understaffing, high absenteeism, non-payment of wages, commodity stock outs, and lack of water that limit their ability to provide services. Private sector providers reach more people, but quality of care and high prices are ongoing concerns.10 In addition to issues of availability, quality, and cost of services, several other important barriers limit the ability of women in particular, but also of men, to access family planning services. Women’s overall limited independence and economic power to make important decisions, their limited mobility, and gaps in spousal communication constrain their use of family planning. Fear of contraceptive side effects is also a very important factor inhibiting both the use and continuation of certain methods. There are also widely held beliefs that family planning is morally wrong or unnatural. Among married women who do not 7 1984-85 estimates from the Pakistan Contraceptive Prevalence Survey (PCPS) by the Ministry of Population and Welfare; 2006-07 estimates from the Pakistan Demographic and Health Survey (PDHS). 8 Ibid. 9 USAID PAIMAN Project Annual Report, 2005-2006. 10 2010 Pakistan MDG Report: Development Amidst Crisis. Family Advancement for Life and Health – Evaluation Report x intend to use family planning methods in the future, the most common reason is that fertility is “up to God.”11 At the policy level, the GOP has committed itself to universal access to reproductive health and gender equity. It is a signatory to the MGDs, and the draft (but not approved) National Health Policy includes a focus on provision of family planning services through the healthcare network and Lady Health Workers (LHW) program. Pakistan adopted a new Population Policy in 2010, and it also prioritizes family planning. USAID awarded the five-year FALAH project to the Population Council in 2007. The goal of the project is to increase the use of family planning and birth spacing methods in Pakistan by removing cultural and other barriers to access, improving understanding of the value of family planning for family health and well-being, increasing knowledge of methods of birth spacing, and improving access to and quality of care in both the public and private sectors. The project had the following specific objectives12:  Use of modern contraceptive methods in the project districts will increase by an average of 10 percentage points by year five.  Eighty percent of public service-delivery points in the project districts will be equipped to provide appropriate family planning services by year five.  Birth spacing and ways of achieving it will be understood and reported as acceptable by three￾fourths of the target population (married women of reproductive age and their husbands).  Public and private sector delivery systems will have sustainable and well-integrated service strategies in the project districts.  Plans for scale up of cost-effective innovations will be articulated and approved by health authorities, with arrangements for financing specified. The project pursues these objectives through four areas of activity or results:  Result A - Increasing acceptance and demand for birth spacing and contraception. Through a national advocacy campaign, FALAH sought to reposition the family planning message from reducing family size to a health intervention essential to reducing maternal and infant mortality. The campaign targeted national and provincial policy makers, government health professionals, and religious scholars and leaders to share new evidence about the health impacts of family planning and discuss their respective roles in improving acceptance and use of family planning in Pakistan. It also used media campaigns and community mobilization to reach the community, household, and individual levels.  Result B - Revitalizing delivery of high quality family planning services in the public sector. At the policy level, Result B focused on creating an enabling environment for working with the public sector by promoting inclusion of HTSP in national family planning policies. Under this result, FALAH also trained health care professionals in family planning methods and practices, strengthened the Lady Health Worker (LHW) program, and trained health facility staff in contraceptive logistics management. 11 DPHS, 2006-07. 12 FALAH Performance Monitoring Plan (Final) November 2009, Updated January 2010 Family Advancement for Life and Health – Evaluation Report xi  Result C - Reinforce high quality family planning services through the private sector. To enhance the quality of private sector family planning services, FALAH mapped public and private sector health care providers to improve district level planning, piloted a Community Based Volunteer (CBV) approach to improve access to HSTP for individuals, and partnered with the Rural Support Programs Network (RSPN) to increase demand for family planning services provided by private sectors suppliers.  Result D - Expansion of family planning services and quality improvement initiatives in the private sector. Through involvement with the FALAH project, Greenstar Social Marketing (GSM) was able to expand its reach outside of urban areas to improve the quality of family planning services to rural franchised providers. GSM trained its service providers to improve the quality of family planning services including Intra-Uterine Contraceptive Device (IUCD) insertion, infection prevention, comprehensive and compassionate counseling, cleanliness, competent treatment, and good management. GSM also collaborated with the Ministry of Population Welfare (MOPW) to organize Sahoolat clinics where private providers offered free contraceptives and services once a month. In November, 2011, USAID commissioned Management Systems International to conduct a final evaluation of the FALAH project to assess the effectiveness, quality, and sustainability of program components designed to reduce the unmet need for family planning and increase the use of modern methods of contraception in the context of HTSP. The evaluation also documented lessons learned for future programming, identified elements that were likely to be sustainable, and highlighted potential areas where the GOP and partners could integrate best practices from the project and scale them up to other parts of the country after the project ends. Key Findings and Conclusions  The FALAH program met or exceeded its targets for planned activities and outputs as outlined in the Cooperative Agreement and subsequent annual work plans and expressed in the approved annual Performance Monitoring Plan (PMP).  Data did not exist at the time of the evaluation13 to assess whether FALAH had achieved key outcome indicators (i.e. Contraceptive Prevalence Rate [CPR]), proportion of MWRAs with unmet need for family planning, proportion of total need met). Project activities, however, had all of the appropriate elements of an enabling environment (communication and social mobilization to increase demand, health worker skills and quality of care training, private sector strengthening to increase contraceptive supply and expand access to quality services, and a new policy environment) that are likely to have achieved these outcomes.  An impact analysis of the FALAH project completed by the Population Council after the evaluation was completed estimated an 8.5 percent increase in CPR across the 14 districts covered by the study. The survey report concluded that the proportionate increase is greatest among rural, uneducated, and the poor and youngest women in the sample. Unmet need decreased by three percent for the cross section of women and by 1.9 percent for panel women in the same period. Total demand for family planning services is now up to 70 percent (69 and 71 percent for cross sectional and panel women respectively). Furthermore, the proportion of satisfied demand has risen by 9 percent from 45 to 54 percent among the cross section of women. Annex 1 presents the results as reported by the Population Council. Because the evaluation was 13 A quantitative survey conducted by Population Council in project districts was underway at the same time the qualitative evaluation was done. Family Advancement for Life and Health – Evaluation Report xii complete and the team disbanded when the Population Council completed the study, the team was not able to assess the statistical significance of the reported changes or the validity of the analysis and results.  According to project documents and key informants who were top GOP officials in the Ministry of Health (MOH) and the MOPW, FALAH collaborated with key decision-makers and stakeholders at the national level, and contributed to repositioning family planning as a health intervention as part of an updated national population policy and strengthened national capacity to deliver high quality family planning services.14 The policy was developed to improve access to quality service delivery; increase advocacy, communication and demand generation; provide for program management and efficiency; and strengthen program monitoring and evaluation. Public and private sector key informants and stakeholders at the national and provincial levels acknowledged that the FALAH-facilitated policy changes that repositioned family planning to HTSP for mother and child health was a “game changer” in their opinion. They said that these significant changes removed many of the religious and cultural barriers to family planning acceptance that will probably contribute significantly to increased CPR. Results will depend on access to adequate supplies of affordable contraceptives for those who want them. This assumption will need to be validated through analysis of quantitative and behavioral studies at the population level over the next several years, including analysis of the Population Council’s quantitative final survey (that was completed in early 2012, but not in time to be included in this evaluation) and the (planned) 2012-2013 Pakistan Demographic and Health Survey.  Security threats in several project districts, a poor economic environment and insufficient supplies of contraceptives at most public health facilities have been the major obstacles to program coverage and access. FALAH did not directly provide contraceptives to either the public or private sectors. Indirectly, it supported its partners GSM, United Distributors (Pakistan) Limited (UDL) and ZAFA to ensure contraceptive availability.  Based on the consensus in responses from scores of interviews with FALAH partners, government policy makers, public and private health and family planning managers, health care providers, community-based providers (including LHWs and CBVs), MWRA and their husbands, and religious leaders from different sects in the four project evaluation districts that were selected to represent a range of Human Development Index (HDI) scores, there appear to be synergies of project activities within the four project results and no single activity emerged as being more valuable than another. The project’s PMP focused largely on operational indicators (activities and outputs) and had no indicators designed to measure behavioral change.  Policy changes, management training and health worker and beneficiary behavior change are probably the most sustainable aspects of the program. Other major aspects of the program such as pre-service and in-service clinical health provider training and engaging religious leaders as supporters of family planning and reproductive health could be sustained if follow-on programs build on the momentum started in FALAH and if the GOP takes ownership of them in their own programs and budget, especially at the provincial and district levels.  Starting in early 2011, after devolution, FALAH began assisting the GOP with the lengthy process of integrating health and population programs in Pakistan at the provincial and district level. Leadership training and assistance in revitalizing District Technical Committees (DTC) with health and population managers have increased collaboration and cooperation between 14 Government of Pakistan, National Population Policy, Ministry of Population and Welfare, January 2010 (Draft) Family Advancement for Life and Health – Evaluation Report xiii managers responsible for both health and family planning services and has shown promise in facilitating the transition towards integrated services.  When the Pakistani legislature abolished the two ministries, provincial and district government authorities associated with the FALAH project suddenly found themselves faced with significant management and financial decision-making responsibilities. Health systems strengthening capacity building provided by FALAH has assisted program managers to meet these new (and somewhat threatening) challenges. Despite the shortened time frame and constantly changing guidance to the project from USAID, FALAH was able to meet their targets for planned activities in the PMP. The FALAH staff and the evaluation team agrees, however, that these accomplishments are “only the beginning”. With just over three years of implementation most activities would not yet be expected to stand on their own without some type of external assistance. Summary of Key Recommendations To build on the momentum of FALAH’s contributions to introducing and scaling up HTSP in Pakistan, USAID should: Scale-up capacity in Pakistan based on lessons learned in FALAH  USAID|Pakistan’s Health, Population and Nutrition (HPN) office should seek ways to build upon, scale up, and sustain the HTSP training approaches developed by the FALAH project within Pakistan in future programs. It can do this by supporting capacity building of Pakistani training institutions to assume responsibility for transferring project capacity building skills from the project to training institutions within Pakistan.  Population Council should capture and analyze lessons learned and best practices identified in the promising community based approaches (LHWs, CBVs, Social Mobilizers [SMs], Sahoolat and mobile clinics) to reaching disadvantaged poor populations that were tested and implemented in the FALAH project. Recommendations for future programs based on these lessons learned should be included in end of project reports. Results of the third party evaluations of the pilot studies in the project should be disseminated amongst USAID’s HTSP partners.  USAID’s HPN office should build upon lessons learned and the enthusiastic response of religious leaders to the capacity building developed by the Extended Service Delivery (ESD) and FALAH projects to engage religious leaders in future HTSP and Maternal, Newborn, and Child Health (MNCH) and include female religious leaders where appropriate.  To promote integration of family planning and MNCH services and encourage synergies of the programs that USAID supports, USAID’s HPN office should clearly state their expectations for collaboration between partners they support in procurement documents such as Requests for Applications (RFAs), Cooperative Agreements, Annual Implementation Plans (AIPs), and PMPs and monitor collaborations during program implementation.  To help Pakistan meet the increased demand for and access to contraceptives created by FALAH and other USAID-supported family planning programs, USAID’s HPN programs should continue their strong support for sustainable contraceptive security in Pakistan, including the supply chain management support currently provided through USAID’s DELIVER project. Demonstrate program impact through strong monitoring and evaluation systems These programs should be required to have strong monitoring and evaluation plans from the beginning that include objectively measurable Social and Behavior Change (SBC) and sustainability indicators. Family Advancement for Life and Health – Evaluation Report xiv USAID should also consider requiring external midterm evaluations for these programs to determine if the project is making sufficient progress towards objectives in time to make corrections/adjustments before the end of the program. USAID Pakistan’s HPN office should strengthen Monitoring and Evaluation (M&E) systems to demonstrate program impact by:  Requiring future HTSP and family planning programs to have strong M&E systems with clear definitions of how coverage and access will be objectively measured using evidence-based indicators that also measure increases in knowledge and behavior change and sustainability.  Considering external midterm evaluations for programs to identify progress towards project goals and objectives and recommend adjustments in the program as needed.  Including measurements of the contribution of private sector and commercial capacity to increasing coverage and access.  USAID|Pakistan should continue support for the PDHS as it remains the only reliable population￾wide measurement of CPR and other key family planning and HTSP indicators and is used by most health and family planning programs in the country. Sustainability of FALAH Project Efforts USAID|Pakistan’s HPN office should support institutionalizing the HTSP model in national health and population programs throughout Pakistan by:  Future USAID|Pakistan’s HPN HTSP programs that include elements that are intended to continue after the project ends should be designed and implemented jointly with those who are intended to sustain the program (GOP, Nongovernmental Organization [NGO], private sector) from the beginning. From the beginning of the program, but before devolution, FALAH worked closely with central government policy makers from both the MOH and the MOPW and the relevant offices in the provincial governments to include their inputs in developing implementation strategies. Health managers at the district level stated that future programs should include them when developing implementation strategies.  Supporting efforts to integrate HTSP and Leadership Training into provincial and district plans and activities, including other health systems strengthening efforts by expanding Leadership and Management Training and support to additional DTCs.  Supporting development of Leadership Training within Pakistani institutions. As requested by USAID, the recommendations section of this report contains more detailed and broader recommendations for future HPN programs in Pakistan. Family Advancement for Life and Health – Evaluation Report 1 INTRODUCTION With an estimated population of 177 million,15 Pakistan is the sixth most populous country in the world. Rapid population growth in the face of declining available national resources is adversely affecting the quality of life of a large population.16 High birth rates contribute to the problem. According to the latest PDHS conducted during 2006-07, on average a Pakistani woman gives birth to four children during her reproductive life. One out of these four children is unwanted. The marital total fertility rate, however, is well above six children both in urban (6.4) and rural areas (6.8). The majority of non-first births (62 percent) fall into a high-risk category, with 38 percent in a single high-risk category and 24 percent in a multiple high-risk category.17 As a result, the infant mortality rate is high at 78 per thousand live births, with over 70 percent of these deaths occurring in the first 28 days after birth. Both neonatal and infant mortality rates are highest among rural, uneducated, and poor segments of the country where birth intervals are also shorter. The PDHS data also show that longer birth intervals reduce the incidence of neonatal mortality from 69 to 34 per thousand births when the birth interval is increased from less than 24 months to over 24 months. In Pakistan, 20 percent of all deaths of women age 12-49 years are caused by complications of pregnancy, childbirth, and puerperium while the maternal mortality ratio is estimated at 276 per one hundred thousand live births.18 Research in developing countries including Pakistan associates narrowly spaced pregnancies with multiple adverse outcomes including death of mothers and their newborns. Research also shows that pregnancy occurring within six months of a live birth increases the risk of induced abortion by 6.5 times; risk of miscarriage by 2.3 times; newborn death (<9 months) by 1.7 times; maternal death by 1.5 times; preterm birth by 0.7 times and still birth and low birth weight by 0.6 times.19 The World Health Organization (WHO) recommends an interval of 24 months from a live birth before attempting another pregnancy in order to reduce the risk of adverse maternal, perinatal, and infant outcomes.20 Further analysis of the PDHS data also indicates that with a 35 percent reduction in fertility alone, maternal mortality can be reduced by one-third.21 In order to reduce fertility, Pakistan started public sector family planning activities during 1960-65 through the infrastructure of health departments in the four provinces of Pakistan. Between 1965 and 1970 Pakistan developed a comprehensive and detailed family planning policy. In 2010 the family planning program was separated from the health sector. Because of devolution, however, the GOP has not yet adopted the policy. Since 2010, the program has continued, experimenting with different approaches to expand family planning services, improve quality, and reach out to underserved areas. In the health sector, the LHW program was also given the responsibility of providing family planning services at the household “door step” level. 15 2011 World Population Data Sheet by Population Reference Bureau, downloaded from www.prb.org. 16 Contech, Impact Evaluation of Community Based Volunteers, 2011. 17 Single high-risk category: If mother’s age is less than 18 years or mother’s age is more than 34 years; and birth interval is less than 24 months or child’s birth order is higher than three. Multiple high-risk category: If mother has two or more characteristics mentioned in the single high-risk category. 18 Pakistan Demographic and Health Survey, 2006-07. 19 Conde-Agudelo, et al, 2000, 2005, 2006; Da Vanzo, et al, 2004; Razzaque, et al, 2005; Rutstein, 2005. 20 WHO, 2006. 21 Sathar, Zeba, 2010. Presentation made at Planning Commission, Pakistan. Family Advancement for Life and Health – Evaluation Report 2 THE DEVELOPMENT PROBLEM AND USAID’S INTERVENTION IN RESPONSE Contraceptive use in Pakistan has increased over the past 22 years (Figure 1).22 However, the 2006-07 PDHS suggests that contraceptive prevalence has stalled at around 30 percent among MWRA. One-fourth of married women have an unmet need for family planning services, 11 percent for birth spacing, and 14 percent for limiting births. Unmet need is higher in Balochistan and Khyber Pakhtunkhwa (31 percent) provinces compared with Sindh (25 percent) and Punjab (23 percent). The use of family planning methods is especially low among poor, uneducated women, and in rural areas.23 FIGURE 2: TRENDS IN CONTRACRPTIVE USE IN PAKISTAN, 1984-2007 9% 12% 18% 24% 28% 32% 30% 0% 10% 20% 30% 40% 1984-85, PCPS 1990-91, PDHS 1994-95, PCPS 1996-97, PFFPS 2000-01, PRHFPS 2003, SWRHFP 2006-07, PDHS Percent of population The gap between ever use and current use of contraception (49 percent versus 30 percent) has never been as wide as reported in the 2006-07 PDHS. One of the major reasons for this dropout lies in the fact that only 33 percent of modern method users were informed about the possible side effects of contraceptives or problems of the method they were using, and only 29 percent were informed about what to do if they experienced side effects. Similarly, only 38 percent were informed about other methods available. These data imply that dropout could be reduced considerably if sufficient information about family planning methods was provided to contraceptive users.24 Although there is support for family planning at the policy level, and the GOP has an extensive network of public sector delivery facilities, these facilities reach only about a third of the country’s population.25 22 Pakistan Contraceptive Prevalence Survey (PCPS) by Ministry of Population Welfare; Pakistan Demographic and Health Survey (PDHS) 1990-91 and PDHS 2006-7, conducted by NIPS; Pakistan Contraceptive Prevalence Survey (PCPS) 1994-95 conducted by Population Council; Pakistan Fertility and Family Planning Survey (PFFPS)1996-97, Pakistan Reproductive Health and Family Planning Survey (PRHFPS) 2000-01, and Status of Women Reproductive Health and Family Planning Survey (SWRHFPS) 2003, conducted by National Institute of Population Studies. 23 PDHS, 2006-07. 24 Ibid. 25 USAID PAIMAN Project Annual Report, 2005-2006. Family Advancement for Life and Health – Evaluation Report 3 Approximately 70 percent of people are served by the private sector. Public sector facilities are plagued by a range of operational health system issues including, but not limited to, understaffing, high absenteeism, non-payment of wages, commodity stock outs, and lack of water that limit their ability to provide services. Private sector providers reach more people, but quality of care and high prices are ongoing concerns.26 In addition to issues of availability, quality, and cost of services, several other important barriers limit the ability of women in particular, but also of men, to access family planning services. Women’s overall limited independence and economic power to make important decisions, their limited mobility, and gaps in spousal communication constrain their use of family planning. Fear of contraceptive side effects is also a very important factor inhibiting both use and continuation of certain methods. There are also widely held beliefs that family planning is morally wrong or unnatural. Among married women who do not intend to use family planning methods in the future, the most common reason is that fertility is “up to God.”27 Multivariate analyses of couples’ responses to different fertility-related questions show that the strongest predictors of contraceptive use are a couple’s joint approval of family planning, the husband’s desire for no more children, and spousal discussion about family planning. Research suggests that a rural background, lack of education, and little communication between spouses may be the limiting factors in fertility decision-making and adoption of contraceptive use.28 At the policy level, the GOP has committed itself to universal access to reproductive health and gender equity. It is a signatory to the MDGs, and the draft (but not approved) National Health Policy includes a focus on provision of family planning services through the healthcare network and LHWs program. Pakistan developed a new Population Policy in 2010 that also prioritizes family planning, but it has not yet been adopted due to devolution. USAID’s Intervention in Response Using the backdrop of information provided in the 2006-07 PDHS, USAID’s FALAH project was designed to remove the major barriers to adopting family planning among those who wish to delay or limit childbearing. These barriers are the product of weak gender relations, social and religious taboos, fear of side effects, non-availability of quality services, limited contraceptive choice, and unaffordable costs. The FALAH program was managed by USAID’s HPN Unit and implemented under a Cooperative Agreement by Population Council in partnership with GSM, HANDS, Jhpiego, Mercy Corps, RSPN, and Save the Children-US. The original time period for the project was from June 1, 2007 until May 31, 2012. However, USAID reduced funding for the project with a revised end date of December 31, 2011 and then granted a no-cost extension through March 31, 2012. The overall purpose of FALAH program was to reduce the unmet need for family planning and increase contraceptive use.29 The HTSP approach was developed by USAID/Washington’s Office of Population and Reproductive Health, Bureau for Global Health to change the emphasis from using contraceptives to reduce the number of births and reduce population growth to delaying and spacing pregnancies to support 26 2010 Pakistan MDG Report: Development Amidst Crisis. 27 DPHS, 2006-07. 28 Mahmood, Naushin. Reproductive Goals and Family Planning Attitudes in Pakistan: a Couple-level Analysis. The Pakistan Development Review, 1998. 29 USAID Pakistan FALAH Evaluation Statement of Work (SOW), June 2011. Family Advancement for Life and Health – Evaluation Report 4 the health and survival of mother and child. HTSP is now one of the Office of Reproductive Health’s six technical priorities. HTSP also repositions family planning as a health intervention thus requiring family planning to be integrated with MNCH programs. Prior to devolution, which eliminated the MOH and MOPW, MNCH programs, under the MOH, were totally separated from the family planning programs that were implemented by the MOPW. FALAH intended to revitalize the family planning program and address stagnation of the CPR as well as address the high unmet need for family planning services documented in the 2006-07 PDHS. FALAH’s program documents state that the project was based on principles of building on existing evidence and experience, including and encouraging a greater number of partnerships, sharing responsibility, and ensuring that models that were implemented were sustainable. The project proposal and the Cooperative Agreement with USAID identified the major objectives to be achieved by the end of the project as:  Increased use of modern contraceptive methods in the 20 project districts by an average of 10 percentage points  Eighty percent of public sector service delivery points in the 20 project districts equipped to provide appropriate family planning services  Birth spacing and ways of achieving it understood and reported as acceptable by three-fourths of the target population  Public- and private sector delivery systems having sustainable and well-integrated service strategies in the project districts  Plans for scale-up of cost-effective innovations articulated and approved by health authorities, with arrangement for financing specified An additional Strategic Objective was added during the first year of the project (2007-2008):  Increase Couple Years of Protection by 50 percent per year in both social marketing and commercial sectors30 The overall goals of the FALAH project were to:  Improved understanding of the value of birth spacing (HTSP) for family health and well-being  Improved access to family planning and birth spacing services that are physically and socially appropriately placed in public and private sectors  Enhanced quality of family planning and birth spacing services Four strategies (called results) were designed to achieve these goals:  Result A - Create increased demand for family planning and birth spacing  Result B - Revitalize high quality birth spacing and family planning services in the public health sector  Result C - Reinforce high quality family planning and birth spacing services in the private sector 30 FALAH Project First Annual Implementation Plan (AIP), June 1, 2007 to May 31, 2008. Family Advancement for Life and Health – Evaluation Report 5  Result D - Social marketing of family planning commodities FALAH identified married women of reproductive age (MWRA) between 15 and 49 years of age and their husbands as the beneficiaries of the program. Changes in the FALAH Project The project modified its approach several times over the life of the project for a variety of reasons. FALAH was originally funded to work in 20 districts31 in all four provinces of Pakistan. In Year 3 (July 1, 2009 to June 30, 2010), project operations were selectively expanded into six additional districts. When USAID notified Population Council on July 31, 2010 that it would terminate funding on December 31, 2011, the project limited its remaining activities to 15 districts during the final 18 months of the program. The reduced implementation period (60 months to 54 months) presented challenges to complete several planned activities, especially quality assurance and training follow-up. USAID reduced the project duration in response to a change in USAID policy to direct more resources to direct government to government support. Security considerations also caused the number and location of project districts to vary from year to year. Consequently, specific project activities were also not implemented for the same amount of time in every district. Annex 2 provides the list of districts for each year of the project. FALAH start-up activities were delayed when some family planning compliance issues related to Tiahart Amendment violations associated with family planning method targets were detected in the public sector component of the project. A rapid assessment conducted in June 2008 confirmed that there were Tiahart Amendment violations at GOP Health and Population Welfare facilities in many of the intervention districts.32 The GOP, USAID, and FALAH took immediate action to address this issue. The Ministry of Health immediately sent a letter to all health posts stating that volunteerism without incentives for targets was GOP policy and that there were to be no targets or coercive activities in GOP programs. FALAH suspended its family planning activities in the public sector and worked with the MOH, MOPW, and local partners to develop a Compliance Monitoring Plan (CMP) and disseminate modified and new policies that complied with the Tiahart Amendment in all of the program’s 20 intervention districts. FALAH resumed family planning activities in the public sector at the end of 2008.33 Constella Futures was the original lead partner for Result A, but elected to withdraw from the project following a disagreement with USAID/Washington about fees.34 Since activities under Result A were 31 Dera Ghazi Khan and Jhelum districts (Punjab Province); Dadu, Ghotki, Larkana, Sanghar, Sukkur and Thatta districts (Sindh Province); Charsadda, Swat, Upper Dir, Buner, Battagram and Lakki Marwat districts (Khyber Pakhtunkwa (KP) Province); Gwadar, Jaffarabad, Khuzdar, Lasbella, Turbat and Zhob districts (Balochistan Province). 32 The Tiahart Amendment prohibits organizations receiving US family planning assistance in the form of funding, technical assistance, commodities, or training from 1) setting quotas or targets for the of total number of births, number of family planning acceptors, or acceptors of a particular method of family planning, 2) paying incentives, bribes, gratuities, or financial reward to individuals to become family planning acceptors or to program personnel to achieve quotas or targets, 3) denying rights or benefits as a consequence of an individual's decision not to accept family planning, and 4) using experimental contraceptive drugs and devices and medical procedures outside of a scientific study in which participants are advised of potential risks and benefits. Furthermore, the amendment requires that projects provide family planning acceptors with comprehensible information on the health benefits and risks of the method chosen. 33 Background information provided in the FALAH Final Evaluation Scope of Work, April 25, 2011, dated June 16, 2011. 34 For-profit companies must normally agree to waive their profit “fees” to participate in USAID Cooperative Agreements. Family Advancement for Life and Health – Evaluation Report 6 critical for other activities to start and other partners were extremely concerned, Population Council along with USAID and the other consortium partners agreed that Constella Futures would leave the consortium and Population Council would assume the role of lead partner for Result A.35 Although devolution of health and population services in Pakistan had been “on the books” for many years, it was not until the Pakistani legislature passed the 18th Amendment in April 2010 that control of family planning programs truly passed to provincial and district governments. The MOH and the MOPW at the central level were abolished, and all responsibilities for programs were abruptly devolved to the provinces and districts in 2011. Fortunately, FALAH had already completed much of the national advocacy and policy revision work that was necessary to introduce HTSP before the ministries were abolished. This meant that activities corresponding with project strategies were already at the district level early in project implementation. Because the project was designed under the old government structure and was “district-focused,” FALAH placed relatively little emphasis on the provincial level until the final 18 months of the project. The FALAH project had representatives and offices in provincial capitals established by the implementing partners such as Save the Children and HANDS. Population Council did not have offices of their own in the provincial capitals. The pace of implementation of FALAH activities in terms of roll-out of activities and budget expenditures started slowly36 and increased dramatically in the last two years up until the end of 2011.37 Late start-up was attributed to several factors, including Constella Futures, a key partner, withdrawing from the project and the unanticipated need to develop a CMP.38 Major changes in national family planning health structures and policies heavily impacted priority activities of the project. Prior to the GOP declaring devolution through the 18th Amendment, however, the MOH had already issued a policy on delivery of family planning services and eliminated targets for male SMs and monitoring performance of individual service providers. FALAH partners facilitated and monitored implementation of these policy changes and shifted more emphasis to the provincial level officials who were not the focus for most of the program. Delays in project start-up and the reduced timeframe required FALAH to introduce many activities late and implement them rapidly in order to finish before the revised end date of December 31, 2011.39 Also as a result of the compressed time-frame, USAID advised starting the end-line survey earlier than planned and eliminating the planned mid-term review. As a consequence, quantitative population￾based data measuring progress towards project objectives were not available until the end of the project. Floods displaced the population in a few districts, especially in Sindh Province, and interrupted project activities in these districts. FALAH provided job aids and supplies to displaced Lady Health Workers in response to the floods. 35 FALAH project First Annual Report, 2007-2008. 36 For example, the Year 1 Annual Report showed expenditures of $3.1 million against a total project budget (at the time) of over $55 million. 37 FALAH project Year 3 and 4 Annual Reports. 38 FALAH project Year 1 Annual Report. 39 FALAH Project Annual Reports Years 1-4 and Key Informant Interviews with FALAH and partner staff, former USAID staff and with former Ministry of Health and Ministry of Population and Family Welfare officials (now working in other government offices). Family Advancement for Life and Health – Evaluation Report 7 PURPOSE OF THE EVALUATION As part of its due diligence, USAID/Pakistan commissioned a final evaluation of the FALAH project to assess the effectiveness of the program components and the resulting impact on reducing unmet need for family planning and increasing contraceptive use, document lessons learned, and identify areas where the GOP could integrated best practices and scale-up. The Statement of Work (SOW)40 included the following questions: 1. To what extent has the FALAH program accomplished its planned activities and outputs as outlined in the Cooperative Agreement and subsequent annual work plans? 2. To what extent has the FALAH program achieved its key outcomes (increased CPR), reduced proportion of Married Women of Reproductive Age with unmet need for family planning, increased proportion of total demand met, increased proportion of MWRA who have discussed family planning with husband, and increased proportion of MWRA intending to use in the future?) 3. What are the trends in FALAH outcome indicators (CPR, proportion of MWRA with unmet need for family planning, proportion of total need met, etc.) in project districts in Pakistan? 4. To what extent is the FALAH program’s technical approach relevant to achieving planned project outcomes in the 20 districts included in the evaluation? 5. What have been the major obstacles to program coverage and access, and what does this imply about how USAID and other donors can facilitate demand for, and utilization of, family planning methods in rural and higher poverty areas? 6. What factors have affected the effectiveness, coverage, quality, and sustainability of FALAH family planning activities and what are the lessons for how USAID designs future programming in Pakistan? 7. To what extent are project results likely to be sustainable and what could the GOP do to ensure continuity, integration, and scaling up of FALAH’s technical advances in project districts?41 EVALUATION METHODOLOGY Management Systems International (MSI) conducted a final evaluation of the FALAH project between November 2011 and January 2012 under the Independent Monitoring and Evaluation Contract (IMEC). The evaluation focused on assessing the effectiveness, impact, and sustainability of FALAH program components. The evaluation incorporated quantitative assessments commissioned by Population Council for selected interventions, but otherwise relied largely on qualitative evidence since little relevant and timely quantitative evidence exists in Pakistan. Population Council completed a follow-up to its baseline surveys in March 2012, after the evaluation was completed. Annex 1 summarizes these results. The evaluation did not focus on cost effectiveness, but where proven cost-effective family planning and health interventions were implemented, the project evaluation team highlighted them in its findings. The evaluation also documented lessons learned, assessed sustainability, identified areas where the GOP could 40 The evaluation Statement of Work is a separate annex to this report. 41 USAID FALAH Statement of Work (SOW), December 2011. Family Advancement for Life and Health – Evaluation Report 8 integrate best practices and scale-up after the project ends, and made recommendations for future USAID reproductive health programs in Pakistan. The evaluation team consisted of an expatriate Reproductive Health Specialist as Team Leader and three experienced Pakistani public health professionals in the roles of Behavior Change Communication (BCC)/Community Health Specialist, Family Planning/MNCH specialist, and Population and Welfare Specialist. IMEC staff provided support and technical supervision. Upon gathering in Islamabad, the team met in a three-day team planning meeting (TPM) to plan the evaluation activity and logistics. Key elements of the TPM included meeting with USAID to discuss mission objectives for the evaluation; meeting with Population Council for project briefing; developing the Getting to Answers matrix (see Annex 3) to guide data collection; developing data collection instruments and protocols; and planning field logistics. The evaluation team collected data through document review;42 key informant interviews with program implementers; group discussions with project-trained LHWs, religious leaders, community mobilizers, and female project beneficiaries; and individual interviews with implementing partner staff and government representatives. Group discussion participants were selected from communities exposed to FALAH interventions and chosen by FALAH project partners who were responsible for project implementation.43 The team visited sites in four districts selected in consultation with USAID and Population Council to represent a range of socioeconomic characteristics and implementation modalities while also considering the local security situation. These districts were Jhelum, Charsadda, D.G. Khan and Thatta (see map in Figure 1). Categories of informants consulted included:  Beneficiary MWRA  Husbands of MWRA (but not necessarily the husbands of the MWRA interviewed by the team)  Health and Family Welfare Service Providers  FALAH partners  Religious leaders  Lady Health Workers and Lady Health Supervisors  Community Mobilizers and Social Mobilizers  Family Welfare Training Institute instructors and students 42 Annex 5 contains a list of the documents the team reviewed. 43 Annex 6 contains the names and locations of group discussion participants. Annex 7 contains the data collection instruments developed for the group discussions and key informant interviews. Family Advancement for Life and Health – Evaluation Report 9  Donors  GOP health and population/welfare officials, and  Private Sector Pharmacists Annex 4 contains the evaluation team’s itinerary and a complete list of meetings and interviews. The team presented preliminary results to Population Council for feedback and clarification and solicited ideas for ways that experiences gained in implementing the project could be applied to future programs in Pakistan. The team presented major findings, conclusions, and recommendations in debriefings to the USAID/Pakistan HPN representative in Islamabad and USAID in Washington, DC. This report summarizes these findings and responses to questions posed during these debriefings. Population Council conducted baseline surveys of randomly selected MWRA and husbands of MWRA44 in each district, but these surveys did not take place at the same time due to different starting dates in each district. Survey reports disaggregated results by several demographic factors, including rural/urban place of residence, but reported results only at the district level and not by province or across the project area. In November 2011, Population Council began endline surveys in the same districts and with the same households that were surveyed in the baseline. Preliminary reports provided to the evaluation team prior to the completion of the surveys in all districts show promising results in terms of increases in CPR for these couples and households.45 Limitations of the Final Evaluation Methodology Because Population Council was just starting data collection for their endline surveys as the evaluation team finished its work, the team could not take most of the quantitative findings into account for their conclusions. Annex 1 contains complete results received after the evaluation was completed. By the time Population Council had completed the surveys, the evaluation team had disbanded and was not able to assess the quality of the results. The evaluation report does not, therefore, include the results as formal findings. FALAH provided preliminary data on three of the project districts and these are included in the evaluation report, but there was insufficient time for the evaluation team to critically analyze and interpret these findings. The evaluation lacked timely data on key family planning-related indicators. The 2006-07 PDHS provides the best available secondary data on national CPR and other family planning-related indicators. However, the 2006-07 PDHS is not timely enough to be relevant to the evaluation. Other national data sources, including the United Nations Population Fund (UNFPA) Multi-Indicator Cluster Survey (MICS), do not use comparable sampling methodologies and most are outdated for evaluation purposes. The planned 2012 PDHS will be an essential source to determine changes in key family planning behaviors and practices at the provincial and national levels. Evaluation fieldwork, including interviews of key personnel and partners, took place several months after many project activities ended and many key personnel and partner management staff were no longer available for interviews. Several project reports, especially those from project partners who had ceased operation at the time of the evaluation, were not available when they were requested by the evaluation 44 Not necessarily the husbands of the MWRA who were interviewed. 45 Annex 1 contains survey results provided by Population Council after the conclusion of the evaluation. Since the evaluation team had disbanded, team members could not assess the quality of the analysis or the results. They appear in the annex as provided by Population Council. Family Advancement for Life and Health – Evaluation Report 10 team. MOH and MOPW officials who were involved when the key national policies were changed are now working for other government ministries. The evaluation team was able to locate and interview some, but not all, of these individuals. Overall, the evaluation team used the best available information from existing sources and from those who now work for USAID, FALAH partners, donors, and GOP offices, but the team was unable to completely address some important evaluation considerations, especially lessons learned from pilot studies and community mobilization. The last year of the project (2011) was a time of significant change and uncertainty in Pakistan’s health and population programs, as well as in the overall national political environment. Security considerations were always in the background, both in the implementation of the program as well as during the evaluation. For some of the fieldwork, only Pakistani nationals could visit the districts included in the evaluation. In other situations, interviews were held when participants were able to travel to a safe location to meet with the evaluation team. Passage of the 18th Amendment that abolished the two major project stakeholders at the national level required the project to adjust programming, and government officials could not tell with certainty what was going to happen within their programs in the near future. FINDINGS AND CONCLUSIONS Findings and Conclusions on Evaluation Question #1 Evaluation Question 1: To what extent has the FALAH program accomplished its planned activities and outputs as outlined in the Cooperative Agreement and subsequent annual work plans? Findings The project’s PMP and Annual Reports identify and track a number of indicators. These relate primarily to Couple Years Protection (CYP) (based on the number of contraceptives distributed by project partners), numbers of individuals of different cadres included in various project trainings, targets for communication strategies, and updated curricula and skills training. As part of USAID’s Substantial Involvement clause in the Cooperative Agreement, USAID would have had to approve the project’s Monitoring and Evaluation plan. Therefore, the evaluation team used the indicators in the PMP as the basis for judging FALAH’s targets and achievements. The evaluation team found no direct link between these indicators and four additional objectives that appear in the Cooperative Agreement (but not in the PMP). Nor did the team obtain any information from project documents or team briefings about measuring progress against these objectives. The objectives include:  Eighty percent of public sector service delivery points in the 20 project districts equipped to provide appropriate family planning services.  Birth spacing and ways of achieving it understood and reported as acceptable by three-fourths of the target population.  Public and private sector delivery systems will have sustainable and well-integrated service strategies in the project districts.  Plans for scale-up of cost-effective innovations will be articulated and approved by health authorities, with arrangement for financing specified. Using the PMP indicator targets as the basis, FALAH either met or exceeded all agreed-upon end-of￾project targets. The PMP did not include indicators of CPR and unmet need and the evaluation team found no assessment of either measurement since the project’s district level baseline surveys conducted in Family Advancement for Life and Health – Evaluation Report 11 2008.46 Table 2 shows key PMP indicators and achievements against targets provided to the evaluation team by the FALAH project. TABLE 2: TARGETS VERSUS ACHIEVEMENTS ON PROJECT INDICATORS Indicators Revised 5 Year Targets Achieved (2007-2011) Achieved as of December 31, 2011 Percentage of 5 Year Target CYP in USG-supported programs 7,903,202 7,962,910 101% Number of people that have seen or heard a specific USG￾supported family planning/reproductive health messages 8,640,697 9,119,273 106% Number of married women reached through household visits for IPC (Interpersonal Counseling) on family planning and birth spacing by GSM 1,044,472 1,245,430 119% Number of married women and men reached through neighborhood meeting on family planning and birth spacing by GSM Men 1,040,517 1,052,998 101% Women 664,611 716,587 108% Total 1,705,128 1,769,585 104% Number of influencers reached for orientation visits on family planning and birth spacing by GSM Men 33,456 35,851 107% Women 41,700 49,320 118% Total 75,156 85,171 113% Men reached through group meetings by SMs of MOPW 306,913 306,913 100% Men reached through Village Health Committees (VHCs) No original targets 41,516 No original targets Women reached through group meetings by female Community Mobilization Officers (CMOs) 94,800 94,800 100% Men reached through group meetings by male CMOs 84,706 84,706 100% Men and women reached through interactive theatre Men 51,537 45,360 88% Women 62,744 64,245 102% Total 114,281 109,605 96% Women reached through group meetings by LHWs 1,825,560 1,952,710 107% Men and women reached through CBVs Men 78,300 81,190 104% Women 78,300 119,349 152% Total 156,600 200,539 128% Number of people that have seen or heard a message through the TOUCH campaign 3,233,081 3,233,081 100% Number of people trained in family planning/reproductive health with USG funds Public Sector Trainings: Total 26,963 26,823 99% Master trainers trained in Client Centered Family Planning Services (CCFPS) - Basic Training 115 115 100% Doctors/paramedics trained in CCFPS Basic Training 3,622 3,622 100% Master trainers trained in CCFPS Advanced Training 72 72 100% Female doctors /paramedics trained in CCFPS Advanced 1,527 1,706 112% Master trainers trained in IUCD skills 37 37 100% 46 Population Council was not scheduled to update the baseline before the end of the project. It was beginning to conduct the follow-up survey at the time of the evaluation. Family Advancement for Life and Health – Evaluation Report 12 Indicators Revised 5 Year Targets Achieved (2007-2011) Achieved as of December 31, 2011 Percentage of 5 Year Target Female doctors/Leahy Health Visitors (LHVs)/midwifes trained in IUCD standard insertion and removal skills 854 905 106% Female master trainers trained in Minilap 23 23 100% Step down trainers trained in Minilap 19 17 89% Male master trainees trained in vasectomy 6 6 100% Step down trainers trained in vasectomy 23 23 100% Master trainers trained for LHWs 114 136 119% Master trainers trained for Emergency Contraceptive Pills (ECP) No original targets 16 No original targets LHWs trained in ECP 3,055 3,055 100% Providers trained in logistics management 747 751 101% Master Trainers trained for Lady Health Supervisors (LHS) No original targets 45 No original targets Social Mobilizers trained in interpersonal communication 400 400 100% LHW/LHS trained on CCFPS 15,213 14,299 94% Leadership Training of Health and Population managers 356 326 92% LHS trained 500 511 102% Master trainers trained for community midwives 48 48 100% Training of Trainers (TOT) for CBVs/VHCs No original targets 94 No original targets Training on Standard Days Method (SDM) for Family Welfare Workers (FWWs) of Population Welfare Departments (PWDs) No original targets 139 No original targets Refresher training for CCFPS advance and IUCD No original targets 389 No original targets Number of Sahoolat Clinics 4,462 4,451 100% Number of clients at Sahoolat Clinics 98,522 106,501 108% Private Sector Training: Total 8,337 8,414 101% Number of providers given refresher trainings 7,343 7,413 101% New Providers Training Training of new female providers in 26 districts (GSM) Rural 479 489 102% Urban 515 512 99% Total 994 1,001 101% Number of USG-assisted service delivery points providing family planning counseling or services 11,619 11,626 100% Note: Original FALAH documents, from which this table was derived, use the term Lady Health Visitor (LHV) instead of the more common term in Pakistan, Lady Health Worker (LHW). The terms are synonymous and LHV is retained in this table to be true to the original project documents. Conclusions  FALAH met or exceeded all project activity and output targets contained in the project PMP. The evaluation team could not determine whether the project had met four additional objectives listed in the Cooperative Agreement. Family Advancement for Life and Health – Evaluation Report 13 Findings and Conclusions on Evaluation Question #2 Evaluation Question 2: To what extent has the FALAH program achieved its key outcomes (increased contraceptive prevalence rate [CPR], reduced proportion of MWRA with unmet need for family planning, increased proportion of total demand met, increased proportion of MWRA who have discussed family planning with husband, and increased proportion of MWRA intending to use in future)? Findings Population Council is relying largely on a comparison of baseline and follow-up surveys in project districts to estimate changes in the five key outcome indicators mentioned in this evaluation question. It conducted the baseline in 2009 in 27 project districts, but was just beginning to conduct follow-up surveys as the evaluation concluded. Since no other sources provide timely or geographically relevant data on these indicators, the evaluation team had little evidence with which to address this question. The evaluation team based its finding for this question on two sources of information.  Preliminary comparisons of baseline and follow-up survey data provided by Population Council for three of the districts that the evaluation team visited (Jhelum, Charsadda, and D.G. Khan). The team did not have time to assess the quality of the data or the estimates.  The team’s assessment of whether the project’s infrastructure and capacity building activities could reasonably be expected to result in the desired positive changes based on international family planning and maternal child health program standards. Tiahart compliance issues prohibited contraceptive targets within activities conducted with the public sector. This was the reason that FALAH gave for providing CYP numbers for only private sector activities in their PMP. Population Council collected baseline estimates of CPR for a randomly selected cohort of MWRA and men in 27 project districts in 2009. The survey instrument included questions designed to address a number of key outcome indicators. These included CPR, unmet need, proportion of total demand met, proportion of MWRA who have discussed family planning with their husbands, and intentions to use family planning in the future. Population Council delivered summary estimates of CPR from three project districts to the evaluation team just as the team was completing the evaluation. The estimates contained no details of the analysis. The evaluation team was not able, therefore, to assess the quality of the estimates. The preliminary data that Population Council provided to the evaluation team show an increase in CPR in all three districts – from 38 percent to 42 percent in Jhelum, 37 percent to 47 percent in Charsadda and 27 percent to 33 percent in D.G. Khan. 47 Figure 3 summarizes results. FIGURE 3: CHANGE IN CONTRACEPTIVE PREVALENCE RATE 47 The results were preliminary and did not include measures of statistical significance. See Population Council Endline Survey Report for results from all project districts. Family Advancement for Life and Health – Evaluation Report 14 38% 37% 27% 42% 47% 33% 0% 10% 20% 30% 40% 50% Jhelum Charsada DG Khan Contraceptive Prevalence Rate (%) Baseline (2009) Endline (2012) Sufficient quantitative data were not available at the time of the evaluation to draw conclusions on the project’s impact on the target population in terms of CPR, unmet need, MWRA discussing family planning with husband, or intention to use family planning in the future. FALAH tested some approaches to overcome known barriers to using family planning, but the only CPR data in project documents came from reports (some still in draft) of third party evaluations that FALAH commissioned for these pilots. FALAH implemented the interventions covered in these studies over very short time periods and in a limited number of districts.48 Due to the limited scope and population coverage, the results cannot be generalized to the entire FALAH project target population. The reports, however, identified some promising practices and included recommendations for investigations that should be undertaken in the future. Conclusions  From the data available, it was not possible for the evaluation team to determine whether CPR increased, if MWRA with unmet need for family planning decreased, what portion of total demand was met, what portion of MWRA have discussed family planning with their husbands, or if the intent to use family planning in the future increased. The endline population survey in each district that will be conducted by Population Council is designed to answer those questions. Results should be available in early 2012 (see Annex 1). Findings and Conclusions on Evaluation Question #3 Evaluation Question 3: What are the trends in FALAH outcome indicators (CPR, proportion of MWRA with unmet need for family planning, proportion of total need met, etc.) in project districts in Pakistan? Findings While the FALAH project tracked implementation process indicators through its PMP indicators (Table 2), the evaluation team found no evidence that the project tracked CPR, MWRA with unmet need or proportion of unmet need in the FALAH project overview presentation, PMP, AIPs, or Annual Reports. 48 Contech, Impact Evaluation of Community-Based Volunteers (draft), December 2011 and Shafaat, M. et al., Impact of Lady Health Worker’s Client Centered Family Planning Services Approach Training: a Third Party Evaluation, 2011. Family Advancement for Life and Health – Evaluation Report 15 The 2010 Midterm Review revised report mentioned plans for a rapid survey in 2010, but neither the subsequent AIP nor the Annual Report mention the survey or results. The current FALAH Chief of Party (COP) told an evaluation team member that the assessment was dropped when the project time was reduced. The evaluation team assessed other primary and secondary data sources to answer the evaluation questions. Table 3 summarizes the characteristics and suitability of each source. None of these sources provided the specific information required at the district level except the FALAH baseline and endline surveys, which were not complete at the time of the evaluation. The MICS surveys that were underway in Punjab and Sindh provinces at the same time as the evaluation include questions to estimate unmet need at the district level. However, the results of these surveys will not be available until the project is over. In addition, the MICS methodology for estimating CPR is not compatible with the PDHS, or with contraceptive prevalence surveys done in Pakistan and elsewhere, so they could not be used for the evaluation even if they were available. The PDHS planned for 2011-12 can provide the necessary information on CPR, proportion of MWRA with unmet need for family planning, and proportion of total need met at the national and provincial levels, but not for individual FALAH districts. Two secondary sources, the Health Management Information System (HMIS) and the Population Management Information System (PMIS) provide some information on contraceptive dispensation, but not on CPR and other relevant indicators. Conclusions  No data are currently available to assess trends in key FALAH outcome indicators for the FALAH districts. The upcoming 2012 PDHS will provide reliable data to determine trends in these indicators at the national and possibly provincial levels. The FALAH project baseline and endline surveys should also provide valuable information on trends at the district level (see Annex 1). TABLE 3: AVAILABILITY OF INFORMATION ON KEY PROJECT INDICATORS BY SOURCE Data Source Relevant Indicators Availability At District Level Most Recent Data PDHS 1. CPR 2. Unmet need No 2006-07 Pakistan Social and Living Standards Measurement Survey (PSLM) Does not include CPR; only pre and post-natal care Yes 2010-11 Multi-Indicator Cluster Survey (MICS) CPR – different methodology, so not comparable with PDHS and PSLM Yes Balochistan: 2009-10 Punjab: 2007-08 Sindh: 2003-04 HMIS Does not include CPR and unmet need but includes trends in facility level data on contraceptive methods Yes Current Population Management Information System Does not include CPR and unmet need but includes trends in facility level data on contraceptive methods Yes Current Contraceptive Performance Does not include CPR. Yes April – June 2011 Family Advancement for Life and Health – Evaluation Report 16 Report by FALAH project Includes CYP which is a different measure from CPR Baseline/endline survey in selected FALAH districts (Annex 1) CPR, unmet need, and proportion of women with unmet need Yes 2009-1010 (baseline) 2011-12 (endline) Findings and Conclusions on Evaluation Question #4 Evaluation Question 4: To what extent is the FALAH program’s technical approach relevant to achieving planned project outcomes in the 20 districts included in the evaluation? Findings FALAH’s technical approach was broken down into four results: Result A - Increase Acceptance and Demand for Birth Spacing and Contraception FALAH organized a national advocacy program to reposition family planning as a health intervention essential to reducing maternal and infant mortality sufficiently to achieve MDGs four and five targets by 2015. The project organized national and provincial level meetings and seminars targeting policy makers, health professionals from the MOH and MOPW, as well as religious scholars and leaders to share new evidence about the health impacts of family planning and highlight what each or their roles should be towards changing how family planning was viewed and implemented in Pakistan. FALAH and other donors funded the Ninth Annual Population Research Conference entitled “Population Dynamics and Security: Public Policy Challenges” in December 2008. The conference provided a platform to share experience with stakeholders for policy and program development and was attended by the Minister of Population and Welfare, senior officials from MOH, USAID representatives, and other dignitaries.49 FALAH’s communications and mobilization strategy focused on facilitating the paradigm shift towards birth spacing as a means of improving the health of women and children. For this purpose the HTSP was introduced at the policy (national) as well as at the community level. Evidence of this successful national policy change was documented in the official GOP publication “Birth Spacing Saves Lives: Strategy for Improving Delivery of Birth Spacing Services by the Public Health System” in December 2009. Interviews with FALAH staff, health care providers, former officials of MOH and MOPW, provincial officials, policy makers, and group discussions with religious leaders in all four FALAH districts visited during the evaluation revealed that until the recent shift of emphasis from limiting family size and decreasing population growth to HTSP, most of the population viewed the purpose of family planning as “limiting family size” and focused on messages designed to convince families to have only two children with little rationale for the advice. Several district religious leaders interviewed by the evaluation team said that in the past they had misperceptions of the reasons why family planning was important. Group discussion participants conceded after they received HTSP training that they had misunderstood the original purpose of promoting family planning all along. In the same interviews, the respondents said that the FALAH project succeeded in introducing the concept of “birth spacing saves lives” and “healthy timing and spacing for the health of mother and child” and did not specify a particular number of children that a family should have. The emphasis on family health, according to religious leaders and MWRA (and others), did not conflict with religious advice in the Quran and made promoting the use of contraception, spacing births, and having only the number of children a family can support acceptable to religious 49 FALAH project Annual Report 2008 -2009, August 2009. Family Advancement for Life and Health – Evaluation Report 17 scholars. The religious leaders said that counseling married adults about their responsibilities to themselves and to their children is an important aspect of their responsibilities. Further objective evidence of the acceptance of HTSP since the project began include:  The use of the word “birth spacing” is now included in the draft National Population Policy 2010; provincial policies are under development.  A statement issued by the Prime Minister on July 11, 2011 making it mandatory for all levels of health facilities to offer birth spacing services to the entire population at a nominal fee or free of cost.  The national and provincial MNCH programs endorsed the birth spacing messages for use in their future communications and advocacy materials.  Religious scholars from all schools of thought, including the Islamic Ideology Council, have endorsed the concept of birth spacing, unequivocally stating that it does not contravene Islamic injunctions. In spite of the need to replace a key project partner for Result A at the beginning of the project (Constella Futures), FALAH and partners said they knew that the communication and social mobilization activities were essential to the success of the other three results and FALAH partners devoted considerable effort to Result A in the first years of the project. Key activities under Result A include: Developing Communication Materials The project communication strategy focused on reaching the target audience through multiple communication channels to reach from the national to the community, household, and individual levels delivering the same 16 HTSP messages. Initial activities were intended to remove misunderstandings about modern contraception (myths and misperceptions). This was followed by detailed explanations about how birth spacing benefits the health and well-being of families. Mass media messages and interpersonal communications materials and activities were designed to enhance and reinforce understanding of ideal practices for health timing and spacing of births (spacing of two years between a delivery and planning the next pregnancy, no births before age 18 or after age 35, etc.) among women, men, religious leaders, gate keepers, providers, policy makers, program personnel, and the society at large. FALAH developed television and radio messages on HTSP that were endorsed by leading health professionals. MWRA interviewed by the evaluation team said that they heard about HTSP from messages on the television as well as from the LHW and health providers at health facilities. Media Campaign Assessment FALAH commissioned Gallup Pakistan to assess the effectiveness of the three week media campaign. The study found that the campaign reached an estimated 48 million adult men and women adults, ages 18 and older. They constituted 66 percent of the total television audience in that age group. The average viewer was exposed to FALAH advertising messages 48 times during the campaign period. The audience consisted of 34 million rural and 14 million urban residents, or just below 30 percent of the country. The messages reached all socioeconomic groups but was broken down to 32 million low income (under Rs.10,000 monthly household income), 18 million middle income persons (monthly household income Rs.10,000 to30,000), and 0.3 million on high income persons (monthly household income over Rs.30,000). Family Advancement for Life and Health – Evaluation Report 18 The study found that the majority of those who heard the messages lived in rural areas and came from the lower economic strata, indicating that the messages reached disadvantaged populations.50 “Those who saw or heard the media messages confirmed that they saw a difference between birth spacing messages that highlight benefits for maternal and child health and the message from previous family planning campaigns that emphasized limiting the number of children. They also said that the new HTSP messages were more acceptable to them and that they would share what they had heard with others.”51The assessment also found that the condom mass media campaign implemented in 2009 significantly decreased embarrassment among men when they tried to purchase condoms, and increased demand for and use of condoms. Exposure to branded messages was also associated with a higher level of belief in the efficacy of condoms and a lower level of perceived difficulty in obtaining condoms.52 Social Mobilization and Communication Through Community Based Health and Population Workers All of the groups of MWRA and married men that the evaluation team interviewed mentioned that the training that they received from LHWs and SMs was very helpful for them to understand the use and side effects of contraceptives. The training materials and job aides provided by the FALAH project also included specific content that explained how HTSP was consistent with Islamic beliefs (including quotations from the Quran). Participants in all group discussions specifically mentioned these materials and messages with appreciation and that the change in focus from limiting the family size to HTSP for health of mothers and children was a new message that they had never heard before. No one said that the new emphasis was not helpful. MWRA attributed their enhanced knowledge about contraceptives and birth spacing to radio, TV, LHWs, and meetings organized by community mobilizers. They also said they were informed that contraceptives of their choice were available either through community mobilizers (condoms only) or LHWs (condoms, pills).53 All MWRA groups were consistent in this response. One group of married men interviewed that did not respond this way was in a Catholic community in Jhelum. FALAH project activities in the district ended several months before the evaluation team and no FALAH staff were working there when the evaluation team visited this community so it was not possible to verify to what extent this community was exposed to project activities. Since FALAH activities in Jhelum ended several months before the evaluation field work, it was not possible to follow up with a project staff member. Community Mobilization Activities The FALAH project used community mobilization activities to promote HTSP and increase use of contraceptives by addressing personal and social barriers to contraceptive use such as side effects or fear of side effects, religious disapproval, and husband disapproval. The FALAH project reported that their mobilization activities sensitized around 5.7 million men and women. The basis of this estimate was not completely clear to the evaluation team, but was assumed to come from a combination of attendees at community meetings and home visits, and estimated audiences for community media. Male community members were approached through male SMs and male VHCs members. FALAH included training the 50 Gallup Pakistan, Viewership of FALAH Media Campaign aired during July – August 2011 Volume I Executive Summary, October 10, 2011. 51 Ibid. 52 Ibid. 53 These comments came from group discussions in areas where LHWs and/or community and SMs were available. Family Advancement for Life and Health – Evaluation Report 19 organizers of the (largely moribund) VHCs on how to conduct effective meetings and how to use these meetings to educate men in their communities. At the national level, The ESD project organized a study tour to Egypt in May 2009 to learn from their successful experience involving religious leaders in reproductive health programs. Representatives of the MOPW, MOH, Pakistan Initiative for Mothers and Newborns (PAIMAN), FALAH, and USAID attended. FALAH and USAID collaborated with the MOPW to follow the study tour with a national seminar on “Strengthening the Involvement of Religious Leaders” to shift the national strategy from promoting family size limitation to birth spacing. FALAH focused on advocacy activities with religious leaders as well as helping the MOPW to roll out a sensitization strategy for important stakeholders, especially the heads of the main sects of Islam. Using a TOT approach, the FALAH project trained over 1,500 religious scholars, or ulemas, to become trainers about HTSP and reduce religious and cultural barriers to family planning, especially among men. The approach expanded upon methods developed by the former USAID ESD project and engaged the most respected national scholars from each sect. TOT included these religious authorities working side by side with a medical doctor. The evaluation team interviewed groups of 5 to 10 male religious leaders from several sects in all four of the evaluation districts. All groups said that in the training they received a copy of a DVD from FALAH that included messages from the chief religious scholars of all of the sects of Islam in the country explaining how HTSP was important for the health of both the mother and the child and was consistent with advice given in the Quran. Leaders from D.G. Khan District volunteered that the training helped them to know each other better, and allowed them to work as colleagues when they attended trainings and promoted HTSP activities in communities. In two of the four religious leader group discussions held during the evaluation (D.G. Khan and Charsadda), male religious leaders recommended that HTSP training activities be expanded to include female religious leaders who are present in every community and have significant influence on MWRA behavior. Religious leaders in each of the discussion groups explained that prior to the FALAH project they had “many false preconceptions” of the purpose of promoting family planning and they were “afraid they were being used or manipulated.” However, once they attended the training, the scholars understood “how HTSP was one way to fulfill their responsibilities to provide guidance on family behavior.” All participants volunteered that they were very happy to have this training. They confirmed that they now promote HTSP in their communities and intend to continue now that the project is over. In Jhelum, D.G. Khan, and Thatta districts, individual ulemas volunteered that they use almost any opportunity “because they see how it really benefits the families and the communities” and “we knew there was information in the Quran about promoting health of mothers and children with HTSP messages and once we understood that family planning could help them to survive and be happy, we were convinced we should promote it.” The religious leaders also said that HTSP training included information on sexually transmitted diseases and obstetric fistula, topics that the religious leaders had heard about but were helpless to counsel on because of their lack of knowledge. They said that their only disappointment was the uncertainty of future opportunities for refresher trainings and that they did not have the resources to scale-up by training other religious leaders in the rest of Jhelum district. Community mobilizers (male social mobilizers of the PWD in this case) said that they were able to cover 50 to 80 percent of their designated union council areas. Those who had access to a vehicle, lived in the union council, and had full LHW coverage within the union council were able to cover more. They said that they dispensed condoms and referred clients to Family Welfare Centers (FWC). They also said that they found references from the Quran to be effective in promoting birth spacing with married couples. Family Advancement for Life and Health – Evaluation Report 20 They said that they leave condoms with a village community member in order to meet increased demand. On the other hand, a former senior official in the MOPW told a member of the evaluation team (who confirmed the finding from his own professional experience) that when FWC social mobilizers had no transportation (which is frequently the situation), many did not go to the communities but continued to collect government stipends as if they did. The evaluation team could not determine whether some Community, Advocacy and Mobilization (CAM) strategies were more effective than others, the relative cost for each activity (media spots were paid), or how skills and knowledge would be refreshed after the project was finished. Several of the activities of FALAH’s social marketing partners, including GSM, ZAFA, and UDL also contributed to creating demand for family planning. These included:  Female outreach staff conducted over one million household visits for face to face Interpersonal Counseling (IPC) with women with an unmet need for family planning.  Male and female outreach workers respectively conducted neighborhood meetings involving over one million men and over 700,000 women.  Male IPC staff conducted orientation meetings with over 25,000 influential male members of the communities including religious leaders, politicians and male teachers.  Female IPC staff conducted orientations with over 33,000 LHWs, Traditional Birth Attendants and female teachers.54 The major objective of these mobilization activities was to motivate women to seek birth spacing services from a nearby GSM provider especially during Sahoolat Clinic days, where services offered were free of cost or highly subsidized during the project. GSM’s communication and social mobilization activities included airing HTSP generic television commercials aimed at creating awareness of HTSP messages, male involvement in family planning decisions, contraceptive brand marketing, and behavior change to seek high quality family planning services. 54 Greenstar FALAH Project Closeout Report, March 2011. Family Advancement for Life and Health – Evaluation Report 21 RESULT B: Revitalization of High Quality Birth Spacing and Family Planning Services in the Public Sector Creating an enabling environment for working with the public sector FALAH facilitated a cross-visit to integrated MNCH/family planning programs in Bangladesh and also sponsored GOP officials to attend an international conference on family planning and Islam in Cairo. FALAH also organized meetings promoting the inclusion of HTSP in national policies and facilitated its inclusion in the statement of the Prime Minister that defined family planning as a health intervention and made HTSP mandatory in all health facilities. National and provincial MNCH programs also went on to endorse HTSP. Revitalizing the public sector to offer quality birth spacing services The project upgraded pre-service family planning curricula for 20 medical colleges (and oriented heads of other medical colleges), four schools of nursing, and 24 community midwifery schools. Project trainers also oriented LHWs in HTSP and updated their training packages. FALAH also provided family planning skills labs, including equipment, computers, job aides, and pelvic models in Islamabad, Dera Ghazi (D.G.) Khan, and Jhelum districts that also included extensive content on infection prevention. Trainers also conducted follow-up and quality assurance visits to these labs to assess whether the quality of the updated pre-service training had improved. These skills labs provided training for IUCD insertion intended to expand the number and type of health providers who are qualified to provide a wider array of contraceptive choices. The project also upgraded in-service training, including providing computers and software to GOP health and family welfare providers in CCFPS as well as IUCD and surgical contraceptive skills. Family The Ulema and the Taxi Driver One ulema participant in an evaluation group discussion cited his experience as a Master Trainer to promote HTSP. He told the team that before the training he was suspicious that the project was trying to use him to promote something that was against his religious beliefs. He said that after the training, when he realized that HTSP could help him to counsel families on parental obligations for healthy and prosperous children with healthy mothers, he became very enthusiastic to do what he could to spread the word about how family planning and birth spacing would help achieve those objectives, stating, “We knew about the passages in the Koran that support these behaviors, but we were not confident on how to advise anyone how to practice them.” The ulema took every opportunity that he could to counsel men about HTSP. Taxi rides often offered such opportunities. One taxi driver told him that he and his wife had tried for a very long time to conceive a child, but her pregnancies kept resulting in miscarriages. The ulema asked him how long he and his wife waited after a miscarriage to conceive a child and he responded that they would try again almost immediately. Based on what he had learned in the training, the ulema advised the man that he needed to let his wife rest for at least six months to allow her body to recover before trying to conceive again and that he should consult a doctor for contraceptives to prevent pregnancy until sufficient time had passed. After many months, the ulema saw the taxi driver again. The man had taken his advice and consulted a doctor, telling the doctor what the ulema had said. The doctor confirmed that the ulema’s advice was good and provided contraceptives. The couple waited six months before trying to conceive again, as the doctor and ulema both advised, and as a result, they now have a healthy child. The ulema was very happy to see the successful application of HTSP recommendations. Family Advancement for Life and Health – Evaluation Report 22 Welfare Training Institute trainers had their skills upgraded, and HTSP messages and compliance monitoring were integrated into their services. All health care providers interviewed during the evaluation agreed that the training was highly effective in improving the quality of services they now provide. The project also provided vasectomy training, but health care providers reported very low demand for vasectomy services. District health officials in all four districts said that the shortage of female doctors in government facilities hinders the ability of the district health facilities to implement these new skills, especially for permanent surgical methods, and also hinders prescribing family planning at the Basic Health Units (BHUs). Even though there are now more female medical graduates than males, the Executive District Officer (EDO) says that hiring for new positions is currently frozen. Leadership training was provided to district population and health policy makers, managers, service providers, and administrators to transform them into effective leaders who will act as change agents emphasizing birth spacing as an intervention for improving client centered reproductive health services in the public and private sectors. Leadership was also extended to FALAH partner NGOs including HANDS and their Community Based Organization (CBO) partners, and Save the Children field staff. Population Council organized 16 leadership trainings for 326 District Health and Population/Welfare Managers from 18 FALAH districts. The results of the workshops included developing district vision statements, integrated plans of actions, and individual pledges/promises of all 126 participants to contribute to improving the population and health systems in their districts. Follow-up meetings were held to strengthen networking among the leadership training participants from different training sessions and sections and for them to share achievements and experiences, helping them to motivate each other. They were also facilitated to consolidate and incorporate vision statements and action plans into integrated district action plans and to assess the impact of leadership training on district population and health systems strengthening. According to key informant interviews and focus group discussion held during the evaluation, the most effective training methods and approaches (including team building, problem solving, skills training interpersonal communication, and job aids) were those where the trainee said that what they learned helped them “to do my job better and more effectively.” In two of the districts visited by the evaluation team, members of the District Technical Committee commented that they had little or no input into the design of the FALAH project activities in their districts. For sustainability, DTC members said it would be important for them to be engaged in the design of future programs from the beginning. Training Social Mobilizers FALAH trained SMs from the MOPW to provide them with the knowledge and skills to conduct effective group sensitization meetings with eligible men and husbands of women who had been sensitized by the LHWs in their areas using information they had learned through the CCFPS training provided by FALAH. Social mobilizer training was contracted out to the National Rural Support Programme (NRSP), an independent local training organization that had also trained the community mobilizers (private sector) of FALAH’s implementing partners. In an interview with eight SMs in one district of Sindh Province, they said that the training provided to them by the FALAH project that included updates on contraceptive methods was better than anything they had ever received during their careers and that it also gave them an orientation on the new messages on HTSP that they needed for their work. One SM interviewed for the evaluation who attended the training said, “I learned more about what I am supposed to do and teach in a few days in this training than in all of the trainings I have had in years of service as a SM.” The evaluation team questioned the sustainability of the effects due to the current lack of supervision and support systems (especially transportation) in current district Population and Welfare offices. Expanding Contraceptive Choices Family Advancement for Life and Health – Evaluation Report 23 In addition to providing logistics management training and private sector contraceptive selection and supply (mentioned elsewhere), FALAH focused on introducing two new methods into the national contraceptive mix – Emergency Contraceptives (EC) and the Standard Days Method (SDM). As noted above, in 2007, FALAH organized a study tour to Bangladesh for the federal and provincial level staff of the National Program for Family Planning and Primary Health Care. The purpose of the study tour was to orient participants on how EC were introduced into the public health system in that country and to learn about problems and obstacles and how they were overcome. As a follow-up to the study tour, FALAH organized a consultative seminar in January 2008 and invited policy makers, doctors, health professionals, and program managers from both the MOH and MOPW from all four of the provinces. Regional experts from India and Bangladesh along with national doctors and policy makers shared their views on EC at this meeting. A major achievement of this seminar was getting the MOH to join the MOPW and the private sector in endorsing and supplying emergency contraception.55 Pilot Training of LHWs on ECPs:“Introduction of Emergency Contraceptive Pills (ECPs) through LHWs” trainings were conducted for 16 Trainers and 3,055 LHWs in D.G. Khan, Mansehra, Thatta, and Gawadar as part of a pilot. After completion of the trainings, LHWs have started providing ECPs to their clients in the four districts. Contraceptives for the test marketing phase were provided by United Nations Population Fund (UNFPA) under a tri-partite agreement with Population Council and the NPFP and Primary Health Care (PHC). Pilot on SDM. FALAH started a pilot to introduce SDM using cycle beads in one union council of Thatta through a public sector health outlet. A one-day training on SDM was also provided to nine community mobilizers belonging to the implementing partners, HANDS and RSPN, who in turn trained 38 CBVs on counseling techniques and provision of SDM in the selected project area. SDM (with bead necklaces for demonstration) was also introduced through providers and LHWs in other parts of the project. The evaluation team noted that both activities had taken place and found that several health facilities and LHWs had SDM necklaces. In a group interview with the DTC in Thatta District, committee members shared they felt both trainings were positive, but they were not sure about how the results of the new activities would be carried forward after the project ends. However, the evaluation team was unable to find any in-depth analysis of the experiences of these pilots and what lessons had been learned and next steps that are needed that could be used as the basis for expansion of both approaches in future programs. Technical Support to the LHW program FALAH provided a research officer to the National Program for Family Planning and Primary Health Care from May 2008 to October 2009 in order to improve operations research and monitoring and evaluation capabilities of the program. FALAH also supported the LHW program by providing 15,000 printed copies of the LHW’s Management Information System manual that is designed to enhance the performance of LHWs and the program as a whole.56 FALAH provided training on CCFPS for health providers and LHWs. The third party LHW study57 commissioned by FALAH, as well as interviews that the evaluation team conducted with LHS, LHWs, and female clients confirmed that they perceived the training to be of good quality and effective. Interviewed LHWs were able to answer questions about HTSP that required a good understanding of the 55 FALAH project Annual Report June 1, 2007 – May 31, 2008. 56 FALAH Project Third Annual Report, Population Council, June 2010. 57Shafaat, M., et al., Impact of Lady Health Worker’s Client Centered Family Planning Services Approach Training: a Third Party Evaluation, 2011. Family Advancement for Life and Health – Evaluation Report 24 content of training curricula. They gave examples of how they used the knowledge and said that they would continue to practice the improved attitudes and skills they received through the project. LHWs provided condoms and Oral Contraceptive Pills (OCPs) at the community level and ECPs (only in the four pilot districts). They also referred clients to BHUs for IUCDs, where available, and for counseling on other methods, including long-term and permanent methods. LHWs and LHS showed the evaluation team copies of the training job aids on CCFPS topics that were contained in the training. The LHW study found that trained LHWs conducted more home visits and community meetings and were much more likely to discuss family planning and contraception during these activities than in districts that were not part of the FALAH project. They also found differences in the source of contraceptive methods between districts. In intervention districts, 44 percent of women who were using contraceptives cited LHWs as the source as compared to 31 percent in non-intervention districts. The 2007 PDHS determined that side effects or fear of side effects were major barriers for women to use contraceptives. Of women who experienced side effects in intervention districts, 64 percent of women sought advice from the LHW as opposed to 41 percent in non-intervention districts and 91 percent knew of emergency contraception versus 22 percent in non-intervention districts. LHWs who were interviewed for the study said that the CCFPS training provided by FALAH had improved their knowledge of family planning and improved their skills in conducting women’s group meetings and in household visits. FALAH’s special emphasis on birth spacing methods, choices of contraception, and side effects appeared to have been (well retained) by the LHW trainees. The specific training on how to conduct the women’s group meetings utilizing supporting material was effective in boosting discussion on birth spacing and its benefits in a group setting in the community. Trained LHWs, however, said that a one-time intensive training requires follow up with supervisory support and periodic refreshers. 58 Client responses about the quality of home visits indicated that “trained LHWs were clearly aware of the appropriateness of a more client centered interaction with their clients.” Clients of trained LHWs reported better quality interactions with LHWs in terms of reaching agreement on issues, wider discussion about household health problems, and in greater privacy. The study concluded that the LHWs changed their attitudes towards clients, had improved family planning knowledge, and the training gave them the impetus to administer family planning services.59 Evaluation group discussions conducted during field visits found that LHSs and LHWs clearly understood the concept of HTSP and also recognized two natural birth spacing methods, the Lactational Amenorrhea Method (LAM) and SDM, even though they did not focus much attention to those methods during their BCC activities with MWRA. In interviews conducted by the evaluation team, LHSs and LHWs said that their efforts as a result of FALAH trainings have 2empowered women through BCC activities “to convince their husbands and identify ways to use contraceptives without disrupting the family environment.” They also said that messages their husbands heard on the radio and (in some cases) community meetings had encouraged them to be more supportive of family planning. MWRA also said in evaluation interviews that the poor economic conditions were another reason that couples were more encouraged to try to space births. They said that information they received from project training helped them understand that there were several types of contraceptives and that if they had side effects from one type of contraceptive they could go to the BHU (where they are available) and ask for another type of contraceptive. 58 Shafaat, M., et al., Impact of Lady Health Workers Client-Centered Family Planning Approach Training: a Third Party Evaluation Study, TNS Aftab Consultants, March 2011. 59 Shafaat, M., et al., Impact of Lady Health Worker’s Client Centered Family Planning Services Training: A Third Party Evaluation, March 2011 Family Advancement for Life and Health – Evaluation Report 25 Of the MWRA who were using contraception that were interviewed in the evaluation group discussions, the majority were already using modern methods. They were aware of natural family planning methods (LAM and SDM), but their understanding of the relationship of breastfeeding for two years as promoted in the Quran, and fertility reduction related to exclusive breastfeeding for the first six months, was not sufficient to have confidence in using the method. They were aware of the SDM, but said that they were not able to get access to the beaded necklaces to try the method. None of the LHWs or LHSs complained of being overwhelmed by extra duties. Both LHSs and LHWs deemed the LHW CCFPS HTSP training as feasible and effective, but the evaluation team was only able to interview LHWs that have been active for more than 10 years. Beneficiary MWRA said that the LHW was not available to them when they needed their help. On the contrary, several MWRA interviewed stated that the LHW was always available when they needed them, even if only for advice by mobile telephone. Logistics Management Training FALAH provided training on contraceptive logistic management for facility level staff in approximately one-third of project districts.60 The logistics training was intended to increase the availability of contraceptives at the facility level by ensuring timely and proper completion of the requisitioning forms and processing that will lead to regular supplies and reduced stock-out events. Linked with the leadership training, the logistics training was designed to help district level officials obtain needed contraceptive supplies from the provincial departments of population welfare (DPW), if they had them, and also to understand how to order contraceptives according to national guidelines. Quality Assurance The project led the revision of the National Standards for Family Planning and the updated version now includes chapters on HTSP, Post-Partum Family Planning, and Infertility. They also conducted workshops with Jhpiego to finalize the quality standards on family planning for FWCs of the MOPW. National family planning recommendations, standards on HTSP, and GOP policy on family planning were added and translated into Urdu. Result C: Reinforce High Quality Family Planning Services Through the Private Sector Mapping Public and Private Sector Health Service Points with Geographic Information Systems (GIS) FALAH conducted GIS mapping to locate public and private health care providers, including the location of LHWs within each district, and produced maps to be used by each of the districts for planning purposes. These maps were used to develop Integrated District Plans that are intended to include both public (including LHWs) and private sector providers for planning purposes. The DTCs in all four of the districts included in the evaluation as well as the GOP officials in the three provincial level offices of Health and Population and Family Welfare judged the GIS mapping to be extremely valuable. Provincial and district Health and Population Welfare officials responsible for planning said that the value of the tool for health systems strengthening in multiple areas has already been recognized. The districts and provinces requested copies of the software so they could apply it to other programs as well as update the maps that have already been produced over time. Family Advancement for Life and Health – Evaluation Report 26 Community-Based Volunteer (CBV) Pilot FALAH piloted a CBV program in six project districts61 in an attempt to develop a model for community mobilization and facilitation and provide access to HTSP for married couples in areas where there are no LHWs (approximately 30-40 percent of the population). An assessment study was conducted in five of the districts62 where the CBV pilot was undertaken. The remaining district was not included due to flooding at the time of the assessment.63 CBVs were “volunteers” who were identified community leaders facilitated by Community Mobilization Officers (CMOs) of RSPN. The report estimated that there were 1,080 CBVs (540 male and 540 female) working in six FALAH districts where the pilot was conducted. These CBVs are trained by the RSPN on GOP’s policy for family planning programs; HTSP messages, contraceptive methods, and their common side effects; addressing the myths and misconceptions associated with use of contraceptives; Islam and family planning; and community mobilization for birth spacing services. In some cases, CBVs provided contraceptives, but in most cases they referred clients to LHWs, who were also hired by the project. Contech, on behalf of Population Council, designed and carried out an impact evaluation survey in five of the six64 CBV intervention districts, three in Sindh and one each in Punjab and KP provinces. The study collected data from random samples of almost 1,990 MWRA, 269 CBVs, and 30 healthcare providers and project managers using both quantitative and qualitative data collection techniques. Responses from all the districts were compiled and integrated to provide an overview and was then compared with previous knowledge and practices against existing national trends. Key findings from the survey included:  Ninety-seven percent of those surveyed were aware of birth spacing methods at the time of survey as compared to only 62 percent at the beginning of the CBV intervention  Thirty-seven percent of women expressed a desire to avoid future pregnancy as compared to 25 percent prior to the program  Fifty-nine percent of those who still wished to have more children said that they had decided to observe an interval of at least two years  Forty-two percent of married women who had been non-users of any family planning methods wished to adopt them in the future  There was no significant change in preferred contraceptive methods by previous users; preferred family planning choices were pills, followed by injections, condoms, and IUCDs  Eighty-eight percent expressed that they selected family planning methods on the advice of CBVs and 67 percent had already visited the sources to obtain them  Only 18 percent of MWRA had misconceptions about contraceptive side effects; ninety-seven percent of those said they overcame their fear of side effects after consulting a CBV 61 Contech, Impact Evaluation of Community-Based Volunteers (draft), December 2011. 62 Intervention districts include Mansehra (KPK Province), D.G. Khan (Punjab Province), Sukkur, Dadu and Thatta (Sindh Province). Sanghar in Sindh Province was not included in the assessment due to severe flooding in the district. 63 Contech, 2011. 64 Flooding in one intervention district caused it to be dropped from the study. Family Advancement for Life and Health – Evaluation Report 27  Twenty percent of family planning users reported side effects from using contraceptives and 72 percent of them preferred to contact CBVs first for information before consulting anyone else  CPR among the surveyed population is estimated at 50.4 percent: far better than the national estimate of 29 percent  Eighty-seven percent preferred using modern methods compared to only six percent who preferred traditional birth spacing methods, and more than 90 percent knew the correct way to use each family planning method  The majority of the CBVs (96 percent) reported consistent availability of contraceptive supplies District health managers and health care providers interviewed for the study said that the CBV program could be strengthened through:  Increased involvement of male CBVs  Better remuneration for all CBVs  Increased frequency of camps (preferably through permanent static service delivery centers managed by LHWs near clients’ home)  Improved coordination between program and public sector health facilities65 The CBV model was not sustainable without external funding. As soon as project funding ended, CBVs, along with the LHWs hired by the project to provide family planning services, stopped working. CBVs were not true volunteers and expected to be paid. CBVs considered the stipends too low and had already started to complain after the pilot that the stipend was insufficient for the amount of work.66 Partnership between Community Mobilization NGO (RSPN) and Private Sector Family Planning Providers. In an effort to increase access to family planning services in poor rural communities without public sector family planning services, RSPN increased demand through community-based organizers who mobilized groups of men and women to access GSM mobile clinics. RSPN managers said that they were successful in creating demand, but GSM mobile clinics could not provide services to many who wanted them. GSM managers could not provide input because they were no longer working for the project at the time of the evaluation. In addition, the mobile clinics and mobilization efforts were discontinued when FALAH project funding ended. Result D: Expansion of Family Planning Services and Quality Improvement Initiatives in the Private Sector Involvement in the FALAH project enabled GSM Network to move outside urban areas and to provide quality improvement services to their franchised providers.  GSM identified and trained 1,020 new franchise providers on family planning. Of these, 502 were in rural areas and 518 were in urban areas. In addition, 7,316 existing GSM providers were given refresher training on infection prevention and IUCD insertion skills. The refresher messages focused on the importance of quality. This included provider quality, such as comprehensive and compassionate counseling and care that clients should expect to receive, as 65 Contech International, Draft Report of Community Based Volunteers: A Third Party Evaluation (2011). 66 Ibid. Family Advancement for Life and Health – Evaluation Report 28 well as service quality, which emphasizes cleanliness, competent treatment, and good management. GSM network providers were given refresher seminars on IUCD insertion skills  GSM collaborated with the MOPW to increase access to family planning services and supplies in rural underserved areas by organizing free “camps” that offered clinical services in Sahoolat (Urdu for convenience) Clinics where project-trained private providers agreed to provide free services once per month. Clients were mobilized by GSM community mobilizers. During the monthly clinics, GSM provided contraceptives free of charge. GSM also organized mobile medical units to provide services to rural areas where there were no providers. In these cases, RSPN conducted the community mobilization.67 Project documents indicate that activities under this result met their targets as indicated in the PMP, but there was no plan in the project design to make activities sustainable, and by the time of the final evaluation at the end of 2011, most activities had already ended. Some activities were incorporated into ongoing programs implemented by HANDS in Sindh Province supported with funds from other donors. Social Marketing of Family Planning Commodities The FALAH project did not procure or distribute contraceptives, but focused on creating demand, capacity building, and skills training to implement and manage family planning public and private service delivery and commodity supply programs. Social marketing work (including GSM camps and Sahoolat Clinics) was intended to increase access to and the quality of private sector services and commodities (GSM and commercial) in addition to the commodities provided through the GOP Health and Population and Welfare programs. One project activity was to increase availability and better marketing of full price contraceptive products for those who can afford to pay for them. Pre-service and in-service curricula and skills training increased the number of health providers able to provide IUCD and surgical contraceptive services and introduced a new fertility awareness method (SDM). The only contraceptives that the FALAH project provided directly were those subsidized commodities provided through GSM.68 Otherwise the project approach mobilized beneficiaries to access supplies provided through either government institutions or private commercial providers (pharmacies) and put more control in the client’s ability to make an informed choice from the available sources. The evaluation team heard feedback from district health managers and health providers in all four districts that there are not sufficient supplies of contraceptives in government family planning programs, especially those provided through health facilities, to meet the current demand, and stock-outs are common. USAID has provided support to the national commodities logistic system through the DELIVER project to move from a “pull system” to a “push system,” which is intended to help alleviate some of these problems. FALAH provided training in logistics management to the districts and at the end of the project was one of the partners working with the central warehouse in Karachi to improve supply and transportation of contraceptives directly to the districts. GSM together with ZAFA and UDL have been major sources of contraceptive commodities to the private sector and essential to filling in gaps in public sector supplies. GSM provides subsidized commodities, 67 FALAH Project Greenstar Social Marketing End of Project Report, March 2011. 68 Annex 8 contains as detailed report of the project’s social marketing activities under Result D provided by FALAH. Family Advancement for Life and Health – Evaluation Report 29 while UDL and ZAFA market to the segment of the population who can afford to pay full price for contraceptives. Throughout the life of the project GSM ran 965 television and radio spots for Sathi and 10,268 spots for Touch (two condom brands). The FALAH final evaluation did not conduct market studies. However, since different FALAH partners sold specific brands, the evaluation team’s observations of brands available in the market provide some evidence of market penetration. An evaluation team member conducted informal site visits to pharmacies in each of the four districts covered by the evaluation field work. Table 4 indicates FALAH-supported brands of contraceptives discovered by the evaluation team member in the pharmacies. MWRA interviewed in LHW groups said that they consider GSM products to be affordable. ZAFA and UDL products are targeted for the general market and are sold at prevailing prices.69 69 Group discussions and interviews with MWRA and key informant interviews with health service providers. Family Advancement for Life and Health – Evaluation Report 30 TABLE 4: BRANDS OF CONTRACEPTIVES PROVIDED THROUGH FALAH Greenstar ZAFA UDL Condoms Sathi Touch Happy Life Condoms Doted Happy Life Collection Pills Emergency Contraceptive pill Novodol Nova Famila 28F Pills Emkit 0.75 (ECP) Emkit DS (ECP) DESOFAM Pill Injections Femiject Megistron Novaject Famila Injections FAM-INJ NORIFAM IUCD: Multiload Safeload Key Market Research Studies GSM told the evaluation team that it had conducted several market research studies. The evaluation team could not review the studies, but GSM reported the following results in reports to USAID. Retail Condom Audit: A retail condom audit was conducted every month by AC Nielson for GSM based on a sample size of 5,000 retail shops (excluding pharmacies). The survey provided information on the up-take of condoms marketed by GSM, commercial companies, and other NGOs; and GSM’s total market share of the condoms, but the results of the survey were not available from FALAH at the time of the final evaluation. Evaluation Study of the "Sathi" and "Touch" Campaigns: In February and March 2009, a social marketing advertising campaign for “Touch” condoms was aired on television and radio in Pakistan. Following the campaign, a nationally representative survey of 1,606 men married to women aged 15-49 was conducted to assess its impact. In addition to social and demographic characteristics of respondents and fertility desires, the survey instrument collected information on family planning behaviors and recall of contraceptive advertisements. Bivariate analysis was conducted to determine whether respondents’ reports of changes in behavior were associated with awareness of the Touch advertisement. Multivariate analysis was conducted to determine whether awareness of the Touch advertisement was associated with improved attitudes towards condoms, condom use, and contraceptive use. About 15 percent of urban married men were aware of the Touch advertisement. Another 16 percent recalled other family planning advertisements. Awareness of the Touch advertisement was associated with a higher level of belief in the effectiveness of condoms, reduced embarrassment in negotiating condom use, reduced embarrassment in purchasing condoms, increased discussion of family planning, and respondents report of having started use of condoms and other contraceptive methods. 70 Even after 70 FALAH project Greenstar Social Marketing End of Project Report, March 2011. Family Advancement for Life and Health – Evaluation Report 31 controlling for a range of other variables including daily television viewership, awareness of the Touch advertisement correlated significantly with condom use and contraceptive use. An extrapolation of the pertinent data suggests that the Touch thematic condom advertising campaign may have generated 218,373 users of condoms and 83,777 users of other contraceptive methods. The findings suggest that the social marketing campaign was effective in changing contraceptive behavior among married men. Short Marketing Studies Such as Concept Testing, Pre-Testing of Materials, and Messages: AC Nielson conducts a retail condom audit every month for GSM. Nielson uses a sample size of 5,000 retail shops (excluding pharmacies). The survey provides information on the off-take of condoms marketed by GSM, commercial companies, and other NGOs. The audit reports indicated an increase in the sales of Sathi and Touch when the products were advertised on mass media, which indicated positive results of the media campaigns on Sathi and Touch. The off-take for Sathi increased more in rural areas while Touch sales increased considerably in urban markets. The sales trend also shows a positive impact of the Touch and Sathi media advertisement. Under informational and promotional activities, Information Education Communication (IEC) materials were kept at the provider outlets for clients. These materials help clients choose the method that matches their reproductive goals and give clients more information about their chosen contraceptive method including its effective use, possible side effects, and ways to manage minor problems. Conclusions Result A  National level policy and advocacy activities that the project did will likely have impact beyond the project target districts, but quantitative data are not yet available to support those assumptions.  Communication and social mobilization activities conducted in the FALAH project were very effective in establishing the enabling environment to reposition family planning as a health intervention as HTSP. HTSP messages have removed significant religious and cultural barriers to the use of contraceptives in family health. FALAH used proven social and behavioral methodologies to accomplish this change. The qualitative assessment of the media campaign suggests that it was successful in reaching viewers of both genders in the intended age group, and raised awareness of the HTSP in the priority rural and low income households. It is likely to have contributed to intended attitudinal and behavioral changes that would be pre-requisites to increase CPR, but did not measure CPR.  Working with religious leaders is especially valuable in changing social norms in the conservative religious and cultural environment of Pakistan. Result B  Pre-service and in-service training to health providers and community based workers in the public sector has increased the number of qualified providers in the project districts. Training methods, materials, and job aides are high quality and appropriate. Curricula changes at national level training institutions will likely ensure that training content will be up to date in institutions that have access to materials and trained faculty. Skills training labs require costly equipment and training job aids. Scale-up to reach other districts would probably require financial and technical support to implement. CCFPS LHW training is feasible and acceptable for experienced LHWs. There is a concern that new LHWs would find the additional training content overwhelming. This would have to be investigated in-depth if the LHW training were to be scaled up. Service Family Advancement for Life and Health – Evaluation Report 32 provider curriculum and training updates that were done by the project are critical pre-requisites to increasing access and coverage of family planning services. Result C  Project activities demonstrated that collaborating and providing capacity building to the private sector, including NGOs and community based workers, can be effective in delivering HTSP services. However, in many cases, service delivery activities as currently designed are unlikely to be sustainable after project funds end. The CBV pilot yielded very promising results, but the CBVs were no longer working when the final evaluation took place. The CBV model is also a vertical family planning program at a time when Pakistan is moving towards integrated MNCH/family planning services. To be sustainable, the costs of the program would need to be part of the national budget. It is possible that NGOs might be contracted to provide CBV services, such as HANDS is doing, but a sustainable source of financing, either GOP or donor, would still be necessary.  The parallel outreach programs, including outreach activities for GSM camps through RSPN, were not sustainable. Outreach through community development organizations was not able to meet the potential reach of the approach during the program because GSM was not able to keep contraceptive supply through mobile clinics at a level to meet demand created by the program. Outcome analysis of the pros and cons of all of the community-based components of the FALAH project would have been valuable, but time constraints prevented a thorough analysis. Result D  Contraceptive supplies through private sector social marketing and commercial firms were the only FALAH activities directed at increasing access to affordable contraceptives. PMP results document that Result D increased CYP significantly and also trained large numbers of new providers in the private sector to provide quality contraceptive services. Activities related to contraceptives in the other results were indirect. The evaluation team concluded that the social marketing activities were effective. GSM facilities confirmed that their service providers are primarily located in urban areas and in rural areas that were not very far from urban centers. Sahoolat clinics increased the number of private providers, but after the project ended free services given by these providers ended and the approach requires private (formal) health care providers to be available, which is often not the case in small rural communities. Although public sector health workers know about GSM products and availability (some health care providers purchase GSM contraceptives for their clinics with their own money when government supplies run out), there is currently no formal referral process from the public sector to the private sector, and LHWs are not instructed on how to inform MWRA where to find contraceptives that are not provided by the government.  While the RSPN-GSM partnership was a laudable attempt at providing services to rural areas without functioning public sector health facilities or private family planning providers, barriers to successful functioning of the model would need to be analyzed before any recommendation to scale-up the approach could be made. Findings and Conclusions on Evaluation Question #5 Evaluation Question 5: What have been the major obstacles to program coverage and access, and what should USAID and other donors do to facilitate demand and utilization of family planning methods in rural and higher poverty areas? “Coverage” refers to the percentage of the target populations reached by FALAH activities, such as community mobilization, trainings, etc. “Access” refers to the percentage of the target population being able to receive family planning services. Family Advancement for Life and Health – Evaluation Report 33 Findings Although the baseline survey measured percentages of MWRA or married men with specific attitudes and behaviors in the use of contraceptives, coverage was not defined elsewhere in FALAH project documents. The PMP relies on whether target numbers are achieved. Implementation strategies were measured by number of activities and individuals reached by an activity (message, training, etc.) in each district, but do not include percentages. The project’s endline survey will collect quantitative data on coverage and access. The project PMP did not include process indicators that would directly measure family planning coverage or access. They did, however, measure many pre-requisites to achieving coverage and access that would indicate an enhanced enabling environment for change in those indicators. The project estimated that over a three-week period approximately 50 million people, primarily from rural and poor households, heard HTSP messages promoted by the project’s media campaign program.71 The communication strategy was designed to significantly increase demand for family planning and promote HTSP. In public health, acceptable measures of population coverage require an accurate measurement of the population to determine the denominator of “susceptible” or “eligible” individuals for the activity targeted by the program (immunization coverage, contraceptive prevalence, etc.). The last census in Pakistan was done in 1998 and population estimates are based on extrapolations from that time or house to house counting done by community workers. It is not known how reliable those estimates are. Without a reasonable estimate of the number of MWRA and their husbands, it is not possible to determine coverage estimates for the population as a whole using the PMP, and measurement must rely on large surveys such as those conducted in the project baseline and final surveys. Sampling methods used in the PDHS are acceptable for estimating coverage. The next PDHS is scheduled be conducted in 2012-2013. To understand and address problems concerning access to family planning services in the public and private sectors, the districts needed to know the locations of existing providers. The GIS mapping exercises and leadership and management training provided by the project with district and provincial health and population/welfare planning officials were intended to provide this essential baseline information. In addition, the GIS software would allow for maps to be updated over time. GOP officials at the district and provincial level who were interviewed for the evaluation said that GIS mapping supported by FALAH has proven extremely valuable in helping them understand some of the access issues that hinder increases in CPR. Some provincial and government health and population/welfare managers in the four evaluation districts who were trained by the project stated that they started to use the mapping program in other public health programs. This is a strong indicator of the value of the management and leadership training program provided in the FALAH project as well as GIS mapping as a valuable planning and health systems strengthening tool. Security: The overall security situation in the country, including several FALAH districts, was a continuous concern throughout the project and substantially affected coverage. USAID augmented the project budget to improve security for all of the partners. The sensitivity of the family planning issue made working in these areas especially challenging and USAID responded by replacing some of the highly sensitive districts with other districts. Many NGO implementing partners were provided offices within government buildings as an extra security measure. Overall, with the exception of Mercy Corp’s activities in Balochistan, the security situation did not cause significant delays in achieving project activity targets. According to interviews with managers from HANDS, Save the Children, and RSPN, NGOs that have worked in areas with poor security for many years have the trust of the communities and are more able to continue and scale-up HTSP services than government or commercial sector programs. 71 Gallup Pakistan, Viewership of FALAH Media Campaign July – August 2011, Volume I. Family Advancement for Life and Health – Evaluation Report 34 Commodity security: GSM was the main private sector partner in the FALAH consortium and provided 91 percent of the CYPs planned for the project. According to district Department of Health (DOH) managers, reliable and affordable contraceptive supply in GOP health facilities remains the major problem for most of the population from poor families. This problem is discussed in other parts of this report, and USAID is providing support to solve the problem through the DELIVER project’s technical assistance to the central contraceptive warehouse in Karachi. The manager of the national commodity warehouse in Karachi said that this assistance is crucial and any support from the national level to the districts will likely help alleviate the stock-out problems as supplies at the national level are deemed adequate to meet the demand. FALAH’s logistic management and leadership training at the district level were designed to provide the ability for district GOP officials to solve problems and address contraceptive supply issues at the district level. Government partnerships: Passage of the 18th Amendment abolishing the national MOH and MOPW toward the end of the FALAH project created great uncertainty within the donor community and implementing partners about how health and family planning programs will be managed now that responsibility has been transferred from the national level to the provinces and districts. Prior to this major change, frequent transfer of key government officials at the federal, provincial, and district levels required the project to conduct continuous briefings and orientations and disrupted the progress of scheduled activities. However, the implementing partners kept abreast of these changes and provided regular feedback to the COP and project team so that by the end of the project, the project had accomplished all targeted activities. While the MOPW’s Social Mobilizers (SMs) that were trained by the program said they were now doing a better job, the evaluation team was unable to objectively determine if this training had significantly contributed to improved performance or quality of services delivered to communities. According to a former MOPW official, there is a perception that many SMs are recruited because of political influence and often do not visit communities because they lack skills and motivation to fulfill their responsibilities. The evaluation could not confirm if this was true everywhere or even a commonly-held opinion among beneficiaries or health care providers. The FALAH project staff said that the rapid quantitative survey that was planned after the Midterm Review in 2010 was canceled after the project timeframe and funding was decreased by USAID. The AIPs did not specify how coverage and access indicators specified in the project Cooperative Agreement were to be monitored. Conclusions  While the project strategy was designed to increase family planning coverage and access, the project PMP was not clearly linked to measuring increases in coverage and access during the life of the project. The final quantitative survey is supposed to measure coverage and access in each FALAH district. Findings and Conclusions on Evaluation Question #6 Evaluation Question 6: What factors have affected the effectiveness, coverage, quality, and sustainability of FALAH family planning activities and what are the lessons for how USAID designs future programs in Pakistan? Which indicators can attribute to sustainability or institutionalization? Findings The project activities and PMP did not clearly define how coverage in project target groups was to be measured. However, USAID officials said that in the beginning of the project, coverage was interpreted as “numbers of districts served” by specific activities and not the percentage of the target population Family Advancement for Life and Health – Evaluation Report 35 reached by project activities within those districts. The project did not assess changes in knowledge or practice of desired behaviors during implementation. The project’s baseline and endline surveys will measure some of these indicators. Group discussion participants and key informants at all levels including national, provincial, and district policy makers and program managers, health care providers, pre-service and in-service trainers, and community providers (LHWs, LHSs, SMs, community mobilizers) all stated that the quality and appropriateness of all the project efforts were very high and improvements in curricula, standards, and job aids will be permanent. Evaluation team members probed for negative findings. The only negative comments were disappointment that the project was ending and it was unknown if there would be refreshers in the future. Strategies at the district level did not include plans to reach a specific percentage of the district eligible population. From a public health standpoint, that makes determining the overall impact of the program in terms of coverage very difficult. Conclusions  The shift to emphasizing family planning/MNCH integration and HTSP for family health has been a major “game changer” in support of acceptability and has enhanced the enabling environment for increased coverage by removing significant religious and cultural barriers to using contraceptives. This change is considered to be sustainable as there is no support, nor ability, to return to the previous system or messages.  Strengthening government and private sector family planning and health programs, generating demand, and fostering ownership of FALAH efforts within government structures were priorities for most of the project, and available evidence seems to support the value of this strategy. Even in the best of circumstances, it normally takes significant time for persuasion, discussion, and permission to introduce innovation in government programs, but once convinced, most interventions have strong potential for continuity and scale-up beyond the limits of the project. Examples within the FALAH project were working with the Ministry and Departments of Health to make family planning a priority, training public health facility staff and LHWs in the CCFPS approach to reproductive health service delivery, expanding method choice by providing emergency contraceptive and fertility awareness methods, and combining the efforts of SMs and LHWs for effective mobilization at the household and community levels. Given that FALAH basically had approximately three years of implementation, it was unlikely that all project activities would be stand-alone and fully sustainable without outside support by the end of the project.  The government structure underwent sudden and radical changes of devolution and the central government offices related to the FALAH project were abolished in the last year of the project. The newly-charged GOP structures that now must absorb project activities were still establishing their programs at the end of the project and the commercial sector is not yet prepared to pick up the services that government cannot provide. FALAH has provided a good foundation in many districts, but there will need to be follow-on activities to turn them over completely.  The evaluation team was convinced that FALAH made every effort to base most of the interventions on the principles of sound programming practices, sustainability, and potential for broad scale-up. Family Advancement for Life and Health – Evaluation Report 36 Findings and Conclusions on Evaluation Question #7 Evaluation Question 7: To what extent are project results likely to be sustainable and what could the GOP do to ensure continuity, integration, and scaling up of FALAH’s technical advances in project districts? Findings The evaluation team found no specific sustainability plan in project documents, except in the initial plans for ZAFA support which were abandoned when Constella Futures withdrew from the consortium early in the project. The project PMP contained no indicators of sustainability. When the timeframe of the project was unexpectedly shortened, several project follow-up and exit activities such as follow-up assessments of the training impact and the rapid quantitative assessment planned after the midterm review had to be dropped. These changes left open questions about who would develop supervisory and monitoring systems to ensure that behavior change in the project’s capacity building activities will continue. Curricula and training materials developed by the FALAH project were judged to be especially high quality but it is uncertain how materials will be scaled up, provided to Pakistani training institutions and programs, and reproduced. Leadership skills and specific technical approaches such as GIS are already in use and health managers at the district and provincial level in all four districts interviewed by the evaluation team said that the skills will be retained. FALAH managers said they will provide the staff that was trained in GIS with the software so they can continue to update the maps over time. Conclusions  The evaluation team was convinced that the change in family planning messaging to HTSP for the health of the family as a result of FALAH activities will result in a sustainable change in perceptions about the purpose and value of family planning. Working through religious leaders was very effective in increasing knowledge about and acceptance of family planning and HTSP and removed barriers and efforts to scale-up this training to include female religious leaders who will continue to support these changed populations within Pakistan. There is wide enthusiasm to scale this approach to the rest of the country, and the effect is likely to be sustainable.  Policy changes that defined family planning as a health intervention required in all MNCH programs, updated family planning standards, increased availability of a wider array of contraceptives, and CCFPS accompanied with interpersonal communication training are all important to be integrated within the GOP standard operating procedures and will lead to sustainable change. Adequate and consistent supplies of an array of contraceptives are essential for sustainability of the behavior changes, whether they come from the public or the private sector.  Expanding these capacity building efforts is likely to have long-term positive health systems impacts. Leadership training participants from DTCs, provincial DPW offices, and partner NGOs told the evaluation team that the skills they developed help them with team building and empowered decision-makers to manage change during the devolution process. Participants remarked on how it “changed their attitudes and gave them a more positive and effective approach to their work.” The most effective and possibly sustainable contribution of FALAH’s capacity building is providing skills to manage change during the devolution process and afterwards. These enhanced management skills can have multiple positive impacts on HPN programs.  Enhanced social marketing increased the availability of a variety of affordable contraceptives in the market where private pharmacies are located. Specific training increased the knowledge and Family Advancement for Life and Health – Evaluation Report 37 skills of facility and community workers to provide delivery of better quality services. LHW and LHS training brought higher quality services to the client’s doorstep, but these improvements will need continued supportive supervision and implementers will need refreshers provided by the GOP. Gaps remain in how to reach remote rural populations where GOP health facilities are not functional, the LHW program is weak, and the commercial sector is largely absent.  Project planning documents contained no defined sustainability strategy with measurable indicators or a specific exit strategy to turn over activities to local structures by the end of the project. Population Council staff said they cut short their plans for post-training monitoring visits when the project time was reduced. Policy changes and repositioning family planning as HTSP will probably be sustained and are "institutionalized," but support will be needed for all health care providers to be updated in these new developments. For Results B and C, the project turned materials and methods over to local institutions for their use without outside support, but this happened relatively close to the end of the project. More time is required to determine whether the local institutions were able to use the materials and implement the training methods after the project support ended.  FALAH was a district level project, and by-passed the provincial level because that was the GOP structure at the beginning of the project. Nationally, FALAH only covered about one-quarter of the districts in the country and did not cover entire districts with all strategies. The evaluation team found no clear plans to scale-up successful piloted interventions. With the passage of the 18th Amendment that moved the management action from the central government to the provinces, FALAH focused more attention at the provincial level. Population Council volunteered that in retrospect it would have been better to have provincial offices from the beginning.  Even though it was mentioned specifically in project objectives, sustainability, coverage, and scale-up were not included as part of the program monitoring plan. However, the evaluation team felt that even in the absence of objectively measurable indicators, several project elements, especially policy change, skills and behavior change of providers and clients, and social marketing of contraceptives would probably be sustained and even scaled-up. The extent of the potential scaling up is not known. Several elements of the project, however, have potential to be sustainable but only if additional investments are made in future programs and this will probably require technical assistance at the provincial planning and management levels. RECOMMENDATIONS Scale-up HTSP in Pakistan based on lessons learned in FALAH  USAID Pakistan’s HPN office should scale up and sustain HTSP within Pakistan by supporting capacity building of Pakistani training institutions to assume responsibility for transferring project capacity building skills from the project to training institutions within Pakistan.  Population Council should capture and analyze lessons learned and best practices in the project’s promising community-based approaches to reaching disadvantaged poor populations (LHWs, CBVs, SMs, Sahoolat and mobile clinics) that were tested and implemented in the FALAH project. Recommendations for future programs based on these lessons learned should be included in end of project reports. Results of the third party evaluations of the pilot studies in the project should be disseminated among USAID’s HTSP partners. Family Advancement for Life and Health – Evaluation Report 38  USAID’s HPN office should build upon lessons learned and the enthusiastic response of religious leaders to the capacity building developed by the ESD and FALAH projects to engage religious leaders in future HTSP and MNCH and include female religious leaders where appropriate.  To promote integration of family planning and MNCH services and encourage synergies, USAID’s HPN office should clearly state their expectations for collaboration between partner procurement documents, and joint efforts should be monitored during program implementation. Demonstrate program impact through strong monitoring and evaluation systems USAID Pakistan’s HPN office should strengthen M&E systems to demonstrate program impact by:  Requiring future HTSP and family planning programs to have strong M&E with clear definitions of how coverage and access will be objectively measured using evidence-based indicators that also measure increases in knowledge and behavior change and sustainability  Considering external midterm evaluations for programs to identify progress towards project goals and objectives and recommend adjustments in the program as needed  Including measurements of contributions of private sector and commercial capacity to increasing coverage and access  Continuing support for the PDHS as it remains the only reliable measurement of CPR and other key family planning and HTSP indicators that allow for nationwide trend analysis Sustainability USAID Pakistan’s HPN office should support institutionalizing the HTSP model in national health and population programs throughout Pakistan by:  Engaging GOP partner offices in project design, implementation, monitoring and evaluation from the beginning  Expanding Leadership and Management Training and support to additional DTCs  Providing support to develop Leadership Training within Pakistani institutions Family Advancement for Life and Health – Evaluation Report 39 REFERENCES Da Vanzo, J., A. Razzaque, M. Rahman, L. Hale, K. Ahmed, M. A. Khan, G. Mustafa and K. Gauzia. The Effects of Birth Spacing on Infant and Child Mortality, Pregnancy Outcomes, and Maternal Morbidity and Mortality in Matlab, Bangladesh. Rand Corporation: Rand Labor and Population Working Paper Series, WR-198 (October) 2004. Retrieved from http://www.rand.org/pubs/working_papers/2004/RAND_WR198.pdf Israr, Syed M. Project Mentality: A Threat to Effective Implementation and Sustainability of Donor Funded Projects in Pakistan. Journal of Pakistan Medical Association, 2005. Retrieved from http://jpma.org.pk/PdfDownload/916.pdf Mahmood, Naushin. Reproductive Goals and Family Planning Attitudes in Pakistan: A Couple-level Analysis. The Pakistan Development Review, 1988. Retrieved from http://www.pide.org.pk/pdf/PDR/1998/Volume1/19-34.pdf Sathar, Zeba. Status of Family Planning in Pakistan: Presentation to UNFPA-ICOMP Regional Consultation, 2010. Retrieved from http://www.icomp.org.my/new/uploads/fpconsultation/Zeba%20Sathar.pdf World Health Organization. Policy Brief on Birth Spacing-Report from a World Health Organization Technical Consultation. WHO Department of Reproductive Health and Research and Department of Making Pregnancy Safer, 2006. Retrieved from http://www.searo.who.int/LinkFiles/Publications_policybrief5.pdf Conde-Agudelo, A. and J. M. Belizan. Maternal Morbidity and Mortality Associated with Inter￾pregnancy Interval: a Cross Sectional Study. British Medical Journal, 321: 1255-1259 (November), 2000. Retrieved from http://www.bmj.com/highwire/filestream/386390/field_highwire_article_pdf/0.pdf Conde-Agudelo, A. and J. M. Belizan, R. Breman, S. Brockman and A. Ross-Bermudez, 2005. Effect of the Inter-pregnancy Interval after and Abortion on Maternal and Prenatal Health in Latin America. International Journal of Obstetrics and Gynecology, Vol. 89, Supplement 1 (April), S34-S40. Conde-Agudelo, A. Ross-Bermudez, and A. C. Kafury-Goeta , 2006. Birth Spacing and Risk of Adverse Perinatal Outcomes: A Meta-Analysis. Journal of the American Medical Association, 295: 1809-1823. Retrieved from http://www.esdproj.org/site/DocServer/JAMA_2006_Conde-Agudelo.pdf?docID=1744 Razzaque, A., J. Da Vanzo, M. Rehman, K. Gausia, L. Hale, M. A. Khan, A. H. M. G. Mustafa, 2005. Pregnancy Spacing and Maternal Morbidity in Matlab, Bangladesh. International Journal of Gynecology & Obstetrics, Vol. 89, Supplement 1 (April), S41-S49. Rutstein, S., 2005. Effects of Preceding Birth on Neonatal, Infant and Under-Five Years Mortality and Nutritional Status in Developing Countries: Evidence from the Demographic and Health Surveys. International Journal of Gynecology & Obstetrics, Vol. 89, Supplement 1 (April), S7-S24. Retrieved from http://factsforlife.org/pdf/BIRTH%20SPACING%20AND%20NUTRITION%20IN%20DEVELOPING% 20COUNTRIES.pdf 2008 National Institute of Population Studies (NIPS) and Macro International Inc. Pakistan Demographic and Health Survey 2006-07. Islamabad, Pakistan. Retrieved from http://measuredhs.com/pubs/pdf/FR200/FR200.pdf Family Advancement for Life and Health – Evaluation Report 40 ANNEXES Annex 1: The Impact of FALAH on Contraceptive Use and on Unmet Need for Family Planning The evaluation team received the material in this annex from the Population Council after the evaluation was completed and the team disbanded. Therefore, the team was not able to assess the quality of the analysis or of the reported results. The remainder of the annex, with the exception of the figures and formatting, is exactly as provided by the Population Council. The impact of the project was measured through an end line survey conducted November 2011 –January 2012. In the end line 9,995 ever married women were interviewed overall; furthermore a panel 6,806 women was also followed in the end line survey. Additional women were recruited for the cross sectional sample in cases where women from the panel had to be replaced. Both samples are statistically representative for each of the 14 districts where the survey was conducted. Contraceptive use rose by 8.5 percent for the cross sectional sample and also for the panel of women in the period of 3.5 years between the baseline and end line 2008-2011. One of the major accomplishments is a rise across the board in contraceptive prevalence across all districts except in the case of Larkana in the cross sectional sample - in the panel there was an increase even in Larkana. The proportionate increase is greatest among rural, uneducated, poor and youngest women in the sample. Unmet need decreased by 3 percent for the cross section of women and by 1.9 percent for panel women in the same period. Total demand for family planning services is now up to 70 percent (69 and 71 percent for cross sectional and panel women respectively). Furthermore, the proportion of satisfied demand has risen by 9 percent from 45 to 54 percent among the cross section of women. This represents a 6 percent increase in the total market potential for contraception in the project areas and reflects that the majority of Pakistani women are in need of family planning services and a little more than half of the need is satisfied. Figures 1 and 2 illustrate changes in key indicators for all districts and by district. Tables 1 and 2 show changes by district and various socio-economic categories. Family Advancement for Life and Health – Evaluation Report 41 FIGURE 1: CHANGE IN KEY FALAH OUTCOME INDICATORS FOR PANEL SAMPLE OF MWRA 35% 29% 64% 46% 33% 38% 71% 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% Unmet need Met need (CPR) Total demand Proportion met Percent of MWRA Baseline Endline FIGURE 2: CHANGE IN KEY FALAH OUTCOME INDICATORS BY DISTRICT 0% 10% 20% 30% 40% 50% 60% Percent of MWRA Met Need (CPR) Baseline Endline 0% 10% 20% 30% 40% 50% Percent of MWRA Unmet Need Baseline Endline 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Percent of MWRA Total Demand Baseline Endline 0% 10% 20% 30% 40% 50% 60% 70% 80% Percent of MWRA Proportion of Need Met Baseline Endline Family Advancement for Life and Health – Evaluation Report 42 TABLE 1: CHANGE IN KEY FALAH OUTCOME INDICATORS FOR THE CROSS￾SECTIONAL SAMPLE OF WOMEN Background Unmet Need Met Need (CPR) Total Demand Proportion Met Baseline Endline Baseline Endline Baseline Endline Baseline Endline District Sukkur 33.0 30.8 30.9 38.3 63.9 69.2 48.4 55.4 Dadu 39.9 42.3 21.0 27.7 60.9 70.0 34.5 39.6 Larkana 33.8 35.3 27.8 25.9 61.6 61.1 45.2 42.3 Thatta 33.8 33.5 20.8 25.0 54.6 58.5 38.0 42.7 Sanghar 32.6 31.1 21.8 34.6 54.4 65.7 40.1 52.7 Ghotki 34.9 41.1 16.5 26.1 51.4 67.2 32.1 38.8 Sawabi 33.6 29.5 36.3 47.1 69.9 76.6 51.9 61.5 Charsada 32.3 24.8 36.9 49.8 69.2 74.6 53.3 66.8 Mansehra 37.3 27.6 28.4 39.8 65.6 67.4 43.2 59.1 Mardan 29.9 23.2 40.6 53.0 70.6 76.2 57.6 69.5 Jhelum 30.6 25.4 37.5 46.9 68.1 72.3 55.1 64.9 DG.Khan 41.7 34.2 26.6 36.4 68.3 70.6 38.9 51.5 Bahawalpur 30.7 26.1 38.3 44.6 69.0 70.7 55.5 63.0 Rajanpur 40.2 36.3 20.4 32.6 60.6 68.8 33.6 47.3 Age group 15 - 24 31.8 32.8 13.8 20.4 45.6 53.2 30.2 38.3 25 - 34 36.7 33.7 29.8 36.3 66.5 70.0 44.9 51.8 35 - 49 34.8 29.0 39.4 46.2 74.2 75.2 53.1 61.4 Literac y Literate 28.5 25.0 36.6 46.3 65.2 71.3 56.2 64.9 Illiterate 37.1 34.6 26.0 33.5 63.1 68.0 41.2 49.2 Education level No education 37.2 34.3 25.9 33.6 63.1 68.0 41.0 49.5 Up to primary 32.9 29.3 32.3 39.9 65.2 69.2 49.5 57.7 Up to Secondary 26.1 23.7 38.9 49.7 65.0 73.4 59.8 67.7 Above secondary 23.5 21.7 41.6 51.3 65.2 73.0 63.9 70.3 Standard of living Low 41.1 39.9 17.4 25.6 58.5 65.5 29.7 39.1 Medium low 37.2 33.1 24.6 34.0 61.8 67.1 39.8 50.6 Medium high 32.6 28.7 32.7 42.7 65.3 71.4 50.0 59.8 High 28.2 25.0 40.6 47.1 68.8 72.1 59.0 65.4 No. of children 0 3.9 4.7 0.8 4.2 4.7 8.9 17.9 47.5 1-2 34.2 29.0 18.5 23.7 52.8 52.7 35.1 45.0 3-4 40.0 34.9 35.9 41.5 75.9 76.4 47.3 54.3 5 or more 42.1 36.1 41.4 48.7 83.5 84.7 49.6 57.4 Residence Rural 36.8 33.5 25.5 34.8 62.3 68.4 40.9 50.9 Urban 26.7 24.8 42.2 46.6 68.9 71.5 61.2 65.2 All 34.7 31.7 28.9 37.4 63.6 69.1 45.4 54.2 Family Advancement for Life and Health – Evaluation Report 43 TABLE 2: CHANGE IN KEY FALAH OUTCOME INDICATORS FOR THE PANEL SAMPLE OF WOMEN Background Unmet Need Met Need (CPR) Total Demand Proportion Met Baseline Endline Baseline Endline Baseline Endline Baseline Endline District Sukkur 33.7 33.7 27.5 38.9 61.1 72.5 44.9 53.6 Dadu 40.3 43.5 21.0 28.3 61.3 71.9 34.2 39.4 Larkana 34.3 35.4 26.0 28.2 60.2 63.5 43.1 44.3 Thatta 34.5 35.4 19.3 24.7 53.8 60.1 35.9 41.2 Sanghar 31.7 31.0 24.9 37.3 56.6 68.3 44.0 54.6 Ghotki 35.6 43.8 16.6 25.5 52.3 69.3 31.8 36.8 Sawabi 32.4 33.7 37.8 47.1 70.2 80.8 53.9 58.3 Charsada 30.1 27.1 37.7 50.3 67.8 77.4 55.6 65.0 Mansehra 39.4 29.0 29.6 43.0 69.1 72.0 42.9 59.7 Mardan 30.5 24.4 43.6 51.8 74.1 76.2 58.8 68.0 Jhelum 33.1 26.8 40.9 46.8 74.0 73.6 55.3 63.6 DG.Khan 41.2 34.6 28.6 37.3 69.7 71.9 41.0 51.9 Bahawalpur 31.5 25.5 39.8 45.5 71.3 71.0 55.8 64.1 Rajanpur 39.6 36.4 21.3 33.6 60.9 70.0 35.0 47.9 Age group 15 - 24 31.2 34.9 13.7 22.8 44.9 57.7 30.4 39.6 25 - 34 36.7 34.9 30.6 35.6 67.3 70.6 45.5 50.5 35 - 49 36.2 30.5 41.9 45.8 78.2 76.3 53.7 60.1 Literac y Literate 29.8 25.2 37.2 49.4 67.0 74.6 55.5 66.2 Illiterate 37.1 36.0 26.4 33.5 63.4 69.5 41.6 48.2 Education level No education 37.1 35.7 26.4 33.8 63.5 69.5 41.5 48.6 Up to primary 34.6 31.0 31.3 39.4 65.9 70.4 47.4 56.0 Up to Secondary 27.4 23.3 40.8 53.7 68.2 77.0 59.8 69.7 Above secondary 23.2 20.6 42.9 60.4 66.1 81.0 64.9 74.6 Standard of living Low 41.7 41.3 17.8 25.8 59.5 67.1 30.0 38.4 Medium low 37.0 34.6 25.6 34.8 62.6 69.4 40.8 50.1 Medium high 33.1 30.5 33.2 43.2 66.3 73.7 50.0 58.7 High 28.3 24.0 40.9 50.2 69.2 74.2 59.1 67.6 No. of children 0 4.0 1.9 0.7 4.1 4.7 6.0 15.5 68.5 1-2 34.0 26.6 17.6 22.8 51.7 49.5 34.1 46.2 3-4 39.3 36.8 37.4 39.7 76.7 76.5 48.8 51.9 5 or more 44.0 37.3 43.0 48.4 87.0 85.7 49.5 56.5 Residence Rural 37.0 35.2 26.1 35.2 63.1 70.4 41.4 50.0 Urban 26.6 23.9 43.3 49.2 69.8 73.2 62.0 67.3 All 35.0 33.1 29.4 37.9 64.4 70.9 45.6 53.4 Family Advancement for Life and Health – Evaluation Report 44 Annex 2: Evolution of FALAH Implementation Districts, 2007-2011 Province Districts Above average HDI of Punjab? June 2007 – June 2009 July 2009 – June 2010 July 2010 – December 2011 Included in the evaluation Punjab 1. D.G. Khan 1. D.G. Khan 1. D.G. Khan 1. D.G. Khan N 2. Jhelum 2. Jhelum 2. Jhelum 2. Jhelum Y 3. Bahawalpur 3. Bahawalpur 3. Bahawalpur N 4. Khanewal 4. Khanewal N 5. Multan N 6. Rajanpur 4. Rajanpur 5. Rajanpur N Sindh 3. Dadu 7. Dadu 5. Dadu 6. Dadu N 4. Ghotki 8. Ghotki 6. Ghotki 7. Ghotki Y 5. Larkana 9. Larkana 7. Larkana 8. Larkana N 6. Sanghar 10. Sanghar 8. Sanghar 9. Sanghar N 7. Sukkur 11. Sukkur 9. Sukkur 10. Sukkur Y 8. Thatta 12. Thatta 10. Thatta 11. Thatta N 13. Jacobabad N 14. Shikarpur 12. Shikarpur N 15. Karachi 13. Karachi72 Y KP 9. Charsadda 16. Charsadda 11. Charsadda 14. Charsadda N 17. Swabi 12. Swabi 15. Swabi Y 18. Mardan 13. Mardan 16. Mardan Y 19. Mansehra 14. Mansehra 17. Mansehra N 10. Swat N 11. Upper Dir 18. Upper Dir N 12. Buner 19. Buner N 13. Battagram N 14. Lakki Marwat N Balochistan 15. Gwadar 20. Gwadar N 16. Jaffarabad 21. Jaffarabad 15. Jaffarabad 20. Jaffarabad N 17. Khuzdar 22. Khuzdar N 18. Lasbella 23. Lasbella N 19. Turbat 24. Turbet-Kech N 20. Zhob 25. Zhob N 26. Quetta N Total 20 26 15 20 25 6 72 Out of three townships (Gadap, Lyari and Orangi), the evaluation will include Gadap and Orangi. Family Advancement for Life and Health – Evaluation Report 45 Annex 3: Initial Getting to Answers Table Data Collection Evaluation question Type of answer/evidence Methods Source Sampling/ selection Data analysis methods 1. To what extent has the FALAH program accomplished its planned activities and outputs as outlined in the Cooperative Agreement and subsequent annual work plans? Comparison between planned and actual activities Document review Project reports and activity records n.a. Narrative and tabular comparison of planned to actual activities and outputs 2. To what extent has the FALAH program achieved its key outcomes (increased contraceptive prevalence rate (CPR), reduced proportion of MWRA with unmet need for family planning, increased proportion of total demand met, increased proportion of MWRA who have discussed family planning with husband, and increased proportion of MWRA intending to use in future)? Cause and effect. Determine the extent to which FALAH activities and outputs have contributed to intended outcomes Document review, interviews, survey data Project reports, interview data, FALAH baseline and follow-up surveys FALAH surveyed a random sample of potential clients in the baseline and will interview the same sample as a follow up. IMEC will select a random sample of clients, LHWs, LHVs, and others for interviews and focus group discussions. FALAH will analyze its survey data and provide results to IMEC. IMEC will analyze qualitative data from interviews and focus groups to qualitatively examine causality. Family Advancement for Life and Health – Evaluation Report 46 Data Collection Evaluation question Type of answer/evidence Methods Source Sampling/ selection Data analysis methods 3. What are the trends in FALAH outcome indicators (CPR, proportion of MWRAs with unmet need for family planning, proportion of total need met, etc.) in project districts in Pakistan? Comparison of indicator values over time. Output indicators are: contraceptive prevalence rate (CPR), proportion of MWRA with unmet need for family planning, proportion of total demand met, proportion of MWRA who have discussed family planning with husband, and proportion of MWRA intending to use in future. Document review, secondary data sources Project baseline documents Potential secondary data sources include District Health Information Management Systems and various national surveys including DHS, 2008 Multi-Indicator Cluster Survey, Pakistan Social and Living Standards Measurement Surveys (PSLM), district level reports prepared by DHIS cells, and the Contraceptive Performance Report.73 n.a. Review secondary data sources and construct as complete a trend as possible for the key outcome indicators at the district and national levels – 2001 to present. 4. To what extent is the FALAH program’s technical approach relevant to achieving planned project outcomes in the 20 districts included in the evaluation? Descriptive – describe efficacy of technical approach (i.e., training, community mobilization, etc.) Semi-structured interviews and group discussions Interviews with health facility staff, government stakeholders, LHWs, LHVs, religious and opinion leaders. Group discussions with clients, LHVs, and LHWs. Purposive sampling for interviews and random samples for group discussions. Description based on qualitative data from interviews. 73 The detailed Scope of Work for the evaluation reviews available secondary sources of data and concludes that they do not contain consistent time series data on the FALAH indicators. Nevertheless, the evaluation team will piece together the trend data that are available from these, and perhaps other, secondary sources. Family Advancement for Life and Health – Evaluation Report 47 Data Collection Evaluation question Type of answer/evidence Methods Source Sampling/ selection Data analysis methods 5. What have been the major obstacles to program coverage and access, and what does this imply about how USAID and other donors can facilitate demand for and utilization of family planning methods in rural and higher poverty areas? Descriptive and recommendation Semi-structured interviews and group discussions Interviews with health facility staff, government stakeholders, LHWs, LHVs, religious and opinion leaders, other donors. Group discussions with clients, LHVs, and LHWs. Purposive sampling for interviews and random samples for group discussions. Description based on qualitative data from interviews. 6. What factors have affected the effectiveness, coverage, quality and sustainability of FALAH family planning activities and what are the lessons for how USAID designs future programming in Pakistan? Descriptive and recommendation Semi-structured interviews and group discussions Interviews with health facility staff, government stakeholders, LHWs, LHVs, religious and opinion leaders, other donors, USAID. Group discussions with clients, LHVs, and LHWs. Purposive sampling for interviews and random samples for group discussions. Description based on qualitative data from interviews. 7. To what extent are project results likely to be sustainable and what could the GOP do to ensure continuity, integration, and scaling up of FALAH’s technical advances in project districts? Descriptive and recommendation Semi-structured interviews and group discussions Interviews with health facility staff, government stakeholders, LHWs, LHVs, religious and opinion leaders, other donors. Group discussions with clients, LHVs, and LHWs. Purposive sampling for interviews and random samples for group discussions. Description based on qualitative data from interviews. Family Advancement for Life and Health – Evaluation Report 48 Annex 4: Evaluation Schedule and Itinerary of Evaluation Team Visits to FALAH Districts Location Time Activity December 12, 2011 – Monday – District Jhelum Departure from Tulip Hotel, Jhelum 9:00 a.m. Arrival at EDO Office, Jhelum Executive District Officer (Health), EDO Health Office, Civil Lines Road, Jhelum 9:30 – 10:45 Meeting with EDO Health, DOH, DC-LHWs, PDHIS Coordinator and other key district health officials Arrival at BHU, Sukhan, Jhelum 11:00 – 12:00 Visit Health Facility to Meet CCFPS trained service providers Observe their work Visit mosques 11:00 – 14:00 Meeting with religious leaders and scholars Arrival at Skills Lab Nursing School, Jhelum 12:15 – 13:45 Site visit to Skills Lab established by FALAH to meet with faculty and students Tulip Hotel, Jhelum 14:00 – 15:30 Luncheon meeting with religious leaders who have been sensitized to become advocates on birth spacing and HTSP concept December 13, 2011 – Tuesday – District Jhelum Departure from Tulip Hotel, Jhelum 9:00 a.m. Arrival at DPWO office Shahrah-e-Mehrban, Civil Lines Road, Jhelum 9:30 – 10:30 Meeting with DPWO, Deputy District Population Welfare Officer (DDPWO), and other relevant staff Visit Family Welfare Centers 10:30 – 14:00 Meeting with trained staff at Family Welfare Centers Arrival at LHW Health House, Janjurarryala 11:00-12:00 Observe women group conducted by trained LHW Arrival at BHU, Janjurarryala, Jhelum 12:30– 02:00 Meeting (FGD) with rural area BHU, LHWs, and trained staff Arrival at DHDC Lady Health Supervisor 11:00 – 12:00 LHS trained on CCFPS and supportive supervision DHDC, EDO Health Office, Jhelum 14:00 – 15:00 Tea meeting with Leadership Fellows December 14, 2011 – Wednesday – District Jhelum Arrival at GSM Sahoolat Clinic site 10:00 – 11:30 Observe Sahoolat Clinic Arrival at LHW Health House 11:45 – 13:30 Meeting (FGD) with clients (MWRA) of LHW’s Arrival at house of a male community activist/volunteer 11:45 – 13:30 Meeting with currently married men Arrival at chemists involved by UDL 11:45 – 14:00 Meeting with chemists December 15 – Islamabad Arrival at Jhpiego Office: House No. 63-B, Street 36 Sector E-11/3, Islamabad, Pakistan Tel: 051-2228830 09:30 – 11:00 Meeting with Dr. Shabana Zaeem, Country Director Pakistan, Jhpiego—an affiliate of Johns Hopkins University Family Advancement for Life and Health – Evaluation Report 49 Location Time Activity Arrival at RSPN office: House No. 7, St. 49, F-6/4, Islamabad Tel: 12:30 – 14:00 Meeting with Ms Shandana Khan, Chief Executive Officer, RSPN Arrival at GSM office: 'House No 1-A, St. 8, Kohistan Road, F-8/3, Islamabad, Tel: 2256068 15:30 – 17:00 Meeting with Mr. Muzaffar Mahmood Qureshi, Resident Director, GSM December 16, 2011 – Friday – Islamabad (Ms. Jean Capps) Arrival at UNICEF office 90 Margalla Road, F 8/2, Islamabad 08:00-09:00 Meeting with UNICEF Health Specialist, Dr. Samia Rizwan Arrival at DFID office British High Commission, Diplomatic Enclave, Islamabad 09:45 –10:45 Meeting with Dr Ayesha Rasheed, Health Specialist DFID Arrival at UNFPA office Serena Hotel, Business Complex Khayaban-e-Suhrawardy, G-5, Islamabad 11:00 – 12:00 Meeting with Mr Rabi Royan, Country Representative UNFPA Arrival at STC office: H # 1, Street # 15A, Main Margalla Road F 7/2, Islamabad 14:30 – 15:30 Meeting with Dr. Amanullah Khan, Deputy Country Director Program Implementation, SAVE December 16, 2011 – Friday - Khyber Pakhtoonkhwa (Dr. Israr and Dr. Fozia Qureshi) Team A: Meeting room, Health Secretariat, Khyber Road, Peshawar 09:00 –10:30 Meeting with key provincial health officials, Chief, Health Sector Reforms Unit (HSRU) Department of Health, Director Reproductive Health Services KPK, Provincial Coordinator LHW program, KPK Arrival at Secretary PWD office: House # 125, Street 7-B, Defense Officer Colony, Peshawar Cantt 11:00 – 12:00 Meeting with Secretary Population Welfare Department KPK and other key provincial officials Team B: Arrival at EDO Health Office Charssada, (Inside DHQ Hospital) Charsadda 09:30 – 12:00 Meeting with DOH Health, DHDC and PPHI in￾charge, and DPWO other district officials from Charsadda December 16, 2011 – Friday – District Charsadda (Mr Mehboob Sultan) Arrival at health facilities, various locations, Charssada 09:45 –10:45 Visit to health facilities, Charsadda Various locations 11:00 – 12:00 Meeting with religious leaders December 19, 2011 – Monday – District Multan (Ms. Jean Capps and team) Departure from Multan 08:00 Arrival at Quaid-e-Azam Medical College, Bahawalpur 10:00 – 11:30 Site visit of Skills Lab Meeting with trained faculty Lunch 12:00-14:00 Meeting with Leadership Fellows, Master Trainers and religious leaders December 19, 2011 – Monday - District D.G. Khan (Pakistani members of evaluation team) Departure from Multan to D.G. Khan 08:00 - Arrival at BHU, Haji Ghazi, GT road, D.G. Khan 09:30 – 10:00 Visit health facility to Meet CCFPS trained service providers Family Advancement for Life and Health – Evaluation Report 50 Location Time Activity Arrival at Rural Health Center (RHC), Sarwar wali, GT Road D.G. Khan 10:15 – 11:00 Visit health facility to Meet CCFPS trained service providers Observe their work Arrival at DHDC D.G. Khan 11:30 – 13:00 Meeting (FGD) with LHS, LHWs Travel to Multan 13:15 December 20, 2011 – Tuesday RHSA Center Multan Address: RHS-A Master Training Center, 1st floor, Nishter Hospital Multan 10:00 – 11:00 Meeting with FALAH Trained Service Providers of RHSA and Minilap Hotel Ramada, Abdali Road Multan 11:30 – 13:00 Meeting with EDO, District Coordinator, DOH, DPWO, and other key officials of D.G. Khan Lunch at Hotel Ramada, Abdali Road, Multan 13:15 - Meeting with D.G. Khan district religious leaders sensitized by FALAH on the birth spacing concept December 22, 2011 – Thursday – Karachi Visit to HANDS office 11:00 – 12:00 Meeting with HANDS Team HANDS office 12:00 – 13:30 Meeting with Chief Executive (CE) Lyari Community Development Organization Meeting with CBVs HANDS office 13:30 – 14:30 Lunch with HANDS Team December 23, 2011 – Friday – Karachi Departure from hotel 09:15 Arrival at RHSA Center, Abbassi Shaheed Hospital, Karachi 10:00 -11:00 Visit RHSA Center, Abbasi Shaheed Hospital Arrival at Health Department, Sindh 12:00 - 13:00 Meeting with Special Secretary Health and other officials, Deputy Secretary DG Health, Provincial Coordinator, National Program Health Education Officer, LHW Program December 24, 2011 – Saturday – Karachi Departure from hotel 09:00 am Arrival at RTI Karachi Street no. 21-22 Block II, Scheme # 05, near Ziauddin Hospital, Cliftion, Karachi. Ph# 021-99251005 09:30 – 10:30 Meeting with RTI principal and faculty Visit to Population Welfare Department, Sindh 61-A, K.C.H Society, Karachi. 11:30 – 13:00 Meeting with Secretary, PWD, additional Secretary, and other key officials Visit to Central Warehouse Plot No. F- 508, Zia Colony Site Karachi 10.00 – 12.00 Meeting with Director, Central Warehouse December 26, 2011 – Monday – Thatta Travel to Thatta 08:30-10:00 Arrival at EDO Office, Thatta 10:30-11:30 Meeting with government officials: EDO,Health Thatta and other official DC LHW, DSM, PPHI, DOH and DPWO Dhabeji, Thatta (Team A) 12:00- 13:30 Meeting with LHWs and clients Dhabeji, Thatta (Team B) 12:00- 14:30 Meeting with male clients and religious leaders Departure for Karachi 15:00 December 27, 2011 – Tuesday – Islamabad Family Advancement for Life and Health – Evaluation Report 51 Location Time Activity Population Council # 7, Street 2, F-6/3, Islamabad Tel: 051-8445566 03:00-04:00 Meeting with Dr. Mumtaz Esker Population Council # 7, Street 2, F-6/3, Islamabad Tel: 051-8445566 04:00-05:00 Meeting with Dr. Chandio, Registrar Health Services Academy/Former Deputy Coordinator of the National Program (NP) for family planning/PHC December 28, 2011 – Wednesday – Islamabad Serena Hotel, Business Complex Khayaban-e-Suhrawardy, G-5, Islamabad 10:00 – 11:00 Meeting with Dr. Nabeela Ali, Chief of Party, PAIMAN. Family Advancement for Life and Health – Evaluation Report 52 Annex 5: Documents Reviewed USAID documents Office of Inspector General, Final Audit Report of FALAH, August 2010 USAID/Pakistan, draft Statement of Work, FALAH Evaluation, revised April 25, 2011 USAID/Pakistan, Health Office Targets, June 2011 USAID/Pakistan, Performance Management Plan, FY 2011 – FY 2013 (chapters on health, and introduction and cross-cutting objectives) USAID TIPS, Constructing an Evaluation Report, 2010 USAID Extending Service Delivery Project: Note on HTSP Strategy FALAH documents FALAH’s Operational Plan Indicators Final FALAH PMP, January 2010 List of FALAH Districts and Partners AIPs, Year 1 to Years 4 and 5 (review at least Years 4 and 5) Annual Reports, Years 1 to 4 and USAID comments on the 2009-10 Annual Report (review at least Year 4 and USAID comments) Quarterly Progress Reports (review at least the most recent two or three reports) FALAH district baseline reports, 27 reports (review at least the reports for districts Jhelum, D.G. Khan, Karachi and Thatta) FALAH district mapping reports, 21 reports (review at least the reports for districts Jhelum, D.G. Khan, Karachi and Thatta) FALAH Compliance Monitoring Plan FALAH Communication, Advocacy and Mobilization Strategy (CAM) for Birth Spacing FALAH Social Mobilization Strategy FALAH training manuals and related documents (review selectively): Training Strategy for Public Sector Health Care Providers, revised January 2009 Acquiring Skills in Family Planning Service Delivery, Trainer’s Notebooks for Modules 1, 2, 3 and 4 Acquiring Skills in Family Planning Service Delivery, Student Notebooks for Modules 1, 2, 3 and 4 Client Centered Family Planning Services (Basic), Trainer’s Manual Client Centered Family Planning Services (Advanced), Trainer’s Manual Client Centered Family Planning Services (Advanced), Participants’ Manual Client Centered Family Planning Services, Overview Workshop Notebook Family Advancement for Life and Health – Evaluation Report 53 CCA Booklet (Urdu) Impact of Lady Health Workers’ Client Centered Family Planning Services Training Gallup Report on Viewership of FALAH Media Campaign Impact Evaluation of Community Based Volunteers Competency Based Clinical Skills for IUCD, Trainer’s Notebook Competency Based Clinical Skills for IUCD, Participants’ Manual Competency Based Clinical Skills for IUCD, Reference Notebook Emergency Contraceptive Pills, Trainer’s Guide (Urdu) Emergency Contraception Manual (Urdu) FALAH Program and Financial Review, January 2010 Greenstar Pakistan Commodity Security Strategy Greenstar Training Strategy and Amended Training Strategy National documents (reviewed selectively) FALAH, Manual of National Standards for Family Planning Services, 2010 (endorsed by the GOP) National Institute of Population Studies, Pakistan Demographic and Health Survey, 2006-07 Pakistan Millennium Development Goals Report, 2010 National Health Policy, 2009: Zero Draft Other Sources: One-page summary of essential evidence-based interventions to reduce, maternal, newborn and child mortality, and promote reproductive health Population and Sustainability Network, 2010. Pakistan Launches a New Population Policy, retrieved from http://www.populationandsustainability.org/990/news/pakistan-launches-a-new-population-policy.html Family Advancement for Life and Health – Evaluation Report 54 Annex 6: List of Participants in Interviews and Focus Group Discussions Meeting with USAID HPN Office - December 8th: Venue Name And Designation of the Participants Contact Details USAID/Pakistan Health Office Dr. Rushna Ravji, Health Team Leader, Asia and the Middle East, USAID Washington Nora Madrigal, Health Development Officer, Office of Health, Population and Nutrition Meeting with Dr. Issa - December 9th: Venue Name and Designation of the Participants Contact Details Serena Business Complex, Meeting Room Dr. Issa, former Contracting Officer’s Technical Representative (COTR) FALAH, currently Public Health Advisor at Aga Khan Foundation Family Advancement for Life and Health – Evaluation Report 55 Names of Participants – Jhelum - Day 1: December 12th, 2011 Venue Name and Designation of the Participants Contact Details EDO Office Dr. Naseer Ahmad, Program Director, District Health Development Centre (DHDC) 0544- 9270261 Dr. Wasim Iqbal, Deputy District Office, Health (DDOH), Jhelum 0544- 9270418 Dr. Imtiaz Shah, District Coordinator, National Program (DCNP) Zamir Minhas, Entomologist, Jhelum Dr. Shahid Tanvir, EDO, Health, Jhelum 0333- 5822251 Dr. Muhammad Imtiaz Darr, Public Health Specialist, MNCH Program, Jhelum 0300- 5440349 Aftab Sarwar, DY, District Population Welfare Officer (DPWO) 0300- 5440350 BHU Sukhan, Jhelum Dr. Saira Khan, Woman Medical Officer Najma Begum, Female Medical Technician Shamim Zohra, LHW Interviews with religious leaders in mosques Hafiz Jamil Iqbal, Khateeb Jamia Masjid, Nazam Din, Bilal Town, Gulshan-e-Feroz, Jhelum Hafiz Muhammad Farooq, Khatib, Jamia Masjid Nizam Din, Behind Police Line, Gulshan-e￾Feroz, Jhelum Group discussion with religious leaders at Tulip Hotel Hafiz Sultan Mehmood, Imam Norani Masjid, Bilal Town, Jhelum Adeel Shahzada Qadri, Khatib Jamia Masjid Al-Hassanian G.T. Road Jadda, Jhelum 0300- 9519607 Allama Syed Rukhsar Hauder Naqvi, Khateeb Masjid-Jafria Kala Jhelum 0333- 5811215 Okasha Madni, Jamia Uloom-e-Arabia, Asria Road, Jhelum (email address oksmadni@yahoo.com), Khateeb and teacher Jamia Asria. 0313- 5402530 Alama Syed Fazal Hussain Naqvi, Khateeb Jamia Jafria Imam Bargah M.C.M, Jhelum 0333- 5824005 Skills Lab, Nursing School Mrs. Hamida Sarwar, Principal Mrs. Nasreen, Community Midwife Tutor Saadia, Nurse Midwife Shazia, Community Midwife Student, 3 rd year Family Advancement for Life and Health – Evaluation Report 56 Names of Participants – Jhelum - Day 2: December 13th, 2011 Venue Name and Designation of the Participants Contact Details DPWO Office Adnan Ashraf, DPWO Aftab Sarwar, DY. DPWO Waheed Iqbal, District Demographer Abdul Majeed AD, DPWO FWC- Jhelum Zulakha Aziz, FWW Bilal Town, Jhelum Tehmina Choudhri, FWW, Shumali Muhalla, Hjelum Robina Naz, FWW, Dina-1 Saeeda Yasmin, Domela Muhalla, Dina-2 BHU Jangu Rarryala Dr. Shama, In-charge Woman Medical Officer Mariam Khaksar, Female Medical Technician Maimoona Bushra, LHW Shabnam Shazia, LHS 10 LHWs trained by LHS LHW Health House, (LHW: Shamim Qadri) Meeting with 11 MWRA DHDC Lady Health Supervisor Haleema, R.H.C. Dina 0333-5816354 Tahira Jabeen, BHU Kotla Faqir 0321-5458670 Riffat Shamim, Pak. U.K. 0343-5730593 Saima Gul, Pak. U.K. 0321-9516208 Roomana Sanam, BHU Chak Khasa 0300-5467388 Shagufta Kazmi, Pak. U.K. 0332-5806812 DHDC: Meeting with Leadership Fellows Zamir Minhas, Entomologist 0321-5441034 Dr. Imtiaz Shah, Deputy Coordinator, National Program for PHC and Family Planning 0300-8510019 Dr. Imtiaz Daar, PHS MNCH Program 0300-5440349 Dr. Aftab Sarwar Dr. Ruidad Irfan, District Health Information System (DHIS)Coordinator Dr. Khalid Mehmood, District Officer, Health (DOH) Dr. Sarwar DHDC: Meeting with Leadership Fellows Dr. Rodab Irfan Majeed, DSC/DHIS Coordinator 0333-5445888 Dr. Capt. Asif, DoH Sh. 0333-8012077 Family Advancement for Life and Health – Evaluation Report 57 Names of Participants – Jhelum - Day 3: December 14th, 2011 Venue Name and Designation of the Participants Contact Details GSM Sahoolat Clinic Site Dr. Fareeha Fawad, Manager Health Services, Greenstar Rawalpindi Khizar Abbas, Zonal Coordinator, GSM Rawalpindi LHW Health House, Shamali MOHalla (LHW: Ishrat Jabeen; LHS: Saima Gul) Group Discussion with 11 MWRA House of Male Community Mobilizer FGD with six currently married men ZAFA Chemists Standard Medical Store, Main Bazar, Jhelum Mian Jee Enterprises, Ram Din Bazar, Jhelum Chaudri Medical Store, Chawk Ahle Hadis Attique Medical Store, Near District Headquarters (DHQ) Hospital Al-Syed Medical Store, Near DHQ, Jhelum Names of Participants – RSPN and Greenstar - December 15th, 2011 – Islamabad Venue Name and Designation of the Participants Contact Details RSPN Office, Islamabad M. Bashir Anjum, National Coordinator, FALAH 0333-400-3811 K. A. Tetlay, Chief Operating Officer, RSPN 0301-846-7906 Shandana Khan, CEO, RSPN 0300-500-9506 GSM, Islamabad Dr. Haroon Ibrahim 0300-854-4716 Muzaffar Mahmood Qureshi 0300-850-8623 December 16th, 2011 – Meetings in Islamabad Venue Name and Designation of the Participants Contact Details UK Department for International Development (DFID) Office, Diplomatic Enclave Dr. Ayesha Rasheed, Health Advisor, DFID a-rasheed@dfid.gov.uk Save the Children (STC) Office Dr. Amanullah Khan, Deputy Country Director, Program Implementation, Save the Children Dr. Masood Ahmed Abbasi, Senior Health Manager, STC UNFPA Office Mr. Rabi Royan, Country Representative, UNFPA UNICEF Office Dr. Samia Rizwan, Health Specialist, Maternal and Child Healthcare, UNICEF Family Advancement for Life and Health – Evaluation Report 58 Names of Participants – Charsadda - Day 1: December 16th, 2011 Venue Name and Designation of the Participants Contact Details EDO Office Dr. Farhad Khan, Deputy EDO Health 0333- 9062377 Dr. Dr Saira Ameen, Assistant District Coordinator, National Programme for family planning and primary health care 0342- 8127881 Dr. Shakeel Manan, FALAL District Coordinator 0332- 9846924 Reproductive Health Services A-type Centre District Hospital Charsadda Ms Musarrat Begum, Family Welfare Counselor 9220097 Ms Shazia Nawab, Family Welfare Worker Ms Rahat Begum, Family Welfare Assistant People’s Primary Health Care Initiative (PPHI) Mr. Muzaffar Khan Wazir, District Support Manager Day 2: December 17, 2011 Group discussion with religious leaders at Masjid District Hospital Charsadda Molana Bashir-ul-Haq, Mudarras Dar-ul-Uloom Islamia, and Khatib Jamia Masjid Qalandar Khail, Tehsil Shabqadar 0302- 5509448 Qari Aurangzeb, Mudrass Darul Uloom Islamia, Charsadda and Khatib Masjid Masood Khail, Charsadda City 0333- 9201406 Qari Nasrullah, Mudrass Darul Uloom Islamia Charsadda, and Khatib Masjid, District Headquarters Hospital, Charsadda 0301- 5937358 Molana Saleem Shah, Mudrass Darul Uloom Islamia, and Khatib Masjid Darul Hifz Islamia, Charsadda 0302- 5519722 Molana Noorullah Jan Farooqui, Khatib Masjid Salar Zai, Rajar, Tehsil Charsadda 0343- 9035455 Mobile Service Unit Ameerabad, Charsadda Neelofar Malik, Field Technical Officer PPHI Basic Health Unit Rajjar, Charsadda Ms Salma Naheed, LHW Mr. Wiqar Ali, Medical Technique PPHI Basic Health Unit Utmanzai Nishat Begum, LHW Samilna Begum, LHW, six years and Female Medical Technician, 18 years Family Advancement for Life and Health – Evaluation Report 59 Meetings at Peshawar Venue Name and Designation of the Participants Contact Details Health Secretariat, Khyber Pakhtun Khuwa (KPK), Peshawar Dr. Shabina, Chief, Health Sector Reforms Unit (HSRU) shabina.raza@gmail.com 03018582457 Dr. Roohullah, Department of Health, Director Reproductive Health Service (RHS) KPK, 03475096196 03005825037 Dr. Ijaz, Provincial Coordinator, LHW program 03339187220 (ppiunwfp@yahoo.com Secretary PWD office House # 125, Street 7-B, Defense Officer Colony, Peshawar Cantt Mr. Ahmad Hanif Orakzai, Secretary, PWD 03005922270 (ahorakzai@gmail.com) Dr. Sartaj Naeem, Principal, Regional Training Institute (RTI), Peshawar 03219133597 (sartajnaeem@hotmail.com) Dr. Saadia Nawab, Deputy Principal, RTI, Peshawar 03339151170 (saadianawab@hotmail.com) Dr. Najma Sultana, Director (Technical), PWD, KPK 091-9211075 Meeting with Population Council – December 17th, 2011 Venue Name and Designation of the Participants Contact Details Population Council Office, Islamabad Dr. Ali Mir, Chief of Party, FALAH Dr. Arshad Mahmood, Director Research, M&E Ms. Seemin Ashfaq, Senior Program Manager, Communications Dr. Gul Rashida, Senior Director, Training Names of Participants – Multan - Day 1: December 19th, 2011-Bahawalpur Venue Name and Designation of the Participants Contact Details Quaid-e-Azam Medical College Dr. Suhail Mehmood Chaudhry, Associate Professor Dr. Naheed Fatima, Professor of Obstetrics and Gynecology Nausheen Adil and Sabahat-e-Gul, final-year students EDO Office Dr. Saeed Asghar, DOH Bahawalpur, previously District Coordinator LHW Dr. Muhammad Naeem-ud-Din, Anesthesia Department, Victoria Hospital, prev. Deputy District Health Officer, Bahawalpur BHU, Jhangiwala Dr. Shazia Waqar, Woman Medical Officer, WMO Rihanna Kausar, LHW Family Advancement for Life and Health – Evaluation Report 60 Day 2: December 20th, 2011-Multan Venue Name and Designation of the Participants Contact Details RHS ‘A’ Master Training Center, Nishtar Hospital, Multan Dr. Rabia Bilal, Woman Medical Officer Mrs. Shagufta Ch., Theatre Nurse Mrs. Zahida Nawab, Theatre Technician Ramada Hotel, meeting with district officials from D.G. Khan Huma Mehdi, DPWO – D.G. Khan 0300-678-2930 Dr. Sajjad Sarwar, DCNP/DDOH 0300-678-2014 Dr. Muhammad Riaz Malik, Public Health Specialist, MNCH D.G. Khan 0344-644-3377 Sajjad Naqvi, CEO, Al-Asar Development Organization 0333-646-4683 Ramada Hotel, meeting with religious leaders Dr. Syed Wajid Ali Taj 0333-857-6592 Sherbaz Husnain Naqvi 0346-286-2531 Qazi Muhammad Ibrahim 0333-605-9218 Hafiz Muhammad Zubair 0333-858-6436 Day 1: 21st, December 2011- Lahore Venue Name and Designation of the Participants Contact Details Punjab Population Welfare Department, Lahore Mr. Javed Amjad, Secretary Dr. Attaya Maroof, Additional Secretary Dr. Aysha Qureshi, Deputy Secretary Dr. Bushra Amjad, Director Technical Regional Training Institute Dr Asma Rana, Principal Dr. Tabassom Mahmood, Dy. Principal Dr. Fozia Masood, Sr. Instructor Mrs Syeda Shamim Akhtar, Sister Tutor Day2: 22 December, 2011-Lahore Health Department, Punjab Civil Secretariat, Lahore Dr, Zafar Ikram, Provincial Coordinator, MNCH Program 03078836259 Dr. Tariq Sultan Butt, Additional Director, MNCH Program 03334805261 Dr. Chauhan, Deputy Provincial Coordinator, LHW Program 03364011515 RHS-a Vasectomy Centre, and RHSA Centre, Sir Ganga Ram Hospital, Lahore Dr. Amjad, In-charge, Vasectomy Centre 03009431284 (ramzan@hotmail.comAmjad_ Dr. Farzana, In-charge, RHSA Center 042-36375880 03334501784 doctorfarzana5@gmail.com Family Advancement for Life and Health – Evaluation Report 61 December 23rd, 2011-Karachi Venue Name and Designation of the Participants Contact Details Regional Training Institute, Clifton Karachi Dr. Razia Begum, Principal 99251005 Dr. Sughra Abbassi, Deputy Principal 0300-253-8747 Sofia Waseem, Instructor 0300-226-0076 Khalida Tasneem, Sister Tutor 021-351-2818 Population Welfare Department, Govt. of Sindh Allah Dino Ansari, Additional Secretary (CTL&S) 0334-371-7148 Ashfaq Shah, Additonal Secretary 0345-255-2751 December 23, 2011-Karachi Venue Name and Designation of the Participants Contact Details Central Warehouse, Karachi Syed Ilyas Haider, Director Planning and Development Division December 24th, 2011-Karachi Venue Name and Designation of the Participants Contact Details RHSA Training Center, Abbassi Shaheed Hospital Dr. Samina Asif, In-charge Training Center 0300-216-5572 Health Department, Sindh Dr. Feroz Memon, Population Council LHW Program, Sindh 0300-308-8112 Dr. Khalid Sheikh, Additional Secretary Development, Health 0300-899-3704 Dr. Suresh Kumar, Special Secretary Health 0333-376-6163 Mrs. Asma Jatoi, Provincial Health Education Officer, Focal Person FALAH 0331-350-4108 Sr. Saqib Ali Sheikh, Section Officer, Public Health 0300-224-7244 Dr. Sahib Jan Badar, PD-MNCH Program, Sindh 0333-214-6464 Family Advancement for Life and Health – Evaluation Report 62 December 26th, 2011-Thatta Venue Name and Designation of the Participants Contact Details EDO Office, Thatta Dr. Maqbool Ahmed, DWNP SFP Dr. Zahoor Ahmed, DOH Admin and Accounts Dr. Agha Iftikhar, DHO District Thatta 0300-212-6089 Dr. Hassan Gandro, District Focal Person (DFP), MNCH Thatta 0321-210-5469 Dr. Ghulam Qadir Sheikh, THO HQ Thatta 0300-268-2689 Dr. Muhammad Hanif Memon, Deputy Taluka Health Officer, Preventive 0321-328-2502 Qazi Nadeem Sarwar, District Population Welfare Officer Hassan Ali Memon, Executive Officer (MEE), PPHI 0301-228-2801 Gharo, Lady Health House 20 MWRA – Zeenat, Vilayat, Niayamat, Tabassum, Baghi, Ghanwa, Shabana, Jhana, Zareena, Azina, Shahnaz, Imamzadi, Chagha, Karima, Zeenat, Chatal, Parveen, Surkhani, Tahira, Nadra Religious leaders, Thatta Molana Muhammad Yaqoob Umrant, Masjad Sona Gali, Thatta City 0323-3842553 Molvi Riaz Hussain Khatib Bukhari Nasjid Dargah Bukhari 0322-2193771 Hafiz Muhammad Ishaq Mamon, Madina Masjid Islampura, Thatta 0306-2695840 Qari Abdul Basit Sayal, Jamia Masjid Shah Jehan, Thatta 0321-2343389 Qari Muhammad Bilal Syed, Jamia Masjid Al-Madina Mir Jo Ghoth 0313-5799948 Meeting with married men Meeting with nine married men held in village in Ismail Jokhio Meeting with community mobilizers of PWD Meeting with seven community mobilizers held in Dhabeji, Thatta, Sindh December 27th and 28th: Meetings in Islamabad Venue Name and Designation of the Participants Contact Details Population Council Office, December 27th Dr. Mumtaz Esker, Director General, Population Cell, Planning and Development Division, Islamabad Dr. Arshad Chandio, Registrar Health Services Academy/Former Deputy Coordinator of the National Program for Family Planning Serena Business Complex, December 28th Dr. Nabeela Ali, COP PAIMAN Family Advancement for Life and Health – Evaluation Report 63 Annex 7: Questionnaires and Guidelines Guidelines for Group Discussion With MWRA =============================================================== Introduction: Structure and Methodology of the FGDs: Group Size: 10-15 MWRA Duration of Session: Each session will be approximately 60-90 minutes in duration. Here again, flexibility will be maintained where sessions may be slightly shorter or longer than the specified time. Such variation may occur depending upon the degree of information, experience, and interest of the participants on the topic under discussion. Data Collection and Transcription: These will later be transcribed in Urdu and then translated in English for final analysis. In addition to the recording, each facilitator will be asked to write notes after each session including her observations, findings etc. This will help in the final analysis of the transcriptions. The FGD Process: An outline of the essential components of the FGD below provides a guideline for conducting the session. The sequence and content of these elements must essentially remain the same across all FGDs. These elements are: 1. The Welcome: Participants are thanked for taking the time out to attend the FGD. 2. Introduction of Team and Purpose of FGD: Facilitator will introduce herself and the note taker. Purpose and the objectives of FGD are explained. (Potential queries at this point should be answered). 3. Ground Rules: Also known as norm setting is clearly outlined. This also included stressing that FGD is not about answering the facilitator’s questions individually but to engage in a discussion with other participants. (It is the facilitator’s job to ensure that these rules are understood and implemented during the session). 4. Introduction of Participants: An icebreaker, this allows participants to feel at ease and allows the facilitator to know the participant somewhat (the facilitator should ensure that members address the group and not the facilitator). Family Advancement for Life and Health – Evaluation Report 64 FGD Questions: MWRA Date: _________2011 Start and end time: _________________________________ Site: _________________________ No. of participants: _____________ Ages: Education: 1. 00 2. Read and write 3. Read only 4. 1-5 yrs 5. 6-12 yrs 6. 13-14 yrs 7. College/University Married/Single: NOTE: PROBE ALL QUESTIONS. Questions: 1. Perception of a healthy pregnancy: What do you think is a healthy pregnancy? 2. Perceived causes/advantages/disadvantages of having many children, less children: a. Why do couples have many children? b. What do you see as the advantages/disadvantages of having many children or fewer children? 3. Decisions about time, spacing, and number of children: Who makes the decisions about time, spacing, and number of children? 4. Knowledge about contraceptives: Have you heard about contraceptives? a. List all mentioned: i. traditional ii. modern (Condoms, pills, IUCD, Injections, ECP, LAM, Beads SDM) b. Do you know of any advantages and/or side effects? Family Advancement for Life and Health – Evaluation Report 65 5. Practice of contraceptive use (Perceptions /trends in the community): a. Have you heard about HTSP? b. Is HTSP/FP used in your community? c. Has use increased recently [after FALAH project]? 6. Knowledge and practice of about HTSP: What does HTSP include? a. Birth spacing – 24 months between two pregnancies (why, why not?) b. Birth spacing – 06 months before next conception after a miscarriage or abortion c. Appropriate age for first pregnancy 7. Knowledge /practice about LAM: Have you heard of LAM? Criteria for LAM: What are the Criteria for LAM? a. Within 6 months of delivery b. Breastfeeding exclusively day and night c. No menstruation 8. FP Information: Have you heard of FP? a. Type of information: What have you heard about FP? b. Type of messages: What messages were you given? c. Source: Where did you hear of FP? i. Meetings ii. Theatre iii. Print and electronic media iv. Counseling v. LHWs vi. Health care providers vii. FHW viii. Relatives/friends 9. Knowledge about FALAH Project: a. Have you heard about the FALAH Project? b. Who, how, when, where: When, where and how did you learn about it? c. Concerns: Do you have any concerns about this project? d. Room for improvement: Would you recommend any changes in the project? 10. Source of obtaining contraceptives: Where do couples get contraceptives from? Do they know where to get supplies? a. Govt Health Facility b. FWC c. Chemist d. Other local source 11. Accessibility: Is it easy to get what you want/need? Family Advancement for Life and Health – Evaluation Report 66 Interview Guidelines - Leadership Fellows 1. Purpose 2. Setting for interview/discussion 3. Target group Questions 1. Can all of you briefly describe your roles and responsibilities one by one? 2. What role do you play in promoting birth spacing and use of contraceptives? 3. Tell us about the leadership training you received from FALAH: a. When was the training held what was its duration? b. What management knowledge and skills you learned? c. What did you like and did not like of that training? d. How that training wasdifferent with other management training, if you had attended earlier? e. What topics that trainingcovered? f. Did that training focus on managing family planning programs? g. Did you manage to apply learned knowledge and skills in your work setting? How? h. What challenges you faced while applying the learned concepts? i. Did you observe any changes? What changes? j. What would you suggest to make thattraining more effective and useful? 4. What is the major concept of “Transformational Leadership Training” you attended? 5. What did you contribute in developing behavior change communication strategies to promote birth spacing and use of contraceptives? 6. Did that training enhance your supervisory skills? If yes, explain. 7. Did that training develop/enhance your team building skills? If yes, explain. 8. Did that training improve your planning skills? If yes, explain. 9. If you get a chance to manage FALAH project as National Project director, how differently will you manage the project? Family Advancement for Life and Health – Evaluation Report 67 10. As we are all aware, provinces and districts will have significantly more responsibility for managing family planning and maternal and child health programs. Do you think the knowledge and skill you received in the Leadership Training will help you to better manage the increased responsibilities you have been given? 11. Will the leadership training contribute to sustaining the changes that have resulted from the training and BCC components of FALAH? Do you feel you have the capacity to sustain and provide refreshers for groups (religious leaders, community groups, etc.) who have been involved in the FALAH project? If yes, how? If no, what would it take for you to have the confidence to carry on? (For example, if they were asked to expand the training of religious leaders to all who need it in the district). 12. Would you recommend this leadership training to your colleagues in other districts who have not yet received it? 13. Do you have any recommendations how future leadership training like this could be improved, made more effective, or relevant to the work that you do? Family Advancement for Life and Health – Evaluation Report 68 Interview Guidelines - Health Care Providers 1. Purpose 2. Setting for interview/discussion 3. Target group Questions 1. Did you receive any training on Family Planning? If yes, a. When, by whom? b. What new knowledge and skills you gained? c. What part/s of that training you liked most? d. What part/s of that training you suggest to further improve? What would you suggest in this regard? e. Did that training cover the concepts of HTSP and LAM? What do you know about HTSP and LAM? f. What do you think about birth spacing as an intervention for healthy mother and child? g. Did you manage to apply learned knowledge and skills in your work setting? How? h. What challenges you faced while applying the learned concepts? i. Did you observe any changes? What changes? j. What would you suggest to make that training more effective and useful? 2. What types of services are provided at this health facility? a. What types of services you provide? (Name all) b. What types of family planning services you provide? (Name all) c. If a desired FP method is not available at your health facility, do you refer the client? d. Where do you refer the clients? e. Do you have any mechanism to get feedback from the referred clients? Do you get the feedback? If no, why not? 3. Do you follow any family planning policy? a. Is this policy available as a written document? b. Is this policy available at all facilities? c. Can you share the policy document? 4. Does your health facility set any family planning targets? a. What are those targets? b. How do you measure those targets? c. What happens to your planning if you meet the targets or do not meet the targets? Family Advancement for Life and Health – Evaluation Report 69 d. Does your health facility get any incentive/s for achieving the set targets? What kinds of incentives? How often? By whom? 5. Does your health facility provide any incentives to clients for using specific FP services? a. If yes, who are beneficiaries? For what kinds of FP services? b. What kinds of incentives are provided? If cash, how much? c. How compliance is determined to provide incentives? d. If a client stops using the specific service, do you also stop providing incentives? 6. In a family of your client, who usually takes the decision to obtain and use family planning services? 7. While providing family planning services, what types of information you provide to the clients? (Proper method to use, Risks, Side effects, Benefits) 8. Are vasectomy and tubal ligations performed at your health facility? a. Is consent is taken on a form? Can you share such form? 9. What do you think about the level of knowledge and awareness about birth spacing and its methods among your clients? a. How do you promote birth spacing? b. Do you provide brochure, pamphlets to you clients (Can you share?) c. Do you use charts and posters? (Let’s see) d. Do you hold awareness raising meetings/sessions with local communities? Who does it? For which target population? How often? e. What challenges you face in this regard? f. What do you think is the most effective way of disseminating information about birth spacing and its methods to your target population? 10. What do you think were major challenges for FALAH Project? a. Acceptability, accessibility, resources, quality, political interference 11. Are you aware of any success story/stories of FALAH Project? a. If yes, can you share with us? 12. Do you think the training program and BCC strategy that FALAH used will be institutionalized by the provincial ministry of health and population? a. If yes, how? b. If no, why not? 13. What do you think are important lessons learnt from FALAH Project that can benefit future reproductive and family health project in Pakistan? Family Advancement for Life and Health – Evaluation Report 70 Discussion Guidelines for Currently Married Men =============================================================== Introduction: Structure and Methodology of the FGDs: Group Size: 10-15 currently married men Duration of Session: Each session will be approximately 60-90 minutes in duration. Here again, flexibility will be maintained where sessions may be slightly shorter or longer than the specified time. Such variation may occur depending upon the degree of information, experience and interest of the participants on the topic under discussion. Data Collection and Transcription: These will later be transcribed in Urdu and then translated in English for final analysis. In addition to the recording, each facilitator will be asked to write notes after each session including her observations, findings etc. This will help in the final analysis of the transcriptions. The FGD Process: An outline of the essential components of the FGD below provides a guideline for conducting the session. The sequence and content of these elements must essentially remain the same across all FGDs. These elements are: a) The Welcome: Participants are thanked for taking the time out to attend the FGD. b) Introduction of Team and Purpose of FGD: Facilitator will introduce herself and the note taker. Purpose and the objectives of FGD are explained. (Potential queries at this point should be answered). c) Ground Rules: Also known as norm setting is clearly outlined. This also included stressing that FGD is not about answering the facilitator’s questions individually but to engage in a discussion with other participants. (It is the facilitator’s job to ensure that these rules are understood and implemented during the session). d) Introduction of Participants: An icebreaker, this allows participants to feel at ease and allows the facilitator to know the participant somewhat (the facilitator should ensure that members address the group and not the facilitator). Family Advancement for Life and Health – Evaluation Report 71 FGD Questions for Men 1. What do you consider a pregnancy as a healthy pregnancy? a. What lead to a healthy pregnancy? 2. What are the advantages/disadvantages of having many children? 3. What are the advantages/disadvantages of having fewer children? 4. Do you think that there should be an interval between two pregnancies? a. What should be the interval between two pregnancies? b. What are advantages/disadvantages of birth spacing? c. If a woman has miscarriage/abortion, after how much she should be conceived again? What should be the appropriate age of marriage for a girl? d. Minimum age? e. Maximum age? f. Advantages/disadvantages of marriage of a girl before the age of 18 or after the age of 35? 5. Who makes the decision to have child/more children? a. You, your wife, your mother, your father, or someone else 6. Who makes the decision to have birth spacing? a. Do your wife, mother and other family members know the advantages of birth spacing? 7. What methods do you know for birth spacing? a. Can you name any advantages of using contraceptives? b. Are there disadvantages of using contraceptives? If yes, are there ways to overcome these disadvantages? 8. What do you know about use of contraceptive practice in your community? a. Roughly what percentages of married couples use contraceptives? b. What types of contraceptives they use? c. What are the reasons for those who do not use contraceptives? 9. Do you know advantages/disadvantages of continuous breastfeeding immediately? a. What do you know the role of continuous BF in birth spacing? i. Probe to discuss: Within 6 months of delivery, breastfeeding exclusively day and night, no menstruation 10. Where do you get your information about birth spacing and use of contraceptives? a. Who provided this information? b. How this information was provided? (Probe for: Community meeting, seminar, theatre, print and electronic media, counseling by a health worker) c. If provided through written messages, were messages easy to understand? Clear? d. If provided through counseling, who provided this counseling? Was the language and messages clear to understand? e. Is there any place or person who can answer questions you have about this information? f. Any comments about the source of information 11. Do you know where you can get contraceptives from? a. Name the source b. How far you have to travel to obtain contraceptive? c. Is it easily available? If not, why? 13. If you received information about birth spacing or contraceptives from the FALAH project, would you recommend that this type of information be provided to other places in Pakistan that were not part of the project? If yes, why? If not, why not? 14. Do you have any recommendations about ways to help married men to become better informed about birth spacing and healthy timing of pregnancy for the health of mothers and children? Family Advancement for Life and Health – Evaluation Report 72 Discussion Guide for Religious Leaders 1) Name and address 2) Have you ever attended any orientation on birth spacing and its impact on maternal and child health? 3) When and where did you get this orientation for the first time? 4) When was the last time that you attended such orientation? 5) Did this orientation change your opinion or you already were of the opinion that birth spacing is beneficial for maternal and child health? 6) I want your opinion on an important subject. Quran says a woman should breastfeed her child for two years. This is better for the mother and the child. A mother is protected from getting pregnant in the first 6 months of child’s birth if the child gets exclusively breastfeeding. After that she can conceive anytime. What should a woman do if she does not want to conceive and continue to breastfeed her child for two years? 7) Do you address your congregation on health matters? 8) On what health subjects do you talk in your congregation? (probe for MNCH and FP) 9) How often do you do this? 10) If no, what are the reasons that you do not talk about this in spite of the fact that you got orientation about birth spacing? 11) In your opinion, what is the best way of motivating married couples to have spacing between pregnancies for better health of the mother and child? 12) Have you ever heard of FALAH? 13) Did you get any orientation about FALAH Project? 14) What was the main focus of FALAH project? 15) If yes, do you think there were activities of FALAH that were particularly beneficial to the health of families in your area? If yes, can you tell me what they were? 16) With regard to these beneficial interventions, are there certain ones that you feel should be offered to other areas of Pakistan that were not included in the project? 17) Do you think that you will be able to sustain including messages about FP/MNCH in your talks, sermons and counseling? If not, what would make it possible for you to do so? Family Advancement for Life and Health – Evaluation Report 73 Discussion Guides for Lady Health Supervisors =============================================================== Introduction: Structure and Methodology of the FGDs: Group Size: 10-15 MWRA Duration of Session: Each session will be approximately 60-90 minutes in duration. Here again, flexibility will be maintained where sessions may be slightly shorter or longer than the specified time. Such variation may occur depending upon the degree of information, experience and interest of the participants on the topic under discussion. Data Collection and Transcription: These will later be transcribed in Urdu and then translated in English for final analysis. In addition to the recording, each facilitator will be asked to write notes after each session including her observations, findings etc. This will help in the final analysis of the transcriptions. The FGD Process: An outline of the essential components of the FGD below provides a guideline for conducting the session. The sequence and content of these elements must essentially remain the same across all FGDs. These elements are: e) The Welcome: Participants are thanked for taking the time out to attend the FGD. f) Introduction of Team and Purpose of FGD: Facilitator will introduce herself and the note taker. Purpose and the objectives of FGD are explained. (Potential queries at this point should be answered). g) Ground Rules: Also known as norm setting is clearly outlined. This also included stressing that FGD is not about answering the facilitator’s questions individually but to engage in a discussion with other participants. (It is the facilitator’s job to ensure that these rules are understood and implemented during the session). h) Introduction of Participants: An icebreaker, this allows participants to feel at ease and allows the facilitator to know the participant somewhat (the facilitator should ensure that members address the group and not the facilitator). Family Advancement for Life and Health – Evaluation Report 74 FGD Questions Date: _________2011 Start and end time: _________________________________ Site: _________________________ No. of participants: _____________ Ages: Education: 1. 00 a. Read and write b. Read only 2. 1-5 yrs 3. 6-12 yrs 4. 13-14 yrs 5. College/University Married/Single: NOTE: Probe all questions QUESTIONS (options listed are specific topics to listen for in the responses): 1. What are your perceptions of a healthy family? (Probe to see if they mention number of children, birth intervals, etc.) 2. What are your roles and responsibilities as a LHS [list]: 3. Can couples limit the number of children that they have? 4. What is the appropriate age for the first pregnancy? 5. Why would families want to limit the number of children they have? (Listen for responses) a. Health of mother: b. Health of child: 6. What is the appropriate Birth Interval to achieve HTSP [listen for between live births / after abortion]?: 7. Promotion of Breast Feeding [early, exclusive, duration]? 8. What methods of birth spacing were you trained on? (e.g. Training on HTSP, LAM, contraceptives): a. Who was responsible for the training? b. What did it cover? c. When did you get the training? d. Are there messages and/or skills you learned that you find useful? e. Dissemination including sites: Do you use HTSP skills and where? If yes, how often? f. Methods of dissemination: i. Meetings ii. Group discussions iii. Individual counseling [method-specific counseling] g. Have you had any post-training follow up or refresher training? If yes, what were they? 9. Challenges: What challenges do you face in your tasks? Family Advancement for Life and Health – Evaluation Report 75 10. Way forward: Would like to recommend any changes that would help you to promote HTSP in your work?? Family Advancement for Life and Health – Evaluation Report 76 Interview Guide for Family Welfare Workers 1. Name of Family Welfare Worker 2. FWC location 3. When were you posted at this facility? 4. How far away is your residence? 5. Is the facility adequately staffed? 6. What contraceptives are available in the facility? 7. What type of contraceptive is in greater demand at this facility? 8. Are there any contraceptives here that clients don’t like? 9. Were there any stock outs during the past 6 months? 10. Are any contraceptives methods were out of stock? a. If yes, which ones? b. For how long? c. What were the reasons for the stock out? 11. Were the tock out problems resolved? Who resolved the stock out problems? 12. Have you heard of FALAH project? 13. Did you have any refresher training under FALAH Project, if yes where and for how much period? 14. What were the main subjects of the refresher training? 15. Do you think the refresher training was relevant and useful in your work? If yes in what ways it was useful? If not, why was it not useful? 16. Can you show me Client Register please? (If Register available, record monthly clients record for the period January 2010-November 2011). 17. Have you heard about Healthy Timing for Spacing Pregnancies? If yes, what does it mean? (If the FWW knows about HTSP ask for the source of her knowledge). 18. What kind of counseling do you provide to the first time FP users? 19. What kind of side effects is reported by clients and how do you respond to their complaints? 20. What kind of treatment facilities is available at this facility? (medicines, equipment etc) 21. Do you need any further training to improve your proficiency? If yes in what areas would you need training? 22. What is the catchment area of this facility? Do you personally know most of your clients, some of your clients or none of your clients? Have you ever tried to build any interpersonal relationship with your clients? Family Advancement for Life and Health – Evaluation Report 77 23. Is there any follow up mechanism established at this facility? If yes, do you have Client’s Record Card available at the facility? 24. Are you satisfied with your job? If not what are the problems that you face here? 25. (ADD Compliance question here if appropriate) Family Advancement for Life and Health – Evaluation Report 78 Discussion Guide for Lady Health Workers’ Supervisors (LHS) Structure and Methodology of the FGDs: Group Size: 10-15 MWRA Duration of Session: Each session will be approximately 60-90 minutes in duration. Here again, flexibility will be maintained where sessions may be slightly shorter or longer than the specified time. Such variation may occur depending upon the degree of information, experience and interest of the participants on the topic under discussion. Data Collection and Transcription: These will later be transcribed in Urdu and then translated in English for final analysis. In addition to the recording, each facilitator will be asked to write notes after each session including her observations, findings etc. This will help in the final analysis of the transcriptions. The FGD Process: An outline of the essential components of the FGD below provides a guideline for conducting the session. The sequence and content of these elements must essentially remain the same across all FGDs. These elements are: i) The Welcome: Participants are thanked for taking the time out to attend the FGD. j) Introduction of Team and Purpose of FGD: Facilitator will introduce herself and the note taker. Purpose and the objectives of FGD are explained. (Potential queries at this point should be answered). k) Ground Rules: Also known as norm setting is clearly outlined. This also included stressing that FGD is not about answering the facilitator’s questions individually but to engage in a discussion with other participants. (It is the facilitator’s job to ensure that these rules are understood and implemented during the session). l) Introduction of Participants: An icebreaker, this allows participants to feel at ease and allows the facilitator to know the participant somewhat (the facilitator should ensure that members address the group and not the facilitator). FGD Questions Date: _________2011 Start and end time: _________________________________ Site: _________________________ No. of participants: _____________ Ages: Education: 6. 00 Family Advancement for Life and Health – Evaluation Report 79 a. Read and write b. Read only 7. 1-5 yrs 8. 6-12 yrs 9. 13-14 yrs 10. College/University Married/Single: NOTE: Probe all questions QUESTIONS (options listed are specific topics to listen for in the responses): 1. What are your perceptions of a healthy family? (Probe to see if they mention number of children, birth intervals, etc.) 2. What are your roles and responsibilities as a LHS [list]: 3. Can couples limit the number of children that they want? 4. What is the appropriate age for the first pregnancy? 5. Why would families want to adopt birth spacing? (Listen for responses) c. Health of mother: d. Health of child: 6. Are you familiar with the term “Healthy Timing and Spacing of Pregnancy and the three messages?” 6. What is the appropriate Birth Interval to achieve HTSP [listen for between live births / after abortion]? 7. Can you list three health benefits of HTSP (healthy baby, healthy mother, economic, time for the baby and family) and three adverse outcomes for short birth intervals after a miscarriage or a live birth (premature birth, low birth weight, miscarriage, maternal death) 7. Promotion of Breast Feeding [early, exclusive, duration]? 8. What methods of birth spacing were you trained on? (e.g. Training on HTSP, LAM, SDM , and other contraceptive methods): h. Who was responsible for the training? i. What did it cover? j. When did you get the training? Family Advancement for Life and Health – Evaluation Report 80 k. Are there messages and/or skills you learned that you find useful? l. Dissemination including sites: Do you use HTSP skills to counsel women about birth spacing and where? If yes, how often? m. After counseling on HTSP, have your clients adopted a family planning method to help them practice HTSP? n. Methods of dissemination: i. Meetings ii. Group discussions iii. Individual counseling [method-specific counseling] o. Have you had any post-training follow up or refresher training? If yes, what were they? 11. Challenges: What challenges do you face in your tasks? 12. Way forward: Would like to recommend any changes that would help you to promote HTSP in your work?? Family Advancement for Life and Health – Evaluation Report 81 Interview Guides for Provincial and District Managers 1. Purpose 2. Setting for interview/discussion 3. Target group  Provincial level: o Special Secretary, Health o Director General, Health o Provincial Coordinator, LHW Program o Provincial Director, MNCH,FP o Secretary, Population Welfare o Donor partners (UNFPA, UKAid, AusAID, Packard)  District Level: o Executive District Officer, Health o District Coordinator, LHW Program o District Population Welfare, Officer o Selected Medical Officer In-charge (2-3) 4. Recording of Information Besides written notes by the note taker, Interview/discussion will be tape recorded if permitted by the respondent INFORMATION TO BE RECORDED FOR EACH INTERVIEW:  Name of Respondent:____________________________  Designation of Respondent:________________________  Province/District:________________________________  Facilitator:_____________________________________  Note Taker:____________________________________  Date & Time:__________________________________  Place of Interview:______________________________ Interview Questions Family Advancement for Life and Health – Evaluation Report 82 14. What do you think are the priority health problems in your province/district? 15. Do you consider Family Planning as a priority health intervention? a. If yes, why? b. If not, why not? 16. What do you think about FALAH Project? a. What are the expected benefits? b. What have been its achievements so far? c. How far it has succeeded in promoting the concept of birth spacing and use of contraceptives? 17. Did you play any role in FALAH Project? a. What role did you play? 18. How much FALAH contributed in: a. Capacity building of health care staff through training? i. Were training programs beneficial? If yes, how? If no, why not? b. Improving quality of FP services? i. What c. Changing knowledge, behaviors and practices of married males and females? i. Using what methods? ii. Who were the target audience? iii. What types of messages used? iv. Did they focus on HTSP and LAM? v. Do you think used methods were effective? If yes, how? If not, why not? vi. Any other comments on the methods used? vii. Do you have any evidence of any change? viii. What were strengths and weaknesses of communication and community mobilization methods? 19. What do you think were major challenges for FALAH Project? a. Acceptability, accessibility, resources, quality, political interference 20. Are you aware of any success story/stories of FALAH Project? a. If yes, can you share with us? Family Advancement for Life and Health – Evaluation Report 83 21. Do you think the training program and BCC strategy that FALAH used will be institutionalized by the provincial ministry of health and population? a. If yes, how? b. If no, why not? 22. What do you think are important lessons learnt from FALAH Project? Family Advancement for Life and Health – Evaluation Report 84 Annex 8: Result – D “Social Marketing of Family Planning Commodities” – Year wise Activities (provided by Population Council)74 Year – 1: (June 1, 2007 – May 31, 2008) A. Lead Partner Change Originally Constella Futures was proposed to be the lead partner for Result D. However by the beginning of the fourth quarter Constella was not able to reach an agreement with USAID/Population Council. In May 2008, Constella Futures withdrew its application from the project. Consequently in June, it was agreed by all Partners that Population Council will take on the role as lead partner for Result D. The strategy for implementing Result D is under preparation by the Population Council. B. Meeting with ZAFA Support to ZAFA was one project intervention area that included a defined exit strategy in the beginning of the project. This was stipulated as support to ZAFA’s distribution, marketing, and brand advertising plan on a decreasing scale from the previous USAID funded project over a period of three years. It was envisaged that ZAFA would develop its capacity to fully take over the full distribution of marketing and brand advertising by the end of three years. Due to Constella Future’s withdrawal from the project, support to ZAFA did not follow the plans that were developed at the beginning of the program, all support from FALAH to ZAFA ended in 2011. C. Greenstar Activities Greenstar Social Marketing (GSM) provided the following results to correspond with activities implemented by Greenstar under FALAH in the provinces of Punjab, Sindh, and Balochistan and the districts of Upper Dir, Swat, Battagram, Buner, Charsadda and Lakki Marwat, in Northwest Frontier Province (NWFP). Couple Years Protection in FALAH-supported GSM activities. Planned Achieved Five year target and Cumulative results (FY 07-12) Targets Actual (Yr1+2+3+4+5) Rural CYPs 80,000 * * * Urban CYPs 1,400,000 * * * Total CYPS 1,480,000 1,140,240 11,455,000 1,140,240 The distinction between rural and urban distribution is not yet available in the management information system. The programming has been completed and the data input will start this year. 74 This annex contains an original project document and appears here largely in its original format with only minor changes to improve readability. Family Advancement for Life and Health – Evaluation Report 85 CPYs by Product Year 1 Condoms 447981.61 39.2% OCs 134003.88 11.75 Injectables 64690.57 5.67% IUDs 320643.5 28.12% VSC 172920.5 15.16% The total sales of contraceptives during the reporting period equals 1,140,240 couple years of protection (CYPs), which is 77 percent of the projected goal for the reporting period. About 39 percent of all CYPs were generated by GS condom sales. To achieve this TV spots, radio spots and print advertisements were made for condoms marketing. In terms of the other methods available, 11 percent of CYPs were generated through sales and distribution of oral hormonal contraceptives; 5 percent of CYPs came from injectable hormonal contraceptives, and 28 percent were from intra-uterine devices. Year – 2: (June 1, 2008 – June 30, 2009) RESULT D: ENHANCED SOCIAL MARKETING OF FAMILY PLANNING COMMODITIES Overall accessibility to contraception must be improved in rural areas for both men and women. Effective contraceptive use is dependent upon client satisfaction and success with a chosen method. By having a range of affordable methods available coupled with provider counseling, women are more likely to find a contraceptive method that suits their individual needs. Product distribution takes place in conjunction with the training of health care providers and communication to increase acceptance and reduce misconceptions, such as associated side effects and efficacy of a contraceptive method. Through a dedicated work force of approximately 231 sales staff, Greenstar distributes high quality, subsidized contraceptives to sales points, which include about 80,000 retail outlets and 20,000 pharmacies. An expanded national contraceptive distribution and sales management structure is now in place and nearly 85% of CYP targets have been achieved for Year 2. Greenstar has achieved 1,830,299 CYPs nationally and 211,515 CYPs in the 20 FALAH districts. The district wise breakdown of CYP targets achieved is given below in Table 11. Greenstar’s strategy is to use a total market approach to product social marketing and improve market segmentation, availability and access, demand for branded products and sustainability. The bulk of interventions during this year targeted low-income underserved urban and increasingly rural Pakistanis. In January 2009 a change was made in the distribution mechanism and the Greenstar higher management decided to shift to a national level distributor to increase the distribution network. This resulted in a slowing down of sales and distribution for a month. Family Advancement for Life and Health – Evaluation Report 86 Table 11: CYP Targets for FALAH Districts District CYP Battagram 1,361 Buner 4,043 Charsadda 8,181 Dadu 17,071 D.G. Khan 34,596 Ghotki 16,553 Gawadar 531 Jaffarabad 3,007 Jhelum 28,264 Khuzdar 952 Larkana 26,411 Lasbella 3,163 Mansehra - Mardan - Sanghar 13,486 Sukkur 29,002 Swabi 10,239 Thatta 12,749 Turbat 1,285 Zhob 621 Total 20 Districts 211,515 National CYP achieved 1,830,299 National CYP target 2,141,760 Percent achieved 85.46 Activity D-1: Social marketing of contraceptives Challenges / Lessons learned: Commodity security continues to be an issue for Greenstar. GS has held five meetings so far with potential donors and stake holders including USAID, UNFPA, FALAH / Population Council, KfW, DFID, JICA, EU, CIDA and the Health Systems Strengthening Project funded by USAID. Discussions are still under way and we hope to receive further support in advocating for donor commodity support from FALAH, USAID and UNFPA. Greenstar estimates a $25 million USD commodity funding gap through mid 2012. Sub-Activity D-1.1: Implementation of annual marketing plans for each commodity (condoms, hormonals and IUCDs) Greenstar has been using mass media to promote its products. The following mass media advertisements were made during the reporting year: 7,665 TV spots, 1,882 radio spots and 44 print advertisements for condoms marketing. To monitor product availability in the market, GSM conducts periodic retail audit which gives an insight into the availability of the various products. 98,280 spots were aired in the buses of Family Advancement for Life and Health – Evaluation Report 87 Niazi bus service, 1,760 spots were shown in 80 cinemas nationally and 100,000 railway tickets were branded in Year 2. The following special promotional campaigns were conducted:  Sathi consumer promotion: Sathi consumer promotion ad was made and on aired on different TV channels and an ad was printed in newspapers in Year 2.  Touch consumer promotion: Touch consumer promotion ad was made and aired on different TV channels and an ad was printed in newspapers in the first quarters of Year 2. CYPs by method, Year 2, FALAH. No. (% of overall CYPs) Condoms 627,267 (34.3%) OCs 151,534 (8.3%) Injectables 110,918 (6%) IUCDs 585,542 (32%) VSC 355,038 (19.3%) Total CYPs 1,830,299 Sub Activity D-1.2: Expansion of Commodities Distribution in large villages Greenstar Social Marketing is collaborating with FALAH partners to expand commodity distribution in large villages. GSM held a coordination meeting on rural expansion with RSPN at the Greenstar Head Office in Karachi on May 4, 2009. During the meeting in light of RSPN’s experience, Greenstar sought input on devising a rural expansion strategy. RSPN provided lists of private practitioners and shop keepers identified by RSPs to Greenstar. GSM agreed to do the assessment of service providers identified by RSPs and build their capacity for provision for service delivery. GSM also agreed to organize the two Clinic Sahoolats per month in Union Councils where RSPs will conduct group meetings. Another option that was discussed was to pilot mobile service delivery to reach far-flung rural communities in one or two districts. This will target the communities mobilized by RSPs. Based on its success it can be replicated in other districts of FALAH. On the subject of social marketing of condoms, it was agreed that RSPN will identify community based distributors at the UC level and GSM will identify stockists in these areas who will serve as focal points for distribution of condoms. The stockists will be linked to a wholesale dealer of GSM at Tehsil /district level. All activities agreed with RSPN will be initiated in Year 3. Sub Activity D-1.3: Pilot Private-Public Partnership: Commodities distribution at BHUs Greenstar will identify BHUs in all 20 districts to start the process of negotiations with the Health Departments of the Government to reach a Memorandum of Understanding for distribution of commodities at Basic Health Units. This activity is planned for Year 4. Sub Activity D-1.4: Trade fairs, expositions and local events Awareness and marketing events were held during trade fairs at Karachi from December 19- 21 2008 and in February in Lahore and were attended by approximately 900,000 people. It was a well attended Family Advancement for Life and Health – Evaluation Report 88 occasion for the promotion of birth spacing products and the corporate image of GSM. Health education was provided to mothers and children on the benefits of small family norms. Sub Activity D-1.5: Hotline for consumers The hotline service offered by GSM provides first hand technical information on methods of contraception. During Year 2, 40,542 calls were received through the hotline operations. The hotline centre is based in Karachi with the toll free number accessible nationally. The number is 0800-111-71. This hotline provides a way to seek counseling from experts without disclosing the individual’s identity. The counselors all (graduate doctors) who provide counseling services at the hot line number have been comprehensively trained on FP. These counselors are equipped with a list of GSM franchised providers, so they can refer clients to a GSM provider who is closest to the client. The hotline number is advertised in all GSM mass media/print advertisements. Most of the callers are married and ask questions on available methods of birth spacing, where can they access to get quality services, and what to do in case of unprotected intercourse. Activity D-2: Expansion of pharmaceutical distribution in the private sector In order to ensure availability of contraceptives to consumers, the products must be in stock at all times at outlets that have been trained or have agreed to stock them. Availability is ensured through frequent visits to outlets by the distribution sales force. Greenstar will aim to ensure that all outlets receive regular monthly visits. Commercial outlets historically do not stock contraceptives due to a variety of factors including limited demands, low profit margins and limited availability of products. Greenstar’s contraceptive distribution network has different components:  Hormonals are distributed to over 20,000 pharmacies and 8,000 health providers  IUCD sales through the GSM sales force to franchised GSM providers. Sub-Activity D-2.1: Detailing non-GSM providers and pharmacies with FP information During Year 2, GSM has worked on expanding its capacity and infrastructure to provide coverage in virtually all urban areas including smaller towns. This will allow informed decision making on how to best serve local needs and to maximize contraceptive accessibility, availability and sales. Condom distribution took place through large distributors who employ almost 200 direct sales agents, thus reaching about 110,000 consumers to meet their condom needs. In total, 189,244 detailing visits were carried out by Medical Representatives and sales teams nationally to GSM and non-GSM sales outlets, out of which 9,891 were carried out in the 20 Project districts. Sub-Activity D-2.2: Contraceptive supply to non GS providers and pharmacies Greenstar works with private distributors to ensure that health products are easily accessible, affordable, and available not only in clinics but also in pharmacies, general stores and local corner stores/kiosks. Greenstar’s distribution network has: a sales force of nearly 200 sales agents that provide sales coverage to over 80,000 retail outlets and 20,000 pharmacies; and to all trained and certified network providers for IUD purchases. As a result, Greenstar’s contraceptive supply represents one third of all modern family planning in Pakistan. Family Advancement for Life and Health – Evaluation Report 89 Activity D-4: Expanding Rural Access to FP by Community Organizations (COs) in the 20 districts This activity is planned for Year 4. Activity D-5: FP Commodities Security for CPR increase Sub-Activity D-5.1: Securing FP commodities for Social Marketing Commodity security continues to be an issue for Greenstar and although their contraceptive security proposal was presented in different forums there has been no outcome of support at present. In total five meetings have been held between GSM and potential donors and stake holders including USAID, UNFPA, FALAH/Population Council, KfW, DFID, JICA, EU, CIDA and the Health Systems Strengthening Project funded by USAID. Sub-Activity D-5.2: Enabling environment for commercial sector's increased contribution to contraceptive security Pending approval of sub-agreements with commercial firms, ZAFA and UDL. Activity D-6: Support Commercial Private Sector for Enhancing Contraceptive Security Revision of the strategy to support the commercial private sector With the decision by Constella Futures (CF) to withdraw from the FALAH consortium, Population Council took on the responsibility for CF’s Scope of Work which included leading the effort to support and build the commercial private sector. This also provided an opportunity to revisit the earlier strategy in light of the changed scenario. After a careful review, PC came to the conclusion that continued direct support in sales and distribution to commercial sector partners would be detrimental to the long term objective of making them independent of donor assistance and take on greater ownership for the future growth of their brands. Therefore a modified strategy needed to be put in place that focuses on building the capacity of ZAFA and UDL’s sales force to make them better and more effective sales agents rather than expecting the project to do their work. The resulting savings in cost from abandoning the sales support structure by the project would be rechanneled to put greater emphasis in communications for demand creation and capacity building within the partner’s contraceptive sales and marketing team. The ultimate goal behind FALAH’s modified strategy is to create an environment that would encourage the growth of the commercial contraceptive market in Pakistan thus contributing to improved contraceptive security in the country. Specifically the project would like to ensure that investments in ZAFA and UDL results in creating greater demand for their contraceptive products and ultimately a private sector supply source of quality contraceptives that would grow and sustain without requiring long term donor assistance. The role of PC/FALAH will be to act as a catalyst in supporting the partner’s initiatives to move their contraceptive products towards greater improvements in quality and sustainability thus contributing to improved contraceptive security in the country. The commercial sector support strategy was revised by PC and shared with USAID. It highlighted FALAH”s revised role in the proposed partnership with ZAFA and UDL. Accordingly, the terms of a draft sub agreement and a budget was developed for support to the respective partners and submitted to USAID for approval on 28 May 2009 Year – 3: (July 1, 2009 – June 30, 2010) Family Advancement for Life and Health – Evaluation Report 90 Result D: Enhanced Social Marketing of Family Planning Commodities 1. Overview FALAH has delivered over 2,851,000 Couples Years Protection (CYPs) nationally against the annual target of 2,811,461 which represents 101% achievement of the annual objective. Compared to the previous year, FALAH’s national CYPs are up by almost 56%. Compared to year 1, it is up by 113%. A vast majority of the CYPs (96%) have been delivered through GSM sales whereas 4% have been contributed by the commercial sector partners. In FALAH districts, total CYPs delivered amounted to 791,602 or 28% of the national total. Compared to the previous year, CYPs in FALAH districts have increased by 275%. It should be noted that FALAH CYPs in year 2 were based entirely on GSM sales and based on 20 districts compared to 26 districts in Year 3. The GSM conducted sales activities with ready stock in the villages to increase rural penetration of its contraceptive products. As a result Greenstar Area Sales Officers were able to increase their coverage to 776 villages/ towns and 15,411 shops. Also during the year, 189,238 visits were made to service providers, pharmacies and retail outlets by the Greenstar medical representatives for product promotion. GSM consumer hotline received over 28,400 calls in the year seeking information on various reproductive health issues. FALAH’s support to commercial sector partners ZAFA and UDL started in September, 2009. Until June, 2010, their combined contraceptive sales have delivered 112,742 CYP’s. During this period, FALAH supported ZAFA to launch a TV commercial to promote the Famila oral contraceptives and visit 5,000 commercial sector providers for providing information on birth spacing and contraceptives. FALAH supported UDL to conduct training of its sales force. As of this year, 48% of the 5 year national CYP target for the project has been achieved. With the rising sales trend and the partnership with the commercial sector, FALAH is very optimistic of achieving its end of project CYP target. 0 1,000,000 2,000,000 3,000,000 Year 1 (2007/2008) Year 2 (2008/2009) Year 3 (2009/2010) CYPs delivered by FALAH CYP Family Advancement for Life and Health – Evaluation Report 91 2. Greenstar Social Marketing Generating demand for GSM contraceptives through the mass media Creating demand for GSM products and brands is a major component of Greenstar’s social marketing efforts. This encourages sustainability of the program through increased sales and revenues. Greenstar made extensive use of the mass media and has also worked on messages related to increasing awareness of birth spacing, male involvement, and promoting behavior change to seek high-quality family planning services. Greenstar extensively aired its Touch Condom advertisement to help grow sales of its higher priced condom brand to help improve cost recovery. Greenstar aired 22,092 television spots and 2,184 radio spots to promote its different contraceptive brands. Concepts for the new Sathi and Sabz Sitara campaign were presented to the FALAH BCC committee early in the year for inputs from committee members and the campaign has been produced which will go on air in Year 4. Product sales and distribution Greenstar sold almost 95 million condoms, over 2,062,000 cycles of pills, 882,881 injections, provided 275,758 IUCDs and performed 41,127 female voluntary sterilizations. Compared to the previous year, condom sales are up by 61%, pill sales by 21%, injections by 41%, IUCDs by 61% and sterilizations by 34%. Greenstar distributes its products through 90,000 retail outlets, over 7,000 active Greenstar franchised providers and other private sector providers, including male and female doctors, female paramedics, and pharmacists visited for product detailing and supply. During the year, Greenstar’s national distributor fully established and expanded its capacity and infrastructure to provide coverage in virtually all urban areas, including smaller towns. Sub-distributors have also been appointed to give coverage to the rural towns. To increase rural penetration, Greenstar conducted sales activities with ready stock in the villages. This allowed for decision making about how to best serve local needs to maximize contraceptive accessibility, availability and sales. As a result of this expanded program, Greenstar Area Sales Officers were able to increase coverage to 776 villages/ towns and 15,411 shops. Almost 50% of the calls were productive that is they lead to a successful sales event. Complete addresses of these outlets were provided to the national distributor to ensure regular supply to these outlets. Table 5: Contraceptive Sales of Greenstar (in units) Product Sales in Units Condoms 94,925,465 Sathi 77,726,595 Touch 17,198,870 Oral Pill-cycles 2,062,511 ECP 1,066,444 Novodol 868,243 Nova 127,824 Injections 882,881 Femiject 133,887 Megestron 309,511 Novaject 439,483 Family Advancement for Life and Health – Evaluation Report 92 IUCD 275,758 Multiload 265,746 Safeload 9,832 Sterilization-VSC 41,127 Informational and Promotional Activities During their follow up visit to the GSM outlets, Greenstar Medical Representatives distributed IEC material to be kept at the provider outlets for clients. These materials help clients choose the method that matches their reproductive goals, give clients more information about their chosen contraceptive method including its effective use, possible side effects, and how to manage minor problems. Specific Promotional and IEC material were produced and distributed to the network providers on combined injectables, Novodol, emergency contraceptives and Multiload. During the year, 189,238 visits were made to service providers, pharmacies and retail outlets by Medical Representatives. Ward activities continued to be a priority and Greenstar regularly held ward activities during the year. The objective of these activities is to emphasize the role of Greenstar in improving the quality of life of low socioeconomic group and create awareness of the benefits of birth spacing in improving maternal/child health. The target audiences are doctors, LHVs75, Registered Medical Officers (RMOs) and DHMS from tertiary care hospitals. A total of 1,640 participants participated in the 69 ward activities held in the year. Greenstar participated in the Dawn Lifestyle Exhibition in Karachi in November, 2009, where approximately 13,000 people visited the Greenstar stall and were given a brief orientation on the Greenstar birth spacing services. Consumer hotline The hotline call centre is based in Karachi but the toll free number (0800-111-71) is accessible nationally. The hotline is open for calls twelve hours a day, seven days a week. Counselors are trained by the Greenstar health services staff in reproductive health issues of women and men. Most of the calls are questions related to use of contraceptive methods. The hotline number is printed on all GSM branded products packages and is advertised on all print and media advertisements of Greenstar. During the year, 28,444 calls were made to the GSM hotline center. Key market research studies and findings Retail Condom Audit: A retail condom audit is conducted every month by AC Nielson for Greenstar. Nielson uses a sample size of 5,000 retail shops (excluding pharmacies). The survey gives information on the up take of condoms marketed by Greenstar, commercial companies and other NGOs and Greenstar’s total market share of the condoms. Evaluation Study of the "Sathi" and "Touch" campaigns: To assess the impact of the thematic condom advertising campaign on condom use in Pakistan a research study was undertaken by Greenstar during year three after the Touch advertisement was aired. The results have been very encouraging and show that about 15% of urban married men were aware of the Touch advertisement. Another 16% recalled other family planning advertisements. Awareness of Touch advertisement was associated with a 75 Original FALAH documents, including the material in this annex, use the term Lady Health Visitor (LHV) instead of the more common term in Pakistan, Lady Health Worker (LHW). The terms are synonymous and LHV is retained in this annex to be true to the original project documents. Family Advancement for Life and Health – Evaluation Report 93 higher level of belief in the effectiveness of condoms, reduced embarrassment in the purchase and use of condom and increased discussion of family planning methods. Respondents reported having started the use of condoms and other contraceptive methods as a result of this exposure. An extrapolation of the pertinent data suggests that the Touch thematic campaign may have generated 218,373 users of condoms and 83,777 users of other contraceptive methods. Short marketing studies such as concept testing, pre-testing of materials and messages: Two qualitative research studies were conducted during the reporting year for pre-testing the concept of the Sathi and Sabz Sitara campaign. The finding of the studies was shared by Greenstar at the BCC committee meeting with FALAH. The campaigns were finalized based on the findings of the evaluation study. Baseline retail tracking study: FALAH and GSM agreed to formulate a research technical committee which will have representation by both partners. Greenstar has completed the methodology of this study so it could be used/ modified to conduct a baseline retail tracking study to gauge availability of both GSM and non-GSM supplied contraceptives. The draft methodology was shared by Greenstar with the research technical committee in the first quarter of Year 3 and inputs were sought to finalize the study design. The study was fielded in the last quarter of Year 3. 3. The Commercial Sector The agreement to support the marketing efforts of FALAH’s commercial partners ZAFA and UDL was signed at the end of the first quarter of this year. Since then, over 35,000 injections, 712,000 oral pills, and 182,500 emergency contraceptive pills and close to 3,940,000 condoms have been sold between them at full or commercial prices thus saving considerable cost in commodity to the donors and the Government of Pakistan Table 6: Contraceptive sales of commercial sector partners ZAFA and UDL Partner/Method Cumulative year 3 (Q3+Q4) ZAFA Famila Injections 35,110 Famila Pills 711,724 Emkit 0.75 (ECP) 175,024 Emkit DS (ECP) 7,449 UDL Happy Life condoms 3,937,024 During this period, FALAH supported ZAFA to launch a TV commercial to promote Famila oral contraceptives through several local and regional satellite channels. Over 18,530 TV spots of Famila were aired. FALAH also supported ZAFA visits to 5,000 commercial sector health practitioners to provide them with information on the benefits of birth spacing and to promote ZAFA’s contraceptive brand portfolio. In the same period FALAH supported UDL to conduct training of its sales force. Twenty UDL sales agents and Managers in charge of condom distribution received training in product selling skills and territory management. In addition, FALAH provided a consultant to educate the sales force on USAID FP compliance issues. Family Advancement for Life and Health – Evaluation Report 94 Table 7: Couple Years Protection (CYP) by Partners – National and FALAH districts District Cumulative CYPs year 3 Gawadar 340 Jaffarabad 3,603 Khuzdar 1,085 Lasbella 3,641 Quetta 36,771 Turbat/ Kech 2,267 Zhob 453 Baluchistan-Total 48,160 Charsadda 18,241 Mardan 80,517 Mansehra 10,713 Swabi 20,565 KP-Total 130,036 Bahawalpur 50,784 D.G. Khan 30,824 Jhelum 34,245 Khanewal 97,402 Multan 149,350 Rajanpur 14,365 Punjab-Total 376,970 Dadu 26,297 Ghotki 29,420 Jacobabad 25,296 Karachi 26,331 Larkana 36,775 Sanghar 22,454 Shikarpur 8,867 Sukkur 40,461 Thatta 19,541 Sindh-Total 235,442 All FALAH Districts 791,602 GSM National 2,738,886 ZAFA 73,372 UDL 39,370 National 2,851,628 Family Advancement for Life and Health – Evaluation Report 95 Year – 4: (July 1, 2010 – June 30, 2011) Result D: Enhanced Social Marketing of Family Planning Commodities FALAH’s private and commercial sector partners, Greenstar, ZAFA, and UDL, continued to implement their activities in Year 4. Greenstar activities came to an end on February 28th, 2011 and were close to the end of the third quarter. Similarly, ZAFA activities ended on March 31st, 2011, while UDL is still working with FALAH in the commercial sector. Major accomplishments and activities under Result D in Year Four are outlined below. 1. Greenstar Social Marketing  Greenstar’s medical representatives visited both network and non-network providers to ensure the availability of products and reinforce the importance of advocating for family planning. Medical detailing was used to increase provider productivity and motivation for offering a family planning method. These detailers make targeted, systematic visits to providers with the aim of increasing Greenstar branded sales and distribution. Detailers also identify new service points and link interested private sector providers with upcoming training opportunities. The objective is to encourage private practitioners to advocate strongly for family planning products and voluntary use of contraceptive services. In Year 4, medical representatives made 137,835 visits to reach out to pharmacies and retail outlets.  Greenstar has a hotline call centre to address reproductive health issues and questions people may have about contraceptive methods. The center based in Karachi received 9,163 calls during the year. The toll free number, which is 0800-111-71, is accessible nationally. The five counselors who are medical doctors provide information and counseling to the callers on reproductive health issues and also refer them to nearby Greenstar franchised outlets. The hotline number is printed on all GSM branded products packages and is advertised on all print and media advertisements of Greenstar. 2. Media Campaign in the Commercial Sector  With its partner ZAFA, FALAH conducted a media campaign on promoting Famila 28F as low dose oral contraceptive pills. Famila was promoted with the message “Main houn Famila Woman” meaning “I am a Famila woman”, where women from different fields were shown balancing work and family life by having decision making power over reproduction. The Famila television commercial was released on major Entertainment channels in Pakistan. A total of 2,796 spots of the famila campaign were run over a period of 3 weeks. Approximately more than 130 spots were run every day on popular entertainment channels such as, HUM TV, Geo entertainment, Samaa, TV One, HBO, AXN, Zaiqa, Awaz TV, Apna channel in their prime time slots. Streamers reflecting this campaign “Main houn Famila Women” were also put up on the main streets of Karachi, Lahore, and Islamabad.  FALAH also supported UDL in airing its recently produced TV and radio commercial promoting a Happy Life condoms extension brand called Happy Life Collection. In total, 11,185 spots were telecast by 10 satellite and cable network channels across the country. In addition, 7,063 radio spots of 15 seconds duration were aired by 20 regional radio stations. 3. Capacity Building and Product Information Family Advancement for Life and Health – Evaluation Report 96  FALAH, with support of its private sector partner ZAFA, organized a seminar for the gynecologists in Multan on March 10th, 2011. Approximately 100 female and 10 male doctors attended this seminar which focused on the status of Family Planning in Pakistan, HTSP, contraceptives available in Pakistan, misconceptions about the use of contraceptive products, side effects of contraceptives and their management as well as an overview of FALAH’s objectives and achievements. As a result of this orientation it is hoped that private sector providers will become more proactive advocates of birth spacing.  To improve effectiveness of its sales calls to providers, ZAFA, supported by FALAH, organized three training sessions of two days duration each for its hormonal sales teams in Karachi, Lahore and Islamabad in October. In all, 60 medical representatives and sales managers employed by ZAFA attended. To build the capacity of the UDL sales force FALAH supported 4 training sessions for 35 sales persons in Karachi, Lahore, Multan and Islamabad.  With support of UDL, FALAH also organized 19 chemist orientation seminars, in Khanewal, D.G. Khan, Bahawalpur, Larkana, Sukkur, Ghotki, Rahimyar Khan, Sargodha, Gujrat, Mansehra, Jhelum, Mandi Bahauddin, Haripur, Layyah, Mian Channu, Burewala and Multan. Approximately 508 leading area chemists attended these seminars which focused on the importance of birth spacing for the mother and the child, contraceptive methods available in Pakistan, and features and benefits of Happy Life Condoms.  To promote FALAH supported products of ZAFA, their hormonal sales force visited 1,500 commercial sector providers to detail them on ZAFA contraceptive products and give evidence based information on the benefits of birth spacing. Fifteen seminars were also organized with 415 Health Providers (73 Doctors and 339 LHVs) Lady Health Visitors in Karachi, Lahore, Mirpurkhas, Hyderabad, Bahawalpur, Sahiwal and Sheikhupura.  Hormonal medical sales representatives visited 15,600 private sector health practitioners (doctors and LHVs) nationally having made 112,980 detailing calls. The team also visited 14,000 pharmacies to detail them on ZAFA contraceptive products and evidence-based information on birth spacing benefits.  UDL merchandizing and salespersons covered 61,360 shops overall to sell the happy life collection. UDL’s sales team visited 53,160 shops, of which 36,960 were pharmaceutical shops and 16,200 were grocery stores. In addition, 8,200 shops were visited though sub distributors. Outdoor advertising through pan flex wall panels was also ongoing over the year.  UDL merchandizing and sales team made Happy Life products available to 10,193 shops directly and indirectly Family Advancement for Life and Health – Evaluation Report 97 through sub-distributors of ‘HAPPY LIFE’ in under-covered towns and areas. Of these 6,953 Happy Life condoms were distributed by UDL Sales and Merchandising teams and 3,240 by the sales teams of the sub-distributors. 4. FALAH Helps Support Launch of a Full-priced Private Sector Condom United Distributors Ltd. (UDL), a commercial sector partner of FALAH, has been expanding their marketing effort to increase sales of their full priced condom brand, Happy Life. Encouraged by the consumer acceptance of the currently popular Happy Life dotted condoms, in July UDL introduced the Happy Life Collection brand which includes dotted, ribbed and contoured condoms in different colors in a single skillet. Supported by USAID under the FALAH project, Happy Life Collection sold over 1.145 million pieces in the first three months of its launch. Together with the Happy Life Dotted condoms, total UDL condom sales have reached over 10.5 million this year. All Happy Life condoms are imported from Malaysia and full priced which means that revenues from sales recover 100 percent of their direct and indirect costs. Since they are financed and sourced by the company, long-term savings to donors in commodity assistance is significant. 5. Couple Years of Protection—Private and Commercial Sector  A total of 1,704,199, or 88 percent, of all CYPs were generated by sales of Greenstar contraceptives, with 7 percent contribution from contraceptives sold by ZAFA and five percent from UDL.  During the year, UDL reported sales of 105,731 of Happy Life condoms, with condom sales by UDL providing over 105,731 couple years of protection. This year sales achievements are 173 percent against the sales target because of the growing popularity of the method and the efforts for the wide availability of this brand in the market by UDL Sales Promotion Officers. Similarly there is a 101 percent increase in sales as compared to the last year.  Sales of ZAFA products stood at 130,095 which is about 97 percent of their annual CYP target.  441,537 CYPs were generated by contraceptive sales in FALAH districts, which represent 23 percent percent of all CYPs delivered by FALAH in Year 4. The CYP breakdown, by FALAH districts, is provided in the table below. Family Advancement for Life and Health – Evaluation Report 98 Table 5: Couple Years Protection (CYP) by Partners and FALAH Districts District Cumulative CYPs as of year 4 Gawadar 316 Jaffarabad 1,402 Khuzdar 751 Lasbella 2,173 Quetta 18,922 Turbat/ Kech 636 Zhob 207 Baluchistan-Total 24,407 Charsadda 18,363 Mardan 41,541 Mansehra 11,170 Swabi 17,671 KP-Total 88,745 Bahawalpur 26,853 D.G. Khan 15,696 Jhelum 25,700 Khanewal 37,921 Multan 85,030 Rajanpur 8,581 Punjab-Total 199,781 Dadu 14,382 Ghotki 17,185 Jacobabad 10,233 Karachi 13,418 Larkana 24,522 Sanghar 15,157 Shikarpur 3,936 Sukkur 20,620 Thatta 9,151 Sindh-Total 128,604 All FALAH Districts 441,537 GSM National 1,704,199 ZAFA 130,095 UDL 105,731 National 1,940,025 Contraceptive Sales by Greenstar: Greenstar sold over 66 million condoms, over a million cycles of pills, 544,984 injections, 65,679 IUDs and performed 8,535 sterilizations in the reporting year. Family Advancement for Life and Health – Evaluation Report 99 Table 6: Product Sales of Greenstar Social marketing (in units) Product Cumulative Condoms (Q1+Q2+Q3) 66,494,104 Sathi 56,754,235 Touch 9,739,869 Oral Pill –cycles 1,393,257 ECP 738,908 Novodol 505,390 Nova 148,959 Injections 544,984 Femiject 77,331 Megestron 215,492 Novaject 251,161 IUD 157,360 Multiload 151,128 Safeload 6,232 Sterilizations 22,315 VSC 22,315 CYP Contribution of commercial partners by Contraceptive Method: Contraceptive sales of FALAHs commercial sector partners ZAFA and UDL helped achieve 235,800 couple years of protection (CYPs) during the year. ZAFA sales represented 55 percent of these CYPs, while condom sales from UDL represented 45 percent of CYPs. Table 7: CYP Contribution of commercial partners by Contraceptive Method Contraceptive Method Quantity CYPs ZAFA Famila Injection 55,260 13,815 Famila Tablets 1,311,661 100,978 EMKIT .75 Tablets 266,900 13,345 EMKIT DS Tablets 32,820 1641 Desofam Pill 4108 316 ZAFA-Total 130,095 UDL Condom 105,731 105,731 UDL-Total Total-ZAFA + UDL 235,826 Family Advancement for Life and Health – Evaluation Report 100 SALES The increase in-market sales and CYP is a clear sign that given the support to commercial enterprise like ZAFA & UDL, it can grow rapidly. SALES & CYP GENERATED BY ZAFA PHARMACEUTICALS – OCTOBER 2009 - MARCH 2011 SALES OF CONTRACEPTIVE PRODUCTS BY BRAND & CYP EX-DISTRIBUTORS QUARTERLY SALES & CYP (OCT 2009 - MARCH 2011) BY THE ZAFA SALES TEAM. PRODUCTS OCT-DEC, 2009 JAN-MAR 2010 APR-JUN 2010 JUL-SEP, 2010 OCT-DEC 2010 JAN-MAR 2011 SALES CYP SALES CYP SALES CYP SALES CYP SALES CYP SALES CYP Famila 28F 341913 26301 342096 26315 334194 25707 381333 29333 439227 33787 492156 37858 Emkit 78826 3941 97620 4881 81177 4059 77012 3851 89081 4454 100794 5040 Emkit DS 7405 370 7652 383 14454 723 11977 599 10876 544 9968 498 DESOFAM 1305 100 1141 88 1323 102 1332 103 1412 109 1366 105 ACNOT 421 32 656 50 711 55 846 65 1055 81 2296 177 FAM-INJ. 21783 5446 19820 4955 16097 4024 15325 3831 18489 4622 21445 5361 NORIFAM 158 13 118 10 177 15 85 7 107 9 123 10 TOTAL CYP 36203 36682 34684 37789 43606 49049 CYP ACHIEVEMENT (From Oct 2009 to March 2011). 238,013 Family Advancement for Life and Health – Evaluation Report 101 SALES & CYP GENERATED BY UDL - SEPTEMBER 2009 TO JULY 2011 EX- DISTRIBUTUORS QTLY. SALES & CYP. (UDL) PRODUCTS QTR-2 OCT–DEC 2009 QTR-3 JAN-MAR2010 QTR 4 – APR-JUN 2010 SALES CYP SALES CYP SALES CYP HAPPY LIFE￾DOTTED 1,328,922 13,289 1,139,538 11,395 1,863,422 18,634 PRODUCTS QTR-1,JUL￾SEP,2010 QTR-2 OCT-DEC, 2010 QTR-3, JAN￾MAR,2011 QTR-4 APR-JUL 2011 SALES CYP SALES CYP SALES CYP SALES CYP HAPPY LIFE￾DOTTED 1,247,122 12471 1404472 14045 1807268 18072 1,565,800 15,658 HAPPY LIFE￾COLLECTION 1,145,901 11459 758916 7589 1132869 11329 2,122600 21,226 TOTAL SALES 2,393,023 23930 2163388 21634 2940137 29401 3,688,500 36,885 Family Advancement for Life and Health – Evaluation Report 102 TOTAL CYP ACHIEVEMENT BY UDL FROM OCTOBER 2009 TO JULY 2011 = 155,169 SALES GRAPHS CYP TARGET VS ACHIEVEMENT TOTAL PRIVATE SECTOR (ZAFA & UDL) CYP TARGET VS ACHIEMENT The CYP achievement shows that the number of couples protected from an unwanted pregnancy during the project period was 393,181 which represent a 21.5% increase over the target for the performance period. ZAFA - TOTAL CYP TARGET VS ACHIEMENT 323,465 393,181 TOTAL PRIVATE SECTOR (ZAFA & UDL) CYP TARGET VS ACHIEMENT TARGET ACHIEVEMENT Series1 234,182 238,013 ZAFA - TOTAL CYP TARGET VS ACHIEMENT TARGET ACHIEVEMENT Series1 Family Advancement for Life and Health – Evaluation Report 103 The CYP achievement of ZAFA Pharmaceutical shows 1.6% increase over the target for the performance period. UDL – TOTAL CYP TARGET VS ACHIEVEMENT The CYP achievement of UDL shows 66% increase over the target for the project period. This increase in CYP is due to the launch of new additional brand of condoms “Happy Life Collection”. 93,465 155,159 UDL - TOTAL CYP TARGET VS ACHIEMENT TARGET ACHIEVEMENT Series1 Family Advancement for Life and Health – Evaluation Report 104 QUARTERLY SALES AND CYP ANALYSIS The increase in-market sales and CYP is a clear sign that given the support to Private sector like ZAFA & UDL, it can grow rapidly and achieve a high CYP. COMBINED QTLY. CYP ACHIEVEMENT OF PRIVATE SECTOR PARTNERS- ZAFA &UDL QTR-2 QTR-3 QTR-4 QTR-1 QTR-2 QTR-3 QTR-4 TOTAL CYP CYP ACHIEVED OCT-DEC 2009 JAN￾MAR 2010 APR￾JUN 2010 JUL-SEP 2010 OCT-DEC 2010 JAN￾MAR 2011 APR-Jul 2011 OCT 2009 - Jul 2011 ZAFA 36203 36682 34684 37789 43606 49049 238013 UDL 13289 11395 18634 23930 21634 29401 36885 155168 TOTAL CYP 49492 48077 53226 61719 65240 78450 36885 393181 NOTE: Activities of ZAFA was till March 2011 and UDL July 2011. Family Advancement for Life and Health – Evaluation Report 105 The CYP growth from quarter to quarter during the project period was as follows; 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 OCT-DEC 2009 JAN-MAR 2010 APR-JUN 2010 JUL-SEP 2010 OCT-DEC 2010 JAN-MAR 2011 QTR-2 QTR-3 QTR-4 QTR-1 QTR-2 QTR-3 Axis Title COMBINED (ZAFA & UDL) QUARTERLY TREND IN CYP GROWTH (OCTOBER 2009 - MARCH 2011). CYP ACHIEVED COMBINED (ZAFA & UDL) QUARTERLY TREND IN CYP GROWTH (OCTOBER 2009 - MARCH 2011).