January 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Yagya Bahadur Karki, Gyanu Shrestha, Ritu Prasad Gartaula and Upasana Shakya under the direction of the Population, Health, and Development (PHD) Group, Sanepa, Lalitpur, Kathmandu, Nepal. MID-TERM EVALUATION OF USAID/NEPAL’S GHAR GHAR MAA SWASTHYA (GGMS) PROGRAM FINAL REPORT i MID-TERM EVALUATION OF USAID/NEPAL’S GHAR GHAR MAA SWASTHYA (GGMS) PROGRAM FINAL REPORT January 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Yagya Bahadur Karki, Gyanu Shrestha, Ritu Prasad Gartaula and Upasana Shakya under the direction of the Population, Health, and Development (PHD) Group, Pvt. Ltd. Indraeni, Sanepa, Lalitpur, Kathmandu, Nepal. ii MID-TERM EVALUATION OF USAID/NEPAL’S GHAR GHAR MAA SWASTHYA (GGMS) PROGRAM FINAL REPORT USAID/NEPAL DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS ACKNOWLEDGEMENT.......................................................................................................................................... iv ACRONYMS ................................................................................................................................................................ v EXECUTIVE SUMMARY ............................................................................................................................................ vii EVALUATION PURPOSE ......................................................................................................................................... 1 EVALUATION QUESTION ...................................................................................................................................... 1 PROJECT BACKGROUND ...................................................................................................................................... 2 EVALUATION METHODOLOGY........................................................................................................................ 4 EVALUATION FINDINGS ........................................................................................................................................ 6 CONCLUSION ....................................................................................................................................................... 19 RECOMMENDATIONS.............................................................................................................................................. 21 TABLES Table 1: Comparison of percent distribution of currently married women aged 15-49 years by current use of contraceptive methods in mountain and hill regions, 2011……………………………… ……….9 Table 2: CRS brands by measure of sustainability, 2014/2015 …………………………………………...11 Table 3: Percent achieved against the projected figures, May 2010 - January 2013……………………....13 FIGURES Figure 1: Number of trainee participants in leadership and management development training in four phases in 2012 and 2013 ………................................................................................................................... 7 Figure 2: Annual CYP results based on five spacing methods distributed by CRS, 2010/011 -2011/013 ……………………………………………………………………..…..…... 13 Figure 3: Percentage share of revenue generated from sales of commodities of three sources, May 2010- January 2013 ……………………………………………….…………..……......…… 14 Figure 4: Number of new Sangini outlets opened in 49 hard-to-reach districts, May 1, 2010- Jan. 31, 2013 and likely to open by Mid 2014…………………………...……………………...… …… 15 ANNEXES Annex – I List of hard to reach districts of Nepal covered by GGMS project …………. ........... 25 Annex – II SOW ………………………………………………………………….......................... 26 Annex – III Team Planning Meeting Schedule ………………………………………………….. 50 Annex – IV Tools ………………………………………………………………………………. 51 Annex – V List of key informants………………………………………………………………. 84 Annex - VI Expansion of NTOs in 49 GGMS districts.......................................................................... 85 Annex - VII Rural field representatives ………………………........................................................... 87 iv ACKNOWLEDGEMENT The Evaluation Team expresses gratitude to USAID/Nepal for providing the opportunity to conduct the "Mid-Term Evaluation of USAID/Nepal’s Ghar Ghar Maa Swasthya (GGMS) Program." The team would like to thank Mr. Daniel Sinclair, Team Leader: HIV/AIDS, Logistics and Social Marketing, Office of Health and Family Planning, USAID /Nepal. The team would like to express its gratitude to Mr. Pangday Yonzone - the Contract Officer’s Representative (COR) for the Ghar Ghar Maa Swasthya (GGMS) Program, for his patience in steering the team and providing guidance. The evaluation team would like to thank the other officials of USAID/Nepal Mr. Murari Adhikari, Evaluation Officer; Mr. Ajaya Parajuli, Acquisition and Assistance Agent, Office of Acquisition & Assistance; and Mr. Gajendra Rai for their valuable suggestions and comments during the evaluation process. The team would like to thank Dr. Mingnar G. Sherpa, Director General, Department of Health Services for his valuable suggestions. Dr. Senendra Raj Uprety, Director, Family Health Division (FHD), who is also the Chair of the Board of Directors of Nepal Contraceptive Retail Sales Company Ltd. (CRS), gave valuable suggestions to improve the CRS program. The directors of Child Health Division (CHD), Logistic Management Division (LMD), and National Center for AIDS and STD Control (NCASC) also gave valuable comments and suggestions for the improvement of CRS and its sustainability. District Health Office/District Public Health Office (DHOs/DPHOs) of the 12 districts visited by the evaluation team members deserve special thanks for their cooperation during these field visits. Mr. Hareram Bhattarai, COP; Ms. Moon Pradhan and Dr. Prakash Dev Panta, FHI 360; and Mr. K. B. Rayamajhi, Managing Director; all staff, especially senior staff, of CRS Company; Binod Khambu, GGMS Program associate; Shambhu Singh, Administration and Finance; and Ms. Pema Lakhe, Program Director, NFCC; all key informants in Kathmandu deserve our special thanks for their cooperation and open discussion with the evaluation team. The evaluation team would like to thank all CRS regional managers, officers, and other key staff members who provided us with valuable information during the field visits. The evaluation team members are grateful to RFRs of CRS for their openness when talking about their work. Many respondents from traditional outlets (TOs) and non-traditional outlets (NTOs) of 12 districts and focus group discussion (FGD) participants contributed substantially to enrich the evaluation report for which the evaluation team is highly grateful. The team would like to express sincere thanks to all those who have assisted directly or indirectly with the development and execution of this study. v ACRONYMS AIDS Acquired Immune Deficiency Syndrome CDCS Country Development and Cooperation Strategy CDK Clean Delivery Kit CHD Child Health Division COP Chief of Party COR Contract Officer’s Representative CRS Nepal Contraceptive Retail Sales Company Ltd. CYP Couple Years of Protection DHO District Health Office DOHS Department of Health Services DPHO District Public Health Office DQA Data Quality Assessment eCON Emergency Contraceptive Pill ECP Emergency Contraceptive Pill EDC Effective Demand Creation EPM Effective Product Marketing ERP Enterprise Resource Planning FCHVs Female Community Health Volunteer FGD Focus Group Discussion FHD Family Health Division FHI 360 Family Health International FP Family Planning FY Fiscal Year GGMS Ghar Ghar Maa Swasthya GIS Geographical Information System GON Government of Nepal HCWM Healthcare Waste Management HFP Health and Family Planning HIV Human Immunodeficiency Virus HRM Human Resource Management IDF Institutional Development Framework IR Intermediate Result IUD Intrauterine Device KAP Knowledge, Attitude and Practice KfW Entwicklungsbank or German Development Bank LMD Logistic Management Division LMD Leadership and Management Development MCH Maternal and Child Health M&E Monitoring and Evaluation MER Monitoring, Evaluation and Research MOHP Ministry of Health and Population NCASC National Center for AIDS and STD Control NDHS Nepal Demographic and Health Survey NFCC Nepal Fertility Care Centre NGO Non-Governmental Organization N-MARC Nepal Social Marketing and Franchising Project: AIDS, Reproductive Health and Child Survival vi NTO Non-traditional Outlet OC Oral Contraceptive OCP Oral Contraceptive Pills ORS Oral Rehydration Salt PHD Population, Health and Development Group PSI Population Services International PMP Performance Monitoring Plan PPP Public Private Partnership RH Reproductive Health RFR Rural Field Representative SBCC Social and Behavioral Change Communication SOW Statement of Work STI Sexually Transmitted Infection TA Technical Assistance TO Traditional Outlets TOR Terms of Reference TOT Training of Trainer TPM Team Planning Meeting TSV Technical Support Visit USAID United States Agency for International Development VAT Visibility Accessibility Touch VDC Village Development Committee vii EXECUTIVE SUMMARY Introduction: USAID/ Nepal’s Ghar Ghar Maa Swasthya (GGMS) Program, or Healthy Homes, seeks to improve the institutional sustainability of the Contraceptive Retail Sales (CRS) Company and also to increase availability and accessibility of quality health products and services in hard-to-reach rural areas. GGMS consists of two components. Firstly, USAID funds CRS for social marketing and social franchising activities that are focused on developing sustainability by using commercial marketing in urban areas and promoting subsidized products in 49 rural hard-to-reach districts. Secondly, Family Health International 360 (FHI 360) - funded by USAID - assists CRS to develop and implement business plans and social behavioral change communication (SBCC) activities. Goal: The goal of GGMS is to assist Nepal's government in expanding the depth, reach, and impact of the private sector in social marketing, by providing a subsidized supply of maternal and child health (MCH), family planning (FP), and HIV prevention products and services. EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the mid-term evaluation was to review the GGMS project progress during the first 33 months of the project period. The evaluation team has made recommendations for modifications and mid-course corrections for the second half of the project. Specifically, this evaluation has examined how successful the GGMS partnership has been in building CRS’s capacity to achieve self-sustainability. The evaluation questions for the mid-term evaluation were (i) how effective has the GGMS program partnership been in building CRS’s capacity to operate as a more independent, local social marketing organization; (ii) given any barriers to capacity development are any changes to the current TA structure or additional TA direction recommended)? And (iii) are there any revisions that should be made in the expectations or demands of CRS? The prime audience of this evaluation report will be the USAID/Nepal Mission, specifically the Health and Family Planning (HFP) team, the Mission Country Development and Cooperation Strategy (CDCS) team, the Asia Bureau, and the implementing partners -CRS and FHI 360. A summary of the evaluation, including the description of methods, key findings, and recommendations will be provided to the Ministry of Health and Population (MOHP), Entwicklungsbank (KfW), and Logistics Management Division (LMD)/Department of Health Services (DOHS) and made available to the public online in a fully searchable form within three months of the evaluation’s conclusion. PROJECT BACKGROUND The United States Agency for International Development (USAID)/Nepal’s Ghar Ghar Maa Swasthya (GGMS) Program, or Healthy Homes, seeks to improve the institutional sustainability of the CRS Company and also to increase availability and accessibility of quality health products and services in hard-to-reach rural areas. The program assists the Nepal Government in expanding the impact of the private sector in social marketing and social franchising activities by providing a subsidized supply of maternal child health, family planning, and HIV prevention high quality health products and services. EVALUATION METHODS The evaluation methodology included a range of methods and approaches for collecting and analyzing data that would best allow the project to be evaluated. The evaluation team reviewed the list of background reports and conducted key informant interviews with stakeholders from the government and GGMS partners. A team planning meeting (TPM) was conducted, which included USAID/Nepal officials and Population, Health, and Development (PHD) Group consultants. Data collection viii methodologies were drafted by the evaluation team, discussed in the TPM, and approved by USAID/Nepal. Evaluation activities included desk reviews, interviews with stakeholders/implementers, meetings, FGDs with beneficiaries, and field observations made by travelling to 12 of the 49 project districts. Appropriate tools were developed and administered by evaluation team members in 12 districts outside of Kathmandu valley and in Kathmandu. The evaluation report includes analyses made using the collected data outlined above. FINDINGS Some of the findings of the mid-term evaluation are presented below:  Although FHI 360 delayed its activities at the beginning, overall, GGMS activities have been effective in promoting and expanding quality FP, MCH, and HIV products and services. The annual Couple Years of Protection (CYP) figures achieved exceeded the targets with a monthly CYP of about 30,000 during the first 33 months of the program (Figure 1). Overall, the results are positive as the CYP achieved met the projected goals.  However, in order to achieve the 5-year projected goal of 1.8 million CYP by the end of the project period CRS will have to achieve CYP of 31,589 per month from February 2013 onwards.  In Year One, FHI 360 organized an Institutional Development Framework (IDF) workshop to identify CRS's capacity needs. The needs identified were the strengthening of monitoring and evaluation systems, human resource development, and supply-chain management.  Under the supply-chain management two types of trainings supported by FHI 360 - effective product marketing (EPM) and effective demand creation (EDC) - were rated by all levels of the staff. The staff expressed being able to present their cases with confidence and accuracy as a result of the training. The technical and theoretical knowledge from the training combined with the staff’s own field experience has positively impacted their work. Based on self-reporting, this training strengthened and enhanced the quality of work of CRS employees.  The Five-phase Leadership and Management Development (LMD) for CRS is nearly complete as only the last phase remains. However, this training has not been so effective with CRS staff. FHI 360 has considered CRS’ request to improve the last phase of the LMD training.  In Year Two CRS installed Microsoft Dynamics Enterprise Resource Planning NAV ERP to integrate existing vertical systems (e.g. finance, logistics, human resources, monitoring and evaluation) into a single platform. With support from FHI 360, CRS succeeded in introducing the system, which is fully operational. All staff using the system have undergone extensive training to operate the system. The CRS Finance Department is particularly happy with the new software. 100.6 104.9 101.6 27 31 30 0.0 50.0 100.0 150.0 2010/011 2011/12 2012/13 Figure 1Annual CYP results based on five spacing methods distributed by CRS, 2010/011–2011/013 CYP achievement % Per month CYP ('000) ix  FHI 360 supported five studies/assessments for CRS. However, the CRS senior management team explained that these studies were not the most useful assessments to efficiently build CRS’ capacity and support its work. The knowledge, attitude, and practice (KAP) study on FP and MCH was too long and contained more data than CRS needed. Data on pricing of contraceptive commodities, menstrual hygiene, and use of sanitary napkin were useful; however, data on contraception, MCH products, and media habits were not necessary, as they could be obtained from NDHS too.  Sangini’s quality assurance "Mystery Client Survey" study findings have been used by CRS to assess service quality, IEC materials, stock of products, and physical facilities and their use.  The findings of the "Human resource management study" by Deloitte India are planned to be used to strengthen CRS HRM, developing an HR Manual, filling vacant positions, developing a talent acquisition system, determining compensation for and the training and development of CRS staff.  The "Assessment of the CRS’ distribution system" study by MART gave strategic recommendations to make CRS' distribution system more effective and efficient in hard-to-reach districts. It has also proposed Self Help Groups, Woman Entrepreneurship and Youth-Based Community Groups models which are built on community networks. The top CRS management team appreciates this study but has not yet prioritized the recommendations given.  Sales of CRS condoms, pills, injectable (Sangini), eCON, and Jadelle were executed as planned in the last 33 months, but the sale of IUDs, CURe STI treatment kit, and MCH products such as Clean Delivery Kit (CDK) and Nava Jeevan (ORS) declined over the years. The sale of Nava Jeevan declined mainly because the supply was interrupted.  By mid-term eCON was classified as a full cost recovery product, while Panther Condom, Nilocon White (OCP), ORS, CDK and Piyush (Water Disinfectant) were labeled as cost recovery products.  The third brand of condoms - D'zire- and sanitary pads have been pre-tested. CRS has already marketed the D'zire condom while sanitary pads will be marketed soon and are also likely to be full cost recovery product.  The recruitment of Rural Field Representatives (RFR) marks a paradigm shift1 in retailing quality health and FP products/services, improving people's attitudes towards condom usage, and increasing access in hard-to reach areas. CRS has the capacity to expand RFRs in hard-to-reach districts.  FHI 360 should provide technical support to CRS to develop its capacity to attract donor funding from sources other than USAID. However, it would be best for CRS to increase the sale of its quality FP and MCH products and services and generate sufficient revenue to run its operations independently. CRS has at its disposal a large sum of capital which it can invest profitably. FHI 360 should provide CRS with the skills to develop a business plan in order to invest successfully. RECOMMENDATIONS FOR THE SECOND HALF OF GGMS Recommendations for FHI 360:  FHI 360 must give priority to M&E to strengthen CRS' M&E systems.  FHI 360 should share the initial big-picture findings of any research/assessment/study as early as possible so that CRS program activities can be strengthened in a timely manner.  FHI 360 should improve the Leadership and Management Development (LMD) course module for the last phase. In addition, LMD training should impart skills to senior CRS staff on developing a sustainable business plan.  The next round of KAP study should focus on sanitary napkins, MCH products, and pricing factors 1 RFR recruitment is a new concept unlike regular organizational staffs. RFR engages community mobilization and behavior change communication activities in local hard‐to‐reach areas which helps non traditional outlets (NTOs) to function better. x and leave out the other related data collection. This way the size of the questionnaire would be shorter which would save resources for use to meet competing needs of CRS.  In the next round of the "Mystery Client Survey," FHI 360 should minimize the replacement of sample sites.  FHI 360 should speed up its activities, but before starting a new activity it should assist CRS in translating results of past studies/assessments of specific activities into actions.  FHI 360 should provide additional TA to CRS to train CRS field staff including RFRs on reporting and recording sales data in hard-to-reach areas and in the rest of the country.  Training on marketing skills, record keeping, and orientation on CRS and its commodities should be provided to the NTOs so that they can better market CRS commodities. Recommendations for CRS:  The "Assessment of the CRS distribution system" conducted by MART India proposed models, namely, Self Help Groups, Woman Entrepreneurship and Youth Based Community Groups. These groups can contribute to CRS’ sustainability, as they are groups founded in community networks. However, CRS has not yet made action plans based on these recommendations. It is recommended that CRS acts upon these recommendations soon.  In order to address the problem of slow decision making at CRS, senior positions should be filled immediately. Within CRS, there should be delegation of power to different divisions for speedy decision making. Improving the management structure is an urgent need. Additionally, the gender ratio of the staff should be more balanced and inclusive.  One of the main objectives of the GGMS project is to increase the access of contraceptives and MCH products to people in hard-to-reach areas. The current Performance Monitoring Plan (PMP) does not tell whether project implementation is progressing towards achieving this goal. Therefore CRS should prepare PMP by stratifying by hard-to-reach areas and the rest of the country.  RFRs should mobilize local Mothers' Groups and Female Community Health Volunteers (FCHVs)and conduct Social Behavioral Change Communication (SBCC) activities locally. The Center should delegate its authority to the area manager and RFRs to conduct SBCC activities. Local media, and, if necessary, local languages should be used to educate the local people on contraception, HIV/AIDS, and maternal and child health.  RFRs should be given an allowance to afford rent for one room in each district. He/she can operate this room as an office. DHO/DPHOs should be contacted, and RFRs should establish affiliations with the district health offices.  GGMS should strengthen the supply chain of its commodities. Currently commodities like CDK, IUD, Nava Jeevan, and CURe kit do not exist in sufficient quantities. CRS should prepare a logistics plan so that the supply of its commodities is not interrupted.  CRS should gradually build its capacity to provide quality FP services, including Sangini on which NFCC has been depending for the past three years. Recommendations for USAID:  Both the Deloitte HRM study and the evaluation team found that the salary scales are low in CRS compared to many similar organizations working in Kathmandu. The evaluation team recommends that this be positively addressed by USAID.  GGMS' goal is to increase access for the people of 49 hard-to-reach districts to FP and MCH products but has not deployed sufficient human resources to carry out various tasks. Policies should be developed to assign at least one RFR to each district. The donor fund would be better utilized this way. FHI 360 can assist CRS in this respect.  The evaluation team sensed that CRS was not able to provide constructive feedback to the TA provided by FHI 360. Therefore CRS should be empowered to critically comment and constructively assess the technical assistance provided by FHI 360. 1 EVALUATION PURPOSE The purpose of the mid-term evaluation was to review the progress of the Ghar Ghar Maa Swasthya (GGMS) Program, or Healthy Homes, during the first 33 months of the project period. This external evaluation will help determine which aspects of the project are going well and which are not and the reasons for the challenges and successes so far. The purpose of this report is also to suggest modifications and mid-course corrections to help guide GGMS during the final half of the project. Specifically, this evaluation examines how successful the GGMS partnership has been in building CRS capacity to achieve the targeted results and move toward self-sustainability. The evaluation has also identified ongoing Technical Assistance (TA) needs, barriers to capacity development, and any recommendations for changes in the structure of the TA contract or cooperative agreement. USAID/Nepal put a strong focus on capacity building in its draft Country Development Cooperation Strategy (CDCS). To this end, the evaluation has come up with some findings that will provide input on the success of the modality of providing capacity building through a partnership between an international organization and a local Non-Governmental Organization (NGO). In summary, the evaluation will help all involved to better understand the initial results achieved in organizational capacity development, to re-focus and strengthen the program as required, and inform the implementation of the CDCS mission. EVALUATION QUESTIONS The evaluation questions for the mid-term evaluation were: A. How effective has the GGMS program partnership been in building CRS’ capacity to operate as a more independent, local social marketing organization, in terms of: i. FHI 360‘s identification of CRS’ capacity needs; ii. The quality and appropriateness of the capacity-building TA offered; iii. CRS’ openness to receive and use the TA offered; iv. Building and operationalizing organizational systems and processes to develop CRS as an independent organization by the end of the project; v. Improving the quality and efficiency of CRS’ service delivery, including customer satisfaction. B. Given any barriers to capacity development, are any changes to the current TA structure or additional TA directions recommended? C. Are there any revisions that should be made in the expectations or demands of CRS? 2 PROJECT BACKGROUND Brief Overview of Nepal CRS Company USAID/Nepal’s work on social marketing began in 1978 with the establishment of the Nepal CRS Company, one of the oldest and most successful social marketing2and social franchising organizations in the world. CRS quickly established itself as a pioneer in health communications and marketing and played a key role in developing markets in Nepal for condoms, oral and injectable contraceptive products, like Depo-Provera, and oral rehydration salts. CRS’s influence has resulted in more than 30 brands of condoms and ten brands of oral contraceptive pills being available on the market. With the successes of CRS, new challenges have emerged. CRS remains primarily dependent on USAID funds to support its operations and activities. Additionally, Nepal’s hilly and mountainous landscape makes travel and product distribution difficult; and, where roads exist, political turmoil has frequently limited access. While CRS has products available in all 75 districts of Nepal, hard-to-reach areas still lack access to essential public health products. The United States Agency for International Development (USAID)/Nepal’s Ghar Ghar Maa Swasthya (GGMS) Program, or Healthy Homes, seeks to improve the institutional sustainability of Contraceptive Retail Sales (CRS) Company and also to increase availability and accessibility of quality health products and services in hard-to-reach rural areas. The program is assisting the Nepal Government in expanding the impact of the private sector in social marketing and social franchising activities by providing a subsidized supply of high quality maternal child health, family planning, and HIV prevention health products and services. GGMS consists of two components. Firstly, USAID funds CRS for social marketing and social franchising activities, focusing on sustainability through commercial marketing in the urban areas and promoting subsidized products in the rural, hard-to-reach areas (see Annex I for districts). Secondly, Family Health International 360 (FHI 360), funded by USAID, assists CRS in developing and implementing business plans and behavioral change communication activities, such as campaigns promoting family planning best practices. USAID Program Strategy and Activities Aimed at Institutional Sustainability of CRS and Increasing Availability and Accessibility of Health Products in Hard-To-Reach Rural Areas The USAID Ghar Ghar Maa Swasthya Program strategy is designed to: a. Expand CRS’ proven track record of operational efficiency to produce an even more cost￾effective distribution system within the existing network of reproductive health product distributors. b. Increase revenue generation and cost recovery through evidence-based price increases, and gradually create a sustainable and self-reliant social marketing and social franchising program. 2 Kotler and Andreasen define social marketing as "differing from other areas of marketing only with respect to the objectives of the marketer and his or her organization. Social marketing seeks to influence social behaviors not to benefit the marketer, but to benefit the target audience and the general society." This technique has been used extensively in international health programs, especially for contraceptives and oral rehydration therapy (ORT). Nedra Kline Weinreich. Hands-On Social Marketing: A Step-by-Step Guide to Designing Change for Good (2nd Ed.). September 15, 2011. 3 c. Employ a marketing approach that improves the “reach” of products and services to the neediest living in the more remote areas of the country. This includes the deployment of new and non-traditional distributors and storage locations, and new retail outlets such as cabin restaurants, groceries, small shops, etc. d. Improve and monitor the quality of services within the network and ensure proper waste management. e. Increase the availability of and access to condoms and other reproductive health products and services to the most at-risk, and in the hot zones of Nepal where the rate of HIV/AIDS infection is rising. f. Bring about behavioral change in the at-risk and hard-to-reach populations, including the youth of Nepal. g. Create a better environment for the distribution of health products and services in Nepal. 4 EVALUATION METHODOLOGY Methodology The methodology of the evaluation is guided by the Statement of Work (see Annex II: SOW) which includes a wide range of methods and approaches for collecting and analyzing the information required to assess the evaluation objectives and answer evaluation questions. The evaluation team has reviewed the project reports and conducted key informant interviews with stakeholders from the government, FHI 360, and other social marketing organizations. A team planning meeting (TPM) was conducted in Kathmandu which was attended by USAID/Nepal officials and PHD Group evaluation team members (Annex III). Data collection methodologies were drafted by the evaluation team, which were discussed in the TPM and consequently approved by USAID/Nepal. The tools (Annex IV) developed were as follows: 1. Key Informant Interview Checklist –MoHP relevant Department, Divisions and Centers 5. Checklist for Observation of Service Sites 2. Interview Checklist for key CRS staff 6. Checklist for Observation of Commodity Outlets 3. Interview Checklist for key FHI 360 Partner staff 7. FGD Guideline - Females 4. Interview Checklist for other Social Marketing Organizations 8. FGD Guideline - Males The evaluation approach was primarily participatory. The evaluators also made site visits to 12 districts and conducted in-depth interviews with DPHOs, CRS field staff, and conducted six FGDs (four female FGDs and two male FGDs). In the Kathmandu Valley, in-depth interviews were conducted with senior, mid-level, and some field-level CRS staff members, FHI 360 staff members, and government officials (see Annex V for the list of key informants). The districts visited by the evaluation team included Ilam, Panchthar, Dhankuta, Terathum, Chitwan, Dhading, Kavre, Dolakah, Ramechhap, Gorkha, Banke, and Dandeldhura (see Map below). 5 The evaluation team collected data from multiple sources to ensure accuracy and that all relevant beneficiaries and stakeholders were considered. The evaluation involved both quantitative and qualitative methods and procedures, including document/desk reviews, a secondary data analysis, workshops, primary data collection through in-depth interviews, and focus group discussions with beneficiaries in field sites and field observations. Limitations The evaluation report is based mainly on the analysis of primarily qualitative data collected in 12 sample districts, a review of documents and reports made available by CRS and FHI 360, key informant interviews with different stakeholders, and field visits by evaluation team members. Key informants included high-level government officials of the Ministry of Health and Population, FHI 360, and CRS (high, middle, and front-line staff members), and other key national and international agencies. This evaluation is limited to the assessment of the effectiveness of the GGMS program, documentation of the GGMS approach in making a difference in the lives of people in hard-to-reach districts, documentation of best practices, informing and recommending for the remaining 27 months of the program period, and TA support. In addition, the findings from this evaluation will improve the programming direction for the remaining project period. EVALUATION FINDINGS The mid-term evaluation focuses on three main questions of the terms of reference (TOR) which are: A. How effective has the GGMS program partnership been in building CRS’s capacity to operate as a more independent, local social marketing organization? B. Given any barriers to capacity development, are any changes to the current TA structure or additional TA directions recommended? C. Are there any revisions that should be made in the expectations or demands of CRS? A. Effectiveness of GGMS Program Partnership in Capacity Building The effectiveness of GGMS program partnership in building CRS' capacity to operate as a more independent, local social marketing and social franchising organization has been analyzed under five headings: i. FHI 360‘s identification of CRS’ capacity needs; ii. Quality and appropriateness of the capacity-building TA offered; iii. CRS’ openness to receive and use the TA offered; iv. Building and operationalizing organizational systems and processes to develop CRS as an independent organization by the end of the project; v. Improving the quality and efficiency of CRS’ service delivery, including customer satisfaction. i. FHI 360's Identification of CRS’s Capacity Needs In Year One (2010/2011) FHI 360 organized an Institutional Development Framework (IDF) workshop to understand CRS better and identify its capacity needs. The capacity needs identified were the strengthening of monitoring and evaluation systems, human resource development, and supply-chain management, and the following were the key sub-areas: financial management, staff development, marketing management, procurement, data collection and processing, reporting mechanism, and the development of operational performance indicators as required by USAID. 7 Discussions held with the CRS top management team revealed that the areas of capacity-building identified in the IDF workshop were good overall; however the most important area the needed technical assistance was the strengthening of monitoring and evaluation systems - the area in which FHI 360 has not provided any technical assistance during the mid-term period. The CRS team explained having asked for TA in M&E from FHI 360 at the beginning of the project. However CRS could not understand the reason behind FHI 360’s failure to provide TA in M&E until now. FHI 360 in turn said that they now have a plan to provide TA in M&E this year, but they did not give clear reasons for the delay. FHI 360 also identified the need for conducting several evaluation studies and/or assessments for CRS regarding sustainability and the need for a subsequent strategic plan for CRS to achieve greater sustainability after 2015. The CRS team acknowledged several studies/assessments conducted by FHI 360 for the company, however, they were not the studies/assessment that CRS would have prioritized. Since CRS and AED (later transformed to FHI 360) submitted their proposals for the 2010/11-2014/15 phase separately, one party did not know what studies/assessments the other party had included in the proposal. Following the approval from USAID for project implementation, no research/study/assessment joint meeting was held to prioritize what studies and assessments were to be conducted. Nevertheless, FHI 360 commissioned five studies/assessments, namely, Mystery Client Survey, Knowledge, Attitudes and Practice (KAP) Survey, Assessment of the CRS’s Distribution System, Healthcare Waste Management (HCWM) Assessment among Sangini Providers, and Human Resource Management (HRM) Assessment. ii. Quality and Appropriateness of the Capacity-Building TA Offered Overall the quality of capacity building TA offered by FHI 360 was satisfactory based on the interviews with area managers in the field, top management, and mid and lower level CRS staff. FHI 360 provided Leadership and Management Development (LMD), effective product marketing, and effective demand creation trainings to CRS staff. The LMD course was offered to top and mid-level staff of CRS. Of the five phases of the LMD course, four phases have already been completed and one phase is still to be conducted. Overall, 90 staff from CRS, FHI 360, and USAID took part in the training in the first four phases (Figure 1). 8 Over one in three trainees were from CRS. Two trainings took place in 2012 and another two in 2013. At the end of each training, the participants were asked to evaluate the training in terms of the quality of presenters, workshop content, and logistics. FHI 360 repeatedly told the evaluation team that the training was rated highly by CRS participants based on the evaluation criteria, which included logistics, presentation, etc. The training required the CRS participants to present their progress report using the “challenge model”, develop an action plan, and detail monitoring progress towards achieving a measurable result. The training reports also included the challenges that they faced, leadership skills that they learned during the previous workshop, and whether they applied them in the workplace or elsewhere, and any evidence of positive change in the organization or in the individual. They had to present this progress report in the following phase of the training. The four phases of the LMD training workshops taught the Leading and Managing Result Model, which included various techniques such as the Challenge Model, SMART Criteria, Five Whys Technique, ORID Methodology, Tuckman's Model, Balancing Advocacy and Inquiry, Ladder of Inference, and Recognizing Sphere of Influence. During the workshop the participants practiced these techniques in house. The evaluation team members inquired about the use of the techniques learned in the workshops in the field, but the CRS team said that they have not yet practiced any of the techniques in the field. The LMD course was rated as satisfactory by the top CRS management team; they said that the LMD course was not included in the 2010 proposal submitted to USAID but afterwards FHI 360 included it in the work plan. One of the interviewees said "Leadership training is very bookish, given to only top level managers and directors." Several members of the top CRS management team said that the LMD course did not meet their expectations and therefore they suggested FHI 360 improve the module for the last phase and also expand it to the lower level staff. The evaluation team suggested that the expectations of CRS should be examined by FHI 360 and USAID; the evaluators think that the suggestion from CRS that 14 5 2 21 16 6 1 23 13 7 3 23 16 6 1 23 0 5 10 15 20 25 CRS FHI 360 USAID Total N u m b e r o f p a r t i c i p a n t s Name of participant organization Figure 1 Number of trainee participants in leadership and management development training in four phases in 2012 and 2013 First Second Third Fourth 9 the LMD training be given to lower level staff is not appropriate as it would not strengthen the future sustainability of CRS. Although different types of techniques were incorporated into the LMD training; the evaluation team, just as CRS’ management, thought that the content of the LMD training were theoretical. Participants had practiced and applied what they learned in the training in the training sessions only. In regards to strengthening supply-chain management, FHI 360 provided technical assistance with two types of trainings - effective product marketing (EPM) and effective demand creation (EDC). Discussions with the top management team, mid-level, and field level CRS staff determined that the staff found these trainings to be very effective. Based on the direct responses from staff at every level, the training helped them present their cases with confidence. Having years of practical field experience, training participants are experts in their area of work, however, they lacked technical and theoretical knowledge on marketing which the training greatly improved. This training has also strengthened and enhanced their quality of work. However, they could not mention to what extent they have improved because no relevant evaluation has been conducted to measure the effectiveness quantitatively. The training was relevant and high in quality, as the courses were fully participatory and contextually relevant for CRS staff. First, the trainings were given to a group of seven staff at the Headquarters by FHI 360 as Training the Trainers (TOT). Then, the master trainees (CRS staff) conducted the training for all 42 field level staff under the logistic support of FHI 360. In Year Two CRS installed Microsoft Dynamics Enterprise Resource Planning NAV ERP to integrate existing vertical systems (e.g. finance, logistics, human resources, monitoring and evaluation) into a single platform. With support from FHI 360, CRS succeeded in introducing the system, which is fully operational. All staff using the system have undergone extensive training to operate the system. The Finance Section of CRS is happy with the new software. FHI 360 will continue to provide CRS technical assistance relating to NAV in order to ensure its maximum use. The system is connected to CRS’ six area offices from where the sales and other data can be directly accessed. Individual interviews and group discussions with CRS’ top management and concerned field staff revealed that NAV ERP is effective and useful for CRS as it has sped up the data collection process and improved financial management. The five studies/assessments mentioned earlier that FHI 360 commissioned were relevant as per the discussions with five top management team members ,however, it also appeared as if they were not the most desired or CRS-specific studies/assessments. They said they were somewhat useful. For instance, the findings of the KAP study (USAID. 2012) on the pricing of contraceptive commodities and menstrual hygiene and the use of sanitary napkins were useful to CRS, while the rest of the issues explored were not that useful. Other issues such as contraception, maternal and child health practices, and media exposure could have been analyzed using the Nepal Demographic and Health Survey (NDHS) 2006 and NDHS 2011 data. Many of the KAP information on contraceptives did not differ much from NDHS 2011 data as shown in Table 1. Therefore, the study could have focused on sanitary napkins, MCH products and pricing and thereby reduced the size of questionnaire, which would have saved resources for use to meet competing needs of CRS. Table 1 Comparison of percent distribution of currently married women aged 15-49 years by current use of contraceptive methods in mountain and hill regions, 2011 Survey Any modern method Female Sterilization Male Sterilization Pill IUD Injections Implant Condom NDHS, 2011 41.0 6.5 11.5 3.9 1.4 10.8 1.9 4.7 10 (Mountains & hills) GGMS Baseline, 2011 45.1 6.6 9.3 5.4 2.1 16.8 1.9 3.1 Difference 4.1 0.1 (2.2) 1.5 0.7 6.0 0.0 (1.6) FHI 360 designed a quality assurance "Mystery Client Survey"(NDRI Consulting and Central Institute of Science & Technology, 2012 and USAID Nepal, 2012) and conducted it in 28 sampled districts (25 hard￾to-reach and three valley districts) using the sample frame of 49 hard-to-reach districts. But the sample frame included only 46 hard-to-reach districts in reality as the sample frame included three districts of Kathmandu valley which are not hard-to-reach districts. Tables were produced by comparing data from hard-to-reach sample areas and those from three valley districts. It was not necessary to compare hard￾to-reach district data with valley district data to establish baseline indicators for the GGMS project which focused on 49 hard-to-reach districts. Besides, of the total 119 Sangini outlets chosen for the study, 48 outlets were replaced, which represents 40% of the total sample; such a high replacement in sampling is questionable. Nevertheless the study findings have been used by CRS to assess service quality, IEC materials, stock of products, and physical facilities and their use. The "Assessment of the CRS distribution system" study conducted by MART India gave strategic recommendations to make CRS' distribution system effective and efficient in hard-to-reach districts. It has also proposed three models namely, Self Help Groups, Woman Entrepreneurship, and Youth-Based Community Groups. These models seem appropriate for Nepal, as they are founded in community networks, which can contribute greatly to CRS’ sustainability efforts. The start-up investment in any of these groups would not be high, but the returns can be substantial if properly managed. The top CRS management team appreciates this study, but they have not yet prioritized the recommendations given. FHI 360 commissioned a "Health Care Waste Management (HCWM) Assessment among Sangini Providers" (VaRG, 2013). The report came out in February 2013. This study is important because there are neither policies nor incentives in place for private health institutions to follow national or international guidelines to manage waste. The HCWM assessment has given seven recommendations, but they have not been prioritized. The CRS management team has not yet managed time to implement any of the study’s recommendations. On the other hand, CRS is happy with the financial and technical support provided by FHI 360 for health care waste management activities, which they accomplished by contracting HECAF-Nepal to dispose off expired USAID-donated commodities collected from CRS’ distribution channels. In order to improve human resources of CRS, FHI 360 commissioned Deloitte Touche and Tomatsu Pvt. Ltd. to assess CRS’ existing HRM system. The recommendations of the study were mainly of three types namely short term, policy, and donor-related. CRS has already implemented the short term recommendations, such as printing identity and visiting cards for all of its employees. Donor-related recommendations, such as the revision of salary scales for employees is in process, as USAID has to give formal approval for that. Policy-related recommendations have not yet been implemented. The findings are planned to be used to strengthen CRS HRM by developing a HR Manual, filling vacant positions, developing a talent acquisition system, compensating CRS Staff, and training and development of CRS staff. FHI 360 is also planning to bring Deloitte India to help CRS implement study recommendations. With the objective of expanding choices for clients, CRS, with TA from FHI 360, conducted research on a third condom brand. Research showed that D’zire was the most preferred brand name, and this need identification, according to CRS management, was very informative for market expansion. FHI 360 has provided technical assistance to CRS since the beginning of GGMS project. Their support, according to CRS staff, has brought improved their capacity in leadership and management development, 11 effective product marketing, and in designing and developing marketing plans. Disposing expired products in an environmentally friendly way and the installation of NAV ERP system are rated as some of the most remarkable achievements of FHI 360’s support. CRS would like FHI 360 to develop more effective business models and techniques to achieve greater cost recovery. They would like FHI 360 to conduct field-based studies to this end to acquire an in depth understanding of how to increase cost recovery. CRS also wants FHI 360’s assistance in translating research findings and recommendations into concrete plans of actions. One of CRS’ concerns was that FHI 360-supported research/studies/assessments took a long time. iii. CRS’ Openness to Receive and Use the TA Offered FHI 360 carried out activities identified under IDF. TA to strengthen human resource development and supply-chain management were openly received by CRS. They are satisfied with EPM and EDC trainings, as the staff could easily apply the knowledge and skills acquired from the training into practice. They commented that these two trainings, specifically, were eye-opening. CRS conducted improved social and behavioral change communication activities and community-level product promotion activities as a result of the training. TA offered by FHI 360 for identifying a third condom brand and scientific disposal of "expired" commodities was also adopted by CRS. CRS was also very satisfied with the ERP NAV program which is fully operational. Discussions with CRS staff suggest that CRS was not very open to receiving the TA on LMD offered by FHI 360. None of the interviewees said that they used the knowledge gained from the LMD training. This indicates that the LMD training was planned without much interaction between the two organizations. The evaluation team found that both parties had different stories. FHI 360 said that the training was effective, while CRS did not rate it highly. Although the LMD training initiative by FHI 360 was necessary, as high quality leadership is required in today's competitive market, the training could have been limited to fewer phases instead of the five required phases of FHI 360’s training. CRS did not seem to be that satisfied with the LMD training but will not openly talk about it. It was also noted that the training materials were not user-friendly; they were in English. Also, CRS asked for LMD support early on in the study, so that their increased knowledge would be able to positively impact the program. For FHI 360, the challenge was to manage the training early on time. Nevertheless, both parties meet to plan annual activities jointly. iv. Building and Operationalizing Organizational Systems and Processes to Develop CRS as an Independent Organization by the End of the Project FHI 360 is providing different types of TA to develop CRS as an independent organization. FHI 360 has been assisting CRS in identifying institutional development priorities. FHI 360 and CRS staff jointly self￾assessed CRS and identified operational needs across multiple dimensions. Based on this, the team decided to focus on human resources development, strengthening the monitoring and evaluation system, and strengthening of CRS’ supply chain management. FHI 360, as mentioned and discussed earlier, also initiated, largely on its own, a number of studies and assessments such as the mystery client study to improve the quality of service, the assessment of the CRS’ distribution system to increase access by people living in hard-to-reach districts to FP/MCH products, the healthcare waste management (HCWM) assessment, the assessment of human resource management (HRM) system, and so on. One of the main goals of GGMS is to assist CRS to advance towards greater sustainability, more specifically illustrated by the transition from USAID-donated commodities to self-financed and self￾procured commodities, the introduction of new products, and building of systems and capacities of employees. In Year One, an attempt was made to understand and measure strengths of, weaknesses of, 12 opportunities for, and threats to CRS by holding a workshop called IDF. Furthermore, the workshop aided participants in envisioning CRS not only from a financial perspective, but also from other aspects – mission and vision, staff development, marketing management, procurement, and human resources. In Year One, a Data Quality Assessment workshop was also held in which a review was conducted on the CRS monitoring and evaluation system, data collection and processing, and reporting mechanisms. Participants also discussed USAID's operational performance indicators related to the GGMS project. In Year One, GGMS classified its products into three categories, viz., subsidized [(Nilocon White (Pill), Dhaal Deluxe (Condom), Sunaulo Gulaf (Pill), IUD, Panther Premium (Condom), Jadelle, Sangini (Injectable), Matri Surakchya Chakki (PPH)], commodity cost recovery3 [(CURe (Male STI treatment kit), Sutkeri Samagri (Clean delivery kit), Virex (Packet: Chlorine disinfection powder) and Piyush (Point￾of-use water chlorination)], and full cost-recovery4[eCON and Nava Jeevan (ORS)]at that time. Under its sustainability plan, CRS has projected that for five products it will achieve commodity cost recovery and another five products full cost recovery (Table 2). Results, however, show after 33 months of project implementation that CRS could yield only five commodity cost-recovery products and one full cost-recovery product. From the field visit and from the CRS central office, it was also found that Nava Jeevan ORS was not available - the reason being that CRS terminated its contract with the supplier on the grounds of unsatisfactory product quality and inability to supply ORS on time as per the contract. This happened in the third quarter of 2012, and since then Nava Jeevan ORS has been discontinued. Table 2 CRS brands by measure of sustainability, 2014/2015 Subsidized brands Commodity cost recovery brands Full cost-recovery brands Dhaal Deluxe IUD Third brand of male condom* Sangini CURe eCON Sunaulo Gulaf Sutkeri Samagri Sanitary napkin brand* Jadelle Piyush Nava Jeevan Panther Premium Nilocon white * New products In August 2011, CRS outsourced a research activity to understand consumer perceptions of designs of its third condom brand name and pack (e.g., shape, size, graphics, logo, fonts, and color) and investigate reservations/apprehensions related to brand name/packs. Consumers' willingness to pay for the new brand was also tested. D’zire brand was the most preferred, and the consumers expressed that they were willing to pay as much as Rs. 35 per pack. CRS is also planning to market sanitary napkin pads, which are in high demand. During the first half of Year Three, CRS, with support from FHI 360, placed significant priority on sustainability issues. CRS entered into an agreement with Karex Industries SDN. BHD. Malaysia for the supply and delivery of five million pieces of dotted premium condoms and five million pieces of plain premium condoms. On January 25, 2013, CRS received its first shipment of 900,000 cycles of oral contraceptive pills from HospitecPharma as a replacement for USAID-donated oral contraceptive pills. This represents a significant milestone in working towards greater sustainability. 3Commodity Cost Recovery is defined as the cost which includes purchase cost of the commodity + cost of packaging material + cost of labor used in the packaging. For example, if a unit of condom costs CRS Rs 2 to purchase, and 50 paisa for packaging material and 10 paisa for the labor to package, CRS needs to recover Rs. 2+0.5+0.1=2.60 under commodity cost recovery. 4Full cost recovery is defined as the cost of commodity +repacking material cost+ repacking labor cost+ overhead cost allocated to the product. For example, if for a piece of condom Rs 1is charged as overhead cost, CRS has to recover Rs. 2.60+1=3.60 to recover the full cost. 13 Additionally, with the financial and technical support of FHI 360, CRS worked with Deloitte Touche Tohmatsu India Private Limited on assessing CRS’ human resource management (HRM) system. The findings and recommendations of different studies and assessments like this one have recently become available. FHI 360 and CRS in the remaining period of the project should translate those findings and recommendations into concrete actions to move towards achieving the goal of becoming a sustainable, independent organization. From discussions with different CRS and FHI 360 staff, it was found that the delay of project activities by FHI 360 in the first year slowed down the program’s progress by nearly one year. Nevertheless, the GGMS intermediate results for the first 33 months are encouraging. According to CRS GGMS project reports, the 15-month results of five-spacing methods (injectable popularly known as Sangini, two brands of condoms, two brands of pills, IUD, and Jadelle) and eCON distributed and marketed by CRS surpassed the target of 407,890 CYP with a CYP of 410,244 in 2010/011. In the following year the CYP was 368,917 compared with the target of 351,685 CYP, and in the first six months, i.e., August 2012 - January 2013 it was 179, 820 again surpassing the target of 176,937. Although the CYP figures look smaller year after year the achievements exceeded the anticipated goal every year (Figure 2). Overall, the results are positive as the CYP achieved met and exceeded the projected goals. However, the pace of the increase of CYP in the first 33 months (29,060 CYP per month) is not sufficient to achieve the 5-year projection of 1,811,875 CYP. In order to achieve 1.8 million CYP by the end of the project period, CRS will have to strive to achieve CYP 31,589 per month from February 2013 onwards. The sale of two brands of condoms - Dhaal Deluxe and Panther Premium –meets expectations, as shown by figures from the last 33 months, which show that sales were above 100 percent of the projected figures. Among the two, Panther condoms faired much better than Dhaal Deluxe (Table 3). The sale of two brands of pills - Sunaulo Gulaf and Nilocon White - were above 90 percent when compared with the projected overall figures; Nilocon White was selling well in the first and second years, but, in the first six months of year three, it only met 77 percent of its target sales. 100.6 104.9 410,244 101.6 368,917 179,820 27,350 30,743 29,970 - 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 110.0 2010/011 2011/12 2012/13 C Y P P e r c e n t Year Figure 2 Annual CYP results based on five spacing methods distributed by CRS, 2010/011–2011/013 CYP achievement % CYP achievement n Per month CYP 14 Table 3 Percent achieved against the projected figures, May 2010 - January 2013 Commodity 1st year 2nd year 1st half, 3rd year 33 months Dhaal condom 100.1 101.0 99.7 100.3 Panther condom 115.0 120.1 118.8 118.0 Sunaulo Gulaf pills 95.9 89.4 88.3 91.2 Nilocon White pills 100.1 103.5 76.8 93.4 Sangini injectable 100.6 107.4 116.0 108.0 IUD 71.0 97.0 28.4 65.5 Jadelle 53.8 219.8 91.3 121.6 eCon pills 106.9 110.1 113.0 110.0 CDK packet 94.6 18.7 48.3 53.9 Virex packet 51.9 18.1 0.3 23.4 Nava Jeevan (Sachet) 143.1 79.1 0.3 74.2 Piyush (Bottle) 97.0 39.8 105 80.6 CURe 19.2 12.8 63.8 31.9 The sale of Sangini (Vial) has been good in all years. IUCD sales have declined over the years, while the sale of Jadelle has increased. The product, eCon, has become increasingly popular over the years, and in the first half of the third year, eCON sales were much higher than the target. The sale of CDK packets is declining and it is expected to decline as institutional delivery is increasing. The sale of Virex packets and Piyush bottles is not doing well and Misoprostole is no longer sold. Nava Jeevan Sachets sold well in the first year but later declined, although it is in demand in the country. The sale of CURe is also very low (Table 3). The GGMS program documents the partnership dynamics that consist of FHI 360 and CRS- the USAID project partners - and CRS and NFCC as core partners. CRS' other donor partner is Entwicklungsbank (KfW). The partnerships combine public sector (Nepal government), USAID, NGOs, and the private sector. Despite good partnership dynamics and achievements in terms of CYP, the total income generated from the sale of commodities has declined during the mid term of the program. In year one, CRS generated Rs. 104,292,727/- of which 43 percent was comprised of income from the sale of USAID-supported products, 50 percent from KfW supported products, and eight percent from CRS products (Figure 3). In the following year, the total income dropped to Rs. 88,118,576/- and the share of USAID supported products was 53 percent, that of KfW was 38 percent, and CRS’ share was nine percent. During the first six months of year three, the total income dropped to Rs. 36,255,013/- and the share of CRS products dropped even further to two percent (Figure 3).This should be taken seriously by both CRS and FHI 360. From different reports it was found that the shortage of Nava Jeevan in particular and the low sales of CDK, CURe kits, IUD, and Nilocon White pills was responsible for this. The decline in revenue in Year Three is attributable to a lengthy procurement procedure. Nevertheless, CRS, with TA from FHI 360, has managed to procure Nava Jeevan, D'Zire condoms, and OCPs, and they should be delivered on time in 2013. 15 For CRS independence does not mean that it should not secure donor funding for its operation; rather it means more independence from USAID support because it has depended on USAID funding for a long time. Therefore it should develop its capacity to attract donor funding from sources other than USAID. FHI 360 should provide technical support to develop CRS in this way too. However, the best option for reducing dependence on USAID would be to increase the sale of its high quality FP and MCH products and generate sufficient revenue to run its operations independently. Given CRS' reputation and credibility this is not impossible. v. Improving the Quality and Efficiency of CRS’s Service Delivery, including Customer Satisfaction. CRS service delivery covers all 75 districts of the country. CRS brands of contraceptives and MCH products are available in major pharmacies or TOs of every district. CRS has expanded its Sangini network in 46 hard-to-reach districts after 2010, and, before that, it had its network in Dhading, Doti, and Pyuthan districts (Karki, et al. 2010). The opening of new Sangini outlets in 49 hard-to reach districts providing Sangini service is likely to be achieved as planned by the end of project period. Until the midterm NFCC expanded its Sangini outlets to reach 3,578, and it should meet the target without any difficulty given the remaining time (Figure 4). 42.8 53.0 60.2 49.5 38.1 37.5 7.7 8.9 2.3  ‐ 10.0 20.0 30.0 40.0 50.0 60.0 70.0 1st year: May 1, 2010 ‐ July 31, 2011 2nd year: Aug 1, 2011 ‐ July 31, 2012 3rd year: Aug 1, 2012 ‐ Jan 31, 2013 Percent Figure 3 Percentage share of revenue generated from sales of commodities of three sources, May 2010 - January 2013 Sales of USAID supported products KfW supported products CRS products 16 The NFCC's main task is ensuring FP compliance and quality service is provided by Sangini outlets to their clients. NFCC entered into a contract with CRS in 2010 for three years, which will end in September 2013. Up until January 31, 2013, CRS trained 2,601 Sangini service providers in 49 mountain and hill districts (see Box). In addition, NFCC also conducts Technical Support Visits (TSV) to Sangini outlets to ensure quality service. In order to increase the access of remote district populations to maternal, child health, and contraceptive commodities in 49 districts, CRS has been gradually opening NTOs through which condoms and Navajeevan ORS packets are sold. A Rural Field Representative (RFR) is assigned two or more districts where he/she is responsible for promoting, set up outlets, and marketing condoms and ORS packets in remote districts. Preceding July 2013, CRS set up 3,422 NTOs (791 in the Mountain region and 2,631 in the mid hill region, see adjoining box).Throughout this areas, there are only ten RFRs. Each RFR motivates local shopkeepers to keep condoms and Nava Jeevan (ORS) to sell in the region. He /she also conducts orientation classes on HIV and AIDS to students of Grades 9 and 10. He/she has to resupply and promote the sale of the commodities kept in the NTO. Given the value of RFR's work CRS should recruit more RFRs to cover all hard-to-reach districts as in the GGMS program. Undoubtedly CRS has the capacity to do this. Interviews with the NTOs and TOs and responses received from the FGD participants suggest that CRS products are of good quality: People prefer CRS products because of the brand image and the relatively low price point. The NTOs are willing to keep CRS products even though the margin provided by CRS is comparatively low to other brands of contraceptives. The FGD responses also suggest that people prefer CRS products and are satisfied using them. 1,051 1,739 788 327 655 600 1,150 850 850 1,110 - 500 1,000 1,500 2,000 2010/011 2011/012 2012/013 (1st 6 months) 2012/013 (2nd 6 months) 2013/014 New Sangini outlets Figure 4 Number of new Sangini outlets opened in 49 hard-to￾reach districts May 1, 2010 - Jan 31, 2013 and likely to open by mid 2014 Achievement Projection Number of Sangini service providers in 49 mountain and hill districts Year Projected Achievement %Achieved 2010/011 240 502 209 2011/012 900 987 110 2012/013 900 1,112 124 2010- 2013 2,040 2,601 128 Expansion of NTOs in 49 GGMS districts: May, 2010 to July, 2013 Targets Achievements % achieved Mountain region 616 791 128.4 Hill region 2,000 2,631 131.6 Total 49 districts 2,616 3,422 130.8 Source: Annex V 17 In hard-to-reach areas, NTO is not financially cost effective but this meets the objective of USAID funding because the intent of their support is to promote family planning in remote areas of the country; for, the people in those communities have the right to know about family planning methods. An intended impact of the project is that after it is finished; people in remote communities will be able to talk about contraceptives openly. However, it was found that even after opening an NTO in a remote hill district, the NTO shopkeeper was hesitant to talk about contraceptives. When asked by an evaluation team member why condoms from CRS were kept under the table an NTO owner said "children play with condoms if they (condoms) are kept at visible places." In another NTO, condoms were kept in the corner of the shop where they were not visible easily. When asked why they were not put on the front table, the NTO owner said, "This is early in the morning; I was going to display them on the front table in the mid day." Interviews with several NTO owners reveal that they are interested in building their capacity to better sell the commodities they obtain from CRS. One NTO (paan shopkeeper) said, "I have never received any training or benefits from CRS. It would be nice to get training on CRS products." The evaluation team has the impression from field visits that most NTOs are interested in getting some kind of training so that they can market CRS products better. The training components for NTOs can include marketing, record keeping, and orientation on CRS and its commodities. The evaluation team members visited 12 districts to observe the service outlets and found that the quality of CRS services was high because the sites were clean, there was running water, and separate rooms for client counseling, etc. The service delivery mechanism was also efficient because these outlets or pharmacies were open for a long time every day, and the NTOs were open as long as local shops were open. This mechanism satisfied customers even if they had to pay for the service and commodities because they say they could access CRS services at any time. A male FGD participant in Dolakha said, "People buy from CRS outlet even though it is not free because government health facility is open only from 10 am to 2 pm while CRS outlet is open for a longer time." B. Given any barriers to capacity development are any changes to the current TA structure or additional TA directions recommended? Feedback from discussions and interviews with CRS staff suggest the TA to focus on improving and enhancing the financial management system and developing better business models and techniques for moving towards cost recovery. Also technical assistance should be provided on how to translate research findings and recommendations into action plans. The evaluation team suggests that for CRS sustainability, FHI 360 TA should be geared towards program sustainability, which requires careful planning and an investment in both human and financial resources. The programs that CRS implements must be cost effective, and, for this, cost analysis should be undertaken. CRS can capitalize on the expertise of FHI 360 in this respect. FHI 360 must come up with a business plan and policies and guidelines that are tailored to CRS to develop a robust cost recovery program. Additionally, FHI 360 plans and areas of targeted support should be aligning with CRS’ work plan. Furthermore, there should be timely dissemination and sharing of study results to enable CRS to prioritize the resulting study recommendations and act on them. TA should make plans keeping in mind the sustainability of CRS with a greater emphasis on improving their business’ capacity for building their market in the hard-to-reach areas. The TA should also emphasize training that enhances CRS’ managerial and functional capacity. 18 CRS prefers training facilitators who have practical experience in relevant areas. CRS would be happy to receive training from local experts with the desired practical knowledge. However, the LMD training was provided by an external expert, who, according to CRS trainees, created a language barrier between the trainer and trainees. C. Are there any revisions that should be made to the expectations or demands of CRS? CRS expects that USAID becomes flexible enough to meet its demands. CRS is not allowed to use the big reserve funds lying in Banks. If USAID allows CRS to use the reserve fund they will be able to move forward on the sustainability path. Since the inception phase of the project, human resource planning was involved. The current organogram shows a total of 66 human resources working for CRS. There are, however, several senior and mid-level staff positions vacant, such as Deputy Managing Director, Finance Director, Finance Manager, Marketing Manager, Franchising Manager, Marketing Officer, and Medical Representative. Of the six Area Mangers there is no Area Manager in Dhangadi, Kailali in the Far-Western Region. In addition, a number of other positions are vacant. Discussion of management of CRS was held with top, mid, and lower level CRS staff. Except for a few, most top level managers said that the organization is functioning well. Organizational rules and regulations are all in place. Discussion with mid-level staff revealed that the organization is moving slowly, decision making is delayed, and, as a result, activities are not completed as planned. The case in point is the shortage of Nava Jeevan. CRS should either remove the DMD position or take prompt action to fill the vacant position. A DMD would actually help CRS as decision-making power can be shared between the MD and DMD and other top division chiefs. Another issue mentioned by the mid￾level staff was low morale because salaries were low compared to other similar organizations in the country. They also mentioned that the salaries of FHI 360 Nepali staff are several times higher than those of CRS staff even though the source of funding is the same. CRS also would like to have Deloitte’s recommendations implemented as soon as possible. One of the main recommendations is the raising of salaries, which will encourage and motivate them to perform better. The low salary is one of the main reasons that several positions have been vacant for a long time. Discussion with front line and some field staff revealed that the organization is moving slowly; communication between lower level and higher level staff is virtually non-existent. They say CRS is a top-down organization: the senior staff do not know the realities of the fields. Of the total staff only 17% are female. The lower level staff say the salary level is too low. The field staff suggest that in every hard￾to-reach district there should be at least one RFR, and he/she should be provided with some budget to open a small office in the district. This will enable coordination with the district government officials and a concrete connection with community members and district specific MCH, and contraceptive education and information activities can be carried out in these location. Meetings with DHOs/DPHOs in the ten districts (two DHOs/DPHOs of Kavre and Ramechhap could not be interviewed because of their unavailability at the time of the visit)by evaluation team members revealed that except for Chitwan and Banke, coordination between CRS and the government does not exist at all in those ten districts, which are hard-to-reach districts. The DHOs/DPHOs, however, said that they would very much like to cooperate with CRS. The CRS Area Managers explained that because CRS does not have offices in the hard-to-reach districts, they have not been in contact with concerned DHOs/DPHOs. The rural field representatives (RFRs) believe that they are overburdened with work. In hard-to-reach 19 districts, RFRs have to open at least one NTO in the village they visit to sell condoms. They say it is very difficult to convince a shopkeeper to put condoms on shop shelves which can be easily seen by customers. Shopkeepers are still reluctant to talk about condoms openly: for them it is an embarrassing matter. This was found in several districts which were visited by evaluation team members. Nevertheless, 3,422 NTOs have already been opened in 49 districts (see Annex VI). In hard-to-reach districts there is an urgent need to carry out awareness programs on family planning methods. The RFRs are assigned more than two districts each and to date only ten RFRs are covering 24 districts. Considering the difficult terrain and that a lot has to be accomplished during the remaining period of the project, CRS and FHI 360 should employ at least one RFR in every hard-to-reach district. FHI 360 should help CRS to prepare an action plan to ensure this happens. In order to be able to compare the progress made in the hard-to-reach districts versus the 26 accessible districts, efforts must be made to collect information by district. CRS covers all 75 districts of Nepal, and its data are generated without separating between the hard-to-reach 49 districts and the remaining 26 districts. CRS should prioritize the importance of arranging the data by district for the hard-to-reach districts and the more accessible districts, which will allow them to evaluate how much progress has been made more specifically so that future activities can be planned accordingly. The current data system does not allow for generating district-level data as the data are lumped together for all the 75 districts. CRS only keeps records of its sales that have been supplied through the center to the distributors in different regions and districts. Therefore, it does not show sales of products that have been made by the distributor. A particular distributor can distribute CRS products to retailers of different regions and districts; however, those sales are not recorded by district and thus CRS does not produce CYP by district. Both CRS and FHI 360 informed evaluation team members that it is not possible to keep records of sales by district because it can be expensive. Also FHI 360 mentioned that it was not in the program design from the start of the project, and, with limited time, it is not possible to incorporate it now. They added that since the profit margin of CRS products is low, the distributors would not be interested in keeping a record of sales to different regions and districts. However CRS does keep records of revenue generated from these sales from each of the distributors and retailers. In order to generate data by district, CRS should also keep a record of district sales while collecting the payments for the products as these data would be valuable to CRS for different purposes: it will be in a position to produce disaggregated data by residence and also measure and compare progress made by district and identify gaps in sales between hard-to-reach districts and the remaining districts.. FHI 360 can provide additional TA to CRS in this respect. CRS field staffs and RFRs can carry out this task. A short training on reporting and recording to the field staffs would be enough for this task. Furthermore, more RFRs can also be employed because, as the current data show, in 49 hard-to-reach districts only 24 districts have been assigned with RFRs (see Annex VII), and 25 districts still need to have RFRs. Keeping records of such data will be an asset for the GGMS project. Discussions with top management suggest that the TA in M&E that is scheduled should include training on district-level data recording and interpretation so that after the program is phased out, CRS can retain its own developed M&E section. CONCLUSIONS The GGMS partnership has been good but there is still much room for improvement. FHI 360 TA has 20 identified a number of useful capacity needs of CRS and accordingly FHI 360 has carried out activities to strengthen CRS’ capacity. However, a few of the capacity needs identified are not that relevant to current sustainability issues facing CRS, which implies an inefficient use of resources. FHI efforts mostly concentrated on training, research, and assessments and CRS accepted them, but in retrospect CRS seem unsatisfied. CRS did not want to criticize FHI openly. All this suggests that CRS was not as open as it could have been to receiving the TA offered by FHI 360. However, the relevance of some of the TA was questioned, as some of it did not have immediate practical applications. FHI efforts were appreciated in helping CRS identify its weaknesses through the IDF workshop, but it did not address CRS' most crucial need -strengthening M&E. Nevertheless, CRS has greatly appreciated the NAV ERP system, the assistance with the disposal of expired products, and the EPM and EDC trainings. The evaluation of CRS current position shows that CRS seems confident in its ability to achieve sustainability by 2015 as it has built a good image over the past 35 years, has a presence in all 75 districts of the country, sells its commodities cheaper than other business entities, and it is planning to reduce product dependency from donors by introducing commercial products to raise profits and cross￾subsidize from the profit gained. However, CRS still needs to strengthen its current position to stand out in the competitive market for which it requires TA. CRS appears weak at this point as it has not been able to make a case for specific kind of FHI 360 assistance. CRS needs to engage itself more and think critically to identify specific areas they need TA in order to support growth. CRS should be made aware that FHI 360 is there only for them, and they have the authority to choose what kind of support they want from FHI 360. They should have the ability to identify and ask for specific assistance clearly. CRS has been receiving the TA offered by FHI 360 but it does not necessarily mean CRS is fully satisfied with the TA offered. It appears that CRS has not fully realized what its position or authority is, where it stands regarding determining the type of TA FHI 360 will provide. The top management team indicated that instead of another organization giving them technical assistance, CRS would prefer to have its own staff carry out technical/highly professional work. The evaluation team thinks that CRS’ decision to give LMD training to lower level staff is not appropriate, as it will not add significantly to the sustainability challenge CRS is facing. The LMD training initiative by FHI 360 was good given that strong leadership is required in today's competitive market, however, the training could have been limited to a few phases instead of the current five phases as proposed by FHI 360. In addition, instead of same people participating in four or five-day training courses repeatedly for five times, it would be more productive for selected employees to participate in a two to three-week long training in reputed institutions possibly abroad. Similarly, one of the evaluation factors should have been the relevancy of the training to the issue of building CRS’ sustainability: How useful the training is for CRS staff based on which they could translate the training knowledge to practical usages. Since the evaluation criteria did not include the relevancy of the training to CRS, the measurement of satisfaction of CRS on this training was not significant. The TA should strongly consider the time frame CRS has: The one year delay in the beginning of the program would not be excused now. There has been lot of work done by FHI 360 but it is limited to piles of files and reports. The problem of FHI 360 lies in starting a new activity (second, third, etc rounds of research, assessment, and study) without translating research findings into actions from the first round studies/assessments. If action plans were drawn and program activities for CRS were implemented accordingly, the outcomes would have been tremendous. The organizational structure of CRS is incomplete and this shortcoming has already been identified by the study conducted by Deloitte. CRS wants to find a solution on this as quickly as possible. Salary was another big concern for CRS. Considering today's inflation, the evaluation team felt the low salaries 21 must be addressed by USAID so that salary scales may be revised. Finally prompt practical implementation of the recommended list of actions by different firms assigned by FHI 360 would surely augment CRS’ capacity as it moves towards self-sustainability. Due to the delay of technical assistance from FHI 360 in the first year, CRS activities have fallen behind schedule. Studies, assessments, and trainings conducted as part of the technical assistance provided by FHI 360 were all delayed, and this affected the activities planned up to the mid-term evaluation. Nevertheless, in line with the PPP objectives, GGMS has involved ‘for-profit’ and ‘not-for-profit’ organizations together in the health sector. The project made strong efforts to facilitate collaborations and networking between public and private sectors. However, in the 49 hard-to-reach districts, DHOs/DPHOs did not know about CRS activities because GGMS had no mechanism in place for coordination with district health offices; CRS does not have offices in the hard-to-reach districts, although CRS activities supplement government programs. Shopkeepers are still embarrassed to talk about condoms openly. This was found in several hard-to￾reach districts visited by evaluation team members. Given the difficult terrain and that a lot has to be accomplished during the remaining project period, RFRs have a daunting task to perform. Results of the mid-term period show good achievements; however, some of the data are not detailed enough to show breakdowns by hard-to-reach districts and the rest of the country. It may be that the achievements in the data are obscuring the weaknesses in the hard-to-reach districts. Decision making is slow at CRS. The top management team is not proactive: its decision making is not fast enough nor does it communicate efficiently with the rest of the staff. During the first half of Year Three, CRS was successful in purchasing condoms, including a new brand, and contraceptive pills on its own from a non-USAID source, which is a significant milestone of progress towards greater sustainability. Sustainability of CRS, however, does not just mean making profit by selling its products in every part of the country. CRS should also develop its internal competency on Sangini -Sangini is one of the main commodities marketed by CRS, and its service quality control is maintained by NFCC. Instead of depending on an external agency for a single commodity like Sangini, it would be wise for CRS to gradually develop technical competency around Sangini service. In moving forward, there are two things that will help CRS complete its work in the second half of the project. FHI 360 conducted various studies and assessments in support of CRS. These were published in the first quarter of 2013. The sooner CRS can analyze them, the sooner the resulting recommendations can be implemented. Top management and mid-level staff have had several rounds of capacity building trainings by now, which should also help CRS to speed up program activities. Once the results of FHI 360’s studies and the skills acquired in the training are synthesized, CRS should be in a solid position to move forward to achieve its goals faster in the remaining period of the program. RECOMMENDATIONS 22 Based on the analysis of information from project reports, research and assessment studies, key informant interviews, and field visits, the following sets of recommendations are made to improve the second half of the GGMS program: Recommendations for FHI 360: 1. FHI 360 has provided TA in human resource development and supply-chain management but not in monitoring and evaluation system (M&E), and now they must prioritize M&E TA. 2. FHI 360 and CRS should jointly work to prioritize the recommendations given in the five studies/assessments for which FHI 360 provided TA. CRS further suggests that FHI should share a summary of big-picture findings as early as possible so that CRS program activities can be strengthened in a timely fashion. 3. The Leadership and Management Development (LMD) course did not meet CRS' expectations, and, therefore, it is suggested that FHI 360 improves last phase module. In addition, LMD training should impart skills to senior CRS staff regarding developing a sustainable business plan. 4. The next round of the KAP study should focus on sanitary napkins, MCH products, and pricing￾related information and leave out the other data collection. This will shorten and strengthen the questionnaire to be more relevant and easy to complete. 5. FHI 360 designed a quality assurance Sangini "Mystery Client Survey" and collected information from 119 sites. However, 40% of sample sites were replaced, and it is recommended that in the next round the replacement of sample sites be minimized. 6. FHI 360 should speed up its activities, and before starting a new activity, it should assist CRS in translating results of studies/assessment of specific activities into action. 7. FHI 360 should provide additional TA to CRS to train CRS field staffs including RFRs on reporting and recording sales data in hard-to-reach areas and in the rest of the country. 8. Training focusing on marketing skills, record keeping, and an orientation on CRS and its commodities should be provided to the NTOs so that they can better market CRS products. Recommendations for CRS 1. The "Assessment of the CRS distribution system" conducted by MART India proposed models, namely, Self Help Groups, Woman Entrepreneurship, and Youth-Based Community Groups, which can greatly strengthen CRS’ sustainability as these groups are all part of strong, existing community networks. The CRS management team appreciates this study but they have not yet made an action plan based on recommendations given. It is recommended that CRS acts upon these recommendations soon. 2. In order to address the problem of slow decision making at CRS, prompt decisions have to be made to fill the senior positions, including Deputy Managing Director, which has been vacant for many years. Within CRS there should be a delegation of power to different divisions for speedy decision making. Therefore, the management structure needs to be urgently improved. The sex ratio of staff should also be more balanced and inclusive. 23 3. One of the main objectives of the GGMS project is to increase people’s access to contraceptives and MCH products especially in hard-to-reach areas. Current PMP do not tell whether project implementation is progressing towards achieving this goal. Therefore CRS should prepare PMP by distinguishing between hard-to-reach areas and the rest of the country. 4. RFR should mobilize local Mothers' Groups and FCHVs and conduct Social Behavioral Change Communication activities locally. The Center should delegate its authority to the area manager and RFR to conduct SBCC activities. If necessary, local media and local languages should be used to educate the local people on contraception, HIV/AIDS, maternal and child health. 5. RFR should be given an allowance to cover a room rent in one district. He/she can operate this room as an office. The RFR should also create affiliations with the DHO/DPHO and the district health office. 6. GGMS should strengthen the supply chain of its commodities. Currently commodities like CDK, IUD, Nava Jeevan, and CURe kit do not exist in sufficient quantities. CRS should prepare a logistics plan in such a way that the supply of its commodities is not interrupted. 7. CRS should gradually build its capacity to provide quality FP service including Sangini for which it had been depending on NFCC the last three years. Recommendations for USAID 1. The Deloitte-conducted HRM study made several recommendations, and one is related to salary scales: The CRS salary scales are too low compared to many similar organizations working in Kathmandu. The evaluation team found the same situation. The evaluation team recommends that this be addressed and improved by USAID. 2. GGMS' goal is to increase access for the people of 49 hard-to-reach districts to contraceptive commodities and MCH products, but it has not deployed sufficient human resources to carry out various tasks. Policies should be made to assign at least one RFR to each district. The donor fund would be better used this way. FHI 360 can assist CRS in this respect. 3. The evaluation team sensed that CRS was not open to constructively critique the TA provided by FHI 360. Therefore CRS should be empowered to critically comment on the quality of FHI 360’s technical assistance. BIBLIOGRAPHY 24 AED. 2011. "1st Semi-annual report, August 2, 2010-February 28, 2011". Contraceptive Retail Sales Private Ltd.no date. "Ghar Ghar Maa Swasthya, First Annual Report.May 1, 2010- July31, 2011". Submitted to USAID/Nepal. Kathmandu. Contraceptive Retail Sales Private Ltd.no date. "Ghar Ghar Maa Swasthya, Second Annual Report.August 2011- July 2012". Submitted to USAID/Nepal. Kathmandu. CRTI, NDRI Consulting Pvt.Ltd. 2012. "Measuring Quality of Service Delivery through Sangini Franchising Outlets". AED. Gopal, Anamika Galli, Baluwatar, Kathmandu, Nepal. Final Report. May Deloitte. 2013. "HRM Assessment of Nepal CRS Company". Final Report. February. FHI Development 360 LLC. 2011. "1st Annual report, August 2, 2010-July 31, 2011". USAID Nepal. August. FHI 360. 2012. "Annual report, August 1, 2011-July 31, 2012". USAID Nepal. FHI 360. 2012. "Semi-annual report, August 1, 2011-July 31, 2012". USAID Nepal. FHI 360. 2013. "Semi-annual report, August 1, 2012-July 31, 2013". USAID Nepal. Karki, Yagya B., Adhikary, Khagendra and Pande, Ashok D. 2010. Evaluation of USAID/Nepal's Key social Marketing and Franchising Proejct: AIDS, Reproductive Health, and Child Survival (N￾MARC). Submitted to Office of Health and Family Planning, USAID, US Emabssy, Kathamndu. March MART. 2013. "Assessment of Distribution System and Costs of Nepal CRS Company’s Health Commodities". Final Report. November, India. Ministry of Health and Population (MoHP). 2009. “Let’s Know Our Epidemic”. National Centre for AIDS and STD Control. Government of Nepal, Teku, Kathmandu. _____________________. 2008. “Annual Report2006/07”.Department of Health Services. Government of Nepal, Kathmandu, Nepal. _____________________. 2007. “Nepal Demographic and Health Survey 2006”. New ERA and Macro International Inc. Ministry of Health (MoH). 2002. “Nepal Demographic and Health Survey 2001”. Kathmandu, Nepal and Calverton, New ERA and ORC Macro, DHS+, Maryland, USA. _____________________. 1993. “Nepal Fertility, Family Planning and Health Status Survey, 1991”. Kathmandu, Nepal. _____________________. 1987. “Nepal Fertility and Family Planning Survey Report 1986, First Report.” Family Planning and Maternal Child Health Division, Planning Research and Evaluation Section, Kathmandu, Nepal. _____________________. 1983. “Nepal Contraceptive Prevalence Survey Report, 1981”. Nepal FP/MCH Project, HMG and Westinghouse Health Systems. _____________________. 1977. “Nepal Fertility Survey 1976. First Report”. Nepal Family Planning and Maternal and Child Health Project, World Fertility Survey. Kathmandu. NDRI Consulting and Central Institute of Science & Technology. 2012. ‘Measuring Quality of Service Delivery through Sangini Franchising Outlets’. FINAL REPORT. Submitted to Academy for Education Development (AED), Kathmandu. Nepal CRS Company. 2013. "Ghar Ghar Maa Swasthya, Intermediate Results, May 2010- January 31, 2013". Nepal CRS Company Pvt. Ltd. Nepal CRS Company, FHI 360. 2012. "Ghar Ghar Maa Swasthya Workplan August 1,2011- July 31,2012". USAID Nepal. 25 Nepal CRS Company, FHI 360. 2012. "Ghar Ghar Maa Swasthya Workplan August 1,2012- July 31,2013". USAID Nepal. Nepal CRS Company, AED. 2011. "Joint 1st Year Workplan". USAID Nepal. Nedra Kline Weinreich. Hands-On Social Marketing: A Step-by-Step Guide to Designing Change for Good (2nd Ed.). September 15, 2011 Pandey, A., Baniya Chetri, S. 2012. "Effective Product Marketing Strategy Training". Training Report. USAID Nepal. 2012. "Mystery Client Survey among Private Providers in the Sangini Social Franchising Network". Kathmandu. USAID Nepal. 2012. "Assessment of Distribution System and Costs of Nepal CRS Company’s Health Commodities". Key Workarounds. November. USAID Nepal. 2013. "Health and Family Planning, Ghar Ghar Ma Swasthya (Healthy Homes) Pogram". January USAID Nepal. "Success Story: Providing Quality Family Planning Services in Remote Areas" USAID Nepal. "Snapshot: Increasing Contraceptive availability in Nepal's mountain Communities" VaRG. 2013. "Health Care Waste Management Assessment at Sangini Outlets". FHI 360/GGMS. Gopal Bhawan, Anamika Galli, Baluwatar, Kathmandu, Nepal. June. VaRG. 2012. "Ghar Ghar Maa Swasthya: Family Planning (FP), Reproductive Health (RH), Maternal and Child Health (MCH), Disinfection of Drinking Water, and Menstrual Hygiene Baseline Knowledge, Attitudes and Practices (KAP) Survey in 49 Hill and Mountain Districts". FHI 360. January. 26 ANNEX I LIST OF HARD-TO-REACH DISTRICTS OF NEPAL COVERED BY GGMS PROJECT Development region SN District Development region SN District Eastern Mountain Mid-western Mountain 1 Taplejung 31 Dolpa 2 Sankhuwasabha 32 Jumla 3 Solukhumbu 33 Kalikot Eastern Hill 34 Mugu 4 Panchthar 35 Humla 5 Illam Mid-western Hill 6 Dhankuta 36 Pyuthan 7 Terhathum 37 Rolpa 8 Bhojpur 38 Rukum 9 Okhaldhunga 39 Salyan 10 Khotang 40 Surkhet Central Mountain 41 Dailekh 11 Dolakha 42 Jajarkot 12 Sindhupalchok Far-western Hill 13 Rasuwa 43 Achham Central Hill 44 Doti 14 Sindhuli 45 Dadeldhura 15 Ramechhap 46 Baitadi 16 Kavrepalanchok 17 Nuwakot Far-western Mountain 18 Dhading 47 Bajura Western Mountain 48 Bajhang 19 Manang 49 Darchula 20 Mustang Western Hill 21 Gorkha 22 Lamjung 23 Tanahu 24 Syangja 25 Myagdi 26 Parbat 27 Baglung 28 Gulmi 29 Palpa 30 Arghakhanchi 27 GGMS PROJECT IDENTIFICATION DATA Project Title: Ghar Ghar Maa Swasthya (Healthy Homes) Project Numbers: (CA) AID-367-A-10-00002; (TA Contract) AID-367-C-10-00001 Project Dates: (CA) May 1, 2010 – April 30, 2015; (TA Contract) August 1, 2010 – July 31, 2015 Life of Project Funding: (CA) $5,800,000; (TA Contract) $9,883,369 Implementing Partners: (CA) Contraceptive Retail Sales Company (TA Contract) FHI 360 Agreement and Contract Officer’s Representative : Pangday Yonzone ANNEX II EVALUATION STATEMENT OF WORK (SOW) TITLE: To do a mid-term and final evaluation of USAID/Nepal’s Ghar Ghar Maa Swasthya (GGMS) program. Purpose/Objective: The purpose of this contract (PO) is to obtain services from a local organization to do a mid-term and final evaluation of USAID/Nepal’s Social Marketing and Franchising Project, GGMS Program. Project Background and Context: The GGMS or Healthy Homes program seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360. The health status of remote rural Nepal is decidedly lower than in the urban areas.5 In remote areas of Nepal, chronic staffing and management problems are compounded by lack of road transport and the high cost of alternative transportation, which leads to inconsistent supply of essential drugs, supplies and equipment. Medicine stock-outs diminish government health workers’ ability to provide services and lower their motivation. The private sector is likewise affected by high transportation costs, resulting in high prices of goods in local markets. Furthermore, patients become discouraged from visiting health facilities when travelling is dangerous, lengthy and arduous, only to find supply stock-outs or health personnel absences. Therefore, social marketing is a vital element in the Government of Nepal’s (GON) vision of ensuring that products and services reach marginalized populations in remote areas. CRS, the pioneer of social marketing and predominant organization performing social marketing in Nepal, has been working in family planning, HIV/STI prevention and maternal and child health. CRS has more than 30 years of USAID support (direct and indirect) and has worked with many USAID partners, including all the key US-based partners with social marketing experience: Academy for Educational Development (AED), Futures Group, John Snow Inc., and Population Services International (PSI). For the past ten years, USAID/Nepal support to CRS has focused on the improvement of family planning services and raising demand in much of the Terai, or plains area of Nepal. These efforts have contributed to increased contraceptive use in the Terai. In accordance with the GON’s request, USAID/Nepal has encouraged CRS efforts in lesser-served areas in the hilly and mountain districts where the GON faces more challenges to improve public sector services. Other Major Social Marketing Players in Nepal 5 See New Social Marketing Program ADD for statistics used in planning and see NDHS 2006 for full picture at the time of planning and 2011 for comparative analysis with current knowledge, behavior and status. 28 Since mid-2006, Population Services International (PSI) has implemented a Global Fund to Fight HIV/AIDS, Tuberculosis, and Malaria(GFATM) project in conjunction with the Epidemiology and Disease Control Division of the Department of Health Services (DOHS)/Ministry of Health and Population (MOHP) PSI has recently received funding from a large anonymous donor to implement social marketing, which they are planning to do through their branded Sun Quality Network, a social franchise network created by PSI with USAID funding. In March 2009, PSI presented their General Operational Plan for a public-private partnership with the MOHP. PSI aims to improve rural demand and access to quality, long-term contraceptive family planning services (intra-uterine devices, implants, comprehensive abortion care and medical abortion) through public and private providers. Specifically, PSI plans to improve remote area services for intra-uterine devices and implants. Currently, PSI does not intend to re￾introduce their No. 1® condom brand, the OK products or other short-term family planning products in Nepal. Marie Stopes International (MSI) is a social marketing organization that operates through their locally registered NGO, Sunaulo Pariwar Nepal. MSI supports clinics and mobile camp operations where they offer voluntary surgical contraception, comprehensive abortion care, medical abortion, Depo-Provera®, their own brand of oral contraceptives and Jodi® brand condoms. MSI expects that upward sales trends of Jodi® condoms, targeted to an upper middle class market, will reach profitability in the medium-term, with eventual cost-recovery. MSI’s goal is to use the revenue generated to cross-subsidize clinical services. Jodi® condoms are sold from 61 MSI clinics, mobile camps, youth-friendly service centers and six kiosks on the border with India. More than 600 sales promoters are reaching traditional and non-traditional outlets in over 65 districts. MSI’s oral contraceptives are at a middle level price range, with good primary sales and unknown secondary sales. MSI is interested in diversification and is examining the possibility of adding voluntary HIV/AIDS counseling and testing services at their clinics. MSI has one clinic in the remote mountain district Jumla. Otherwise, to-date their clinics are all in more accessible hill and Terai districts. MSI plans a gradual extension into remote locations. Project Intent: USAID/ Nepal’s Ghar Ghar Maa Swasthya Program, or Healthy Homes, seeks to improve the institutional sustainability of CRS and also to increase availability and accessibility of health products in hard-to-reach rural areas. The program is assisting the Government of Nepal to expand the impact of the private sector in social marketing by providing a low cost supply of maternal child health, family planning and HIV prevention products and services. The Ghar Ghar Maa Swasthya (GGMS) Program strategy is designed to: a. Expand CRS’s proven track record of operational efficiency to produce an even more cost-effective distribution system within the existing network of reproductive health product distributers. b. Increase revenue generation and cost recovery through evidence-based price increases, and in this way gradually create a sustainable and self-reliant social marketing program. c. Employ a marketing approach that improves the “reach” of products and services to the neediest living in the more remote areas of the country. This includes the deployment of new and nontraditional distributors and stocking points, and new outlets such as cabin restaurants, groceries, small shops, etc. d. Improve and monitor the quality of services within the network and ensure proper waste management. e. Increase the availability and access to condoms and other reproductive health products to most at-risk, and in the hot zones6 of 6 A hot zone is a cluster of hot spots where sex trade is conducted. Development Hypothesis If USAID/Nepal applies more liberalized regulations on use of revenue by CRS from social marketing program and CRS improves its business capacities and social marketing behavior change communication skills, then CRS will use previous revenue and new profits from commercial products from urban areas to cross￾subsidize necessary MCH/FP products and services in remote rural areas, thereby requiring less donor support in the future. 29 Partial Cost recovery products:  Panther Condoms IUD Full cost recovery products: Third brand of condom Nilocon White ORS Emergency Contraceptive pills Sanitary Pads Bed nets Nepal where the rate of HIV/AIDS infections is rising. f. Bring about behavioral change in the at-risk and hard-to-reach populations, including the youth of Nepal. g. Create a better environment for the distribution of health products and services in Nepal. Project’s Result Areas: a) IR-1 Increased supply of selected high-quality family planning, reproductive health and maternal and child health commodities and services in hard to reach rural areas through private sector health providers: i. CRS expects to achieve 1.8 million Couple Year Protection (CYP through the sales of both subsidized and commercial products over 5-year period and reduction on donor dependence on commodity supply. ii. CRS's product cost share is expected to increase from 6% in 2011 to 80% in year 2015. b) IR-2. Increased availability of HIV/AIDS/STI prevention commodities and services among most at risk groups in designated hot zones. c) IR-3. CRS achieves at least full cost recovery, or profit, with several products, and product cost recovery (recovery of the product cost but not all of the services or overhead involved in provision of the product) with all products not targeted to rural and hard to reach areas, by the end of project. d) IR-4. CRS's marketing and strategic behavior change communication activities successfully increase sales. Note to IR-3: In spite of a long history of successful achievements CRS is still dependent on donor support due in part to specific limitations in organizational capacity. CRS was granted the authority to use program-generated revenue/credit funds only in 2009. For a social marketing organization to attain self-sustainability, it is essential that it have profit-making authority so that they can re-invest profit as cross-subsidization can re-invest profit as cross￾subsidization to non-profit-making elements in the portfolio. CRS is in process of completing their first product procurement under GGMS, and will need more guidance to develop full competency in this activity. Other organizational capacities that CRS needs to enhance to become more self-sustainable include: long term strategic planning, product analysis, and designing and conducting behavior change communication campaigns. Project Approach and Implementation: The GGMS social marketing program is national in scope and is a fully integrated program in which all components are coordinated, inter-linked and mutually supportive of USAID/Nepal’s and GON health goals. Ghar Ghar Maa Swasthya program is comprised of two components: a. Under the first component, USAID/ Nepal has a Cooperative Agreement with Nepal CRS Company for social marketing activities, focusing on sustainability through commercial marketing in the urban areas and promoting subsidized products in the rural, hard-to-reach areas. CRS implements strategies to cost￾effectively expand product and service availability. Partnerships have been important for CRS, with the most important being their public-private partnership with the Government of Nepal. Members of the CRS Board include representatives from the MOHP, DOHS, Social Welfare Council, the Private Sector, Professional Organizations (Health) and Non-Governmental Organizations (NGOs). Through their relationship with FHI 360 (see below), CRS is expected to develop greater competency in program monitoring based on the joint GGMS Performance-based Monitoring System. CRS also has partnerships with private distributors throughout Nepal. CRS also has worked with other donors, e.g. in 1997, they entered into an assistance agreement with KfW Entwicklungsbank (KfW) to support family planning and health activities through social marketing programs.. As the leading social marketing organization in Nepal, CRS advances national family planning, maternal and child health, and HIV/AIDS plans in Nepal, through a national network that raises awareness and use of health and family planning products among poorer segments of the Nepali population using modern marketing tools and techniques. b. Under the second component, USAID/ Nepal has a Contract with FHI 360, (a) to provide technical assistance to CRS to build their capacity and to achieve increased self sustainability and (b) to design and 30 implement behavior change communication activities, such as generic campaigns promoting family planning best practices. Under the original award, part (a) was to be for the first three years only. Due to legal complications beyond the scope of the USAID/Nepal GGMS contract, FHI 360 was delayed in starting several key aspects of the technical assistance to CRS7 . FHI 360 requested and USAID/Nepal is in process of reviewing an amendment of the Technical Assistance component to allow for the capacity-building activity to extend through the entire 5-year award period, rather than strongly tapering off in the third year as planned. The amendment presents the need for this change as being primarily due to program constraints imposed by USAID on former partner organization, AED, before they were purchased by FHI 360. FHI 360 supports CRS in continuing its path of transformation into the stronger, more independent local social marketing organization that can achieve the results expected by its supporters, particularly USAID. FHI 360 is mentoring CRS, through the Deputy Chief of Party and others, on planning, marketing, and behavior change communication methodologies to help CRS enhance organizational capacities to recover product costs for most of its social marketing interventions, recover full costs for some products, and generate “surplus revenue” from a new set of commercial initiatives. FHI 360 also supports CRS to serve the underserved and at-risk populations with a range of subsidized products and services. FHI 360 will help CRS scale up promising rural and community-based marketing initiatives through the engagement of cost-efficient human resources such as women, youth entrepreneurs, and experienced NGOs to perform direct selling to the hard-to-reach and at-risk, including in hot zones. FHI 360 will also develop CRS capacity to plan and implement generic behavior change communication (BCC) campaigns with the GON and corporate sponsors, strengthening their public-private partnerships. Under the second component in GGMS program, FHI 360 takes a four-pronged approach towards building CRS's capacity to become a sustainable social marketing organization, and to create greater demand for health products with a particular emphasis on hard-to-reach areas: a. Increasing CRS's Independence and Sustainability: CRS seeks to recover product costs for most of its social marketing interventions and will explore new commercial initiatives to generate revenue that help achieve full cost recovery for several products. b. Focusing Distribution and Marketing on Rural, Hard-to-Reach Areas: CRS focuses program efforts on underserved and most-at-risk populations, scaling up promising rural and community-based marketing initiatives, and engaging non-governmental organizations and commercial distributors to increase product accessibility in hard-to-reach areas. c. Building CRS's Organizational Capacity: FHI 360 supports CRS in continuing its path of transformation into a stronger, more independent, and local social marketing organization. d. Implementing Cutting-Edge Strategic Behavior Change Communication: FHI 360 helps build CRS's capacity to implement innovative, evidence-based, BCC programs to support its existing product portfolio. In addition, FHI 360 manages generic BCC and market-building initiatives in collaboration with local partners and the GON to ensure program success and sustainability. CRS and FHI 360 work closely together in the implementation of the GGMS program. CRS and FHI 360 have joint monthly meetings, a joint Performance Monitoring Plan and joint Work plans, however they report separately to USAID/ Nepal. The CRS Cooperative Agreement ends in April 2015 and the FHI 360 Contract ends in July 2015. Evaluation Purpose: 7 The award was initially awarded to AED, who was very shortly afterwards suspended by USAID. 31 A single Contractor must conduct the following activities: a. Mid-term Evaluation: This external evaluation will come at the chronological mid-point of the GGMS project. It is a mid-term, formative evaluation whose objectives are to help determine what components and project aspects are working well and why, which perhaps are not and why, and to make modifications and mid-course corrections if necessary, to help guide GGMS over its second half. Specifically, the Contractor must examine how successful the GGMS partnership has been in building CRS’s capacity to achieve the targeted results and move toward self-sustainability. The evaluation should also identify ongoing TA needs, barriers to capacity development and any recommendations for changes in the structure of the TA contract or cooperative agreement. USAID/Nepal has given a strong focus on capacity-building in its draft Country Development Cooperation Strategy (CDCS). In this regard, the evaluation must provide input on the success of the modality of providing capacity-building through a partnership between an international organization and a local NGO. In summary, the evaluation will help all involved to better understand the initial results achieved in organizational capacity development, to re-focus and strengthen the program as required, and inform the implementation of the Mission CDCS. b. Final Evaluation: This external evaluation will come at the end of the project’s life cycle and is meant to serve a tri-fold purpose: i. to assess whether social marketing has been an effective means of increasing supply and demand for high-quality family planning, reproductive health, maternal and child health, and HIV/AIDS/STI prevention commodities and services in remote areas, ii. to assess and recommend with regard to CRS’s progress towards self-sustainability, and iii. to assess FHI 360’s performance in capacity-building interventions. Note: USAID and the Contractor must review the mid-term evaluation to identify whether it warrants changes to the final evaluation purpose and question. Audience and Intended Use: The prime audience of this evaluation report will be the USAID/Nepal Mission, specifically the Health and Family Planning (HFP) team, the Mission CDCS team, the Asia Bureau, and the implementing partners, CRS and FHI 360. A summary of the evaluation, including description of methods, key findings and recommendations will be provided to the MOHP, KfW, and LMD/DOHS and made available to the public on-line in a fully searchable form within three months of the evaluation’s conclusion. USAID/Nepal will use the report to make changes to the project (mid-term) and to shape any follow-on activity (final). Both FHI 360 and CRS will learn about their strengths and weaknesses and how to adjust their programs accordingly. MOHP will learn how effectively CRS has increased availability and accessibility of health products in hard-to-reach rural areas, while becoming a more independent organization. This will give any necessary input to MOHP for review of national policies on provision of products and services by private organizations. Evaluation Questions: a. Mid-term Evaluation: i. How effective has the GGMS program partnership been in building CRS’s capacity to operate as a more independent, local social marketing organization, in terms of:  FHI 360‘s identification of CRS’s capacity needs;  Quality and appropriateness of the capacity-building TA offered;  CRS’s openness to receive and use the TA offered;  Building and operationalizing organizational systems and processes to develop CRS as an independent organization by the end of the project;  Improving the quality and efficiency of CRS’s service delivery, including customer satisfaction. 32 ii. Given any barriers to capacity development are any changes to the current TA structure or additional TA directions recommended? iii. Are there any revisions that should be made in the expectations or demands of CRS? b. Final Evaluation: i. To what extent have GGMS program achievements within Intermediate Results 1-4 endorsed the USAID/ Nepal strategy and the strategies of the Nepal Health Sector Plan (NHSP) II for making positive health (family planning, maternal and child health, and HIV prevention) impact in remote rural areas through social marketing? Was behavior change communication a vital or minor part of the achievement? Describe any shortcomings, why these occurred, and what could be done to prevent these in the future. Describe any recommendation for the MOHP with regards to the evaluation findings and the place of social marketing in the NHSP II and upcoming NHSP III. ii. To what extent has CRS become financially and technically self-sustainable? What is recommended for CRS to function past the life of the GGMS program?  Does CRS have the technical and organizational capacities to sustain activities efficiently and effectively without FHI 360’s TA?  What were the strengths and weaknesses in the CRS management structure and systems in terms of efficient implementation of the program? Were the management structure and systems of the technical units relevant to the overall program objectives and results? What could have been done to better strengthen systems and build capacity of staff to manage technical and administrative functions?  In terms of financial management, is CRS adequately able to segregate commodity accounts? Are internal controls strong, transparent and accountable? Were any of their financial sustainability strategies more profitable or helpful than others? Have they been able to lower their dependency on outside donors? iii. Has FHI 360 satisfactorily incorporated and successfully implemented the recommendations for TA from the mid-term evaluation? Evaluation Design and Methodology: The firm or evaluators must use a wide range of quantitative and qualitative methods and approaches for collecting and analyzing the information which will be required to assess the evaluation objectives and answer the presented questions. The firm or evaluators must specify the method(s) that will be used to collect data for answering each of the evaluation questions separately. The evaluation team must be required to start its work with a paper review of all the documents cited in the suggested background documents (preliminary list provided in Annex II)8 . The evaluation team must spot-check GGMS Final Reports information on product and services availability in remote areas during the evaluation team remote area field visits. The evaluation team should also be prepared to conduct interviews with key informants (preliminary list provided in Annex III), as well as conducting site visits and team planning meetings. Data collection methodologies and instruments are subject to approval by USAID/ Nepal prior to the start of the evaluation. The mid-term and final evaluations must be carried to the extent possible in a positive and participatory approach. The evaluation team leader, assisted by one or more evaluation team members, must facilitate and conduct thorough team planning meeting(s) (estimated at two days) before starting the evaluation. USAID/ Nepal’s focal person will participate in the team planning activities and other HFP staff may be involved as appropriate. The 8 The evaluators must go through the two agreement documents very carefully to distinguish the responsibilities of the two implementing partners. 33 following are illustrative activities for the evaluation team:  Clarify team members’ roles and responsibilities, working styles, logistics, administration and dispute resolution;  Review and develop final evaluation questions;  Finalize data collection plans and tools;  Finalize an evaluation work plan and assignment timeline and request USAID Nepal feedback;  Develop a preliminary draft outline of the team’s report; and  Finalize the report Data Collection Methods and Sources: It is suggested that the project evaluation team methods may include, but not be limited to:  Review GGMS program documents, work plans, M&E plan, annual and semi-annual reports, performance monitoring plan, demobilization plans, marketing plans, capacity building and collaboration plans, BCC and behavior change intervention (BCI plans and campaigns and other GGMS program related technical documents and studies, as well as other documents/reports made available by the HFP Team  Review detailed data summaries from GGMS partners on distribution of commodities, coverage of services, marketing activities, and behavior change communication interventions.  Review detailed data summaries from GGMS on product and cost recovery, and other descriptions of the financial status of CRS  Interview key stakeholders including donors, government counterparts and implementing agencies and Nepali who utilized the commodities and services provided by GGMS  Interview GGMS key staff  Interview USAID/ Nepal Social Marketing team -Conduct specific field visits (including at least one remote area) and observe the activities in actions Data Analysis Methods: Prior to the start of data collection, the evaluation team must develop and present a data analysis plan. The data analysis plan is subject to USAID review and approval and must detail the following: a) How stakeholder interviews will be transcribed and analyzed; b) What procedures will be used to analyze qualitative data from key stakeholders interviews; and c) How the evaluation will weigh and integrate qualitative data from these sources with quantitative data from project monitoring records to reach conclusions. Deliverables: The evaluation team must produce the following deliverables for the mid-term and final evaluation: a) Planning meeting documents: The team leader must submit to USAID/Nepal for approval at the conclusion of the team planning meeting(s) a finalized work plan/timeline for the evaluation, finalized data collection plans and tools, finalized evaluation questions, finalized format of the draft final report including table of contents, defined roles and responsibilities of team members, including agreed upon writing responsibilities for the final report. b) Mission in-briefing and final debriefing: The team must conduct an in-briefing with USAID/Nepal Social Marketing managers as part of the initial planning meeting(s), as well as a de-briefing. At the de-brief meeting with USAID/Nepal, the team will present their main findings and recommendations, a CD-ROM with a power point of the presentation, a hard copy of the presentation and a hard copy draft final report and include time for discussion. The draft final report will also include a separate ‘sensitive but unclassified’ annex that contains sensitive information that should not be published in an open, public document, but which contains important information for USAID’s internal use. 34 c) Draft Report: The Contractor must submit the first draft of the evaluation report at the full team’s debriefing at USAID/Nepal so that comments and feedback can be incorporated into the draft report that will be left with the Mission. The draft will conform to the agreed upon format and include findings, conclusions and recommendations, as well as annexes with additional reports, copies of data collection tools, lists of key informants, etc. USAID/Nepal will review and provide one set of comments to the draft report within 10 working days upon receipt d) Final Report: The Contractor must submit the final draft evaluation report to USAID/Nepal within 5 working days after the team receives comments from USAID/Nepal on the draft report. Once the final draft report is reviewed and accepted, the Contractor must submit a print-ready Final Report version to USAID/Nepal for approval before publication. Any procurement sensitive information or recommendations made by USAID/Nepal must be removed from the public Final Report and provided to the Mission as an Internal Memo for USAID/Nepal Use Only. e) Raw data and records of the evaluation report collected by the evaluation team (e.g. interview transcripts, survey responses etc.) in electronic form (CD ROM) and submitted separately from the report. Reporting Guidelines: USAID/Nepal requires that the team review USAID/Nepal’s Criteria to Ensure the Quality of the Evaluation Report, which can be accessed online at: http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf As mentioned above, findings from the evaluation will be presented in a draft report at a full briefing with USAID/Nepal. The format for the evaluation reports are as follows: a) Executive Summary – Concisely state the most salient findings and recommendations (1-1.5pp); b) Table of Contents (1pp); c) Introduction – Purpose, audience, and synopsis of task (1-2pp); d) Background – Brief overview of CRS in Nepal, USAID program strategy and activities implemented to improve the institutional sustainability of CRS and also to increase availability and accessibility of health products in hard-to-reach rural areas, purpose of the evaluation (2-3pp); e) Methodology – Describe evaluation methods, including constraints and gaps (1-2p); f) Findings/Conclusions (10-15pp); g) Recommendations/Future Directions (2-5pp) h) References Includes bibliographical documentation, meetings, interviews and focus group discussions; i) Annexes – Annexes that document the evaluation methods, schedules, interview lists and table – should be succinct, pertinent and readable. The final report must be clear and grammatically correct to be accepted by USAID/Nepal. It is required that the final report be prepared and orally presented by a native or highly proficient English speaker. A full version of the evaluation report must be submitted to USAID/Nepal in hard copy as well as electronically. The report format must be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 30 pages, excluding references and annexes. The evaluation team leader must also submit one electronic or hard copy of the mid-term and final evaluations no later than 30 days after completion to the Development Experience Clearinghouse (DEC) with a cover sheet indicating the type of evaluation and the design. The DEC evaluation submission must also include a 3-5 pages summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, 35 recommendations and lessons learned (if applicable) from the evaluation. Development experience documentation may be submitted Online: http://www.usaid.gov/results-and-data/information-resources/development-experienceclearinghouse-dec; By mail (for pouch delivery): USAID Development Experience Clearinghouse M/CIO/ITSD/KM/DEC RRB M.01-010 Washington, DC 20523- 6100 For questions on DEC submissions, contact M/CIO/ITSD/KM/DEC Telephone: +1 202-712-0579 E-mail: DocSubmit@usaid.gov Team composition: In ensemble the team members must meet the following criteria:  Strong team skills  Ability to work with persons of diverse professional and social backgrounds with varying opinions  Strong written and verbal English communication skills  Demonstrated strong technical and analytical skills  Ability to travel and work for short durations in challenging environments  Working knowledge of health product and social marketing and social franchising in Nepal and in the region. Team Leader: Dr. Yagya Bahadur Karki The team leader must be a local senior level evaluator with a Master of Public Health or Master of Business Administration or other relevant higher education degree, and five years of experience in designing and conducting development project evaluations. Alternatively, a Bachelor degree in a relevant academic field plus fifteen years of relevant experience is acceptable. The team leader must have successful experience showing competency to supervise two local experts and produce a high-quality written evaluation report. Particular expertise and experience should include: - Strong monitoring and evaluation experience in public health activities - Strong knowledge of social marketing and social franchising and public/private partnerships - Prior experience as the leader of large development project evaluations - Strong team management and supervisory skills - Demonstrated strong cross-cultural communication skills - Demonstrated lead writer with strong technical English writing skills - Experience working with a range of government officials, local NGOs, private and public sector enterprises in project implementation and management Local Expert(s): Gyanu Shrestha and Ritu Prasad Gartaula The local expert team member(s) must have at least a Bachelor of Public Health or Bachelor of Business Administration or other relevant academic degree with at least five years of experience. The local expert(s) must work under the supervision of the team leader and will be responsible for logistics, including organizing meeting/site visits with key commercial sector partners and government counterparts. Particular expertise and experience should include the following. The sum of skills can be mixed among the team members. - Institutional capacity development expertise and experience - Behavioral change communication expertise and experience - Monitoring and evaluation expertise and experience 36 - Ability to translate select documents and interviews - Ability to handle general team logistics - Extensive marketing skills - Financial management expertise, including accounting skills - Versatility with business management systems and practices - Ability to translate select documents and interviews Logistics: The Contractor must make all logistical arrangements, such as flight reservations for in-country travel, airport pick-up/drop, lodging and meetings. The key documents and background materials for reading is listed in Annex I. USAID/Nepal will send these via email to the Contractor. USAID/Nepal will help arrange the in-briefing and debriefing when the award is made. The USAID/Nepal’s HFP staff will provide contacts for meetings and a list of the suggested site visits for the team to arrange meetings. USAID/Nepal may accompany the team in key meetings with Senior GON officials and with selected stakeholders and possibly attend some field visits. Mission Point of Contact Information: The COR and AOR of GGMS programs is Mr. Pangday Yonzone. The Contractor shall contact the COR and AOR via email to pyonzone@usaid.gov. Timeline: The Contractor must conduct the mid-term evaluation between May-August 2013 and the final evaluation between November 2014-January 31, 2015. USAID/Nepal anticipates that the preparation days, in-country work including site visits, as well as drafting and finalizing the evaluation reports will be completed according to the following illustrative timeline. A 6-day work week is authorized while working in country. The timeline for each evaluation is not to exceed the below: Task Days Document review/preparation 5 days Team planning meeting (in KTM) 2 days Interviews and discussion with key stakeholders 6 days Field studies at project implementation sites-possibly 2-3 flights and car rental involved 15 days USAID debriefing 1 day Draft report writing 5 days Draft-to-final report revisions, based on Mission comments 5 days Total consultation period 39 days (Team leader) 60 days The above tasks represent adequate time for consultations, exchanges of findings among the team, team planning, report preparation and discussion with Mission staff. The evaluation must be completed within 7 weeks, including preparation days, all in-country work, and draft report writing and finalization. However, final editing and formatting may take an additional 2-3 weeks. Terms and Conditions for the Contractor: The Contractor must sign a Non-Disclosure Agreement (Annex IV), inclusive of not being part of any potential future bid related to the GGMS program with USAID/Nepal. The Contractor must also sign a Disclosure of Real or Potential Conflict of Interest for USAID evaluation (Annex V) 37 Budget: The final and negotiated cost is stated below for the completion of both evaluations. Mid Term Evaluation Final Evaluation Total Cost LOE (days) or in Units Daily Rate/ Item Rate Cost LOE (days) or in Units Daily Rate/ Item Rate Cost I. Direct Labor Team Leader 39 255 9,945 39 281 10,940 20,885 Local Expert 1 35 110 3,850 35 121 4,235 8,085 Local Expert 2 28 125 3,500 28 138 3,850 7,350 Direct Labor Total 17,295 19,025 36,320 II. Travel & Per Diem a. Airfare-in-country (4 site visits) Team Leader 4 125 500 4 138 550 1,050 Local Expert 1 4 125 500 4 138 550 1,050 Local Expert 2 4 125 500 4 138 550 1,050 b. Lodging & M& IE (4 site visits/4 days each) Team Leader 16 30.00 480 16 33 528 1,008 Local Expert 1 16 20.00 320 16 22 352 672 Local Expert 2 16 20.00 320 16 22 352 672 Ground Transportation (taxis etc.) 20 110 2200 25 121 3025 5,225 Travel Total 4,820 5,907 10,727 III. Other Direct Costs Communication 250 275 525 Materials/Supplies 800 880 1,680 Financial (bank fees, wire transfer costs, etc.) 125 137.5 263 Delivery/Couries 100 110 210 Editing 1500 1650 3,150 Printing/Photocopying 450 495 945 Insurance 300 330 630 Total Other Direct Costs 3,525 3,878 7,403 Total Costs 25,640 28,809 54,449 Fees/Profit % for the firm 10 2,564 2,881 5,445 28,204 31,690 59,894 VAT % 13 3,667 4,120 7,786 Grand Total Costs 31,871 35,810 67,680 38 Contract Type, Amount and Payment Plan: USAID intends to offer a “Firm Fixed Price” contract. The total cost of the Contract is $67,680. The total cost of the mid-term evaluation is $31,871 and the total cost of the final evaluation is $35,810. That means a total cost for all the activities described in the statement of work However, USAID may provide payments upon completion of the following benchmarks:  25 percent upon receiving a finalized work plan/timeline for the evaluation, data collection plans and tools, evaluation questions, format of the draft final reporting. (Deliverable a).  25 percent upon submission of the first draft of the evaluation report at the full team’s debriefing at USAID/Nepal so that comments and feedback can be incorporated into the draft report that will be left with the Mission. (Deliverables b and c)  50 percent upon submission of the final report and upon approval of the report by USAID. (Deliverables d and e). Payment shall be made in Nepali Rupees for costs close to 30 days upon receipt of a valid invoice and SF-1034 voucher by the Office of the Controller as per the prompt payment provisions of FAR 52.232.25. Accounting and Appropriation Data: Operating Unit (OP): Nepal Distribution Code: 367-M Program Area: A11, A26 Program Element: A052, A140 BORN: 367018200HFP03; 367018200HFP06 Fund: GH-C 2009/2010 Amount: $67,680 Technical Direction: Technical directions for this work will be provided by the in country POC. Technical directions, as used herein, are directions to the Contractor that fill in details, suggest possible lines of inquiry, or otherwise complete the general scope of work. Technical directions must be within the terms of this PO, shall not be changed or be modified in any way, and shall not constitute changed within the meaning of the clause FAR 52.243-4 Changes (Jun 2007). Note: The Contractor must notify the Contracting Officer through the POC of any changes in the performance of the contract per FAR 52.243-7 Notification of Changes (APR 1984). Purchase Order Clauses: This purchase order incorporates one or more clauses by reference, with the same force and effect as if they were given in full text. Upon request, the Contracting Officer will make their full text available. Also, the full text of a clause may be accessed electronically at this address: http://www.arnet.gov. Drug-Free Workplace, FAR 52.223-6 Availability of Funds, FAR 52.232-18 Prompt Payment, FAR 52.232-25 Changes-Fixed-Price, FAR 52.243-1 Termination for Convenience of the Government (Fixed-Price), FAR 52.249-2 Excusable Delay, FAR 52.249-14 39 EXECUTIVE ORDER ON TERRORISM FINANCING (FEB 2002): The Contractor/Recipient is reminded that U.S. Executive Orders and U.S. law prohibits transactions with, and the provision of resources and support to, individuals and organizations associated with terrorism. It is the responsibility of the contractor/recipient to ensure compliance with these Executive Orders and laws. This provision must be included in all subcontracts/subawards issued under this contract/agreement. Acceptance: Name: _____________________________ Signature: _________________________ Date: _____________________________ 40 ANNEX I: List of Preparatory Materials For purposes of proposal preparation the following documents are available (further, GGMS program documents of USAID/Nepal will be available if required by the awardee): a) The Cooperative Agreement and Contract. b) GGMS Joint Work plans for 2011-2012 and Performance-based Monitoring System. c) “The National Action Plan for HIV/AIDS Control (2005-2006).” Please refer to the following URL for this document. d) “The National Communication Strategy for Maternal, Newborn and Child Health 2011 – 2016. e) “National Communication Strategy for Adolescent Sexual and Reproductive Health (ASRH) Nepal, 2011- 2016. f) Reproductive Health Social Marketing Strategy 2002 – 2006 by PSI. g) National Family Planning Service Delivery Guideline (unofficial translation). 41 ANNEX II: List of Key Current Stakeholders and Partners S.N. Organization 1. Department of Health Services 2. NCASC 3. Family Health Division 4. Child Health Division 5. Logistics Management Division 6. KfW 7. SAATH SAATH Project 8. Nepal CRS Company 9. FHI 360 10. DELIVER II 12. Nepal Fertility Care Center 13. Population Services International 14. Marie Stopes International 15. Family Planning Association of Nepal 42 ANNEX III: USAID Evaluation Policy Criteria to Ensure the Quality of the Evaluation Report 43 ANNEX IV: Non-Disclosure Agreement PRECLUSION FROM FURNISHING CERTAIN SERVICES AND RESTRICTION ON USE OF INFORMATION With respect to proposal submitted dated February 25, 2013 in response to solicitation SOL-36713-000006 of USAID/Nepal’s mid-term and final evaluation of Ghar Ghar Maa Swasthya (GGMS) dated February 4, 2013, the undersigned hereby agrees and certifies to the following: (a) This SOW calls for the Contractors to furnish important services in support of the mid-term and final evaluation of the GGMS. In accordance with the principles of FAR Subpart 9.5 and USAID policy, the Contractor shall be ineligible to furnish, as a prime or subcontractor or otherwise, implementation services under any contract or task order that results in response to findings, proposals, or recommendations in the evaluation reports within 18 months of USAID accepting the reports, unless the head of the contracting activity, in consultation with USAID’s competition advocate, authorizes a waiver (in accordance FAR 9.503) determining that preclusion of the Contractor from the implementation work would not be in the Government's interest. (b) In addition, by accepting this contract, the Contractor agrees that it will not use or make available any information obtained about another organization under the contract in the preparation of proposals or other documents in response to any solicitation for a contract or task order. (c) If the Contractor gains access to proprietary information of any other company in performing this evaluations, the Contractor must agree with the other company to protect the information from unauthorized use or disclosure for as long as it remains proprietary, and must refrain from using the information for any purpose other than that for which it was furnished. Contractor must provide a properly executed copy of all such agreements to the contracting officer. Signature: ________________________ Name Typed or Printed: ________________________ Date: ________________________ 44 ANNEX V: Disclosure of Real or Potential Conflict of Interest for USAID Evaluations Instructions: Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest.9 For external evaluations, all evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated.10 Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s). In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.11 Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 9 USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part 9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct. 10 USAID Evaluation Policy (p. 11) 11 FAR 9.505-4(b) 45 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Title Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 46 ANNEX VI: Checklist for Evaluation Report Review Title of study being reviewed: __________________________________ GOOD PRACTICE ELEMENTS OF AN EVALUATION REPORT12 Keyed to USAID’s 2011 Evaluation Policy EVALUATION REVIEW FACTOR 1 2 3 4 5 Comments STRUCTURE OF THE REPORT 1. Does the evaluation report have a cover sheet attached indicating the type of evaluation conducted (e.g. performance evaluation or impact evaluation) and general design? 2. If a performance evaluation, does the evaluation report focus on descriptive and normative evaluation questions? 3. If the evaluation report uses the term "impact evaluation," is it defined as measuring the change in a development outcome that is attributable to a defined intervention (i.e. impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual)? 4. Regardless of the type of evaluation, does the evaluation report reflect use of sound social science methods? 5. Does the report have a Table of Contents (TOC)? 6. Do Lists of Figures and Tables follow the TOC? 7. Does the report have a Glossary of Terms? 7.1 Are abbreviations limited to the essential? 8. Is the date of the report given? 9. Does the body of the report adhere to the 20 page guide? 10. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 11. Does the report‘s presentation highlight important information in ways that capture the reader‘s attention? 12. Is the report well written (clear sentences, reasonable length paragraphs, no typos, acceptable for dissemination to potential users)? 13. Does the evaluation report focus on the essential issues concerning the key questions, and eliminate the "nice to know", but not essential information? 14. Does the evaluation report disclose either lack of a conflict of interest by all evaluation team members and/or describe any conflict of interest that existed relative to the project being evaluated? 15. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? EXECUTIVE SUMMARY 16. Does the evaluation report begin with a 3- to 5-page standalone summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if 12 In addition to the USAID 2011 Evaluation Policy, good practices in evaluation reporting have also been drawn from: Morra Imas, Linda and Ray C. Rist. 2009. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC.: The World Bank. Scriven, Michael. 2005. Key Evaluation Checklist. Stufflebeam, Daniel L. 1999. Program Evaluations Metaevaluation Checklist. 47 applicable) of the evaluation? 17. Does the Executive Summary concisely state the main points of the evaluation? 18. Does the Executive Summary follow the rule of only saying what the evaluation itself says and not introducing new material? INTRODUCTION 19. Does the report introduction adequately describe the project? 19.1 Does the introduction explain the problem/opportunity the project was trying to address? 19.2 Does the introduction show where the project was implemented (physical location) through a map? 19.3 Does the introduction explain when the project was implemented? 19.4 Are the "theory of change" or development hypotheses that underlie the project explained? (Does the report specify the project‘s inputs, direct results (outputs), and higher level outcomes and impacts, so that the reader understands the logical structure of the project and what it was supposed to accomplish?) 19.5 Does the report identify assumptions underlying the project? 19.6 Does the report include sufficient local and global contextual information so that the external validity and relevance of the evaluation can be assessed? 19.7 Does the evaluation report identify and describe any critical competitors to the project that functioned at the same time and in the project‘s environment? 19.8 Is USAID‘s level o f investment in the project stated? 19.9 Does the evaluation report describe the project components funded by implementing partners and the amount of funding? 20. Is the purpose of the evaluation clearly stated? 21. Is the amount of USAID funding for the evaluation indicated? 22. Are all other sources of funding for the evaluation indicated as well as the amounts? 23. Does the report identify the evaluation team members and any partners in the evaluation? 24. Is there a clear statement of how the evaluation will be used and who the intended users are? 25. Are the priority evaluation questions presented in the introduction? 26. Does the evaluation address all evaluation questions included in the Statement of Work (SOW)? 26.1 Are any modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline indicated in the report? 26.2 Is the SOW presented as an annex? 26.3 If so, does the annex include the rationale for any change with the written sign-offs on the changes by the technical officer? SCOPE AND METHODOLOG 27. Does the report provide a clear description of the evaluation‘s design? 27.1 Is a design matrix or similar written tool presented in an annex that shows for each question/subquestion the measure(s) or indicator(s) used to address it, the source(s) of the information, the type of evaluation design, type of sampling if used, data collection instrument(s) used, and the data analysis plan? 28. Does the report state the period over which the evaluation was conducted? 29. Does the report state the project time span covered by the evaluation? 30. Does the evaluation report indicate the nature and extent of consultation 48 on the evaluation design with in-country partners and beneficiaries? 31. Does the evaluation report indicate the nature and extent of participation by national counterparts and evaluators in the design and conduct of the evaluation? 32. Does the report address each key question around which the evaluation was designed? 33. Is at least one of the evaluation questions directly related to gender analysis of outcomes and impacts? 34. Are data sex-disaggregated? 35. In answering the questions, does the report appropriately use comparisons made against baseline data? 36. If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness? 36.1 As appropriate, does the report include financial data that permits computation of unit costs and analysis of cost structure? 37. Is there a clear description of the evaluation‘s data collection methods (summarized in the text with the full description presented in an annex)? 37.1 Are all tools (questionnaires, checklists, discussion guides, and other data collection instruments) used in the evaluation provided in an annex? 37.2 Does the evaluation report include information, as appropriate, on the pilot testing of data collection instruments? 37.3 Does the evaluation report include information, as appropriate, on the training of data collectors? 38. Are all sources of information properly identified and listed in an annex? 39. Does the evaluation report contain an section describing the "strengths"and "limitations" associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc)? 40. Does the evaluation report indicate the evaluation methodology took into account the time, budget, and other practical considerations for the evaluation such as minimizing disruption and data burden? 41. Does the report have sufficient information to determine if the evaluation team had the appropriate methodological and subject matter expertise to conduct the evaluation as designed? 42. If an impact evaluation was designed and conducted, does the evaluation report indicate that experimental methods were used to generate the strongest evidence? Or does the report indicate that alternative methods for assessing impact were utilized and present the reasons why random assignment strategies were not feasible? 43. Does the evaluation report reflect the application and use to the maximum extent possible of social science methods and tools that reduce the need for evaluator-specific judgments? 44. Does the evaluation scope and methodology section address generalizability of the findings? ANALYSIS 45. Are percentages, ratios, cross-tabulations, rather than raw data presented, as appropriate? 46. When percentages are given, does the report always indicate the number of cases used to calculate the percentage? 46.1 Is use of percentages avoided when the number of cases is small (<10)? 47. Are whole numbers used or rounding-off numbers to 1 or 2 digits? 48. Are pictures used to good effect? 48.1 Relevant to the content 48.2 Called out in the text and placed near the call-out 49. Are charts and graphs used to present or summarize data, where relevant? 49 49.1 Are the graphics easy to read and simple enough to communicate the message without much text? 49.2 Are they consistently numbered and titled? 49.3 Are they clearly labeled (axis, legend, etc.) 49.4 Is the source of the data identified? 49.5 Are they called out in the text and correc tly placed near the call-out? 49.6 Are the scales honest (proportional and not misleading by virtue of being "blown-up")? FINDINGS 50. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 50.1 As appropriate, does the report indicate confirmatory evidence for FINDINGS from multiple sources, data collection methods, and analytic procedures? 51. Are adequate data provided to address the validity of the "theory of change" or development hypothesis underlying the project, i.e., cause and effect relationships? 52. Are alternative explanations of any observed results discussed, if found? 53. Are unplanned results the team discovered adequately described? 54. Are opinions, conclusions, and recommendations kept out of the description of FINDINGS? CONCLUSIONS 55. Is there a clear distinction between CONCLUSIONS and FINDINGS? 56. Is every CONCLUSION in the report supported by a specific or clearly defined set of FINDINGS? 57. Are the CONCLUSIONS credible, given the FINDINGS the report presents? 58. Can the reader tell what CONCLUSIONS the evaluation team reached on each evaluation question? RECOMMENDATIONS 59. Are RECOMMENDATIONS separated from CONCLUSIONS? (Are they highlighted, presented in a separate section or otherwise marked so that the reader sees them as being distinct?) 60. Are all RECOMMENDATIONS supported by a specific or clearly defined set of FINDINGS and CONCLUSIONS? (Clearly derived from what the evaluation team learned?) 61. Are the RECOMMENDATIONS practical and specific? 62. Are the RECOMMENDATIONS responsive to the purpose of the evaluation? 63. Are the RECOMMENDATIONS action-oriented? 64. Is it clear who is responsible for each action? 65. Are the RECOMMENDATIONS limited/grouped into a reasonable number? LESSONS LEARNED 66. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 67. Are the LESSONS LEARNED highlighted and presented in a clear way? 68. Does the report indicate who the lessons are for? (e.g., project implementation team, future project, USAID and implementing partners, etc.) BOTTOM LINE 69. Does the evaluation report give the appearance of a thoughtful, evidence￾based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 70. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? 71. Is the evaluation report structured in a way that will promote its 50 utilization? 72. Does the evaluation report explicitly link the evaluation questions to specific future decisions to be made by USAID leadership, partner governments and/or other key stakeholders? 73. Does the evaluation report convey the sense that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? 51 ANNEX III TEAM PLANNING MEETING SCHEDULE Ghar Gharmaa Swasthya (GGMS) Program: Mid-Term Evaluation Team Planning Meeting Schedule June 5, 2013 Venue: Hotel Shangri-la, Lazimpat, Kathmandu Prepared by: GGMS Evaluation Team: Dr. Yagya. B. Karki, Mrs. Gyanu Shrestha and Dr. Ritu Gartaula Time Agenda Activities 9:30-9:45 Arrival of Participants/Tea 9:45-10:00 Introduction Formal Introduction 10:00-10:15 Scope of Work Highlight by Activity Manager from USAID/Nepal 10:15-10:30 Background/Objectives Briefing by GGMS Evaluation Team Leader 10.30-10.45 Understanding the Roles and Responsibilities of the Team Members Team Leader briefs about the roles and responsibilities of Team Members 10.45-1.00 Developing a Conceptual Framework Discussion on Data Collection Plans and Tools (Key Stakeholders, Partners and Beneficiaries) 1.00-1:45 Lunch 1.45-2.45 Developing a Conceptual Framework (continued) Data Collection Plans and Tools Finalization 2.45-3.00 Tea Break 3.00-3.45 Timeline Discussion on the entire evaluation program timeline (work plan) 3.45-4.45 End Product, wrap up Discussion on preliminary draft outline and logistical & administrative procedures Submission of TPM Documents to USAID/Nepal for approval 52 ANNEX IV EVALUATION TOOLS Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: DG, DOHS, MOHP Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes program seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 20 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Director General, DoHS, MOHP, Teku Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to DG. Issues to discuss: General/Organisational 1. Government policy on private sector development 2. Ghar Ghar Ma Swasthya (GGMS) project 3. Level of involvement of government in implementing projects/programs of GGMS and partner organizations (FP products, MCH, HIV/AIDs products) 4. Achievements 5. Effectiveness 6. Logistic Management 7. Co-ordination , co-operation and monitoring from the government 1 2 53 8. What contribution is GGMS project making in achieving government health goals a. In contraceptive distribution b. Child health service products c. Maternal health service products d. STI/HIV/AIDS e. ORS f. Water Purification 9. Would you say that GGMS project is effective in meeting the needs of target groups in FP, child health, maternal heath/products, STIs/HIV and AIDS? a. If yes, how? b. If no, why? Others 10. Do you get regular information about GGMS / social marketing activities? a. If yes, how? b. If no, how would you like to get informed? 11. How is the social marketing activity conducted by CRS? 12. What do you think of the capacity of CRS for social marketing of FP and MCH products? Is their capacity high medium or low? 13. Are you aware of any strategy in place to sustain social marketing activities conducted by CRS? 14. Which aspects of GGMS activities – social marketing, private sector strengthening, partners' capacity building …. do you think needs strengthening for the remaining period of GGMS in Nepal? 15. Do you have any particular suggestions to make social marketing/GGMS project activities more effective? Thank you. 54 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: FHD, DOHS, MOHP Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Director, FHD, Teku Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Director. Issues to discuss: General/Organisational 1. Government policy on private sector development 2. Ghar Ghar Ma Swasthya (GGMS) 3. Partners of Ghar Ghar Ma Swasthya /Family Health International(FHI) 360 project such as Contraceptive Retail Sales (CRS), National Fertility Care Center (NFCC), other Non Governmental Organisations (NGOs) and INGOs 4. Level of involvement of government in implementing projects/programs of GGMS and partner organizations 5. Please share your views on your position as CRS coordination committee Chairperson 6. What level of contribution is GGMS project making in achieving government health goals a. FP contraceptive products and distribution b. Maternal health service products Family Planning /Contraceptives 1 2 55 7. CRS products are sold but the government distributes them free. How does this affect clients' demand for contraceptives? What comments would you have on this situation? 8. Do you think GGMS and its partners through social marketing and franchising helping increase access of target groups (MRP) to contraceptive use? 9. Where are CRS products popular; urban or rural? 10. Would you be able to tell us the share of GGMS and its partners’ in condom distribution in the country? Would you say that it has the largest share – if so what percentage? Or would you say that it needs to increase its share? 11. What about the share of GGMS and its partners’ in other FP products (pills, Emergency Contraceptives (EC), long acting spacing methods (Depo, IUD, Jadelle) in the country? 12. Do you think GGMS and its partners have the capability to meet the increasing demand of consumers of FP/MCH products? 13. What about the other private and NGO sectors? Do they have FP outlets? If yes, which organizations are doing it? 14. Is the government monitoring Qualitative assurance in FP products that CRS distributes? Safe Motherhood 15. We hear that Nepal CRS Company is promoting “Matri Surakchya Chakki” to prevent post partum haemorrhage? Can you elaborate on this? In what places is it promoting and distributing “Matri Surakchya Chakki”? Is it part of GGMS project activities? 16. GGMS and its partners also promote Clean Delivery Kits(CDK) and it is said that CRS does it quite extensively? 17. Does the government promote CDK too? 18. The last (2011) NDHS data showed only 14% of non- institutional deliveries using CDK while the 2006 NDHS showed 17.6%? Would you have something to say in this context? 19. How is CRS promoting safe motherhood in context of cost recovery? Others 20. The government has Public/private partnership policy; how would you relate this policy with CRS activities? 21. Does the government endorse social marketing? 22. Which aspects of GGMS activities – social marketing, private sector strengthening, CBO capacity building …. needs strengthening in the next phase of GGMS? 23. Your views on sustainability of CRS in future or sustainability after USAID support is withdrawn? 24. Do you have any particular suggestions to make GGMS project activities more effective? Thank you. 56 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: LMD, DOHS, MOHP Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Director, LMD, Teku Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Director. Issues to discuss: Government/Partnership 1. Government policy on private sector development 2. What level of contribution is GGMS project making in achieving government health goals a. Child health service products b. Maternal health service products c. STI/HIV/AIDS (STI treatment kit, CURE) d. ORS e. Water purification (PIYUSH, VIREX) 3. How are RH, child survival and STI/HIV/AIDS health products of GGMS distributed in the country? 4. Do they acquire commodities from your division? 5. Do they follow your target /protocol on commodity distribution, store and disposal system? 6. Has the government endorsed social marketing of health products/commodities policy? 7. Your views on public/private partnership in logistics management 1 2 57 ORGANISATIONAL 8. Does CRS carryout social marketing based on the logistic forecasting? 9. Would you like to suggest on the logistics management of CRS products? 10. What about the quality of CRS products? 11. CRS markets FP/MCH products but the government distributes most of them free. What would you say about this? 12. What about the coverage of CRS products in the country – what percentage of the country do they cover? 13. Related to this is market share. What percentage of the market do CRS products have? a. Share of Sangini (depo) b. Share of Jadelle c. Share of Matri Surakchya Chakki (Safe Motherhood Pills) d. Share of Condoms e. Share of CDK f. Share of pills g. Share of ORS h. Share of Water Purification Products (Virex, Piyush) 14. What impact do you think CRS products have made in the country? 15. Would you say CRS approach is innovative? a. If yes, how? b. If no, how can they be innovative? 16. Effectiveness of CRS activities 17. Lessons Learnt 18. Do you feel that CRS is capable to sustain its results in future even after USAID support? a. If yes, how? b. If no, how can it sustain, would you have any suggestions? Recommendations 19. Which aspects of GGMS activities – social marketing, private sector strengthening, partners' capacity building …. do you think need strengthening for the remaining period? 20. Do you have any particular suggestion to make GGMS/CRS project activities more effective for the remaining period? Thank you. 58 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: CHD, DOHS, MOHP Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Director, CHD, Teku Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Director. Issues to discuss: General/Organisational 1. Government policy on private sector development on child health issue 2. Ghar Ghar Ma Swasthya (GGMS) project, 3. Partners of GGMS/Family Health International (FHI) project such as Contraceptive Retail Sales (CRS), National Fertility Care Center (NFCC), other Non Governmental Organisations (NGOs) and International Non Governmental Organisations(INGOs) 4. Level of involvement of government in implementing projects of GGMS and partner organizations (Planning, Organising, Staffing Co-ordination, Reporting, Budgeting, Monitoring and Evaluation ) 5. What level of contribution is GGMS project making in achieving government health goals a. Child health service products b. Maternal health service products c. Oral Rehydration Solution (ORS) d. Water purification/piyush 6. Would you say that GGMS project is effective in meeting the needs of target groups in Child Health, 1 2 59 a. If yes, how? b. If no, why? ORS related 7. What about child survival? Are CRS products effectively reaching the target group such as the children suffering from diarrhoea? 8. Are ORS packets free in government health facilities? a. If yes, who pays for ORS b. If no, how much do you charge clients who buy ORS? 9. What is the difference between ORS products distributed free of cost and socially marketed ORS products sold by CRS? 10. Do you know how much CRS charges for a packet of ORS? 11. Do you think the price CRS charges are affordable to the beneficiaries? If no, why? 12. Do you have any comments on CRS distribution outlets? If, yes, what comments do you have? 13. What about Non Traditional Outlets (NTOs) such as pan shop for the distribution of CSR products? 14. Do you have any comments on the capacity of CRS for the distribution of child survival products? 15. Do you think ORS is distributed by CRS among children who need it the most? 16. Do you think it is necessary to use social marketing for ORS, to reach the intended groups? 17. They say there are different brands of ORS in market, do you agree? If yes, what are they? 18. Which brand of ORS, in your opinion is liked by people? a. Brand name………………… b. It is supplied by …………. 19. Are you aware of any Behaviour Change Communication (BCC) activity conducted by CRS to promote ORS? If yes, what are they? 20. Do you think GGMS and its partners can do more in future for child health program of the government? Others 21. What about child and maternal health products? Are these products effectively distributed to women? 22. GGMS and its partners also promote Clean Delivery Kits (CDK) and it is said that Nepal Contraceptive Retail Sales (NCRS) does it quite extensively, what are your views on this? (Government Policy on CDK) 23. We hear that Nepal CRS Company is promoting “Matri Surakchya Chakki” to prevent post partum haemorrhage? 24. Are you aware of it? 25. If yes, Can you elaborate on this? 26. In what places is it promoting and distributing “Matri Surakchya Chakki”? 27. CRS is now supported by USAID and other donors but this support is for limited period. Do you think CRS can sustain in future even after donor support? If yes, how? If no, why not? 28. Do you have any suggestions for GGMS for future programming on child health? Thank you 60 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: NCASC, MOHP Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. DIRECTOR, NCASC, MOHP, Teku Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Director. Issues to discuss: General /Organisational 1. Government policy on private sector development 2. Level of involvement of NCASC in implementing projects/programs of GGMS and partner organizations 3. What level of contribution is GGMS project making in achieving government health goals a. In FP contraceptive distribution b. Child health service products c. Maternal health service products d. STI/HIV/AIDS 4. Are there any events that NCASC and GGMS or its partners conduct jointly? 5. If yes, what are they? 6. What is the relationship of your division with GGMS and its partners? 7. How are RH, child survival and STI/HIV/AIDS health products of GGMS distributed in the country? 1 2 61 8. Do they follow your target /protocol on commodity distribution, store and disposal system? 9. Your views on public/private partnership in logistics management STI/ HIV and AIDs 10. Would you say that GGMS project is effective in meeting the needs of target groups? a. If yes, how? b. If no, why? 11. What about STI treatment kits (CURE)? 12. Do you think GGMS and its partners are helping to increase access of target groups (Maximum Retail Price) to condom use through social marketing and franchising? 13. Are you aware of GGMS and its partners’ condom promotion activities? a. If yes, what are they? b. Are they effective in reaching the target groups? c. What about costs of condoms? Are they affordable by the lower socio-economic groups 14. What is the share of GGMS and its partners’ in condom distribution in the country? Would you say that it has the largest share – if so what percentage? Or would you say that it needs to increase its share? 15. Which target group in particular is benefiting from condom distribution network? (poor/middle class/rich) Others 16. How is the BCC activity in social marketing promotion? 17. Has CRS promoted dual responsibility for condoms? (STI/HIV and AIDs? Family planning? Male participation?) 18. Preference of branded contraceptives in hard to reach areas, female sex worker , MSM, transport workers and migrant workers 19. CRS markets FP/MCH products but the government distributes most of them free. What would you say about this? 20. What do you think of the capacity of CRS? 21. Which aspects of GGMS activities – social marketing, private sector strengthening, partners' capacity building …. do you think need strengthening for the remaining period? 22. What are your views on the prospects of sustainability of CRS in future? 23. Do you have any particular suggestions to make GGMS project activities more effective for the remaining period? Thank you. 62 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: CRS Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Nepal CRS Company Pvt. Ltd. Fix an appointment Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Managing Director (MD), Section Chiefs of admin/logistics, field op director, marketing (BCC) and finance director Mid level staff (Central) , Managerial level - admin, logistics, training and franchising Issues to discuss: Organisational 1. Can you please explain about the goal of CRS (strive to become a profitable private sector company operating independent of USAID funding) 2. What about the objectives of Contraceptive Retail Sales (reach low socioeconomic group)? 3. Programs implementation strategies (Private Sector Programme, network, experts) 4. Monitoring and evaluation plan/system (Performance Based Monitoring System - PBMS, Performance Monitoring Plan -PMP, Business Plan, Collaboration Plan, BCC Plan/Strategy) 5. Can you give us an insight on co-ordination and co-operation mechanisms between this company and  Government  Local partners 1 2 63 6. What about Capacity Building, Training, Orientation, Behaviour Change Communication (BCC), guidelines (Quality Of Care -QOC), protocols), etc 7. Types of support from Ghar Ghar Ma Swasthya (GGMS) FHI 360 8. Number and types of products and distribution network 9. Coverage – district, urban, rural 10. Results/ Impact, share (different products) of market 11. Innovations regarding brands, prices, promotion of products (Give examples) 12. Achievements 13. Effectiveness 14. What are the products you sell? Name them? 15. Are the products dispatched in time? 16. What is the condition of commodities? (expired, disposal etc) 17. Management of waste products  Percent of products wasted - trend (2010 to 2012)  Management of waste products  Provision of Recycling 18. Anything special about CRS approach (strength) 19. Problems faced (challenges/constraints) 20. Steps taken to solve the problems Strategies 21. What strategies do you have to work with government and how do you report them? 22. Do you ever engage government officials in preparing work plans/programs at national / district level? Technical support 23. Did CRS conduct Training Needs Assessment? (Ask for report) 24. Was the technical support provided by FHI as per your needs assessment? a. If yes, what types (leadership, BCC, social marketing , management etc) of technical assistance were provided by FHI 360 to CRS? Can you name those areas? b. If no, why and what types? 25. Was the technical assistance provided to CRS put into use? a. If yes, how did you apply it? Can you explain it with some examples? b. If not, what were the reasons behind it? 26. Did you take part in any of the above trainings? a. If yes, in which one? b. How useful was that course for you? c. Has it increased your work efficiency? d. Can you give us an example of technical assistance that was useful to you (it can be a case study) 27. Can you explain on the capacity building of the staff /technical support, contents, resource materials, duration, experts/trainer etc? Additional Question for Mid Level (Officers) only 28. Do you get enough support from your supervisors? If Yes, How? If No. Explain? External Assistance 29. Which organisations besides FHI are supporting you externally? (commodity, protocols, etc) 30. How long is the commitment for this program? 64 BCC 31. Can you explain about CRS Behaviour Change Communication (BCC) Programs? 32. Where do you get technical support for BCC programs? Financial Matters 33. What are your funding sources? (Internally generated income? Donor Fund? Trend 2010, 2011 and 2012) Others 34. Is the Quality of Care (QOC) guideline followed and updated? 35. Can you explain how do you plan supply of commodities and human resources? 36. Do you think people prefer CRS’s socially marketed branded products than other products? If yes, please give reasons? 37. How do you plan to sustain after 2015? 38. Would you like to share other matters apart from what we discussed? Thank you. 65 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: FHI 360-GGMS Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. FHI 360-GGMS Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Resident Advisor. Issues to discuss: General/Organisational 1. Program goal 2. Objectives of Ghar Ghar Ma Swasthya (GGMS) (reach low socioeconomic group) 3. Programs implementation mechanism (Private Sector Program , network, experts, matching fund￾sustainability) 4. Capacity Building, resource management, Training (own and outsourced), Orientation, Behaviour Change Communication (BCC), guidelines, etc 5. Achievements (IR1, IR2, IR3 and IR4) 6. Effectiveness 7. Anything special about GGMS approach (strength) 8. Problems faced (constraints) 9. Steps taken to solve the problems 10. What strategies do you have to work with the government and do you report to them? Behaviour Change Communication (BCC) 1 2 66 11. Do you have any result oriented BCC programs for social marketing of CRS? 12. Do you conduct BCC programs for beneficiaries also? 13. Have you provided evidence based BCC programs to support CRS’s activities? If yes, give examples If No, please explain why Technical Assistance 14. Did you conduct needs assessment of CRS before providing technical assistance? 15. How openly did CRS receive your technical assistance? 16. How effective is the GGMS program in providing technical assistance to CRS? 17. What types of technical assistance do you provide (social marketing, capacity building, BCC, waste management and sustainability)? 18. Has CRS operationalized the trainings/ technical assistance you provided? If yes, can you give some examples? If no, why and what were the barriers? 19. Do you think CRS will sustain themselves after your technical assistance? If yes, how for which products? If certain products are not sustainable what are the reasons for it? 20. Up to what level of CRS structure do you provide your technical assistance (Central, district and below district)? 21. Can you explain on Co-ordination and Co-operation between you and CRS? Lessons learned 22. Any lessons learned after implementing GGMS for some 2 years? 23. Recommendations you have for future sustainability of social marketing in Nepal 24. Any other matters you think need to be taken into account? Thank you. 67 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: NFCC Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Nepal Fertility Care Centre (NFCC) Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Managing Director. Issues to discuss: Organisational 1. Do you get any support from Ghar Ghar Ma Swasthya (GGMS) project? a. If yes, what are they? 2. What is your role/responsibility in implementing GGMS? What activities do you carry out? (Training, monitoring, etc) 3. How effective are your activities? Can you explain to us with examples? (percent of clients complaining about side effects declined - give evidence) 4. Achievements of GGMS/CRS/NFCC 5. Can you explain about GGMS in relation to Quality of Care Government /Partnership 6. Do you do partnership with government? a. If yes, how? b. If no, why not? 1 2 68 7. Do you ever engage government officials in preparing work plans/programs at national and /or district level? 8. Whether NFCC foresee the possibility of future government involvement in private sector family planning quality control activities? a. If yes, in what ways? Sustainability / Recommendations 9. Anything special about CRS approach (strength) 10. Problems faced (weakness) 11. Steps taken to solve the problems 12. Lessons learned 13. What are your views on sustaining the results achieved by CRS in years to come? a. Do you think it will sustain? i. If yes, how, can you explain? ii. If no, how can it change its approach to make it sustainable? 14. Any other matters you would like to mention? Thank you. 69 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: FPAN Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. DG, FPAN Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to FPAN MD. Issues to discuss: Organisation/Planning 1. Does USAID support FPAN in any activity? a. If yes, what are those activities? 2. How does your organization manage contraceptive and other reproductive health commodities and services for your centres and districts? (condoms, pills, Depo, Jeeval jal, CDK, STI/ Treatment etc) 3. Do you charge a price for your commodities and services? 4. What systems do you have for making products and services in hard to reach areas? 5. What systems do you have for making products and services in hot zones? 6. What mechanisms do you have in place for disposal and waste management? Percentage of commodities wasted/ expired in a year? 7. Does your organization do any social marketing? a. If yes, how? b. If no, have you heard of social marketing of RH commodities in Nepal? 8. I am sure you know about CRS but do you have any working relation with CRS? 1 2 70 a. If yes, how does it work? b. If no, are you aware of their activities? c. What about GGMS? Are you aware of their activities? 9. CRS markets FP/MCH products but the government distribute most of them free. What would you say about this? 10. What about the quality of products of CRS? 11. Are you aware of BCC activities of CRS? a. If yes, what are they? b. Does your organization and CRS share BCC activities? Sustainability/Lessons Learnt 12. Would you have anything to say about the strengths of CRS 13. Any comments on their constraints 14. Any lessons learnt from CRS 15. CRS/GGMS aims to reach hard to reach areas of the country and they have completed half of 5 year project period. Would you have any suggestions for the remaining period? 16. How do you envision CRS social marketing continuing without external support which will end by end of 2015? 17. Recommendations you have for future sustainability of social marketing in general? 18. Any other matters you think have been left out? Thank you. 71 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: MSI Form No. Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Marie Stopes International Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to Country Director. Issues to discuss: 1. MSI programs 2. Does USAID support MSI in any activity? a. If yes, what are those activities? 3. How does your organization manage contraceptive and other reproductive health commodities and services for your centres and districts? (condoms, pills, Depo, Jeevan jal, CDK, STI/ Treatment etc) 4. Do you charge a price for your commodities and services? If yes, how much do you charge? (condoms, pills, Depo, Jeevan jal, CDK, STI/ Treatment etc) 5. What systems do you have for making products and services available in hard to reach areas? 6. What systems do you have for making products and services in hot zones? 7. What mechanisms do you have in place for disposal and waste management? 8. Percentage of commodities wasted/ expired in a year? 1 2 72 9. Does your organization do any social marketing? i. If yes, how? ii. If no, have you heard of social marketing of RH commodities in Nepal? 10. What mechanisms do you have for ensuring program sustainability? 11. What percentage of your program is sustained now? % self generated income % donor funds 12. I am sure you know about CRS but do you have any working relation with CRS? i. If yes, how does it work? ii. If no, are you aware of their activities? iii. What about GGMS? Are you aware of their activities? 13. CRS markets FP/MCH products but the government distribute most of them free. What would you say about this? 14. What about the quality of products of CRS? 15. Are you aware of BCC activities of CRS? i. If yes, what are they? ii. Does your organization and CRS share BCC activities? 16. Do you use Contraceptive Retail Sale’s branded products? Yes/No i. If yes , please explain ii. If not, why 17. Do you think social marketing and service provision in one place is it effective or not? 18. Do you think CRS can sustain themselves with their activities 19. Anything special about CRS approach(strength) 20. Duplication of efforts 21. Capacity of CRS in terms of human resource and logistic management 22. Any other matters you think have been left out? Thank you. 73 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: PSI Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. Population Services International Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to PSI Country Representative. Issues to discuss: General/organizational 1. Current activities in social marketing in Nepal? 2. If yes, what are your funding sources? 3. Your link with USAID/Ghar Ghar Ma Swasthya (GGMS)? 4. Does USAID support you in any activity? 5. Any noticeable differences between PSI and GGMS approaches to social marketing? 6. Where do you get contraceptive products from? 7. Duplication of efforts? Contraceptive Retail Sales (CRS) 8. Past experience in social marketing in Nepal especially with CRS company 9. Strength /Constraints /Lessons learned from working with CRS in past? 10. How do you envision CRS social marketing continuing without external support? 11. GGMS is half way through in its 5-year project; have you any comments on their work? 12. Do you think CRS will be able to sustain when the project period is over? 13. If yes, how will it do so? 14. If no, why do you say that? 15. Partnership with CRS 16. Recommendations you have for future sustainability of social marketing in general? 17. Any other matters you think have been left out? Thank you. 1 2 74 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GHAR GHAR MA SWATHYA) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: DHO/DPHO/DRHCC DISTRICT MEMBER Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GHAR GHAR MA SWATHYA or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GHAR GHAR MA SWATHYA program is composed of two components: direct implementation of social marketing activities by Nepal CRS Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. DHO/DPHO /DRHCC president Fix an appointment, Explain about the purpose of visit, Hand over a copy of letter given to Evaluation Team from USAID/Nepal to PSI Country Representative. Issues to discuss: Contraceptive Retail Sales (CRS) 1. Are you aware of Contraceptive Retail Sales (CRS) products and their outlets in this district? 2. Is CRS district office a regular participant of DRHCC? 3. Do you think the non traditional outlets promoted by Contraceptive Retail Sales (CRS) are appropriate for promotion of Family Planning /Maternal Child Health/HIV/AIDs products? 4. Are you aware of Behaviour Change Communication (BCC) activities of CRS? a. If yes, what are they? 5. In your opinion what is the market coverage of CRS products in your district? a. What percentage of the market do CRS condoms have in this district? b. Share of Sangini (depo) c. Share of Matri Surakshya Chakki (Safe Motherhood Pills) d. What about share of CDK e. And share of pills and f. Share of ORS g. Share of Piyush/Virex 1 2 75 6. CRS markets FP/MCH products but the government distributes most of them free. What would you say about this? 7. Do you think CRS has helped in improving health status of the people (improving maternal and child health, reducing HIV/AIDs and STIs, prevention of diarrhoea)? 8. About CRS outlets. Do you think CRS outlets are located at appropriate places? 9. What about beneficiaries. Are CRS products reaching beneficiaries? 10. Do people prefer CRS branded products to others? Co-ordination 11. Would you have any comments on the co-ordination mechanism between you and CRS? 12. Do you promote CRS products too when people come to receive services? Strengths/ Weaknesses/Lessons Learnt 13. What do you think are the strengths of CRS? 14. What about their weaknesses? 15. Have you any particular experience with CRS? Recommendations 17. Do you have any particular suggestion for CRS in the district? Thank you. 76 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: CRS AREA MANAGER/FIELD OFFICER/ RURAL FIELD REPRESENTATIVE Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. CRS AREA MANAGER/FIELD OFFICER/ RURAL FIELD REPRESENTATIVE Fix an appointment Issues to discuss 1. Technical training on social marketing (when, how long, where, by whom) a. Years worked in this branch b. Years worked with CRS 2. Your role as Branch Manager/District Manager/Incharge (Area Manager) a. Number of CRS outlets in this district - TO and NTOs 3. Communication / Guidance and support from and to the central level (headquarters), timely response 4. CRS commodities distributed/sold from this branch (data sheet for last three years) 5. Income and expenditure (data sheet for last three years) 6. How much fee do you charge for your services? (Retailers and service providers) - Sangeeni Outlet 1 2 77 - Drug Outlet - NTOs (condoms and ORS) - SEWA Network 7. Motivation for beneficiaries, if yes what types of motivation? (Sangeeni Didi Neighbourhood Program) 8. Types of BCC activities a. Are BCC activities enough for your work? 9. Management of waste products a. Percent of products wasted - trend (2010 to 2012) b. Management of waste products 10. Market share of CRS FP/MCH products in the district 11. Co-ordination with the DPHO/RHCC and other institutions/clubs 12. How is your co-ordination with central office? Do they respond to you on time? Do they listen to your suggestions? 13. Monitoring system 14. Strengths of your program 15. Constraints/problems faced 16. Lessons learnt 17. Sustainable sale of CRS products 18. Any other matters you would like to share 19. Recommendations for the future Thank you. 78 Confidential, information to be used for research purposes only Ghar Ghar Maa Swasthya (GGMS) Project Mid-Term Evaluation SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group STAKEHOLDER INTERVIEW GUIDE: CRS AREA MANAGER/FIELD OFFICER/ RURAL FIELD REPRESENTATIVE/DISTRIBUTOR Time discussion started: Hour: _____ Minute: _____ INTRODUCTION AND CONSENT The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. The interview usually takes around 30 minutes. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Your participation in this study is voluntary and you can choose not to answer any individual question or all of the questions. However, we hope that you will participate in this survey since your views are important. May I proceed with the questions? RESPONDENT: AGREES TO BE INTERVIEWED DOES NOT AGREE TO BE INTERVIEWED END. Name of interviewer: ................................... Date (d/m/y).................................. STOCKIST / DISTRIBUTER Fix an appointment Issues to discuss 1. How many districts do you cover? 2. What CRS products do you distribute? 3. What are the most popular products in this region? 4. How do you determine the price of CRS products? (If the distributor decides) Ask how do you do it? 5. What percentage do you get from CRS and what profit margin do you give to retailers? 6. Do you have anything to say about the packaging of CRS products? 7. How frequently do you distribute CRS products to the retailers? 8. How do you get your products from CRS to your centre? 9. How do you transport CRS products to retailers? 10. What is the percentage of stock out or overstock? 11. What is the percentage of CRS products wasted from your Centre? 12. How do you manage waste products? 13. Do you have anything to suggest for better waste management? Thank you. 1 2 79 Confidential, information to be used for research purposes only GHAR GHAR MAA SWASTHYA (GGMS) PROJECT MID-TERM EVALUATION SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group Interview Guidelines for “Non-Clinical Products” INTRODUCTION AND CONSENT Hello. My name is …………………………and my friend is ………………………….…. We are from the Population, Health and Development (PHD) Group, which is conducting a study for USAID. The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. We are here to find out how this project has been helping people in your community. We would very much appreciate your participation in this study. This information will help USAID to improve its program in the country. The discussion usually takes around half an hour. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Products: CDK, Navajeevan, Piyush, Condoms No. of Outlets: TOs – 1 (Pharmacy) NTOs – 1 (Restaurants, Bus Parks, Bhatti, Music/CD Shops, Tyre Shops, Groceries, paan pasal (Betel Counters), Mini Departmental Stores, clubs etc.) Key Questions  Availability/ Opening/Closing Hours?  Different brands of products sold and price of each brand?  Types of Customers?  Sufficiency in Supply?  Within the last 2 months did you have problems with Stock out/ Overstock?  Expiry Problems?  Profit Margin?  Others…………………………………………………… Observation Points  Product Visibility o (Eye Catching, Clear, Prominent)  IEC Material Display (Point of Purchases) o (Prominently, Satisfyingly)  Others…………………………………………………… Capacity Building Any orientation/training? If yes,  Who?  When?  Was it helpful, do you have any suggestions for improvement? Others, if any. 80 Confidential, information to be used for research purposes only GHAR GHAR MAA SWASTHYA (GGMS) PROJECT MID-TERM EVALUATION SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group Interview Guidelines for “Hot Zones” INTRODUCTION AND CONSENT Hello. My name is …………………………and my friend is ………………………….…. We are from the Population, Health and Development (PHD) Group, which is conducting a study for USAID. The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. We are here to find out how this project has been helping people in your community. We would very much appreciate your participation in this study. This information will help USAID to improve its program in the country. The discussion usually takes around half an hour. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Products: Condoms Areas to Visit: Vulnerable Areas (NTOs-Restaurants, Bus Parks, Bhatti, Music/CD Shops, Tire Shops, Groceries, paan pasal (Betel Counters), Mini Departmental Stores, etc.) No. of Outlets: NTOs – 1 Key Questions 1. Is condom available? Yes. No. 2. If yes, what brands? 3. Brand preferences and price? a. Most preferred brand …………………….. Price (packet of …. pieces)………… b. 2nd most preferred brand ……………… Price (packet of …. pieces)………… c. 3rd most preferred brand……………… Price (packet of …. pieces)………… 4. Types of customers? (Driver, men in general, women/girl, uniform person, etc) 5. Supply Source? 6. Sufficiency in supply? 7. Stock problems? 8. Profit for selling? 9. Price of service and commodities (Charge from client and proportion paid back to CRS) 10. Expiry problems? (what % of stock expires) 11. Others…………………………………………………… Observation Points 12. Product Visibility a. (Eye Catching, Clear, Prominent) 13. IEC/BCC Material Display a. (Prominently, Satisfyingly) 14. Others…………………………………………………… Capacity Building 15. Any orientation/training? If yes, 16. Who? 17. When? Others, if any. 81 Confidential, information to be used for research purposes only GHAR GHAR MAA SWASTHYA (GGMS) PROJECT MID-TERM EVALUATION SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group (CLINICAL PRODUCTS) INTRODUCTION AND CONSENT Hello. My name is …………………………and my friend is ………………………….…. We are from the Population, Health and Development (PHD) Group, which is conducting a study for USAID. The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360-GGMS. We are here to find out how this project has been helping people in your community. We would very much appreciate your participation in this study. This information will help USAID to improve its program in the country. The discussion usually takes around half an hour. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. Products: Sangini, Norplant, Jadelle, Emergency Contraceptive Pills, STI Treatment Kit (CURE), Pills No. of Outlets: TOs – 1 (Pharmacy) Key Questions 1. Availability? 2. Types of Customers? 3. Sufficiency in Supply? 4. Stock Problems? 5. Expiry Problems? 6. Pills/Emergency Contraceptive Only a. Brand Preference? (the highest) b. Training or Counseling 7. Others…………………………………………………… Observation Points 8. Product Visibility: (Eye Catching, Clear, Prominent) 9. IEC Material Display (PoPs) (Prominently, Satisfyingly) Quality of Care (Sangini, Jadelle & Norplant) c. Cleanliness d. Hand-washing facility e. Puncture proof container f. Needle disposal container (basket) g. Separate counseling room h. Clean hand towel i. Proper lighting in counseling room j. Informed choice of poster displays k. Trained manpower availability l. Display of certificate m. Others…………………………………………………… Capacity Building 2. Any orientation/training? If yes, 3. Who? 4. When? 5. Suggestions? Others, if any. 82 Confidential, information to be used for research purposes only GHAR GHAR MAA SWASTHYA (GGMS) PROJECT MID-TERM EVALUATION SOL-367-13-000006 Conducted for USAID/Nepal by Population, Health and Development (PHD) Group (FGD Guideline: Male Group - age 15-49) INTRODUCTION AND CONSENT Two persons conducting it: Moderator and note taker Introduction Hello. My name is …………………………and my friend is ………………………….…. We are from the Population, Health and Development (PHD) Group, which is conducting a study for USAID. The GGMS or Healthy Homes project seeks to strengthen private sector marketing and distribution of health products in rural and hard to reach areas. The GGMS program is composed of two components: direct implementation of social marketing activities by Nepal Contraceptive Retail Sales Company (CRS) and Technical Assistance (TA) provided to CRS by FHI 360. We are here to find out how this project has been helping people in your community. We would very much appreciate your participation in this study. This information will help USAID to improve its program in the country. The discussion usually takes around half an hour. I assure you that your name will not be shared with anyone else and your answers to my questions will be combined with answers from many other people so that no one will know that the answers you give me today belong to you. Your privacy is protected, and I assure that your answers will be kept confidential. My friend will try to take notes of major things that you express. However, it would be difficult to remember everything and take notes that you say during the discussion. For this reason we would like to use a tape recorder. This will capture all your ideas. I hope you would not mind us using tape recorder. To moderator: Make sure that the participants do not have objection against the use of tape recorder. If they give their consent please continue, if not thank them and stop here. IDENTIFICATION 1) Name of District: ………………………………………………… 2) Name of VDC/Municipality: …………………………………………… 3) Ward Number: ………………………………………………………… 4) Name of Village/Tole: …………………………………………………… FGD FACILITATOR (Name)........................................... NOTE TAKER (Name)...................................................... Date FGD conducted (d/m/y)............................................ Time FGD started: Hour: ___ Minute: ___ 83 Before starting our discussion, can each of you please give me some personal information? 1. Background information SN. Name of Participants Age Ethnicity Marital Status Education Occupation Nearest CRS outlet (TO/NTO) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Thank you very much. It is alright. Let us now start our discussion. Topics for discussion Warming-up  Talk about farming season  Talk about the weather It is nice to know more about each of you. We are interested to talk more with you all. That is why we are here in your area. We have a lot of ground to cover, so let us move onto the subject. During discussion if you want to add anything, you can stop me. To FGD moderator: Encourage every participant to respond to every query.  (To FGD moderator: Take what you get and probe :) What about that? Can you explain that? How do you feel about that? Tell me more about that. (Probe why, why, why or why not.) FGD guidelines for male group - Talk about their families, children, age at marriage, when women marry, men marry, how many children they have, what is their social norm, how many children they prefer to have, whether they plan number of children to have, etc - Advantages and disadvantages of planning families - If they plan their families how do they do it - Whether they are aware of CRS company - Any TO or NTO near their place Probe about each of the following product turn by turn with respect to availability, price, preference, use, etc Commodities: Condoms 1. Dhaal Deluxe Condom 2. Panther Condom 3. Commercial condom Clinical contraceptives 1. Sangini Injectable Contraceptive 84 2. Intrauterine Devices (IUDs) 3. Norplant Implants Oral contraceptive pills 1. Nilocon White Oral Contraceptive Pills 2. Sunaulo Gulaf 3. Econ Emergency Contraceptive Pill Maternal and child health products 1. Oral Rehydration Salt 2. Nava Jeevan (ORS) 3. Clean Delivery Kit 4. Sanitary Pads Other items 1. Insecticide treated bed net 2. Virex Chlorine Powder 3. Piyush Water Disinfectant 4. STI Kits Questions 1. Do any of you use any contraceptive methods( condom, pills, sangeeni, jadelle , norplant , IUD etc) If Yes, what method are you using? 2. Where did you get this product/service? (CRS, Government , Private, NGO) 3. Why did you prefer this product from this service outlet? (If CRS outlet is mentioned ask) Was it from NTO(paan pasal.. grocery etc) or TO (pharmacy) Why are you using CRS product/service? (it is good , easy , affordable, attractive packaging, confidentiality, etc) 4. Do your friends/ neighbors also use these methods in your community? If Yes, why? If No, why? 5. How do people in your community take CRS products like the ones you are using? (If the answer is positive , ie. They mentioned liking the product) Why do they like it? Pricing, quality, convenience, availability, attractiveness etc 6. How did you come to know about the product/s you or your communities are using? (Radio, TV, Poster, Newspaper, Friends, Neighbors, Brochures , Sangeeni Didi Char Chimek Program , Government facilities, etc) Thank you very much for your help. We appreciate your concerns and frankness. Namaste. Time interview ended: ……….. hour ………minute 85 ANNEX V LIST OF KEY INFORMANTS (GOVERNMENT OFFICIALS AND OTHER PARTNERS) S.N. Name Designation Organization 1 Dr. Mingnar G. Sherpa DG DOHS 2 Dr. Senendra Raj Uprety Director FHD 3 Dr. Naresh P. K.C. Director LMD 4 Dr. Tara Pokhrel Director CHD 5 Dr. Bal Krishna Subedi Director NCASC 6 KB Rayamajhi Managing Director CRS 7 Rohit Shrestha Administration and Operations CRS 8 Mahesh Pokhrel Field Operations In-Charge CRS 9 Umesh Regmi Marketing Head CRS 10 Jyoti Bhagat Shrestha Finance Manager CRS 11 Indira Chitrakar Executive Secretary CRS 12 Sushma Chitrakar Training Manager CRS 13 Mr. Sanjay Raj Sharma Admin and Finance Officer CRS 14 Niraj Khanal Field Manager CRS 15 Rishikesh Kafle Logistics Manager CRS 16 Sunil KC Field Supervisor, Chitwan CRS 17 Dhruba Paudel Senior Field Officer, Illam CRS 18 Subhash Chapagain Sales Promoter/ RFR CRS 19 Durga Baral RFR CRS 20 Ganga Gurung Sales promoter/RFR CRS 21 Shambhu Singh Administration and Finance NFCC 22 Pema Lakhe Program Director NFCC 23 Binod Khambu GGMS Program Associate NFCC 24 Bal Krishna Subedi Director NCASC 25 Yogesh Gurung IEC Officer PSI 26 Sabitri Training Officer MSI 27 Hare Ram Bhattarai COP FHI 360 28 Dr. Prakash Pant Research Manager FHI 360 29 Moon Pradhan Training Manager FHI 360 30 Dr. Mohammed Daud Senior DHO DPHO, Dhading 31 Bishwo Ram Shrestha Senior DPHO DPHO, Gorkha 32 Basanta Adhikari DPHO DPHO, Dolakha 33 Dr. Bishnu Basnet DHO DHO, Tehrathum 34 Dr. Pawan Jang Rayamajhi DHO DHO, Panchthar 35 Ghana Shyam Paudel Senior Public Health Administrator DPHO, Illam 36 Bhanu Yengnam Senior DPHO DPHO, Dhankuta 37 Rita Bhandari Joshi Senior DPHO DPHO, Dadeldhura 38 Keshar Singh Goddar Senior DPHO DPHO, Chitwan 39 Jeevan Malla Senior DPHO DPHO, Banke 86 ANNEX VI EXPANSION OF NTOS IN 49 GGMS DISTRICTS May - 2010 to July - 2011 Aug 2012 to Jul 2013 May - 2010 to July - 2013 Mountain region Districts Targets Achieve ments % achievemen ts Targets Achieveme nts % achievemen ts Targets Achievem ents % achievemen ts 1 Sankhuwasabha 28 37 132 14 24 171 42 61 145 2 Solukhumbu 25 17 68 12 17 142 37 34 92 3 Taplejung 5 24 480 18 32 178 23 56 243 4 Manang 3 3 100 6 6 100 9 9 100 5 Mustang 2 3 150 8 25 313 10 28 280 6 Dolpa 17 0 - 10 13 130 27 13 48 7 Humla 18 1 6 10 10 100 28 11 39 8 Jumla 20 4 20 20 62 310 40 66 165 9 Kalikot 15 13 87 20 26 130 35 39 111 10 Mugu 5 2 40 12 18 150 17 20 118 11 Bajhang 204 220 108 4 6 150 208 226 109 12 Bajura 19 29 153 14 27 193 33 56 170 13 Darchula 33 56 170 10 11 110 43 67 156 14 Dolakha 16 36 225 12 28 233 28 64 229 15 Rasuwa 3 6 200 16 16 100 19 22 116 16 Sindhupalchok 3 4 133 14 15 107 17 19 112 Total 416 455 109 200 336 168 616 791 128 Hill region Target Achieve ment % achieved Targets Achieveme nts % achievemen ts Targets Achievem ents % achievemen ts 1 Bhojpur 15 10 67 45 48 107 60 58 97 2 Dhankuta 20 8 40 60 65 108 80 73 91 3 Ilam 10 27 270 30 40 133 40 67 168 4 Khotang 15 5 33 45 47 104 60 52 87 5 Okhaldhunga 15 5 33 45 46 102 60 51 85 6 Panchthar 15 29 193 45 66 147 60 95 158 7 Tehrathum 20 26 130 60 66 110 80 92 115 8 Sindhuli 20 34 170 60 57 95 80 91 114 9 Arghakhanchi 20 34 170 60 72 120 80 106 133 10 Gulmi 20 25 125 60 67 112 80 92 115 11 Palpa 15 38 253 45 41 91 60 79 132 12 Baglung 15 50 333 45 48 107 60 98 163 13 Gorkha 15 6 40 45 58 129 60 64 107 14 Lumjung 15 1 7 45 49 109 60 50 83 15 Myagdi 15 24 160 45 46 102 60 70 117 16 Parbat 20 37 185 60 67 112 80 104 130 17 Syangja 20 14 70 60 65 108 80 79 99 18 Tanahun 15 4 27 45 45 100 60 49 82 19 Dailekh 15 28 187 45 44 98 60 72 120 20 Jajarkot 15 7 47 45 33 73 60 40 67 21 Pyuthan 10 47 470 30 43 143 40 90 225 22 Rolpa 15 84 560 45 52 116 60 136 227 23 Rukum 15 96 640 45 87 193 60 183 305 24 Salyan 15 43 287 45 55 122 60 98 163 25 Surkhet 10 58 580 30 57 190 40 115 288 26 Achham 10 16 160 30 45 150 40 61 153 87 27 Baitadi 10 38 380 30 31 103 40 69 173 28 Dadeldhura 20 25 125 60 63 105 80 88 110 29 Doti 5 18 360 15 23 153 20 41 205 30 Dhading 20 20 100 60 65 108 80 85 106 31 Kabhre 15 27 180 45 46 102 60 73 122 32 Nuwakot 15 17 113 45 47 104 60 64 107 33 Ramechhap 10 12 120 30 34 113 40 46 115 Total 500 913 183 1500 1718 115 2,000 2,631 132 M+H 916 1,368 149 1,700 2,054 121 2,616 3,422 131 88 ANNEX VII RURAL FIELD REPRESENTATIVES (RFRS) Table 4.12 Distribution of rural field representative by district SN District District Sex SN District District Sex 1 Taplejung Indra Rai M 26 Parbat None 2 Sankhuwasabha Sabitra Khadka F 27 Baglung None 3 Solukhumbu None 28 Gulmi Bijaya Sapkota M 4 Panchthar Indra Rai M 29 Palpa Bijaya Sapkota M 5 Illam Indra Rai M 30 Arghakhanchi None 6 Dhankuta Sabitra Khadka F 31 Dolpa None 7 Terhathum None 32 Jumla Laxmi Kanta Pande M 8 Bhojpur Sabitra Khadka F 33 Kalikot Laxmi Kanta Pande M 9 Okhaldhunga None 34 Mugu None 10 Khotang None 35 Humla None 11 Dolakha Subash Chapagin M 36 Pyuthan Lauti Kumari Tharu F 12 Sindhupalchok None 37 Rolpa Lauti Kumari Tharu F 13 Rasuwa None 38 Rukum Sunil Thapa M 14 Sindhuli None 39 Salyan Sunil Thapa M 15 Ramechhap Subash Chapagin M 40 Surkhet None 16 Kavrepalanchok None 41 Dailekh Laxmi Kanta Pande M 17 Nuwakot None 42 Jajarkot None 18 Dhading None 43 Achham Chakra Pd. Pant M 19 Manang Surya Bika M 44 Doti Chakra Pd. Pant M 20 Mustang None 45 Dadeldhura None 21 Gorkha None 46 Baitadi Laxmi Dutta Bhatta M 22 Lamjung Surya Bika M 47 Bajura None 23 Tanahu Surya Bika M 48 Bajhang None 24 Syangja None 49 Darchula Laxmi Dutta Bhatta M 25 Myagdi None U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Or USAID/Nepal US Embassy, Kathmandu GPO Box 295, Maharajgunj Kathmandu, Nepal