November 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Circey Trevant, Shareef Ghoneim, and Sabreen Alikhan on behalf of Social Impact, Inc. End of Project Performance Evaluation USAID/Egypt Maternal and Child Health Integrated Program (MCHIP) i Cover Photo: Focus Group Discussion with SMART Community Health Workers in Qena (September 2014). ii USAID / EGYPT MATERNAL AND CHILD HEALTH INTEGRATED PROGRAM (MCHIP) END OF PROJECT PERFORMANCE EVALUATION November 18, 2014 This report will also be made available on the Development Experience Clearinghouse. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii ACKNOWLEDGMENTS This evaluation could not have been possible without the contributions of dedicated USAID, implementing partner, and beneficiary organization staff, whose inputs were instrumental to the evaluation team’s efforts. Firstly, the team expresses its deepest gratitude to Monitoring and Evaluation Specialist and Contracting Officer’s Representative, Ms. Shadia Attia, for her tremendous support and guidance from the evaluation’s inception to its close-out. The team would also like to thank Dr. Shahira Hussein, Senior Program Manager, and Dr. Nabil Alsoufi, Director of the Office of Health, for taking the time to provide critical information to the team, as well as thoughtful feedback on deliverables. Thanks are also due to Ms. Nancy Yassa for her invaluable assistance to the team during the evaluation start-up. The team also wishes to acknowledge the support it received from SMART project staff and staff of the implementing partner, Save the Children. Specifically, the team expresses its appreciation for the generosity of SMART Chief of Party, Dr. Issam El Adawi, and SMART M&E Director, Dr. Farouk Salah, in their willingness to welcome the team, provide critical input to numerous inquiries, supply project data, and facilitate introductions to partner organizations. The team also acknowledges Save the Children home office staff for their timely support in responding to the team’s additional requests. The team would like to convey its bottomless appreciation for the staff of each CDA sampled for this evaluation, as well as the mothers, husbands, grandmothers, and community health workers who made themselves available to participate in focus group discussions. The team is especially appreciative of the flexibility of CDA staff to accommodate the evaluators into their busy schedules. These staff and beneficiaries consistently greeted the team with warmth and enthusiasm, making available time and resources to provide the information upon which this evaluation is based. Last, but not least, the team would like to thank James Fremming and Caroline Perkinson for their home office technical and administrative support throughout the duration of this work. Special thanks are also extended to the team’s local logistics coordinator, Ahmed Elrefai, whose support contributed a critical component of this evaluation effort. iv CONTENTS ACKNOWLEDGMENTS .......................................................................................................................................................... iii ACRONYMS ................................................................................................................................................................................ vi MAP OF PROJECT AREA .......................................................................................................................................................... i EXECUTIVE SUMMARY ............................................................................................................................................................. i EVALUATION PURPOSE & EVALUATION QUESTIONS .............................................................................................. 1 Evaluation Purpose .................................................................................................................................................................. 1 Evaluation Questions .............................................................................................................................................................. 1 PROJECT BACKGROUND ...................................................................................................................................................... 2 Program Description .............................................................................................................................................................. 2 EVALUATION METHODS & LIMITATIONS ...................................................................................................................... 5 Methodology ............................................................................................................................................................................. 5 Limitations ................................................................................................................................................................................. 6 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ................................................................................................ 8 Evaluation Question 1: What are the most significant factors that enabled or constrained the implementers’ ability to achieve the desired project outcomes? ................................................................................ 8 Findings .................................................................................................................................................................................. 8 Conclusions ........................................................................................................................................................................ 14 Evaluation Question 2: What evidence exists to substantiate suppositions that the project had any positive effects? ...................................................................................................................................................................................... 15 Findings ................................................................................................................................................................................ 15 Conclusions ........................................................................................................................................................................ 27 Evaluation Question 3: What are the most significant determinants that influence the likely sustainability of project benefits in target populations? ............................................................................................................................. 29 Findings ................................................................................................................................................................................ 29 Conclusions ........................................................................................................................................................................ 30 Evaluation Question 4: To what extent did the project use its resources efficiently? ......................................... 31 Findings ................................................................................................................................................................................ 31 Conclusions ........................................................................................................................................................................ 32 Evaluation Question 5: Considering the project’s design constraints and possible counterfactual alternatives, to what extent does the constellation of project interventions represent the most effective way of reducing stunting? .................................................................................................................................................... 33 Findings ................................................................................................................................................................................ 33 Conclusions ........................................................................................................................................................................ 34 LESSONS LEARNED ................................................................................................................................................................. 35 RECOMMENDATIONS ........................................................................................................................................................... 37 ANNEXES .................................................................................................................................................................................... 39 v vi ACRONYMS ABWA Asyut Business Women’s Association ANC Antenatal Care BCC Behavior Change Communication CDA Community Development Association CHW Community Health Worker COP Chief of Party CPR Contraceptive Prevalence Rate EBF Exclusive Breastfeeding EDHS Egypt Demographic and Health Survey ENC Essential Newborn Care ET Evaluation Team FP Family Planning HBB Helping Babies Breathe IEC Information, Education, and Communication IFA Iron and Folic Acid IYCF Infant and Young Child Feeding KMC Kangaroo Mother Care LBW Low Birth Weight LCDA Local Community Development Association LE Lower Egypt M&E Monitoring and Evaluation MCHIP Maternal and Child Health Integrated Program MiL Mother-in-law MNCH-FP Maternal, Newborn, and Child Health-Family Planning MOHP Ministry of Health and Population MOSS Ministry of Social Solidarity MOU Memorandum of Understanding NGO Nongovernmental Organization ORS Oral Rehydration Solution PNC Postnatal Care PPFP Postpartum Family Planning SBCC Social Behavior Change Communications SBC Social Behavior Change SMART Community-based Initiatives for a Healthy Life Takamol Integrated Reproductive Health Services project TIPS Trials of Improved Practices TOT Training of Trainers UCDA Umbrella Community Development Association UE Upper Egypt USAID United States Agency for International Development U2 Children Under Two Years of Age WHC Warm Hug Care i MAP OF PROJECT AREA i EXECUTIVE SUMMARY INTRODUCTION AND BACKGROUND USAID’s Egypt Mission contracted Social Impact to conduct a final evaluation of the Community-based Initiatives for a Better Life (SMART) Project which was implemented by consortium led by Save the Children, in partnership with Jhpiego, PATH, and JSI from October 2011 through December 2013, with a no-cost extension until June 2014. The overarching objective of SMART was to implement effective health communication strategies across target areas through proven, life-saving community interventions. SMART was funded by USAID’s Egypt Mission with $10,400,000.00 within the framework of the MCHIP (Cooperative Agreement Number GHS-A-00-08-00002-00); in order to address Egypt’s critically under-performing nutrition and child health indicators (2008 Egypt Demographic and Health Survey (EDHS) revealed an increase in child malnutrition from 2005, with 29% of under-fives stunted (<- 2 HAZ), and13% severely stunted (<-3 HAZ), SMART was designed to focus just on stunting. Building on community-based outreach activities implemented under previous USAID health and nutrition programs, SMART worked through and with local non-governmental organizations (NGOs) to complement and create demand for public sector health services, and increase adoption of key healthy practices. With a focus on stunting, SMART sought to build capacity to engage local organizations to target improve communities’ abilities to utilize and sustain community-based strategies to improve maternal and child health, neonatal health, family planning and nutrition. SMART’s main activities were: - Community health outreach and communication activities, which aimed at increasing target families’ awareness and knowledge about the importance of adopting proper MNCH-FP-Nutrition behaviors; - Nutrition education and rehabilitation classes, which aimed to address maternal malnutrition and childhood malnutrition and stunting (6-24 months); - Home-based neonatal care through a package of simple interventions using trained Community Health Workers (CHW) to counsel mothers for newborn care, including resuscitation, cord care, kangaroo mother care for low birth weight, and initiation and exclusiveness of breastfeeding; - Build capacity of local CDAs1 to respond to health needs with focus on sustainability. SMART identified one “umbrella” CDA (UCDA) per district to serve as a mentor organization to smaller, local CDAs and trained them in financial, administrative and program management systems;2 - An in-depth study to understand the underlying issues for the increased stunting levels in Egypt. SMART focused geographically in six priority governorates, including Qalyubia and Sharqia in Lower Egypt; and Asyut, Beni-Suef, Qena and Sohag in Upper Egypt. In those six governorates, SMART was implemented in 12 districts and 102 villages, targeting 438,539 women of reproductive age (WRA), 62,836 pregnant women, 97,582 children (< 3 years) and 54,640 newborns. 1 Community Development Associations (CDAs), are Grassroots/Community-based NGOs, commonly manage and fund social welfare activities that serve their community members. 2 CDA Training included program planning, monitoring and evaluation, situational analysis, needs assessment, and fund raising. ii EVALUATION PURPOSE, QUESTIONS, AND METHODOLOGY The purpose of this utilization-focused performance evaluation is three-fold: (1) To evaluate the extent to which the SMART project achieved program objectives; (2) To identify lessons learned from project implementation and local stakeholder relationship-building in order to inform USAID future investments; and (3) To assess the sustainability of SMART interventions at the community level. The evaluation specifically addressed the following questions: 1. What are the most significant factors that enabled or constrained the implementers’ ability to achieve the desired project outcomes? 2. What evidence exists to substantiate suppositions that the project had any positive effects? 3. What are the most significant determinants that influence the likely sustainability of project benefits in target populations? 4. To what extent did the project use its resources efficiently? and 5. Considering the project’s design constraints and possible counterfactual alternatives, to what extent does the constellation of project interventions represent the most effective way of reducing stunting? A four-member team of two evaluators, a technical advisor, and a logistician conducted the evaluation from August 26 to October 15, 2014, which entailed both in-country and remote work. There were three elements of the evaluation methodology: desk review of pre-existing data and project documents, in-country primary data collection through in-person interviews and discussions, and additional primary data collection through e-mail and telephone questionnaires. In the field, the evaluation team visited eight districts and twelve villages within the six governorates. The evaluation team selected key informants from a list of names provided by USAID, as well as using snowball sampling from referrals received during KIIs. Due to the availability of local CDAs and/or logistical/safety reasons, the team visited “replacement” CDAs, as directed by UCDAs, when necessary. Local CDAs assisted the evaluation team by recruiting participants for focus group discussions; the team conducted separate focus group discussions (FGDs) with SMART-recruited CHWs, community beneficiaries and respective Umbrella and Local CDAs’ board and staff members. During its work, the evaluation team encountered the following limitations: ● SMART activities officially ended two months prior to the commencement of the evaluation, which limited the team’s ability to access some project staff and data. ● As a result of missing data, the evaluation team was unable to independently verify some data found in project reports (instances of incomplete or missing are explicitly stated throughout the report.) ● Due to this evaluation’s reliance on qualitative inputs where quantitative data is unavailable, the generalizability of conclusions drawn from qualitative evidence is nonetheless limited in scope. KEY FINDINGS AND CONCLUSIONS Despite the difficulties that faced SMART in coordinating with the Ministry of Health and Population (MOHP), SMART was able to achieve its objectives by forging strategic partnerships with local CDAs and MOHP officials, as well as with relevant national associations, syndicates and universities. Engaging through a network of local entities created a solid implementing web that provided critical antenatal and nutrition care to beneficiaries in need, upgraded skills of local service providers in MNCH, FP and nutrition, and improved maternal and child nutrition and health in target communities. iii Because of SMART’s behavior change communication (BCC) efforts, there were tangible behavioral changes were observed not only at the household level, but also in the community, especially with respect to exclusive breastfeeding, improved household nutrition practices, pregnant women seeking regular antenatal care (ANC), and shifting family attitudes toward maternal health. CHWs and staff members of CDAs reported that SMART’s capacity building not only increased their technical knowledge and professional skills, but also their abilities to be effective community leaders and advocates for women’s health and education. Despite delayed implementation of the operational research the TIPS studies generated valuable data on nutrition-related behaviors that was directly incorporated into SMART BCC materials. After SMART ended, the majority of CDAs received grants from domestic and foreign donors to continue some or all of SMART’s original activities, enabling SMART’s sustainability. SMART effectively raised the supply (using mobile clinics) and demand (through raised awareness) of MNCH-Nutrition services. In the short term, SMART’s health messaging has sustained impact within households, as mothers continue to practice improved health behaviors for children, and at the community level, where community attitudes and practices toward nutrition and gender parity are beginning to improve. SMART maximized project efficiency by leveraging existing community systems and entities, as well as by updating and using health materials from previous health projects. While the evaluation team was not provided with detailed budget data, cost estimates demonstrate that, with a compensation totaling approximately 5% of overall SMART funds, the CHW component alone achieved a significant portion of SMART’s observed results. SMART’s implementation period was too brief to evaluate its effectiveness, especially with regard to stunting, neonatal and maternal health and/or mortality; however, respondents cited observable changes in health-seeking behaviors and child feeding practices. SMART suffered from poor M&E, data management, and documentation, making it difficult to determine the extent of SMART’s progress against targets. RECOMMENDATIONS 1. Establish a hybrid community based health model, to capitalize on strengths and resources of the civil society, government and private sector. 2. Target adolescents with messages about MNCH/FP/Nutrition before marriage and first pregnancy in order to ensure early understanding and practice of good health behaviors. 3. Increase resources (relevant educational materials and events) and remuneration for CHWs, both for their professional development and sustained commitment to high performance. 4. Combine MCH and socioeconomic development programs, to ameliorate effects of poverty on the health status of women and children. 5. Consider adding male CHWs to penetrate conservative communities more effectively and enable family-level changes in attitudes and health-seeking practices. 6. Partner with rural health units to operate mobile clinics in order to improve health iv service accessibility and systematize knowledge-sharing between government and private providers. 7. Maximize coverage and interaction of health messages through social media such as mobile phone SMS and Facebook. 8. Leverage the use of information communication technology (ICT) so that future M&E systems can use electronic data collection and reporting systems to reduce human error and secure data compilation and management for decision making. 9. Develop skills of medical doctors and nurses early on in their careers (e.g. before being licensed) to limit new graduates’ exposure to misinformation and poor medical practices. 1 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this utilization-focused performance evaluation is three-fold: (1) To evaluate the extent to which the SMART project achieved program objectives; (2) To identify lessons learned from project implementation and local stakeholder relationship-building in order to inform USAID future investments; and (3) To assess the sustainability of SMART interventions at the community level The audience of this performance evaluation is USAID/Egypt Mission, specifically the health team, the Global Health Bureau, the Middle East Bureau, global MCHIP and local stakeholders including program beneficiaries in both Upper and Lower Egypt regions. The evaluation’s action-oriented recommendations and lessons learned are intended to be used by the primary evaluation users to understand the extent of the project’s contribution to improved MNCH outcomes in Egypt, and to incorporate lessons learned in order to maximize the effectiveness of future MNCH-FP-Nutrition programming at the national and regional level. USAID/Egypt will use findings from this evaluation to inform the design of future projects. The evaluation report will be disseminated widely among relevant stakeholders and project beneficiaries as well as submitted to the USAID Development Experience Clearinghouse (DEC). EVALUATION QUESTIONS In pursuit of the complementary objectives, the evaluation specifically addressed the following questions: 1. Factors Affecting Results: What are the most significant factors that enabled or constrained the implementers’ ability to achieve the desired project outcomes? 2. Evidence of Results: What evidence exists to substantiate suppositions that the project had any positive effects? 3. Sustainability: What are the most significant determinants that influence the likely sustainability of project benefits in target populations? 4. Efficiency: To what extent did the project use its resources efficiently? 5. Effectiveness: Considering the project’s design constraints and possible counterfactual alternatives, to what extent does the constellation of project interventions represent the most effective way of reducing stunting? 2 PROJECT BACKGROUND PROGRAM DESCRIPTION The 2008 Egypt Demographic and Health Survey (EDHS) revealed an increase in child malnutrition from 2005, with 29% of under-fives stunted (<-2 HAZ), and13% severely stunted (<-3 HAZ). Despite a marginal decrease in the neonatal mortality rate (NMR) (18.6 per 1,000 live births in 2005 to 16.3 per 1,000 live births in 2008), indicators of poor neonatal health remain obstinately high. To address Egypt’s critically under-performing nutrition and child health indicators, MCHIP/Egypt’s SMART project approach built on previous community outreach activities that were implemented under the Communication for Healthy Living (CHL) and Integrated Reproductive Health Services (Takamol) projects, which ended in 2010 and 2011, respectively. Each program worked through and with local non-governmental organizations (NGOs) to complement and create demand for public sector health services, and increase adoption of key healthy practices. Focused on combatting stunting, SMART used a similar approach of engaging local organizations to ensure that communities are able to utilize community-based strategies and approaches to improve maternal and child health, neonatal health, and nutrition. USAID/Egypt implemented the SMART project through field support funding to the MCHIP Cooperative Agreement Number GHS-A-00-08-00002-00 from October 2011 through December 2013, with a no-cost extension until June 2014. Total SMART funding was $10,400,000 over a 33 month period, and was implemented by a consortium of entities led by Save the Children, including Jhpiego, PATH, and JSI. The overarching objective of SMART was to implement effective health communication strategies across target areas of Egypt through proven, life-saving community interventions focused on improving childhood nutrition and decreasing neonatal mortality. Figure 1. SMART Results Framework 3 Specifically, SMART activities included: 1. Community health outreach and communications activities. Community health outreach and communications activities were designed to increase families’ and households’ awareness and knowledge of maternal and neonatal risk factors, emphasize the practice of key maternal, neonatal and child health behaviors and appropriate care-seeking practices thereby creating demand for related services as in antenatal care classes, post-partum and post-natal care, birth-spacing, etc. 2. Nutrition education and rehabilitation program. A nutrition education and rehabilitation program at the community level was implemented to address maternal nutrition and childhood malnutrition and stunting (6-24 months). The package of activities included identifying malnourished children in the community, providing food and nutrition education, home visits by volunteers to follow up children’s nutritional status, medical checkups and laboratory investigations for children to detect parasitic infestation and training volunteers to implement program’s activities. 3. Home-based neonatal care through a package of simple interventions. Promote a home-based neonatal care through a package of simple interventions that can save newborn lives, especially those delivered at home. Trained outreach workers were to counsel mothers for newborn care, including resuscitation, cord care, kangaroo mother care for low birth weight, and initiation of breastfeeding. The home-based intervention package consisted of antenatal care, iron/folate tablet distribution, safe delivery, postnatal care and family planning use. 4. Test new approaches in community health outreach and communication. Introduce and test new approaches in community health outreach and communication that will improve neonatal survival and young child nutrition. 5. Build capacity of local CDAs3 to respond to health needs with focus on sustainability. Using the following selection criteria, SMART sought to identify and assess organizational and technical capacity of potential CDA partners in the targeted governorates. SMART identified one “umbrella” CDA (UCDA) per district to serve as a mentor organization to smaller, local CDAs. SMART provided CDAs with training in areas including financial management, program planning, monitoring and evaluation, situational analysis, needs assessment, and fund raising. 6. Implement an in-depth study to understand the underlying issues for the increased stunting levels in Lower Egypt. SMART focused geographically in six priority governorates, including Qalyubia and Sharqia in Lower Egypt; and Asyut, Beni-Suef, Qena and Sohag in Upper Egypt. These governorates were selected on the basis of the following factors: 1) high malnutrition rates per 2008 EDHS; 2) high poverty rates; and 3) under-performing health indicators (e.g. contraceptive prevalence rate, antenatal care, birth spacing, teenage pregnancy, deliveries by skilled providers, and availability of antenatal care and maternal mortality). SMART activities were designed to engage approximately 8% (1.96 million of the 23.8 million) of the people living in the targeted six governorates (12 districts and 101 villages). The population in the 3 Community Development Associations (CDAs), as NGOs are known in Egypt, commonly manage and fund social welfare activities that serve their community members. 4 target area was projected to have: 438,539 women of reproductive age (WRA), 62,836 pregnant women, 97,582 children (< 3 years) and 54,640 newborns.4 4 As the report elaborates later, the evaluation team understood from the SMART PMP, staff interviews, quarterly and final reports, and SMART management, staff and consultants/experts, that the project’s indicator targets were not be derived from calculations of realistic project reach. Furthermore, not all data collected from the project was consistent with PMP indicators, such that some indicators could not be matched against targets in the PMP. These inconsistencies and information gaps were among the challenges the evaluation team faced in drawing conclusions on SMART’s observable achievements. 5 EVALUATION METHODS & LIMITATIONS METHODOLOGY A detailed evaluation methodology, including strategies and limitations, may be found in Annex II. Per USAID guidelines, a performance evaluation focuses on descriptive and normative questions, such as what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period), how it is being implemented, how it is perceived and valued, whether expected results are occurring; and other questions that are pertinent to program, design, management and operational decision making. Due to the lack of valid control groups of local organizations, a non-experimental performance evaluation design was employed for this evaluation. A four-member team of evaluators, a technical advisor, and a logistician conducted evaluation fieldwork from August 26 to October 15, 2014, which entailed both in-country and remote work. There were three elements of the evaluation methodology: desk review of pre-existing data and project documents, in-country primary data collection through in-person interviews and discussions, and additional primary data collection through e-mail and telephone questionnaires. Desk Review: The evaluation commenced with a desk review of key project documents provided by USAID, including work plans, quarterly reports, performance management plan, and others (see Annex II for complete list of documents reviewed). While in-country, the evaluation team conducted additional desk review of documents and data obtained during field work. Review of Pre-existing Data: The evaluation team also reviewed project data obtained from the SMART monitoring and evaluation (M&E) system, including baseline and endline performance data and service data reported by CDAs. Primary Data Collection: The evaluation team collected primary data through key informant interviews (KIIs), focus group discussions (FGDs), semi-structured telephone questionnaires administered to doctors and nurses, and structured e-mail questionnaires administered to CDAs. FGDs were conducted during site visits to UCDAs and local CDAs; site visits encompassed FGDs with CDA staff, community health workers (CHWs), and community beneficiaries (women of reproductive age, husbands, and grandmothers). The evaluation team visited all six governorates in which SMART was implemented, and within those, selected eight districts and twelve of the 101 villages where SMART activities took place. The evaluation team selected key informants from a list of names provided by USAID, as well as using snowball sampling from referrals received during KIIs. From each project governorate, USAID selected one district for the evaluation team to conduct site visits. The team randomly selected local CDAs within each district to visit, and coordinated with each district’s representative UCDA to arrange meetings with sampled CDAs. Due to the availability of local CDAs and/or logistical/safety reasons, the team visited “replacement” CDAs, as directed by UCDAs, when necessary. Thus, the team primarily employed convenience sampling in conducting site visits for data collection. Local CDAs assisted the evaluation team by recruiting participants for focus group discussions; the team conducted separate FGDs with SMART-recruited CHWs and community beneficiaries. Table 1 below provides a breakdown of the qualitative data sample by method of data collection and respondent category. 6 Table 1. Qualitative Data Collection Sample Method of Data Collection Description of Respondent Group Sample Size Key Informants National partners (Associations) 5 USAID/Egypt Officials 4 SMART staff: (Chief of Party, Team Leaders, Deputy Chief of Party, M&E Officer, Grant Manager) 9 SMART Contractors 3 Government of Egypt: Ministry of Health and Population (MOHP), Ministry of Solidarity 2 Total no. of Key Informants 24 Focus Groups UCDA board and SMART-related staff members 7 LCDA board and SMART-related staff members 22 CHWs 104 Mothers of Children 101 Grandmothers 6 Husbands 7 Doctors 5 Nurses 8 Pharmacists 3 Total no. of Focus Group Participants 263 Semi-structured Questionnaire Beneficiaries of SMART Training (27 Doctors, 8 Nurses and 7 Interns) 42 UCDAs 5 Total no. of Questionnaire Respondents 47 The evaluation team reviewed secondary project data alongside results of primary data collection in order to triangulate findings. The team used project reports to analyze trends in service use and demand at the village level. The team recorded summary notes of KIIs and FGDs and conducted qualitative coding using the online software package Dedoose. For interviews and focus groups conducted in Arabic, Arabic-speaking team members translated summary notes to English prior to qualitative coding. The team compiled quantitative results from the questionnaire administered to doctors and nurses, which explored the extent to which information from SMART trainings is valued and incorporated into current job functions. Similarly, the team compiled and summarized results of the questionnaire administered to UCDAs regarding characteristics of assigned local CDAs. Data from the questionnaires was processed using Microsoft Excel to create summary frequency tables (see Annex IX for phone interview summary data). LIMITATIONS The evaluation team notes the following limitations: ● SMART activities officially ended two months prior to the commencement of the evaluation, which limited the team’s ability to access some project staff and data. ● The evaluation team requested data that was either provided late in the fieldwork phase (e.g. SMART monitoring data), or not at all (e.g. SMART expenditures by activity, complete rosters of training participants). Data received late limited the amount of time the evaluation team could dedicate to analyzing results. As a result of missing data, the evaluation team was unable to independently verify some data found in project reports. Instances where data is incomplete or missing are explicitly stated throughout the report. 7 ● In the absence of a random sample selection, the generalizability of findings from qualitative interviews is limited in scope. Additionally, qualitative interviews are inherently subject to specific biases: recall bias, wherein responses are affected by respondents’ ability to recall past experiences; and the Hawthorne effect, whereby respondents aware of the study modify their behavior. The evaluation team used best practices in evaluation to minimize bias and subjectivity to enhance the rigor of the evaluation results. Due to this evaluation’s reliance on qualitative inputs where quantitative data is unavailable, the generalizability of conclusions drawn from qualitative evidence is nonetheless limited in scope. 8 FINDINGS, CONCLUSIONS & RECOMMENDATIONS EVALUATION QUESTION 1: WHAT ARE THE MOST SIGNIFICANT FACTORS THAT ENABLED OR CONSTRAINED THE IMPLEMENTERS’ ABILITY TO ACHIEVE THE DESIRED PROJECT OUTCOMES? FINDINGS Activities under the SMART project have taken place in a challenging environment, amidst changes in government, social unrest and security concerns. Despite these intermittent disruptions, CDAs continued to provide services to improve MNCH and nutrition outcomes of women and their children in SMART communities. This section will analyze both positive and negative aspects of the external and internal factors that affected the results.5 Internal Factors ● Trust in CHWs: One of the most frequently cited enabling factor of SMART results was the use of CHWs as community behavior change agents. CDA staff and beneficiaries overwhelmingly noted that the selection of CHWs from target communities was effective in gaining people’s trust. Mothers and CHWs reported that at the beginning of the project, many husbands and mothers-in-law were reluctant to give permission for mothers to attend health education classes outside of the home; however, following a few home visits from SMART CHWs, many relatives reported granting such permission. CHWs, beneficiaries (mothers), and doctors interviewed reported anecdotal evidence of CHW efficacy seen through increased demand for antenatal services at MOHP rural health units, as well as services provided through SMART-funded mobile clinics. CDAs and CHWs also reported rising participation in nutrition classes over the course of SMART – evidence of the effectiveness of CHWs in encouraging beneficiaries to seek healthier nutrition behaviors. ● Availability and quality of services: The majority of respondents reported that SMART not only met a need in their respective communities, but that SMART effectively met the increased service demand with increase service availability. Specifically, SMART-funded mobile clinics were noted among beneficiaries for their popularity (increasing number of patients over the life of the project, as reported by CDAs), reliability (alternatively, community beneficiaries consistently expressed reservations about antenatal care at MOHP health units), and service quality. Beneficiaries reported receiving high quality care from CDA clinics (both mobile and permanent), expressing the desire for mobile clinics to become permanent community fixtures. Beneficiaries also reported receiving consistent health messaging from mobile clinic health providers and CHWs, which respondents noted was a factor in encouraging behavior change. Some beneficiaries (n<10) reported receiving errant health messages from doctors or nurses at MOHP rural health units, which they recognized as incorrect due to the efficacy of CHW home-based services and visits to mobile clinics. ● Further, beneficiaries consistently described SMART’s nutrition classes and cooking 5 Internal factors refer to those that are intrinsic to the project (e.g. administrative, design, financial) and external are those beyond the project’s control (e.g. political environment). 9 demonstrations as a highly useful intervention. CHWs at every group discussion reported instances where grandmothers would bring their grandchildren to nutrition classes or mobile clinics in the event that the mother was ill or otherwise unable to attend, indicating a high perceived quality of SMART services by family members originally resistant to behavior change messaging. ● Project management: SMART operated a decentralized field implementation structure in which SMART Team Leaders housed directly within UCDAs facilitated constant capacity building while coordinating closely with UCDA staff to ensure the completion of project activities. CDA staff consistently reported that SMART staff’s close proximity to the field allowed for prompt intervention when needed to address issues that arose during implementation of the project activities. ● Short implementation period: On the negative side, the most commonly cited factor affecting results was SMART’s short duration. All CDAs interviewed reported that the constrained implementation period (an average engagement of 15 months per CDA), and consequently small field budget, resulted in CDAs reaching fewer beneficiaries than desired. While each district exceeded its target number of beneficiaries reached, the majority of CDAs reported a desire to expand their respective catchment areas – a desire that was constrained by time and funding. Considering that CDAs first received capacity building training and took time to recruit and train CHWs, actual field implementation of health message dissemination and service delivery was further limited. Even though beneficiaries reported that they still practice some key behaviors (e.g. exclusive breastfeeding (EBF), hygiene, and seeking care for sick children) at the time of the final evaluation, the project’s short duration could not definitively demonstrate significant changes in key outcome indicators. ● Weak information management: SMART’s information management at large was a general weakness that CDAs reported. While the project made an effort to harmonize the project database and data collection forms, quarterly reports did not reflect routine data analysis showing progress of the outcome indicators against targets. Further, the evaluation team did not find documentation of service provision analysis conducted by CDAs on data collected by CHWs during their daily home visits and monthly activities. One quarterly report alludes to a “large volume” of data tracking over 60,000 women and children, and the challenges associated with managing such a “huge amount of information.” However, the discussion does not address how the project used the data for decision-making. CDAs were instructed to send all field data to Cairo for analysis, yet CDAs reported not receiving feedback on said analysis in any way that would allow them to track their own progress toward achieving project objectives. Similarly, one informant (former senior-level SMART staff) reported that once the target of reaching 7,000 beneficiaries (women and children receiving MNH-FP services from mobile teams organized by CDAs) was met, the project ceased to continue documenting further progress on that indicator.6 Incomplete documentation of results limits the extent to which SMART achievements can be considered credible, which, in turn, affects data uptake for use in future program design. External Factors 6 This comment is related to beneficiaries receiving care from mobile clinics. The evaluation team notes the possibility of double-counting individuals visiting mobile clinics over the life of the project, as the referenced indicator measured the number of visits to mobile clinics, as opposed to the number of unique individuals visiting mobile clinics. It was impossible to verify how the 38,000 recorded beneficiaries were counted in the absence of the records. Senior SMART staff also reported that the mobile clinic activity was under-budgeted; SMART targeted 7,000 beneficiaries over the life of the project, which was already met within the first year of implementation. By the end of the project, SMART reported serving 38,000 beneficiaries, many of whom were serviced by CDAs who continued mobile clinic services using their own funding or through partnerships with MOHP. 10 ● Partnership modality: The majority of key respondents noted that without the cooperation of and partnerships with local entities (CDAs, MOHP health units, National Associations, Community Committees (Mandaras), SMART would not have been able to penetrate communities as effectively as it did. CDAs were particularly instrumental in taking initiative toward expanding SMART’s reach through innovative partnerships: CDAs in Upper Egypt reported forming a joint network to formalize knowledge- and experience-sharing, and the majority of Upper Egypt CDAs visited reported either using Mandaras as platforms for spreading SMART messaging (particularly gender-related), or forming partnerships with local MOHP health units to sustain or expand mobile clinics in their respective communities. Cooperation between CDAs and national and governorate level MOHP officials led to Memorandums of Understanding (MOUs) that enabled greater community access to MOHP mobile clinic vans in Sohag and Qena. These fully equipped MOHP owned vans conducted outreach activities to marginalized communities that would have otherwise remained untouched by SMART. ● SMART project staff and technical consultants reported that the utilization of national associations was also instrumental in allowing SMART to tap into broader networks beyond direct beneficiaries to reach service providers with training. While the project did not directly measure provider service quality post-training, the associations did conduct pre- and post-tests to measure changes in service provider knowledge levels. Increased clinical knowledge does not necessarily translate into improved clinical practice; therefore, it is difficult to identify a correlation between the SMART training and improvement in quality of services post-training. At the project sites, however, anecdotal evidence revealed that demand for prenatal services was sustained at a high level, as beneficiaries reported a strong preference for the quality of services at mobile clinics as opposed to public clinics. According to beneficiary focus group respondents, SMART's mobile clinics provided specialized medical check-ups, consultations and prescriptions by pediatricians and obstetricians/gynecologists: specialists most relevant to the target beneficiaries. Beneficiaries reported that public clinics provide primary health care services from a "generalist" with occasional visits from MOHP specialists who operate within a tight schedule that only allows for a quick examination of patients before referring them to hospitals outside the village. Additional comments from beneficiaries confirmed low levels of public trust in the quality of care at rural health units, citing such reasons as poor incentives for government employees to perform well. ● Political Transition: Egypt’s recent governmental transitions inevitably impacted SMART’s start-up, as reported by key informants. Difficulty engaging directly with MOHP at the project’s commencement caused delays in activity implementation, though this hurdle was later overcome. Respondents among CDA staff reported that the political situation initially hindered the acceptance of SMART messaging in rural communities. Though CHWs helped to assuage community distrust over the course of the project, respondents conceded that this external factor negatively affected SMART achievements. Furthermore, changes in political administration precipitated a disruption in family planning (FP) commodities, resulting in the stockout of these supplies in some areas. Anecdotal evidence from FGDs with beneficiaries and CHWs confirms that modern FP options were not available for many beneficiaries who sought them, despite improved FP messaging and counseling from CHWs. ● CDA capacity: SMART engaged CDAs of varying capacity levels. While this provided a number of smaller CDAs the opportunity to gain invaluable capacity building (as reported by CDA directly), these local CDAs were admittedly less capable of implementing SMART activities than their larger, more developed UCDA counterparts. While local CDAs contributed significantly to the achievement of SMART results, smaller CDAs had fewer human and financial resources to draw from in sustaining achievements. 11 Table 2. Summary of Key Challenges, Mitigation Strategies, and the Results Key Challenges Strategies/Measures adopted by SMART and/or partner CDAs to overcome them Results/Outcome of SMART' Mitigation Measures At the national level A.1 Initial reluctance of MOHP to cooperate with SMART ● Generally, SMART worked through CDAs and Ministry of Social Solidarity; and at the governorate and district levels depended on the personal relationships between SMART and/or Partner-CDAs’ staff and MOHP officials; and ● Some national level officials and/or heads of health programs or departments were invited to participate in SMART events (conferences and training workshops) ● Most CDAs at the governorate and district levels implemented the project interventions efficiently; and were able to mobilize resources to sustain the interventions beyond SMART; ● Governorate and district level officials with whom SMART collaborated helped mobilize MOHP resources (such as services of the Rural Health Units and MOHP￾run Mobile clinics) to support SMART activities in the communities within their jurisdictions; and ● Some national level officials and/or heads of health programs or departments participated in SMART events (conferences and training workshops); and advocated within the MOHP for using SMART materials and techniques in MOHP facilities and programs. A.2 Political transition and security situation ● Decentralization of SMART’s management; i.e., most decisions made for SMART governorates were made by the Team Leaders assigned to those governorates; thus, enabling them to manage situations locally resulting from political transition activities that made it hard to travel from HQ to the field. ● At the local/field level, the project carried on its activities during the political transition with minimal interruption, since decisions regarding activities and service provision were left at the discretion of the local CDAs unaffected by national-level administrative changes. At the Governorate and/or local level B.1 Local traditions, beliefs and behaviors that contradict SMART messages ● Conveying SMART messages through well-trained, educated young women from the same target communities; and ● The adverse attitudes and behaviors of target beneficiaries were substantially changed (per beneficiaries and CHWs); hence, tangible positive changes in the mothers and children’s health and nutrition were documented and reported. For example, EBF and better care seeking for children, as well as intake of IFA, were 12 ● Raising awareness of, and working with, local leaders who have the confidence of their communities/ constituents and are against adverse local traditions and beliefs. mentioned in almost all the focus group discussions as observed practices that changed for the better in the community as a result of SMART. This is linked to one of the expected results that MNCH-FP-Nutrition behaviors would improve for both men and women. Also, the project reported that “during the final quarter of CHW activities, 82.69% of pregnant women could correctly identify at least 3 danger signs during pregnancy and 78.89% could identify at least 3 danger signs of newborns.”7 B.2 Rumors in target communities/villages in Upper Egypt that the IFA causes sterilization of women. ● Supporting the messages of the CHWs about the IFA tablets by messages from the mobile clinics doctors who have full trust from the local beneficiaries. ● Demand for IFA tablets among pregnant women rose substantially during the project, as reported by CHWs and doctors. One CHW reported a case of IFA tablet stockout at a rural health unit. This health unit used to regularly dispose of expired IFA tablets due to low demand, but since SMART, now experiences stock depletion. B.3 Vast geographic coverage with limited human, material and financial resources, which CDAs often compensated for at organizations’ expense ● Using mobile clinics of other institutions or programs (MOHP, UNICEF and local universities) to reach out to the remote areas and provide medical services to hard-to￾reach beneficiaries. ● Coverage of most SMART-related services exceeded target; CDAs reached areas beyond SMART target areas out to remote villages. B.4 Other than IFA tablets, there were no medications given to sick mothers or children at mobile clinics, which was the major beneficiary complaint about mobile clinics ● Reports from CDAs of mobilizing additional resources (independent of SMART) to make available essential medication for mothers and children, either for free or at subsidized prices, through agreements with local pharmacies or donations from local philanthropists. ● Beneficiaries reported appreciation for additional resources; one of the most frequently cited recommendations was to disburse a wider range of medication at mobile clinics in the future. B.5 MNCH-FP- Nutrition services at risk of low financial sustainability beyond SMART because target beneficiaries unable to pay for essential services at the local level and distrust free services at MOHP ● CDAs, using technical assistance from SMART, applied for grant funding from other donors (mainly Social Fund for Development) to sustain SMART activities. ● Most CDAs, especially in Upper Egypt, report continuing to provide services initiated by SMART after the project’s close using funds obtained from the SFD and other donors. The majority of CDAs engaged in SMART have been able to secure grant funding in some form, which CDAs credit to the quality of SMART capacity building. 7 Quarterly Report FY14 Q1 13 health units. 14 CONCLUSIONS Internal and external factors influencing the results of SMART have been both positive and negative. One of the strongest factors affecting the results of SMART – a behavior change-based project – was the use of CHWs as change agents. Over the course of SMART, beneficiaries (mothers, in particular) became more willing to adhere to health messaging originating from CHWs than clinical providers at MOHP health units. Beneficiaries trusted CHWs recruited from their own communities, which enhanced the effectiveness of CHWs’ work, and ultimately, SMART’s health messaging. As CHWs promoted better care-seeking behaviors and increased demand for services, CDAs were able to meet those needs through mobile clinics that were instrumental in gaining the cooperation of communities. Particularly in Upper Egypt, family trust in CHWs and their messages enabled women and mothers to leave the house more often in pursuit of nutritional information (classes held at CDAs) and prenatal examinations, whereas, prior to SMART, husbands and grandmothers were more resistant to such a level of maternal mobility. In this way, strong community-level trust and activities perceived as highly useful affected gender norms within beneficiary households. Despite the difficulties faced during project start-up with respect to high-level coordination with MOHP, SMART capitalized on lower-level strategic partnerships with CDAs, national associations and universities, and local MOHP health officials to maximize the achievement of results. Partnerships with a network of local entities created a solid implementing web that helped to extend critical antenatal and nutrition care to beneficiaries in need, as well as to upgrade the skills of service providers in MNCH/FP (namely, nutrition), and build the capacity of local CDAs to improve nutrition and child health in their respective communities. Delayed start-up due to political instability limited SMART’s period of implementation, but the project mitigated this constraint by: a) engaging with as many CDAs and partners as were available and qualified at the time, b) by building upon and adapting materials from previous funded projects to quickly develop technical guidelines and training materials for SMART use (thereby saving time and money), and c) building the capacity of CDAs and CHWs through in-depth training. The skill set of the project staff and their personal relationships with local government officials greatly helped as well to establish a positive visibility for SMART activities in the community. 15 EVALUATION QUESTION 2: WHAT EVIDENCE EXISTS TO SUBSTANTIATE SUPPOSITIONS THAT THE PROJECT HAD ANY POSITIVE EFFECTS? FINDINGS In this section, the evaluation team will present evidence of the extent to which each SMART objective and/or expected result was achieved. This evidence will be supported by quantitative and anecdotal data comparing the achievements documented by SMART (from quarterly reports, final report, endline study, and others) with the outcome indicators and targets of SMART’s performance management plan (PMP), as well as triangulating SMART data with qualitative data gathered from the field. Objective 1: Improved access to and quality of key MNCH-FP-Nutrition services by private, community-based providers To achieve objective 1, SMART adopted the following strategies and interventions: 1. Using mobile clinics8 and private lab facilities to deliver specialized medical services (mainly related to Ob/Gyn and Pediatrics) to target beneficiaries. Teams, comprised of a gynecologist, a pediatrician, nurses, and lab technicians, operated once a month at designated spaces of a local CDA (or, in some cases, spaces in the community volunteered by individuals) to offer free medical care to pregnant women and mothers of children under two years of age. 2. Using CHWs from among educated young women in target communities to provide quality health education/awareness, basic care and referrals regarding antenatal, postnatal and neonatal care to target beneficiaries. CHWs worked to effect behavior change in communities initially resistant to messages that contradicted traditional health and nutrition practices. 3. Training of health services providers in MNCH-FP-Nutrition in such new techniques as Helping Babies Breathe (HBB) and Kangaroo Mother Care (KMC)9, as well as updated evidence-based guidelines for treatment of malnourished babies/children and pregnant women. 8 SMART used the term “mobile clinic” to primarily mean a visiting health care provider who operated a makeshift clinic from a non-clinical space, such as at a local CDA or another community space. Traditional mobile clinics (i.e. a clinic on wheels) were also used by SMART, but these were minority instances in which CDAs partnered with local MOHP officials to share the operation of vehicular mobile clinics. 9 These techniques, while not new innovations generally, were approaches newly introduced in Egypt through SMART. Kangaroo Mother Care was renamed Warm Hug Care to fit the Egyptian context. 16 Evidence from SMART’s Documents/Reports: Table 3. Mobile Clinic Services from November 2012 to December 2013 in 101 Villages in 12 Districts Client Total Services PMP Target Pregnant women* 12,079 ANC and Post-Partum Not specified10 Mothers and Children 18,205 Health Services: medical checkups and referrals to lab services by gynecologists and pediatricians; prescription and distribution of IFA tablets, growth monitoring Not specified Total number of beneficiaries 30,284 700011 *First time pregnant and those with previous pregnancies Mobile clinics were key in providing access to quality MNCH-FP-Nutrition services at the community level: SMART documented over 30,000 visits to mobile clinics by mothers and their children (no age specified) over an average of 15-month engagement per CDA. However, SMART does not specify whether these visits correspond to 30,000 individual mothers, or whether mothers made multiple visits to clinics that added to 30,000 total visits. In the absence of rosters or other monitoring records, the evaluation team cannot verify that this indicator refers to 30,000 individual 10 The evaluation team was informed during the key informant interviews with former SMART staff and SMART M&E consultant that the PMP underwent multiple rounds of revision and was approved later than scheduled. The evaluation team found documentation in the quarterly reports that the SMART team received international technical support from consortium members on M&E, but that some input was not addressed. 11 This target was not disaggregated by types of services, and beneficiaries who received them including men, women, children, youth. 17 clients, as is assumed in the SMART project’s end-of-project report. At the time of this report’s writing, records of the exact number of clients, repeated visits, or types of services provided during mobile clinic visits were not made available to the evaluation team. Unfortunately, the mobile clinic data was not segregated by services and types of beneficiaries (1st time pregnant, multiple pregnancies, children and their age groups, etc.). Findings related to the effectiveness and efficiency of services is limited to the fact that the target was met and that services were provided as planned, but the evaluation team cannot independently assess service quality nor demonstrate demand trends over time, as such data is unavailable to the team at the time of the report’s writing. It is documented that mobile clinics were held once monthly in 101 villages. Furthermore, it is clear that the target was underestimated, as several respondents noted that the basis for setting the exact target is unknown. Moreover, one respondent reported that once a target was met, the project stopped documenting further achievements. Table 4: Services provided by the CHWs to target beneficiaries Service Provided by CHWs # of beneficiaries Health Education/Awareness-Raising, during Group Counselling and Monthly Home Visits 149,000 Basic PNC12 within two days of delivery by CHWs 4,241 Average number of home visits per CHW/month13 8 Source: Calculations based on data from SMART’s End-of-Project Report There has been tangible improvement in access to MNCH and nutrition services in target communities, and all five output indicators show improvement from baseline to endline, while only one indicator fell short of its target: postnatal visits within the first seven days after delivery. Figure 2: Output Indicators for Improved access and quality of MNCH-FP-Nutrition Source: SMART PMP 12 To check on the health of the woman and her baby, provide her with information on postpartum and newborn danger signs, how to care for herself and her infant, and to explain the importance of exclusive breastfeeding. 13 This number was calculated by dividing the number of CHWs by the total number of beneficiaries, then dividing by 15 months to arrive at an average number of home visits per month 18 SMART engaged 12 Umbrella CDAs to develop a cadre of service providers to conduct training-of￾trainer (TOT) sessions within their respective organizations. A total of 3,202 doctors, nurses, and CHWs received training in MNCH/FP and nutrition topics promoted by SMART. Table 5. Summary Training Achieved by National Institutions - SMART Partners (2012 to 2014) Training Topic Planned Achieved HBB TOT 45 TOT Physician 80 – physicians TOT HBB Workshops 100 Physicians 100 Nurses 420 Individuals14 170 – Physicians 200 – Nurses & Physicians 200 Physicians and Health Workers Pneumonia TOT, Breastfeeding, Nutrition 25 TOT Physicians 300 Physicians and Health Workers 720 Individuals 25 (TOT) 300 Physicians 720 Participants (women) KMC 24 TOT Physicians, 150 Physicians & Health workers 288 Individuals KMC 24 physicians – TOT 150 Physicians and Health Workers 288 Individuals Infection Control/Healthy Nutrition 25 TOT, 300 nurses, 720 Individuals 25 TOT 300 Nurses 720 Individuals15 Total trained: 3,202 (1,474 Physicians, Nurses and Health workers, 1,728 Individuals) Source: FY13 (Q3_ April to June 2013) – Update on Service Agreements Report, FY14 (Q1_ October 1, 2013 through December 31, 2013) Through training of physicians, nurses and CHWs, SMART achieved an increase in knowledge of CHWs about optimal care and practices from 28% in a baseline pre-test to 65.9% in post-test.16 Determining the long-term effectiveness of HCW training from this indicator is difficult, however, as SMART did not implement subsequent rounds of HCW examination. Evidence from the Field In all FGDs with project stakeholders, CHWs and beneficiaries, respondents confirmed that mobile clinics and CHWs provided beneficiaries with timely and quality services, and beneficiaries consistently agreed that mobile clinics and CHWs should be retained in any future MCH program. The majority of CDAs visited sustained mobile clinics (either vehicular, through partnerships with local MOHP health units, or with visiting doctors) and CHWs, though with less frequent home visits, after SMART ended. These CDAs reported maintaining these SMART activities with either existing resources, new funding obtained from other donors, or through agreements with other providers, 14 The progress reports from the partners allude to “individuals” to distinguish them from doctors and nurses; these maybe be non-clinical staff or simply mothers, husbands, or mothers-in-law. The report specifically mentioned awareness workshops where these individuals were trained, but did not specify the audience type at the workshops. 15 Ibid 16 Source: SMART End-of-Project Presentation 19 such as the local universities or MOHP rural health units. When asked to compare quality of SMART services and CHWs to existing community services at MOHP health units (and MOHP-funded Raedat Refeaat17), beneficiary mothers overwhelmingly expressed that SMART’s services were superior to existing alternatives. Further, when asked why mobile clinics and CHWs are preferred above existing public services, beneficiaries unanimously reported that low confidence in MOHP rural health units drives communities to private services (that are often unaffordable) and CDA￾owned clinics, which operated at a subsidized cost. The mobile clinics were cited for the use of medical specialists who provided free check-ups and IFA supplements; the cost of these services was borne by SMART. CHWs reported the training they received not only increased their technical knowledge and professional skills, but also helped them develop as effective community leaders and health and education advocates. Moreover, every FGD conducted with CHWs revealed that SMART was the first-ever employment opportunity for at least one of the young women. One FGD composed of young CHWs (< 25 years) revealed that engagement in SMART allowed them to develop “strong personalities” while helping spark newfound interest in continuing community work in the future. The evaluation team conducted a brief questionnaire of 42 service providers to assess the results of this training and its effect on clinicians’ practices: 41 of 42 respondents reported adopting aspects of SMART training into their practices – specifically, HBB, feeding practices for children, and exclusive breastfeeding (EBF). The self-reported questionnaire also revealed that 45% of surveyed doctors (n=19) report training others in SMART competencies, primarily through on-the-job training of clinical colleagues. Objective 2: Increased knowledge and use of key MNCH-FP-Nutrition behaviors by women and men To achieve objective 2, SMART adopted the following behavior change communication (BCC) strategies and interventions: 1. SMART recruited CHWs from among educated young women in target communities to disseminate messaging regarding nutrition and antenatal/natal/postnatal health; 2. Training doctors in evidence-based clinical practices in the management of stunting, child nutrition, and antenatal/postnatal care to reinforce messages delivered by CHWs; 3. Conducting nutrition classes to teach mothers about inexpensive, nutritious meals and personal and food hygiene, based on the results of SMART-funded operations research (e.g. TIPS); 4. Raised awareness among men and women regarding the importance of proper MNCH-FP￾Nutrition, as well as gender-specific issues, using Community Committees and gender￾focused seminars Evidence from SMART’s Documents/Reports: Key indicators demonstrating changes in beneficiary attitudes and behaviors pertaining to health care and nutrition of pregnant women, neonates and children, are highlighted in the table below. 17 These are CHWs who are salaried under the MOHP system. 20 Table 6: SMART PMP Program Indicators vs. Targets INDICATORS BASELINE ENDLINE TARGET18 Result 1: Increase access to and to quality of key MNCH-FP Nutrition Services Outcome Indicator: % of newborns, delivered in the last 2 years who received Essential Newborn Care (ENC) 17.5% 22.5% 69.8% 1.1 Number of women and children receiving MNH-FP services from mobile teams organized by CDAs. NA 38,000 7,000 1.2 Percent of CHWs who can correctly identify the seven danger signs for newborns 28.0% 52.8% 95.0% 1.3 % of women with children under 2 who consumed 90+iron-folate tablets during their last pregnancy (children who were delivered prior to the baseline/end line surveys 31.4% 34.1% 50.0% 1.4 % of women with children under 2 receiving at least four ANC visits from trained health personnel during their previous pregnancy 75.3% 87.7% 79.4% 1.5 % of women with children under 2 who had a medically assisted delivery (doctor, nurse, midwife) 89.6% 99.7% 91.0% 1.6 % of mothers with children under 2 who received their first postnatal care home visit within two days of delivery 35.0% 62.0% 75.0% 1.7 % of mothers with children under 2 who received their first postnatal care home visit within seven days of delivery 23.1% 30.4% 75.0% 1.8 % of women with children under 2 whose newborns received a postnatal care visit at home within two days of birth by CHW NA 41.7% 75.0% 1.9 % of husbands and wives who received at least one FP counseling session during pregnancy Women: 50.7% Men: 14.5% Women: 80.1% Men: 34.7% Women: 60.0% Men: 30.0% Outcome Indicator: % of children 6- 23 months who are underweight (low weight for age) 15.3% 12.2% 6.1% Outcome Indicator: % of children 6- 23 months who are stunted (low height for age) 22.3% 9.4% Increase knowledge and use of key MNCH-FP-Nutrition behaviors by women and men 2.1 Percentage of mothers with children under 2 who are currently using a modern method of FP 72.2% 67.9% 72% 18The ET was unable to confirm the rationale behind the targets through the documentation and the key informant interviews of those who worked on the PMP, other than the fact the PMP was finalized and approved before the baseline results were processed. This is the only plausible explanation that the evaluation team has to offer for the mismatch between the results and end of project targets. 21 INDICATORS BASELINE ENDLINE TARGET18 2.2 % of women received at least 90 IFA/folic acid tablets with improved level of hemoglobin in their third trimester of pregnancy NA 89.3% 1019 2.3 % of women with LBW (2500g) newborn who practicing Kangaroo mother care for 24 hours/7 days for less than 7 days of baby life NA 21.8% NA 24 hours/7 days for the first 7 days of baby life NA 2.7% 5% 2.4 % of women with children under 2 who can identify at-least 3 danger signs of newborns 14.5% 55.7% 25% 2.5 % of women with children under 2 who sought care from a health care provider for newborns with danger signs 83.8% 82.5% 25% 2.6 % of women with children under 2 with diarrhea in last 2 weeks who provided appropriate care 30.7% 96.5% 10% increase from baseline 2.7 Prevalence of children 6-23 months receiving minimum acceptable diet 37.3% 47.9% 3% increase from baseline 2.8 % of mothers with children under 2 withholding pre-lacteal feeds 57.1% 53.6% 45% 2.9 % children under 2 who are exclusive breastfed in the first 6 months 50.8% 55.6% 37% 2.10% of mothers initiating BF within 1 hour of delivery 35.6% 56.8% 40% Source: SMART’s PMP End-Of-Project Report, September 2014 Note: Boxes highlighted in pink represent figures that fell below targets. 19The baseline did not capture this indicator; therefore, there is no value for it. The target was set prior to the processing of the baseline results; this target intended for the program to achieve at minimum a 10% increase from whatever the baseline value would be over the life of the project. Therefore the ET is unable to assess the performance of this indicator. 22 While the final PMP shows positive trends between baseline and endline results for a majority of indicators, the evaluation team is unable to independently verify the sources and validity of this data. The evaluation team also identified inconsistencies in indicator measurement that have likely resulted in imprecise progress monitoring. Primarily, growth monitoring data was collected by CHWs during home visits. While CHWs were trained in precise measurement techniques at the beginning of the project, the evaluation team notes from discussions with CDA staff that data collected by CHWs commonly contained errors, and that CDAs did not conduct follow-up training for CHWs in data collection.20 Relatedly, a potential explanation for the project’s observed increase in stunting and malnutrition rates is the improvement of measurement techniques over time: as CHWs gained experience obtaining anthropometric measurements of children over the course of the project, measurement techniques were likely to have improved, and measurements themselves were more likely to have become precise and standardized across the target population. Findings from the Field: Notably, SMART’s PMP targets appear disjointed from field realities; thus, comparing endline values to indicator targets are not necessarily instructive in evaluating the project’s achievements. The most tangible outcome of SMART, as obtained through FGDs with CHWs and beneficiaries, is the positive change it induced in target beneficiaries’ attitudes and behaviors regarding MNCH and Nutrition. CHWs and beneficiaries reported seeing marked changes in community nutrition practices following the implementation of SMART activities – namely, widespread behavior change message dissemination from CHWs compared to the attitudes and behaviors at baseline. Beneficiaries consistently reported that CHWs were the key to influencing positive behavior change in communities, and that the availability of mobile clinics helped reinforce the importance of prenatal health. Further anecdotal evidence reveals that many beneficiaries – particularly, mothers of infants – became “knowledge multipliers” in their respective communities, taking individual initiative to pass along nutrition and health information to other relatives and neighbors. Mothers reported that even if they did not have children within the SMART target age group, they applied lessons learned from nutrition classes within their family unit. This notion was corroborated in a focus group conducted among husbands of beneficiaries in Qena, who unanimously reported observing their wives changing nutrition habits in the household (e.g. preparing more nutritious food for the family and exercising improved food hygiene). Change in attitudes and behaviors was not limited to mothers and children; FGD participants overwhelmingly reported shifting attitudes and behaviors of husbands and grandmothers in response to effective SMART messaging and activities. Most notably, CHWs and beneficiaries reported that husbands and grandmothers gradually become more accepting of women delivering in health facilities as opposed to at home. In every beneficiary focus group, the evaluation team heard firsthand accounts from mothers confirming that their children born before SMART were delivered at home, while children born after were delivered in health facilities. These mothers also reported that facility-delivered babies appeared healthier at birth than their other children (born at home). Family planning (FP) was not a dominant theme in any FGD among SMART stakeholders; rather, CHWs and beneficiaries most often mentioned FP as one of SMART’s weakest achievements. Most frequently, CHWs noted that mothers frequently requested more information on family planning methods. This was confirmed by reports from beneficiaries that FP was an area they would like expanded in future programs. It was noted that during the political transition FP commodities experienced a chronic stockout due to unavailability of the various products on the market. This partially explains why the SMART reports show essentially no change in the prevalence of beneficiary 20 Anthropometric measurements are universally known to be difficult to obtain with precision in the field, and normally require extensive training of specialized data collectors. 23 FP use between 2012 and 2013 (70% and 68%, respectively); however, due to SMART’s short implementation frame, it may not be possible to observe a marked difference in any single indicator so soon as a direct result of SMART. Discussions with some SMART staff revealed that FP strategies were never considered the forefront of SMART; rather, staff knew that prevailing cultural sensitivities may have hindered the acceptance of comprehensive FP messaging, so SMART instead focused primarily on initiatives considered more socially agreeable. Objective 3: Increased capacity of CDAs to implement community-based strategies to improve MNCH-FP-Nutrition To achieve Objective 3, SMART implemented a rudimentary capacity assessment tool to identify CDAs worthy of project engagement. Upon selection, SMART instituted a series of trainings aimed at building the institutional capacities of 12 umbrella CDAs (UCDAs) and 101 local CDAs (LCDAs). Partner CDAs received a comprehensive capacity building training and technical assistance package focused on strengthening organizational management (including financial), fundraising, and governance. Evidence from SMART’s Documents/Reports: The following statements are extracted from SMART’s End-of-Project Report regarding the coverage and effectiveness of CDA capacity building: ● “SMART provided UCDAs with training in project management, budgeting, financial and resource management, fund raising, strategic planning, time and resource management, report writing, and staff development and sustainability planning. They, in turn, trained and supported the local CDAs. A total of 112 umbrella and local CDAs were supported under the SMART program. Although some established umbrella CDAs already had organizational capacity, they still benefited from SMART training and support in how to integrate health as a part of the package of interventions provided to the local community.”; ● “As a demonstration of the impact of and support for the work of the CDAs and CHWs, the Social Fund for Development and other donors provided an additional two years of funding to replicate the [SMART] model in areas not currently covered by the SMART program in all six governorates. Many CDAs have also successfully raised funds to extend or expand their activities: 32 CDAs received a total of 33,767,000 Egyptian Pounds (US$4.74 million) to scale up the SMART model. Additionally, 15 SMART CDAs received funding for 28 complementary projects with a total value of 17,239,215 Egyptian Pounds (US$2.42 million) in five of the program governorates. Additionally, by presenting what SMART reported on CDA capacity building in the table below, the evaluation team notes that SMART did not give much emphasis to following up on and measuring the progress made in the CDAs’ capacity. Therefore, the first indicator and target set by SMART were not consistent with the types of training provided by the capacity building program as described in the End-of-Project report; CDAs received trainings in a wide range of skills such as administrative and financial management, but data measuring CDA capacity in these skills are not recorded. Moreover, the ability of the CDAs to raise funds to continue to provide MCH services beyond SMART (which is consistent with the purpose of the training they received) was not among SMART' PMP's indicators and targets. Using this example, the evaluation team illustrates one of the weaknesses of SMART’s M&E, which made it challenging for this evaluation exercise to ascertain some numbers of the results reported in the PMP table and in other reports. 24 Table 7. SMART Indicator Progress for CDA Capacity Development Indicator Target Achieved # of UCDAs who are capable of conducting gender analysis at the community level21 6 6 % of LCDAs successfully completed the program interventions for MNCH-FP-Nutrition services as defined in the work plan 90% 100% Evidence from the Field: All CDAs (umbrella and local) visited for interview reported that SMART’s capacity building efforts were beneficial, and that SMART’s technical assistance helped improve the performance of the organization. Some CDAs reported greater benefits than others; for example, those CDAs with higher existing capacity levels (UCDAs) noted that SMART’s technical assistance was more impactful for smaller, less developed CDAs. Likewise, many local CDAs confirmed this point firsthand. Corroborating evidence from SMART documentation, local CDAs reported that fundraising training was particularly beneficial in helping to obtain grant funding after SMART (for example, from the Social Fund for Development). The majority of local CDAs reported positive experiences with their assigned umbrella CDAs. CDAs in Upper Egypt in particular reported being accustomed to the SMART model of inter-CDA knowledge/experience sharing and mentoring; thus, the SMART model of CDA mentorship was not challenging for these CDAs to adapt to. As mentioned earlier, local CDAs in Upper Egypt established a formal network to capitalize on information sharing and continue the progress initiated by SMART’s capacity building efforts. Umbrella CDAs also reported that housing SMART project staff (project and grant officers) directly within the UCDA was beneficial to the organization’s performance, as UCDA staff learned additional skills working alongside project staff. The evaluation team found that CDAs’ strategic planning was a general weak point: most CDAs’ strategic plans were either nonexistent or elementary (i.e., did not contain proper mission statements or strategic objects). Of the CDAs that did have strategic plans, few had mission statements that reflect an expressed commitment to MNCH-FP-Nutrition outcomes, posing a potential risk to the sustainability of SMART-related activities beyond project close-out. Objective 4: Increased the knowledge base of factors associated with stunting (including those that are gender-specific) and approaches to reduce stunting and neonatal mortality To achieve objective 4, SMART contracted technical subject experts to conduct a number of operations research activities aimed at investigating the behavioral and contextual factors that enable stunting and malnutrition in Egypt. The studies consisted of in-depth focus groups with mothers and grandmothers, in-home observation of maternal behaviors, and follow-up growth monitoring of newborn children. Evidence from the Field: Interviews with SMART staff and technical consultants revealed that operations research encountered significant implementation delay. The in-depth focus groups conducted as part of the TIPS studies generated valuable data on nutrition-related behaviors in Upper and Lower Egypt. SMART reportedly extracted key lessons from these studies and incorporated them into SMART materials. The study reports themselves were also developed into project briefs for dissemination to clinical providers; thus, SMART’s qualitative operations research was found to have been useful for 21This indicator is evident from the fact that SMART was able to work with 12 UCDAs however it does not measure the capacity of the UCDA to implement the gender analysis. The ET heard through the interviews with the CDAs that they did receive the training in gender and that some even went to imminent such component. 25 tailoring elements of SMART’s health messaging to the Egyptian context. The longitudinal study on stunting, on the other hand, was designed to follow study participants (mothers and newborn babies) for one year, monitoring child growth while counseling mothers on best nutritional best practices. At the time of this report’s writing, final analysis of this study is underway; however, the evaluation team did not obtain a draft study report to analyze; since the study ended at the end of June 2014, the study team informed the ET that the data was still being analyzed and therefore not available for sharing. The study was planned earlier in the project, and the project was extended beyond its original end-date to allow for its completion. SMART staff, however, did not provide the evaluation team with specific dates regarding when the study was slated to have started and ended. However, results from the Stunting Study-Junk Food Brief and TIPS -Stunting Report proved that the nutritional counselling, observation, frequent growth monitoring and nutrition/feeding classes resulted in positive child health outcomes, which were more pronounced in Upper Egypt's children. Though junk food was a problem in both Lower and Upper Egypt, it was more prevalent in Lower Egypt. In the study sample, only 11% of children were stunted, with slightly greater proportion of stunted children in Lower Egypt (12%) than in Upper Egypt (10%). Acknowledging the particular cultural and socio-economic differences in both Upper and Lower Egypt, the evaluation team learned the findings of this study were especially useful in guiding SMART messaging. Objective 5: Improved awareness of the impact of gender roles in improving MNCH-FP-Nutrition outcomes To achieve Objective 5, SMART addressed gender inequality in the following ways: 1. SMART worked with UCDAs to help them carry out a basic gender analysis in each governorate, which served as a program planning tool and guide for examining social relations in communities and households in order to maximize the health outcomes within specific communities, allowing interventions to be locally appropriate and women￾empowering; 2. Based on the gender assessment, SMART conducted workshops in the six governorates to introduce the gender analysis tools to SMART partners. 3. SMART drafted a program-wide gender equality strategy, which framed SMART’s gender approach across all activities and with implementing partners; 4. In December 2012, SMART developed the Family Solidarity Module using a training methodology that builds on the principles of adult learning and active participation. SMART built the capacity of CHWs and community health committees using the Family Solidarity Module to raise awareness and initiate dialogue around gender relations in the communities where they worked; and 5. SMART encouraged local organizations to integrate an awareness of gender roles within their routine activities through a series of Training of Trainers workshops—first at the national level, and then in each governorate. 26 Evidence from SMART’s Documents/Reports: Table 8: Level of achievement of SMART targets regarding messages about gender Target Achieved # of fathers and mothers who have participated in at least family solidarity session linked to SMART Interventions Was not specified in SMART’s PMP 10,000 of fathers and 10,000 mothers-in-law22 # Social Behavior Change (SBC) materials reflecting gender notion 70 90 Source: SMART’s End-of-Project Report, September 2014 According to SMART documentation, the project reached 10,000 fathers with gender-related messaging; however, the evaluation team cannot independently verify this figure. An interview with a former high-level SMART staff member revealed that once a predetermined target was achieved, SMART ceased tracking the indicator’s progress. In the case of gender-related project indicators, the evaluation team notes that due to the inability to obtain rosters of gender seminar participants or any other data to substantiate the figures reported in SMART’s final PMP, it is not possible to know the precise number of individuals sensitized to gender messages through SMART. Evidence from the Field: There is evidence, both from SMART data and firsthand information gathered from the interviews and focus groups that the gender awareness campaigns have been perceived as successful; beneficiary women and husbands reported that attitudes surrounding gender norms are beginning to shift in their communities. Despite the fact that SMART’s gender activities were introduced late in the course of the project (gender was the last training module developed for SMART), CDAs reported that the module was widely accepted by communities and considered beneficial. In particular, female beneficiaries noted that similar gender-related seminars should continue with any future project, and many CDAs (in Upper Egypt especially) have built upon SMART’s gender module in continuing to host gender-related discussions in community forums (Mandaras). SMART also engaged local community leaders in spreading messages of gender parity. The evaluation team interviewed two local religious leaders, who both separately confirmed that they have continued spreading gender-sensitive messaging, such as the teachings of Islam and Christianity regarding women/mothers status in the family and how they should be treated with respect and care ( e.g., “Paradise is at the footsteps of Mothers” and “Women and Men are equal before God” - Prophet Muhammad) in their communities after SMART close-out, and that these gender activities have effected positive change in their respective communities. A focus group conducted with husbands of beneficiaries in Qena demonstrated that men in traditionally conservative communities are beginning to welcome their wives’ participation in CDA health activities and mobile clinic visits, where before, wives were admittedly discouraged from leaving the home. With respect to evidence of SMART’s overall positive effects, the evaluation team asked SMART beneficiaries – specifically, CHWs and mothers who participated in SMART activities – about significant changes observed in their respective communities as a result of SMART. Mothers cited the widespread application of knowledge gained from CHW visits and nutrition classes as a positive impact on family health. Mothers across all governorates consistently cited instances of behavior 22 Only mothers-in-law and fathers were in attendance at these solidarity sessions, as they were intended to attend together. The calculation counts the total number of fathers and mothers-in-law from the attendance lists of family solidarity sessions held by the CHWs. 27 change, particularly by way of exclusive breastfeeding, preparation of more nutritious meals, and regular antenatal care, as evidence of SMART’s effect. CHWs cited similar changes observed in their communities. The most significant changes observed in SMART-implemented communities, as reported by CHWs, were the uptake of exclusive breastfeeding practices among mothers, improved family nutrition, and increased antenatal visits to health units and mobile clinics. Other behavior changes cited included improved recognition of neonatal and infant danger signs, consistent child growth monitoring (informally by mothers, and during clinic visits), and mothers’ adherence with prescribed IFA supplement regimens. Figure 2 below depicts a graphical representation of citation frequency of the most significant changes observed in communities as a result of SMART, where graphic size corresponds with the frequency of mentions in interview data. Figure 2. Most frequently cited behaviors changed in community due to SMART, as reported by CHWs CONCLUSIONS While the project’s data management challenges make it difficult to assess the SMART’s achievements from numbers alone, evidence from qualitative interviews and focus groups suggest that the constellation of SMART activities were wholly successful. The evaluation team found the most successfully adopted behaviors among beneficiaries were exclusive breastfeeding, regular antenatal care, improved household nutrition and hygiene. Also, higher stunting rates at endline may, to some extent, be a function of improved surveillance techniques CHWs were effective in providing quality health awareness target families, and were ultimately beneficiaries of SMART themselves. Mobile clinics expanded rural communities’ access to critical maternal health and nutrition services, which, in turn, allowed beneficiaries to continue exercising positive health practices. Because SMART trained different types of stakeholders – doctors, nurses, CHWs, mothers, grandmothers, and husbands – the project ensured that health messages remained consistent. Community leaders were similarly instrumental in propagating evidence-based health messages throughout target areas; SMART was successful in leveraging existing community resources to effect health behavior change. The two indicators SMART used to measure CDA capacity did not represent the breadth of the skills that the CDAs received. Alternatively, even though it was not in the mandate of the project to follow up progress of the CDAs’ capacity in depth, at least other more appropriate indicators reflecting specific training would have been more appropriate (e.g. number of CDAs with a well￾defined budget, or accounting system in place, or the like). Thus, the indicators chosen to measure CDA capacity effectively undermine the comprehensive investment that the project made in building the capacity of the CDAs. Training of health care providers was not only effective in improving the clinical services they provided, but also ensuring the sustainability of the “know-how”, as trained individuals continued to train others after SMART ended. Further, popularity of SMART’s activities, from nutrition classes to mobile clinics, speaks to the quality of the interventions. The quality of capacity building activities also strengthened local CDAs’ response to community malnutrition. 28 SMART’s operations research activities constituted a large undertaking, as a well as a significant portion of SMART funding. Qualitative research was instrumental to refining ongoing elements of SMART messages as the project was underway, which was an efficient use of data The ET estimates that the results of the longitudinal study once they are published will help guide future programs. Though by virtue of its definition, a longitudinal study does not include a control group, yet in the case of SMART, it could have been an opportunity to study control groups to compare groups from communities where SMART intervened against the cohort in the longitudinal study that did not. This could have leveraged the Longitudinal Study's conclusions. Because, the duration of SMART's longitudinal study was too short. This is why the ET suggested that not having a "Control Group" in SMART's Longitudinal Study was a missed opportunity. SMART demonstrated that subjects that are traditionally difficult to discuss, such as family planning and gender, can be effectively addressed from a health perspective. While beneficiaries found SMART’s family planning elements insufficient, anecdotal reports of increased demand for FP information demonstrates that future opportunities for promoting modern FP methods exist. SMART was particularly effective in directly incorporating gender into activities; similarly, increased demand for gender-related seminars among beneficiaries and CDAs show SMART has opened the door for future activities that build on the project’s gender work. Evidence for SMART’s positive effects comes directly from mothers engaged in SMART activities, who reported significant and lasting behavior changes as a result of gaining new knowledge, as well as CHWs, who observed widespread change in behavior and attitudes in their respective communities. Anecdotal evidence of behavior change, particularly from mothers who reported changing child rearing practices, demonstrates the effectiveness of SMART’s health messaging in a context where harmful old practices are traditionally difficult to overcome. 29 EVALUATION QUESTION 3: WHAT ARE THE MOST SIGNIFICANT DETERMINANTS THAT INFLUENCE THE LIKELY SUSTAINABILITY OF PROJECT BENEFITS IN TARGET POPULATIONS? FINDINGS The evaluation team assessed sustainability from three perspectives: financial, institutional and final outcomes. Financial In terms of financial sustainability, SMART data shows that 32 of the project’s 101 CDAs have successfully acquired funding to either continue SMART activities or implement new projects. The indicator used to measure performance in CDA fundraising did not identify which of the 112 SMART CDAs improved their capacity to sustain SMART activities; rather, the indicator simply measures how many CDAs received funding to continue activities. It is possible that these 32 CDAs had a higher level of existing capacity than other CDAs and would have received funding anyway due to their demonstrated financial viability. Table 8. Summary of CDAs engaged in scaling up SMART Activities Governorate # of CDAs # of projects Total value in L.E. Lower Egypt Sharquia 1 1 1,000,000 Qalyubia 3 3 3,000,000 Upper Egypt Beni-Suef 4 4 4,000,000 Asyut 8 9 8,430,000 Sohag 11 11 12,000,000 Qena 5 6 5,337,000 Total 32 34 33,767,000 Discussions with CDA staff revealed that the capacity building received from SMART – particularly, fundraising – was reportedly beneficial to CDAs’ ability to secure funding from sources such as a Social Fund for Development. Some CDAs visited for interviews reported continuing SMART activities using existing funding, including continuing to pay CHWs for home visits and health messaging. One Umbrella CDA reported terminating CHW home visits after SMART ended, but using CHWs as education awareness agents in the community in support of a newly-funded project. Institutional CDAs reported that participating in SMART strengthened their organizations, enabling them to strengthen their respective communities. CDAs in Upper Egypt established networks formalizing knowledge and experience sharing initiated by SMART. CDAs also reported that participation in SMART rose their profiles in the community, and that increased credibility has helped sustain messages promoted by SMART. However, the evaluation team found that only three of the CDAs visited by the evaluation team had changed their missions to include MNCH as a result of SMART, which has implications on the sustainability of CDAs' commitment to MNCH in the future. SMART project staff and consultants reported that the MOHP and some national associations engaged with SMART ultimately incorporated SMART technical materials into their own training curricula for health providers. Additionally, one academic doctor trained by SMART reported incorporating SMART-introduced techniques such as KMC into his medical school lectures. The same doctor has also independently initiated university-funded research to investigate the impact of 30 KMC in rural communities surrounding Sohag. A gap in institutional sustainability is the fact that SMART did not have a deliberate strategy to directly upgrade the skills of service providers at MOHP rural health units in order to raise confidence in public service quality, and thus increase usage of these facilities. Instead, SMART shifted demand from the public health system and costly private clinics to CDA-sponsored mobile clinics.23 During the interviews and focus group discussions, beneficiaries indicated preference for those mobile clinics and were still less inclined to use the public clinics. While SMART did not engage in activities to explicitly improve the quality of services offered at rural health units, the evaluation team confirmed that SMART operated in one district that overlapped with complementary public clinic activities implemented by UNICEF. Specifically, the UNICEF project worked to improve health unit performance monitoring and data use, in addition to using BCC materials from SMART to sensitize public sector health care staff and their clients. For the one district in which this programmatic overlap occurred, SMART’s institutional sustainability may have been bolstered, though this could not be confirmed with health facility or population-level data. Outcomes Even six months after SMART activity close-out, the evaluation team found sustained demand for MNCH-Nutrition services initiated by SMART. Focus groups with different project stakeholders revealed that beneficiaries continue to request health information from CDAs. Regardless of whether they are still actively employed as CHWs, all CHW FGD participants unanimously reported that they continue to serve as health advocates in their communities. A number of SMART CHWs reported being stopped on the street or reached directly at their homes by beneficiaries continuing to seek health advice. A number of CHWs also reported becoming mothers after SMART, which gave them the opportunity to directly apply and benefit from the lessons they had been sharing throughout the community. CONCLUSIONS While the financial sustainability of CDAs is indeed important for the continuation of SMART activities, the most sustainable element of SMART has been the sustained demand for MNCH￾Nutrition services. Public awareness of evidence-based strategies to combat stunting and malnutrition continues to spur demand for activities that meet community needs, and a cadre of trained CHWs now exists to meet that need. Further, the multiplying effect of SMART’s health messaging has sustained impact within households, as mothers continue to improve health behaviors for future children, and at the community level, where mothers continue to demonstrate best practices to their neighbors and relatives. The evaluation team notes that while the use of SMART materials in training curricula and universities is a significant achievement for project sustainability, participating institutions are limited in their ability to independently modify national training curricula (specifically, for medical/nursing school students) without an explicit mandate from the Ministry of Higher Education.24 SMART’s training for medical health professions was well-received, though there is little evidence to demonstrate the widespread effect on improved health care practices. 23 The evaluation team notes that direct engagement with public service providers, namely by way of skills training, was hindered by delayed engagement with MOHP during SMART’s start-up phase. SMART project staff noted that this delay prompted a shift in project focus: working directly through CDAs. 24 Direct coordination with and approval from the Ministry of Higher Education is a necessary prerequisite for modifications to any national curriculum. 31 EVALUATION QUESTION 4: TO WHAT EXTENT DID THE PROJECT USE ITS RESOURCES EFFICIENTLY? FINDINGS Efficiency in the context of SMART was examined from the viewpoint of synergy and leveraging existing resources, and to a lesser extent, on use of funds beyond CDA grants.25 Although the evaluation team was not able to determine the total value of funding saved by adapting educational materials from previously funded USAID projects (e.g. CHL and Takamol, which ended in 2010), it is assumed that the project saved labor costs by updating existing materials. SMART also drew on existing community entities and networks (e.g. CDAs, Mandaras) to directly implement its activities; this strategy ensured SMART did not establish a duplicative system, and relied upon a health structure already adapted to the local Egyptian health context. Utilizing existing community structures also saved SMART from building implementation entities from scratch; minimal investment in capacity building activities was necessary, as CDAs with fundamental organizational structures already existed in target communities. CHWs also represent existing resources that SMART leveraged, as young women were recruited directly from the communities they would serve. This meant that transportation costs (to the project) remained low, and CHWs could capitalize on innate local knowledge to make SMART messaging more effective to target beneficiaries. The only financial data that was made available to the evaluation team provided a rudimentary breakdown of project line items. Table 9 below presents a calculation to show that the CDA budget was approximately one third of the field budget, and that it cost SMART approximately 5% of its total budget to motivate 1,200 CHWs at a cost of less than a $1 per day for their contributions to achieving the project outcomes. Table 9. Calculation of CHW vis-à-vis Total SMART Budget Total SMART Funding: $10,400,000 for 32 months Field Budget26: $8,583,007 CDA Contracts: $2,783,872 (32% of Field Budget) Calculation of CHW Motivation  1200 CHWs x $30/month = $36,000/month  Total Cost (average 15 months27) from 2011 to 2013: $36,000/month x 15 months = $540,000  ($540,000 / $10,400,00) * 100 = 5.2% of total SMART budget After SMART funding ended, many CDAs were unable to independently sustain CHW outreach activities at the same level as SMART. However, having gained proposal development capacity from SMART, nearly every CDA reached by the evaluation team reported successfully acquiring funding from such sources as the Social Development Fund to continue some activities initiated through SMART. In this respect, SMART’s capacity building component was an efficient use of funding, as it equipped CDAs with the ability to continue community outreach activities without growing a dependency on USAID funding. 25 At the time of this report’s writing, the evaluation team has not been provided detailed SMART budget figures with which to conduct economic analyses. 26 The field budget can be summarized into these main categories: Admin Total Budget of $4,079,910 consisted of (Personnel cost including benefits, travel, equipment, materials and supplies, program costs and indirect costs), and contracts including CDAs' contracts for a total of $4,503,097. 27 This is an average figure; some CDAs engaged with SMART for as long as 18 months, while others participated for as few as 11 months. 32 All interviewed CHWs, regardless of current occupation or project affiliation, reported informally continuing health message dissemination in their respective communities, chiefly possible as a result of the visibility and credibility imparted by their participation in SMART-sponsored community activities. After SMART’s termination, CHWs reported instances of being sought after for health advice on the street, via telephone, and even at their own homes. Many CHWs noted that women in their communities expressed greater trust in them than in medical doctors. At a significantly low cost to the project, SMART created a cadre of health workers whose effects on the community continue to pay dividends after the project’s end. CHW compensation was originally intended as a stipend rather than full-time or part-time salary; the evaluation team notes, however, that SMART’s compensation (between $25 and $50 a month) was below the national minimum wage ($174/month28). A dominant theme of every focus group discussion, as well as the majority of key informant interviews, was concern for the CHWs’ level payment. A glimpse of the job description of the CHWs reveals that they carried a heavy workload relative to compensation. A GLIMPSE AT A CHW’S WORKLOAD ● CONDUCT FOUR (4) WEEKLY COUNSELING SESSIONS FOR WOMEN IN FIRST PREGNANCY ● CONDUCT MONTHLY AWARENESS SESSIONS FOR ALL PREGNANT WOMEN ● CONDUCT HOME VISITS FOR FIRST AND HIGH-RISK PREGNANCIES ● CONDUCT TWO (2) POSTPARTUM VISITS (2ND AND 7TH DAYS) ● CONDUCT FOUR (4) GROWTH MONITORING PROMOTIONS FOR CHILDREN ● CONDUCT MONTHLY COOKING DEMONSTRATION SESSIONS ● CONDUCT TWO (2) HOME VISITS FOR AT RISK CHILDREN ● EACH CHW CONDUCTS A MINIMUM OF TEN (10) HOME VISITS PER WEEK Unfortunately, data on the costs of individual SMART activities, such as trainings, clinic upgrades, BCC material development and dissemination, or studies, was not provided to the evaluation team; thus, the team could not determine the cost efficiency of each project component relative to its observed outcomes. CONCLUSIONS SMART maximized project efficiency by leveraging existing systems and entities. CDA contracts, which included compensation for CHWs, constituted approximately a third of the total field budget and were responsible for enabling the entire project’s observable outcomes (ultimately, SMART’s results). While the evaluation team is unable to estimate the added value of SMART’s other components, such as detailed nutritional studies, it is notable that the lowest allocation of resources was reserved for CHWs, the “engines” producing SMART’s results in the community. Therefore, the evaluation team’s conclusion on resource efficiency is limited only to the fact that the most important vehicle for achieving the SMART’s objectives received the least amount of financial resources. While it is true that SMART achieved critical health behavior outcomes with limited financial input, this efficiency is unsustainable; future projects will need to provide a more equitable wage in order to attract and retain CHWs for similar work henceforth. 28 As of January 2014. Previously, Egyptian minimum wage was $102. 33 EVALUATION QUESTION 5: CONSIDERING THE PROJECT’S DESIGN CONSTRAINTS AND POSSIBLE COUNTERFACTUAL ALTERNATIVES, TO WHAT EXTENT DOES THE CONSTELLATION OF PROJECT INTERVENTIONS REPRESENT THE MOST EFFECTIVE WAY OF REDUCING STUNTING? FINDINGS The evaluation team considers the project’s implementation period too brief to observe the effect of the package of interventions on stunting and maternal health; thus, effectiveness is examined mainly through the viewpoint of the ability of the SMART intervention to achieve its objectives. Overall, results demonstrate that SMART was effective in sustaining demand for services, especially from mobile clinics and home visits; however, the evaluation team relied chiefly on perceptions of high satisfaction that beneficiaries expressed during focus group discussions. As for the 3,200 physicians and nurses trained through the associations using SMART’s training materials, it is not possible to verify that trainees applied the knowledge they gained. Table 10. Level of achievement of SMART objectives SMART Objectives Interventions Level of achievement Improved access to and quality of key MNCH-FP-Nutrition services by private facility and community-based providers Community health outreach and communications activities; Nutrition education and rehabilitation program High access; quality unverifiable Increased knowledge & use of key MNCH￾FP-Nutrition behaviors by women and men. Community health outreach and communications activities; Nutrition education and rehabilitation program; Home￾based neonatal care through a package of simple interventions High for both knowledge and use Increased capacity of Community Development Associations to implement community-based strategies to improve MNCH-FP-Nutrition. Build capacity of local CDAs to respond to health needs Relatively higher in Upper Egypt; relatively lower in Lower Egypt29 Increased knowledge base of the causes, including gender-specific causes, of stunting and how to implement programs to reduce stunting. Implement an in-depth study to understand underlying issues for the increased stunting levels in Lower Egypt High for knowledge generation Improved critical awareness of the impact of gender roles in improving MNCH-FP￾Nutrition outcomes. Community health outreach and communications activities High for awareness 29 CDAs in Upper Egypt reported more experience in working with foreign donors and development programs, as well as inter-organizational learning, than in Lower Egypt. The evaluation team learned that international donor programs over the past three decades have focused more Upper Egypt than in other areas. 34 Source: Evaluation team's qualitative findings Limitations: While the TIPS study results provided a preliminary knowledge base for the underlying causes of stunting, its use in program design is reserved for future projects, as well as the results of the longitudinal study. Preliminary results thus far are not able to demonstrate quantitative effect on stunting partly because the implementation period was very short, and the data on the cohort of the 300 children followed is still being processed. Further, there is no comparison group against which to measure the effectiveness of SMART interventions over time; thus, the study will not produce quantitative evidence to suggest how effective the SMART model is in combatting stunting in Egypt. Another limitation is that the evaluation team was unable to rely on the results of SMART’s endline study for determining the impact of SMART interventions. The endline study identified “comparison” districts against which to measure intervention districts; however, the evaluation team does not consider these comparison districts to be a true counterfactual. While the study did attempt to match districts on a number of socioeconomic indicators for comparability, matching at the district level may be too high a level at which to attain statistical power. Further, the project’s short implementation period and small number of direct beneficiaries relative to governorate population makes identifying a significant change in key outcome indicators extremely difficult. Both the baseline and endline surveys were conducted by two different firms, which introduced the potential for methodological discrepancy. Conclusions derived from analysis of the endline survey, therefore, do not speak to SMART’s true effectiveness. Also, a cost effectiveness analysis could be conducted on the package of SMART interventions, as detailed budget data was not made available to the evaluation team. CONCLUSIONS Evidence for the effectiveness of the SMART model is limited to the local perception on effectiveness, namely testimony of beneficiaries who attested to changes in their behaviors and those observed in their communities. All stakeholders expressed that CHWs were effective, and that mobile clinics and nutrition/health education classes were beneficial. Anecdotal evidence suggests that behavior change continues to spread throughout target areas well after SMART’s close-out. 35 LESSONS LEARNED 1. Strategic Partnerships and Scale Up: Partnerships built over time with between SMART and agencies/associations in health and community development were instrumental in allowing SMART to expand beyond its geographical coverage. Also, tapping into the local expertise of researchers and practitioners in health and community development benefitted the research studies on malnutrition and stunting. The TIPS studies used local researchers with existing knowledge about Egyptian culture and cooking to inspire the recipes that the CHWs used during the nutrition classes. In the future, such partnerships are critical to scale up community-based initiatives and for diversifying funding and other resources, as these institutions could also contribute money and other in-kind resources that projects cannot independently sustain overtime. In that light, the experience also reinforces other types of partnerships that should also be cultivated early on with the local government (e.g. MOHP, Ministry of Social Solidarity, and Ministry of Education) as well as private sector (e.g. pharmacies, telephone service providers, and banks) all in an effort to address health and community development more systematically. The evaluation team heard through the interviews how some pharmacies provided discounts on iron/folic acid tablets, and how CDAs in Sohag and Asyut were able to use mobile vans from the local MOHP to conduct their monthly outreach clinics. If future programs consider expanding the SMART model, all of these types of partners should be aligned from the very beginning of the program. 2. Community Based Health Programs: One fundamental lesson learned is that a project cannot increase demand without subsequently providing services to meet demand. SMART was strategic in ensuring that CDAs would provide services to sustain the demand raised by the effective messaging though the CHWs. The mobile clinics enhanced the work of the CHWs and simultaneously facilitated the adoption of the new practices that they promoted --- hence, making behavior changes possible. Improving knowledge alone does not translate into behavior change; what made the difference in SMART were the messengers (CHWs as agents of change), community trust in the messenger (CHWs originated from the communities), message consistency, and repeated contact with beneficiaries (repeated home visits, interaction at the market, by phone, word of mouth, etc.), use of local products for ingredients and local recipes (nutrition classes used existing products already consumed locally), and engagement of the entire family unit (husbands and mothers in law became involved in care of the mother and child). Monetary incentive was not the only factor that kept CHWs motivated. Despite the fact that CHWs were compensated at a very low rate (average of $1/day), they remained committed and persevered with their duties of home visits and counseling. During the FGDs, they shared with the evaluation team that their motivation was sustained by several factors: a) giving back to the community; b) religious duty to serve others; c) personal development (opportunity to gain employment experience), new knowledge and skills in health and nutrition; d) work experience, especially for those who had never worked before; e) and for those who were out work, it was an opportunity to do something worthwhile. While future projects should consider an enhanced compensation package for CHWs, incentives should not be viewed in strictly monetary terms; other schemes of compensation should be explored, such as education subsidies. Gender (family solidarity) equity/equality was served early in the project through indirect means, by introducing concrete evidence to family members (husbands, mothers in law) that caring for and helping mothers during pregnancy and childcare improved the health of 36 children and mothers. Gender was a strongly desired program component; many stakeholders reported wishing that gender training had been introduced earlier. However, even prior to the introduction of SMART’s formal gender activities, beneficiaries reported experiencing positive effects of SMART’s messaging in terms of husbands and grandmothers becoming more lenient with mothers leaving the home to attend activities at CDAs. Family solidarity effectively preceded the formal sensitization sessions on gender. The evaluation team is not suggesting that sensitization on gender was not useful; on the contrary, it should be introduced at the very beginning. However, this experience suggests that there are other strategic ways to address gender apart from conducting seminars that can be both subtle yet impactful. 3. Centralized information management: Compiling project data solely at the central level for analysis provided a disservice to the project in that the CDAs were not empowered to track their own progress against targets. For example, the evaluation team did not find evidence that the PMP was consistently updated summarized in quarterly reports and shared with the project team and partners at large. While a large amount of project data was collected, it was not used in timely decision-making for modifying program activities, strategies, or targets, which hindered the extent to which SMART can claim success. 37 RECOMMENDATIONS While it is not certain that a new cadre of community health workers like the SMART agents of change can be retained to work only on health activities, that the renovations of local health units to family health units will scale up nationally over the next five years, or that civil society will regain enough trust in public sector services to seek them out for quality care, now is an opportunity to rethink how a future community-based health program should look. Overall, as the evaluation team recognizes that USAID and other development partners have a role to play in building the community’s trust again in the public health system, it recommends that future health and development projects should not only focus on the quality of care, but also be anchored in sustaining behavior changes not just for mothers, but also for service providers. Specific recommendations are as follows: Recommendation 1. Build a hybrid of community based health model. This type of model should combine the best elements of what civil society, government and private sector have to offer. The CDAs already have the community’s trust, as they are deeply rooted there. This can be capitalized on to engage more development workers to be agents of behavior change. Other elements of the program should include: a) Targeting adolescents for messages about MNCH/FP/Nutrition before marriage and first pregnancy in order to nurture an early understanding of good health behaviors emphasizing the role of proper nutrition for the welfare of the mother and the child. By targeting adolescents who may be more open and less set in their ways, the likelihood of adopting those key behaviors at an early age will ensure sustainability. b) Increase resources for CHWs: The budget for CHWs should be enhanced so that their motivation is more comparable to MOHP standards. Also, since they expressed interest in personal development, future projects should look into a compensation package that could be part monetary, but also include access to specialized training that would advance their earning potential and/or access to a micro-credit program aligned to their interests. As per their request, a toolkit containing detailed technical manuals, for example, could be used to support their messages during health education classes and also gain more credibility from the families that they visit. Audiovisual aids could also be useful during health education training and nutrition classes during mobile clinics and/or during monthly sessions at the CDAs office. As needed refresher trainings are also key to keeping CHWs updated on the latest evidence-based strategies and guidelines. c) Leverage socioeconomic development activities: Because health interventions are not always the answer to resolve health issues facing communities, future health projects should also be combined with socioeconomic development projects that help maximize the benefits of higher income, such as accessing private health providers, the ability to provide more nutritious foods for the family and purchasing iron/folic acid for pregnant women. Therefore, USAID should deliberately align geographically with other types of community-based initiatives funded by USAID and other development agencies. In areas where there is no such initiative, the project should then advocate with the development agencies to invest in those communities where both health and socioeconomic needs are still not met. d) Consider adding male CHWs. Although the CHWs were able to eventually gain trust of the families that they served by winning over the consent of the husbands and mothers in law to allow young women to participate in health education and nutrition classes, future projects could benefit by introducing a cadre of male CHWs to work alongside female CHWs to promote the family solidarity concept/strategy that SMART utilized in its campaign for gender sensitization. In addition, the family solidarity module should automatically 38 incorporated in the training agenda for all CHWs so that it is not seen as a side topic. e) Strengthen linkages with public health system: As citizens recognize their rights to quality health services, they learn to demand more from their public service providers and local governments. On one hand, SMART was able to achieve that in BeniSuif, where citizens demanded the change in the service provider at a health unit. This came about because of the awareness that SMART created through health education and classes. This can be done on a wider scale if the next project systematically creates a mechanism to evaluate those situations where poor service is provided and equip citizens to make such demands. Another way to collaborate with the public health system is through the use of the “mobile clinics”. Where there is an actual mobile van, the CDAs should proactively seek to use it, and in the absence of such van, the CDA should work with local rural/family health unit to use their trained specialists to serve at a designated location once a month. f) Maximize coverage of health messages through social media: Use of social media through collaboration with private service providers of cell phones is essential for any future program. The way that individuals interact and obtain their health information has changed worldwide, particularly in Egypt. Using sites like Twitter and Facebook, people instantly communicate with large networks of friends, colleagues, and even strangers. Therefore, there is merit for future health projects to publicize activities using social media, as well as using it to engage with newer, younger audiences. Recommendation 2. Data management: Leverage the use of information communication technology in future programs. Future M&E systems should be built on electronic data collection systems that allow for reduced human error in data reporting, reliable and secure data compilation and management, advanced analytics capability, ease of data dissemination, and above all, ease of use. A host of free software is available to development project implementers worldwide, and should be explored for use in any future project, specifically, projects with built-in operations research. Electronic data collection would allow CHWs to submit home visit data on tablets, for example, that CHWs could also use to show mothers video demonstrations, interactive health data, or other audiovisual aids. Recommendation 3. Target service provider capacity building to improve quality of services: Training for medical doctors and nurses should start early on in their careers (e.g. before being licensed), during the internship period for doctors and for nurses, while they practice what they learned in the classroom. Instead of blanket training for doctors at any career stage, future projects should consider targeting these internship programs to build into the curriculum and/or seminar schedule updates on the various topics (e.g. MNCH/FP/Nutrition, WHC) relevant to the intern’s specialty. Furthermore, a future project should explore a mechanism by which students not yet licensed can access these trainings at no cost, while doctors already in practice pay a small fee. This fee can be determined by the association that sponsors those trainings to subsidize cost of the trainers, materials, and other administrative fees. This will help to reduce the number of individuals who repeatedly reap the benefits of free trainings without valuing the knowledge or skills gained. A client satisfaction survey should also be incorporated into a future program as a means of measuring how these skills are being used to improve quality of services. 39 ANNEXES I. Evaluation Scope of Work II. Detailed Evaluation Methodology III. Final Evaluation Tools IV. SMART Performance Management Plan (PMP) V. Summary Findings from Key Informant Interviews VI. List of Key Informants VII. List of Site Visits VIII. Summary of CDA Assessment Update IX. Summary Results of Telephone Survey X. Summary of Key Lost Opportunities (Conclusions) XI. Evaluation Meeting Schedule XII. Disclosure of Any Conflicts of Interest 40 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 41 ANNEXES I. EVALUATION STATEMENT OF WORK STATEMENT OF WORK End of Project Performance Evaluation of USAID/Egypt Maternal and Child Health Integrated Program I. Background Information A. Identifying Information Project Title: Maternal and Child Health Integrated Program (MCHIP) Award Number: Cooperative Agreement # GHS-A-00-08-00002-000 Award Date: October 1, 2011–June 30, 2014 Geographic Areas: Qalyubia and Sharqia in Lower Egypt; and Asyut, Beni-Suef, Qena and Sohag in Upper Egypt Funding: $10,400,000 Implementing Organization: Save the Children (SC) Activity Managers: George Sanad USAID/Egypt (during the period October 2011 through May 2013) Shahira Hussein, USAID/Egypt (June 2013-present) Evaluation Program Manager: Shadia Attia, USAID/Egypt B. Development Context Background The 2008 Egypt Demographic Health Survey (EDHS) estimates neonatal mortality to be 16 per 1000 live births, a decrease in mortality of 33% since 2000. For the same period, infant mortality declined by 43% and under-five mortality by 48%, indicating a more rapid decline of infant and under-five mortality in comparison to neonatal mortality. Neonatal mortality contribution to under-five mortality has therefore increased from 44% to 58% during the same time period. Malnutrition in children is surprisingly high in Egypt; according to EDHS, stunting prevalence (low height-for-age) is a significant problem in Egypt. Stunting prevalence among children under five increased from 23% in 2005 to 29% in 2008. It is known that in many developing countries, stunting occurring before two years of age will have a long-term and irreversible effect on adult height and other outcomes such as educational achievement, productivity, and income. Stunting is related to a 10% reduction in lifetime earnings and contributes to a loss of national gross domestic product of 2-3 percent for all nutrition problems. The quality of diet (protein, essential 42 fatty acids, micronutrients, and other chemicals) may be important factors in promoting growth in length/height. However, it is clear that malnutrition in children is associated with inadequate feeding practices. While almost all infants are breastfed, only half are exclusively breastfed in the first six months. Feeding children during the complementary feeding period, 6-23 months was not optimal. Only 68% of children were consuming the minimum number of food groups (a proxy for the quality of the diet) in 2008, and only 50% of children were being fed the minimum number of meals per day (a proxy for energy intake). As a result, it is estimated that only 41% of all children are fed a minimally adequate diet in Egypt. Only 42 percent of newborns are weighed at birth and only about half are breastfed in the first hour. While most newborns are screened for iodine deficiencies, other newborn care interventions are not routinely provided. The major causes of neonatal death are preventable or treatable with simple, cost￾effective interventions. Preventing unintended pregnancies, particularly through pregnancy spacing, is a critical component to improve the health, nutrition and survival of both the mother and the infant. According to the 2008 EDHS, 58% of currently married women 15-49 years of age were using a modern method of FP. While contraceptive use in Egypt has been increasing, overall trends in birth spacing have not improved. In Upper Egypt in particular, the median birth- to- birth spacing is only 25 months, meaning half of postpartum women become pregnant before their child is two years old. Data from the 2008 Egypt Demographic and Health Survey suggest that a significant proportion of the Egyptian population seeks healthcare from the private sector, particularly for maternal and child health services. Such health care seeking patterns place a financial burden on patients, particularly those from the poorest segments of society. The economic challenges brought on in the post-2011 revolution era exacerbate financial burdens. According to the gender assessment of the USAID/Egypt health program conducted in August 2010, there are persistent gender inequalities in Egypt that continue to contribute to poor health outcomes of women and children, in particular, but that also affect men’s health. Unequal power relations based on gender are evident within families and communities, and also reflected in health, educational, judicial, and economic institutions. This severely affects women’s capacity to make optimal decisions about their healthcare and the prevention of illness for themselves and their children. For instance, the 2008 DHS reported that only 21 percent of ever married women had knowledge of danger signs during pregnancy and childbirth- vital information that should have been given to them during ante-natal check- ups. Women’s and men’s different roles, normative behavior, and identities may restrict or facilitate their access to and use of health services, prevention of illness, and risk of morbidity and mortality. In Egypt, these differences are based on unequal power and control of resources, which result in women’s and girls’ lower social status and restricted rights, thus limiting their ability to access health services. This is particularly important during pregnancy, childbirth and the postpartum period, when skilled service providers are needed, but also necessary for them to receive Family Planning information and services. Within this context, USAID/Egypt requested the services of USAID/Washington’s global Maternal and Child Health Integrated Program (MCHIP). In 2011, USAID/Egypt, through a field support fund to the global MCHIP, started the MCHIP/ Choices for Healthy Living 43 (SMART) in Egypt. SMART is a two-year unilaterally-awarded initiative, that started on October 1, 2011 and was implemented in six governorates of Egypt- Qalyubia and Sharqia in Lower Egypt, and Beni Suef, Asyut, Qena and Sohag in Upper Egypt. Intervention areas were selected by considering malnutrition rates and low health indicators related to neonatal, child and maternal health. MCHIP/Egypt’s SMART aimed at improving key maternal, neonatal and child health and nutrition behaviors and increasing use of community-based MNCH-FP-Nutrition services. Development Hypothesis The hypotheses underlying the MCHIP project focuses on reduction of stunting and neonatal mortality. By working with local non-governmental organizations that conduct community￾based education programs on a constellation of MNCH and nutrition behaviors, including gender-related considerations, men and women in the target areas will adopt improved individual and family-related behaviors and improved health care seeking practices. These improved health behaviors and practices will lead to reduced stunting and reduced neonatal mortality. Strengthening the capacity of local organizations will enable them to continue educating men and women in their geographic areas and continuing the improvements in health status. Program Expected Results SMART is aligned with USAID/Egypt’s Office of Health and Population results framework, the U.S. foreign assistance framework and the Global Health Initiative. The activity focuses on improving childhood nutrition and decreasing neonatal deaths. SMART’s expected results are: 1. Increase knowledge, skills and practices of healthy maternal, newborn and child behaviors; 2. Mobilize community action, support and demand for the practice of healthy maternal, neonatal and child health (MNCH) behaviors; 3. Increase demand for antenatal care in select communities; 4. Increase demand for post-partum care in select communities; 5. Increase demand for post-natal care in select communities; 6. Decrease childhood malnutrition in select communities; 7. Increase exclusive breastfeeding practices for children under-six months of age; 8. Promote birth-spacing for improved MNCH outcomes; and, 9. Strengthen the capacity of local nongovernmental organizations to implement health communications activities for reduced malnutrition and neonatal death. SMART’s Results Framework is presented below. 44 SMART’s Results Framework Strategic Objective: To increase the use of key maternal, neonatal and child health and nutrition behaviors and use of community-based MNCH-FP-Nutrition services R1.Improved access to and quality of key MNCH-FP￾Nutrition services by private, community-based providers R2. Increased knowledge and use of key MNCH-FP-Nutrition behaviors by women and men R3. Increased capacity of CDAs to implement community-based strategies to improve MNCH-FP￾Nutrition To improve neonatal health and child nutrition outcomes R4. Increased understanding within the development community of the causes, including gender-specific, of stunting and how to implement programs to reduce stunting and neonatal mortality R5.Improved critical awareness of the impact of gender roles in improving MNCH-FP-Nutrition outcomes 45 Program Approach MCHIP/Egypt’s SMART program approach built on previous USAID-supported successful community outreach activities that were implemented under the Communication for Healthy Living (CHL) (http://www.jhuccp.org/whatwedo/projects/communication-healthy-living-chl) and Integrated Reproductive Health Services (Takamol) (http://hci.com.eg/core￾competencies/major-projects/integrated-reproductive-health-services-project-takamol) projects. Each project worked through and with local non-governmental organizations to complement and create demand for public sector health services, and increase adoption of key healthy practices. SMART provided an integrated package of interventions that directly benefits key vulnerable populations including low income households and women and children. It addressed key technical areas such as newborn health, FP, and nutrition, particularly stunting in children younger than two years of age. SMART worked through and with Community Development Associations (CDAs) to improve quality and expand coverage of their existing health services. CDAs are not-for-profit organizations that offer services and engage in other activities that promote and support community development. Egyptian CDAs were established in 1960s and are registered by Ministry of Social Solidarity. The program worked with private health providers at the CDA facility level and through outreach workers in the community. SMART focused on building the organizational and technical capacity of governorate-level umbrella NGOs to enable them to support village level CDAs. SMART also reached out to private pharmacies as a way to improve the availability and use of key commodities such as FP methods, particularly for postpartum women, who have the highest levels of unmet need, and Iron-Folic Acid (IFA) supplements for pregnant women. SMART built the capacity of pharmacists to provide counseling to women and couples on the appropriate use of FP methods and IFA. SMART identified barriers that hinder the practice of key MNCH-FP-Nutrition behaviors through improving knowledge, attitudes and skills of men and women to practice key behaviors. SMART also identified and addressed gender-specific and other barriers and facilitators for improving the desired behaviors. SMART started a four component study to explore the causes of stunting among young children in Lower Egypt as an integral part of SMART’s monitoring of program processes and outcomes. The study aims at providing better understanding of the causes of stunting, describing perceptions of the problem in Egypt, and suggesting how to address the problem. The main objective of SMART in relation to gender was to improve the critical awareness of the impact of gender roles in improving MNH-FP-Nutrition outcomes. Accordingly, SMART utilized transformative gender strategies that examine, question, and seek to change rigid gender 46 norms and imbalance of power as a means of reaching health as well as gender equality objectives to ensure constructive male involvement. SMART implemented a baseline survey, and planned to conduct an end-line survey three months before conclusion of the project. The baseline and end-line surveys aim to provide informative and comparative analysis of the availability, accessibility and utilization of MNH-FP services in the six targeted governorates before and after project interventions. They also document changes in the knowledge, behavior and quality of MNH-FP nutrition services. These two surveys include questions that would reflect the improvement in SMART outcome indictors over its course of implementation. Both qualitative and quantitative approaches are adopted in conducting the baseline and final surveys. The project also uses Control groups, where no SMART interventions are made; these are tracked in the baseline and final surveys. Critical Assumptions The successful implementation of USAID –supported MCHIP/SMART project assumed that:  CDAs have the capacity and maintain a strong willingness for increasing the awareness of population of healthy behaviors; and  Private sector and non-governmental organizations have the enabling environment to improve the availability and use of health services. A. Project Management Modifications SMART was initially planned to end in September 30, 2013. Political and social turmoil in the country followed January 25, 2011 revolution and June 30, 2013 event caused significant delays in the implementation. Initial approvals granted by the Ministry of Social Solidarity to Umbrella NGOs and local CDAs were delayed for more than six months. Street violence and disruption of the railroads services prevented Project’s staff and consultants based in Cairo from traveling regularly to intervention governorates. In May 2013, USAID extended the completion date of the project to December 31, 2013 at no additional cost to allow for completion of the planned activities. However, with the continued unrest, SMART sought USAID approval to extend the project to June 2013 with no additional cost to achieve the following: 1. Complete the Stunting Study and End line Survey; 2. Document/prepare program briefs and publications; and 3. Disseminate project results at governorate, national and international levels. The majority of CDAs closed by the end of December 2013, but selected CDAs working in the areas where the stunting study is taking place will continue activities through March 2014. The final months will also include specific dissemination activities to ensure that the outcomes of the project and study are shared broadly within Egypt. The SMART team will develop strategic messaging for key stakeholders including service providers, local community development organizations, to ensure that momentum started under the project will not be lost at the end of the project. While all programmatic activities (including sub-awards) will end by March 2014, a few key staff at the Egypt MCHIP office will remain with SMART through May 30, 2014, to finalize all 47 dissemination activities as well as programmatic and financial reporting and closeout. MCHIP headquarters staff supporting the program will continue through June 30,2014 close-out to finalize reporting and close out documentation. B. Relevant Documentation The evaluation team should consult a broad range of background sources apart from project documents, USAID/ MCHIP headquarters and SMART staff. These may include documents that relate to MNCH-FP-Nutrition, legacy review of 30 years of investment in Egypt (http://www.ghtechproject.com/files/Egypt_Health_and_Population_Legacy_Review.pdf) , and a gender assessment of the USAID/Egypt health portfolio completed in 2010 (http://www.healthpolicyinitiative.com/Publications/Documents/1410_1_Egypt_Gender_Assess ment_Final_FINAL_acc.pdf) The team may also find the MHIP website and global MCHIP mid-term evaluation (http://www.ghtechproject.com/files/MCHIPReport%20Final%20508%20Document%209.08.11 .pdf) useful. USAID and SMART team will provide the evaluation team with a package of briefing materials, including, in full, prior to the team’s arrival in Egypt:  MCHIP/SMART original concept note a. Project’s agreement and amendments b. Project quarterly reports, annual and ad-hoc work plans and review document developed as part of routine monitoring. c. Budget information d. Gender Analysis e. Project’s Performance Monitoring Plan f. Project’s baseline and line surveys data sets g. Contact information for key informants The evaluation team should use the baseline and end-line raw data and provide independent analyses and comparisons. II. Evaluation Rationale A. Purpose The USAID/Egypt Mission is planning to conduct a performance evaluation of its MCHIP/SMART project. The purpose of this evaluation is to: (1) Review, analyze, and evaluate the effectiveness of the SMART project in achieving program objectives; (2) Identify lessons learned in terms of implementation and relationships with counterparts in order to inform USAID future investments; and 48 (3) Assess the sustainability of the interventions at the community level. Findings and recommendations of this evaluation will guide future investment in the areas of MNCH-FP-Nutrition. B. Audience and Intended Uses The audience of the evaluation report will be the USAID/Egypt Mission, specifically the health team, the Global Health Bureau, the Middle East Bureau, global MCHIP, and stakeholders in Egypt. USAID/Egypt will review and share the executive summary, expanded executive summary, final report, and recommendations (see IV. A. Deliverables) with Egyptian stakeholders working on MNCH-FP-Nutrition, and the general public via the Development Education Clearinghouse (DEC). USAID will address the evaluation report findings and recommendations in future relevant project activities and share lessons learned with other stakeholders. Global MCHIP will incorporate lessons learned to improve future activities in the area of MNCH-FP-Nutrition. . C. Evaluation Questions The evaluation will answer the following questions: 1. Given that the program was designed as a large infusion of funds in a short period of time, what are the most significant factors that enabled or constrained the implementers’ ability to achieve the desired project outcomes? 2. What evidence exists to substantiate suppositions that the project had any positive effects? 3. What are the most significant determinants that influence the likely sustainability of project benefits in target populations? 4. Since the project involved a large amount of funding and a short implementation period, to what extent did the project use its resources efficiently? 5. Considering the project’s design constraints and possible counterfactual alternatives, to what extent does the constellation of project interventions represent the most effective way of reducing stunting and neonatal mortality? III. Evaluation Design and Methodology A. Evaluation Design This performance evaluation is intended to focus on how SMART has been implemented, what it has achieved, and whether expected results have occurred according to the project’s design and in relation to the development hypothesis. The evaluation will focus on identifying lessons learned that will guide future USAID investments. The evaluation will also assess the sustainability of the interventions at community level. Evaluators will use a mix of quantitative and qualitative data collection and analysis methods as well as the project’s baseline and end-line data and analyses to generate answers. The evaluation 49 team will use the project’s baseline and end-line data set and do their independent analysis and comparison. Evaluators will use USAID Evaluation Policy (http://www.oecd.org/derec/unitedstates/USAID_Evaluation_Guidelines.pdf) as a guideline in the evaluation design. B. Data Collection Methods The evaluation team should develop data collection tools that are consistent with the evaluation questions to ensure high quality analysis. The evaluation team is required to share data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. These tools may include a combination of the following: • Desk review of relevant documentation cited above in Section D; • Analyses of the project’s baseline and end-line data sets • Site visits to SMART intervention areas; • Key informants interviews; • Focus group discussions with SMART community outreach workers, beneficiaries, and other counterparts and stakeholders; and • Independent analyses of baseline and end-line data sets. Desk Review The international evaluation team will start work on a paper review of all, but not limited to, resources cited in the “Relevant Documentation” section above, prior to arriving in Egypt. The local evaluation team members should complete the paper review prior to the international team’s arrival. Interviews Key Informant Interviews will include, but may not be limited to:  USAID/Egypt Health Team – including Activity Manager  SMART staff  Global MCHIP key staff  Community participants of SMART The evaluation team will provide a more detailed explanation of the proposed methodology for collecting the data. The Evaluation Team may be accompanied by a staff member from USAID/Egypt, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. C. Data Quality Standards The evaluation team shall ensure that the data collected clearly and adequately represent answers to the evaluation questions. Collected data should also be sufficiently precise to present a fair picture of performance, and at an appropriate level of details. 50 D. Data Analysis Methods Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan that details how focus groups and key informant interviews will be analyzed; what procedures will be used to analyze qualitative and quantitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data from indicators and project performing monitoring records to reach conclusions about the effectiveness and efficiency of the activities conducted by SMART. The Mission expects the evaluation team to present strong quantitative and qualitative analysis, within data limitations, that clearly addresses key issues found in the research questions. The Mission is looking for new, creative suggestions regarding this evaluation, and it is anticipated that the implementer will provide a more detailed explanation of the proposed methodology for carrying out the work. The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information required to assess the evaluation objectives. The methodology will be discussed with and approved by USAID/Egypt Activity Manager and the Evaluation Program Manager prior to implementation. E. Methodological Strengths and Limitations Key informant interviews and focus group discussions are suggested as a primary data source for this evaluation. It is anticipated that some interviews may be conducted in the presence of at least one or more outside observers, including project and USAID staff, and that interview responses could be affected by the presence of these observers. USAID expects that all issues affecting validity be discussed and documented in the evaluation planning stage – including measures to minimize precision and validity issues. Measures to mitigate these issues will be addressed with all team members and USAID team in the implementation phase and detailed in the final report IV. Evaluation Products A. Deliverables Work Plan: During the team planning meeting the evaluation team will discuss a detailed work plan, which will include the methodologies to be used in the evaluation, timeline, and detailed Gantt chart. The work plan will be submitted to both the SMART Activity Manager and the Evaluation Program Manager for approval no later than the sixth day of work. Methodology Plan: A written detailed methodology and data analysis plan (evaluation design, data analysis steps and detail, operational work plan, see sections III. C and D) will be prepared by the team and discussed with USAID during the planning meeting. 51 List of Interviewees and Schedule: USAID will provide the evaluation team prior to the team’s arrival in Egypt with a stakeholder analysis that includes an initial list of interviewees, from which the evaluation team can work to create a more comprehensive list. Prior to starting data collection, the evaluation team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. Data collection tools: Prior to starting fieldwork, the evaluation team will share the data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion and approval. In-briefing and Mid-term brief with USAID: The evaluation team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in-brief, the team should have the list of interviewees and schedule prepared, along with the detailed Gantt chart that maps out the evaluation through the report drafting, feedback and final submission periods. At the mid-term brief, the partner should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. In addition, to facilitate a smooth implementation of the data collection and analysis phases, the evaluation team will be expected to coordinate and communicate with the Mission’s POC on evaluation team ongoing basis. Discussion of Preliminary Draft Evaluation Report: The team will submit a first draft of the report to the USAID Evaluation Program Manager, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Egypt through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of methodology, findings, achievements and issues as well as any conclusions, and recommendations. The team will consider USAID/Egypt comments and revise the draft report accordingly, as appropriate. Debriefing with Partners: The team will present the major finding of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The debriefing will include a discussion of achievements, activities, and recommendations for possible modifications to project approaches, results, or activities. The team will consider partners' comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the team leader’s departure from Egypt. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide comments on the draft report within two weeks of submission. 52 Final Report: The team will submit a final report that incorporates the team responses to Mission comments and suggestions no later than five days after USAID/Egypt provides written comments on the team’s draft evaluation report (see above). If USAID/Egypt determines that its comments on the first draft have not been satisfactorily addressed, it will provide further feedback for the team to address within five days. The evaluation report will be deemed final only with USAID/Egypt’ approval. The format will include an executive summary, table of contents, methodology, findings, and actionable recommendations. The report will be submitted in English, electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. Expanded Executive Summary: The team will submit an expanded executive summary to accompany the final report that will include a background summary on the evaluation purpose and methodology, and an overview of the main data points, findings, conclusions, and recommendations. The expanded executive summary should be easy to read for wide distribution to local audiences and the partner is encouraged to look for creative presentation styles, formatting and means of dissemination. The expanded executive summary will be submitted in English and Egyptian Arabic, in hard copy (50 copies) and electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this evaluation will be presented to USAID on a flash drive to the Evaluation Program Manager. All data on the flash drive will be in an unlocked, editable format. A two-day team planning meeting will be held in Egypt before the evaluation begins. This meeting will allow USAID and the evaluation team to confirm mutual expectations and understanding, for example, around the purpose, expectations, and agenda of the assignment. In addition, the team will:  Clarify team members' roles and responsibilities;  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  Review and develop final evaluation questions;  Review and finalize the assignment timeline and share with USAID;  Present data collection methods, instruments, tools, and guidelines (materials should be developed prior to this meeting);  Review and clarify any logistical and administrative procedures for the assignment;  Develop a preliminary draft outline of the team's report; and,  Assign drafting responsibilities for the final report. B. Evaluation report requirements The format for the evaluation report is as follows: 1. Executive Summary—concisely state the most significant findings and recommendations (3 pp); 53 2. Table of Contents (1 pp); 3. Introduction—purpose, audience, and summary of task (1 pp); 4. Background—brief overview of MCHIP/SMART project in Egypt, USAID project strategy and activities implemented in response to the problem, brief description of SMART, purpose of the evaluation (3 pp); 5. Methodology—describe evaluation methods, including threats to validity, constraints and gaps (1 pp); 6. Findings/Conclusions/Recommendations— for each evaluation question, the report will state findings, conclusions and recommendations in clearly demarcated sub￾sections;, also clear distinctions will be made between findings, conclusions, and recommendations (20-25 pp); 7. Challenges—provide a list of key technical and/or administrative, if any (1–2 pp); 8. Future Directions (3–4 pp); 9. References (including bibliographical documentation, meetings, interviews and focus group discussions); 10. Annexes—annexes that document evaluation scope of work, evaluation methods and limitations, copies of the actual data collection tools, documents reviewed, schedules, interview lists and tables— should be concise, relevant and readable. Annexes should also include a disclosure of any conflict of interest by evaluation team members. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/policy/evalweb/evaluation_resources.html). The final evaluation report will conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy. The Evaluation Program Manager will determine if the criteria are met. This evaluation will not conclude until the Evaluation Program Manager has confirmed, in writing, that the report has met all of the quality criteria. The final version of the evaluation report will be submitted to USAID/Egypt electronically. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 40 pages, excluding references and annexes. V. Team Composition USAID encourages the participation of local experts on evaluation teams, including in the roles of evaluation specialist and team leader. All attempts should be made for the team to be comprised of male and female members. Team members will be required to provide a written disclosure of conflicts of interest (per USAID Evaluation Policy). The evaluation team will be composed of four members- a team leader, two consultants one of whom is a local specialist, and a local logistics coordinator. The evaluation team must have in one or more team member(s) the following experience:  Behavior Change Communication experience;  Nutrition programming experience in developing countries;  Implementing and evaluating similar USAID health programs; 54  Demonstrated experience in Egypt;  Egyptian Arabic fluency;  Demonstrated quantitative and qualitative data analysis skills. All the team members must have Monitoring and Evaluation experience. It is desirable to have in one or more team member(s) the following qualifications:  Medical anthropology / ethnology  Maternal neonatal and child health programming  Community health education programs  Complexity science / systems thinking Offerors may comprise their team as they see appropriate, so long as all required experience mentioned above is represented by an appropriate number of team members. All other factors being equal, maximizing a team’s ability to fulfill more qualifications in the “desirable” criteria might enhance an application. Team Leader: a senior consultant with extensive experience in leading and conducting USAID health program evaluations. S/he should have an MPH or related post graduate degree in public health. S/he should have at least 10 years senior level experience in at least one of the qualifications the evaluation team must have. Excellent oral and written skills are required. The Team Leader should also have experience in leading evaluation teams and preparing high quality documents. The Team Leader will: - Finalize and negotiate with USAID/Egypt the evaluation work plan; - Establish evaluation team roles, responsibilities, and tasks; - Facilitate the Team Planning Meeting (TPM) - Ensure that the logistics arrangements in the field are complete; - Manage team coordination meetings in-country and ensure that team members are working to schedule; - Coordinate the process of assembling individual input/findings for the evaluation report and finalizing the evaluation report; - Lead the preparation and presentation of key evaluation findings and recommendations to USAID/Egypt team prior to departing Egypt Local consultant: The local technical specialist is expected to be fluent/professionally proficient in spoken Egyptian Arabic. The local specialist will assist the team to better understand different cultural and social issues related to nutrition, maternal, neonatal and child health in Egypt. S/he will also assist in communications and interviews with local stakeholders. Local Logistics Coordinator: The Logistics Coordinator should be a local staff member for handling all in country travel related logistics and providing administrative and translation support to the technical team members. The Logistics Coordinator will also be responsible for setting up meetings with USAID and stakeholders. 55 Required qualifications include:  Demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision.  Capacity for effective time management and flexibility.  Must be able to interact effectively with a broad range of internal and external partners, including international organizations and host country government officials.  Must be fluent in both English and Arabic.  Proven ability to communicate clearly, concisely and effectively both orally and in writing. VI. Evaluation Management A. Logistics USAID will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID will assist in arranging meetings with key stakeholders identified by USAID prior to the initiation of field work. The evaluation team is responsible for arranging other meetings as identified during the course of this evaluation and advising USAID/ Egypt prior to each of those meetings. The evaluation team is also responsible for arranging transportation as needed for site visits in and around Cairo and other governorates. USAID can assist with hotel arrangements if necessary but the evaluation team will be responsible for arranging its own work/office space, computers, internet access, printing, and photocopying. The evaluation team is also responsible for procuring and paying for translation services for interviews, reports and any other evaluation related task. Evaluation team members will be required to make their own lodging and travel payments. USAID personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. B. Scheduling Work is to be carried out over a period of approximately 9 weeks, beginning on or about (o/a) June 01, 2014, with field work completed July 3, 2014 and final report and close out concluding o/a July 31, 2014. 56 Timeline and LOE Task Week 1 Week 2 Week 3 Week 4 Week 5 Week 8 Week 9 Planning Detailed Preparations Field Implementation Data Analysis Reports Task Team Leader/ Senior international consultant Senior Local expert Mid-level international expert Logistics Manager Total Days by Task % of total Eval days Planning 10 10 10 3 33 17.93% Detailed Preparations 6 6 6 4 22 11.96% Field Implementation 15 15 15 5 50 27.17% Data Analysis 13 13 13 0 39 21.20% Reports 16 10 14 0 40 21.74% Total LOE 60 54 58 12 184 100.00% 57 II. DETAILED EVALUATION METHODOLOGY The purpose of this utilization-focused performance evaluation is three-fold: (1) To evaluate the extent to which the SMART project achieved program objectives; (2) To identify lessons learned from project implementation and local stakeholder relationship￾building in order to inform USAID future investments; and (3) To assess the sustainability of SMART interventions at the community level. The audience of this performance evaluation will be the USAID/Egypt Mission, specifically the health team, the Global Health Bureau, the Middle East Bureau, global MCHIP and local stakeholders including program beneficiaries in both Upper and Lower Egypt regions. The evaluation will culminate in a comprehensive report designed to be shared with all relevant stakeholders and the general public via the Development Education Clearinghouse (DEC). The evaluation’s action-oriented recommendations and lessons learned are intended to be used by the primary evaluation users to understand the extent of the project’s contribution to improved MNCH outcomes in Egypt, and to incorporate lessons learned in order to maximize the effectiveness of future MNCH-FP-Nutrition programming at the national and regional level. USAID/Egypt will use findings from the evaluation to inform the design of future projects. The evaluation report will be disseminated widely among relevant stakeholders and project beneficiaries as well as submitted to the USAID Development Experience Clearinghouse (DEC). KEY EVALUATION QUESTIONS In pursuit of the complementary objectives, the evaluation will specifically address the following questions: 6. What are the most significant factors that enabled or constrained the implementers’ ability to achieve the desired project outcomes? 7. What evidence exists to substantiate suppositions that the project had any positive effects? 8. What are the most significant determinants that influence the likely sustainability of project benefits in target populations? 9. To what extent did the project use its resources efficiently? 10. Considering the project’s design constraints and possible counterfactual alternatives, to what extent does the constellation of project interventions represent the most effective way of reducing stunting and neonatal mortality? THEMATIC AREAS Four broad themes stand out from the key evaluation questions: I. Project Achievements (has the project achieved its outcomes, have the target been met? Internal and external factors affecting the results) II. Assessment of Project Effectiveness (e.g. appropriateness of interventions, project design including program strategy, effectiveness in reducing stunting and neonatal mortality) III. Assessment of project efficiency (e.g. synergy with other partners, complementarity) IV. Sustainability (e.g. local ownership, behavior change) EVALUATION METHODOLOGY 58 A preliminary review of the PMP indicates that there is a mix of output and outcome level indicators. This evaluation will focus on outcomes, such as those demonstrating the results of the mix both clinical and preventive of interventions that were implemented to address stunting and neonatal mortality. The team will explore in depth the evidence behind those intended outcomes, the reasons behind outcomes that were not achieved and enabling factors for those that were, as well as unintended results. Social Impact’s evaluation methodology will combine a comprehensive, rigorous analysis of existing quantitative data with customized qualitative techniques designed to elicit primary data from a wide range of counterparts, partners, beneficiaries and other stakeholders. This mixed-method approach will allow for the triangulation of complementary data to elucidate linkages between project inputs, outputs, and outcomes. The evaluation team will analyze quantitative and qualitative data in the context of the MCHIP results framework to investigate the extent to which evaluation findings substantiate the logic underlying MCHIP’s development hypothesis, including the sustainability of interventions at the community level. Specifically, the team will use: 1) secondary data and existing project information, such as quarterly and annual reports and other technical reports, baseline and endline household data, and all project databases; and 2) primary data collected through detailed KIIs, and focus group discussions FGDs. The SI team recognizes that nutrition and child health are inexorably linked to gender norms in Egypt. Indeed, the foundation of MCHIP’s activities is derived from the intent to improve awareness of the interplay between gender roles and key MNCH-FP outcomes. In acknowledgment of the critical role gender plays in the achievement of MCHIP objectives, as well as the extent to which MCHIP activities have influenced broader gender considerations, the SI team will employ customized gender-focused qualitative inquiry. Specifically, qualitative interview guides for key informants and focus group participants will include questions designed to elicit information on perceptions of gender roles with respect to MNCH-FP and nutrition. The evaluation will use this data in conjunction with MCHIP’s gender analysis and household survey data to present a detailed picture of MCHIP’s performance framed within the context of delicate cultural nuances. To the extent possible, the evaluation will also analyze and present sex-disaggregated data, in accordance with USAID Evaluation Policy. DATA COLLECTION METHODOLOGY Mixed methods The evaluation team will use a mixed method approach (both quantitative and qualitative techniques) to address the key evaluation questions and the sub-questions, ranging from review of secondary data from e-library, to collecting primary data through interviews and focus groups discussions, site visits and direct observations; conducting organizational assessments of a sample of NGOs/CDAs (UCDAs and/or LCDAs) that participated in SMART (to measure the actual increase in the NGOs/CDAs’ institutional capacities; as a result of the training, technical assistance, and resources they received from SMART). To address each of these Key Evaluation Questions, the evaluation team will rely on a variety of data sources and data collection methods. The Data Collection and Analysis Matrix in Annex 1 is organized around each of the evaluation’s five Key Questions (KQs), and provides a description of data collection methods to be used. Annex 1 further describes the variety of data to be extracted and analyzed, the sources(s) of that data, and the types of analyses that will be undertaken to inform findings and conclusions. The main data collection instruments are as follows: 59 (a) Desk Review of Documents. The e-library consists of project documents, including strategic documents (proposals), past evaluations, work plans, various reports (baseline assessment, quarterly and annual), operational documents, partners reports and other related M&E documents. The evaluation team (ET) will assess the extent to which this secondary data can be used to answer the evaluation questions and then identify data gaps which need to be addressed as primary data collection during the field visit. (b) Collection and review of secondary data in the field. In addition to the desk review before the field visit, the ET will gather additional documents from stakeholders and partners in the field. From these they will extract the quantitative and qualitative secondary data which will serve as the key sources of information for this evaluation including country level performance data for the various interventions in which MCHIP was engaged. An assessment of data constraints will also be documented and whereas possible the ET will mitigate. (c) Site visits. Sites will be selected in order to assess the performance of the MCHIP interventions using selection criteria that represent the geographical regions, the package of interventions and gather perspectives from a wide range of stakeholders in order to fill data gaps identified by the ET as well as gaps in MCHIP’s programming. QUALITATIVE METHODS Key Informant Interviews Key informants constitute main data source for this evaluation. Findings gleaned from key informant interviews provide critical contextual data with which to gain detailed understanding of MCHIP effectiveness, as well as to triangulate with existing quantitative data. Key Informant Interviews will include, but may not be limited to: (See Annex 4 for stakeholder analysis):  USAID/Egypt Health Team – including Activity Manager  SMART staff  Global MCHIP key staff  Community participants of SMART (health facilities, community health workers, etc.), users and beneficiaries of the services;  Board and staff members of umbrella CDAs that lead the NGOs/CDAs participation in SMART Illustratively, the SI team will ask community participants of SMART (e.g. UCDAs’ board and staff members; community leaders; community health workers [CHWs] and beneficiaries) for their perceptions of a CDA or health facility’s capacity to sustain the project’s objectives beyond SMART closeout. The team will also ask CHWs and direct beneficiaries about the perceived effectiveness of SMART activities, providing invaluable data on the actual change in the KAP level of the direct community beneficiaries that may not be captured in quantitative surveys alone. CHWs are uniquely positioned to speak to the impact of SMART-facilitated trainings; thus, additional lines of inquiry will be related to which SMART interventions are perceived as most beneficial, and where demand for specific skillsets remains unmet. Similarly, SMART project staff are well suited to speak about the determinants of implementer’s ability to achieve project outcomes, further addressing the evaluation’s key questions. SMART staff can also provide detailed information on resource efficiency and lessons learned given their experience working directly with CDAs. The SI team will pursue these inquests as part of a preliminary work plan, with further proliferation following a comprehensive document review and consultation with USAID/Egypt. The list (Annex 4) will integrate gender considerations to allow generation of information on women (women’s groups), men, boys and girls from different groups (e.g. beneficiaries, implementers, etc.) and avoid the reinforcement of gender discrimination and unequal power relations. 60 It is the intention of the ET that the key informant interviews be conducted at the office of the interviewee (although alternative arrangements can be made if necessary), and, based on the semi￾structured interview guide, last between 45 minutes to one hour. The SI team will use a semi-structured questionnaire guide to gather the views of the stakeholders on the key themes of the evaluation exercise to allow adequate and uniform coverage of topic areas while encouraging the natural evolution and expansion of the iterative qualitative data collection process. The guide has been designed with universal questions that elicit detailed description for the relevant evaluation questions, and the team will tailor the questions to each type of stakeholders using the rationale for selecting the interviewee and knowledge of their context. While the evaluation team expects some interviews to take place in English, the inclusion of local Egyptian specialists on the evaluation team will allow interviews to be conducted in Arabic, when necessary. The review of program data, in consultation with USAID/Egypt, will inform the selection of participants for key informant interviews and the protocols for these interviews. The team acknowledges two inherent biases associated with this type of data collection. One limitation is the possibility of recall bias amongst key informants. The team will take steps to reduce recall bias in the protocol design phase. This will include framing questions to aid accurate recall. Where possible, the team will use alternative sources to corroborate interview findings. The team also acknowledges the potential for bias due to subjectivity of respondents. In order to address this potential bias, the SI team will purposively recruit a diverse sample and triangulate responses with other data sources. Since the team will not be able to avoid all bias in the data, persistent biases will be accounted for during the analysis and interpretation phase and will be well-documented during analysis and dissemination. The SI team will employ qualitative analysis software Dedoose as a means of constructing response categories and identify patterns in data. Coding qualitative data through using electronic software will allow the evaluation team to analyze interview transcripts with speed and efficiency, easily cataloging and documenting emergent themes from among respondents. Focus Group Discussions (FGDs) FGDs are particularly useful for supplementing KIIs and quantitative data by gleaning valuable information from discussions among group participants. The evaluation team will conduct FGDs with various cadres of MCHIP stakeholders, from a sample of MCHIP-supported CDAs and health facilities, to gain an in-depth understanding of their experiences with project interventions, and any large grouping that occur within its operations (i.e. health workers, mothers, men, and others as identified by the ET). Each FGD will consist of a group of participants that have common characteristics of their engagement with MCHIP – type of intervention/benefit received and also taking into consideration balanced views by geographical distinction (region, governorate, etc.). The team will also facilitate a community mapping exercise in which community beneficiary women draw the locations of local family planning or antenatal care services. The objective of this exercise is to inform the evaluation team about the knowledge that these women have about those types of services and also serve to confirm that the women know where to get services when they need to. Likewise, a focus group composed of community men can elicit valuable information about the degree to which men are aware of and participant to postnatal child health care. These types of exercises are essential to determining the effects that SMART interventions have had on health provider and beneficiary experiences and service preferences, which will prove useful to the design of future MNCH-FH￾Nutrition programming in Egypt. The ideal number of participants for the FGDs is between 6 and 12, which allows for a wide discussion of opinion without over-crowding one another. During the FGDs, the convening member of the team will guide the process using a discussion guide to ensure that the discussions remain relevant, but will encourage participants to elaborate on key points that they make so that depth can be achieved in the responses. The team member will ascertain that opinions are representative of the whole group and encourage wide participation, rather than relying on answers of the most vocal. 61 Findings from FGDs will provide insight into perceptions of the SMART approach and its effectiveness, as well as how SMART activities have influenced behavior change at the household and community levels. FGDs are subject to biases similar to those common to KIIs (i.e. recall bias and subjectivity), and have the added challenge of being dominated by the most powerful voices in a group. Power dynamics between individuals based on status and sex will be a key consideration for the evaluation team when constructing and moderating focus groups. Sex-disaggregated focus groups, namely among groups of community beneficiaries, may be used in order to mitigate challenges of this kind. Data from FGDs will be transcribed and coded using qualitative analysis software, which lends itself to rapid and efficient analysis and reporting for short-term field evaluations. Site Visits Site visits will allow the evaluation team to confirm data found in project reports and/or key information interviews, as well as to make pertinent observations of facility and staff performance. Site visits also present the opportunity to glean information directly from community beneficiaries; consenting expectant mothers visiting a health facility may be asked about antenatal care access and quality, as well as MNCH-FP-related gender dynamics in the household. A USAID staff member may be requested to accompany the evaluation team during field visits as an observer; however, in order to maintain the objectivity and independence of the evaluation, this individual will not directly participate in data collection. The evaluation team will travel to the six governorates in which SMART was implemented – Qalyubia and Sharqia in Lower Egypt, Beni Suef, Asyut, Qena and Sohag in Upper Egypt in order to conduct site visits at participating CDAs and health facilities. The selection of facilities and communities to be visited will be determined in consultation with USAID/Egypt and with inputs from MCHIP and SMART staff. From a complete frame of SMART-supported facilities and communities, the evaluation team will purposively select a sample of sites based on criteria developed in conjunction with USAID/Egypt. Selection will include factors such as:  Local CDA: A selection of both high and low capacity performing and low performing facilities will help to understand factors contributing to success and barriers to achievement.  Logistical convenience within districts: ease of traveling, location of the facility, security considerations, etc.  Budget for the evaluation exercise  % of the district targeted  Availability of UCDA Selected Districts for Site Visits Governorate Selected Districts Who/Where/What Lower Egypt Who: (UCDA, LCDA, Beneficiaries, CHW, Pharmacists) Where: One (1) village per district What: One (1) health center per district Qalyubia, Qanater Sharqia, Belbeis Abo Hammad Upper Egypt Asyut Asyut El-Fath Beni-Suif Beni-Suef El-Fashn Qena Qous Naqada Sohag Sohag El Maragha Mapping of Stakeholders for Key Informant Interviews (KII) and Focus group discussions (FGD) No Type of respondents KII/FGD Sample Tool # Used 1. Umbrella Community Development Association (UCDA) Board members (Chair, Treasurer, Secretary, Executive director, other staff) FGD 42 2 2. Local CDA Executive Director, Health facility staff (doctor, nurse) FGD 42 2 CHW FGD 140 4 3. Pharmacists FGD 10 3 4. Beneficiaries Pregnant women and recently delivered, Mothers in law, husbands FGD 210 1 5. SMART Staff M&E, COP, Team Leaders KII 8 6. Local Partners * KII 18 7. USAID KII 5 8. International NGO partners KII 5 9. MoH (National – Cairo) KII 4 10. SMART Technical Consultants KII 5 QUANTITATIVE METHODS Independent analyses of baseline and endline data sets The existence of baseline and endline household survey data, among intervention and control communities, provides a potentially strong source of quantitative evidence of SMART project effects. In order to be considered a true control group, non-intervention communities must possess a high degree of similarity in observable characteristics to communities in which SMART was implemented. If the non-intervention communities surveyed do not constitute a suitable control group, it will not be possible to attribute changes in intervention community health outcomes directly to SMART. However, non-intervention communities may still be used to illustrate important differences between communities and their respective uptake of SMART behavior change strategies regardless of a lack of statistical significance. Such inferences would affect the evaluation team’s ability to generalize results beyond those areas in which SMART was implemented. Pre-post intervention analyses may be conducted using baseline and endline household surveys, which will allow for changes in key program outcomes to be mapped over time. Program effectiveness may be gleaned from changes in outcome indicators, such as the percent of children 6 – 24 months who are underweight or stunted. Assuming data quality can be independently verified by the evaluation team, estimations of the number of cases of underweight or stunted children averted may also be possible. Analysis of data from existing databases, such as the Community￾Based Routine Information System and Facility-Based Routine Information System, if available, will be used to provide additional analysis of project achievements against targets. Given the potential that documented health outcomes may not be directly attributable to SMART interventions, the evaluation team will take several steps to mitigate this limitation. First, the evaluation team will review SMART data quality procedures to ensure the adequacy of household survey data, as well as all project data. The evaluation team will then employ contribution analysis as a means of identifying all possible explanations for observed project outcomes. Volatile political or economic conditions, for example, may affect child stunting independent of SMART activities, which would be documented as a threat to valid causal attribution. Similarly, changes in observed outcomes may be due in part to other programs in the same area or environmental factors producing population variability. The evaluation team will investigate the most probable contributions to observed project results and the degree to which such contributions are likely to have influenced results, which will be documented in the final evaluation report. ANALYSIS PLAN The ET has assessed both the availability and the quality of the data during its initial desk based review and will continue this with information received during the evaluation period. Using a Data Summary template, the content of the KIIs and FGDs responses will be assigned into categories based on the evaluation themes/questions. The categories are then analyzed for frequency of responses from stakeholders in order to identify the main messages. Once this is done the primary qualitative information can be compared with the secondary quantitative information to interrogate, corroborate and expand on the findings from the secondary sources and then draw conclusions. This process will be ongoing during the evaluation so that key themes in the responses can be extrapolated for the production of the preliminary findings and recommendations at the end of the field visit. Triangulation and complementarity methods as per Stern et al, 2012, definitions will be used to check and clean the data collected.30 Information for each sub question will be gathered and used to remove outliers, irregularities and subjective responses, fill information gaps, and determine the reliability of the data contributing to the recommendations. Where similar findings are obtained across the different data collection methods, the team can confirm the credibility of the results and demonstrate the confidence it has in the eventual assessments and recommendations. Any findings that the team comes across, but which have not been corroborated through the triangulation or complementarity methods – such as suggestions from single sources for future programs -- will contain a note describing that the data is from a single source and the reason for its inclusion. However, to avoid this, the ET will make every effort to reinforce the reliability of the information, and will perform further document review. The ET will use multi-methods - including tables, graphs, photos, network maps, diagrams, and case studies - to display the data behind the findings in evaluation report. Summary narratives for each interview will be used to outline the salient issues and each will be linked to existing secondary data. During the evaluation, the summary narrative will be used to identify new questions that require further exploration and these will be added into evaluation plan. Recurring themes/ideas will be coded in broad categories to facilitate drawing conclusions. Data from the questionnaire will be processed using Excel and STATA statistical package to compile summary statistics from the findings, which will be presented in tables and graphs. The team will also use existing graphs, maps, diagrams to process the newly collected information so that the findings can be displayed in the geographical coverage of the MCHIP operations. Tables will be used to summarize the number of beneficiaries (targets groups) and stratified by gender, age groups, and activity in the targeted geographical areas. When possible, photos will depict actual project sites with beneficiaries and other activities. The recommendations in the evaluation report will be based on the measured achievements of the MCHIP program and be linked where appropriate will be linked, where appropriate. QUALITY ASSURANCE In order to ensure that data of the highest quality is collected and analyzed, the evaluation team will first consult with MCHIP and Mission staff to determine the extent to which available data is complete and likely to be accurate. The identification of potential weaknesses in available data at the onset of the evaluation will aid the team in focusing refining its data collection tools to ensure that data gaps are adequately addressed. The consistent triangulation of quantitative and qualitative data in the data analysis phase will ensure that findings are drawn from evidence of the highest possible quality. 30Triangulation confirms and corroborates results reached by one method with other results reached by another method. For instance, when beneficiaries of a project’s services state that they judge it good (or bad); this can be cross-checked by collecting quantitative data on coverage and accessibility of the service.” Complementarity refers to the way in which results obtained by a method help better understand those obtained by another method. In-depth theory based approaches (such as focus group discussions and key informant interviews) will allow the ET understand reasons why a project led to unexpected results; qualitative methods may help clarify concepts and define variables; large-scale data sets may be analyzed by multivariate and case-based methods to provide a context within which a small number of intensive case studies can be interpreted.” SI employs a three-stage QA process for all of its evaluations to ensure high quality, evidence-based results that are useful for program improvement, accountability, and learning purposes. Each stage of the evaluation is reviewed and vetted through checklists and direct feedback is given to the Team Leader and field team. Stage I: Work plan — The Senior Technical Advisor will review the feasibility and rigor of the proposed methodology and work plan and adequacy of the dissemination plan. Stage II: Draft Evaluation Report — Report structure and logical linkages among findings, analysis, conclusions, presentation of qualitative and quantitative data, and actionable recommendations will be assessed. Stage III: Final Report — A 40-point quality check of the executive summary, program, and methodology description; adequacy of findings analysis, conclusions, and final recommendations; full compliance with USAID evaluation policies; and report presentation, e.g. charts, graphs, and annexes will be conducted. REPORTING Following fieldwork, the evaluation team will prepare and deliver a presentation to USAID/Egypt consolidating data collected into formulation of preliminary findings, conclusions and recommendations. Based on feedback from the presentation, the team will draft a high-quality evaluation report consistent with the standards set forth in USAID’s Evaluation Policy. Following acceptance of the final evaluation report, the ET will submit to USAID all qualitative data (in the form of summary notes) and details of quantitative analyses. III. FINAL EVALUATION TOOLS HEALTH FOCUS GROUP DISCUSSION (FGD) GUIDE Tool 1: Community beneficiaries Date: ________________________ Location (Governorate/District/Village)_____________________________________ Purpose: To understand how the community appreciates SMART (ANC, ENC, Nutrition, raising awareness). To identify new practice adopted, health status, if any. Sustainability of behavior change, services, etc. Recommendations for future programs related to unmet needs. Participants: Mothers who have recently given birth during SMART implementation and who had children before SMART, Mothers In Law, Husbands, Community leaders. 1. Tell me your impressions about the home visits from the CHWs? The mobile clinics? 2. Did the CHWs influence you to do something different? a. Probe 1: Change in behavior b. Probe 2: Change in practices 3. Is there anything else that you wish that the CHW had done during the home visit? If so, what? 4. Has the CHW made your health better? If yes, how? If not why not? [Alternative question: can you tell me how you have benefited from the SMART project/program? Give 2 to 3 examples] 5. Do you think that mobile clinics should be part of a community based health care system? If so, will you be you willing to pay for that service? [For Mothers: who previously has children and also gave birth during and/or after SMART – in order to compare their experience and also to help measure how the program] 6. How has SMART made a difference in your experience during this recent pregnancy in comparison to previous pregnancies? [For Mothers in law – to gauge how their belief systems have changed vis a vis the SMART messages on ENC, ANC, Nutrition, etc.] 7. What if anything has changed in your belief system? [For Husbands of the mothers who have recently given birth] 8. Have you noticed any difference in care, services in your wife recent pregnancy/delivery during SMART? 9. Are you willing to pay for services in the next pregnancy? 10. Do you think that the new practices adopted during SMART will continue? If so, how? (Probe: will the mothers teach the children?] Tool 2: Community Development Association (CDA) Umbrella and Local ) Date: ________________________ Name of CDA:______________________________________________ Location: (Governorate/District)_____________________________________ Purpose: To understand the CDA‘s perception of SMART in terms of benefits to the organization itself, the community at large, the effectiveness of the program, factors that affected the results, lessons learned and perspective for the future (sustainability and recommendations for future programs). Participants: Board members, staff (service providers, executive directors). 1. Factors that influenced how SMART was implemented and the results in your district? a. Probe 1: Internal factors b. Probe 2: External factors c. Probe 3: What could have been done to overcome constraints? 2. In your opinion, how can the CDA capitalize on these successes? a. Probe: future plans of the CDA 3. What changes have you observed in the community as a result of SMART? Give two to three examples. 4. What effect did SMART have on your organization? a. Probe: strategic plan, mission, quality of services, higher coverage, access 5. What resources currently exist in your community that can be used to capitalize on some of the SMART activities going? What are they? 6. How could SMART have helped you to achieve greater impact? 7. Please describe the relationship with Local CDAs 8. Did you benefit from the trainings workshops, direct TA from SMART staff located in your office? a. Probe: effect on systems (management, financial) mission, etc. Local CDA: describe how the TA from the program and grant officer made a difference, if any, in your work? Give examples. Lessons learned: What would you do differently? Tool 3: Pharmacists Date: ________________________ Name of Pharmacy:________________________________ Location: (Governorate/District)_____________________________________ Purpose: To understand how the pharmacist benefited from SMART. Participants: pharmacist who were trained by SMART 1. What made you interested in participating in the SMART project? 2. What was your contribution to SMART? 3. Are you still practicing anything that you learned from SMART? If so, what? What are your motivations for doing it? 4. What are the most frequently asked questions that you get from women (pregnant, mothers, husbands). Tool 4: Community Health Worker (CHW) Date: ________________________ Location: (Governorate/District)_____________________________________ Purpose: To understand the work of the CHWs in the communities, their perception of change in the community as a result of SMART. Participants: CHWs 1. What made you interested in participating in the SMART project? 2. What was your contribution to SMART? 3. Are you still practicing anything that you learned from SMART? If so, what? What are your motivations for doing it? 4. How have you benefited from the training received by SMART? Give examples. 5. What changes if any have you seen in the community? Please give specific examples? 6. What was your relationship with the CDA? a. Probe: Supervision, capacity building, other support received by CDA 7. So now that SMART has ended, what are the implications for the community and for you? a. Probe: what practices will be easier to sustain Lessons learned: What would you do differently? Tool 5: Doctors/Nurses Date: ________________________ Location: (Governorate/District)_____________________________________ Purpose: To understand the perception of the service providers (doctors, nurses) outside of the CDAs who have received training from SMART. Participants: Service Providers (doctors, nurses) including other partners. 1. What made you interested in participating in the SMART project? 2. What was your contribution to SMART? 3. Are you still practicing anything that you learned from SMART? If so, what? What are your motivations for doing it? 4. How have you benefited from the training received by SMART? Give examples. 5. So now that SMART has ended, what are the implications for the community and for you? a. Probe: what practices will be easier to sustain Lessons learned: What would you do differently? QUESTIONNAIRE FOR DOCTORS AND NURSES TRAINED BY SMART Objective: To assess how doctors and nurses have benefited from SMART training. For example, if they have changed anything in their practice or how they have disseminated the information received. A core of nurses and doctors received TOT with the intention to train other service providers who in turn were expected to hold “seminars” for others. Name: Organization: Nurse _____ Doctor______ Specialty of Medical Practice:_________________ 1. What SMART training did you receive? a) TOT b) Helping Babies Breathe (HBB) c) Healthy Nutrition d) Care of the newborn e) Care of the sick child f) Pneumonia g) Care of premature or underweight h) Breastfeeding practices/skills i) Exclusive breastfeeding (EBF) (NOTE: We really want to hear them use the term “exclusive”) j) Feeding practices for children (supplementary feeding) k) Kangaroo Mother Care l) Infection Control m) Other_____________________________________ 2. As a result of the training received, did you do something different in your work? a. Yes _____ No______ 2.1 If yes, which training made you do something different in your work/practice? a) Helping Babies Breathe (HBB) b) Healthy Nutrition c) Care of the newborn d) Care of the sick child e) Pneumonia f) Care of premature or underweight g) Breastfeeding practices/skills h) Exclusive breastfeeding (EBF) (NOTE: We really want to hear them use the term “exclusive”) i) Feeding practices for children (supplementary feeding) j) Kangaroo Mother Care k) Infection Control l) Other_______________________________ 3. If you did TOT, did you train others? a. Yes____ b. No______ 3.1 If Yes, How: i. on the job training ii. seminars iii. formal training workshops iv. Other__________________________ 3.2 If yes, what subject: 3.2.1 Helping Babies Breathe (HBB) 3.2.2 Healthy Nutrition 3.2.3 Care of the newborn 3.2.4 Care of the sick child 3.2.5 Pneumonia 3.2.6 Care of premature or underweight 3.2.7 Breastfeeding practices/skills 3.2.8 Exclusive breastfeeding (EBF) (NOTE: We really want to hear them use the term “exclusive”) 3.2.9 Feeding practices for children (supplementary feeding) 3.2.10 Kangaroo Mother Care 3.2.11 Infection Control 3.2.12 Other_________________________ 4. Recommendations for future programs: (We would like at least one). a. __________________ b. __________________ c. ___________________ CDA “STATUS UPDATE” MATRIX SENT TO UMBRELLA CDAS FOR COMPLETION CDA Assessment Update Governorate/District Name of Local CDA Duration of SMART Activities (Months) CDA owned clinic (Y/N) Referral to private and public clinic (Y/N) Currently Working or Operational (Y/N) Current Mission includes MCH (Y/N) Currently conducting mobile clinics (Y/N) Worked with pharmacists trained by SMART (Y/N) Exchange Visits (Local/reg ion) L/R/Both Type of other projects (Sector) Source of funding Private/ Grant/ other V. SMART PERFORMANCE MANAGEMENT PLAN (PMP) OUTPUT INDICATORS SMART TARGET BASELINE ENDLINE Outcome Indicator: % of newborns, delivered in the last 2 years who received Essential Newborn Care (ENC) 17.5% 22.5% 69.8% 1.1 Number of women and children receiving MNH-FP services from mobile teams organized by CDAs. NA 38,000 7,000 1.2 Percent of CHWs who can correctly identify the seven danger signs for newborns 28.0% 52.8% 95.0% 1.3 Number of persons trained: doctors, nurses and CHWs (male and female) by SMART on MNH-FP Nutrition services - 5,525 1,500 1.4 % of women with children under 2 who consumed 90+ iron-folate tablets during their last pregnancy (children who were delivered prior to the baseline/end line surveys 31.4% 34.1% 50.0% 1.5 % of women with children under 2 receiving at least four ANC visits from trained health personnel during their previous pregnancy 75.3% 87.7% 79.4% 1.6 % of women with children under 2 who had a medically assisted delivery (doctor, nurse, midwife) 89.6% 99.7% 91.0% 1.7 % of mothers with children under 2 who received their first postnatal care home visit within two days of delivery 35.0% 62.0% 75.0% 1.8 % of mothers with children under 2 who received their first postnatal care home visit within seven days of delivery 23.1% 30.4% 75.0% 1.9 % of women with children under 2 whose newborns received a postnatal care visit at home within two days of birth by CHW NA 41.7% 75.0% 1.10 % of husbands and wives who received at least one FP counseling session during pregnancy Women: 50.7% Men: 14.5% Women: 80.1% Men: 34.7% Women: 60.0% Men: 30.0% Outcome Indicator: % of children 6 -23 months who are underweight (low weight for age) 15.3% 12.2% 6.1% OUTPUT INDICATORS SMART TARGET BASELINE ENDLINE Outcome Indicator: % of children 6-23 months who are stunted (low height for age) 22.3% 9.4% 2.1 Percentage of mothers with children under 2 who are currently using a modern method of FP 72.2% 67.9% 72% 2.2 % of women received at least 90 IFA/folic acid tablets with improved level of hemoglobin in their third trimester of pregnancy NA 89.3% 10% (this one is not correct; it meant 10% increase) 2.3 % of women with LBW (2500g) newborn who practicing Kangaroo mother care for 24 hours/7 days for less than 7 days of baby life 21.8% 24 hours/7 days for the first 7 days of baby life 2.7% 5% 2.4 % of women with children under 2 who can identify at-least 3 danger signs of newborns 14.5% 55.7% 25% 2.5 % of women with children under 2 who sought care from a health care provider for newborns with danger signs 83.8% 82.5% 25% 2.6 % of women with children under 2 with diarrhea in last 2 weeks who provided appropriate care 30.7% 96.5% 10% increase 2.7 Prevalence of children 6-23 months receiving minimum acceptable diet 37.3% 47.9% 3% increase 2.8 % of mothers with children under 2 withholding pre-lacteal feeds 57.1% 53.6% 45% 2.9 % children under 2 who are exclusive breastfed in the first 6 months 50.8% 55.6% 37% 2.10 % of mothers initiating BF within 1 hour of delivery 35.6% 56.8% 40% INDICATOR SOURCE ACHIEVEMENT TARGET Outcome Indicator: % of all LCDAs mobilizing non-SMART resources in￾support of improved MNH-FP-Nutrition services and outcomes in their communities Organizational Assessment tool 100% 80% 3.1 #of UCDAs who are capable of conducting gender analysis at the community level Gender Analysis report 6 6 3.2 % of LCDAs successfully completed the program’s interventions for MNH-FP￾Nutrition services as defined in the work plan Quarterly progress reports 100% 90% Outcome Indicator: % of women in the communities with increased awareness about the causes of stunting Regular monitoring 85.8% 70% 4.1 % of LCDAs implementing interventions to reduce stunting CDA assessment 100% 95% 4.2 # of dissemination events conducted to share the results of the stunting study Event report – to be finalized - 4 4.3 # of new messages designed based on research findings of the stunting study Messages developed – to be finalized - 4 4.4 A set of recommendations developed based on the results of the study about causes of stunting and shared with stakeholders. Set developed – to be finalized - Outcome Indictor: Number of community-based committees who actively disseminate gender-related messages to enhance women’s status and protect women’s inviolable rights and needs Regular monitoring 84 75 5.1 Number of fathers and mothers-in-law of who have participated in at least one family solidarity session linked to SMART-supported MNH-FP-Nutrition interventions Regular data collection 10,000 men 10,000 MiLs 5.2 Number of SBCC materials reflecting the gender relations and notions of masculinities that are developed jointly by the community-based committees and married couples to be widely disseminated at the community level. Final report of the nutrition study – to be finalized 90 70 Baseline Endline 5.3 % of men and women who know three advantages of healthy timing and spacing of pregnancies Household survey Women: 65.5% Women: 84.4% Men: 87% 25% VI. SUMMARY FINDINGS FROM KEY INFORMANT INTERVIEWS AND FGDS EVALUATION QUESTION 1: FACTORS AFFECTING RESULTS a) Short implementation of project activities: Preparation phase was too long and therefore decreased preparation phase. b) SMART recruited its own CHWs instead of using the existing MoH Raedat Refeyat. The MoH Raedat Refeyat are paid a monthly salary and are based at the local health units and/or family health units. The MoH is currently planning to upgrade about 2000 existing local health units into family health units. c) SMART had sound technical inputs from local technical consultants and international partners on the consortium. d) SMART adapted guidelines and IEC education materials from previous USAID project (Healthy Mother and Health Child) which were already approved by MoH and updated them. This strategy saved time and money for additional testing. Also this strategy helped save time in terms of waiting for approval from MoH. e) SMART worked directly with CDAs to enter communities instead of waiting for MoH approval and using the MoH system to kick start the project. Working with CDAs who already had presence and recognition among the communities to increase outreach to vulnerable groups. f) We also learned that CDAs have mixed capacities and this issue needs to be further investigated. g) Reluctance of MoH to ‘join’ SMART may have affected the stat up phase therefore causing some delays. h) There is a general impression that there may have been too many partners at the international level (in the consortium) which also play a role in the delay of the project. i) Finalization of the PMP which took some time was also challenged by too many inputs from the different partners on the consortium. j) The political atmosphere in Egypt at the time of the project was difficult for civil society in particular; though it was less stressful for the health sector, it was still not looked upon favorably if local NGOs partnered with international NGOs because the partnership was considered instigative in nature. The fact that SMART was a health project, it was more acceptable for the political climate at the time. k) In additional personal relationship (between SMART COP and MoH ministers at the governorate level) facilitated the acceptance of SMART at the MoH at the governorate level. l) The high turnover of government cabinets during 2011to 2013 did not help the project gain traction at the MoH (national level) since each minister had different priorities and the short tenure at the cabinet did not allow them to fully appreciate the contribution of SMART during their tenure. m) Working with the universities was very critical more many reasons: sustainability; access to service providers to get a chance to retrain them and influence the curricula at those universities. EVALUATION QUESTION 2: EVIDENCE OF THE RESULTS a) There are anecdotes that the updated guidelines and IEC materials have been incorporated into universities curricula. b) The awareness raising around maternal and child health issues has “gone well”, some topics more than others; the perception is that FP scored the lowest in terms of uptake of services and ANC, ENC postpartum care scored the highest, and Nutrition scored medium. Note: Though the team intended to verify data from registers of the mobile clinics to compare results before SMART and after SMART, it was not possible. The data was not collected as such. c) It is not clear to what extent raising awareness has been translated into specific behavior change. d) Also it is not clear how the CDAs, the service providers (doctors and nurses) have improved their skills and capacity to provide better services. e) Incomplete data on mobile clinics: Need to know about the mobile clinics in terms of services provided. (Ideally, if there were mobile clinics before SMART, it would have been interesting to compare the uptake of services at mobile clinics after SMART started). It was mentioned that the number of mobile clinics events was more than what the project intended since it was appreciated by the community and in high demand. f) The analysis of the longitudinal study data is still incomplete at the time of the evaluation. g) Clear changes in mothers in caring for themselves and for their newborn h) CHWs seems to have been effective in disseminating to mothers i) CHWs have gained the trust of the mothers, husbands and mothers in law j) Several mentioned how the mothers in law have advised to take the sick child to the doctor k) Many mentioned wanting to have other permanent facilities as alternative to government clinics; it seems that even after upgrading the health facility to family health units, there is still distrust in government health system l) Key practices that seem to have changed: a. Care for themselves (e.g. taking folic acid and iron supplementation, TT2 compliance, better hygiene practices, better nutrition) b. Exclusive breastfeeding (EBF) c. Feeding practices d. Greater confidence in caring for the children Capacity of the CDAs 1. CDAs don’t seem to have long term plan to incorporate maternal and child health in their mission 2. There is mixed capacity among the CDAs; it seems that the CDAs were able to get just the necessary skills to implement SMART; so beyond SMART, it is not clear what they plan to do 3. Reporting does not seem to have been strong EVALUATION QUESTION 3: SUSTAINABILITY a) Seems to be the weakest point of SMART b) Institutional Capacity: even though the CDAs have been trained, there is a general sense that their capacities are mixed. It is not clear how they will continue to sustain the health activities undertaken by SMART c) Also it is not clear that the trainings of the services providers are translated into skills (the know-how) even through their knowledge has been “updated” on clinical protocols. d) Beneficiaries: not sure to what extent they have changed their behaviors and if they have which behavior they will sustain. e) Not clear how and if the materials that the universities have taken from SMART will be institutionalized. f) Financial sustainability of the CDAs is questionable. g) Not sustainable; LCDAs will not be able to support the CHWs with home visits, training, etc. The LCDAs will not be able to continue with the monthly mobile clinics held at their offices. h) However there seems to be enough momentum to believe that those women who have indicated that they changed their practices regarding child care, breastfeeding, adapted healthier eating habits will not change as they have support from their husbands and their mothers in law. i) What will be sustained after SMART is the knowledge base (all levels: community awareness, the doctors, nurses, and the CHWs) and the changes in practices and behaviors that the people have made. Once the attitude is changed, the new behaviors/practices become [part of everyday life]. - MoH is very interested in the SMART approach and would like to train all service providers in the SMART techniques. EVALUATION QUESTION 4: EFFICIENCY a) Original proposal was modified. The Evaluation team needs to verify how the activities were actually implemented against what was planned. b) The baseline was conducted after the project started; there is doubt about the usefulness of the baseline. c) Management of SMART (staffing, delegation ,decision making, centralized versus decentralized), d) Looks like the plan has been followed and activities were implemented according as scheduled; for example, the CHWs were trained, they conducted the home visits, the doctors and nurses held the mobile clinics, referral of the patients were made to private clinics and local health units and family centers. EVALUATION QUESTION 5: EFFECTIVENESS a) Appropriateness of the activities; nutrition was not appropriate; they should have applied positive deviance model. This is a missed opportunity. b) Awareness level seems to be higher c) Cost effectiveness cannot be explored in this context since the elements such as details of the costs for the various program activities could not be disaggregated with the financial data made available to the evaluation team. d) SMART Staff provided TA to the UCDAs who in turn passed on the skills to the LCDAs. RECOMMENDATIONS 1. Follow-up study to on the practices that SMART has promoted at the community level to see the impact on the children. The study would provide further details about additional barriers around child nutrition and feeding practices. Also if in fact the mothers kept the practices, it would provide some information about the effectiveness of SMART. 2. Case study on children who are “stunted” in order to know what mothers did. This study will allow us to identify some key issues about the origin of the stunting. 3. In order to change anything at the university level, the Project should work with Ministry of higher education which can give the directives to the universities to include new information in the curricula of medical education 4. Imams are good change agents for family planning but not for mother health and nutrition. The Imam can talk with the men about acceptance of family planning. 5. Mass media is really important for disseminating messages as well. It should not just be print materials. 6. Work with the ministry of social development and solidarity to upgrade the skills of the Radafyat and also now the government has the conditional cash transfer which is a cash support given to lower income women; they are making it mandatory to attend a health session monthly in order to receive cash. So maybe, the program can also support this type of mandatory requirement for health education. 7. The program should use a focus group discussion style for disseminating health information; for example, during the session, the educator can ask the mothers about their concerns and focus on those while keeping the messages or lessons for that day. 8. The radafyat should be trained and given some motivation in order to be effective. 9. Messages should be uniform for all: community level, doctors, nurses; all NGOs should have the same messages so that on all fronts the messages are reinforced. Most likely, women listen to doctors; in Upper Egypt for instance, where the socioeconomic situation is less, the use of the rural health unit is more because people don’t have the money to go to private providers. This is different from Lower Egypt where the rural health unit is less used and private providers are used more. The focus should be working with the public health system to make sure that the doctors are updated in their knowledge as well the radafyat. 10. Currently, the radafyat are not motivated; they have too much work for the salary that they receive and besides there is no money for transportation. Maybe a possibility is to help the ministry of social work and solidarity to upgrade the system somehow. ??? I n the event that this does not work, then the best is to work with NGOs directly who can train CHWs directly from the communities and give them some incentives and do some follow-up work with them for quality control. 11. All health educators should come from the local communities!!!! It is very important that they use the same language that the people use in terms of reference to food, and other jargon related to health; otherwise the people will not feel comfortable and may not follow￾up with the counseling. 12. Understand the context of the mothers very well before giving them advice and be flexible about it, otherwise people will be resistant. 13. There are a lot of jaundice cases in Egypt, and sometimes the mothers can’t nurse. The program should take that into consideration and see how they can help the mothers overcome social stigma related to women who cannot breastfeed for various reasons. 14. Train all grandmothers, not just the mother-in-law. The mother of the wife should be trained as well. In Egypt, both are influential. 15. The mothers who have C-sections should be targeted differently and be given a lot of support: make sure that the mother is comfortable and also be available to put the bay to the breast as soon as possible. 16. Advocacy (child rights) activities should be included in the program 17. Continuity of economic empowerment of the families at the village level 18. Incorporation of nutritional classes (meals, recipes) in the schools 19. Should find ways to get MoH on-board early on and work closely during implementation. Health care should not be left in the hands of the NGOs. The program has to strike a balance between the two entities; for example the NGO could focus on raising awareness and let the government provide the services. Health providers need to be upgraded in terms of their knowledge and skills. 20. It is not clear that the program is reaching the poorest of the poor since it seems that these women were accessing private providers and they had to pay 83 VIII. LIST OF SITE VISITS SEPTEMBER 2014 – SMART EVALUATION TEAM Date Governorate District Organization 15/09/2014 Qalyobeya Kanatr Egyptian Association for Human & Environmental Development (EAHED) Local Community Development Association in Agh-hor El￾Soghra Charitable Pillars of Goodness Association in El Monira 16/09/2014 Sharkaya Belbis Community Development Association for the Improvement of the Situation of Women and Children Association for Preservation of the Quran in Meet Hamal Om Elkora - Eladleya 18/09/2014 Banisuef Banisuef Good Youth Association for Development and Services Community Development Association in Sherif Pasha Egyptian Youth Association for Development and Services in Tizmant El Sharqia 21/09/2014 Assuit Assuit Giving Without Limit Association 22/09/2014 Assuit Assuit Community Development Association in Doronka (Gamayat Tanmiat Al-Mogtama’ be Dronka) Community Development Association in Manqabad (Gamayat Tanmiat Al-Mogtama’ be Manqabad) 23/09/2014 Sohag Sohag Sohag Community Development Association for the Improvement of the Situation of Women and Children Quality for Development Association in Tunis (Gamayat El Gawda men Agl El-Tanmiah be Tunis) 24/09/2014 Sharkeya Abuhammad Egyptian Society for Cultural Development (ESCD) Qena Neqada Banner of Islam Association 25/09/2014 Qena Qous Rural Women's Development Association in Al-Ayayshah (Gamayat Tanmiat Al-Mara’a Al-Refiah bel Ayayshah) Women’s Association for Women’s Development in Nagaa Al-Awari - Hijazah Qebli (Al-GamayahAl-Nessa’iah le Tanmiat Al-Mara’a beNagaa Al-Awari - Hijazah Qebli) 84 IX. SUMMARY OF CDA ASSESSMENT UPDATE Respondents Rate 45%  (46 out 101 Local CDAs) Inclusion of MCH in current mission statement 21 Operating Mobile clinics 10 CDA owned Polyclinics 7 Types of grants MCH type grants: 21 Other: 8 Grants from Social Fund for Development 21 All worked with pharmacists Duration of SMART activities 16 to 20 months Governorate /District (# of CDAs) Grants from Social development Type of Grants Mobile Clinics Inclusion of MCH in Mission 6 out of 10; others are from other sources 4 for family health 2 environmental health   1 microenterprise for young women 10 are operating clinics; 9 are covering cost from own resources; 2 CDAs have own polyclinics 3 CDAs   Beni Suef/Beni Suef (8) 2 out of 8; others are from other sources (6) are doing Community awareness activities on health of mother and child (1) Community Development   (8) are conducting Mobile clinics (1) CDA has own polyclinic 8 CDAs   Quena/ Nakada (10) 6 out of 10 ; others are from other sources 4 ‐ Health 2 ‐ Environmental health 1 Microenterprise for girls 2 out of 10 are doing mobile clinics;   2 CDAs own polyclinic 3 CDAs Sharkia/Abu‐ Ahmad 1 Child Labor No mobile clinics None own polyclinics Only 1 CDA Quena/Qous (10) 3 out of 10; others are from other sources 3 ‐ MCH   No mobile clinics 2 CDAs own clinics polyclinic 3 CDAs Qanatr (8) 4 out of 8 ; others are from other sources 4 MCH No mobile clinics None own polyclinics   4 CDAs 85 X. SUMMARY RESULTS SUMMARY RESULTS OF TELEPHONE SURVEY OF SERVICE PROVIDERS TRAINED USING SMART MATERIALS The survey was administered by telephone to 42 service providers (doctors and nurses) trained by national professional associations. Of those surveyed, only one person did not change anything gin practice after receiving the training. The top three training topics that made them something different were: HBB (39), Feeding practices for children (12) and breastfeeding (22). A few doctors (3) mentioned that the trainings were not useful at all because the information was too shallow. Total Respondents 42 Doctors 27 Nurses 8 Intern 7 Did something different in practice after training 41 out of 42 Training Received using the SMART materials: TOT 14 Helping Babies Breathe (HBB) 33 Healthy Nutrition 2 Care of the newborn 1 Care of the sick child 2 Pneumonia 1 Care of premature or underweight 3 Breastfeeding practices/skills 3 Exclusive breastfeeding (EBF) 4 Feeding practices for children 4 Kangaroo Mother Care 4 Infection Control 1 Other people trained by these doctors and nurses: Students 80 86 Doctors 555 Nurses 90 Women 750 Total 1475 How they were trained: Job training Seminars Workshops 16 9 8 Top 5 recommendations: a) Expansion of the training in future programs and public hospitals and also offer more hands on practice opportunities during the training; b) manual should be available in Arabic; c) training will be more effective for interns and not doctors; d) put standards to measure success of the trainings: pre and posttest; e) The courses should be repeated more frequently. 87 XII. SUMMARY OF KEY LOST OPPORTUNITIES: CONCLUSIONS BASED ON EVALUATION TEAM’S FINDINGS Key Areas of Lost Opportunities Comments/Conclusions Longitudinal study The key lost opportunity is that fact the project was not able to evaluate how the results from the TIPS studies benefited the longitudinal portion of the stunting study; the recipes and other insights from the TIPS studies were applied to the cohort the 300 hundred children and their families who were followed. Also the longitudinal study only followed the children until 24 months old and not beyond. The ET suggests that these children and their families should still be followed until the children are 5 five years old at most in order to fully appreciate the effectiveness of the interventions in combating stunting. Training of service providers (doctors and nurses) Questionnaire to doctors to assess the service providers used the training though not very insightful was just the same revealing in that SMART should have gathered the pre and post test data from those participated to gage areas some level of knowledge acquisition at best. Though it was much too short time span for the project to assess impact, the ET felt it was a lost opportunity not to have focused on the public service providers at the district levels serving at the local health units. This would have been a good partnership with MOHP to increase the capacity of their staff and quality of services at those units who already suffer from poor reputation as they compete with private clinics. While it is not a guarantee that quality would have improved, the training certainly would have been an incentive for those service providers; the ad hoc manner in which the providers were selected by the national professional associations from their private pool of memberships does not lend itself to guarantee that the providers trained would be serving the clients who need them most and who can least afford them. The ET suggests that future training should be more purposeful and more targeted in order to facilitate changes in quality of service and impact the population at large. Currently the general impression from the impact of the training is diluted and it’s not known how many of these providers who serve at those 101 88 mobile clinics over an average of 15 to 20 months received updates from the SMART updated training curricula. Data management and use: Ad hoc documentation and inconsistent reporting SMART lost the opportunity of using consistent documenting of tis progress by having a good PMP (e.g. good indicators) to make timely decision and adjust its activities accordingly. Had the indicators being adequate and also had the central M&E system being more robust, it would have prompted the program staff to adjust the targets early to be more realistic with the situation on the ground. At best, had the CDAs data been analyzed and used more consistently in SMART quarterly reports, with key indicators across all the districts, the SMART program team would have benefited by gaining insights about performance at the village level and also intervened more timely especially for more growth monitoring training and support. Mobile clinics The fact that mobile clinic data was not disaggregated by service and client types (e.g. first pregnancy, etc.) posed a lost opportunity for SMART to measure the breadth and depth of its outreach activities in terms of demands for services (trend during project lifespan), measure performance of the CHWs in terms of growth monitoring , and other useful data that would have given the project more insights in terms of adjusting its targets for example and perhaps would have documented more than 38,000 clients that it claimed to have served. The ET was unable to confirm any of those figures and suspects that the 38,000 could have been 38,000 visits and that the actual number of clients served was less or it could be the other way around (e.g. more clients than the 38,000) since it was documented and confirmed during the interviews that the database could not sustain the amount of data that was being generated and so therefore the project team gave up and stopped collecting data once the 7000 target was reached. Therapeutic feeding of women and children The Et was unable to collect data at the CDA level to measure how the effectiveness of these interventions. This initiative was not reported either in the SMART quarterly report nor was it followed by the M&E team. This is a lost opportunity for the project because the messaging around improving maternal and child nutrition could have been easily documented in terms of real improvement in weight gain or those graduating from these therapeutic feeding sessions and/or the biological indicator in terms of iron level in blood or anemia level in mothers. None of this data was exploited and it was impossible for the ET to reconstitute for this evaluation purpose. Status of the LCDA capacity Though the ET conducted a short assessment on the CDAs, it was impossible to gage with concrete evidence (hard quantitative data) how these CDAs performed pre and 89 post SMART capacity building. This is a lost opportunity for such a large investment in time, human resources (team leader, program manager and accountant), and financial resources. While the ET was not able to put a dollar value for this capacity building activity, it can safely conclude that SMART was vested to ensure that these groups had the right tools to do their job as per the agreement with SMART. Anecdotally, staff members as well as board members of these organizations have testified receiving great benefits for the community at large and for personal development, but it is not enough to substantiate the investment made in the 101 local CDAs and 12 umbrella CDAs. Alternatively, the ET a more systematic monitoring system and better indicators to measure progress and performance these organizations. Optimization of technical expertise in the SMART consortium Although SMART received many visits from HQ to the field for technical input and training of the SMART staff and CDAs, the project lost the opportunity to capitalize on those inputs and optimize its performance; case in point, these technical experts from HQ should have caught on early on the deficits in the PMP, as much as the inconsistencies in reporting. The documentation made available to the ET does not demonstrate that any efforts were made to update the PMP, nor provide the technical guidance needed to optimize the use of the data and adjust the systems accordingly as in making use of the project data for decision making. The ET did not find any evidence in the quarterly reports of a quarterly summary synopsis of what was happening at the CDA level in terms of their service provision, the monthly data collection of the CHWs, etc. Instead the ET found out accidentally through an interview with one of the SMART Team Leaders that there were regular reports from the CDAs and the ET received a couple of samples from in Upper Egypt; those revealed and confirmed the ad hoc reporting. Though this discussion is about the lost opportunity in the use of technical experts in the consortium, the ET uses the case of the poor data management as a recurring theme in SMART for many of its lost opportunities. Because of this, the ET suggests that future should keep a PMP that is more dynamic and flexible to allow for changes as needed. 90 XIII. MEETING SCHEDULE Date Governorate District CDA / Office Activities Briefing/Team Planning Meeting Meeting with Ms. Shadia Attia ‐ Monitoring and Evaluation Advisor Meeting with Dr. Shahira Hussein ‐ Program Manager Meeting with Dr. Nabil Alsoufi ‐ OHP Director MeeƟng with Dr. Taissir Hossam ‐ Monitoring and EvaluaƟon specialist of MCHIP MeeƟng with Dr.Farouk ‐ Monitoring and EvaluaƟon specialist of MCHIP Meeting with Dr. Issam El Adawi ‐ Previous COP MOHP Meeting with Dr. Mohamed Nour ‐ Director of MCH Save The Children Meeting with Dr. Amani Saleh ‐ EVE Project Manager Office Meeting with Dr. Amal Hamouda ‐ Team leader SMARTProject SMART/MCHIP Meeting with Dr. Gulsen Saleh ‐ Clinical Nutritionist‐ NNI UNICEF Meeting with Dr. Magdy Elsanady ‐ Health Specialist Office Meeting withDr. Samia Farghaly‐ Ministry of Health and Population Meeting with Dr. Nevine Hassanein‐ Reproduvtive Health Consultant Meeting with Dr. Sahar Mourad‐ Development Consultant Meeting with Dr. Abla Alalfy ‐ Regional Advisor for Middle Egypt RCPCH Egyptian Association for Human & Environmental Development (EAHED) Local Community Development Association in Agh‐hor El‐Soghra Charitable Pillarsof Goodness Association in El Monira Community Development Association for the Improvement of the Situation of Women and Children Association for Preservation of the Quran in Meet Hamal Om Elkora ‐ Eladleya Good Youth Association for Development and Services Community Development Association in Sherif Pasha Egyptian Youth Association for Development and Services in Tizmant El Sharqia 21.09.2014 Assuit Assuit Giving without limit association Community Development Association in Doronka (Gamayat Tanmiat Al‐Mogtama’ be Dronka) Community Development Association in Manqabad (Gamayat Tanmiat Al‐Mogtama’ be Manqabad) Sohag Community Development Association for the Improvement of the Situation of Women and Children Quality for Development Association in Tunis (Gamayat El Gawda men Agl El‐Tanmiah be Tunis) Sharkeya Abuhammad Egyptian Society for Cultural Development (ESCD) Qena Neqada Banner of Islam Association Rural Women's Development Association in Al‐Ayayshah (Gamayat Tanmiat Al‐Mara’a Al‐Refiah bel Ayayshah) Women’s Association for Women’s Development in Nagaa Al‐Awari ‐ Hijazah Qebli (Al‐GamayahAl‐ Nessa’iah le Tanmiat Al‐Mara’a beNagaa Al‐Awari ‐ Hijazah Qebli) 24.09.2014 25.09.2014 Qena Qous MCHIP Evaluation Team Meetings Schedule from Sep.8, 2014 till Oct.4, 2014 22.09.2014 Assuit Assuit FGD 23.09.2014 Sohag Sohag 15.09.2014 18.09.2014 16.09.2014 Qalyobeya Kanatr Sharkaya Belbis Banisuef Banisuef 10.09.2014 Cairo 11.09.2014 Cairo Office USAID MCHIP/SMART 08.09.2014 Cairo 09.09.2014 Cairo 91