Contracted under RAN-I-00-09-00016, Task Order Number AID-663-TO-15-00001 Authors: Dr. Rosemary Barber-Madden, Dr. Mesfin Beyero, and Mr. Steven Hansch. International Business and Technical Consultants, Inc. EVALUATION Final Performance Evaluation Ethiopia Food by Prescription Final Performance Evaluation Ethiopia Food by Prescription Final Performance Evaluation Ethiopia Food by Prescription Final Performance Evaluation Ethiopia Food by Prescription Final Performance Evaluation Ethiopia Food by Prescription Final Performance Evaluation Ethiopia Food by Prescription ACKNOWLEDGMENTS IBTCI would like to thank the staff of the Federal Ministry of Health of the national Government of Ethiopia, Regional Health Bureaus and health facilities visited in Addis Ababa, Amhara, Tigray, SNNPR, and Oromiya Regions, and the beneficiaries for their cooperation in this performance evaluation of USAID/Ethiopia’s Food by Prescription activity. We would also like to thank the staff of the USAID/Ethiopia Health, HIV and AIDS, Population, and Nutrition (HAPN) Office. We are particularly grateful for the guidance and support of Josephine Francisco and Sileshi Kassa. We appreciate the cooperation and collaboration of the prime implementing partner, Save the Children US. IBTCI also extends its thanks to Gordon Mortimore for his contributions during the beginning of this evaluation. A special thanks goes to Farrah Ahamad for her unyielding administrative support with this evaluation. FINAL PERFORMANCE EVALUATION OF ETHIOPIA FOOD BY PRESCRIPTION March 25, 2015 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Table of Charts .................................................................................................................................................................. i Acronyms ........................................................................................................................................................................... ii Executive Summary ........................................................................................................................................................ v Evaluation Purpose and Questions........................................................................................................................... 1 Activity Background ....................................................................................................................................................... 2 Evaluation Methods and Limitations ....................................................................................................................... 4 Findings...............................................................................................................................................................................6 Annexes............................................................................................................................................................................ 27 - Annex I: Scope of Work From Task Order ....................................................................................................... 27 - Annex II: Save the Children US M&E Plan ...................................................................................................... 37 - Annex III: List of References ................................................................................................................................. 38 - Annex IV: Evaluation Plan and Analysis ........................................................................................................... 39 - Annex V: Data Collection Instruments for KIIs and FGDs .......................................................................... 57 - Annex VI: Sources of Information ……….............................................................................................................64 - Annex VII: Food By Prescription Data Quality Assessment Form.............................................................69 - Annex VIII: Conflict of Interest Forms..................................................................................................................70 i | P a g e TABLE OF CHARTS Table 1. Total KIIs conducted disaggregated by level of the health system.............................................4 Table 2. IR1: FGDs conducted disaggregated by region, respondent group, and type of health facility....................................................................................................................................................................................5 Table 3. Goal: Improved Nutrition, Clinical and Functional Outcomes among PLHIV Adults, Pregnant and Postpartum Women and OVC........................................................................................................7 Table 4. IR1: Improved Use of Therapeutic and Supplemental Nutritional Interventions by PLHIV and OVC……….....................................................................................................................................................................7 Table 5. IR 2: Capacity Strengthened for the Implementation of Quality FBP Interventions through Technical Leadership....................................................................................................................................8 Table 6. IR2: Strong Collection, Analysis and Use of Data Informing Policy and Planning for Provision of Nutritional Care in the Context of HIV...........................................................................................9 Table 7. Clinically Malnourished HIV-Positive Clients and OVC Who Received Food Supplements and Graduated, 2009-2014........................................................................................................................................10 Table 8. Cumulative Number and Percent of Clinically Malnourished HIV-Positive Clients and OVC Who Relapsed After Graduation, 2009 -2014..........................................................................................11 Table 9. Clinically Malnourished HIV-Positive Clients and OVC who Received Food Supplements and (a) Non-Responded, (b) Lost To Follow-Up, (c) Defaulted or Died (by age and gender), 2009- 2014 ....................................................................................................................................................................................12 Table 10. Data Quality Assessment at Facility Level.........................................................................................14 ii | P a g e ACRONYMS ACC/SCN UN Administrative Committee for Coordination, Subcommittee on Nutrition AIDS Acquired immune deficiency syndrome ART Antiretroviral therapy BMI Body mass index CCC Comprehensive Care Center CHW Community health workers CMAM Community Based Management of Acute Malnutrition CRS Catholic Relief Services CSB Corn-Soya Blend DFID Department for International Development DHO District Health Office EC Exited client ES Economic Strengthening FANTA-2 Food and Nutrition Technical Assistance II FAO UN Food and Agricultural Organization FBF Fortified blended flour FBP Food by Prescription FGD Focus group discussion FMoH Ethiopian Federal Ministry of Health GOE Government of Ethiopia HAPCO Federal HIV/AID Prevention and Control Office of the FMoH HEW Health Extension Worker HMIS Health Management Information Systems HFA & Ht/Age Height-for-age HIV Human immunodeficiency virus HSB Health-seeking behavior IEC Information, education and communication IP Implementing Partner iii | P a g e IR Intermediate Result IBTCI International Business and Technical Consultants, Inc. KI Key Informant KII Key Informant Interview LOP Life of project LOT Length of treatment LTF Lost to follow-up M&E Monitoring and Evaluation MAM Moderately malnourished MCC Millennium Challenge Corporation MOH Ministry of Health MUAC Mid Upper arm circumference NACS Nutritional Assessment Counseling and Support NHSSP National Health Sector Strategic Plan OI Opportunistic infection OVC Orphans and vulnerable children P/PP Pregnant and postpartum PEPFAR US President’s Emergency Fund for AIDS Relief PFSA Pharmaceutical Fund and Supply Agency, Federal Government of the Republic of Ethiopia PLHIV People living with HIV/AIDS PMTCT Prevention of mother-to-child transmission RA Research Assistants RDA Recommended daily allowance RDF Revolving Drug Fund RHB Regional Health Bureau RUSF Ready-to-Use Supplementary Food RUTF Ready-to-Use Therapeutic Food SAM Severe acute malnutrition SCMS Supply Chain Management System SNNP Southern Nations, National Peoples Region iv | P a g e SOW Scope of Work TB Tuberculosis USAID United States Agency for International Development VCT Voluntary Counseling and Testing WFA or Wt/Age Weight-for-age WFH or Wt/Ht Weight-for-height WFP World Food Programme WHO World Health Organization v | P a g e EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the Food by Prescription (FBP) Performance Evaluation was to quantify and analyze the results within two groups of interest—HIV-positive persons that received Ready-to￾Use Therapeutic Food (RUTF) and/or Ready-to-Use Supplementary Food (RUSF) (demand side)—and the health service delivery system (supply side). The evaluation examined results of integrating Nutrition Assessment Counseling and Support (NACS) into HIV care and treatment services, and reviewed the systems in place to integrate the quantification, management, and distribution of RUTF and RUSF commodities into the Logistics Management Information System (LMIS) at the national and regional levels. The evaluation assessed the effectiveness of the activity’s design and implementation vis-à-vis achieving FBP’s stated objectives. Recommendations that may help inform future programming of HIV and nutrition activities supported by USAID/Ethiopia, the Government of Ethiopia (GOE), and other relevant stakeholders, were also made. Target audiences for this evaluation include (a) USAID/Ethiopia—to inform the design of future HIV/AIDS Care and Support and Nutrition programs; (b) Ethiopia’s Federal Ministry of Health (FMoH) —to demonstrate the relevance and effectiveness of the FBP model in the national response; and (c) Save the Children US (the FBP Implementing Partner (IP))—to provide information on FBP’s performance and the achievement of its strategic objectives and overall goal. ACTIVITY BACKGROUND The FBP activity provides technical assistance for the integration of NACS into the routine care and treatment services for people living with HIV/AIDS (PLHIV). The activity supports the provision of RUSF and RUTF to moderate and severely malnourished adult PLHIV, including pregnant and lactating mothers, and Orphans and Vulnerable Children (OVC). With Save the Children US as the prime IP, FBP supports five strategic areas: (1) commodity sourcing, procurement and distribution of RUSF and RUTF; (2) capacity building of key stakeholders and health facility staff and communities to deliver FBP activities; (3) support for adherence and behavioral change through information, education and communication (IEC); (4) increasing coordination of HIV and nutrition interventions and policy issues with key stakeholders; and (5) monitoring and evaluation (M&E) systems to support FBP programming. The development hypothesis is that improved nutritional, clinical and functional outcomes of malnourished HIV-positive adults and Orphans and Vulnerable Children (OVC) are achieved through strengthening NACS, creating effective linkages to community resources and economic strengthening (ES) initiatives, and linking food support beneficiaries to community- vi | P a g e level ES activities as a long-term measure to prevent further malnutrition. To benchmark the progress, USAID/Ethiopia identified three intermediate results (IRs), as well as strategies to achieve them: IR1: Increase the provision and access of therapeutic and supplementary nutritional products to PLHIV; IR2: Strengthen capacity to implement FBP and; IR3: Strengthen the collection, analysis and use of data to inform policy and planning for provision of nutritional care in the context of HIV. EVALUATION QUESTIONS According to the Statement of Work (SOW), four domains guided the FBP final performance evaluation as listed below (NB: For a listing of the evaluation questions see the body of the report):  Analytical Domain 1: Effectiveness of Activity Implementation and Management  Analytical Domain 2: Sustainability  Analytical Domain 3: Relevance  Analytical Domain 4: Gender METHODOLOGY This evaluation used a mixed methods approach, employing both quantitative and qualitative techniques. Quantitative methods included analysis of secondary data extrapolated from FBP activity documents (such as M&E Plan and quarterly and annual reports), and national reports. Qualitative methods on the other hand included key informant interviews (KIIs) and focus group discussions (FGDs). The sampling universe encompassed 494 health facilities across five regions and two city administrations. USAID/Ethiopia selected four regions, namely Amhara, Oromia, SNNPR and Tigray and one city administration, i.e., Addis Ababa. Health facilities were selected based on case load, geographical representation, and feasibility of roadside access. Eighteen months of facility records over the five-year period of performance were randomly selected and the records were checked for completeness and consistency. Seventy KIIs with key stakeholders at three levels were carried out. KII participants were representatives of the GOE, other donors, and Save the Children US. At regional level, KIIs were conducted with Regional Health Bureaus (RHBs), Save the Children US regional program coordinators and IPs. Representatives from 27 health facilities at the local service delivery level took also part in the KIIs. Of the KII participants, 13 were at national, 21 at regional and 36 at local service delivery levels. Thirty-two FGDs with selected FBP beneficiaries were also carried out. HIV+ women (age 15+) who represented 59% of all FGDs, women at prevention of mother￾to-child transmission (PMTCT) comprised 9% of the FGDs, HIV+ men (age 15+) accounted for 25% of the FGDs were among the participants in FGDs. FGDs with OVC and a mix of male and female beneficiaries were also among the 32 FGDs. vii | P a g e The team conducted a descriptive analysis to illustrate FBP’s various types of interventions and other characteristics; a content and thematic analysis of qualitative data to identify common issues, themes and patterns for each evaluation criteria (relevance, effectiveness, and sustainability); a comparative analysis to examine findings across different regions and to identify challenges; an analysis of gender implications to ascertain that activities addressed gender issues; and a secondary analysis of M&E plan, M&E system, and data quality to analyze data management system. LIMITATIONS There were two main drawbacks to this evaluation. First, the 32 FGDs did not include sufficient representation from different beneficiary groups, particularly OVC and women in PMTCT services. It was a challenge to have sufficient numbers of OVC during the time of the site visits. According to health facility staff, OVC infrequently seek care for their primary health problems, and they tend to discontinue the RUTF/RUSF when their health conditions improve. Similarly, PMTCT beneficiaries are relatively few in number compared to women in Antiretroviral Therapy (ART), and as well, have specific appointment dates for antenatal care (ANC) compared to ART attendees. This reduced the number of clients from this beneficiary group on the day of our site visit. Site selection was the second drawback. Roadside access influenced site selection, but was mitigated by the number of regions selected, and facilities with high case load in consultation with the RHBs. To minimize interviewer biases the team collected data in pairs and conducted thematic analysis as a group. MAJOR FINDINGS Domain 1: Effectiveness of Activity Implementation and Management For Goal 1— Improved Nutrition, Clinical and Functional Outcomes FBP exceeded all of its targets with 64 percent (target 60 percent) of clinically malnourished PLHIV including PMTCT and OVC clients graduating from the activity, and 0.8 percent (target < 5 percent) of PLHIV (including PMTCT and OVC clients) who died during the course of treatment. For IR1—Improved Therapeutic and Supplemental Nutrition Interventions by PLHIV and OVC- the activity met five of its six targets. FBP exceeded its targets for the number of PLHIV, including PMTCT and OVC clients, who (1) received a nutritional assessment (200 percent); (2) received nutritional counseling (165 percent); (3) were clinically assessed and found to be severely malnourished (107 percent); (4) were clinically assessed and found to be moderately malnourished (117 percent); and (5) were clinically malnourished PLHIV clients who received therapeutic and/supplementary food (153 percent). FBP underperformed on its target for the number of graduated malnourished clients that benefited from economic strengthening (ES) (56 percent). For IR 2—Capacity strengthened for the implementation of quality FBP Interventions through technical leadership —FBP met or exceeded four of its seven targets. This included the number viii | P a g e of service providers trained in nutrition and HIV/AIDS (149 percent); and the number of data clerks trained on M&E of NACS (242 percent). FBP met its target for data/reports shared with the GOE and other partners (100 percent), and for the percentage of health facilities and distribution outlets that collect and report FBP specific data (100 percent). Although Save the Children US did not achieve three of its other targets, FBP did rank over 95 percent performance on the number of health facilities ready to provide FBP services (98 percent), the number of health facility store managers and pharmacists trained on NACS for logistics (98 percent), and number of case managers trained on nutrition and HIV/AIDS (95 percent). Eighty-eight percent of the health facility staff interviewed reported that after their NACS training, their role was to conduct nutrition assessments and body mass index (BMI) measurements, to counsel PLHIV clients, and to prescribe Plumpy Nut/Sups to clients whose weight was below normal standard with subsequent weight gain. We could not confirm these reports in our record review at 10 health facilities due to poor recordkeeping and reporting systems. All FGD participants (100 percent) reported that they had been weighed, counseled, received Plumpy Nut/Sups, and as a result, gained weight. However, the team was not able to verify these reports through health facility record review, due to common reporting errors. For IR 3 Improved collection, analysis and use of data—FBP achieved its target for percentage of health facilities and distribution outlets collecting and reporting FBP data (100 percent), with 94 percent having shared data and reports with governmental and other partners. But it underachieved on the proportion of health facilities handed over to the government (87 percent). The evaluation team did not have access to data that would permit analysis of the efficiency of the commodity management and supply chain process handled by the PFSA. Procurement of commodities was the most substantial cost, which was amplified by the recurring nature of providing large volumes of food. Save the Children US’s coordinating role accounted for roughly 30 percent of total costs. FBP underachieved on the number of PLHIV clinically malnourished, including PMTCT and OVC, who graduated from the activity. The numbers and proportions of those who graduated appear to have been higher in year 4 than in year 5. The FBP M&E Plan showed that the target of the number of graduates linked to the Economic Strengthening (ES) intervention was not achieved. Of the beneficiaries who graduated from the FBP activity, 9,327 were linked to the ES during the five years of the activity. The team found that FBP information management data is reported at different levels, primarily as part of the regular health system, yet there is no feedback mechanism from the regional level back to the health facilities. This contributes to inadequate continuity of care over time for individual clients, as well as an inability to track clients. In addition, there was insufficient data about the target population to make reliable estimates about whether the activity influenced extended survivorship or disability, either positively or negatively. ix | P a g e Domain 2: Sustainability M&E Plan data indicates FBP met its targets for data and number of reports shared with government and partners, and for the percentage of facilities and distribution outlets collecting and reporting data to FBP. Yet FBP underperformed on the number of health facilities handed over to government. The Health Management Information System (HMIS) recently incorporated three indicators related to the FBP activity: (1) the number and percent of PLHIV who are nutritionally assessed and moderately malnourished; (2) those nutritionally assessed and severely malnourished; and (3) the number and percent of those who received nutritional support. This is an important step forward since the inclusion of additional indicators into the national HMIS is very challenging, and takes time to negotiate due to the length of the HMIS list. The inclusion of these indicators that are specific to NACS activities means that health facilities are accountable and they are expected to continue reporting on these indicators after the phasing out of the activity. As reported above, the activity met two of five targets for strengthening capacity for implementing quality FBP activities, and exceeded the number of service providers trained on nutrition and HIV/AIDS and the number of data clerks trained on M&E of NACS. The activity underperformed on the number of facilities ready to provide FBP services. The overall impact of not achieving these targets is not significant compared to what has been achieved during the five years of the activity. Facilities with high case load were selected in consultation with the RHBs - this was the primary criterion for site selection, according to one of the activity staff. Therefore, it is unlikely that high volume sites were left out. In the health facilities visited, there was at least one trained staff member assigned to the ART and/or PMTCT clinics. One health facility manager said: “We were not conducting nutritional assessments nor counseling on better nutrition and healthy eating before the training but this has now changed.” FBP reports showed there are sufficient numbers of master trainers in all RHBs. The team visited Mekele University in Tigray region, where NACS training was consolidated into the training curricula for doctors and health workers. At Hawassa University in SNNPR, NACS was introduced in the pre- service curricula of doctors and mid-level health workers (nurses and midwives) but was later discontinued. At Jimma University in the Oromia Region, a NACS training was planned, but did not materialize. The integration of NACS training into pre-service curricula at targeted universities is varied. For example, Save the Children US Annual Progress Reports indicated that there were funding delays limiting the launch of these activities. In-service training was developed and carried out by RHB-trained master trainers. The team found no shortage of RUTF/RUSF, which is being prescribed as long as there is a supply. A senior Ethiopian FMoH nutrition advisor emphasized that “the government will not be able to continue supplying these products because of cost.” Another FMoH nutrition case team coordinator reported that there has been no discussion on the sustainability of the FBP activity. On the other hand, other FBP-supported RHBs and health facility staff have discussed the sustainability aspect and are concerned about the likelihood to continue providing RUTF/RUSF without external support. x | P a g e Domain 3: Relevance FBP’s objectives were to (1) improve nutritional, clinical, and functional outcomes among PLHIV adults and pregnant and lactating women, and OVC through periodic nutritional assessment and counseling, and (2) provide therapeutic and supplementary food support to malnourished PLHIV. There has been no emphasis on counseling for dietary diversification or increasing micronutrient intake using the locally available sources. Evaluators interviewed 13 RHB staff and 26 health staff at 26 of the selected health facilities (from the 26 health facilities, between zero and two health staff were interviewed). RHB and health facility staff responses demonstrated a high level of knowledge about the nutritional needs of malnourished PLHIV, and commitment to meet their needs. One hundred percent of FGD groups (32) indicated satisfaction with, and effectiveness of, FBP services and RUFT/RUSF. All agreed that RUFT/RUSF helped them to better manage their health, and that “within a few days,” they noticed “increased appetite and gained 5- 7 kg,” “improved water intake and CD4 count,” and “improved adherence to ART.” One man said: “I was not able to order my body to work. Even I felt very tired when I walked a short distance. After taking it (Plumpy Nut) I was able to do hard work.” The team collected data showing that FBP and the health facilities met the immediate needs of the targeted beneficiaries, but the data does not provide sufficient information to conclude that improvements will be sustained. The respondents described the health and nutrition benefits in terms of health outcomes. All health facility staff interviewed (27) agreed that FBP is relevant to their work. Their responses showed in-depth understanding of the implications of malnourishment and how to best treat clients. All (100 percent) health facility staff participants in KIIs reported having seen the effect of RUFT/RUSF on their patients, that is, “decreased prevalence of opportunistic infections,” “increased effectiveness of the ART drugs,” and “improved health, nutritional status and survivorship.” When these staff were asked about the challenges of implementing NACS, 63 percent of them reported that their workload increased significantly. Despite this, more than seventy percent of the complaints of the workload applauded the positive changes they witnessed among their patients as opposed to the workload they bear. All FGD groups (100 percent) reported that they either consume RUFT/RUSF as prescribed by health workers directly from the sachet, or eat it with buttered bread. Among FGDs 22 percent reported sharing nutrition supplements, with most saying “we share it with the children.” Among those who reported that they did not share the supplements (63 percent), they explained that, “every member of the family knows that the supplements are prescribed,” and that “it’s a drug, not possible to share.” Nonetheless, Save the Children US and health facility staff emphasized that a much higher number of beneficiaries share or sell much of their foods. Domain 4: Gender We found no evidence that gender-related issues were examined, flagged or tracked over time. Despite this, M&E Plan data shows that the number of adult females enrolled exceeded the number of adult males enrolled, which likely reflects the epidemiology of HIV in the country, with females slightly outnumbering females among the adolescent population. According to RHB representatives, health facilities disaggregated data and reported to FBP by gender and age. More importantly, we found that the FBP activity did not have a specific plan to address xi | P a g e gender issues, whether male or female, nor did the health worker training plans cover gender related issues. Among the 10 RHB representatives interviewed, four indicated that although the data is collected on a disaggregated basis by sex and age, it is not used for activity analysis or planning. One representative indicated that “the activity is facility-based and client-need based, and those who are in need of the drugs and are eligible are coming to the facilities and are addressed. We don’t get women directly and therefore we are not specifically targeting them.” When asked if FBP activities had any influence on the status of women and men, only two RHB representatives responded. One reported that “those that found work have clearly improved their status and in follow-up we find they are now able to afford more food, school fees.” The other indicated that “FBP tries to integrate mothers who are graduated from the FBP activity with economic support activities/the Back-to-Work program through different organizations such as NAP+, and linked to the Back-to-Work program were women, limited only in a few towns.” CONCLUSIONS Domain 1: Effectiveness of Activity Implementation and Management The FBP activity successfully achieved its outputs, having exceeded targets in years 4 and 5. However, the outcome of improved survivorship remains unknown. Based on the information available, we were unable to understand why individuals drop out. Nor could we understand the apparent inability to track individuals with personal identification records that are carried across regions. Many FBP supply issues were outside Save the Children US’s manageable control, since supply management, logistics and distribution to health facilities were facilitated by PFSA. Overall, we did not have sufficient data to assess the efficiency of the output results. There appears to be an underestimate of the true relapse rate, since some proportion of the participants who graduated inevitably relapsed but did not return to the activity, perhaps because of the distance to the health facility, or the gravity of their illness (e.g., severely ill or terminal). Referral of clients to ES activities was the least well-developed component of the FBP, and did not reach a substantial number of graduated clients, despite reports from RHBs and local NGOs that conduct ES activities, citing the importance of Save the Children US technical and financial support. In the M&E Information and Management System, there was a substantial number of irregularities in activity data reporting, with variation from facility to facility, and possibly between health facility staff, which casts doubt on the validity and reliability of data reporting from facilities to the FBP activity, and from facilities to RHBs. Domain 2: Sustainability The inclusion of NACS-related indicators in the HMIS is a significant step toward sustainability that will promote integration of the activity within the public health system and beyond FBP￾supported health facilities. NACS’s role is well recognized in the care and treatment of PLHIV, and a high percentage of HCWs (76.5%) show a willingness to integrate NACS services as an integral part of their routine activities despite the workload. While the training built the capacity of health workers, and they gained knowledge and skills in NACS and were implementing the skills learned at health facility level, the limited integration of NACS training into the pre-service xii | P a g e curricula in most of targeted universities considerably undermines its sustainability (only one out of the three universities targeted by the project integrated NACS training into the pre￾service curricula). RHBs reported there are a sufficient number of master trainers at their level, and the in -service training can still be applied to a critical mass of people, given that one master trainer can train up to 171 people per session and can conduct multiple sessions in a year. Financial viability to sponsor training sessions over the medium term, though, is questionable. The quality of the services might be at risk given reports that NACS would be limited without Plumpy Nut/Sup or another substitute, and considering reports of high staff turnover and internal rotations within health facilities, especially hospitals. The NACS component of FBP might be challenging to maintain if continued training and reinforcement are not available. In summary, there is no guarantee that the GOE will be able to maintain this type of service given the concerns expressed about the cost of RUFT/RUSF supplements. Domain 3: Relevance There is sufficient evidence from the analysis of FGD data to conclude that the FBP activity provided satisfactory services which have yielded positive results for participating beneficiaries. All FGD groups reported that the products helped them to gain weight and improve their nutritional status, regain their strength, improve ART adherence and go back to work, enhance their appetite for other foods, and gave them hope of being alive. Based on the team’s analysis of stakeholder interview data, RHBs and facilities will likely continue the FBP activities given the positive outcomes, if financial resources are made available. According to our analysis of FGDs, RUFT/RUSF helped to improve weight and nutrition status of beneficiaries. Domain 4: Gender The FBP activity did not have a specific plan to address gender needs of enrolled men and women, nor was the disaggregated facility data used when Save the Children US revised its plan and programming. For example, examination of gender differences in relapses and lost to follow-up (LTF) rates or death rates between men and women may have illuminated issues which, in turn, could have been addressed. The positive step, however, is that facilities disaggregated data and reported to FBP by sex and age making future use to tackle gender issues possible. 1 Bradshaw, M. (2011). Innovative teaching strategies in nursing and related health professions (5th ed., p. 538). Sudbury, Mass.: Jones and Bartlett xiii | P a g e RECOMMENDATIONS Domain 1: Effectiveness of Activity Implementation and Management Sustain and Scale-up: Based on the successes evidenced in this evaluation, USAID should continue to support the scale-up and integration with other nutritional outreach efforts, while working with the GOE to explore other options for subsidizing supplemental foods. The ES component lacks sufficient evidence of effectiveness to be scaled-up. Explore Alternate Foods: USAID and the GOE should collaborate on analyzing alternative, less expensive food products, giving more attention to those that can be manufactured from local production, to sustain FBP activity beyond its conclusion. Plumpy Nut/Sup are expensive commodities that the government is unlikely to continue supplying. These products currently rely upon the health system to be distributed, but the local production of food supplements most likely falls under the responsibility of other government entities, bearing in mind that NACS support and referrals to and from the health sector for supplements will continue to be necessary. USAID should (a) provide transitional and technical assistance to explore the options for scaled-up production of fortified, specialty foods. This should be based on the costs of procurement, packaging, quality control, distribution and other cost factors, and (b) communicate with the GOE and other donors and organizations that provide technical and financial support, to provide less expensive, supplementary foods. Improve M&E Information System and Introduce Systematic Operations Research: USAID and its IPs should work together to reinforce data collection and reporting to improve M&E systems writ large. In addition, there are a number of questions which arose from our evaluation about effectiveness, efficiency, usage, referrals, compliance, and others, such as why clients drop out, who relapses, and the different needs, outputs and outcomes for men and women by age, geographic location, and religious practices. The answers to these questions might influence activity effectiveness, how gender issues might best be addressed, understanding the reluctance of universities to be involved, better local food substitutes, and the amount of other food intake necessary in order to rule out some confounding factors. Operations research is needed to answer these questions. Domain 2: Sustainability Work jointly to assure routine nutrition assessment and counseling: The FMoH and HAPCO should work closely with RHBs, USAID’s Food and Nutrition Technical Assistance (USAID/FANTA III), World Food Programme (WFP) and other stakeholders working in nutrition and HIV/AIDS, to ensure that all health facilities routinely carry out nutritional assessment and counseling. Reinforce use of government systems: USAID and its partners should reinforce use of government systems, such as PFSA for supply chain management, and public health facilities for service delivery, to increase the likelihood of sustaining services and enhancing government ownership. xiv | P a g e Reinforce HCWs capacity building: USAID should work with the MOH and universities to secure financial resources to enhance inclusion of NACS in the curriculum. Frequent rotation to other facilities should be addressed in order to sustain trained workers in the facilities. A motivational system should be put in place to increase acceptance of the workload due to the integration of NACS services as an integral part of routine activities. Domain 3: Relevance Include NACS in ART training: USAID and FMoH should assure the inclusion of NACS into the standard ART training, with appropriate indicators to ensure that every health worker trained on ART is also trained on NACS. Over the long-term, the FMoH, HAPCO, and USAID/FANTA III need to consider integration of NACS into the pre-service curriculum of health science colleges and medical schools. Future Directions Facilitate public – private partnerships to engage graduates of the Back-to-Work program, which is also in tandem with the poverty reduction strategy. Work closely with town level advisory committees to identify locally available resources, and reinforce engagement of graduates with all available ES activities. Explore the feasibility of the inclusion of nutrition supplementation coverage in the expansion of the GOE’s community-based insurance system, as it does with other drugs. 1 | P a g e EVALUATION PURPOSE & QUESTIONS EVALUATION PURPOSE The Food for Prescription (FBP) Performance Evaluation aims to quantify and analyze the results achieved at the level of each of the two beneficiary groups—HIV-positive persons that received Ready-to-Use Therapeutic Food (RUTF) and/or Ready-to-Use Supplementary Food (RUSF)—and the health service delivery system level. We examined results of integrating Nutrition Assessment Counseling and Support (NACS) into HIV care and treatment services, and reviewed the systems in place to integrate the quantification, management, and distribution of RUTF and RUSF commodities into the Logistics Management Information System (LMIS) at the national and regional levels. This evaluation assessed the effectiveness of the activity’s design and implementation vis-à-vis achieving FBP stated objectives. We make recommendations to inform future programming of HIV and nutrition activities supported by USAID/Ethiopia, the Government of Ethiopia (GOE), and other relevant stakeholders. Target audiences for this evaluation include (a) USAID/Ethiopia—to inform the design of future HIV/AIDS Care and Support and Nutrition programs; (b) Ethiopia’s Federal Ministry of Health (FMoH) —to demonstrate the relevance and effectiveness of the FBP model in the national response; and (c) Save the Children US (the FBP Implementing Partner (IP))—to provide information on FBP’s performance and the achievement of its strategic objectives and overall goal. EVALUATION QUESTIONS According to our contractual Scope of Work (SOW), four analytical domains guided the FBP final performance evaluation (See Annex I: Scope of Work from Task Order). Those domains and their associated evaluation questions are outlined below: Analytical Domain 1: Effectiveness of Activity Implementation and Management Q1. What have been the achieved outputs versus planned outputs? Q2. How efficiently have the output results been achieved against inputs and budgets used? Q3. What is the number and percent of clinically malnourished HIV-positive clients who received food supplements and graduated from the activity? Q4. Of those that graduated from the activity, what number relapsed? 2 | P a g e Analytical Domain 2: Sustainability Q5. To what extent have NACS indicators been integrated into FBP-supported government and health facility reporting formats and work streams? Q6. What is the capacity of supported Regional Health Bureaus (RHBs) and health facilities to plan for and provide NACS services in the absence of FBP support? Q7. To what extent has NACS been integrated into in-service training programs for nurses, midwives and medical doctors in targeted universities? Analytical Domain 3: Relevance Q8. To what extent are the objectives of FBP consistent with the needs of the activity’s beneficiaries/target groups? Q9. To what extent do stakeholders (RHBs, health facilities), including beneficiaries, buy-in to and own the goal, objectives and FBP implementation methods? Q10. To what extent are RUTF and/or RUSF palatable and useful to beneficiaries? Analytical Domain 4: Gender Q11. To what extent do FBP interventions address gender issues that expose women to HIV/AIDS or malnutrition? ACTIVITY BACKGROUND The FBP activity provides technical assistance for the integration of NACS into the routine care and treatment services for people living with HIV/AIDS (PLHIV). The activity supports the provision of RUSF and RUTF to moderate and severely malnourished adult PLHIV, including pregnant and lactating mothers and malnourished HIV-positive or exposed children. The estimated HIV prevalence rate in 2000 was 7.3 percent. This figure declined to 1.5 percent by 2011 (EDHS 2011), with an adult HIV prevalence for 2013 expected to be 1.3 percent, according to the FMoH’s epidemiological projections from 2012. The number of PLHIV who are in need of care and treatment is estimated to be 734,048 adults and children. Women are considered more vulnerable to HIV infection and malnutrition compared to their male counterparts due to biological, social, economic, and cultural factors. While there has been some progress in improving nutrition in Ethiopia over the past several decades, stunting, wasting and micronutrient deficiencies are still prevalent across the country. Nutrition surveys have documented high levels of acute malnutrition among women, girls, and infants. Under-5 stunting is above 40 percent and the national prevalence rate for wasting is 10 percent, with higher rates in some regions (EDHS 2011). The prevalence of malnutrition is more severe in rural areas where most of the population lives. 3 | P a g e The FBP activity has been implemented by Save the Children US and supports five strategic areas: (1) commodity sourcing, procurement and distribution of RUSF and RUTF; (2) capacity building of key stakeholders and health facility staff and communities to deliver FBP activities; (3) support for adherence and behavioral change through information, education and communication; (4) increasing coordination of HIV and nutrition interventions and policy issues with key stakeholders; and (5) monitoring and evaluation (M&E) systems for support of FBP programming. The development hypothesis is that improved nutritional, clinical and functional outcomes of malnourished HIV-positive adults and orphans and vulnerable children (OVC) are achieved through strengthening NACS, as well as creating effective linkages to community resources and economic strengthening (ES) initiatives. To monitor progress toward these outcomes, the FBP activity has identified three necessary intermediate results (IRs) and developed strategies to achieve each one.  Intermediate Result 1: Increase the provision and access of therapeutic and supplementary nutritional products to PLHIV  Intermediate Result 2: Strengthen capacity to implement FBP  Intermediate Result 3: Strengthen the collection, analysis and use of data to inform policy and planning for provision of nutritional care in the context of HIV The activity has been implemented over three phases, beginning in September 2009. Phase Two began in September 2012 and ended in September 2013. The final phase was implemented until September 2014, with a no-cost extension given for activities lasting until December 2014. Since inception, the FBP activity has undergone 12 modifications with USAID, including the approval of the final two modifications in 2012 and in 2013. The majority of these modifications revised the number of targets (beneficiaries and facilities), as well as budget realignments (Modifications 2, 3, 5, 6, 9, and 12). By the end of fiscal year 2014, the FBP activity aimed to reach a total of 219,630 beneficiaries over the five-year period, and to provide technical support in the form of NACS to at least 500 health facilities within either two city administrations or five regional states, targeting populations in urban and peri-urban areas. 4 | P a g e EVALUATION METHODS & LIMITATIONS METHODOLOGY This evaluation used a mixed methods approach, employing both quantitative and qualitative techniques. Quantitative methods included analysis of secondary data extrapolated from FBP activity documents, (such as quarterly and annual reports and M&E Plan) and national reports (refer to Annex II). Qualitative methods included multi-level data collection methods—key informant interviews (KIIs) and focus group discussions (FGDs) (See Tables 1 and Table 2). The sampling universe encompassed 494 health facilities across five regions and two city administrations. USAID/Ethiopia selected four regions, namely Amhara, Oromia, SNNPR and Tigray and one city administration, Addis Ababa. The selection criteria for selecting the health facilities were based on case load, geographical representation, and feasibility of roadside access. Facility records were reviewed for a data quality assessment. A total of 18 months of facility records over the five￾year period of performance were randomly selected. These records were checked for completeness and consistency. The evaluation team conducted 70 KIIs with key stakeholders at three levels: national, with representatives of GOE, other donors, and Save the Children US; regional, with Regional Health Bureaus (RHBs), Save the Children US regional program coordinators and IPs, and; at the local service delivery level with health facilities, where FGDs were carried out with FBP beneficiaries. At the national level, the team conducted 13 in-depth interviews with representatives from the central government, USAID/Ethiopia, Save the Children and at the Pharmaceutical Fund and Supply Agency, Federal Government of the Republic of Ethiopia (PFSA). Thirty-percent (21) of all KIIs were conducted at the RHB level. The team interviewed 36 health staff across the 27 health facilities we visited (see Table 1). Therefore, over half of the KIIs were health workers and of these, 20 worked in hospitals and 16 worked in health centers. The team conducted 32 separate FGDs across four targeted beneficiary groups: HIV+ women (age 15+) which represented 59% of all FGDs, women at prevention of mother-to-child transmission (PMTCT) comprised 9% of all FGDs, HIV+ men (age 15+) were 25% of the total. There only one OVC FGD, and one mixed beneficiary (male and female) FGD. Table 1: Total KIIs conducted disaggregated by level of the health system Region Health Facility Regional National Total Addis Ababa 4 4 8 Amhara 4 2 6 Oromia 13 4 17 SNNPR 11 6 17 Tigray 4 5 9 Total 36 21 13 70 5 | P a g e Table 2: FGDs conducted disaggregated by region, respondent group, and type of health facility Region Women Men PMTCT OVC Mixed Total HC Hospital HC Hospital HC Hospital HC Hospital HC Addis Ababa 2 2 1 2 - - - - 7 Amhara - 2 - - 1 - - 1 1 4 Oromia 1 3 1 2 - 1 - - 9 SNNPR 1 4 - 2 1 - - - 8 Tigray 1 3 - - - - - - 4 Total 5 14 2 6 2 1 - 1 1 32 DATA COLLECTION Quantitative analysis of secondary data included review and analysis of FBP activity documents and studies, national reports, M&E Data Plans, and Save the Children US service data and studies (see Annex III: List of References). Qualitative methods for primary data included multi￾level data collection using KIIs and FGDs (see Annex IV: Evaluation Plan and Analysis Matrix). The team conducted KIIs with key stakeholders at three levels; national, with representatives of the GOE, other donors, and Save the Children US; regional, with RHBs, Save the Children US regional program coordinators and IPs, and; local with health service facilities, where FGDs were carried out with beneficiaries and at health services sites. Our team visited supply chain storage facilities where they divided into two sub-teams. We used a checklist which asked two questions: (1) Briefly descried the condition of the storeroom where FBP commodities are stored; and (2) Briefly describe the storage of FBP commodities. One team conducted KIIs with national stakeholders, as well as interviews and FGDs with regional and health facility staff in the Amhara and Tigray regions. The other team conducted regional and health facility interviews and FGDs with beneficiaries in Oromia and SNNPR. Together, both teams conducted KIIs with stakeholders and health facility staff, and FGDs with beneficiaries in Addis Ababa, in order to standardize qualitative tools and interview techniques in the first three days of field work. One team conducted KIIs with national, regional and health facility staff while the other team carried out FGDs with beneficiaries in five regions: Addis Ababa, Amhara, Tigray, Oromiya and SNNPR. The team completed 70 KIIs with national, regional, and local health service staff, and 32 FGDs with 245 beneficiaries (See Annex V: Data Collection Instruments for KIIs and FGDs and Annex VI: Sources of Information). Data Analysis Data analysis procedures included descriptive analysis to illustrate FBP’s various types of interventions and other characteristics; a content and thematic analysis comprising the core of qualitative data analysis of documents reviewed, and KIIs with national, regional and health facility staff and FGDs to identify common issues, themes and patterns for each evaluation criteria (relevance, effectiveness, and sustainability). Procedures also included a comparative analysis to examine findings from secondary data across different regions and themes, and to 6 | P a g e identify challenges; an analysis of gender implications of FBP approach(es) to ascertain the extent that the FBP activity addressed and met both women's and men's needs; and a secondary analysis of M&E Plan data, and other Save the Children US M&E system data, and data quality assessment to analyze data management systems. LIMITATIONS OF METHODS There were two main drawbacks. First, the 32 FGDs did not include sufficient representation from different beneficiary groups, particularly OVC and women in prevention of mother-to-child transmission (PMTCT). The most challenging element was finding sufficient OVC during the time of site visits, which, according to health facility staff, was due to their infrequent visits for primary health problems, and their tendency to discontinue the RUTF/RUSF when their health conditions improved. According to health facility staff, PMTCT beneficiaries are relatively few in number compared to women in Antiretroviral Therapy (ART), and have specific dates of appointment for antenatal care (ANC) compared to ART attendees, which made it difficult to ensure clients’ presence on the day of any evaluation site visit. Sites selection was the second drawback. Roadside access influenced sites selection, but was mitigated by the number of regions selected and facilities with high case load in consultation with the RHBs. To minimize any interviewer bias associated with qualitative data collection, the team conducted data collection in pairs, and thematic analysis as a group. FINDINGS Domain 1—Effectiveness of Activity Implementation and Management Q1. What have been the achieved outputs versus planned outputs? Overall, FBP achieved or underachieved 12 of its 16 targets, as shown in Tables 3-9 below, where ratings for achieved targets are shown in green (100% or >), ratings for underachieved targets are highlighted in yellow, (90%--99%) and ratings for targets not achieved are highlighted in red (<90%). Noteworthy is that of the four underachieved targets, each achieved 94 to 98 percent of target. The Life of Project (LOP) targets were achieved for Goal 1. Improved Nutrition, Clinical and Functional Outcomes with 64 percent (target 60 percent) of clinically malnourished PLHIV including PMTCT and OVC clients graduating from the activity, and 0.8 percent (target <5 percent) of PLHIV including PMTCT and OVC clients who died during the course of treatment. 7 | P a g e Table 3. Goal: Improved Nutrition, Clinical and Functional Outcomes among PLHIV Adults, Pregnant and Postpartum Women and OVC Indicator Target 2009-2014 Actual 2009-2014 Difference Percentage of clinically malnourished PLHIV including PMTCT, and OVC clients who graduated from the activity. >60 percent 64 percent (+4 percent) Percentage of PLHIV including PMTC and OVC clients who died during course of treatment <5 percent 0.8 percent (+4.2 percent) Source: Monitoring & Evaluation (M&E) Plan Data, FBP, September 2009 – December 2014, USAID/Ethiopia. For IR1, Improved of Therapeutic and Supplemental Nutrition Interventions by PLHIV and OVC, the activity met five of its six targets for improved use of therapeutic and supplemental nutritional interventions by PLHIV and OVC clients. FBP exceeded its targets for the number of PLHIV, including PMTCT and OVC clients, who (1) received nutritional assessment (200 percent); (2) received nutritional counseling (165 percent) (3) were clinically assessed and found to be severely malnourished (107 percent); (4) were clinically assessed and found to be moderately malnourished (117 percent), and (5) were clinically malnourished PLHIV clients who received therapeutic and/supplementary food (153 percent). FBP underperformed on its target for the number of graduated malnourished clients that benefited from ES (56 percent). Table 4. IR1: Improved Use of Therapeutic and Supplemental Nutritional Interventions by PLHIV and OVC Indicator Target 2009- 2014 Actual 2009- 2014 Rating Number of PLHIV including PMTCT, and OVC clients who received nutritional assessment 771,177 1,545,530 +774,353 200% Number of PLHIV including PMTCT, and OVC clients who received nutritional counseling 914,476 1,513,644 +600,396 165% Number of PLHIV including PMTCT, and OVC clients who were clinically assessed and found to be severely malnourished 63,539 68,462 +4,923 107% Number of clients including PLHIV, PMTCT, and OVC clients who were clinically assessed and found to be moderately malnourished 147,091 172,564 +25,473 117% Number of clinically malnourished PLHIV clients who received therapeutic and/or supplementary food 156,000 240,022 +84,022 154% 8 | P a g e Table 4. IR1: Improved Use of Therapeutic and Supplemental Nutritional Interventions by PLHIV and OVC Indicator Target 2009- 2014 Actual 2009- 2014 Rating Number of graduated malnourished clients benefited from ES 16,420 9,327 -7,093 (-56% ) Source: Monitoring & Evaluation (M&E) Plan Data, FBP, September 2009 – December 2014, USAID/Ethiopia. FBP exceeded four of its targets for IR 2: capacity strengthened for the implementation of quality FBP interventions through technical leadership, that is, number of service providers trained in nutrition and HIV/AIDS (149 percent), and number of data clerks trained on M&E of NACS (242 percent). FBP met its target for data/reports shared with the GOE and other partners (100 percent), and percentage of health facilities and distribution outlets that collect and report FBP specific data (100 percent). Although the activity did not achieve three of its other targets, FBP did rank over 95 percent performance on the number of service health facilities ready to provide FBP services (98 percent), the number of health facility store managers and pharmacists trained on NACS for logistics (98 percent), and number of case managers trained on nutrition and HIV/AIDS (95 percent). Table 5. IR 2: Capacity Strengthened for the Implementation of Quality FBP Interventions through Technical Leadership Indicator Target Actual Rating Number of health facilities sites ready to provide FBP services 500 494 -6 98% Number of service providers trained on nutrition and HIV/AIDS 4,790 7,182 +2,392 149% Number of Health facilities Store Managers, Pharmacists trained on NACS for Logistics 1,670 1,640 -30 98% Number of Case managers trained on nutrition and HIV/AIDS 3,320 3,169 -151 (-95%) Number of data clerks trained on M&E of NACS 628 1,522 +894 242% Source: Monitoring & Evaluation (M&E) Plan Data, FBP, September 2009 – December 2014, USAID/Ethiopia. In addition, as shown in Table 6 below, FBP achieved its IR 3 target for percentage of health facilities and distribution outlets collecting and reporting FBP data (100 percent), with 94 percent having shared data and reports to government and other partners, while underachieving on the proportion of health facilities handed over to the government (87 percent). 9 | P a g e Table 6. IR3: Strong Collection, Analysis and Use of Data Informing Policy and Planning for Provision of Nutritional Care in the Context of HIV Indicator Target Actual Rating FBP data report shared to government and other partners 18% 17% 94% Percentage of health facilities and distribution outlets collect and report data specific to FBP 100% 100% 100% Proportion of Health Facilities Handed over to GOE 100% 87% (-13%) Source: Monitoring & Evaluation (M&E) Plan Data, FBP, September 2009 – December 2014, USAID/Ethiopia Q2. How efficiently have the output results been achieved against inputs and budgets used? Save the Children US incurred costs of $19 million over the course of the five-year FBP activity, of which $7.4 million was for other direct costs. RUTF costs $50 per carton of 150 sachets ($.33 per sachet), while RUSF costs $48 per carton ($.32 per sachet). Hence, the procurement expense associated with a prescribed six-month course of therapy comes to $177 (1.33x90 days plus $.64x90 days). A senior FBP staff member at Save the Children US said: “FBP used government systems and, as such, is very relevant and meets government requirements for donor-funded projects and activities.” The evaluation team did not have data to analyze the efficiency of the commodity management and supply chain process handled by the PFSA, a unit of the central government. Data from interviews with PFSA and other stakeholders indicated that the system in place did not promote efficiency or high quality of commodities used by FBP, and that there were substantial problems with leakage and loss issues. Commodity Management and Supply Chains PFSA handles commodity management and supply, and reported that the system is integrated with their routine activity of transporting drugs and medical supplies to health facilities. However, Plumpy Nut and Plumpy Sup are bulky and require more space, which increased transportation costs. One fourth of health facilities visited by our team did not have enough storage space because the storage room was not designed to be a pharmacy store, resulting in poor handling. The team found that Plumpy Nut cartons were torn and the sachets were spread over the floor, and cartons were stored in a room with no ventilation. Pharmacists/druggists and store managers in many of the facilities visited indicated that Plumpy Nut and Plumpy Sup were managed just like any other drugs, although they acknowledged that there was no system in 10 | P a g e place for balancing the prescribed amount against the amount dispensed, and this was reaffirmed by PFSA. Save the Children US changed the original memoranda of understanding with each partner to contracts and subcontracts with all IPs. Q3. What is the number and percent of clinically malnourished HIV-positive clients who received food supplements and graduated from the activity? According to data from the Save the Children US Management Information System (Table 7), 56,533 of the clinically malnourished HIV-positive clients and OVC who received food supplements, graduated from FBP. Among the 113,609 clients whose status is known (47.3% of the total), this represents a 49.8% percent graduation rate. Forty-eight percent of both males and females in the under-5 age group graduated. Close to 56 percent of both males and females in the 5-14 age group graduated. For the 15-17 age group, 55 percent males, 47 percent female PMTCT clients, and almost 57 percent other females graduated. Table 7. Clinically Malnourished HIV-Positive Clients and OVC Who Received Food Supplements and Graduated, 2009-2014 Age Group (by years) Gender Received Food Supplements & Graduated No. of clients who received therapeutic and/or supplementary food with known outcomes No. of graduates Percent graduated <5 M 20,537 10,030 48.8 F 21,561 10,344 48.0 Total U5 42,098 20,374 48.4 5-14 M 4,306 2,419 56.2 F 3,748 2,064 55.1 Total 5-14 8,054 4,483 55.7 15-17 M 739 408 55.2 F (PMTCT) 36 17 47.2 F Others 658 372 56.5 >18 M 24,144 11,769 48.7 F (PMTCT) 1,575 761 48.3 F Others 36,305 18,349 50.5 Total M 49,726 24,626 49.5 F(PMTCT) 1,611 778 48.3 F( Others) 62,272 31,129 50.0 Grand Total 113,609 56,533 49.8 11 | P a g e Q4. Of those that graduated from the activity, what number relapsed? Of the total number of clients graduating, a much smaller proportion relapsed, 2,314 (2 percent) of total graduating (Table 8). For example, women in PMTCT (0 percent) ages 15-17 and 0.8 percent for PMTCT over 18 years, and for male (.6 percent), and female (.7 percent) under 5 years. This may be due to regular surveillance of both groups, especially for those registered for PMTCT. For non-PMTCT females, the percentage is higher in both the age groups 15-17 (4 percent) and over 18 (3 percent). For males in each of these age groups, the differences were not significantly different. Table 8. Cumulative Number and Percent of Clinically Malnourished HIV-Positive Clients and OVC Who Relapsed After Graduation, 2009 -2014 Age Group (by years) Gender Relapsed after Graduation No. Percent <5 M 61 0.6% F 77 0.7% 5-14 M 60 2.5% F 53 2.6% 15-17 M 8 2.0% F (PMTCT) - - F Others 15 4.0% >18 M 318 2.7% F (PMTCT) 6 0.8% F Others 559 3.0% Total M 447 1.8% F(PMTCT) 6 0.8% F( Others) 704 2.3% Grand Total 2314 2.0% In addition, the team analyzed the number of clients who were non-responders, lost to follow￾up (LTF), defaulted and died. Clients who were LTF were the largest group at 25,469 (10 percent), followed by those who defaulted, 20,556 (eight percent) with more men than women defaulting, and non-responders were at 9,267, with more women not responding than men, and 1784 for those who died with more men dying than women. 12 | P a g e Box 1 “FBP has given job opportunities to others who did not have any job and strengthened public, private and community partnership. But, beneficiaries developed [a] dependency syndrome, and many do not have skills, so it was difficult to link them to factories that need certain skills. Also, private companies pay low salaries and the beneficiaries are not interested to get employed.” NGO Partner, Oromia “We are linking FBP graduates to different opportunities through the ‘Back-to-Work’ initiative. It requires lots of negotiations with company owners and patience. Some people will start the work and discontinue, others are too selective or not interested. Some companies demand our beneficiaries to have the required equipment to be able to do the work, and because there is no matching fund from the side of the project, it was very difficult to support them with the required equipment.” NGO Partner, SNNP Table 9. Clinically Malnourished HIV-Positive Clients and OVC who Received Food Supplements and (a) Non-Responded, (b) Lost To Follow Up, (c) Defaulted or Died (by age and gender), 2009- 2014 Age Group (by years) Gender Non-Responder Lost to Follow Up Defaulted Died No. Percent No. Percent No. Percent No. Percent <5 M 1118 0.3% 5262 11.5% 4004 8.8% 123 0.3% F 1225 2.6% 5353 11.2% 4506 9.4% 133 0.3% 5-14 M 397 4.5% 890 10.1% 560 6.3% 40 0.5% F 326 4.4% 799 10.7% 537 7.2% 22 0.3% 15-17 M 76 5.7% 152 11.3% 95 7.1% 8 0.6% F (PMTCT) 1 1.5% 12 17.9% 6 9.0% - - F Others 75 6.3% 92 7.7% 97 8.2% 22 1.9% >18 M 2322 4.8% 5068 10.6% 4238 8.8% 747 1.6% F (PMTCT) 92 2.1% 300 6.8% 401 9.1% 21 0.5% F Others 3635 4.8% 7541 9.9% 6112 8.0% 668 0.9% Total M 3913 - 11372 11.0% 8897 8.6% 918 0.9% F(PMTCT) 93 0.4% 312 7.0% 407 9.1% 21 0.5% F( Others) 4036 - 8432 10.4% 6746 8.5% 712 0.6% Grand Total 9267 - 25469 10.6% 20556 8.5% 1784 0.7% What percentage of those who graduated, benefited from ES? According to M&E Plan data and Activity Annual Progress Reports, FBP did not directly implement an ES Component. Beginning in year 2, only 429 NACS graduated clients were referred, with 1,584 in year 3, 5,058 in year 4, and 2,089 in year 5, resulting in 56 percent of target achieved. FBP referred clients to World Food Programme (WFP) through the Federal HIV/AID Prevention and Control Office of the FMoH (HAPCO), where they were subject to a needs assessment based on selection criteria established by a HAPCO- created joint committee. 13 | P a g e FBP did provide financial support to some NGOs to promote ES activities. We interviewed four NGO partners that received referrals from FBP (see Boxes 1-2 for selected KII respondent quotes). Not all FGD groups responded to questions, and ES activities were designed primarily for clients who graduated, while the FGD participants interviewed were active clients receiving supplements. Among FGDs, the main question asked was, “Have you been linked by FBP to economic strengthening activities?” Only nine FGD groups reported having been linked. Reports from these FGDs varied: “When I graduated, they sent me to MOMs2 for income generation training, and I received three days of training about poultry and fattening goat. Afterwards they did not contact me, but they gave loan to other trainees,” “got training and three days per diem (50 Birr),” “first HAPCO gave us 25 kg flour, egg, oil and vegetable every month for six months, then collaboration with ES officer gave us seven days training and helped us save money.” Other group responses were, “I forgot the training but at the end they gave 900 Birr each to do some business whatever we like,” and “have not graduated yet,” and “took the training and got saving skill,” and WFP gave me training in business skills for one week.” When asked in FGDs whether they were informed and linked to ES activities, 14 groups responded positively with the majority of the participants within each FGD asserting they were referred. It is worth noting that among those 14 groups, 44% were FGDs with women+; although this beneficiary group represents close to 60% of all FGDs. All of the respondents within these FGDs were referred, save for a couple of FGDs where at least half of the participants were referred to ES activities and there was one group where only one person was referred. (See Box 3 for an illustrative quote.) 2 Mums for Mums (MOMs): The local partner for ES activities in Tigray region. Box 2 “The collaboration with FBP was small by HIDA standards but [it] was a valuable one. From 2011-2014, we received annual grants from Save the Children US. The seed capital was too little 1,000-3,000 birr and there was no follow-up.” NGO Partner, Addis Ababa “‘Back-to-Work’ was a really innovative strategy, and helped with long term financial security. Without the inputs from Save the Children US, we would not have established ‘Back-to-Work’. Budget delays at the beginning of the project impacted ability to implement planned activities. The demand for ‘Back￾to-Work’ was high. In one town, we had 2,000 applications. There was insufficient time to allow for ‘Back-to-Work’ to reach demand and not enough resources were [allocated]. NGO Partner, Tigray Box 3 "…they first gave me 3,000 birr but deposited 2,000 [of that amount] into a bank. They advised me to start a business with the remaining 1,000 birr. That is not enough to start anything.” Male+ FGD Respondent, Oromia 14 | P a g e Nonetheless, 18 groups (56%) stated that they were not informed or referred. Over half of the FGDs (56%) were among the women+ beneficiary group and 28% were among the men+ group. Within these negative responding FGDs, only one participant was referred to ES activities. Illustrative responses among those not referred were: “no one told us,” “nobody referred us,” “we do not know,” and “nurse told me the organization didn’t request them to send graduates to ES. What was the effectiveness of the FBP M&E and Information System? The team carried out the Data Quality Assessment at 10 FBP facilities in four regions, for 18 randomly selected months over the five-year period (see Table 10). All the facilities reviewed reported on eight required NACS indicators, however only four reported monthly, with four others having a varied reporting schedule. The number of clients receiving therapeutic or supplementary foods was reported differently by different facilities. Five facilities reported only new recipients each month, while two reported a mix of new and old recipients. This resulted in double counting of recipients, affecting the overall number of beneficiaries reported. In one of the facilities, there was a mismatch between the number of PLHIV diagnosed as malnourished and numbers who were receiving RUTF or RUSF. Table 10. Data Quality Assessment at Facility Level Name of health facility Region Reported on all Eight Required Indicators Report on clients’ who received therapeutic and/or supplementary food Reporting period Consistency of reporting Reason for reporting gaps Adama Hospital Oromia + New cases each month Monthly 2012-2013 reports for many months not available Data clerks not replaced Shasheman e Hospital Oromia + New cases each month Monthly Consistent n/a Shasheman e Health Centre Oromia + New cases each month Monthly Consistent n/a Sodo Hospital SNNPR + New cases each month 20th of month to 31st of next month Data before 2010 not available n/a Sodo Health Centre SNNPR + Mismatch between # diagnosed and # supplemented with RUTF/RUSF: September 2012 and August 2014, 8 diagnosed, none 20th of month to 31st of next month Health worker reports in registration book did not match data Reported miscommun ication between data clerk and health workers. 15 | P a g e Table 10. Data Quality Assessment at Facility Level Name of health facility Region Reported on all Eight Required Indicators Report on clients’ who received therapeutic and/or supplementary food Reporting period Consistency of reporting Reason for reporting gaps supplemented. reported Hossana Health Centre SNNPR + Reporting some months only new recipients of RUTF/RUSF Other months new & old reported: December 2013 both old & new reported, other months only new cases reported 20th of month to 31st of next month Inconsistent Unknown St.Luke Hospital Oromia + Both old and new recipients of RUTF/RUSF reported 2010 -2012 report quarterly 2013 monthly Consistent n/a Jimma Hospital Oromia + New cases for the month Monthly, every two month or quarterly. *not possible to review records for sampled months Inconsistent Unknown Dessie Referral hospital Amhara + Reporting months did not match the period of sampled months Mekele Ayder Referral Hospital Tigray + Reporting months did not match the period of sampled months “+” means reported on all eight indicators; “n/a” means non-applicable In addition, we visited 27 health facilities sites. Overall, the team found health facilities reporting to the respective RHBs. However there was no regular feedback mechanism from RHBs to health facilities, which could enhance quality service provision. At health facilities where FBP management was handed over with transition of responsibilities to the RHBs, the monitoring and supervision was reduced significantly. Some health facilities reported that no one had 16 | P a g e supervised the activity over a period of a year. There is no adequate data about the target population from which to make robust estimates on whether the activity influenced extended survivorship or disability, either positively or negatively. Data is managed and reported at different levels, primarily as part of the regular health system. While Save the Children US was implementing FBP, there was a separate reporting system for activity monitoring, using formats specifically developed for this purpose. Data used by FBP activity relied extensively on that reported on a quarterly basis from health facilities, with problems in terms of the timing of data collection, including variations between the regions. Only two of the five RHBs interviewed in KIIs i.e., Amhara and Oromia reported how data is managed and monitored. In Tigray, a coordinator said that “the RHB has a technical working group made up of RHB, PFSA and all stakeholders and NGOs. They meet monthly and FBP results are fully integrated into the TWG planning and reporting schedule.” In the Oromya region, an RHB representative reported that “there is some level of carelessness from the data clerks and health workers for the data is parallel to the government system, and there is documentation problem. Sometimes, they do not leave any copies for their own and do not document. We need this data for the following year’s planning.” A representative from Amhara region reported that “the FBP activity shares the data every quarter and based on these discussions, effective plan and implement activities at all levels are made.” Among 27 health facility staff interviewed, 40 percent (11) responded "yes” when asked if NACS increased your workload? 37 percent (10) indicated “no,” with comments such as, “the work is not difficult and you see positive outcomes,” “it’s part of our job,” “the work gradually improved,” and, “need is great.” Four facility staff reported “yes” because the facility does not have manpower, and two staff did not respond. Health facility staff did emphasize that measuring height and weight of clients and manually calculating body mass index (BMI) is a time￾consuming activity. Since there are many other patients waiting for their turn, it increases the chance of facility staff making errors. As a result, records might not necessarily reflect the true nutritional status, resulting in false inclusion in, or exclusion from the activity. Because stocks are poorly tracked after their arrival at the dispensary of each health center, it was not possible for us to determine how each FBP beneficiary was able to receive their intended doses of RUTF/RUSF, especially since stocks were also given out for other purposes. Some pharmacists and druggists reported that RUTF/RUSF were being prescribed to adult malnourished patients suffering from other chronic illnesses such as cancer, and to surgical patients to enhance recovery. Of the 27 health facilities visited, the team completed data quality assessment using a Data Quality Analysis Tool (See Annex VII: Food By Prescription Data Quality Assessment Form) at 10 facilities during the review of health facility records and found a large number of irregularities. The team found that the monthly reporting period varied from facility to facility. For example, at four facilities visited, the team found one facility reporting monthly data, while three others reported every 40 days; from the 21st of one month to the last day of the next month. In addition, two facilities reported only new recipients of RUTF/RUSF for the month, while two 17 | P a g e others were reporting both old and new. Thus FBP would not know exactly how many people benefited from the activity. Three data clerks reported that their records sometimes did not match that of health workers. They mentioned, for example, those clients that are moderately malnourished and receive RUSF are being prescribed RUTF and vice-versa. In some facilities that were visited, data clerks were not available or there were none on staff, and the time it took to find a replacement resulted in data loss for periods of time. In fact, numerous errors were found in data recording, including wrong dates and measurements. This resulted in the incorrect classification of clients concerning whether they were normal, or moderately or severely malnourished. These findings were corroborated in the Inspector General Office Audit of USAID/Ethiopia HIV Care and Treatment Activities (2014), evidencing a lack of maintenance, by officials, of adequate or organized records at all the health facilities visited, as well as health facility underreporting for quarters, and non￾standardized reporting periods. Domain 2—Sustainability Q5. To what extent have NACS indicators been integrated within FBP-supported government and health facility reporting formats and work streams? According to FBP M&E data, the activity fully met two of five targets for strengthening capacity for implementing quality FBP, that is, the number service providers trained on nutrition and HIV/AIDS (149 percent), and the number of data clerks trained on M&E of NACS (242 percent). FBP achieved nearly 100 percent of targets for the number of facilities ready to provide FBP services (98 percent), the number of health facility store managers and pharmacists trained on NACS for logistics (98 percent), and the number of case managers trained on nutrition and HIV/AIDS (95 percent). NACS is integrated into the public health system. First, the FMoH adopted three NACS indicators into the HMIS. These include a number and percentage of PLHIV who are nutritionally assessed and moderately malnourished; nutritionally assessed and severely malnourished and number and percent of those who received nutritional support. This requires every health facility to report on these indicators and be accountable to carry out the NACS activities. This is an important step in assuring NACS sustainability within the public health system. Secondly, capacity building in NACS through training of health workers and provision of materials for nutritional assessment such as weight scales and length boards provide a measure of sustainability within the public health system for the short- to medium-term. Thirdly, the FBP activity is implemented in 90 percent of public sector health facilities. Q6. What is the capacity of supported RHBs and health facilities to plan for and provide NACS services in the absence of FBP support? Our team found that in all of the five RHBs and health facilities visited, NACS is part of the work plan and discussed in multi-disciplinary meetings, and that it will continue as long as the supply 18 | P a g e is made available. In all the facilities visited, there was at least one trained staff member assigned to the ART and/or PMTCT clinics. Similarly, all 36 health workers interviewed reported that the training improved their knowledge and skill on nutrition assessment, diagnosis, and counseling of PLHIV. Of these, 21 reported their role in the FBP project as nutrition assessment, BMI measurement, counseling and the prescribing of Plumpy Nut supplements, while three had both managerial and clinical responsibilities. There was no single focal person for NACS in the RHBs. It is a shared responsibility of the nutrition, HIV and in some cases the curative and rehabilitation clinical officers. The team did not find shortages of RUTF/RUSF at health facilities visited. RHBs and health facility staff reported that they will continue prescribing as long as the supply is available. The cost of RUTF and RUSF is high and is the main reason that the activity might not be sustainable as it currently stands. According to the Supply Chain Management System (SCMS), the full course of treatment for severe acutely malnourished (SAM) cases costs about $180.00 per beneficiary, and that for moderately malnourished (MAM) cases, it costs about $60.00 per beneficiary. A senior FMoH nutrition advisor emphasized that “the government will not be able to continue supplying these products because of cost.” He added that the WFP and the Clinton Foundation are researching new and cheaper chickpea-based RUTF & RUSF products. However, a WFP staff member raised questions about this initiative since the supply of chickpeas available in the country is not as expected. She said that NACS will continue as the result of the capacity-building efforts, and that cheaper but nutritious products like Corn-Soya Blend (CSB) will be a better alternative. The FMoH nutrition case team coordinator reported that there has been no discussion on the sustainability of the FBP activity. At the same time, a senior official at Save the Children US informed that NACS seems to now be integrated into the overall HIV counseling, care and support. He was not sure, however, how RUTF/RUSF will be continued, and stated that it is necessary to look for cheaper, and preferably local, products. These findings suggest that health facilities will continue to provide NACS to malnourished PLHIV but may not be able to provide RUTF/RUSF without external support once FBP is discontinued. FBP reports there are sufficient numbers of master trainers in all RHBs that could continue the training even after the phasing out of the activity, although the financial viability to cover the cost of trainings is questionable. Q7. To what extent has NACS been integrated into pre-service training activities for nurses, midwives and medical doctors in targeted universities? FBP informed us that the activity was designed to allow the consideration of the inclusion of NACS in the pre-service curricula of doctors and mid-level health workers (nurses and midwives) to assure continuity of the training in a sustainable manner. One initiative was set up in Hawassa Box 4: “We were not conducting [a] nutritional assessment nor counseling on better nutrition and healthy eating before the training. This has now changed. We assess and counsel clients and tell them that the Plumpy Nut is a drug, and not to be shared.” Health Center Manager, SNNPR 19 | P a g e University for the first year of the activity where one group of graduate students received pre￾service training, but this was later discontinued. Jimma University was another institution where the training was planned but did not materialize. Our team also visited Mekele University in Tigray region, where NACS training was consolidated into the curricula for training of doctors and health workers. Because of this integration, new graduates will be equipped with the knowledge and skills needed to provide the service. Domain 3—Relevance Q8. To what extent are FBP’s objectives consistent with the needs of the activity’s beneficiaries/target groups? FBP’s objectives were to improve nutritional, clinical, and functional outcomes among PLHIV adults and pregnant and lactating women, and OVC, through periodic nutritional assessment and counseling, and provision of therapeutic and supplementary food support to malnourished PLHIV. At the activity launch, FBP carried out a baseline survey showing that a majority of adults (79.5 percent) was never assessed for malnutrition, but more than three quarters received nutritional counseling. There was, however, no counseling for dietary diversification or increasing micronutrient intake using the locally available sources. A FBP internal evaluation showed that NACS were seen to be well integrated into HIV care and were a routine activity. Apparently, the activity did not monitor the impact of NACS. One hundred percent of FGD groups (32) indicated satisfaction with and effectiveness of FBP services. All FGD groups (100 percent) also indicated that they were satisfied with RUFT/RUSF. All agreed that RUFT/RUSF helped them to better manage their health, and that “within a few days” they noticed “increased appetite and gained 5- 7 kg,” “improved water intake and CD4 count,” and “improved adherence to ART.” FGD groups in Addis Ababa, Tigray, and Amhara emphasized that “before Plumpy Nut, we could not eat and had insomnia, but after, the nausea and vomiting ceased.” One man said: “I was not able to order my body to work. Even I felt very tired when I walked a short distance. After taking it [Plumpy Nut] I was able to do hard work.” Other FGD participants expanded on health and nutrition improvements since taking RUFT/RUSF. One male FGD participant at Adare hospital in Hawassa, SNNPR said: "I lost energy and could not work. I began to feel exhausted when walking for short distances. My face turned pale and was forced to cover afraid of what people might say about my appearance. After the RUTF, I started to feel healthier and stronger.” Another male FGD participant from Shashemane hospital, Oromia region said: “I am a farmer and used to pay others to do the ploughing. After I took RUTF, I began to plough my own land and harvest crops." In addition, a female FGD participant from Sodo Health Centre in the SNNPR said: "I was not able to walk, let alone do some household chores. After the RUTF, I became healthy and strong and started my own small business and began to generate income.” KIIS with 36 health facility staff from 27 facilities, as discussed above, indicated that NACS provided at facility level met the needs of severely and moderately malnourished PLHIV adults, 20 | P a g e pregnant and postpartum women and OVC, and brought about improvements. However, the data collected did not provide sufficient information to conclude that improvements will be sustained over time for clients interviewed in FGDs. An RHB representative in Addis Ababa emphasized that “the activity is key in the nutritional rehabilitation of PLHIV and also makes great contribution in the prevention of some other infections which could have occurred if FBP was not there. In addition, the project is linking the beneficiaries to ES opportunities to let them back to work. The activity has significantly improved adherence to ART drugs and also resulted in the reduction of mortality.” In Amhara, a Health Development, Promotion and Nutrition Officer at RHB noted that “the FBP project in Amahara and Amhara RHB are strong partners. We’ve been working hand–in-hand together. And the Support FBP activity was very helpful in supporting and accelerating the already started HIV and nutrition related healthcare services in the region.” In Tesfaye, a nutrition focal person reported that “coordination was excellent from the very beginning of FBP although implementation was slow at the beginning. The RHB was involved in planning, capacity building.” Q9. To what extent do stakeholders (RHBs, health facilities), including beneficiaries, buy￾in to and own the goal, objectives and FBP implementation methods? We interviewed 13 RHB staff and 27 health facility staff members. RHB and health facility staff responses demonstrated a high level of knowledge about the nutritional needs of malnourished PLHIV, and commitment to meet their needs, as discussed under effectiveness of M&E System, above. They described the health and nutrition benefits in treating severely and moderately malnourished PLHIV. In terms of health outcomes, health workers reported increased appetite, weight gain, and increased adherence to ARV treatment. RHB representatives emphasized the importance of FBP’s approach in promoting integration, system strengthening and capacity building of health staff. FBP was viewed by all (100 percent) as having improved, within a short time, appetite and nutritional status, and increasing strength to be able to work. Stakeholder responses showed in-depth understanding of the implications of malnourishment and how to treat clients. All (100 percent) reported having seen the effect of RUTF/RUSF on their patients, that is, “decreased prevalence of opportunistic infections”, “increased effectiveness of the ART drugs,” and “improved health, nutritional status and survivorship.” For example, the ART program coordinator at Hawassa University Referral hospital in SNNPR said: “It is relevant to my work because I have seen rapid progress in improvement of health conditions. The RUTF made them healthy and strong and made ART drugs more effective.” Two RHB representatives noted that NACS also “prevented opportunistic infections like tuberculosis, and decreased death rates.” Of the 27 health facility staff (40 percent) interviewed, 11 reported that their workload had not increased since the introduction of NACS, but emphasized that it is meeting a need. Another 37 percent responded that their workload had increased, but that NACS was not difficult and there are positive outcomes, outweighing the increase in workload. Only four staff members (14 percent) responded that workload increased because the health facility did not have sufficient manpower, and two health facility staff did not respond. 21 | P a g e According to the PFSA staff, FBP integrated smoothly into their routine activity of transporting drugs and medical supplies to health facilities. However, the stock and distribution officer at the Addis Ababa branch said that RUTF/RUSF are bulky and require more space for both transportation and storage. This opinion is also shared by the FBP focal person at the SCMS, a USAID contractor working on the supply side of the activity. The utilization of government systems for storage and distribution, and service delivery is described as recognition of the activity’s relevance in government systems. Q10. To what extent are RUTF and/or RUSF palatable and useful to beneficiaries? All FGD groups (100 percent) reported that they either consume RUFT/RUSF as prescribed by health workers directly from the sachet, or eat it with buttered bread. Some beneficiaries did not like the taste. Most reported that the imported version (the Plumpy Nut in particular) is better than those produced locally. Many beneficiaries reported that RUTF is salty compared to RUSF while others reported vice-versa, but either way, they gradually got accustomed to the taste. According to FGDs in one region, there was a difference in preference reflected in market resale price of the two versions of RUTF, with the imported commodity costing 6 birr per sachet and the locally produced selling at 4 birr. One key finding was that the RUTF was popular among both family members, and in the community in general, so that beneficiaries generally shared their FBP foods with their children. Several FGD participants said, “Yes, we are sharing Plumpy Nut to our children because they cried if we are not sharing them.” Another participant shared with her children because she hated “to eat Plumpy Nut or other foods without sharing with them.” Among FGD participants, 21.8 percent reported sharing nutrition supplements, with most saying: “we share it with the children.” Sixty￾two percent of FGDs reported they do not share. Participants emphasized that “every member of the family knows that the supplements are prescribed” and “it’s a drug, not possible to share.” Yet it is clear from observations of Save the Children US and health facility staff that a much higher number of beneficiaries share or sell their foods. Both KIIs with health facility staff and beneficiaries interviewed in FGDs indicated that there is a steady flow of RUTF and RUSF into communities through local merchants, who appear to be supplied by FBP participants. Of the 32 FGD groups, 15 reported that the product is available in local shops, and three FGD groups indicated that “the government now forbids it.” Only one FGD participant reported that children access Plumpy Nut at local shops, saying that her “son is accustomed to the RUTF because he can access it from shops.” Of the few FGD participants who reported members of their community were selling it but they did not indicate at what stage they sell these products, nor their reasons for selling. Government bodies like bureau of health, trade and the police were working closely with Save the Children to mitigate selling of RUTF/RUSF. No KII or FGD participant discussed whether intake of other foods, that is, foods purchased by the beneficiaries or grown by their families, counted toward their daily dietary intake. This would be relevant for determining which foods, in the future, should be promoted. There is inadequate field level, anthropometric or operations research to determine the patterns of how RUTF/RUSF 22 | P a g e fit within the diets of the target households. Moreover, it is likely that these patterns are evolving, as local appetites become more accustomed to RUTF. Domain 4—Gender Q11. To what extent do FBP interventions address gender issues that expose women to HIV/AIDS or malnutrition? We found no evidence that gender-related issues were examined, flagged or tracked over time. Moreover, our team found that the FBP activity did not have a specific plan to address gender issues, either male or female, nor did the health worker training plans cover gender-related issues. The M&E Plan data show that the number of adult females enrolled exceeded the number of adult males, even though males outnumber females slightly in the adolescent population. According to RHB representatives, facilities disaggregated data and reported to FBP by sex and age. Of the RHB representatives interviewed, four indicated that although the data are collected on a disaggregated basis by sex and age, they are not used for activity analysis or planning. One representative noted that “the gender issue was not addressed in NACS training.” Another indicated that “the project is facility-based and client-need based and those who are in need of the drugs and are eligible are coming to the facilities and are addressed. We don’t get women directly and therefore were not specifically targeting them.” When asked if FBP activities have any influence on the status of women and men, only two RHB representatives responded. One reported that “those that found work have clearly improved their status, and in follow-up we found they are now able to afford more food and school fees.” The other indicated that “FBP tries to integrate mothers who are graduated from the FBP activity with economic support activities/Back-to-Work program through different organizations such as NAP+. Many of these who were linked to the Back-to Work-program were women. However, this activity was limited only in few towns.” CONCLUSIONS Domain 1—Effectiveness of Activity Implementation and Management The FBP activity demonstrated success in achieving its outputs, having exceeded targets in Years 3 and 4. The outcome of improved survivorship remains unknown. There is a need to improve understanding about why individuals drop out, and the apparent inability to track individuals with personal identification records that are carried across regions. Many FBP supply issues were outside the control of the IP, since supply management, logistics and distribution to health 23 | P a g e facilities were facilitated by PFSA. Overall, our team did not have sufficient data to assess the efficiency of output results. There appears to be an underestimate of the true relapse rate, since some proportion of the participants who graduated inevitably relapsed but did not return to the activity, perhaps because of the distance to the health facility, or the severity of their illness (severely ill or terminal). The referral of clients to ES activities was the least well-developed component of the FBP and did not reach a substantial number of graduated clients, despite reports from RHBs, and local NGOs that conduct ES activities, citing the importance of Save the Children US technical and financial support in promoting ES activities. In the M&E Information and Management System, there were a substantial number of irregularities in activity data reporting with variation from facility to facility, and possibly between health facility staff, which casts doubt on the quality and reliability of data reporting from facilities to the FBP activity, and from facilities to RHBs. Domain 2—Sustainability Integration of indicators in the HMIS was an important first step to ensuring that FBP services will continue to be provided after the activity has ended. Yet, there is no guarantee that the GOE will be able to maintain this type of service, given the concerns expressed about the cost of RUFT/RUSF supplements. The inclusion of NACS related indicators in the HMIS is a first step toward sustainability that will promote integration of the activity within the public health system, and beyond FBP-supported health facilities. Although NACS’s role is well recognized in the care and treatment of PLHIV, most health workers did not consider it an integral part of their routine activities. Training helped to build the knowledge and capacity of health workers to implement NACS. All respondents reported that NACS will be limited without Plumpy Nut/Sup or a substitute, and considering reports of high staff turnover and internal rotations within health facilities, especially hospitals, the quality might also be affected. RHBs reported there are a sufficient number of master trainers at their level, and the in -service training can still be applied to a critical mass of people, given that one master trainer can train up to 173 people per session and can conduct multiple sessions in a year. Financial viability to sponsor training sessions over the medium term, though, is questionable. The NACS component may be challenging if continued training and reinforcement are not available. The reasons for inadequate integration of NACS training into pre-service curricula at targeted universities are varied. For example, Save the Children US’s Annual Progress Reports indicated that there were funding delays limiting the launch of these activities. Regardless, limited integration of NACS training into the pre-service curricula at targeted universities may undermine NACS sustainability. 3 Bradshaw, M. (2011). Innovative teaching strategies in nursing and related health professions (5th ed., p. 538). Sudbury, Mass.: Jones and Bartlett 24 | P a g e Inclusion in the pre-service curriculum of universities would have meant that every graduate (health worker) would have the necessary knowledge and skills to implement NACS wherever assigned to work, resulting in less dependency on RHB in-service training. This is an area that will require renewed efforts for the inclusion of NACS in the curriculum. It will be difficult to sustain trained workers in the facilities, particularly given reports of frequent rotation to other facilities. Domain 3—Relevance There is sufficient evidence from the analysis of FGD data to conclude that the FBP activity provides satisfactory services that yield welcome and positive results for participating beneficiaries. All FGD groups reported that the products helped them to gain weight and improve their nutritional status, regain their strength, improve ART adherence and go back to work, enhance their appetite for other foods, and gave them hope of being alive. Based on the team’s analysis of stakeholder interview data, RHBs and facilities will likely continue the FBP activity given the positive outcomes if financial resources and nutrition supplements are available. Domain 4—Gender The FBP activity did not have a specific plan to address gender needs or issues of men or women enrolled, nor were the disaggregated data reported by health facilities analyzed to examine differences such as relapse, LTF rates, or death rates between men and women that may have illuminated issues that could have been addressed. RECOMMENDATIONS Domain 1: Effectiveness of Activity Implementation and Management Sustain and Scale-up: Based on success evidenced in this evaluation from our analysis of effective data and feedback from beneficiaries and key informants, USAID should continue to support the scale-up of activities and integration with other nutritional outreach efforts, while working with the GOE to explore other options for supplemental foods. The ES component lacks sufficient evidence of effectiveness to be scaled-up. Explore Alternate Foods: USAID and the GOE should collaborate on analyzing alternative, less expensive products, giving more attention to those which can be manufactured locally to sustain the activity beyond the life of the current activity. Plumpy Nut/Sup are expensive commodities that the government is unlikely to continue supplying. These products are currently dependent on the health system, yet local production is likely the responsibility of other government entities, bearing in mind that NACS support and referrals to and from the health sector for supplements will continue to be necessary. USAID should (a) provide transitional and technical 25 | P a g e assistance to explore the options for scaled-up production of fortified, specialty foods. This should be based on the costs of procurement, packaging, quality control, distribution and other cost factors, and (b) communicate with the GOE and other donors and organizations that provide technical and financial support to provide less expensive, supplementary foods. Improve M&E Information System and Introduce Systematic Operations Research: USAID and its IPs should work together to reinforce data collection and reporting to improve M&E systems writ large. In addition, there are a number of questions which arose from our evaluation research about effectiveness, efficiency, usage, referrals, compliance, and others, such as why clients drop out, who relapses, and the different needs, outputs and outcomes for men and women by age, geographic location, and religious practices. The answers to these questions might influence activity effectiveness, how gender issues might best be addressed, understanding the reluctance of universities to be involved, better local food substitutes, and the amount of other food intake necessary in order to rule out some confounding factors. Operations research is needed to answer these questions. Domain 2: Sustainability Work jointly to assure routine nutrition assessment and counseling: The FMoH and HAPCO should work closely with RHBs, USAID’s Food and Nutrition Technical Assistance (USAID/FANTA III), WFP and other stakeholders working in nutrition and HIV/AIDS, to ensure that all health facilities routinely carry out nutritional assessments and counseling. Reinforce use of government systems: USAID and its partners should reinforce use of government systems, such as PFSA for supply chain management, and public health facilities for service delivery, to increase the likelihood of sustaining services and enhancing government ownership. Reinforce HCWs capacity building: USAID should work with the MOH and universities to secure financial resource to enhance inclusion of NACS in the curriculum. Frequent rotation to other facilities should be addressed in order to sustain trained workers in the facilities. A motivational system should be put in place to increase acceptance of the workload due to the integration of NACS services as an integral part of routine activities. Domain 3: Relevance Include NACS in ART training: USAID and FMoH should assure the inclusion of NACS into the standard ART training, with appropriate indicators to ensure that every health worker trained on ART is also trained on NACS. Over the long-term, the FMoH, HAPCO, and USAID/FANTA III need to consider integrating NACS into the pre-service curriculum of health science colleges and medical schools. 26 | P a g e Future Directions Facilitate public – private partnerships to engage graduates of the Back-to-Work program, which is also in tandem with the poverty reduction strategy. Work closely with town level advisory committees to identify locally available resources, and reinforce engagement of graduates with all available ES activities. Explore the feasibility of the inclusion of nutrition supplementation coverage in the expansion of the GOE’s community-based insurance system, as it does with other drugs. 27 | P a g e ANNEX I: SCOPE OF WORK FROM TASK ORDER BACKGROUND The Food by Prescription (FBP) project provides technical assistance for the integration of Nutritional Assessment Counseling and Support (NACS) into the routine care and treatment services for PLHIV. The activity supports the provision of Ready to Use Supplementary Food (RUSF) and Ready to Use Therapeutic Food (RUTF) to moderate and severely malnourished adult PLHIV respectively, including pregnant and lactating mothers; and malnourished HIV- positive or exposed children. Nutrition Assessment and Counseling to PLHIV in care services is also an important component of the FBP Project. FBP is a five year project implemented in three phases. Phase one covers the first 3 years of the project and began in S e p t e m b e r 2009. The subsequent one year extensions were determined annually. Over the first phase, the activity targeted 96,000 beneficiaries to receive therapeutic and supplementary food in the two city administrations of Addis Ababa, and Dire Dawa, and the five Regional States of Oromia, Amhara, Harari, Tigray and Southern Nations, Nationalities, and Peoples Region (SNNP). In phase 1, the project also targeted reaching 400 health facilities that provide ART services with PEPFAR support. The amount of target beneficiaries for food support over a five year period was 219,630. The project is currently in phase 3, which will end in September 2014. CONTEXT Ethiopia has a population of 82.64 million and is the second most populous country in Africa, with 81.4 percent of the population in rural areas and a current growth rate of 2.4 percent. The estimated HIV prevalence rate of 7.3 percent determined in 2000 steadily declined to 1.5 percent by 2011 (EDHS 2011). Also, according to 2012 epidemiological projections carried out by the Ethiopian Health and Nutrition Research Institute and the Ethiopian FMOH, the adult HIV prevalence for 2013 is expected to be 1.3 percent. Needless to say, this still reflects a huge number of people living with HIV in need of care and treatment (an estimated total of 734,048 adults and children are living with HIV in Ethiopia). Further, women are more vulnerable to HIV infection and malnutrition as compared to their male counterparts due to biological, social, economic, and cultural factors. While there has been some progress in improving nutrition in Ethiopia over the past several decades, stunting, wasting and micronutrient deficiencies are still very prevalent across most of the country. Nutrition surveys have also documented high levels of acute malnutrition specifically in women, girls, and infants, as well as a complex number of gender related root causes. Under 5 stunting is above 40 percent and national prevalence of wasting is 10 percent, 28 | P a g e (EDHS 2011) with even higher rates in some regions. The prevalence of malnutrition is worse in rural areas where most of the population lives. The Government of Ethiopia (GOE) has invested significantly in nutrition planning over the past few years and has demonstrated a commitment to resolve nutritional problems though the development of a National Nutrition Strategy and its Implementation Plan (2008), strengthened coordination through the Nutrition Technical Working Group, and expansion of long-term programs such as the Productive Safety Net Program (PSNP). USAID/Ethiopia has been implementing significant food and nutrition activities to support HIV affected persons and orphans and vulnerable children (OVC) since 2003. The current array of activities includes Nutritional Assessment, Counseling and Support (NACS) for PMTCT clients, PLHIV (adults and children) and OVC, work in the area of livelihoods, as well as food security initiatives. FBP is a large element of ongoing food and nutrition support, largely implemented by Save the Children US, with a smaller portfolio implemented under the Urban HIV/AIDS Nutrition and the Food Security Program by the World Food Program (WFP). A “PEPFAR/Ethiopia Nutrition Programming Portfolio Review” was conducted in February 2010 which resulted in a number of recommendations for strengthening the effectiveness of nutrition services. At the same time, the food security and nutrition programming of USAID/Ethiopia has been expanding to respond to persistent under-nutrition trends in the general population, with non￾PEPFAR nutrition programs helping to strengthen nutrition services within the primary health care system and at community level. Previously, FBP extended limited support to introduce NACS into the tuberculosis (TB) program, with pilot activities in a few health facilities where there was a heavy case load of TB patients and high levels of malnutrition. However with declining resources, this support will not be scaled up, unless it gains financial support from USAID’s TB program. Feed the Future (FTF) agricultural programs are also supporting increased food production and marketing, as well as building resilience of vulnerable households. FBP supports linkage of food beneficiaries to economic strengthening (ES) activities (FTF, PEPFAR, or with other funding) as a long term measure for food security. FOOD BY PRESCRIPTION The FBP activity, implemented by Save the Children US, ensures improved clinical nutrition and functional outcomes for HIV+ individuals, pregnant and postpartum women, and OVC in Ethiopia. The activity supports five strategic areas:  Commodity sourcing, procurement and distribution of fortified supplementary food;  Capacity building of key stakeholders and health facility staff and communities to deliver FBP activities;  Supporting adherence and behavioral change through information, education and communication; 29 | P a g e  Increasing coordination of HIV and nutrition interventions and policy issues with key stakeholders; and  Monitoring and evaluation systems for support of FBP programming. The activity also aims to develop concrete, functional linkages to a range of medical, social and economic opportunities for malnourished PLHIVs and OVC. THEORY OF CHANGE The development hypothesis is that improved nutritional, clinical and functional outcomes of malnourished HIV positive adults and OVC are achieved through strengthening NACS, as well as creating effective linkages to community resources and economic strengthening initiatives. To mark progress toward this hypothesis, the FBP project has identified three intermediate results and strategies to achieve them: 1. Increase the provision and access of therapeutic and supplementary nutritional products to people living with HIV/AIDS (PLHIV) by: a. Strengthening commodity sourcing, procurement and distribution of food products and b. Integrating RUTF and RUSF into the national drugs and pharmaceutical commodities supply system. 2. Strengthen capacity to implement FBP including: c. The capacity of key stakeholders to deliver FBP services to clients. d. The capacity of health facility staff and communities to deliver FBP activities. Increased capacity will support adherence and behavioral change through information, education and communication, and increase coordination on HIV, nutrition interventions, and policy issues with key stakeholders. 3. Strengthen the collection, analysis and use of data to inform policy and planning for the provision of nutritional care in the context of HIV. This is accomplished by strengthening M&E systems for support of FBP programming, establishing an activity monitoring system, and conducting baseline surveys, as well as operational studies. Those who are moderately or severely malnourished receive a “prescription” for supplementary or therapeutic foods. The prescription mechanism serves to draw beneficiaries into health facilities, where they receive additional health care, nutritional assessment, and counseling and support. Patients on food support are expected to improve their BMI progressively. Failure to respond to food support as expected may be the first indication that a patient is developing an opportunistic infection or is not responding to treatment, which calls for clinical investigation. USAID/ FBP supports health facilities in Ethiopia to integrate NACS into comprehensive HIV care and treatment services with the aim of improving adherence to, and uptake of, antiretroviral therapy thereby improving response to treatment, and ultimately improving the 30 | P a g e nutritional and health status of people living with HIV/AIDS (PLHIV). The project also seeks to link clients graduating from the supplementary and therapeutic feeding program and their households to economic strengthening activities. FBP IN ETHIOPIA During the first three years, FBP focused on building capacity at the health facility level to implement NACS. This involved training health facility clinical staff, case managers, peer educators and volunteers in ART and PMTCT to deliver NACS as part of standard HIV services. Pharmacists and logistics officers were also trained. These groups of care providers, mainly based at the facility level, also enhance and create a linkage between the facility and community-based health extension workers (HEWs). The HEWs supported under USAID’s broader health projects provide the community link through which follow-up of patients who have missed appointments in HIV and nutrition services can be traced. By September 2012, the project had trained 7,000 health care providers, including doctors, nurses, midwives, case managers, data clerks, pharmacists, policy makers, local leaders, and community volunteers in NACS. By this same time, the project had also reached a total of 96,323 clients with therapeutic and supplementary food. Of these, 95,637 beneficiaries were PLHIV, including PMTCT clients and OVC. 686 were HIV negative TB patients in a pilot study to integrate NACS into the TB program. In its third year FBP scaled up services to support 400 health facilities and began to focus on strengthening the quality of NACS in HIV care and treatment services through supportive supervision linked to a strong monitoring system. The project staff also coach and mentor site￾level staff. FBP also introduced structured continuous quality improvement (CQI) activities in selected health facilities, with a plan to scale this up in all supported sites during the two option years if exercised. The CQI principles and initiatives implemented for NACS are also applicable to other health care services at supported facilities, and every opportunity is being used to involve the facility management and staff in CQI activities for facility-wide quality assurance practices. One of the project goals is to link food support beneficiaries to community level economic strengthening activities as a long term measure to prevent future malnutrition. Through partnerships with local entrepreneurs, NGOs, and other partners, some beneficiaries are able to return to work. This may occur either via ES activities funded under PEPFAR or FTF, or with local entrepreneurs and community-based organizations. FBP has, in some cases, negotiated re￾employment of clients graduating from food support that previously lost their jobs due to poor health, and has supported skills training for those PLHIV who were physically able to work but had no skills to market. FBP child support includes adolescent targeted BCC (Behavior Change Communication) through multi- media, adolescent ES and supported pediatric clinics, as well as family centered approaches in FBP supported facilities. After phase 1 of the project, it was determined that though the project had focused on scaling up of services, more technical assistance was required to strengthen GOE institutions and 31 | P a g e prepare them for eventual ownership and oversight of the NACS in HIV services. USAID/Ethiopia then exercised the first and second available option years and set new targets for the project, while narrowing the project scope to focus on quality of service, as well as on strengthening GOE readiness to oversee NACS services. Pre-service education was phased out as more partners came on board to support pre-service education. Provision of safe water was also dropped to avoid duplication with the Preventive Care Project implemented by World Vision International. The ES component was also scaled down and focused on a Back-to-Work Initiative that seeks employment for clients graduating from food support. Participants in this activity are individuals of age who were previously employed but lost their job due to sickness, or who had never been employed but were willing to obtain new skills for employability. FBP then works with local private entrepreneurs or GOE institutions to provide skills for these individuals and assists them in finding employment or supports them to go back to school if they have the potential to enter formal education. Additionally, given funding constraints, the project has ended food support to HIV negative TB patients at the end of year four (the first option year. The targets for therapeutic and supplementary food were 65,000 and 58,630 in the first and second option years respectively, subject to availability of food commodities. PURPOSE OF THE EVALUATION The purpose of this Performance Evaluation is to quantify and analyze the results that the project has achieved at both the levels of the beneficiaries (namely, HIV positive persons that receive RUTF and/or RUSF) and the health service delivery system. The evaluation will examine results in terms of integrating Nutrition Assessment Counseling and Support into HIV care and treatment services, and the systems put in place to integrate the quantification, forecasting, management and distribution of RUTF and RUSF commodities into the country’s Logistics Management Information System at the national and regional levels. The evaluation findings will be used to:  Assess the appropriateness of the activity design and implementation in reaching the stated objectives.  Make recommendations to inform future related HIV and nutrition programming in Ethiopia by USAID/Ethiopia, the Government of Ethiopia, and other relevant stakeholders. EVALUATION QUESTIONS Our domains of interest include assessment of: 1. Effectiveness of operational activity implementation and management. The review will assess the results and the progress of the activity in terms of the effectiveness (achieved outputs versus planned outputs) and the efficiency (output results achieved against inputs and budgets used) of implementation. a. A recent study determined that consistent collection of activity monitoring data has been suboptimal. What is the number and percent of clinically malnourished HIV￾positive clients who received food supplements AND graduated from the activity? Also, of those that graduated from the activity, what is the number that relapsed? 32 | P a g e 2. Handover to and sustainability by the Regional Health Bureaus (RHBs) and health facilities including areas that might need continued limited external technical assistance. a. To what extent have NACS indicators been integrated in FBP supported government and health facility reporting formats and work streams? b. What is the capacity of supported Regional Health Bureaus and health facilities to plan for and provide NACS services in the absence of FBP support? c. To what extent has NACS been integrated into in-service training activities for nurses, midwives and medical doctors in targeted universities? 3. Relevance of activity objectives to stakeholders (government, health facilities, and beneficiaries). a. To what extent are the objectives of FBP consistent with the needs of the activity’s beneficiaries/target groups? b. To what extent do stakeholders (RHBs, health facilities), including beneficiaries, buy-in to and own the FBP goal, objectives and implementation methods? c. To what extent are RUTF and/or RUSF palatable and useful to beneficiaries? d. To what extent do FBP interventions address gender issues that expose women to HIV/AIDS or malnutrition? EXISTING INFORMATION AND DOCUMENTATION The Performance Evaluation is seen as an open, transparent learning process for all stakeholders/beneficiaries involved in the activity. It will allow for achieving a common understanding of the institutional and working environment, the current status – achievements, pitfalls and constraints – as well as opportunities of the activity. USAID Ethiopia will gather and provide all crucial reference documents, such as:  Program documents  Work plans  Semi-annual and annual progress reports  Financial reports  Documents produced over the course of activity implementation by USAID, Save the Children US, the GOE and other entities.  Raw data as needed to answer the proposed questions (including facility level data, reporting data)  PEPFAR/Ethiopia Nutrition Programming Portfolio Review  Automated Directives System (ADS) guidance as appropriate  Gender analysis documents USAID Ethiopia will generate a stakeholder list of all the relevant in-country stakeholders including their physical address, email contacts, and telephone contacts. USAID Ethiopia will provide the evaluation team with relevant resource and reference documents including:  USAID Evaluation Policy  How to Prepare Evaluation Reports 33 | P a g e  Evaluation Report Template  USAID TIPS – Key Documents EVALUATION DESIGN AND METHODOLOGY Methodology A mixed methods approach including multi-level data collection (RHBs, facilities, beneficiaries) and both quantitative and qualitative methods is recommended. This will include review of available secondary data (quantitative - project and facility records) and gathering of primary data via focus groups and interviews (qualitative). Focus groups should target beneficiaries and gather information about the impact of the activity, specifically in the area of economic strengthening. Interviews will be with relevant project and RHB staff as well as others as needed. Gender disaggregation of any data collected and analyzed should take place to determine if impact due to the intervention is different by gender. Further, any gender sensitive indicators should be collected and analyzed when applicable to the evaluation objectives.2 Analytical Steps 1. Review key documents and data related to project 2. Conduct interviews with key staff in relevant offices 3. Collect primary data via focus groups from beneficiaries 4. Analyze data (primary and secondary) 5. Write report and summarize Data Sources Primary data should be collected from beneficiaries and others (project staff, RHB staff) as needed to answer the evaluation questions. Secondary data will be provided by:  FMoH  Federal and Regional HIV/AIDs Program Coordinating Offices (HAPCO)  Regional Health Bureaus (RHBs)  Individuals from partner agencies who contribute to or collaborating with FBP  Hospital and clinic staff EVALUATION TEAM SKILLS AND QUALIFICATIONS A strong evaluation team will be comprised of two consultants (1 international, 1 local) with experience in the field of nutrition and HIV/AIDS and evaluation. The evaluation team should include the support of a local logistics coordinator who should be able to manage logistical arrangements for the evaluation team members and evaluation implementation, and may 34 | P a g e include local research assistants for data collection. However, Offerors will not be required to select a logistics coordinator or research assistants by the time of proposal submission. The evaluation may also have virtual support from technical experts from USAID/Washington for aspects such as review of data analysis and draft reports. Profile of Evaluation Team 1. Team Lead The team lead will have the overall responsibility for the expected results of the performance evaluation. S/he will be an international consultant with more than 10 years of experience, including some work in Africa. The team lead will be responsible for team performance and for ensuring the timeliness and quality of deliverables. Strong candidates for these positions will have experience with USAID programs and PEPFAR. Strong team lead candidates will have led at least two external performance evaluations. Strong writing, evaluation methods, and analytical skills are required of both international staff. The consultant will hold conference calls with the other team members and USAID/Ethiopia representatives before and after the visit to Ethiopia in order to develop the evaluation methodology and take the lead in developing the evaluation report. The team lead is expected to present preliminary findings of the evaluation to USAID/Ethiopia and FBP staff prior to departure from the country. The team lead will be supported by a local consultant whose skills must complement the evaluation. Technical knowledge in the areas of HIV/AIDS, nutrition, and evaluation methods will be necessary for a comprehensive team. The international team lead should have a range of skills including the following:  Significant experience in nutrition and HIV/AIDS programming, preferably in Africa  Performance evaluation design, implementation, and analysis; preferably as principal investigator (PI) or co-investigator with evaluation experience preferably in Africa, including peer-reviewed publications validating experience  Development and use of qualitative and quantitative evaluation methods  Experience with the issues affecting genders differently in health programs and specifically in HIV/AIDS programming  Cultural sensitivity and ability to work in a cross-cultural team  Strong English presentation and writing skill 2. Local Consultant (1) The local consultant should possess the following skills:  Experience in nutrition and/or HIV/AIDS programming  Performance evaluation design and implementation  Quantitative and qualitative data collection and analysis (developing evaluation methodologies/tools and performing data collection, management, and analysis)  Experience with the issues affecting genders differently in health programs and 35 | P a g e specifically in HIV/AIDS programming in Ethiopia  Understanding of the local health system and structures (FMoH, RHBs, etc.)  Fluency in written and spoken Amharic  Strong English language and writing skills ESTIMATED LEVEL OF EFFORT (LOE) An agreed tentative schedule will be drafted and the working team(s) will work with designated experts from FBP. The actual task distribution will be finalized once the team is established and according to specific expertise of the consultants. A six-day work week will be approved when the consultants are working in-country. Weekend travel may be necessary. Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort and proposed timing for each task. LOGISTICS USAID/Ethiopia will provide overall direction to the team and will provide key documents and background materials. USAID/Ethiopia is unable to provide workspace for the evaluation team at the Mission except for initial in-briefing, debriefing and meetings with USAID staff. External meeting space and printing and photocopying services may be provided by the partner or through local hotels and printers. USAID will schedule the internal meetings. Save the Children US (the implementing partner) will be responsible for assisting in scheduling site visits (though not in site selection or sampling) and the provision of internal activity documents. The consulting firm will provide logistical arrangements such as flight reservations and tickets (business class is authorized), country cable clearance, in-country travel funds, and airport pick-up, lodging and daily transportation, as appropriate. Any staff outside the F B P team, any USAID staff who participate in parts of the evaluation, or those listed in this evaluation statement of work that wish to participate in this evaluation (develop methodology, perform data collection and/or analysis, or contribute or review and edit the report must be approved by USAID/Ethiopia Program Office, USAID HAPN Office and technical team, as well as the evaluation team lead. A clear scope of work and deliverables are expected in addition to this approval for these additional team members. 36 | P a g e ENVIRONMENTAL IMPACT FBP itself has received a categorical exclusion as of 08/11/2012. Though this is the status of FBP with regard to environmental impact, the final evaluation falls in the following categories meriting a “Categorical Exclusion” and not requiring an Environmental Examination under 22 CFR 216, § 216.3, Title 22 – Foreign Relations. CHAPTER II – AGENCY FOR INTERNATIONAL DEVELOPMENT: (i) The action does not have an effect on the natural or physical environment; (ii) The action involves nutrition, health care or population and family planning services and does not include activities directly affecting the environment (such as construction of facilities, water supply systems, waste water treatment, etc.); (iii) The action is intended to develop the capability of recipient countries to engage in development planning and will not result in activities directly affecting the environment (such as construction of facilities, etc.) As previously stated, this evaluation will include activities that are categorically excluded according to 22 CFR 216. RELATIONSHIPS AND RESPONSIBILITIES The Evaluation team shall closely work with the USAID/Ethiopia FBP team. The USAID/Ethiopia FBP team shall meet with the investigators prior to the start of the evaluation process and review/approve all deliverables referenced in Section C [of the Request for Task Order Proposal]. 37 | P a g e ANNEX II: SAVE THE CHILDREN US M&E PLAN (UNDER SEPARATE COVER) 38 | P a g e ANNEX III: LIST OF REFERENCES 1. USAID/Food by Prescription Nutritional Assessment Baseline Report (NB: Date not indicated on the report.) 2. USAID/Food By Prescriptions Contract Modifications #1 to #14. 3. USAID/Food By Prescriptions Annual Reports Years 1 to 5. 4. Ethiopian National Nutrition Strategy, Federal Ministry of Health, January 2008. 5. Ethiopian National Nutrition and HIV/AIDS Implementation Reference Manual, Federal Ministry of Health, September 2008l. 6. Ethiopian National Guidelines for HIV/AIDS and Nutrition, Revised September 2008. 7. National Nutrition Programme, Government of the Federal Democratic Republic of Ethiopia, June 2013 – June 2015. 8. Measuring the Impact and Cost-Effectiveness of Prescribed Food on Recovery from Malnutrition and HIV Disease Progression Among HIV+ Adult Clients in Ethiopia, Tufts University and Save the Children US in Ethiopia for USAID/FBP, October 2012. 9. USAID/ Food by Prescription Performance Management Plans (PMPs). 10. Save the Children US Annual Progress Reports, Years 1 -4. 11. Global Nutrition Report 2014, International Food Policy Research Institute, Washington D.C. 12. Terminal Internal Evaluation Reports of USAID/ Food By Prescriptions, Save the Children US International. 13. Nutrition Exchange, Emergency Nutrition Network, July 2014 – Issue 4. 14. Office Of Inspector General, Audit Of USAID/Ethiopia’s HIV Treatment Activities, Audit Report No. 4-663-14-006-P, May 23, 2014. 15. Bradshaw, M. (2011). Innovative teaching strategies in nursing and related health professions (5th ed., p. 538). Sudbury, Mass.: Jones and Bartlett. 39 | P a g e ANNEX IV: EVALUATION PLAN & ANALYSIS INTRODUCTION International Business and Technical Consultants, Inc. (IBTCI) submits this evaluation design and methodology and work plan to USAID/Ethiopia as a deliverable under AID-663-TO-15-00001, following the Final Performance Evaluation Statement of Work (SOW) (See Annex A). PURPOSE AND TARGET AUDIENCE Purposes of this final performance evaluation of the Food by Prescription (FBP) project are to: 1. Quantify the results that the project has achieved at the level of individual beneficiaries (HIV-positive persons, Orphans and Vulnerable Children who receive Ready to Use Therapeutic Food (RUTF) and/or Ready to Use Supplementary Food (RUSF)), including how well they are generating a sustained living income. Qualitative methods will be used to gain an understanding of how the activity was developed and implemented from national, regional and local health stakeholders, and beneficiaries, as well as from other stakeholders. A target of 219,630 beneficiaries reached with food support was set over a five-year period, across three phases. 2. Quantify and qualify, using mixed methods, to determine the extent of capacity building efforts the activity achieved at the health service delivery level (facilities including hospitals and health centers from both public and NGO sectors), as well as with the associated staff to ensure sustainable RUFT and RUSF services through the integration of Nutritional Assessment Counseling and Support (NACS) into the routine care and treatment services for People Living with HIV (PLHIV). A target of supporting 500 health facilities across two city administrations (Addis Ababa, Dire Dawa) and five Regional States (Amhara, Harari, Oromia, Southern Nations, Nationalities and Peoples Region (SNNPR)) and Tigray was set over a five-year period. 3. Measure the results the project has achieved at the level of the systems and supply chains that quantify, forecast, manage, procure and distribute RUTF and/or RUSF commodities and integration into the Logistics Management Information System at the national and regional levels. Furthermore, the FBP program will be evaluated for its sensitivity and inclusiveness to help mitigate barriers to help women access available RUTF and RUSF services. 40 | P a g e This performance evaluation will inform USAID/Ethiopia about best practice options for program design and implementation. It will also point out areas that require greater attention in future programming. More broadly, the evaluation will speak to lessons learned about the FBP model that may have implications for other important stakeholders in the government and civil society as the ownership of the program shifts to national actors more fully. Finally, we expect that the evaluation results will elucidate organizational learning for all stakeholders. The target audiences for this evaluation include: • USAID/Ethiopia: to inform the design of future HIV/AIDS Care and Support and Nutrition programs. • Ethiopia Federal Ministry of Health (FMoH): to demonstrate the relevance and effectiveness of the FBP model in the national response by the government. • Save the Children US: to inform on the overall performance of the FBP project and the achievement of the strategic objectives and overall goal of the project. The final evaluation report will be disseminated widely with relevant stakeholders and project beneficiaries, as well as submitted to the Development Exchange Clearing House (DEC). BACKGROUND The FBP project provides technical assistance for the integration of NACS into the routine care and treatment services for PLHIV. The program supports the provision of Ready to Use Supplementary Food (RUSF) and Ready to Use Therapeutic Food (RUTF) to moderately and severely malnourished adult PLHIV, including pregnant and lactating mothers; and malnourished HIV positive or exposed children. In Ethiopia this FBP program has been implemented by Save the Children (via the US chapter) and supports five strategic areas:  Commodity sourcing, procurement and distribution of RUSF and RUTF.  Building capacity of key stakeholders and health facility staff and communities to deliver FBP programs.  Supporting adherence and behavioral change through information, education and communication.  Increased coordination of HIV and nutrition interventions and policy issues with key stakeholders.  Monitoring and evaluation systems for support of FBP programming. The project has been implemented over three phases, beginning in September 2009. Phase two began in September 2012 and ended in September 2013. The final phase was implemented until September 2014, with a no-cost extension given for activities lasting until December 2014. 41 | P a g e The project aimed to reach a total of 219,630 beneficiaries over the five-year period and to provide technical support in the form of NACS to at least 500 health facilities within either two city administrations or five regional states targeting populations in urban and peri-urban areas. The development hypothesis is that improved nutritional, clinical and functional outcomes of malnourished HIV positive adults and OVC are achieved through strengthening NACS, as well as creating effective linkages to community resources and economic strengthening initiatives and to link food support beneficiaries to community level economic strengthening activities as a long term measure to prevent future malnutrition. To mark progress toward this hypothesis, the FBP project has identified three intermediate results and strategies to achieve them: Intermediate Result 1: Increase the provision and access of therapeutic and supplementary nutritional products to PLHIV by:  Strengthening commodity sourcing, procurement and distribution of food products.  Integration of RUTF and RUSF into the national drugs and pharmaceutical commodities supply system. Intermediate Result 2: Strengthen capacity to implement FBP including:  The capacity of key stakeholders to deliver FBP services to clients.  The capacity of health facility staff and communities to deliver FBP programs. Increased capacity will support adherence and behavioral change through information, education and communication and increase coordination on HIV, nutrition interventions, and policy issues with key stakeholders. Intermediate Result 3: Strengthen the collection, analysis and use of data to inform policy and planning for provision of nutritional care in the context of HIV.  Strengthened monitoring and evaluation systems for support of FBP programming, establishing a program monitoring system, and conducting baseline surveys as well as operational studies. Since inception, the FBP project has undergone twelve modifications with USAID, including the approval of the final two one-year phases in 2012 (Modification 5) and 2013 (Modification 8). The majority of these modifications revised the number of targets (beneficiaries and facilities) and budget realignments (Modification 2, 3, 5, 6, 9, and 12). 42 | P a g e EVALUATION QUESTIONS The following key questions will guide the final performance evaluation: Analytical Domain 1: Effectiveness of Program Implementation and Management Q. What have been the achieved outputs versus planned outputs? Q. How efficiently have the output results been achieved against inputs and budgets used? Q. What is the number and percent of clinically malnourished HIV-positive clients who received food supplements and graduated from the program? Q. Of those that graduated from the program, what is the number that relapsed? Analytical Domain 2: Sustainability Q. To what extent have NACS indicators been integrated within FBP-supported government and health facility reporting formats and work streams? Q. What is the capacity of supported Regional Health Bureaus and health facilities to plan for and provide NACS services in the absence of FBP support? Q. To what extent has NACS been integrated into in-service training programs for nurses, midwives and medical doctors in targeted universities? Analytical Domain 3: Relevance Q. To what extent are the objectives of FBP consistent with the needs of the program’s beneficiaries/target groups? Q. To what extent do stakeholders (RHBs, health facilities), including beneficiaries, buy-in to and own the goal, objectives and FBP implementation methods? Q. To what extent are RUTF and/or RUSF palatable and useful to beneficiaries? Analytical Domain 4: Gender Q. To what extent do FBP interventions address gender issues that expose women to HIV/AIDS or malnutrition? EVALUATION DESIGN This evaluation will use mixed qualitative methods and a multi-level and multi-site participatory approach that will allow us to pull together data and findings from different geographic and health delivery sites as well as different stakeholder levels of the project. 43 | P a g e EVALUATION TEAM A team of two experts, one international and one Ethiopian, four Research Assistants (RAs) and a logistics specialist, supported by senior technical experts, will conduct the Performance Evaluation of FBP Project, as follows: Mr. Gordon Mortimore, Team Leader, will be responsible for management of the FBP performance evaluation, and site selection and sample size determination. He will be responsible for key informant interviews (KIIs) mainly with national-level stakeholders, including government officials, the USAID and FBP management team, and government officials in the regional health bureaus of Addis Ababa, Amhara and Tigray regions and Save the Children sub-offices in the respective regions. He will be engaged in the assessment of selected health facilities supply chain systems and procedures and coordinate the data collection in Amhara and Tigray regions. He will guide the analysis and synthesis of qualitative data. Dr. Mesfin Beyero Hirbaye, Health and Nutrition Expert (local consultant), will collaborate with the Team Leader and assist in site selection and sample size determination. He will be responsible for KIIs with government officials in the regional health bureaus of the SNNPR and Oromia regions and Save the Children sub-offices in the respective regions. He will also be engaged in the KIIs with health workers, assessment of selected health facilities supply chain systems and procedures, and will coordinate data collection in the SNNPR and Oromia regions including site and sample selection for FGDs. He will assist the team leader in the analysis and synthesis of the data. Ms. Hiwot will coordinate logistics planning, set up the schedule for the team to conduct KIIs with national stakeholders, interviews with Regional Save the Children sub-offices and Health Bureaus, Zonal Health Departments and FGDs with project beneficiaries. She will also organize the schedule for the FBP performance evaluation plan, including scheduling transport for each sub-team for data collection in the five regions. Mr. Andenet Haile, Ms. Eyerusalem Girma, Ms. Meskerem Fisseha and Mr. Mesfin Tesfay will serve as Research Assistants. They will assist in qualitative data collection, in particular conducting FGDs with beneficiaries and KIIs with health workers in selected sites. They will assist with data transcription, data entry and analysis. Assisting both in Addis Ababa and from IBTCI HO, Steve Hansch will provide technical guidance on nutrition, health, food security, supply chain and statistical analysis, including pattern and regression analysis with statistical software. There will be two teams: Team one includes Mr. Gordon, Mr. Andenet and Ms. Meskerem, and Team two are Dr. Mesfin, Mr. Mesfin and Ms. Eyerusalem. Both teams will conduct KIIs with the regional health bureau and health workers, and FGDs with beneficiaries in Addis Ababa to standardize assessment tools and procedures before branching to their respective regions. 44 | P a g e METHODOLOGY Mixed Methods: The methods approach will allow for an in-depth understanding of key issues, enhance the quality and credibility of findings and conclusions through convergence and overlapping of different data sources and methods of data collection. The triangulation of evidence from different sources will allow the team to cross-check key findings more reliably. The data collection considers meaningful representation of the population size and cultural differences between and within the regional states. Data will be collected from one city administration, Addis Ababa, and the four big Regional States of Amhara, Oromia, SNNPR and Tigray where the therapeutic and supplementary food intervention is being implemented. SAMPLING Selection Strategy The evaluation sites are selected purposively to capture the range of variation seen across Ethiopia within this program. These include the four biggest regions in terms of population size and the Addis Ababa City Administration which is the federal capital. Within the regional states, the regional capital and zonal towns are selected to encompass possible cultural differences within the regions. The sampling strategy will be finalized in consultation with USAID/Ethiopia. Selection of Hospitals and Health Centers The evaluation team selected three of the regional capital cities and one major city within Oromia as the regional government currently resides within Addis Ababa (Table 1). In addition to the regional capital cities, 10 zonal towns have been selected from the four regions. A hospital and one health center will be assessed in each of the regional capitals and zonal towns. The number of beneficiaries in these health facilities is expected to be sufficient for the team to be able to conduct FGDs at any point in time. At least four FGDs will be conducted in the regional capitals for the four category of beneficiaries i.e. PMTCT, Men with HIV, Women with HIV and OVCs. In the zonal towns, however, two FGDs will be conducted for the two categories of beneficiaries. The category will be decided depending on the number of clients available while maintaining the proportionality within the regions. The teams will conduct KIIs with Regional and Zonal Health Department officials, Save the Children sub-offices, and the PMTCT/ART health worker within the health facilities. Selection of Respondents We will draw a purposeful sample among the range of the FBP national, regional and zonal health officials and Save the Children sub-offices and health workers responsible for the PMTCT or ART clinics in the respective health facilities. FGDs with the beneficiaries receiving the services from each of the above facilities will be conducted, as described under data sources (Table 1). 45 | P a g e Table 1: KIIs and FGDs: National, Regional, Zonal and Beneficiary Levels Multi-levels Qualitative Methods National Level KIIs FGDs USAID/Ethiopia staff 2 0 Federal Ministry of Health 3 0 Federal HIV/AIDS Program Coordinating Offices (HAPCO) staff 1 0 Save the Children FBP management 2 0 Save the Children implementing staff 3 0 World Food Programme 1 0 Other HIV implementing partners 3 Pharmaceutical Fund and Supply Agency (PFSA) staff 2 0 Private sector representatives (producers of commodities) 2 0 Staff of Principal donors/stakeholders,(e.g. Global Fund, UNAIDS, UN Women, UNICEF, WHO, UNFPA 8 Staff from CDC, PEPFAR 2 Staff other stakeholders (NGOs/FBOs) 2 0 Sub Total 31 0 Regional, Zonal and Facility levels Qualitative Methods KIIs FGDs Regional Health Bureau staff 15 0 Regional HIV/AIDS Program Coordinating Offices (HAPCO) staff 5 Save the Children US Staff regional staff 10 Hospital/Health Center Medical Directors and ART/PMTCT focal person (2 per hospital or health center proportion to the number of hospital in the region) 20 Regional SC staff 5 0 Sub Total 55 0 Project Beneficiaries Qualitative Methods KIIs FGDs Women at PMCTC, health service sites 0 8 groups/8 participants Women living with HIV (age 15+) selected at service 0 8 groups/8 participants Men living with HIV (age 15+) selected at service 0 8 groups/8 participants OVC beneficiaries 0 8 groups/8 participants Sub Total 0 32 groups/256 participants 46 | P a g e Data Sources Key data sources for the evaluation will be: Desk Review Including but not limited to:  FBP program documents, such as Annual Performance Reports, Quarterly Progress Reports, work plans, FBP contract documents, M&E plans, instruments and tools, communication materials, and Presidential Emergency Plan for AIDS Relief (PEPFAR) annual reports.  FBP studies including baseline and midline reports, and other studies carried out by Save the Children and its sub-contractors.  National reports such as the FMoH and the national HIV/AIDS Program Coordination Office (HAPCO), and relevant literature, and studies such as Demographic and Health Survey. Secondary Data Sources The secondary data analysis entails review of PMP and project data, and facility records when available in order to see the changes over the life of the project and associated factors related to the change to inform future project direction. Service data and records of patient care and achievement will be collected first-hand at each health facility visited, and compared against national records for the same population. Key Informant Interviews will be conducted at multiple levels (national, regional, and zonal and facility) to include:  USAID/Ethiopia’s Office of Health, AIDS, Population and Nutrition (HAPN)  Save the Children FBP management team and program personnel  Ethiopian government officials from the FMoH and Regional Health Bureaus  Save the Children regional sub-offices  Health workers Focus Group Discussions will be conducted with four categories of beneficiaries under HIV/AIDS Care and Treatment programs, that is  Women at PMCTC, health service sites  Women living with HIV (age 15+) selected at service  Men living with HIV (age 15+) selected at service  OVC beneficiaries 47 | P a g e DATA COLLECTION INSTRUMENTS Key Informant Interview Questionnaire: The KII instrument examines opinions and perceptions of national, regional, zonal and facility-based stakeholders about the degree that the project achieved the planned objectives and results in nutritional rehabilitation of moderately and severely acutely malnourished PLHIV. It also examines the effectiveness of the FBP Project design, implementation and management, and the degree to which clinical nutritional assessment and counselling services are integrated into public health facilities. The questionnaire assesses how FBP influenced strategy, programming, and policy at different governmental levels, the project’s contributions to capacity building in nutrition counselling and therapeutic feeding at national, regional and district levels, and the potential for continuity of these approaches after the project is terminated. Focus Group Discussion Guides: The team’s prepared guide for the facilitation, conduct and flow of the Focus Group consists of a set of questions for each of the four different categories of individuals. The FGD guide explores the degree to which the FBP project addressed the needs of men, women including PMTCT clients, and OVC beneficiaries and perceived changes in service delivery as result of FBP activities. This includes their opinions about the relevance and effectiveness of therapeutic and supplementary food products, effectiveness of nutrition counselling and possible linkages to economic strengthening opportunities. Data Quality Assurance: Following the in-briefing with USAID/Ethiopia and preliminary approval of draft data collection instruments, the evaluation team will finalize data collection instruments, conduct training for all team members and RAs in the use of instruments, and measures for transcribing and summarizing KII and FGD. The two teams will pre-test all instruments at the FBP site in Addis Ababa and the two experts will jointly conduct the KIIs at the Addis Ababa regional health bureau, which will be finalized thereafter for fieldwork in the regions. To assure the quality of data collection, the two RAs within each team will jointly conduct the FGDs, one will be a facilitator while the other write down notes. The team’s RAs will also conduct interviews with selected health workers. The two experts will conduct KIIs with government officials, Save the Children sub-offices, and interviews with health workers and facility assessments. Each team will meet daily after data collection in the regions to assess progress and challenges. The evaluation team will design a data summary matrix in Excel to be used by all team members at the end of each workday, to facilitate uniformity of data reporting. ETHICAL CONSIDERATIONS The Evaluation Team will utilize an informed consent form for all KIIs and FGDs and all interviews will be carried out on a voluntary basis. Interviewees will be given the option to opt-out of particular questions or the whole interview, if at any time they believe a response would contain sensitive information. The information provided as part of these interviews and discussions will not be linked to any specific person in the Final Report and all information provided will be kept confidential and used for planning purposes only. Only general identifying information 48 | P a g e (organization, geographical unit, gender, and age if reported voluntarily) will be utilized. Any information that could be directly linked to an individual will not be used. Only members of the Evaluation Team and the RAs will have access to the transcripts and raw data. The FGDs will be conducted in a health facility which is an environment the beneficiaries are used to and avoid possible stigmatization. Each Evaluation Team member and RA will sign a non-disclosure form (see Annex IV). The Final Report will be a synthesis of the Team’s analysis drawn from interviews from numerous respondents. Any included quotes to highlight particular issues will not include names. LIMITATIONS There are clear advantages to mixed qualitative methods design that will give the room to triangulate the generated data and in-depth examination of the quality of services provided. At the same time, qualitative methods have their own limitations that will have to be mitigated, such as:  Some stakeholders may not be available for consultations during field visits. Response: In the event that some stakeholders are unavailable the team will firstly communicate through email and secondly, to find alternates.  Sufficient numbers of beneficiaries might not be available at one point in time for conducting FGD given the tight schedule we have for the field visit. Response: FGDs at health facilities are primarily planned for morning sessions when the majority of beneficiaries are present. Forward planning and coordination with Save the Children US staff at national and regional levels will also mitigate this limitation.  Although this evaluation considers the effectiveness and efficacy of the FBP model, the types of evidence being collected will not provide a robust insight into the physiological outcomes of the intervention. Response: From secondary data it may be possible to draw some conclusions.  There is a bias in the site selection due to the necessity to reach sites that are easier to reach and have a sufficient caseload to conduct FGDs. Response: Site selection has included facilities that have been graded as both well-performing and those that have underperformed. Sites have also included a broad geographic range and cultural differences.  The ability to account for and interpret drop-out patterns, particularly those due to death, will be weak. The team will be interviewing survivors, which introduces a distinct bias. 49 | P a g e Response: Triangulation from FGDs and interviews with community health workers will be able to assist in interpretation of dropout rates and the reasons why drop outs have occurred.  Logistically, the evaluation team will be hard pressed to reach every target location in case of any contingencies (vehicle breakdown, facility closures, etc.) given the tight timeframe. Response: Flexibility and close coordination between the teams and the logistics coordinator. DATA ANALYSIS FBP is in its fifth and final year of implementation, and has generated a considerable amount of data over the past four years. The following methods of data analysis and synthesis will be employed: Descriptive analysis will be used to understand the contexts in which the FBP project has evolved, and to describe its various types of interventions and other characteristics. Results – focused progress analysis: Quantitative data from secondary sources will be used to confirm and identify associations and correlate factors of success and limitations to achieving results. Trend analysis of existing quantitative data on key performance outcome and output indicators will also be analyzed. Content and theme analysis will constitute the core of qualitative data analysis. Documents and qualitative data emerging from the KIIs and FGDs will be analyzed by the evaluation team to identify common trends, themes and patterns for each evaluation criteria (relevance, effectiveness, and sustainability). Content analysis will also highlight diverging views and opposite trends. Comparative analysis will be used to examine findings across different regions, themes or other criteria. It will also be used to identify best practices, innovative approaches, and lessons learned, as well as challenges hindering FBP project progress. This type of analysis will be used to examine information and data from stakeholder KIIs, document review and literature review. Analysis of gender implications FBP approach(es): The evaluation will assess the extent that the FBP project addressed and met both women's and men's needs. The assessment will examine FBP documents to determine how the project identified, gender-specific approaches recognizing the special needs of women versus men (for example with regard to access and compliance), analyzed and utilized sex-disaggregated data in developing specific activities and interventions, and analyze results, giving special attention to the Economic Strengthening program linkage. Considerations will be given to gender influences, male involvement and traditional/cultural issues with regard to project interventions, particularly during FGDs. 50 | P a g e M&E systems and data quality assessment: Assessment will be conducted on data management systems for data quality. The objectives of the assessment will be:  To determine the extent to which indicators and tools used to monitor and measure progress toward results are adequate.  To identify improvements that can be made to better capture progress. The team will assess the M&E system of FBP using a DQA question guide. The DQA will focus on both Save the Children and its implementing partners. In accessing these systems the team will sample select indicators from the thematic area. These indicators will be assessed on a range of issues including their data validity, reliability, precision, integrity, accessibility, confidentiality, security and timeliness. To enable comparison of systems in the analysis, the team will access the same indicators in each of these organizations. To assess the M&E system, the team will assess the organizations’ M&E plans, indicators and their definitions, data collection protocols, databases, data backup mechanisms and reporting procedure. Data analysis will be organized to answer the acceptability and appropriateness of the program design and implementation in reaching the stated objectives at different levels. These include: 1. Assessment of the results the project has achieved at levels of the beneficiaries i.e. HIV positive persons that receive RUTF and/or RUSF, 2. At the level of the health service delivery system and the associated staff ,and 3. At the systems that quantify and forecast, manage and procure the RUTF and/or RUSF (Pharmaceutical Fund and Supply Agency [PFSA] /RHB), taking into consideration that the actual procurement has for the most part been under another USAID mechanism. Data analysis will be ongoing in the field. Final analysis will be conducted jointly as a team, in which findings will be reviewed, compared, confirmed and synthesized, providing biases for evaluation conclusions and definition of recommendations for the report. Where necessary, the team may revert to key stakeholders, USAID and/or FBP staff to confirm or address issues raised during fieldwork. EVALUATION REPORT The Final Evaluation Report will be structured around the four analytical domains with the findings, conclusions, and recommendations for each question clearly delineated and summarized. The report will focus on the qualitative analysis of the FGDs and KIIs. The 14 key performance indicators will be analyzed to determine if the project met its targets and Intermediate Results by intervention. Qualitative analyses from the KIIs and FGDs on perceptions of FBP and NACS will support the recommendations for future programming of FBP. The objective is to demonstrate clearly the perceptions and views of beneficiaries from the selected 51 | P a g e regions as to the acceptability, appropriateness and relevance of FBP on both improved health status and increased economic activity. Together, the quantitative and qualitative data will provide the basis for lessons learned, best practice, conclusions and recommendations for future programming. The report will document how the project has responded to challenges and sought solutions. The evaluation report shall include the following, with the main report from introduction to recommendations not being more than 40 pages and additional information and data presented in the Annexes. 1. Executive Summary (3 – 5 pages) 2. Background; 3. Introduction; 4. Methodology; 5. Findings, including Lessons Learned; 6. Conclusions; 7. Recommendations; 8. Annexes, including: i. Scope of Work ii. Data collection tools iii. List of key informants iv. Documents consulted 52 | P a g e TIMELINE The Scope of Work specifies six deliverables, each of which will be completed as outlined in the timeline below: Table 2. Evaluation Timeline Tasks Home In￾Country Dates Deliverables* Document review, draft evaluation design, methodology and tools X Nov 10-21, 2014 Review SOW & background reading; discuss evaluation report, work plan & individual assignments; review logistics Develop evaluation design & methodology, sampling, analytical plans /tools, work schedule. Travel – Team Leader arrives X Nov 17, 2014 Kick-off meeting with USAID X Nov 19, 2014 Prepare Evaluation Framework and Work Plan X Nov 21-24, 2014 Submit Evaluation Framework and Work Plan including design, methodology and tools to USAID In-brief Nov 25, 2014 In-briefing with USAID Team Planning Meeting X Nov 26-27, 2014 Finalize Workplan, clarify roles and responsibilities Final Workplan Nov 27, 2014 Submit Final Workplan Field Work X Nov 28 – Dec 12 2014 Data analysis and report writing X Dec 13-16, 2014 USAID Debrief X Dec 17, 2014 Team presents Debriefing to USAID Travel Dec 17, 2014 Team Leader depart Team report writing X Dec 29, 2014 – Jan 16 2015 Draft Evaluation Report Jan 19, 2015 Submit Draft FBP Evaluation Report with annexes to USAID for review 53 | P a g e Table 2. Evaluation Timeline Tasks Home In￾Country Dates Deliverables* USAID review X Jan 19-23, 2015 Evaluation team addresses USAID comments X Jan 26-29, 2015 Final Evaluation Report Jan 30, 2015 IBTCI submits final report for USAID final review USAID final review and comments Feb 2-6, 2015 Submit final report to DEC Feb 13, 2015 Incorporate final comments and submit final report with annexes Table 3. Draft Schedule for final performance evaluation of USAID’s FBP program 24 November – 17 December 2014 Date TEAM 1 TEAM 2 Team Leader Gordon Mortimore Research Assistants Mesfin Tesfay Tekle, Meskerem Fisseha Birhanu Team Leader Mesfin Beyero Research Assistants Eyerusalem Girma Andenet Halle Godana Mon Nov 24 Addis Finalization of work plan Addis Finalization of work plan Tue 25 Addis USAID In-Briefing Addis USAID In-Briefing Wed 26 Addis KII with FMoH, Save US Addis KII with FMoH, Save US Thu 27 Addis AM: Team Induction PM: KII Implementation partners Addis AM: Team Induction PM: FGD Field Test Back Lion Hospital Addis AM: Team Induction PM: FGD Field Test Back Lion Hospital Addis AM: Team Induction PM: FGD Field Test Back Lion Hospital Fri 28 Addis AM: KII Addis Regional Health Addis AM: FGD Gandi Hospital Addis AM: KII Addis Regional Health Addis AM: FGD Zewuditu Hospital 54 | P a g e Table 3. Draft Schedule for final performance evaluation of USAID’s FBP program 24 November – 17 December 2014 Date TEAM 1 TEAM 2 Team Leader Gordon Mortimore Research Assistants Mesfin Tesfay Tekle, Meskerem Fisseha Birhanu Team Leader Mesfin Beyero Research Assistants Eyerusalem Girma Andenet Halle Godana Bureau PM: KII Save US Addis Sub￾office PM: FGD Zembaba Hospital Bureau PM: KII Save US Addis Sub￾office PM: FGD St. Poulos Hospital Sat 29 Addis Review of methodology, transcribe notes Addis Review of methodology, transcribe notes Addis Review of methodology, transcribe notes Addis Review of methodology, transcribe notes Sun 30 Mon Dec. 1 Addis KII national stakeholders Addis AM: FGD Bole 17 HC PM: FGD (TBC) Addis AM: FGD Beleteshachew PM: (TBC) Addis AM: FGD Beleteshachew PM: (TBC) Tue Dec. 2 Addis AM: KII national stakeholders PM: Fly to Tigray Travel to Tigray Addis AM: KII Oromiya Regional Health Bureau PM: Travel to East Oromiya Addis AM: KII Oromiya Regional Health Bureau PM: Travel to East Oromiya Wed 3 Makelle AM: KII Tigray Regional Health Bureau PM: KII Save US & partners Makelle AM: FGD Makelle Hospital PM: FGD: Ayder Hospital Adama AM: FGD Adama Hospital PM: FGD St. Aklesia Hospital Adama AM: FGD Adama Hospital PM: FGD St. Aklesia Hospital Thu 4 Adigrat AM: PM: Return to Addis Adigrat AM: FGD Adigrat Hospital PM: Health Center visits Shasemene AM: FGD Shasemene Hospital PM: Travel to SNNPR Shasemene AM: FGD Shasemene Hospital PM: Travel to SNNPR Fri 5 Addis KII with national stakeholders Axum AM: FGD Axum Hospital PM: Health Center Visits Hawassa AM: KII SNNPR Regional Health Bureau PM: KII Save US & partners Hawassa AM: FGD Adare Hospital PM: FGD Hawassa Referral Hospital Sat 6 Addis Drafting of findings Axum Transcribing, data cleaning and Hawassa Drafting of findings Hawassa Transcribing, data cleaning and 55 | P a g e Table 3. Draft Schedule for final performance evaluation of USAID’s FBP program 24 November – 17 December 2014 Date TEAM 1 TEAM 2 Team Leader Gordon Mortimore Research Assistants Mesfin Tesfay Tekle, Meskerem Fisseha Birhanu Team Leader Mesfin Beyero Research Assistants Eyerusalem Girma Andenet Halle Godana analysis analysis Sun 7 Travel to Amhara Travel to Amhara Travel to Sodo Travel to Sodo Mon 8 Gondar AM: FGD Gondar University Hospital PM: Health Center Visits Gondar AM: FGD Gondar University Hospital PM: Health Center Visits Sodo AM: FGD Ottona Hospital PM: Health Center Visits Sodo AM: FGD Ottona Hospital PM: Health Center Visits Tue 9 Bahir Dar AM: KII Regional Health Bureau PM: KII Save US & partners Bahir Dar AM: FGD FelegeHilwot Hospital PM: Health Center Visits Hosanna AM: FGD Hosanna Hospital PM: KII Hosanna Hospital Hosanna AM: FGD:Hosanna Hospital PM: KII Hosanna Hospital Wed 10 AM: Return to Addis PM: KII with national stakeholders Travel to Woldaeya Travel to Jimma, West Oromiya Travel to Jimma, West Oromiya Thu 11 Addis KII with national stakeholders Woldaeya AM: FGD Woldaeya Hospital PM: Health Center Visits Jima AM: FGD Jimma Hospital PM: Health Center Visits Jima AM: FGD Jimma Hospital PM: Health Center Visits Fri 12 Addis KII with national stakeholders Dessie AM: FGD Dessie Referral Hospital PM: Health Center Visits Woliso AM: FGD St Luke Hospital PM: Health Centre Visits Woliso AM: FGD St Luke Hospital PM: Health Centre Visits Sat 13 Addis Drafting of findings Travel to Addis Travel to Addis Travel to Addis Sun 14 Mon 15 Addis Prepare draft findings & recommendations Addis Prepare draft findings & recommendations Addis Prepare draft findings & recommendations Addis Prepare draft findings & recommendations Tue 16 Addis Addis Addis Addis 56 | P a g e Table 3. Draft Schedule for final performance evaluation of USAID’s FBP program 24 November – 17 December 2014 Date TEAM 1 TEAM 2 Team Leader Gordon Mortimore Research Assistants Mesfin Tesfay Tekle, Meskerem Fisseha Birhanu Team Leader Mesfin Beyero Research Assistants Eyerusalem Girma Andenet Halle Godana Prepare draft findings & recommendations Prepare USAID presentation Prepare draft findings & recommendations Prepare draft findings & recommendations Prepare USAID presentation Prepare draft findings & recommendations Wed 17 Addis Debriefing with USAID Addis Debriefing with USAID Addis Debriefing with USAID Addis Debriefing with USAID 57 | P a g e ANNEX V: DATA COLLECTION INSTRUMENTS FOR KIIS AND FGDS KEY INFORMANT INTERVIEW (KII) SCHEDULE FOR NATIONAL AND REGIONAL STAKEHOLDERS Region: District/Town: Organization: Date: Sex: Position: Venue: Time Started: Time Ended: Stakeholder Type4 : Introductory Remarks Thank you for agreeing to talk with us. International Business & Technical Consultants, Inc. (IBTCI) has been contracted by the United States Agency for International Development (USAID) to carry out this final performance evaluation of the Food by Prescription (FBP) project. The purpose of this evaluation is to assess the achievements of the FBP project, its successes and challenges. We believe that you are in a good position to tell us about your organization and what it is doing in relation to the FBP project, hence this interview. We anticipate the interview will last about an hour or less and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and the information you give us will be reported without names. Your participation is voluntary and you can refuse to answer any question or all the questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any benefit or consequence. Do you have any questions? Can we begin now? 4 Stakeholders include: Save the Children US staff at national and regional level, FBP implementing partners, FMoH, Regional Health Bureau, external stakeholders: civil society, multi-laterals, bilaterals, CDC, WFP etc. 58 | P a g e Introduction 1. Please give us a background of your organization (for implementing partners & external stakeholders) your department (for FMoH& regional health bureau) and what it does? 2. How long has your organization been involved with the FBP project? Probe: Start and end date of the involvement. 3. Please specify in what activities was your organization involved with the FBP project? Analytical Domain 1: Effectiveness of Activity Implementation and Management To what extent did the project achieve the planned objectives and results? 1. In your opinion, what have been the major achievements of the FBP project? Guidance: How did service(s) provided under these activities meet community needs? How accessible were these services? What activities worked best? How? 2. What factors facilitated the achievement of results? What were their effects on project achievement? 3. What factors inhibited the achievement of results? What were their effects on project implementation? 4. What were the major challenges/risks? How were these risks mitigated? To what extent were the project design, implementation, and management effective and why? 1. In your opinion, what components of the project design and/or implementation were the most effective and why? Guidance: What have been the effects of these components on the project outcomes? How adequate was the implementation of the planned interventions? 2. In your opinion, what components of the project management were the most effective and why? 3. Overall, what are the lessons learned on project design, implementation, and management and their effects in accomplishing the project’s targets and outcomes? 4. In your opinion, what were the innovative activities implemented by FBP? Explain? Note for Interviewers: The analytical domains provide a range of questions to provide information from a range of stakeholders. Some questions may be more relevant for specific stakeholders and the interviewer should use his/her understanding of the stakeholder background to decide which specific questions to ask. 59 | P a g e Analytical Domain 2: Sustainability: What progress has been made towards ensuring the sustainability of FBP’s approaches and specifically NACS? 1. To what extent did the project influence strategy, programming, and policy at the national, sub-national and community levels? 2. To what extent have NACS indicators been integrated within health facilities and government reporting formats? 3. Do you know if any other stakeholders adopted NACS in their programming? (e.g., government counterparts, the Global Fund, UN etc.) 4. To what extent did the project build the organizational and technical capacity of local implementing partners, Regional Health Bureaus and health facilities? 5. Do you think that the facilities supported by FBP will continue to provide NACS services? Will they be able to do so without the need for more capacity building? 6. To what extent has NACS been integrated into in-service training programs for nurses, midwives and medical doctors in targeted universities? Analytical Domain 3: Relevance: To what extent are the objectives of FBP consistent with the needs of the activity’s beneficiaries? 1. To what extent did FBP meet the needs of the beneficiaries in the communities in which the project operated? Why? (OVC, PMTCT – Pregnant and lactating mothers, PLHA Men, PLHA Women) 2. How relevant is the FBP project to your own work? 3. Have you received any training/capacity building from Save the Children US or a FBP implementing partner? 4. How relevant were the interventions to the achievement of the planned outcomes? 5. How relevant was the project partnership to the achievement of the planned outcomes? Analytical Domain 4: Gender: To what extent did the interventions address gender issues that expose women to HIV/AIDS or malnutrition? 1. Does the FMoH provide guidance concerning gender sensitive HIV and/or nutrition programming? Has the project followed these guidelines? 2. To what extent has the project integrated gender considerations into its activities? If at all, how? 3. Explain the quality of gender activities provided by the activity? 4. How accessible were these services? 5. Did FBP activities have any influence on the status of women and men? 6. To what extent did FBP develop measures to enhance women’s participation in the project? 60 | P a g e Supplementary Analytical Domain5 : Data Management. To what extent has strategic information (SI) activities informed the planning, implementation and monitoring of the project? 1. What data were used to inform FBP project: a. planning; b. implementation; c. monitoring for results and decision making? Guidance: (i) How did you use the data? (ii) How often did you use the data? 2. What were the key data sources? Guidance: How often did you collect the data? 3. To what extent was the monitoring and evaluation system used for activity improvement? 4. Was the data collected, transmitted, collated and interpreted in a timely manner to inform project decision-making? Explain? 5. In your opinion, which performance monitoring methods and tools were effective in monitoring FBP outputs and results? 6. How useful and appropriate were the data and information to FBP implementers, government, and health facility staff? In what ways? 7. What studies have been commissioned? What was the quality of designs, data collection and findings? How were study findings used for planning and implementation? 8. Have there been any external reviews or evaluations of FBP? 9. Has there been any qualitative research undertaken? Examples of best practices documented? 5 Additional questions for management and M&E staff at Save US, Implementing partners, Regional Health Bureau 61 | P a g e Focus Group Discussion Guide My name is ----------------. I come from IBTCI. IBTCI is conducting the final performance evaluation of the Food by Prescription (FBP) activity in selected sites of the project implementation. IBTCI is seeking views and information in relation to the project from selected persons in this community. Because we cannot ask everybody, you have been randomly selected to represent the views of other project beneficiaries in this community. The reason for your coming here is to get your views and ideas related to the project. I request you to openly discuss issues that are related to the aspects of the implemented project. Your views will be of great help to USAID to readjust the project’s implementation process so as to meet its intended objectives in other similar projects in the future. Do you agree? Yes (continue with discussion), No (end the discussion) 1. Participants level of awareness of the FBP activity: a. What do you call the RUTF/RUSF products? b. Do you know the admission and discharge criteria to and from the FBP activity? What are the criteria that were applied for your selection? c. Do you know the amount of RUTF/RUSF that has been prescribed to you? d. Are you receiving the same amount regularly? If ‘No’, what are the gaps? 2. Activity effectiveness: a. To what degree were inputs (RUTF/RUSF) provided / available on time? Any shortages? b. Were there any gaps/delays in the service delivery? If yes, what were the gaps? c. Do all those who need the services have access the project services? If not, where is the problem? d. Are you satisfied with the FBP service? e. What is the gender composition of the beneficiaries? Do all beneficiaries (men, women, male and female children) have equal access? Date Region Zone/Sub City Woreda Facility Beneficiary type Men HIV Women HIV PMTCT OVC Participants FGD moderator's name (List number of the participants, their age range and gender) 62 | P a g e 3. Referral system a. How is the referral system between the health facility and the community working? b. Any gaps? How best can it be improved? c. Are there any issues with linking FBP beneficiaries to economic strengthening activities? 4. Utilization of Nutrition supplements and clinical follow up: a. What do you do with the RUTF/RUSF? Do you consume it as prescribed? Was there any problem in the taste or flavor? b. Are there people who are selling these nutrition supplements? Why do they sell it? c. Are there people who are sharing the nutrition supplements with other family members? If ‘yes’, what are the reasons for the sharing? d. How frequently is your nutritional status being monitored in the health facility? e. Were there any unplanned positive/ negative effects of the nutrition supplements on the beneficiaries? If ‘yes’, what were they? To what extent did the health workers take appropriate measures on the negative effects, if any? 5. How relevant is the FBP activity? a. Does the project respond to your needs? b. Have the project activities been the best way to achieve the objectives? If not, which are the alternative options? c. Are there best practices/treatment outcomes that could be shared? d. Any challenges with the activity? How can it best be improved? Thank the participants and finish the FGD. INFORMED CONSENT STATEMENT Good day. My name is ___________________, and we are conducting an evaluation of the Food by Prescription Project with USAID and other stakeholders. The purpose of the evaluation is to determine the effectiveness and relevance of the interventions of FBP and document what has worked well and what has not. Lessons from this evaluation will be integrated in future programming of USAID. You were selected as a Key Informant to provide information for this evaluation. The information collected will only be used for the evaluation. All the information is strictly confidential. [Interviewer collects signed consent forms]. I would also like to clarify that this interview is voluntary and that you have the right to withdraw from interview at any point without consequence. Thank you very much. 63 | P a g e At this time, do you have any questions? Are you willing to participate in this study? Yes 1) Proceed No 2) Thank the KI and STOP HERE May I begin the discussion now? Yes 1) Continue with the Key Informant Interview No 2) STOP HERE Start Time: ____:____ Interviewee signature _____________________________ Date ___________________ Interviewer signature _____________________________ Date ____________________ Thank you 64 | P a g e ANNEX VI: SOURCES OF INFORMATION NATIONAL KEY INFORMANT NAME/ORGANIZATION 1. Dr. Ferew Lemma, Senior Nutrition Adviser, FMOH 2. Mr. Paul Emes, Deputy Country Representative, Save the Children US 3. Mr. Biarra Melese, Coordinator, Nutrition Case Team, FMOH 4. Dr. Ephrem, Director MCH, FMOH 5. Mr. Yalemsew Derib, FBP Focal Point, SCMS 6. Mr. Loren Wiley, DCOP SCMS 7. Dr. Gideon Chohen, FBP Ex-COP, Save the Children US 8. Mr. Aschalew Kassahun, FBP M&E Adviser 9. Dr. Yared Abebe, FBP DCOP/Senior Nutrition Adviser, Save the Children US 10. Dr. Abdulaziz Ali, FBP COP Save the Children US 11. Mequaninit Nega, Stock and Distribution officer, PFSA Addis Ababa Branch 12. Mrs. Beza Abebe, Livelihood Adviser, Save the Children US INTERVIEWEE/POSITION/NAME OF HEALTH FACILITY/TOWN/REGION 1. Tsegaye Bekele/Clinical Nurse, ART Prescriber and FBP Focal Person/Bole 17 Health Center/Health Center/Addis Ababa/Addis Ababa 2. Gemechis Kebede/TB/HIV Co-coordinator and FBP Focal Person/Zembaba Hospital/Addis Ababa 3. Menase Legese/ART and FBP Focal Person/Dessie Health Center/Health Center/ Dessie/Amhara 4. S/r Atakelti Berhane/ART and FBP Focal Person/Ayder Referral Hospital Hospital/Mekelle 5. S/r Zuriash Halefom/Case Team Leader and FBP Focal Person/Mekelle General hospital/Hospital/Mekelle/Tigray 65 | P a g e 6. Rekik Mengiste/HIV/AIDS, ART, Social Mobilization Co-coordinator and FBP Focal Person/Woldia Health Center/Health Center/Woldia/Amhara 7. ART Coordinator/Hawassa University Referral Hospital/Hospital/Hawassa/SNNPR 8. ART Coordinator/Shashemene Abosto Health Center/Health Center/Shashement/Oromiya 9. S/r Aberash Fesha/ART & FBP Focal Person/Tekelesewat Health Center/Adigrat/Tigray 10. Sr Hana/Clinical Nurse and ART Focal Person/ Aklesia Memorial Hospital/Adama/Oromia Region 11. Bizunesh Guracha/ART Focal Person/Adare Hospital/ Hawassa/SNNPR 12. Getahun Alemu,/ART Nurse/Shashemane Hospital/Shashemane/Oromia Region 13. Asiya Jeylan/ART Focal Person and Treating Clinician/Adama Hospital/Oromia Region, Ayele Cherinet/Disease Prevention and Health Promotion Core Process Owner/Yekatit 12 Hospital/Addis Ababa 14. Dr.Jiksa Dabessa/ Chief Clinical Officer (Medical Director)/Adare Hospital/ Hawassa/SNNPR 15. Clinical Nurse/Head of Disease Prevention and Control/ Beletishachew Health Center/Addis Ababa 16. Dr.Cherinet Leka/Clinical Service Directorate Director, Berhanu Kolisho/ART Focal Person 17. Ayelech Zana/ART Nurse, Wolayita Sodo Teaching Referral (Otona) Hospital/ Sodo/SNNPR 18. Sr.Demekech Mengesha/Head of Health Centre/Sodo/ SNNPR 19. Berhanu Lagebo/OPD Case Team Coordinator/Sodo Health Centre/Sodo/SNNPR 20. Temesgen Abate, Manager, Hossana Health Center, Lemlem Kifleyesus ART Focal Person 21. Dr.Adane Desta, Chief Clinical Officer, SNNPR, Hossana (Nigist Eleni Mohammed Memorial) Hospital, SNNPR 22. S/r Zewdu Abay/ART/FBP Focal Person/Axum Health Center/Health Center/Axum/Tigray 23. Dr.Fekadu Assefa, CEO, 1 year, Ato Feyera Gebisa, Supply Chain Director, Jimma Hospital, 4 years (since inception) 24. Ato Tarekegn Woyessa, ART Coordinator/Jimma University Hospital/Jimma 25. Zenash Demas, ART Focal Person, Chalchise Seboka, Case Manager Dereje G/Sellasie, ART Dtaff/Jimma Health Centre/Jimma/Oromia Region 66 | P a g e 26. Dr. Martha Lunardi, Internist and TB/HIV Coordinator, Meseret Biratu, ART Focal Person/St.Luke Hospital/Wolliso/Oromia 27. Benti Ifa, ART Focal Person/ Wolisso Health Centre/Wolliso/Oromia 28. Mesgananw Adugna/Gondar Referral Hospital/Hospital/Gondar 29. S/R Yemiserach/Bahardar Health Center/Health Center/BahrDar KII REGIONAL HEALTH BUREAUS, REGIONAL NGOS, SAVE THE CHILDREN US REGIONAL OFFICES 1. Taye Wondimu, Nutrition Focal Person, Tsegaye Tesfaye, TB/HIV Program Expert 2. Dereje Assefa, Save the Children US Addis Regional Coordinator 3. Tsegaye Tesfaye, TB/HIV Program Expert 4. Miriam, Clinical HIV Team Leader, Afework Kahsay, HIV C&S Coordinator, Tigray RHB 5. Tadios, Organisation: HIDA Program Coordinator, Addis Ababa 6. Henok Program Director, Asmera, Volunteer, Organisation Mums for Mums Tigray 7. Mrs Beza Abebe, Livelihood Adviser, Save the Children US 8. Wzo. Fatuma Jemal/ Executive Director/NOSAP+ Ato Abebaw Deribew, Program Coordinator/NOSAP+/Hawassa/SNNPR 9. Baylegn Masresha, Health Development, Promotion and Nutrition Officer, Amhara Regional State Health Bureau 10. Fikru Seneshaw, FPB Activity Regional Coordinator, Amhara Regional State 11. Mersha Birara, Family Health Sub-process 12. Senait Assefa, TB/HIV officer, Addis Ababa Regional Health Bureau 13. Tadesse Degefa, Adama Branch Program Coordinator, Oromia, Mekdim Ethiopia National Association 14. Adugna Mamo, East Oromia Regional Coordination, Oromia, Save the Children US 15. Aknaw Kawza, Health Development Plan, Monitoring and Evaluation Process Owner 16. Fisseha Tekle, Regional Coordinator, Region: SNNPR, Save the Children US Isseha Tekle, Regional Coordinator Region: SNNPR, Save the Children US 17. Fikirte Abera, Fikirte Abera, Curative and Rehabilitation Services Core Process Owner Delegate 67 | P a g e 18. Estifanos Geribo, Curative and Rehabilitation Services Clinical Officer Region, SNNPR, Regional Health Bureau 19. Curative and Rehabilitation Services Core Process Owner Delegate 20. Estifanos Geribo, Curative and Rehabilitation Services Clinical Officer Region: SNNPR Regional Health Bureau 21. Fisseha Tekle, Regional Coordinator, SNNPR, Save the Children US FGD GROUP (MEN HIV, WOMEN HIV, PMTCT/OVC) /HEALTH FACILITY/TOWN/REGION/NO. OF PARTICIPANTS/AGE RANGE/MODERATOR'S NAME 1. Women HIV/Zembaba General Hospital /Addis Ababa/Addis Ababa/6/25 to 40 years/Meesfin T. & Meskerem F. 2. Men HIV/Bole 17 health center/Addis Ababa/3/25-49 years/Mesfin T. & Meskerm F. 3. Women HIV/Bole 17 health center/Addis Ababa/3/25-49 years/Mesfin T & Meskerm F. 4. OVC/Referal Hopspital/Dessie/Amhara/4/7 to 14 years ld/Meskerm F & Mesfin T/02 5. Women HIV/Woldia Hospital/Woldia/Amhara/6/30 to 55 years old/Meskerm F. & Mesfin T. 6. Women HIV/Ayder Referal Hospital /Mekelle/Tigray/2/35 & 40 years/Meskerem F. & Mesfin T. 7. Women FGD/Mekele General Hospital/Mekelle/Tigray/05/29 to 42/Meskerm F & Mesfin T. 8. Women FGD/Health Center/Adigrat/Tigray/6/29 to 42/Meskerm F & Mesfin T. 9. FGD with Men HIV, Adare Hospital, Hawassa, SNNPR, 4 partitcipants, Age range from 28-40, Andenet 10. Men HIV/Shashemene Hospital/Shashemene/Oromiya/eight participants/30-50 years/Andenet 11. Women HIV Adama Hospital/ Adama/Oromia 12. Women HIV/ Adaram Hospital Hawassa/7/25-35 13. HIV Women/Hawassa Referal Hospital/Hawassa/SNPPR/11/25-45/Eyerusalem 14. Men HIV/ Shashemane Hospital/Shashemane/Oromia/7/Eyerusalem & Andenet 15. PMTCT(Women)/ Wolaita Sodo Health Center/Wolaita Sodo/SNNPR/9 participants/20-30 16. Men/Wolaita Sodo Hospital, 5/30-45 68 | P a g e 17. PMTCT Women Lactating Wolaita Sodo Health Center SNNPR/ 8/22-35 18. Men HIV , St Paul's Hospital/Addis Ababa/14/06/Eyerusalem Girma 19. Women HIV,Zewditu Memorial Hospital/Addis Ababa/14/06/Andinet Haile 20. Women HIV, Beletishachew Health center/Addis Ababa/14 /5/ Andinet Haile 21. Women HIV/ Ottona Hospital/Sodo/SNNPR/9/30-35/Eyerusalem 22. Women HIV/ Sodo Health Center/Sodo/SNNPR/12/20-50/Eyerusalem 23. Women HIV/Nigist Eleni Mohamed Hospital/Hossaena/SNNPR/12/25-60/Eyerusalem 24. Women HIV/Axum Hospital/Axum/Tigray/6/25 to 30/Mesfin T. & Meskerm F. 25. Women PMTCT/Gondar Health Center/Gondar/Amhara/6/25 to 32/Meskerm F & Mesfin T. 26. Women HIV/Felegehiwot Hospital/BahrDar/Amhara/5/25 to 36/Meskerm F. T. 27. PMTCT/Jimma Hospital/Jimma/Oromiya/3 participants/24-36/Andenet 28. Men HIV, Jimma Health Center/Jimma/ Oromiya/5 parttipants/28-35/Andenet 29. ADULT HIV(Mixed group), Woliso Health Center/Woliso/Oromiya/12 participants/30- 45/Andenet 30. Women HIV/Jimma Health Center/Jimm/Oromia 31. Women HIV/Jimma Referral Hospital/Jimm/Oromia 32. WomenHIV/St Luke Catholic Hospital and Nursing School/Wolisso/Oromia/ 73/Eyerusalem/ 69 | P a g e ANNEX VII: FOOD BY PRESCRIPTION DATA QUALITY ASSESSMENT FORM Region: _________________ Name of facility: ________________ 2010 2011 2012 2013 2014 Jul Nov Feb May Jun Sep Mar May Jun Sep Mar Jun Sep Dec Jan Apr May Aug # clients received nutritional assessment and counseling # clients clinically assessed and severely malnourished # clients clinically assessed and moderately malnourished # clinically malnourished and received therapeutic and/or supplementary food # referred clients to WFP and other ES partners # clinically malnourished PLHIVs and OVCs clients who were graduated from the program # clinically malnourished clients defaulted # PLHIVs and OVCs clients who died during course of treatment 70 | P a g e ANNEX VIII: CONFLICT OF INTEREST FORMS 71 | P a g e 72 | P a g e 73 | P a g e 74 | P a g e