EVALUATION USAID/Egypt: End of Project Performance Evaluation of Avian and Pandemic Influenza Program DECEMBER 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by D. Lucey, A. Forman, D. Bennett, and W. Anwar through the GH Tech Bridge II Project. Cover Photo by Wagida Abel Rahman Anwar EVALUATION USAID/Egypt: End of Project Performance Evaluation of Avian and Pandemic Influenza Program DECEMBER 2012 Global Health Technical Assistance Bridge II Project (GH Tech) USAID Contract No. AID￾OAA-C-12-00027 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-02-016) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00027. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM i ACKNOWLEDGMENTS The evaluation team is grateful for the assistance of all the people we met who gave their time and opinions so freely and frankly. We are particularly grateful for the assistance and guidance of the USAID/Egypt Health Team, for the assistance of the personnel of the General Organization for Veterinary Services and the Food and Agriculture Organization who accompanied us on field trips, and for the logistics support of Progress2. ii USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM iii CONTENTS ACRONYMS ........................................................................................................................................ v EXECUTIVE SUMMARY ................................................................................................................. vii I. INTRODUCTION ..................................................................................................................... 1 II. BACKGROUND ........................................................................................................................ 3 III. METHODOLOGY ..................................................................................................................... 7 Evaluation Methods .................................................................................................................... 7 Data Limitations .......................................................................................................................... 7 Gaps and Constraints ................................................................................................................ 7 IV. FINDINGS AND CONCLUSIONS ....................................................................................... 9 Extent to which the Program Achieved Intended Goal and Results .............................. 9 Assessment of Monitoring and Evaluation Systems ..........................................................10 Progress Toward Desired Outcomes and Impacts ..........................................................12 Assessment of Management and Administration Arrangements ...................................25 V. CHALLENGES .......................................................................................................................... 27 The Political Environment ......................................................................................................27 Community Perceptions Following the H5N1 and H1N1 Emergencies .....................27 The Long-term Risk-reduction Approach ..........................................................................27 The Implications of Partial Success .......................................................................................27 The Prospect of a High-mortality Human Pandemic ........................................................28 VI. LESSONS LEARNED AND RECOMMENDATIONS ...................................................... 29 ANNEXES Appendix A. Scope of Work .......................................................................................................... 33 Appendix B. Persons Contacted ................................................................................................... 55 Appendix C. References .................................................................................................................. 59 Appendix D. Data Collection and Analysis ................................................................................. 63 FIGURES Figure 1. Indicators for the SAIDR Project ................................................................................... 5 iv USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM v ACRONYMS AIVEP Avian Influenza Vaccine Efficacy Project API Avian and pandemic influenza ARDS Acute respiratory distress syndrome AWSO Arab Women Speak Out BCC Behavior change communication CAHO Community animal health outreach CHL Communication for Healthy Living CHPL Central Public Health Laboratory COW Community outreach worker ECTAD FAO Emergency Centre for Transboundary Animal Diseases EHCS Egyptian Health Communication Survey FAO United Nations Food and Agriculture Organization FGD Focus group discussion GOE Government of Egypt GOVS General Organization for Veterinary Services H1N1 Subtype of influenza A virus H5N1 Subtype of influenza A virus HPAI Highly pathogenic avian influenza IDI In-depth interview IPC Interpersonal communication JHU/CCP Johns Hopkins University, Center for Communication Programs KAP Knowledge, Attitudes, and Practices (research survey) M&E Monitoring and evaluation MOALR Ministry of Agriculture and Land Reclamation MOE Ministry of Education MOHP Ministry of Health and Population MOI/SIS Ministry of Information/State Information Service NAMRU-3 U.S. Navy Medical Research Unit - 3 NGO Nongovernmental organization NLQP National Laboratory for Quality Control of Poultry Production NSC National Supreme Committee for Avian and Pandemic Influenza OFFLU FAO/OIE network of avian influenza expertise OIE World Organization for Animal Health (Office International des Epizooties) vi USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM PDSR Participatory disease surveillance and response PHC Primary health care PMP Performance monitoring plan PPE Personal protective equipment RTE Real-time evaluation SAIDR Strengthening Avian Influenza Detection and Response Project SMS Short message service SOW Scope of work STOP AI Stamp Out Pandemic and Avian Influenza TA Technical assistance TOT Training of trainers UNICEF United Nations Children’s Fund USAID United States Agency for International Development WHO World Health Organization USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM vii EXECUTIVE SUMMARY The purpose of this performance evaluation of the USAID/Egypt Avian and Pandemic Influenza (API) Program was to review, analyze, and evaluate its effectiveness in achieving objectives and to assess its contribution to improving API prevention and control. This API Program began in 2007 after the recognition in 2006 of highly pathogenic avian influenza (HPAI) in Egyptian poultry caused by H5N1 influenza virus, and of human infections with the same virus. By 2008 it was documented that the disease had become endemic in poultry in Egypt. It has also become endemic in Vietnam, China, Indonesia, Bangladesh, and India. As in other nations that have a high density of poultry (the density in Egypt is twice that in Vietnam), interventions such as culling without compensation and vaccination focused on household poultry have failed in Egypt. It was soon recognized that emergency control measures could not prevent the spread of H5N1 HPAI. As a result the national strategy was revised in 2010 to put much greater emphasis on long-term risk reduction via farm biosecurity and market chain hygiene. Given the prevalence of the disease in poultry, it is not surprising that laboratory-confirmed human H5N1 infections in Egypt have occurred every year since 2006. The cumulative total of 168 human cases to date is second only to Indonesia’s 191 cases. Notably, however, in Egypt he fatality rate is much lower (6%) in patients younger than 15 than in Indonesia (~75%). This is probably related to several factors: Ministry of Health and Population (MOHP) case management protocols include rapid initiation of antiviral drug treatment before laboratory confirmation and shortly after onset of symptoms, when children are typically brought to medical attention, and widespread recognition of poultry as a risk factor for human H5N1 infection. Due to concerns about a pandemic caused by a highly fatal airborne-transmissible H5N1 virus, Egypt and the world were prepared for a worst case scenario. By 2008 the MOHP had completed its Preparedness Plan for Pandemic Influenza, with support from the USAID API Program. Thus, a year later Egypt was better prepared when the first influenza pandemic since 1968 did occur, although it was caused by H1N1 (2009) influenza virus rather than the H5N1 virus. Fortunately, this pandemic was relatively mild. The combination of a mild pandemic due to H1N1 and lack of sustained human-to-human transmission of H5N1 virus as of 2012, however, has contributed in Egypt and globally to a diminished sense of urgency about the threat of H5N1 influenza. Thus, communication messages need to be carefully designed if any future H5N1 pandemic threat program is to be effective. The design of this performance evaluation focused on how the multiple components of the API program were implemented, the extent to which the expected results occurred, and how program activities were perceived, valued, and sustained. Data collection methods included review of program-related documents provided by USAID/Egypt, in-depth interviews, and focus group discussions. In addition to data collection in Cairo, sites in Gharbeya and Fayoum governorates were visited. The evaluation team was in Egypt from October 29 to December 1. However, this relatively short time frame was a surmountable challenge in terms of being able to assess the performance of the API Program sufficiently. This performance evaluation addressed five questions. A summary follows of general key lessons learned and recommendations. Detailed responses to the evaluation questions can be found in sections 4.1-4.4. A complete list of the eight recommendations can be found in section 6. viii USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Taken as a whole, the API Program was only partly successful in achieving its intended goals of preventing H5N1 infection in humans and birds. In the face of overwhelming constraints, such as the very high density of poultry in Egypt, the low biosecurity of poultry production, and complex informal marketing systems, only modest strides have been made toward prevention and control of H5N1 transmission within poultry and humans. A redirection of strategic focus to long-term risk reduction, such as better farm biosecurity and market chain hygiene, instead of emergency outbreak measures, such as culling, is more appropriate for managing the disease. Future support for HPAI control should seek a commitment to follow the modified Animal Health and Livelihood Sustainability Strategy of 2010 that has a long-term risk reduction focus. Consideration should also be given to a new program management structure, to strengthen both coordination between implementing partners and monitoring and evaluation (M&E). In parallel, any future program should continue technical assistance to regulatory veterinary services to secure gains already made and further build up professional capacities. Animal health and behavior change communication (BCC) professionals need to design risk reduction approaches for any new program that are both acceptable and effective—and their impact on the incidence of HPAI should be measurable. Targeted surveillance for epidemiological monitoring is essential for planning risk reduction measures. Behavior change should be institutionalized through policy and regulatory actions. Approaches to changing poultry husbandry and marketing practices on commercial farms, in households, and along the value chain should be pro-poor and gender-sensitive, and stakeholders should be able to recognize clear economic benefits. Planning for future programs should emphasize social investment in partnerships, with government, civil society, and poultry industry stakeholders an essential part of making progress in HPAI control. In Egypt an alliance of public health and veterinary epidemiology and laboratory expertise should be supported to strengthen influenza virus molecular epidemiology. Targeted testing is needed for H5N1 virus mutations that have potential for human-to-human transmissibility, including those found in Egypt in 2010 and emphasized in recent publications. The Preparedness Plan for Pandemic Influenza (2008) should be updated to incorporate the information gained from the 2009 avian influenza and pandemic influenza experiences. Reasons for success in Egypt, relative to some other nations, in keeping the case fatality rate down to just 6% in children under 15 should be shared globally. Much work remains to be done to build on the API Program achievements, given the scale of the H5N1 influenza problem in poultry and ducks, and the documented potential for an Egyptian H5N1 virus to emerge that can be transmitted from person to person. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 1 I. INTRODUCTION This evaluation was undertaken to assess the performance of the Avian and Pandemic Influenza (API) Program, funded by USAID and implemented in Egypt between October 2007 and September 2011. The purpose of the evaluation was to  Review, analyze, and evaluate the effectiveness of USAID-funded API activities to date in achieving program objectives and completing deliverables; and  Assess USAID/Egypt’s contribution to improving API prevention and control. The task of the evaluation was to produce findings that would inform the direction of future USAID/Egypt API funding and become the basis from which to draft a scope of work (SOW) for a follow-on integrated API program. The evaluation addressed the following five questions: 1. To what extent has the API Program with its different components achieved the intended goals and results? 2. To what extent were the monitoring and evaluation (M&E) systems of the API Program, with its different components, effective for monitoring progress? 3. To what extent were the technical components and approaches of the API Program effective in achieving the desired outcomes/impacts? 4. To what extent were the management structures, administrative support, and partnerships effective? 5. What are the lessons learned, best practices, and the corresponding recommendations for improving the efficiency and effectiveness of the API Program? The audience for the evaluation report was USAID/Egypt, specifically the Office of Health; USAID/Washington; and future implementing partners of API activities. The evaluation was also to provide feedback to each of the partners to help them understand both their strengths and areas where technical, administrative, and management efforts could be improved. In addition, Government of Egypt (GOE) counterparts would learn how to better benefit from implementing partner technical assistance (TA). The four members of the evaluation team were: Dr. Daniel Lucey, API specialist and team leader Dr. Anthony Forman, zoonotic disease control specialist Ms. Dee Bennett, behavior change communication specialist Dr. Wagida Anwar, Egyptian technical specialist The evaluation was conducted between October 24 and December 11, including the assessment trip to Egypt October 28–December 1. After a review of program documentation, the team interviewed a range of stakeholders, including personnel of the Ministry of Health and Population (MOHP), the Ministry of Agriculture and Land Reclamation (MOALR), the Food and 2 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Agriculture Organization (FAO), Communication for Healthy Living (CHL), STOP AI (Stamp Out Pandemic and Avian Influenza); staff of veterinary departments in two governorates; poultry farmers; and householders and others (see Annex B). The team briefed USAID/Egypt personnel periodically and conducted a partners’ briefing on November 27. The team leader stayed in Egypt until December 5th to meet with the multidisciplinary medical team at the Cairo hospital near Heliopolis that cared for 51 children < 6 with H5N1 virus infection between 2006–2012, all of whom survived. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 3 II. BACKGROUND Highly pathogenic H5N1 avian influenza (HPAI) was first detected in Egypt in February 2006 and spread rapidly to most governorates. By May 2012 outbreaks had been detected in 1,065 commercial farms, 1,410 household flocks, and 19 live bird markets. More than 40 million birds had been culled. The first human case of H5N1 infection was diagnosed in March 2006 and by April 2012 Egypt had reported 168 cases, with 60 fatalities. Poultry production is a huge industry in Egypt, employing more than 1.4 million people and involving five to seven million households. Household production represents 80% of poultry production. Most of the husbandry is handled by women, which gives them control of this resource, which represents a very significant contribution to animal protein in diets and overall food security. The primary risk factor for HPAI is poultry density, and Egypt has higher poultry density (3,600 per sq km) than any other endemically-infected country, almost twice as high as Vietnam (G. Dauphin, personal communication). It also has as other contributing factors a high duck population and a complex and poorly regulated poultry marketing system. From the human health perspective, there was concern about people being exposed to H5N1 virus and becoming sick or dying. Experience had shown that early detection and prompt and thorough case management could significantly reduce mortality. Globally, the concern was that an H5N1 virus could develop the ability to spread through the air between humans and possibly cause a high mortality pandemic. In 2009 pandemic influenza caused by the H1N1 virus did surface. It was generally mild but may have expanded the threat of reassortment between two co-circulating viruses, producing a virus with pandemic potential and higher mortality. By the end of the API Program, H1N1 influenza had become part of seasonal influenza and Egypt still has widespread, endemic HPAI in poultry populations in both commercial and household production. There is still the specter of a mutant virus causing a high-mortality human pandemic. While public health services are well-resourced in Egypt, and in fact early detection and good case management of H1N1 infections in humans resulted in lower recorded mortalities in Egypt than in other HPAI-endemic countries, veterinary services are poorly resourced, and there is no robust line of command from national to governorate levels. The veterinary services were therefore poorly equipped to take on the huge task of HPAI control. It had become apparent that international assistance would be of great value in supplementing resources and technical assistance (TA). USAID made a major contribution to the international effort, implementing the API Program in many countries. In Egypt it was implemented as two national projects and national components of four global projects between October 2007 and September 2011. The total budgetary commitment was over $28 million:  Strengthening of Avian Influenza Detection and Response (SAIDR), which had three components (animal health, human health, and communication), October 2007 to September 2011, budgeted at $23.8 million 4 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM  Avian Influenza Vaccine Efficacy Project (AIVEP), June 2008 to June 2011, budgeted at $2.74 million  Stamping Out Pandemic and Avian Influenza (STOP AI), a global project operating in Egypt July 2009 to September 2010, budgeted at $111,000  Improved Biosecurity and Hygiene at Production, Collection Points and Live Bird Markets Including Decentralization, a global project operating in Egypt February 2009 to June 2010, budgeted at $575,000  Developing and Maintaining Public-Private Partnerships for the Prevention, Detection, and Control of Highly Pathogenic Avian Influenza H5N1 and Other Emerging Infectious Animal Diseases, a global project operating in Egypt February to September 2009, budgeted at $312,000  USAID Deliver Project, a global project operating in Egypt since October 2007, budgeted at $633,250. Implementing partners were  The Ministry of Health and Population (MOHP)  The Ministry of Agriculture and Land Reclamation (MOALR), specifically the General Organization of Veterinary Services (GOVS) and the National Laboratory for Quality Control of Poultry Production (NLQP)  USAID grantee Johns Hopkins University Center for Communications Programs (JHU)  The Food and Agriculture Organization of the United Nations (FAO)  The joint FAO and the World Organization for Animal Health (OIE: Office International des Epizooties) network of expertise on influenza (OFFLU)  Development Alternatives Incorporated. The development hypothesis for the program was to produce improved and sustainable avian and pandemic influenza (API) prevention and control, thereby eliminating the necessity for technical assistance in achieving a situation with API in Egypt in which the disease no longer represents a significant threat to human health and in which measures implemented by producers and supported by regulatory authorities minimize the impact of the disease on the industry, livelihoods, and food security. Critical assumptions for successful implementation of the program were that  The national and provincial authorities, at all levels, realize the importance of stopping HPAI at its origin, maintain a strong willingness for investing in the emergency plans, commit resources, and participate in control activities;  The national authorities are willing to provide suitable human resources in an acceptable technical environment and provide the basis for capacity building;  The national authorities are willing to collaborate with all the stakeholders involved in the prevention, detection, and control of the avian influenza emergency; USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 5  The national authorities, at all levels, participate in the required household awareness activities, and stakeholders agree on recommended rehabilitation measures; and  Donors are prepared to invest in the follow-on program. Goals and objectives were stated for the different projects. The SAIDR Project had the goal, subgoals, and expected Intermediate Results (IRs) depicted in Figure 1. Because the goals and objectives of the other projects could readily be accommodated within the same structure, these IRs were adopted as a means of considering the findings and drawing conclusions about the program in Section 4.3 of this report. Figure 1. Indicators for the SAIDR Project IR1.2: Prevent future outbreaks in poultry IR1.1: Improve preparedness and planning to contain bird-to￾bird and bird-to-human transmission of H5N1 IR1.3: Improve detection of H5N1 in wild birds and poultry IR1.4: Improve containment measures in poultry populations IR1.5: Limit exposure of bird￾outbreak investigators, poultry cullers, and poultry vaccinators to H5N1 virus IR2.1: Improve preparedness and planning for a human influenza pandemic IR2.2: Improve detection of suspect H5N1 cases in humans IR2.3: Improve containment measures in human populations IR2.4: Limit exposure of health staff investigating outbreaks and treating patients to the H5N1 virus Subgoal 1: Prevent H5N1 infections in birds Goal: Prevent H5N1 infections in humans Subgoal 2: Prevent human exposure to H5N1 virus IR3.1: Improve awareness of avian and pandemic influenza IR3.2: Decrease high-risk behaviors associated with transmission of H5N1 among birds and humans 6 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 7 III. METHODOLOGY EVALUATION METHODS The team designed an evaluation methodology and data analysis plan to accommodate the time￾frame, access to key stakeholders, and API program documents (see Appendix D for details). Team activities were divided into qualitative and quantitative data collection and into two phases: the first phase, data gathering, began with a desk audit of all the materials provided by USAID that were produced under the API Program. These ranged from quarterly, annual, and final reports to M&E and performance monitoring plans (PMP), and communication materials. PMPs were found for all projects except the DELIVER Project. The team identified and reviewed additional materials, either provided by people interviewed or cited in documents. Following the desk audit the team designed a general questionnaire that captured the five evaluation questions to be addressed. This was in two parts: first were general questions for all and then specific technical questions for each sector and each audience. This was supplemented by focus group discussion (FGD) guides specific to each audience. In-depth interviews (IDIs) were conducted with stakeholders from the MOH, MOALR/GOVS, FAO, NLQP, the World Health Organization (WHO), the U.S. Navy Medical Research Unit 3 (NAMRU-3), UNICEF, and USAID and from closed projects STOP AI and Communication for Healthy Living (CHL). The evaluation team also traveled to Fayoum and Gharbeya to meet with district and governorate animal health officials and conducted IDIs with members of Community Animal Health Outreach (CAHO), rapid response teams, commercial poultry farmers, and trainees from NLQP satellite laboratories and epidemiology and response teams. The team visited two commercial farms (for layers and boilers) and talked with the farm owners. In Cairo the team led an FGD with 22 commercial poultry farmers (Sectors 2 and 3) and an FGD with trainers and beneficiaries of training. In finding answers to all five evaluation questions the team used information gained from the desk audit and in-depth interviews; the FGDs primarily assisted with question # 3 (technical issues) and question # 5 (lessons learned and recommendations). DATA LIMITATIONS Time constraints meant that there were limited opportunities to gain a comprehensive overview of some aspects of the program; in particular there was limited opportunity for onsite visits outside Cairo. Though this reduced the amount of detail obtained, the team believes it did not compromise the integrity of the findings. GAPS AND CONSTRAINTS The biggest problem was difficulty locating full documentation for certain activities (e.g., M&E) now that the API projects have been over for at least a year. The SAIDR Project had an M&E strategic plan that required submission of quarterly and annual plans for activities. STOP AI acknowledged that there had not been an assessment of its trainings or activities. AIVEP training was assessed by on-the-job performance. DELIVER had shipping documents for its personal protective equipment (PPE). CHL’s Egypt Health Communication Surveys and the two GOVS Knowledge, Attitudes, and Practices Surveys (KAPs) in 2009 captured awareness more than measuring behavior change. In mid-December the team leader gathered additional information by phone, Skype, and email from participants in the CHL project (see Appendix C for all resources used). 8 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 9 IV. FINDINGS AND CONCLUSIONS EXTENT TO WHICH THE PROGRAM ACHIEVED INTENDED GOAL AND RESULTS The program was assessed by considering the goals, subgoals, and Intermediate Results (IRs) outlined for the SAIDR Project in the scope of work (SOW) annexed as Appendix A. The program was implemented in an environment in which HPAI was already widespread and entrenched in all poultry sectors and human cases of H5N1 infection had already occurred. Although efforts were very much directed toward improving the capacity of government authorities, the initial approach in the field was an emergency intervention. The program development hypothesis and the SAIDR project goal, while appropriate, were unlikely to be achieved during the program cycle and there was no significant progress toward the subgoals of preventing H5N1 infections in birds and preventing human exposure to H5N1 virus. Of particular concern is that the extent of any progress was not measurable, either from the animal health or public health perspective. In both cases, any perceived reduction in disease incidence could have been real or could equally have been an artifact of reduced surveillance and loss of community interest when the worst-case projections for H5N1, and subsequently the H1N1 pandemic, did not come to pass. However, there was some substantial progress in terms of the IRs for the SAIDR Project. There was considerable improvement in building the capacity of veterinary services and in strategic planning (IR1.1). Promotion of risk reduction practices (IR 1.2), while limited in application, may have already helped with improved HPAI prevention, especially with regard to commercial poultry production. Field and laboratory capacity for HPAI outbreak detection and diagnosis, especially the latter, has been effectively addressed (IR 1.3) and a better approach to outbreak containment (IR 1.4) has potential for gaining producer acceptance. Provision of personal protective equipment (PPE) satisfactorily addressed IR 1.5 and IR 2.4. Preparedness and planning for pandemic human influenza (IR 2.1) was overtaken by the H1N1 pandemic and would benefit from being revisited with a new plan. Early detection of human H5N1 cases (IR 2.2) was apparently satisfactory, with Egypt recording lower mortality rates than other countries. Case management and containment measures (IR 2.3) are very good. But while there was a high level of awareness with respect to both the H5N1 (IR 3.1) and H1N1 viruses, there was only a modest indication of decreased high-risk behaviors as a result of behavior change communication efforts (IR 3.2). HPAI remains widespread and endemic in Egypt: the number of its laboratory-confirmed human infections as reported to WHO has been both the highest of any nation in the world every year since 2009, and the total number of such patients is greater (N=117 cases) than all other nations combined (N=96 cases). This does not necessarily reflect badly on the government efforts supported by the USAID-funded program and other partners, since Egypt also has the highest density of poultry of any endemically infected country—the primary and most important predisposing factor—together with the factors shared by all endemically infected countries, including a diverse and poorly regulated value chain and inadequately resourced veterinary services. However, Egypt also has a well-established nationwide clinical and laboratory detection 10 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM system that makes it likely that most human cases will be clinically suspected, laboratory-tested, and reported to WHO. Conclusion: Given overwhelming constraints to effective disease control, only modest gains have been made to prevent and control H5N1 transmission, within poultry and to humans. ASSESSMENT OF MONITORING AND EVALUATION SYSTEMS Each of the six projects in the program had documented M&E arrangements. SAIDR had a detailed plan guided by the following principles:  Align goals and objectives with the context of the larger project.  Provide consistent, accurate feedback for program management, accountability, and advocacy purposes.  Build on existing monitoring mechanisms.  Monitor progress along the entire input-process-output-impact continuum.  Balance routine and longer-term data collection efforts. The plan identified constraints for project implementation, facilitating factors, and critical assumptions. The goal, subgoals, and IRs were clearly documented, together with tabulated inputs, processes, outputs, outcomes, and impact. A logical framework matrix showed indicators, measurement parameters, sources of information, and regularity of data analysis. A Gantt chart was developed each year (Year 3 chart examined) showing the timing of all activities for each IP. M&E plans for other projects were mainly in the form of a logical framework matrix with accompanying narrative. For some globally implemented projects, the final reporting of country￾specific progress that was available to the team was in summary form and difficult to fully evaluate. PMPs were available for each of the six projects in the API program except STOP-AI. FAO and GOVS had a dedicated M&E officer in their management teams to collate and document results. Biweekly meetings with the USAID Program Manager included M&E updates, and progress was also reviewed at quarterly meetings of the Supreme National Committee1 for Combating Avian Influenza. Backstopping by FAO headquarters technical supervisors tracked project progress. MOHP also had an M&E specialist. The MOHP has both manual and electronic databases, with good data quality and analyses. CHL also had a dedicated M&E specialist and databases with good quality data (personal communications of team leader with Douglas Storey, Ron Hess, and Gavin MacGregor-Skinner). Data analysis and specific assessment of CHL activities in Egypt on avian influenza 2006–09 is provided at the end of Annex D. 1 This coordinating committee was established in 2005 and now comprises the Ministers of Health, Agriculture, and Environment, representatives of the Ministries of Foreign Affairs, Interior, and Information, the Army and the Police, the governors of the seven worst affected governorates, and representatives of WHO and FAO (Government of Egypt, 2007a). USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 11 The SAIDR website home page can still be found (www.govs.gov.eg/saidr1) but not most of the supporting pages. Unfortunately, the “Search the database of printed materials” link no longer opened. Some links do still open, such as the “Biosecurity” poster and two videos on “How to protect yourself and your family” from influenza and “Flu prevention while traveling” (e.g., on the Hajj). “Partners” listed on the SAIDR homepage are MOHP, MOALR, FAO, USAID, CHL, and Stop AI. Gender analysis was performed where appropriate, such as in disaggregating data on training of animal health personnel, which showed a positive bias toward women, consistent with their role in interacting with female household members responsible for poultry husbandry. The substantive issue was discussed frequently in FGDs and other interviews with animal health personnel, particularly in relation to HPAI outbreak prevention and control (see below, Section 4.3.1; IR 1.4). Gender issues related to women traveling to areas for outbreak investigation and control were emphasized and must be taken into consideration for future planning and funding. The development of communication materials and methods also demonstrated a sound and appropriate focus on women in the household sector. Likewise, the MOHP data analyses on human cases included gender-specific H5N1 virus infection and mortality rates. For example, the majority of adults (59%) with H5N1 virus infection are women. Examination of quarterly and annual project reports and back-to-office reports of FAO backstopping officers showed that there was dynamic and effective monitoring of project progress, with adjustments made as required. GOVS also provided quarterly reports to USAID. FAO undertook evaluation of the impacts of training on biosecurity and PDSR (participatory disease surveillance and response). The PDSR training, which had been adopted from the Indonesian model, was modified to better suit Egyptian needs as a community animal health outreach (CAHO) program. In the area of BCC, there was limited documentation to trace how the communication program, including messages and interventions, was amended to reflect recommended new practices, although several examples could be recalled and were conveyed orally by CHL (team leader, personal communication by phone with Marwa Kamel, Ron Hess and Gavin Macgregor-Skinner, and Douglas Storey). During the course of the program FAO undertook two global real-time evaluation (RTE) exercises, in both of which Egypt was included as a special case. A UN Joint Assessment mission (UN, 2010) also reviewed progress in late 2009. A weakness in the M&E data analysis was the inability to measure HPAI incidence in a consistent way over time to assess progress toward the subgoal of preventing H5N1 infections in birds. This weakness was also identified in the UN Joint Assessment Report. Apparent changes in outbreak report numbers from year to year did not necessarily reflect a reduction in HPAI incidence. Particularly after the civil disturbances in 2011, a reduction in disease reporting was most likely due to reduced ability of field staff to perform their duties. This is a complex problem, at the heart of which is the overwhelming extent of HPAI incidence and distribution and the almost impossible task of trying to identify each outbreak. The second FAO RTE recognized a need to address the problem, perhaps by having more realistic indicators for impact. A sensitivity analysis of the CAHO program was designed so that the proportion of 12 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM outbreaks detected to all outbreaks occurring could be estimated. This analysis was underway at the time of the current evaluation. The second FAO RTE and the UN Joint Assessment reports also recognized the need to move away from emergency response to outbreaks to a longer-term risk reduction strategy. FAO and GOVS responded to this by drafting a revised strategy in 2010 that was approved by MOALR and resulted in a change in the direction of the project in its last year. Conclusion: There was considerable monitoring and evaluation of progress at the project level, with external reviews and evaluations providing a broader context. The difficulty of measuring impact at the subgoal level was a challenge yet to be fully addressed. PROGRESS TOWARD DESIRED OUTCOMES AND IMPACTS 1. Animal Health IR 1.1 Preparedness and panning The first objective of the SAIDR Project was to ensure use of the regularly updated National Integrated Plan (NIP) for design of project involvements where viral load reduction can be addressed. This indicates that even at the earliest stage of project planning, it was recognised that HPAI was entrenched and it was necessary to reduce the levels of virus circulating before any attempt at eradication could be contemplated. Appropriate activities were identified: assessing capacity, updating strategies, and focusing on improving communication. It was recognized within project planning that there was a serious lack of capacity and capability within the veterinary services, both at the national (GOVS) and governorate levels. This was a major focus of project activity that was addressed by providing within the program training in disease surveillance, outbreak response, epidemiology, laboratory diagnosis, biosecurity, vaccination, participatory disease search, coordination, and communication. There were 490 training events with a total of 10,038 participants,2 of whom 5,365 (53%) were women. The involvement of women was particularly important because they are the most important interlocutors for household poultry producers, who are mostly women. From discussion with staff at many levels, including in an FGD, it appears that the training was well-conducted and contributed significantly to increasing veterinary staff capability. Through cascade training, considerable numbers of staff were trained, but it must be recognized that with the very large number of government veterinary staff (about 10,000), it has been impossible to comprehensively upgrade veterinary personnel capability in all the areas addressed by the training initiatives. Also, only limited refresher training was possible. Highlights of training have been in laboratory diagnosis, for which capability is now excellent, and in participatory epidemiology within the CAHO program, one of the most important components for disease surveillance and engaging communities in collaborative approaches to disease control. Essential materials supplied by the SAIDR project have included laboratory equipment and reagents, computers and peripherals for epidemiology units, field sampling materials, and rapid diagnostic kits for use in the field. This was also an essential contribution to building veterinary service capability. 2 Many attended more than one training, so the total staff trained is a smaller number. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 13 Management and coordination mechanisms within the veterinary services are weak. While addressing the structural elements of these constraints was outside the scope of the program,3 SAIDR project implementation procedures were negotiated between the director of GOVS and the FAO Emergency Centre for Transboundary Animal Diseases (ECTAD) Unit to enable much of the field activity to be undertaken in the governorates, with inclusion of GOVS personnel. Although this was a necessary initiative, it did lead to some concerns expressed about the potential for exclusion of GOVS from some activities (possibly due to communication deficiencies within GOVS). Also, allowances given to government staff undertaking project￾supported activities (negotiated with and accepted by USAID) threatened to cause resentment among excluded personnel. The replacement of the SAIDR Program Management Unit at the end of the project with an Avian Influenza Emergency Management Unit within GOVS, at the recommendation of FAO, has provided a central focal point for coordination of efforts with other national units (such as NLQP) and with governorate veterinary staff. The strategy for project implementation was consistent with the National Integrated Plan. This gave priority to disease surveillance (including coordination between veterinary and public health surveillance activities) and HPAI outbreak control. However, there were many constraints to this approach and after two years it became clear that little progress was being made in controlling HPAI. After the program was reviewed by USAID and a UN Joint Assessment Mission, project personnel drafted a revised strategy that the MOALR approved in June 2010 (MOALR 2010). It emphasizes preventive measures, particularly improved biosecurity of poultry farms and measures along the market chain to limit virus spread and avoid human infection. These activities have much greater potential for reducing viral load and ultimately reducing HPAI incidence and the risk of human infection. Surveillance activities, including value-chain-based surveillance, were directed more to collecting epidemiological information to improve planning and decision-making. The Developing and Maintaining PPP project in Egypt aimed to encourage integrated partnerships between government and the private sector to support the poultry health and production systems most affected by HPAI. The main output was development of a compensation strategy through two workshops attended by public and private sector participants. With the change in strategy from emergency response to HPAI risk prevention, this became less important, since compulsory culling of poultry was greatly reduced. In addition, a series of biosecurity training events was conducted in all six districts of Fayoum governorate, complementary to those conducted elsewhere through the Improved Biosecurity and Hygiene Project. Conclusion IR 1.1: The redirection of strategic focus from emergency response to long￾term risk reduction is a more appropriate direction for managing the disease in an endemic situation. IR 1.2 Preventing future outbreaks Risk reduction became increasingly important to the program as its greater potential for HPAI control became accepted. Recognizing the need to demonstrate benefits to poultry owners, household studies were conducted (Fasina et al., n.d.) that demonstrated the cost-effectiveness 3 There is a global initiative for assisting to upgrade veterinary service structures and capacity by the Office International des Epizooties (OIE) using a Performance, Vision and Strategy Assessment followed by Gap Analysis. Egypt has been a beneficiary of these activities. 14 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM of basic biosecurity measures. For Sector 2 and 3 producers,4 a simplified guide to good practices was drafted (GOVS, 2010) to focus farmers on the most important elements of biosecurity. Nursery, duck, and Balady chicken farms represent the highest risk for spreading H5N1 virus (AbdelHakim, 2011). Farmers were trained through the Improved Biosecurity and Hygiene project, and evaluation of the training as a SAIDR activity (AbdelHakim and Jobre, 2011) demonstrated a statistically significant reduction in HPAI risk among trained compared with untrained farmers. The STOP AI Project focused on helping poultry producers to upgrade the biosecurity of their farms. Although this was a small-budget project, it provided high-quality local technical expertise that was effective in engendering modified practices, especially in relation to preventing introduction of H5N1 virus (bioexclusion). To gain consumer support, technical assistance targeted biosecurity enhancements as a means to increase farm profitability. The project also adopted PPP by engaging two producer associations to co-host biosecurity workshops, thus engendering a collaborative approach to addressing HPAI control. Poultry producers at an FDG were very positive about the benefits of the training they received through USAID-funded projects. Such discussions, verified by farm visits, indicated that significant improvements have been made in biosecurity, particularly in Sector 2 farms. The Improved Biosecurity and Hygiene Project aimed to improve farm-gate biosecurity (barrier control) on selected commercial poultry farms in seven governorates. While initially the plan was to decontaminate live bird markets, because of the government policy to close these markets the focus was changed to farms. A comprehensive approach trained farm workers and addressed not only disinfection activities but also other barrier controls, such as structural needs. A total of 2,732 long- and short-cycle farms were selected and decontamination exercises undertaken. The work was assisted by workshops, posters, and printed guidelines, which were also used to extend the knowledge to other farms. A later assessment found that 65% of targeted farms showed good or moderate improvement in farm gate biosecurity, while the remainder showed no significant improvement (AdbelHakim & Jobre 2011). Any reduction in HPAI incidence as a result of improved biosecurity practices has not been measured. The reluctance of farmers to report suspected HPAI and a post-revolution increase in unregistered Sector 3 farms5 are among the reasons why it is not possible to measure changes in HPAI incidence or attribute them to particular interventions. GOVS has little influence on vaccination within the commercial poultry sector except in licensing imported vaccines. However, the government has sponsored vaccination within the household poultry sector. Poor acceptance of vaccination (early vaccination activities, outside of the program, probably contributed to the spread of infection), inability of vaccination staff to obtain good coverage, and poor monitoring of vaccination responses rendered Sector 4 4 FAO has a widely accepted means of categorizing poultry production on the basis of biosecurity measures, from Sector 1 (fully integrated industrial production with effective isolation of flocks) through Sectors 2 and 3 (commercial farms with less ability to apply full biosecurity measures) to Sector 4 (household or backyard production in which only minimal measures can be applied). 5 It was reported that about 80% of poultry farms are not registered. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 15 vaccination inefficient and unreliable, and probably just masked the presence of disease. Project personnel successfully encouraged government to end these vaccinations. The recent launch of a new recombinant vaccine offers the prospect of vaccinating day-old chickens and ducks with a single dose, to provide long-lasting and cost-effective immunity efficiently. The potential benefits of widespread use of this vaccine warrant its consideration in formulating future vaccination policy. A continuing challenge is to understand the factors contributing to dissemination of the virus along the value chain6 in order to design preventive measures that are both effective and acceptable to industry participants and consumers. Studies conducted within the project covered assessment of the household producer value chain (Geerlings, 2011) and identification of risk factors in the value chain (AbdelHakim and Jobre, 2011; van Engelen, 2011; Wilsmore, 2008). The virus is probably maintained in household flocks by carriage in ducks and spread over short distances by local market activity. Spread from infected commercial farms is more likely to be over longer distances. Live bird markets, where poultry of different species, of different ages, from different sources, are destined for either slaughter or relocation to another production facility, represent a critical control point. Though officially banned in cities, because they are vital to livelihoods they will probably continue. The value chains are so complex and diverse that local solutions are probably required on the basis of the differing magnitude of the various risk factors. More detailed analysis is needed for appropriate interventions to be designed. Conclusion IR 1.2: Animal health and BCC professionals need to design risk reduction approaches that are both acceptable and effective, and their impact on the incidence of HPAI should be measurable. IR 1.3 Detection of H5N1 in poultry and wild birds The first detected occurrence of HPAI in Egypt was at a time (early 2006) when H5N1 was being detected along migratory flyways in northern and central Europe in dead water birds, migratory and resident. Though it is thought that the virus was introduced into Egypt by migratory birds, this is speculative. In any event, once HPAI became established in Egypt, any spread by wild birds was of little relevance because there were ample alternative ways for it to spread. Poultry populations have been monitored for outbreak detection and epidemiological investigations. Outbreak detection consists of reports of disease from owners or others in local communities (passive surveillance) and by purposive disease search (active surveillance). Passive reporting has been minimal; people were reluctant to report disease when a positive diagnosis of HPAI meant their flock would be culled. Nevertheless, during 2010 passive reporting accounted for 69% of outbreak reports in a study area (Wilsmore, undated), which may reflect minimal penetration of active surveillance. The most successful approach for the latter has been the CAHO program, where teams combined disease search with engagement of the community in communicating advice for prevention of HPAI and human H5N1 virus infection. CAHO personnel were women, since they were mostly interacting with female poultry tenders. The program trained 108 veterinarians who were deployed in 53 districts (30% of all districts in 6 The value chain comprises the supply chain of inputs (chicks, feed) and the market chain of product (poultry and eggs) going to and through markets to other producers or for consumption. 16 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Egypt) in 15 medium- and high-risk governorates. In 2010 through their efforts 185 outbreaks of HPAI were detected, representing 22% of all outbreaks in those districts. While it is impossible to judge the sensitivity of outbreak detection,7 it is almost certain that only a very small percentage of outbreaks were detected. With poultry owners reluctant to report and travel difficulties for personnel in the field, it is almost certain that most outbreaks were not detected. Infected ducks are difficult to identify clinically, and suboptimal vaccination masked clinical disease in chickens. Even when surveillance resulted from a human being infected, detection of a poultry outbreak was rare. In 2009, 78 investigations were undertaken following human cases and only in three was HPAI detected in poultry (Wilsmore, 2008). The National Laboratory for Quality Control of Poultry Production (NLQP), comprising a central and four satellite laboratories, provided diagnostic services for HPAI, and SAIDR resources helped to provide the laboratory with disposable items. The testing was of high quality and the quality control standards were good, including collaboration with international reference laboratories through the OFFLU8 network. The AIVEP Project was designed to identify circulating H5N1 viruses to select candidate vaccine strains and evaluate the efficacy of current vaccines against field strains. Staff of the NLQP and the Central Laboratory for Evaluation of Vaccines built up effective capability to undertake this work with training in their own central laboratory and at the USDA Southeast Poultry Research Laboratory in Athens, Georgia. Gene sequencing of the H5 gene of 102 field isolates indicated that they were all of clade 2.2.1, but late in 2007 a variant, clade 2.2.1.1, arose mainly within commercial poultry farms. It later disappeared again. The classic clade 2.2.1 virus was mainly isolated from the household sector. The fact that the variant clade 2.2.1.1 was mostly identified on commercial farms suggests that it emerged by the selection pressure of vaccination (Swayne, 2011). Antigenic cartography was used to select strains for challenge tests to evaluate vaccines in use. It was determined that vaccines produced using classic vaccine strains protect against all classic field strains, although some variant strains were resistant. Genetic and antigenic data and methodology developed with AIVEP Project support was shared through the OFFLU network and contributed to global intelligence of H5N1 virus evolution. A need was identified for closer linkages between national laboratories (GOVS, MOHP, universities, and private laboratories) in order to collate diagnostic, epidemiologic, and virus characterization data. This would allow faster identification of emerging virus variants. During the course of the project increasing attention was given to epidemio-surveillance— undertaking surveillance primarily to increase understanding of how the H5N1 virus is maintained and spread in different production sectors and along the value chain. Live bird markets were an obvious target, being the point of congregation along the market chain. In one study, 2,000 samples were collected from 225 markets throughout the country. H5N1 virus was detected in 109 (5.5%) of the samples, and 94% of the sampled birds came from households (Wilsmore, 2008). 7 A post-project analysis of CAHO surveillance in currently underway to determine the sensitivity of outbreak detection. 8 OFFLU is the OIE/FAO network of expertise on avian influenza, substantially funded by USAID. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 17 The project has vigorously supported building up epidemiology by setting up a central epidemiology unit within GOVS and units in each governorate and district. Using project funds, each unit has been equipped with computers and TADinfo9 software, and staff has been trained in use of the software. Unfortunately, staff in decentralized units do no data analysis; they simply record information. In the case of at least one district unit (in Fayoum) they use TADinfo for a variety of diseases but not for HPAI. The recording of outbreak investigation data is poor, and it was only possible to obtain basic information, such as the number of outbreaks reported each year, disaggregated into commercial farm and household. With uncertainty about the uniformity of surveillance activity, and particularly with disruption to government services, including HPAI surveillance, after the 2011 revolution, it is unrealistic to attempt to draw any conclusions from this information. Conclusion IR 1.3: Targeted surveillance for epidemiological monitoring meets an essential need for planning risk reduction measures. IR 1.4 Improve outbreak containment measures Outbreak containment has been one of the most challenging issues for the program, for a number of reasons. 1. From a livestock disease perspective, compulsory slaughter is not generally appropriate in a situation where disease is widespread and at high incidence, and especially when it is apparent that the disease is not well-reported. The justification for HPAI in the current endemic situation in Egypt was therefore only to prevent human exposure to infection. The likelihood that human infection would be mitigated by stamping out only the small proportion of HPAI cases that are detected is minimal. 2. Whenever compulsory culling is undertaken as a public good, compensation should be paid for private loss. But if the culling is extensive, compensation is unaffordable, as has been demonstrated in Egypt, where funds were rapidly exhausted. Also, administration of a large compensation scheme with proper accounting and equitable application is extremely difficult. FAO has contributed to proposals for a suitable compensation scheme but the hurdles have proven insurmountable. 3. Knowing that they will have to bear a loss, poultry owners have been reluctant to report suspected HPAI. Even where there is a suspected or confirmed case of human infection, to avoid the consequences, including social stigmatization within their community, the affected household will often hide or destroy its poultry before an investigation can begin. 4. Household poultry production provides a major source of animal protein for many Egyptian families, and HPAI and other poultry diseases threaten this valuable food resource. Although well-intentioned, some emergency outbreak control measures (such as culling apparently healthy poultry in dangerous contact with infected flocks) may cause more harm than the disease itself. That harm may have a particularly high impact on low-income families, especially single-parent families with a female head of household. Conclusion IR 1.4: HPAI containment by compulsory culling without compensation did little to prevent human exposure to H5N1 virus. In planning control measures, there is a need for consideration of food security and an awareness of gender roles in household poultry production. 9 TADinfo is an FAO-developed software package for recording, storage, transmission, and analysis of data for transboundary animal diseases. 18 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM IR 1.5 Limiting exposure of personnel to H5N1 SAIDR project funding provided large numbers of disposable plastic clothing, N-95 masks, and gloves. The USAID DELIVER Project also provided PPE and disinfectant supplies and assisted with logistics for storing and distributing supplies and training personnel in safe packaging of infectious materials for transport. There was little documentation available to assess the DELIVER Project contribution, but one record showed an order for 3,000 PPE kits, 746 decontamination kits, 492 5-gallon pails of Virkon disinfectant, and 100 pairs of gloves. Personnel indicated that supplies of these materials were adequate. Conclusion IR 1.5 The program was successful in providing adequate protection to personnel engaged in activities that might expose them to H5N1 infection. 2. Human Health Component Of the six projects in the USAID/Egypt API program, the primary one with a human health component was the SAIDR project (October 1, 2007, to Sept 30, 2010) implemented by the MOHP. USAID staff managed the implementation letters for this part of the project. The primary goal of the SAIDR project was to prevent H5N1 infections in humans. The subgoal was to prevent human exposure to the H5N1 virus. There were four intermediate, or outcome￾level, results: IR2.1, IR2.2, IR 2.3, and IR 2.4. IR 2.1 Improve preparedness and planning for a human influenza pandemic SAIDR efforts to prepare for pandemic influenza appropriately involved a broad spectrum of technical approaches. For example, the initial focus was on reducing bird-to-human as well as person-to-person transmission, with poultry viewed as the primary source of human infection. Nevertheless, with USAID support the 139-page Egyptian National Preparedness Plan for Pandemic Influenza, which was finalized in 2008 and used for simulations and planning exercises, appropriately emphasized human-to-human transmission. Fortuitously, this national plan was already being used for training purposes in 2009 when the unexpected pandemic of influenza A/H1N1 began. In addition, the program helped to provide well-trained epidemiologists for the MOHP through the Field Epidemiology Training Program. By the end of the SAIDR project the MOHP estimated that 20 epidemiologists, also trained in medical statistics, had graduated. Enhanced training in pandemic influenza was also provided for pre-hospital ambulance transport workers; for nurses and physicians, to improve clinical management of patients with influenza; and for persons involved with isolation and quarantine of patients. Primary health care (PHC) workers were similarly trained across the country, and the MOHP created a specific manual for Primary Health Care Units, “Micro-Planning Plan of Primary Health Care Units for Combating Pandemic Influenza.” Again, like the national preparedness plan, this PHC plan was completed before the onset of the H1N1 pandemic in 2008 and thus was available for immediate use. Conclusion IR 2.1: The Avian and Pandemic Influenza Program succeeded in enhancing pandemic preparedness and response starting in 2007–09, before the first influenza pandemic in 41 years suddenly began (due not to H5N1 but to an unanticipated virus). IR 2.2 Improve detection of suspect H5N1 cases Among the many objectives addressed here were enhanced human surveillance for H5N1 infection, with tracing and follow-up of each confirmed case using the National Egyptian Disease USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 19 Surveillance system. Computer training and information technology support was provided for staff involved in surveillance across the country, and district rapid response teams were created and trained that could deploy immediately to investigate and implement control measures when a new case was diagnosed. Notably, the laboratory diagnostic capacity for suspected H5N1 cases was increased by opening new regional laboratories, for example in Alexandria and Menia Governorates, so that testing could be performed quickly because patient samples did not have to be transported to Cairo, and clinical care decisions could be made rapidly. At the same time, the capacity of the CPHLin Cairo was increased to diagnose H5N1 virus infection and later the pandemic H1N1 virus. The CPHL reported that it began testing for H5N1 virus in humans on January 19, 2006. According to the MOHP, the number of isolated suspect cases of H5N1 was 1,991 in 2006, 1,829 in 2007, 1,907 in 2008, 5,583 in 2009, 622 in 2010, 445 in 2011, and 324 so far in 2012, for a total of 12,571 isolated suspect cases. Of these 168 were confirmed. Since the H1N1 pandemic began in 2009 the CPHL has also identified 10,201 samples as positive for H1N1 influenza. Conclusion IR 2.2: Enhanced laboratory diagnosis and surveillance systems for H5N1 influenza were established and available for the unanticipated A/H1N1 pandemic in 2009. IR 2.3 Improve containment measures in human populations Specific training courses were provided across the nation for intensive care unit management of acute respiratory distress syndrome (ARDS). As part of the enhanced capability for the care of patients such as those infected with the influenza virus, physicians and nurses were also trained on mechanical ventilation for patients with respiratory failure. The MOHP also contributed to increased AI awareness with seminars and educational messaging at schools and universities in order to mitigate transmission in the community. Conclusion IR 2.3: Ability to provide high-quality care for patients needing intensive care, including mechanical ventilation, was enhanced. IR 2.4 Limit exposure of health care staff Influenza infection prevention and control practices were improved at hospitals at all levels, from central to governorate to district. This extensive effort was accomplished by holding training courses across the country. The MOHP Epidemiology and Disease Surveillance Unit stated in its SAIDR Completion Report that 6,500 units of PPE sets were made available. The 2008 Preparedness Plan for Pandemic Influenza included a chapter on infection control procedures that dealt with how to prepare and use disinfectant, guidance for the Emergency Department, optimal design of patient isolation rooms, other hospital rooms, and precautions for specific high-risk procedures like intubation and respiratory suction for laboratory staff safety when handling patient samples and mortuary services. Conclusion IR 2.4: The objective of providing infection control and prevention measures was achieved both for H5N1 and pandemic H1N1 influenza. 20 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM The API program contained an appropriate mixture of technical interventions to achieve the largest beneficial impact on human health. For example, in Egypt the H5N1 virus case fatality rate for young persons under 15 is 6% (5 deaths/83 total cases) compared with 76% in Indonesia (data based on an initial 116 patients, of whom 33% were under 15; Sedyaningsih et al., 2008). Even more strikingly, the single regional referral hospital in Cairo for children 6 and under who had laboratory-confirmed H5N1 virus infection reported zero fatalities in 51 children from 2006 through 2012. Notably, the multidisciplinary team caring for these patients, including those requiring intensive care and mechanical ventilation, reported giving five daily treatments with commercially available intravenous immune globulin (IVIG) in addition to the standard antiviral drug oseltamivir (Tamiflu) to the 21 clinically “sickest” children (personal communication to team leader during hospital site visit). The dramatic 100% cure rate for what might be called “The Egyptian Model” of treating children with H5N1 virus infection should be emphasized globally, and the possible role of IVIG analyzed to see how it might have contributed to the zero fatality rate. Another example is the high level of preparedness for the 2009 pandemic of influenza A/H1N1 based largely on avian and pandemic preparedness from 2006–09. This nationwide effort was epitomized by the detailed guidance provided in the 2008 Egyptian Preparedness Plan for Pandemic Influenza and the national response when the pandemic began in 2009. As one outcome measure the MOHP has analyzed by gender the number of and the fatality rates for human H5N1 infections. Of the 168 laboratory-confirmed human cases of H5N1 infection in Egypt from March 2006 to November 2012 the majority (59%) were women. Analyzed by age, 55% of under-15 cases were males, but 73% (62 cases) of over-15 cases were females, The fatality rates were similar for both sexes, although slightly higher in females, 66% vs. 61% for those over 15, and 8% vs. 4% (but based on very few deaths) for those under 15. The API program appeared to be responsive to stakeholder needs and was valued highly by MOHP counterparts. For example, when the unanticipated influenza A/H1N1 pandemic began in the spring of 2009, USAID approved the reallocation of $400,000 to help MOHP respond immediately to the first influenza pandemic in the 41 years since 1968. However, when the evaluation team met with the MOHP, government staff expressed a strong preference for being made aware of all NGO efforts, whether centrally or at the governorate or district level, related to influenza preparedness and response in order to optimally coordinate such efforts. During that initial meeting the MOHP also provided the evaluation team with a two-page letter that identified five areas in which USAID had previously supported MOHP via the API program and a similar five areas where future support would be appreciated to sustain what has been achieved in terms of avian and pandemic influenza preparedness and response. The five and five future areas generally overlap: 1. Surveillance: past identification of human H5N1 cases and rapid clinical management, now looking forward to improved surveillance systems (online reporting) across the country down to the district level, to cover influenza-like and severe acute respiratory illness programs (for both there are currently eight sentinel sites across the country) 2. Animal and human health: past collaboration in both the field and the laboratory, now looking forward to multi-sectoral coordination between epidemiology and laboratory in both human and animal health, as has developed since the spring of 2011 in Egypt as “4-way linking” (also with WHO and FAO participants in Cairo) USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 21 3. Laboratory capacity-building: past efforts to provide H5N1 diagnostic testing quickly, both in the governorates and the CPHL, now looking forward to increasing the capacity of the CPHL in Cairo to include a biosecure laboratory and gene sequencing (“to help decrease the overload on reference lab [NAMRU-3]”) 4. Rapid response and field epidemiology: teams working in the past across the country, now looking forward to enhancing the quality and capabilities of the nation-wide rapid response and field epidemiology teams 5. Infection control and prevention: past achievements include minimizing risk of exposure and infection by H5N1 and H1N1 influenza viruses; now looking forward to infection prevention and control of influenza and other respiratory pathogens. An example of the latter is the recent “novel coronavirus” first reported September 2012 in Jeddah, Kingdom of Saudi Arabia (KSA) and now in eight other patients from KSA, Qatar, and Jordan. Egypt recently obtained throat swabs from 750 pilgrims returning from the Hajj over the past month to test for this “SARS-family” coronavirus. 4.3.3 BCC component IR3.1 Improved awareness of avian and pandemic influenza SAIDR communication activities were managed by Johns Hopkins University’s Center for Communication Programs’ (JHU/CCP) Communication for Healthy Living (CHL) program that covers a wide array of public health issues other than avian influenza. During the 2005–06 H5N1 outbreak in Egypt, USAID added avian influenza to CHL’s portfolio to help the MOHP and MOI/SIS respond to the emergency. CHL was the chair of the multipartner (donor and GOE ministries) National AI Communication Committee and the entity that designed the National AI Communication Plan. Based on a unifying communication platform there was heavy emphasis on national mass media (radio and television ads), edu-tainment (TV shows), and print materials along with community and district interpersonal communication (IPC). CHL provided master trainings (TOT) for private pharmacists, educators and their supervisors (in association with the MOE), and the MOHP Raeedat Refiat, community outreach workers who operate at national, district, and unit levels. All were trained in communication techniques and key messages on H5N1 virus, how to recognize symptoms, and the importance of reporting. The community workers were supplied with materials developed by CHL for the project, such as JobAIDS, flip charts, and other ephemera. This training laid the groundwork for community BCC activities. At the central level CHL worked with GOVS and FAO to conduct TOT in communication for rapid response teams. CHL also conducted H5N1 information workshops for religious leaders to introduce behavior messages into religious homilies and Friday sermons. They conducted media training for central and governorate spokespeople in the Veterinary Services as part of preparation for outbreak situations. CHL tapped into its network of family planning partners— local and international NGOs—to distribute print materials and deliver messages on the H5N1 virus, and as needed CHL provided TA to GOVS and STOP AI on material development. CHL helped set up and manage a SAIDR website that was transferred to partners in Egypt at the end of the project. The home page is uplinked (www.govs.gov.eg/saidr1) but not most of the supporting pages. Some links do still open, such as the “STOP AI Biosecurity” poster at the upper right side, and two videos on “How to Protect Yourself and Your Family” from influenza and “Flu Prevention While Traveling” (e.g., on the Hajj). Unfortunately, the “Search the database 22 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM of printed materials” link no longer opens. “Partners” listed on this SAIDR homepage are MOHP, MOALR, FAO, USAID, CHL, and Stop AI, while “Allies” include UNICEF, UNDP, Red Crescent, and Save the Children. As part of the original public awareness campaign for a limited time CHL introduced SMS and texting with MobiNil. UNICEF was a member of the National AI Communication Committee and engaged in community- and national communication outreach. It also trained agriculture extension workers, community workers, educators, and NGO partners working with MOHP, MOALR, and MOE. In keeping with its mandate, UNICEF produced communication and education materials that emphasized mothers and children. UNICEF was not funded by USAID. Under separate funding MOHP created a TV ad that starred a well-known Egyptian singer that aired on satellite and local stations. CHL conducted a knowledge, attitudes, and practices (KAP) research survey, the Egypt Health Communication Survey (EHCS), that covered all its public health programs. The avian influenza communication campaigns were part of the Egypt Health Communication Surveys in 2006 and 2008. UNICEF conducted a KAP in 2007 that also captured awareness. GOVS fielded two KAP surveys in October 2009—the last M&E for the SAIDR project. In surveying household poultry producers, the EHCS found high awareness of AI, particularly in 2006, which was at the peak of the H5N1 emergency response; awareness then leveled off to about 65% quarterly (EHCS 2006 and 2008 and Hess undated). It is normal for an emergency public awareness campaign to have high peak awareness and then to stabilize. The later surveys reflect flu fatigue and flu complacency. Concern about the severity of the disease went down from 25% in 2006 to 21% in 2007 among women household poultry farmers, but their confidence in their ability to deal with it went up from 33% to 43%. The ECHS 2008 findings among household poultry farmers (urban and rural) are also insightful for future programs and program budgets. Recall of print materials was 24%, and the materials were most often seen at health clinics. Community- meetings on avian influenza had 4% attendance in the first year (2006) and then went up to 9% (2008). Conclusion IR 3.1 Public awareness of H5N1 peaked during the response to the emergency in 2006 and 2007. IR3.2 Decrease high-risk behaviors associated with transmission of H5N1 among birds and humans Working with community outreach workers on both the human and animal health sides was designed to move household poultry producers from simple public awareness to adopting best practices in animal husbandry. STOP AI, which was part of the API program for 16 months, focused on macro changes by engaging the private sector in H5N1 work and beginning advocacy activities to determine regulatory changes that would benefit the poultry industry and minimize H5N1 transmission. Household poultry farmers are mainly women and their daughters. Village and community mobilization worked through Village Health Committees, Raeedat Refiat, and Arab Women Speak Out (AWSO), a CHL initiative that has been successful with other public health issues. Extension workers, veterinarians, and health care workers, including pharmacists, many of whom are women, made up the network of experts to support behavior change. The National Program Coordinator for MOHP Community Outreach spoke very highly of the AWSO and its positive results for helping village women learn decision-making and income generation relevant to avian influenza. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 23 On the animal health side FAO and GOVS created the CAHO program, which was comprised of veterinarians specially trained in interpersonal communication to complement their animal health expertise. There were also rapid response teams consisting of an epidemiologist, a communication expert, and a culling expert. They too were trained in interpersonal communication techniques. CAHO repeatedly received high praise for its work in communities, but its staff expressed frustration that because of the culling practices and the evil eye tradition, household producers often blocked them from gaining entry or seeing the poultry.10 FGDs with representatives from response teams and CAHO requested new training to help them deal with these situations and “practical” training with “real-life” applications. CAHO communication training was provided by ILRI. CAHO was also proactive in established its own Facebook page and used SMS and texting to share work information, ask questions, and update information. Government policies in the early days of the H5N1 virus outbreak (culling and no compensation) are one reason households and small farmers hide their flocks and generally distrust government representatives like CAHO and Raeedat Refiat. In interviews with representatives from the different projects and government ministries and a review of annual reports from the three SAIDR components, it appears that MOHP was very pleased with the partnership and the technical support from CHL. An MOHP report on avian influenza (MOHP, undated) itemizes communication activities, including 17 TV and radio ads, print materials and their distribution, and communication training. In pilot communities where the communications was implemented, there was acceptance of correct practices related to separation of poultry and hand washing and sanitation during the emergency and post-emergency period (ECHS 2006 and 2007, Hess PPT). The pilot communities had a high concentration of effort to mobilize for HPAI. Grassroots activities to create behavior change are easier to achieve when supported by regulatory or policy changes. Advocacy efforts to effect policy changes that would lead to the larger and more lasting behavior change were limited. STOP AI, however, did propose advocacy activities and had taken initial steps by holding thought-leader workshops and engaging the commercial and private sector in meetings to help to create the enabling environment that would lead to addressing larger regulatory and policy issues. GOVS and STOP AI indicated that when they asked CHL for technical assistance with materials and training, they received it. From central to district levels, however, GOVS felt that the BCC component was a gap in the project and needed to be strengthened. GOVS stressed how much it wanted assistance in behavior change. (The GOVS KAP of October 2009 substantiates that at￾risk audiences are adopting best practices, but only slowly.) In the first meeting with GOVS the discussion kept returning to the need for BCC, advocacy, and cross-ministry communication, as well as communication from the central to governorate to district levels. The general impression was that GOVS would have liked more communication assistance, especially for behavior change. There also was a lost opportunity among the partners in terms of what CHL or communication could contribute to planning. For example, the GOVS Year II Report (Summary of Activities SAIDR Project September 2008–October 2009) describes planning meetings 10 Evil eye is a common problem for community health workers. A simplistic explanation is that “seeing is killing”: if your neighbors see how much livestock you have they will curse it, and the animals will die, reducing your income and well-being. In a village in Gharbeya when household producers were asked how many chickens they had, they laughed and said 5. CAHO vets said most have around 50. 24 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM on the poultry vaccination program and simulation and outbreak response planning. Attendees included FAO, GOVS, STOP AI, and NQLC, but not CHL or a communication technical lead. The explanation for this omission was that this was a technical meeting. There was typical push and pull between technical and communications experts on messages. Animal health experts worried that there was too much emphasis on caging and more sophisticated bio-security methods, and less attention to promoting realistic and affordable biosecurity practices. In addition to what UNICEF and CHL were doing, there were governorate communication activities that the MOHP was not aware of. In Egypt there were numerous communication activities funded from different sources, and lack of coordination led to duplication (mass media, education materials and community outreach materials); multiple and not always consistent messages; and lower-quality work. Before the H5N1 outbreak a series of national technical committees were established, and at the onset of the outbreak a National Avian Influenza Communication Committee was established that was chaired by CHL. Committee members were government ministries, UN agencies, including UNICEF and FAO, and USAID. The representatives were not always practitioners, and there were no representatives from civil society or the private sector. Nor was there formative research conducted, which would have given insight into target audiences and segmented household producers and other stakeholders by behaviors as well as setting a baseline to measure behavior change. The MOHP was proud of its communication work, done with technical assistance from CHL. There has been long-standing collaboration between JHU/CCP and MOHP and MOI/SIS. The animal health experts were more reserved. GOVS and STOP AI representatives said that when assistance was requested, CHL provided it. GOVS felt behavior change was missing, and behavior change was critical to preventing and managing the virus. FAO indicated collaboration was weak. There are several documented examples where CHL was not in attendance at animal health planning meetings for community outreach and the animal vaccine campaign; these activities would have benefited from communication TA. A review of training attendee documents shows gender splits based on the profession being trained. Samples from Using the GOV Annual Report for Year 2 (2008) show that spokesperson training had 15 men and 1 woman; a TOT for administrative staff had 3 women and 20 men; a Step-down training was all women (276). There are examples where the split was 50–50 and others where it is one-third women to two-thirds men. The API program has been most effective in mobilizing stakeholders and building the networks and platforms necessary to prevent H5N1 in humans. Future endeavors to create the necessary behavior change should bring in more representatives from civil society, the private sector, and education, especially medical and professional schools and universities. Expanding to include more national institutions will contribute to sustainability as well as institutionalize wide behavior change by teaching and standardizing correct practices from grassroots to governing institutions. These objectives—sustainability and building the capacity of national institutions, organizations, and businesses—follow closely the USAID Forward agenda of building national capacity, M&E, sustainability, innovation, and women-focused activities (http://www.usaid.gov/results-and-data/progress-data/usaid-forward). Conclusion IR 3.2: Behavior change for maximum affect should be institutionalized through policy and regulatory actions and in schools, universities, and businesses. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 25 ASSESSMENT OF MANAGEMENT AND ADMINISTRATION ARRANGEMENTS A management structure was established for the program with USAID/Egypt managing the SAIDR project. For the animal health component there were four successive program managers, two of whom were very actively engaged in project activities, which was appreciated by partners and led to USAID/Egypt having a very good understanding of the challenges the program faced. Animal health and the communication component had separate program officers, which led them to function as independent entities. While the global projects, Deliver/AI and STOP AI, were managed from USAID/Washington and FAO headquarters, the FAO ECTAD team leader also had direct contact with USAID/Washington staff to negotiate country issues as the operational environment evolved. Coordination meetings were conducted quarterly with MOH, MOALR GOVS and NLQP, FAO, STOP AI, and CHL to encourage collaboration. They were only partly successful. Coordination between MOH and MOALR was not always optimal. For example, reportedly that there were often delays between identification of a suspected human case of H5N1 infection and transmission of this information to GOVS, resulting in delays in investigating associated poultry disease. These appeared to be institutional and operational issues that were outside the scope of program management. While they might have been addressed on the government side through the National Supreme Committee, this body ceased operations early in 2009. FAO took a programmatic approach to ensure that the four projects it implemented in collaboration with GOVS, NLQP and individual governorates were complementary. For example, two projects (Improved biosecurity and hygiene and Developing and maintaining PPE, both global), were modified to meet Egypt’s priority needs in biosecurity training and implemented in different governorates. STOP AI biosecurity work was also complementary, addressing farmers’ needs in Sectors 1 and 2 while the FAO focused on Sectors 3 and 4. STOP AI produced a biosecurity CD that was available to the program for training purposes. FAO formed an ECTAD country team with an international technical team leader and operations manager and national consultants for technical and administrative support, and GOVS formed a Program Management Unit (PMU), which included personnel for management, administration, finance, monitoring and evaluation, surveillance, training and communication. When the program’s activities ended, the PMU was replaced by an Avian Influenza Emergency Management Unit, which is still operational. Budget allocations for the SAIDR Project were negotiated with USAID so that FAO and GOVS received separate allocations for the first three years. FAO financial reporting was through FAO headquarters to USAID/Washington. GOVS reported to the USAID Mission and had satisfactory audits for the three annual budget allocations. Delays in provision of funds (six months in Year 1, four months in Year 2, two months in Year 3) delayed GOVS activities. FAO managed the delays by advancing funds from other sources. Management and coordination mechanisms within the veterinary services are minimal. While addressing their structural elements was outside the scope of the program,11 some project 11 The Office International des Epizooties (OIE) has a global initiative for upgrading veterinary service structures and capacity using a Performance, Vision and Strategy Assessment followed by gap analysis. Egypt has been a beneficiary of these activities. 26 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM procedures were negotiated directly between FAO and individual governorates, with the approval of the GOVs Director and with GOVS personnel part of project activities. This is a departure from standard FAO policy, which is to partner with the national government. However, it was an appropriate way to achieve operational functionality within the current capacity and organization of the veterinary services. Early in the program, slow delivery of inputs through FAO compromised program implementation, and short-term consultancies appeared to lack coordination. This was completely changed with the appointment of the current ECTAD management team, which has demonstrated excellent communication with government counterparts and partners, well￾coordinated technical guidance, and efficient control of operations, including impressive program documentation. CHL implemented the communications component of the SAIDR Project, which, because of the way the project was funded, did not lend itself to natural or easy coordination across the program. JHU/CCP, which managed CHL, has worked for many years with MOH and MOI/SIS providing TA on a wide variety of public health communication issues. Avian influenza was one of several issues in the CHL portfolio and the general sense from all stakeholders was that it was understaffed to take on the H5N1 and eventually the H1N1 portfolio and thus underperformed. Collaboration with FAO appears to be weak. Reports from GOVS and STOP AI were that when asked, CHL would provide TA to projects. As is typical in message development, there was a push and pull between technical and communication experts on messages and promotion of techniques, such as caging, which was considered a sophisticated and expensive practice and not the only behavior to be promoted. Conclusion: The program management and coordination structure was not sufficient to overcome weaknesses in collaboration between government departments. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 27 V. CHALLENGES THE POLITICAL ENVIRONMENT Almost two years after a major change in government, Egypt is still in a period of transition. This has been manifest by multiple changes in leadership at the ministerial, governorate, and department levels in key government areas that would be involved in future programs. Linkages between government departments may have weakened during this transitional period, demanding that any new program have a strong coordinating function. There are indications that a new-found freedom within sections of the community has led to people being less likely to follow official directives, which may have undermined compliance with some regulations. It could also compromise attempts to introduce tighter controls on farms and along the value chain as disease risk reduction measures. COMMUNITY PERCEPTIONS FOLLOWING THE H5N1 AND H1N1 EMERGENCIES The global concern for both H5N1 and H1N1 viruses was an influenza pandemic with high mortality. In neither case did that occur. In Egyptian communities, there is a feeling of disillusionment and distrust; from the human health perspective, H5N1 infection risk was lower than anticipated and for H1N1, the severity of disease was much less than expected. Even from the perspective of the disease in poultry, which is still widespread and highly fatal and likely still the greatest poultry disease concern in Egypt, the behavior and perceptions of people in the household sector were often dismissive. The team heard reports of people hiding their poultry to avoid culling, even when there was human illness in their household, and dismissing H5N1 as not a real disease. It will be a significant but critical challenge to alter such perceptions. Even the poultry disease alone is a significant threat to food security, especially the role of women as the main contributors to poultry husbandry in the household sector. It will call for rigorous assessment of prevailing community attitudes at the start of the program and careful planning to craft an appropriate communication strategy. THE LONG-TERM RISK-REDUCTION APPROACH The process of reducing HPAI to the level where it no longer represents a significant threat to human health will be protracted, probably extending over a decade or more. A shorter program will at best deliver only progressive improvement. It is a more sophisticated approach than stamping out individual outbreaks, more demanding in research and planning, and requiring far more cooperation between different sectors of the community. The change of approach must be transparently explained and negotiated with all it might affect. In particular, high-risk practices, such as raising ducks and marketing live birds, will be targets for change in practice, and the societal and economic implications of making such changes will need to be carefully considered. THE IMPLICATIONS OF PARTIAL SUCCESS Over time, large commercial producers will be progressively less affected by HPAI. They will improve their biosecurity so that outbreaks in Sectors 1 and 2 are rare and new vaccination 28 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM technology may provide them with high levels of flock protection so that the impact of outbreaks is limited. These trends may reduce HPAI in all sectors, allowing the goal to be reached of preventing human infection. However, it is also possible that HPAI could persist in the household and small commercial sectors, thus impacting those who are most food insecure but reducing political will to maintain efforts to eliminate the disease. THE PROSPECT OF A HIGH-MORTALITY HUMAN PANDEMIC As laboratory experiments have demonstrated, it is possible that an influenza virus may emerge that is a mutant of H5N1 or a reassortant with genes contributed by more than one virus subtype that could result in the feared global event that has prompted international concern for the past 10 years. Will Egypt be prepared for it? It is worth giving the prospect serious consideration and taking into account all possible lessons learned from past events in preparing for the future. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 29 VI. LESSONS LEARNED AND RECOMMENDATIONS Management of the API program was difficult because of the emergency nature of project implementation, poor management and coordination between public authorities, and fragmentation of communication efforts between implementing partners, within and outside the USAID Program. A future program would benefit from a stronger and dedicated management structure to address these weaknesses, which have been exacerbated by current changes the public sector is now undergoing. A robust and uniform approach to M&E should be a central element of management. Recommendation 1: Give careful consideration to program management structure, to strengthen both coordination between implementing partners and M&E. Continuing to emphasize the long-term risk reduction measures that were developed during the program cycle holds much better prospects than emergency responses for reducing HPAI incidence and limiting both poultry-to-poultry and poultry-to-human transmission. This includes improved farm biosecurity and veterinary hygiene along the value chain. New vaccine technology also has potential for reducing HPAI risk. Recommendation 2: Seek a commitment in future support for HPAI control to following the modified Animal Health and Livelihood Sustainability Strategy of 2010. Measuring any improvement in HPAI incidence has proved elusive. Successful disease control requires assessment of the impact of interventions, and that should be part of the design of any new project. Experience has shown that surveillance in order to detect outbreaks is very difficult, and it is likely that monitoring the value chain in some way will be more effective in providing a more reliable, though indirect, indicator of disease incidence. Recommendation 3: Ensure that the design of any new program takes into account that changes in disease incidence should be measurable and attributable to interventions to evaluate their impact. BCC efforts have been moderately successful with commercial producers but less successful at the household level in modifying high-risk practices. Behavior change as a critical component of the program requires a more strategic and integrated approach that reaches the most at-risk audiences throughout the value chain. The most effective way to create behaviour change is by policy and regulatory actions and institutionalizing training in correct practices in educational and professional institutions. Individual household behaviors are more difficult to change; it is necessary to first identify the motivations and beliefs that affect the behaviour choices of household producers and encourage good practices by putting in place support systems that lead to personal gains. In the household sector, the majority of producers are women and young girls, poultry production is critical to their food security as well as livelihoods, and they must be integrated into program’s activities, especially since 60% of adult H5N1 virus infections occur in women. HPAI prevention and control strategies need to be designed to accommodate these critical considerations, including gender. 30 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Recommendation 4: To change poultry husbandry and marketing practices on commercial farms, in households, and along the value chain, use approaches that allow stakeholders to can recognize clear economic benefits and that are pro-poor and gender￾sensitive. Although government veterinary services at all levels are still very weak, the API program has led to considerable improvement in capability and a revitalization of field services. Especially in the current transition environment, there is a risk that the improvements will be lost if there is not continuing external support. High priority should be given to increasing epidemiological expertise and supporting veterinary extension services, perhaps by expanding the role of CAHO personnel to monitoring market and other value-chain activities. There is an opportunity to expand capacity building to embrace other diseases, consistent with a One Health approach, and to further address veterinary structural deficiencies identified in the OIE gap analysis. Recommendation 5: Continue technical assistance to regulatory veterinary services to secure gains already made and reinforce professional capacities. It has proven difficult to engage the poultry industry and create trust between private and public sectors. However, the program had some successes and that is evidence that improvements in farm biosecurity has probably reduced the risk of H5N1 virus dissemination. It is unlikely, however, that real progress can be made without committed industry cooperation and acceptance of regulation measures, so continued efforts to foster cooperation between public and private sectors is necessary. An appropriate approach for household producers could be formation of poultry cooperatives supported by local veterinary extension services that provide broad support for poultry production and health. Recommendation 6: In future program planning, emphasize social investment in partnerships of government, civil society, and poultry industry stakeholders as being essential to controlling HPAI. Genetic modification of H5N1 viruses has been found to lead to mammal-to-mammal airborne transmissibility. This raises concern, particularly for Egypt, where virus isolates (e.g., in 2010 from a household duck in Sharkia governorate) have been identified with mutations that can promote transmissibility. Development of a four-way linkage between epidemiology and laboratory groups in the animal health and public health sectors, nurtured by OFFLU, and promoted by FAO and WHO, shows promise of bringing together the expertise required to characterize viruses with knowledge of their epidemiological history, better understand to possibility of virus strains with pandemic potential emerging, and act as an early warning mechanism. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 31 Recommendation 7: Support an alliance of public health and veterinary epidemiology and laboratory expertise in Egypt to strengthen influenza virus molecular epidemiology. Targeted testing is needed for H5N1 virus mutations that promote human-to-human transmissibility, such as those reported in Egypt. The MOHP detailed many lessons learned in the public health sector, both positive and negative, in its SAIDR project final report. These lessons were gained both from experience with sporadic human cases of H5N1 infection arising from poultry between 2006 and 2012 and from the human influenza A H1N1 pandemic of 2009, the first such in 41 years. They have not been formally consolidated into an updated pandemic influenza plan. The most recent MOHP “Preparedness Plan for Pandemic Influenza,” catalyzed by the threat that avian influenza H5N1 would become capable of transmission in a sustained manner from person to person, is from 2008—just before the 2009 influenza A/H1N1 pandemic. Recommendation 8: Support an update of the Preparedness Plan for Pandemic Influenza (2008) that incorporates the many lessons learned from the avian influenza H5N1 and pandemic influenza experiences in Egypt. 32 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 33 APPENDIX A. SCOPE OF WORK GLOBAL HEALTH TECHNICAL ASSISTANCE BRIDGE PROJECT GH TECH CONTRACT NO. AID-OAA-C-12-00027 SCOPE OF WORK End of Project Evaluation of USAID/Egypt Avian and Pandemic Influenza Program as modified by USAID/Egypt 11/01/12 OUTLINE: I. BACKGROUND INFORMATION: The first section of this guide covers SOW elements that describe the program to be evaluated. This includes any relevant background information on what the project intended to accomplish, who it was intended to benefit, and any changes that have occurred during implementation. Typically, all of this information is drawn from existing documents. II. EVALUATION RATIONALE: The second section addresses the fundamentals of an evaluation, including its purpose, its intended audience and uses, and the evaluation questions it is expected to address. USAID policy envisions the development of these aspects of an evaluation SOW as an iterative and collaborative process that begins in the design phase and involves in-country partners and stakeholders. III. EVALUATION DESIGN AND METHODOLOGY: The third section focuses on technical aspects of an evaluation SOW, namely the evaluation’s design and the methods that are to be used for data collection and analysis. IV. TEAM COMPOSITION: The fourth section details what USAID expects will be the intended size of an evaluation team, the roles and responsibilities of team members, and the specific qualifications that team members are expected to possess. V. EVALUATION PRODUCTS: The fifth section of this document provides information on what the evaluation team is responsible for delivering to USAID, both throughout the evaluation and upon its completion. Specifically, this refers to the final evaluation report, which must meet USAID’s reporting criteria, as well as any other reports, research instruments, or briefings required by the Agency. VI. EVALUATION MANAGEMENT: The final section addresses the management elements of an evaluation SOW, not the least of which is USAID’s budget for the evaluation. Additional SOW elements in this section include the evaluation logistics, timeline, and period of performance. 34 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM VII. PROGRAM INFORMATION Identifying Information USAID/Egypt API program is composed of the following six projects/components: Project Title Award Number Award Date Funding Implementing Organization Agreement/ Contract Officer’s Representative Strengthening Avian Influenza Detection and Response (SAIDR) IL # 1 Oct. 1, 2007–Sept. 30, 2011 $23.8 million Animal Health Component IL#1 Oct. 1, 2007–Sept. 30,2010 $2.5 million General Organization of Veterinary Services (GOVS) FAO ILs were managed by USAID staff AID-263- IO-11- 00001 Oct. 1, 2007– Sept.30, 2011 $9.3 million Andrew Clements Human Health Component IL#1 Oct. 1, 2007– Sept. 30, 2010 $7 million Ministry of Health and Population (MOHP) ILs were managed by USAID staff Communication Component (Communication for Healthy Living Project) 263-A-00- 03-00053- 00 Oct. 1, 2007- –Sept. 30, 2010 $ 5 million Vikki Stein Avian Influenza Vaccine Efficacy Project (AIVEP) IL # 2 June 2008– June 2011 $313,500 National Laboratory for Quality Control of Poultry Production (NLQP) ILs were managed by USAID staff June 2008- June 2011 $2,416,500 FAO Stamping Out Pandemic and Avian Influenza (STOP AI) Project EDH-I-00- 05-00004- 00 July 2009– Sept. 2010 $111,000 Development Alternatives. Inc. Andrew Clements Improved Biosecurity and Hygiene at Production, Collection Points and Live Bird Markets (LBM), including Decentralization Centrally managed agreement Feb 2009– June 2010 $575,000 FAO Andrew Clements Developing and Maintaining Public-Private Partnership (PPP) for the Prevention, Detection, and Control of Highly Pathogenic Avian Centrally managed agreement Feb. 2009– Sept. 2009 $312,000 FAO Andrew Clements USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 35 Project Title Award Number Award Date Funding Implementing Organization Agreement/ Contract Officer’s Representative Influenza H5N1 and Other Emerging Infectious Animal Diseases USAID Deliver Project OAA-TO￾11-00015 Oct. 1, 2007- Ongoing $633,250 USAID/W Deliver Project Robert Blanchard Activity Manager: Dr. Akmal Elerian, USAID/Egypt; Evaluation Program Manager: Shadia Attia, USAID/Egypt Program Description Background Highly pathogenic H5N1 avian influenza (HPAI) was first confirmed in Egypt in February 2006, from backyard birds in several governorates. Since February 2006, Egypt has accounted for a significant proportion of the world’s reported H5N1 poultry outbreaks and human cases, ranking number two in the world behind Indonesia for both avian and human cases. Outbreaks continue to be reported primarily in the Nile Delta area; however, H5N1 has been confirmed in 24 of the current 27 governorates and is considered endemic. In the last five years there have been significant investments by USAID (close to $30 million), the World Bank ($7.14 million), and the European Union (€ 2.5 million to the Ministry of Agriculture and Land Reclamation and reallocation of $3.2 million from an ongoing Health Sector Reform Project). These investments were directed to strengthening influenza control and pandemic preparedness. However, HPAI H5N1 remains highly prevalent in poultry throughout the country and, apparently, within the poultry production sector. With 25,000 commercial poultry farms, an estimated 7 million household production units, and poultry being the highest consumed meat in Egypt, the poultry industry represents a critical contributor to lives, livelihoods, and food security. From February 2006 until May 2012, outbreaks of H5N1 HPAI had been detected in 1,065 commercial farms, 1,410 household poultry operation and 19 live-bird markets, resulting in more than 40 million birds being culled. Through April 2012 Egypt reported 167 human A/H5N1 cases with 60 fatalities. The animal and public health significance posed by the endemic H5N1 virus alone was compounded by the emergence of the pandemic novel influenza A H1N1 virus and its arrival in Egypt in June of 2009. Now the major concern, and what has the global scientific community on full alert, is that Egypt (along with Indonesia, Vietnam, China, and Bangladesh) is not only burdened with the H5N1 virus, which is extremely fatal but not transmissible from human to human, but also the H1N1 virus, which is mildly fatal but easily transmissible from human to human. Such circumstances result in the increased likelihood of co-infection with both viruses, and in turn the likelihood of the emergence of an extremely fatal virus that has sustained human￾to-human transmission. From FY 2007 to FY2011, in response to the growing avian influenza crisis, USAID committed about $30 million of support for avian and pandemic influenza (API) prevention and control 36 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM activities in Egypt. Despite these huge investments, the number of human cases is still on the increase, and HPAI has become an endemic disease, causing considerable economic loss to the Egyptian poultry industry, and posing an imminent public health threat. Program Description Development Hypothesis: Produce improved and sustainable avian and pandemic influenza (API) prevention and control, thereby eliminating the necessity for technical assistance in achieving a situation with API in Egypt in which the disease no longer represents a significant threat to human health and in which measures implemented by producers and supported by regulatory authorities minimize the impact of the disease on the industry, livelihoods, and food security. USAID began providing API assistance to the Government of Egypt (GOE) in 2006. Since that time, implementing partners have included the Ministry of Health and Population (MOHP), the Ministry of Agriculture and Land Reclamation (MOALR), and USAID grantees Johns Hopkins University/Center for Communications Programs (JHU), the Food and Agriculture Organization of the United Nations (FAO), OFFLU (the joint FAO and World Organization for Animal Health network of expertise on influenza), Development Alternatives Incorporated, the DELIVER Project, and the National Laboratory for Quality Control of Poultry Production (NLQP). Since 2002 USAID, in partnership with Naval Medical Research Unit #3 (NAMRU-3), has supported the establishment of the Egyptian National Infection Control Program, the Epidemiology and Surveillance System, and the National Disease Surveillance and Response Project in the MOHP. In its initial response to avian influenza, the MOHP was able to draw on this previous USAID/Egypt support. Since 2006 USAID funding has strengthened MOHP and MOALR AI surveillance and response capacity through the establishment of rapid response teams at central and governorate levels, training of health workers and veterinarians, and provision of supplies. USAID, through its cooperative agreement with JHU, has worked in partnership with the GOE since 2003 to develop and build an integrated, strategic approach to communication across health issues. As a result, the JHU/Communication for Healthy Living (CHL) project assisted the GOE (MOHP and SIS) to launch an immediate communication response and maximize outreach when H5N1 emerged. USAID/Egypt worked with the MOHP and SIS to develop the first national Avian Influenza Communication Strategy, which included an AI website and a national communication campaign. The campaign included posters, flyers, television spots, and community mobilization activities. Critical Assumptions: The successful implementation of USAID–supported API projects assumed that  the national and provincial authorities, at all levels, realize the importance of stopping HPAI at its origin, maintain a strong willingness for investing in emergency plans, commit resources, and participate in control activities;  the national authorities are willing to provide suitable human resources in an acceptable technical environment and provide the basis for capacity building;  the national authorities are willing to collaborate with all the stakeholders involved in the prevention, detection, and control of the avian influenza emergency; USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 37  the national authorities, at all levels, participate in the awareness activities required at household level, and stakeholders agree on recommended rehabilitation measures; and  donors are prepared to invest in the follow-on program. Program Components: 1. Strengthening Avian Influenza Detection and Response (SAIDR) Project: Oct. 1, 2007–Sept. 30, 2011 $24 million The overall objective of the project is to "minimize the risk to human health and to reduce the impact on people’s livelihoods from HPAI through its effective prevention and control.” The SAIDR project is focused on three main components: animal health (Sub-Goal 1), human health (Sub-Goal 2), and communication as a cross-cutting area (IR3). The direct beneficiaries of the project are MOALR, MOHP, and SIS in general, and the GOVS and the Epidemiology Surveillance Unit (ESU)/MOHP in particular. Other beneficiaries of the project include  the general population, benefiting from reduced risk of HPAI;  health care workers, hospital physicians, nurses, laboratory teams, and those engaged in other surveillance activities;  national veterinary and veterinary public health services benefiting from strengthened government agencies, including management, technical understanding, and responsibilities;  poultry owners, producers, and traders benefiting directly from a reduced incidence of HPAI and improved poultry disease prevention and control methods; and  the international community, benefiting from reduced risk of the spread of HPAI and of a human influenza pandemic. 38 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM IR1.2: Prevent future outbreaks in poultry IR1.1: Improve preparedness and planning to contain bird-to￾bird and bird-to-human transmission of H5N1 IR1.3: Improve detection of H5N1 in wild birds and poultry IR1.4: Improve containment measures in poultry populations IR1.5: Limit exposure of bird￾outbreak investigators, poultry cullers, and poultry vaccinators to H5N1 virus IR2.1: Improve preparedness and planning for a human influenza pandemic IR2.2: Improve detection of suspect H5N1 cases in humans IR2.3: Improve containment measures in human populations IR2.4: Limit exposure of health staff investigating outbreaks and treating patients to the H5N1 virus Subgoal 1: Prevent H5N1 infections in birds Goal: Prevent H5N1 infections in humans Subgoal 2: Prevent human exposure to H5N1 virus IR3.1: Improve awareness of avian and pandemic influenza IR3.2: Decrease high-risk behaviors associated with transmission of H5N1 among birds and humans USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 39 1. Animal Health Component – GOVS: Implementation Letter Oct. 1, 2007–Sept. 30, 2010$2.5 million – FAO: PIO grant Oct. 1, 2007–Sept. 30, 2011$9.3 million The main focus of the SAIDR project is on the animal health component pertaining to the prevention, detection, and response to HPAI outbreak in poultry. The activities under the program are expected to improve General Organization of Veterinary Services (GOVS) preparedness and capacity in planning and containing outbreaks of HPAI. The Food and Agriculture Organization of the United Nations (FAO), through its Emergency Centre for Transboundary Animal Disease Operations (ECTAD) in Egypt, is providing technical expertise support for the project. GOVS is the main implementing agency for the project through its Project Management Unit (PMU), and it administers the contributions that have been separately allocated by USAID/Egypt though an Implementation Letter (IL). The sharing of the project implementation responsibilities between FAO and MOALR serves the purpose of introducing project sustainability features from the start. It was envisaged that the share of MOALR-executed activities would progressively increase over time in order to enhance the handover of project responsibilities to the national and decentralized structures. Intended results of the animal health component are  improved preparedness and planning to contain bird-to-bird and bird-to-human transmission of H5N1;  prevention of future outbreaks in poultry;  improved detection of H5N1 HPAI in poultry and wild birds;  improved containment measures in poultry populations;  limited exposure of bird-outbreak investigators, poultry cullers, and poultry vaccinators to the H5N1 virus. 2. Human Health Component – MOHP: Implementation Letter Oct. 1, 2007–Sept. 30, 2010$7 million The human health component of the SAIDR Project was designed with the overall goal of reducing the impact of HPAI on humans. Intended results of the human health component were  improved preparedness and planning for a human influenza pandemic;  improved detection of suspect H5N1 cases in humans;  improved containment measures in the human population; and 40 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM  limited exposure of health staff investigating outbreaks and treating patients to the H5N1 and H1N1 virus. Implemented activities pertained to strengthening surveillance networks; improving H5N1 influenza case management skills and infection control systems in local hospitals; making significant investments in curative care by providing additional ventilators for selected general, chest, fever, and pediatric hospitals; improving outbreak investigation and response methods, with a view to detecting human-to-human transmission at the earliest stages; assisting governorate-level planning for pandemic preparedness, including strengthening quarantine systems; and strengthening the capacity of the Central Public Health Laboratory. In June 2009, WHO raised the influenza pandemic alert to phase 6; USAID/Egypt approved reallocation of $400,000 to support the MOHP response to the A/H1N1 pandemic. The MOHP activated its Pandemic Preparedness Plan and stepped up surveillance activities at health care facilities and airports for rapid detection and management of cases, mitigation measures in schools, comprehensive communication campaigns, development of case management protocols, strengthened management capacities of 220 general and district hospitals, and improved infection control practices. 3. Communication Component – The Johns Hopkins Bloomberg School of Public Health (JHU), Center for Communications Programs (JHU/CCP), Communication for Healthy Living(CHL) Project,Cooperative Agreement Oct. 1, 2005–Sept. 30, 2010 $5 million Building upon previously established health communication infrastructure, CHL in collaboration with the Government of Egypt (GOE) prepared a full-scale response prior to the avian influenza outbreak on February 17, 2006. This included establishment of the National Supreme Committee and development of behavior change communication (BCC) activities directed toward the prevention of and response to API. Egypt was in a unique position to respond swiftly and effectively to the avian flu crisis largely due to the more than 25-year partnership and cooperative mechanisms established between the MOHP and the MOI/SIS with the support of USAID. A practical communication strategy developed under the auspices of the National Supreme Committee, chaired by the MOHP with representatives from all the concerned national bodies (including the former MOI and the Ministry of Agriculture) as well as international partners (such as USAID, NAMRU-3 and WHO), guided mobilization efforts. Intended results of the communication component were  Improved awareness of avian and pandemic influenza; and  Decreased high-risk behaviors associated with transmission of H5N1 among birds and humans. The two intended results provided crosscutting communication support to different components of the SAIDR project. The first result aimed to improve awareness, and the second result to decrease high-risk behaviors and improve proper behaviors associated with prevention of HPAI and preparedness for pandemic influenza. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 41 CHL worked with providers of human healthcare to improve preparedness, management, and treatment of API cases by focusing on all cadres of government and private sector health workers as well as other sources of health information, specifically private pharmacists, to improve their knowledge of API, emphasizing early referral, improve their communication skills, and improve compliance with infection control, case management, and reporting protocols. Activities were also directed toward the poultry industry to improve management and prevention outcomes of H5N1 transmission in birds. This was accomplished by focusing on improving communication skills among veterinary workers (veterinarians, agricultural extension workers, rapid response teams, etc.). In addition, job aids were developed for service providers to assist them in their work. Through CHL’s community outreach program, implemented by Save the Children, CHL worked with other community members, such as women backyard poultry producers, school, and others to increase their awareness of API and improve their behaviors. This was done through a variety of message formats, such as media spots, outreach activities, village seminars, school activities, and women’s groups. 4. Avian Influenza Vaccine Efficacy Project (AIVEP) Joint project: June 2008–June 2011 – National Laboratory for Quality Control of Poultry Production (NLQP) Implementation Letter $313,500 – FAO: PIO grant $2,416,500 The general objective of the project was to assist the GOE in its efforts to control and eradicate HPAI in the domestic poultry population and avoid the risks of human infection through identification of antigenic variants among field variants and determination of the ability of available/used vaccines to provide protection against each identified isolate. FAO provided technical support through AIVEP, as vaccination is considered part of the overall efforts to reduce HPAI infections and spread of the virus in Egypt, with the potential to benefit other countries in the region and worldwide. Specific project objectives were to conduct 1. Screening and evaluation of genetic and antigenic variants among existing H5N1 HPAI field strains collected from 2006 until early 2008; 2. Intensified collection of H5N1 HPAI field isolates from newly confirmed outbreaks since mid-2008; 3. Challenge testing of antigenic variants in specific pathogen-free (SPF) birds and currently used avian influenza (AI) vaccines in Egypt; and 4. Controlled transmission trials in the laboratory on birds raised and vaccinated in commercial poultry farms. 5. Development Alternatives, Inc., Stamping Out Pandemic and Avian Influenza (STOP AI) Project – Contract with USAID/W July 2009–Sept. 2010 $111,000 42 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM The primary goal of this project was to implement interventions to improve biosecurity in selected vulnerable areas of the poultry value chain that reduce the transmission of HPAI. Specific objectives were to 1. establish a private-public partnership (PPP) with the Poultry Union, individual Poultry Union members, or other commercial partners to provide technical assistance in biosecurity upgrading and other services for Sector 2 and 3 broiler grow-out, layer, hatchery, and breeder farm managers and owners; 2. provide coordinated support for FAO’s proposed decontamination program; and 3. provide training materials for use by FAO and GOVS to provide biosecurity and outbreak response training to GOVS veterinarians. Efforts focused on measurable impact from relatively small-scale risk reducing and cost-effective interventions that can be promoted and replicated. They have concentrated activities on poultry producers in Sectors 2 and 3. 6. Improved biosecurity and hygiene at production, collection points, and live bird markets (LBM), including decontamination – FAO: PIO grant Feb. 2009–June 2010 $575,000 The objective of this project was to develop and implement an integrated cleaning and disinfection program in selected LBMs and other collection points in Egypt aiming to minimize the risk for human health and reduce transmission and spread of HPAI virus. Biosecurity measures were designed to reduce the amount of the agent (HPAI virus, in this case) that was moved from place to place. Specific activities included developing disinfection guidelines and protocols; providing training for staff/workers; and implementing and monitoring cleaning and disinfection operations at commercial poultry farms and collection points. 7. Developing and maintaining public-private partnerships (PPP) for the prevention, detection and control of Highly Pathogenic Avian Influenza H5N1 and other emerging infectious animal diseases – FAO: PIO grant – Feb. 2009–Sept. 2009 $312,000 Objectives of the project were to create, strengthen, and maintain PPP to support poultry health and production systems, within a functional animal health system led by official veterinary services. Activities included: (a) strengthening the capacity of the public veterinary services to lead the development and management of the animal health system to prevent, detect, and control HPAI and other animal diseases, and (b) creation of forums and networks of public-private stakeholders for enhanced communication, education, information dissemination, and awareness for the prevention, detection and control of HPAI and other animal diseases. 8. USAID DELIVER Project – Oct. 1, 2007–ongoing Contract with USAID/W $633,250 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 43 In 2006 USAID was tasked with creating and maintaining the Avian Influenza International Stockpile (AIIS) to ensure that countries could receive the commodities. The USAID DELIVER Project, Task Order 2, funded by USAID/W, was awarded on March 21, 2007, to manage the USAID AIIS and to distribute avian influenza commodities to recipient countries around the world. By ensuring the availability of personal protective equipment (PPE), decontamination equipment, and other supplies, the project supports surveillance, outbreak response, and aggressive decontamination activities in countries at risk for and currently experiencing an HPAI outbreak. In support of Egypt’s fight against HPAI, more than 70,000 sets of PPEs have effectively been delivered to date. Relevant Documentation USAID team and the six API project teams will provide the evaluation team with a package of relevant materials, including  SOW for each USAID API project  Program/project strategies  Quarterly and annual reports  Work plans  Project PMPs  Financial records and reports  2008 Egypt Demographic and Health Survey, UNICEF KAP study, and other relevant studies  External audit reports  API Inter-Ministerial Conference, 2008, with focus on Government of Egypt final statement  Assessment reports  UN API Coordinator final report, 2010  AI gender analysis report  Any other reports and documents reflecting USAID/Egypt’s API work VIII. EVALUATION RATIONALE Evaluation Purpose The USAID/Egypt Mission is planning to conduct a performance evaluation of its six projects/components of the Avian and Pandemic Influenza (API) program. The purpose of this evaluation is to 1. review, analyze, and evaluate the effectiveness of the USAID funded API activities to date in achieving program objectives, completing deliverables; and 2. assess USAID/Egypt’s contribution to improved API prevention and control. 44 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM The findings of the evaluation will inform the direction of future USAID/Egypt API funding and will be utilized to develop an SOW for a follow-on integrated API program. Audience and Intended Uses The audience of the evaluation report will be the USAID/Egypt Mission, specifically the health team, USAID/Washington, and the future implementing partners of API activities. The executive summary, expanded executive summary, and final report will be provided to the Ministry of Health and Population (MOHP), the General Organization for Veterinary Services (GOVS), the Ministry of Agriculture and Land Reclamation (MOALR), the National Laboratory for Quality Control of Poultry Production (NLQP), the State Information Service (SIS), and other donors in Egypt working on API. USAID/Egypt will integrate the evaluation recommendations into future API activities and share lessons learned with other stakeholders. The evaluation will also provide important feedback to each of the partners that should help them to understand both their strengths and areas where technical, administrative and management efforts could be improved. In addition, Government of Egypt counterparts will learn how to better benefit from implementing partner technical assistance. It is expected that the MOHP, GOVS, MOALR, and NLQP will have the opportunity to discuss how the USAID-supported API program assisted them and how these types of projects could better assist them in the future to tackle the API epidemic. Evaluation Questions The evaluation will answer the following Illustrative questions: 1. To what extent did the API Program with its different components achieve the intended goals and results in the area of API? a. Was the program design appropriate to support the Development Hypothesis? If yes, why? If not, why not? Please give specific examples to support your answer. b. Has the program been effective in adjusting activities to reflect changes to the environment in terms of the disease, the poultry industry, and the GOE? If yes, why? If not, why not? Please give specific examples to support your answer. c. What were the key constraints or setbacks that may have hindered the achievement of results? Are there any constraints that may hinder future work? 2. To what extent were the monitoring and evaluation systems of the API program with its different components effective for monitoring progress? Please substantiate your answer with specific examples. a. Was a PMP developed for each project? b. Were data-gathering methods adequate for monitoring progress and indicators? c. Are the data disaggregated and analyzed by gender or is the outcome/impact on males and females measured whenever applicable? d. How was the information used for program management and improvement? USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 45 3. To what extent were the technical components and approaches of the API program effective in achieving the desired outcomes/impacts? a. Did the program activities contain an appropriate mix and focus of technical interventions to produce the largest impact on human health, animal health, and API control? If yes, why? If not, why not? Please give specific examples to support your answer. b. What were the outcomes/impact on males and females? c. Was the program responsive to stakeholder needs? If yes, why? If not, why not? Please give specific examples to support your answer. d. How is the program perceived/valued by the Government of Egypt counterparts? 4. To what extent were the management structures, administrative support, and partnerships effective? a. Did the program’s administrative and management structures appropriately support the implementation of the activities of its different components? If yes, why? If not, why not? Please give specific examples to support your answer. b. Has the API program facilitated synergy, collaboration, and coordination among all USAID implementing partners as well as the GOE? If yes, why? If not, why not? Please give specific examples to support your answer. 5. What are the lessons learned and best practices and the corresponding recommendations for improving the efficiency and effectiveness of the API program? a. Based on current conditions and lessons learned, what are the essential activities that should be implemented to achieve improved and sustainable API prevention and control? Please substantiate your answer with specific examples. b. What changes, if any, should be considered by USAID/Egypt to make the API program more responsive and effective? Please substantiate your answer with specific examples. c. Were the program activities sustainable without further USAID funding? If yes, which component(s), and why? If not, why not? Please give specific examples to support your answer. IX. EVALUATION DESIGN AND METHODOLOGY Evaluation Design This is a performance evaluation and is intended to focus on how the six components/projects of the API program have been implemented, what they have achieved, whether expected results have occurred according to project designs and in relation to the development hypothesis, whether the projects were cost-effective, and how activities are perceived, valued, and sustained. Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to generate answers. Data Collection Methods The evaluation team should consider a range of possible methods and approaches for collecting and analyzing the information that is required to assess the evaluation objectives. The evaluation 46 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM team shall share data collection tools with USAID for review, feedback, and/or discussion with sufficient time for USAID’s review before they are applied in the field. The data collection methodology will include a mix of tools appropriate to the evaluation’s questions. These tools will include a combination of document review, in-depth interview with key informants, and focus group discussions. The evaluation team will do site visits to some governorates Document Review: USAID/Egypt will provide the evaluation team with electronic access to key program-related documents mentioned above in the relevant documents section prior to the start of in-country work. All team members shall review these documents in preparation for the initial Team Planning Meeting. Interviews and Site Visits: The evaluation team will conduct in-depth interviews and focus group discussions, at a minimum, with the following organizations/staff:  MOALR  GOVS  MOHP  NLQP  ECTAD Egypt, FAO  UNICEF  NAMRU -3  WHO  World Bank  CHL/JHU (Save the Children)  Poultry producers within all sectors  USAID staff  Other beneficiaries The evaluation team will provide a more detailed explanation of the proposed methodology for collecting the data. Proposed governorates for the site visits are Kalyoubia, Giza, Fayoum, Gharbia, and Behira. Mode of transportation is by car. USAID will provide a detailed in-country interview schedule prior to the assignment’s inception. The evaluation team may be accompanied by a staff member from USAID/Egypt, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 47 Data Quality Standards The evaluation team shall ensure that the data they collect clearly and adequately represent answers to the evaluation questions, be sufficiently precise to present a fair picture of performance, and be at an appropriate level of detail. Data Limitations Having the evaluation done for the six components of the API program within a relatively short timeline may jeopardize the required level of detail that is expected to give a clear picture of program performance. It is anticipated that some interviews may be conducted through translators for the international evaluation teams. As a result, some differences of the language might lead to not capturing the full intent or meaning offered by the interviewees. It is also anticipated that some interviews may be conducted in the presence of one or more outside observers, including project and USAID staff. As a result, the interview responses might be affected by the presence of these observers. USAID expects that all threats to validity be discussed and documented in the evaluation planning stage, including what will be done to minimize threats to validity, and detailed in the final report. Data Analysis Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan that details how focus group interviews will be transcribed and analyzed, and how the qualitative data from the focus group discussions and in-depth interviews with the key informants and other stakeholders will be integrated with quantitative data from the different related documents to reach conclusions about the effectiveness and efficiency of the API program. The Mission expects the evaluation team to present strong quantitative and qualitative analysis, within data limitations, that clearly addresses key issues found in the research questions. The Mission anticipates that the evaluation team will provide a more detailed explanation of the proposed methodology for carrying out the work. X. TEAM COMPOSITION USAID encourages the participation of local experts on evaluation teams. USAID staff are also encouraged to participate on evaluation teams, as are MOHP, MOALR, and GOVS or other stakeholders when their participation would be beneficial for skill development and not present a conflict of interest or a threat to validity, or their engagement in the evaluation would help to ensure the use of evaluation results within USAID. All attempts should be made for the team to be comprised of an equal number of male and female members.  Team Leader: a senior international consultant with extensive experience in leading and conducting USAID health program evaluations.  Team members (3): A mix of senior and mid-level consultants with the following areas of expertise: Avian and pandemic influenza (human and animal components), zoonotic disease control, and behavior change communication.  One local technical consultant with an excellent understanding of the Egyptian public health system as well as USAID programs, who is fluent in Arabic. 48 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM  One local logistics coordinator to handle travel-related logistics and provide administrative support to the technical team members. At least one member should also have strong expertise in monitoring and evaluation. The team leader (TL) should be an independent consultant but the technical specialists may be USAID/W Global Health staff. Team Leader/Evaluation Methods Expert: a senior consultant with extensive experience in leading and conducting USAID health program evaluations. Should be an independent consultant and have an MPH or related post-graduate degree in public health. S/he should have at least 10 years senior-level experience working in infectious diseases (preferably API) in a developing country. S/he should have extensive experience in conducting qualitative and quantitative evaluations/assessments. Excellent oral and written skills are required. The Team Leader should also have experience in leading evaluation teams and preparing high-quality documents. This specialist should have wide experience in implementation of USAID-funded infectious disease programs. S/he should also have a good understanding of project administration, financing, and management procedures. The Team Leader will  Finalize and negotiate with USAID/Egypt the evaluation work plan;  Establish evaluation team roles, responsibilities, and tasks;  Facilitate the Team Planning Meeting (TPM);  Ensure that logistics arrangements in the field are complete;  Manage team coordination meetings in-country and ensure that team members are working to schedule;  Coordinate the process of assembling individual input/findings for the evaluation report and finalizing the evaluation report; and  Lead the preparation and presentation of key evaluation findings and recommendations to USAID/Egypt team prior to departing Egypt. Avian and Pandemic Influenza Specialist: This specialist will have a public health degree and at least 7–10 years of experience in management of or consulting on avian influenza. S/he should have a proven background and experience in avian influenza control programs and a strong understanding of the challenges facing such programs in Egypt or Asia. S/he should also have a good understanding of relevant national programs in API prevention and control, including both public and private sectors. This specialist will be responsible for assessing the ability of the program to achieve outcomes in the animal health component. This specialist will also assess the technical quality of the human health interventions. S/he will document key lessons learned and provide recommendations for modifications in approach, results, or activities. Zoonotic Disease Specialist: This specialist will have a veterinary degree and at least 7–10 years of experience in management of or consulting on zoonotic diseases, including avian influenza. S/he should have a proven background and experience in avian influenza control programs and a USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 49 strong understanding of the challenges facing such programs in Egypt or Asia. S/he should also have a good understanding of relevant national programs in API prevention and control, including both public and private sectors. This specialist will be responsible for assessing the ability of the program to achieve outcomes in the animal health component. This specialist will also assess the technical quality of the animal health interventions. S/he will document key lessons learned and provide recommendations for modifications in approach, results, or activities. Behavior Change Communication/Community Mobilization Specialist: This specialist should have wide experience in implementation of behavior change communication (BCC) and community mobilization programs in the areas of API. S/he should have a postgraduate degree in health promotion sciences or a related field with a minimum of 5-10 years of experience working with USAID-supported BCC mobilization programs in developing countries. S/he will analyze the program’s behavior change interventions and assess the effectiveness and appropriateness of the approaches adopted by the project to improve API knowledge, health￾seeking behavior, and health outcomes. S/he will also assess the technical foci of BCC activities and whether they are the appropriate mix and topics for intervention communities. Local Technical Specialist: The local technical specialist is expected to have an MPH or related postgraduate degree in public health or community medicine and at least 7–10 years of experience in working in or consulting on infectious disease (preferably zoonotic disease/API). S/he should have an excellent understanding of the Egyptian public health system and the national program in API prevention and control, including both private and public sectors. S/he should also have proven experience in conducting evaluations and assessments and drafting high￾quality reports. The local specialist will help the team to better understand different cultural and social issues related to API in Egypt. S/he will also assist in communications and interviews with local stakeholders. S/he will participate in different evaluation activities and may be assigned specific tasks by the Team Leader as appropriate. Local Logistics Coordinator: The logistics coordinator should be a local staff member for handling travel-related logistics and providing administrative support to the technical team members. The logistics coordinator will also be responsible for setting up meetings with USAID and stakeholders. Required qualifications include  demonstrated ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments, and perform tasks independently with minimal supervision;  capacity for effective time management and flexibility;  ability to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts;  fluency in both English and Arabic; and  proven ability to communicate clearly, concisely, and effectively both orally and in writing. 50 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM XI. EVALUATION PRODUCTS Deliverables Team Planning Meeting: A one-day day Team Planning Meeting (TPM) will be held in Egypt at the outset of the evaluation. This meeting will allow USAID/Egypt to discuss the purpose, expectations, and agenda of the assignment with the evaluation team. In addition, the team will  clarify team member roles and responsibilities;  review and develop final evaluation questions;  review and finalize the assignment timeline and share it with USAID/Egypt;  present and discuss data collection methods, instruments, tools, and guidelines; and  review and clarify any logistical and administrative procedures for the assignment. Work Plan: The team will draft a detailed work plan prior to travelling to Cairo, to be shared and discussed with USAID/Egypt team during the TPM. The detailed work plan will include the methodologies to be used in the evaluation, timeline, and detailed Gantt chart. The work plan will be submitted to the API Activity Manger and Evaluation Program Manager at USAID/Egypt for approval no later than the 4th day of work. Methodology Plan: A written methodology and data analysis plan (evaluation design, data analysis steps and detail, and operational work plan) will be prepared before the team travels to Cairo, to be discussed during the TPM with and approved by USAID prior to implementation. List of Interviewees and Schedule: USAID will provide the evaluation team with a stakeholder analysis that includes an initial list of interviewees, from which the evaluation team can work to create a more comprehensive list. Prior to starting data collection, the evaluation team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The evaluation team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. Data Collection Tools: Prior to starting fieldwork, the evaluation team will share the data collection tools with the USAID Evaluation Program Manager for review, feedback, and/or discussion and approval. In-briefing (TPM) and Mid-term Brief with USAID: The evaluation team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in-brief, the evaluation team should have the list of interviewees and schedule prepared, along with the detailed Gantt chart that maps out the evaluation through the report drafting, feedback, and final submission periods. At the mid-term brief, the evaluation team should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. Discussion of Preliminary Draft Evaluation Report: The evaluation team will submit a preliminary draft of the report to the USAID Evaluation Program Manager, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with Partners: The team will present the major finding of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 51 the team’s departure from the country. The debriefing will include a discussion of achievements and activities only, with no recommendations for future programs. The team will consider partners’ comments and revise the draft report as appropriate. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Egypt through a PowerPoint presentation after submission of the draft report and before the team’s departure from the country. The debriefing will include a discussion of achievements and issues as well as recommendations for future activity designs and implementation. The team will consider USAID/Egypt comments and revise the draft report accordingly. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the team’s departure from Egypt. The written report should clearly describe findings and conclusions. Recommendations for future programming will be addressed in a separate internal memo. USAID will provide written comments on the draft report within 5 working days of receiving the document. Final Report: The evaluation team will submit a final report that incorporates responses to Mission comments and suggestions no later than three working days after USAID/Egypt provides written comments on the team’s draft evaluation report (see above). If USAID/Egypt determines that there are still content issues to be addressed or that previous feedback has not been satisfactorily addressed, the final unedited report will be considered a second draft and further feedback will be given to the team no later than 10 days of receipt of the second draft. If USAID/Egypt determines that there is no need for further changes, the report will be considered a final unedited draft and no further feedback will be given. This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format will include an executive summary, table of contents, glossary, methodology, findings, and conclusions. The report will be submitted in English, electronically, and then disseminated within USAID/Egypt for final approval. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. Data Sets: All data instruments, data sets, presentations, meeting notes, and the final report for this evaluation will be presented to USAID on three (3) flash drives to the Evaluation Program Manager. All data on the flash drive will be in an unlocked, editable format. See table 1 for the timelines of the deliverables. Reporting Guidelines  The evaluation report should represent a thoughtful, well-researched, and well- organized effort to objectively evaluate what worked in the project, what did not, and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 52 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) and what is being done to mitigate threats to validity.  Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented—organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. Evaluation Report Requirements The format for the evaluation report is as follows: 1. Executive Summary—concisely state the key findings (2 pp); 2. Table of Contents (1 pp); 3. Introduction—purpose, audience, and summary of task (1 pp); 4. Background—brief overview of AP program in Egypt, USAID program strategy and activities implemented in response to the problem, brief description of AI projects/components, purpose of the evaluation (3–4 pp); 5. Methodology—describe evaluation methods, including threats to validity, constraints and gaps (1 pp); 6. Findings/Conclusions—based on the evaluation questions; also include data quality and reporting system that should present verification of spot checks, issues, and outcome (20-24 pp); 7. Challenges—provide a list of key technical and/or administrative, if any (1–2 pp); 8. References—including bibliographical documentation, meetings, interviews, and focus group discussions; 9. Annexes—annexes that document the evaluation methods, schedules, interview lists, and tables should be succinct, pertinent, and readable. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports: (http://www.usaid.gov/policy/evalweb/evaluation_resources.html). The final evaluation report will conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy. The Evaluation Program Manager will determine if the criteria are met. This evaluation will not conclude until the Evaluation Program Manager has confirmed, in writing, that the report has met all of the quality criteria. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 53 The final version of the evaluation report will be submitted to USAID/Egypt electronically. The report format should be restricted to Microsoft products, and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 40 pages, excluding references and annexes. XII. EVALUATION MANAGEMENT Logistics USAID/Egypt will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID/Egypt will assist in arranging meetings with key stakeholders as identified by USAID/Egypt prior to the initiation of field work. The evaluation team is responsible for arranging all other meetings as identified during the course of this evaluation and advising USAID/Egypt prior to each of those meetings. The evaluation team is also responsible for arranging vehicle rental and drivers as needed for site visits around Cairo, but USAID/Egypt may facilitate travel to sites in the governorates. USAID/Egypt can also assist with hotel arrangements if necessary. Evaluation team members will be required to make their own payments. USAID/Egypt personnel will be available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Scheduling and Level of Efforts Work is to be carried out over a period of approximately 13 weeks, beginning on or about (o/a) October 18, with field work completed in November and final report and close out concluding in December. Timeline and LOF table is attached. Budget GH Tech Bridge will draft the activity budget. 54 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 55 APPENDIX B. PERSONS CONTACTED USAID Dr. Thomas Easley, former HPAI coordinator (now USAID/Uganda) Dr. Andrew Clements, USAID/Washington Mr. Randall Kolstad Dr. Akmal, API Program Manager Dr. Nabil Alsoufi Ms. Shadia Attia, Evaluation Program Manager MOHP Dr. Md Genedy, DG Communicable Disease Control Dr. Samir Abdel Aziz Refaey, Director, Epidemiology and Surveillance Unit Dr. Amr Kandeel, First U/Secretary, Preventive Affairs and Endemic Diseases Dr. Nasr Elsayed Dr. Amal Zaki Dr Manal Labib Fahim Dr. Amany Elgohary Sheta, Head, Virology, Central Public Health Laboratory Dr. Amel Mohamed Naguib, Head, Virology Molecular Unit, Central Public Health Laboratory Dr. Mohamoud Abdel Magid, General Director, Abbasia Chest Hospital MOALR/GOVS Dr. Soheir Hassan AbdelKader, Head, Central Administration, Preventive Medicine Dr. Safaa El-Fadaly, General Coordinator, AIEMU Dr. Mahomed Atea Ezz-Eldin, Surveillance Manager, AIEMU Dr. Shereif Abd Elkhakek, Training, AIEMU Dr. Gehad Salah, Biosecurity, AIMEU Dr. Tarek Zakaria, Epidemiology, AIMEU Dr. Walaa Ibrahim, Biosecurity, AIMEU Dr. Mary Amin Mansoor, Biosecurity, AIEMU NLQP Dr. Soad Abd El Aziz, Head of Laboratory Dr. Mahomed K. Hasan, Technical Manager Dr. Abdul Selim, Head, Virology Dr. Abdel Satar Arafa Md, Head, Gene Analysis Unit Dr. Zakaria Elkanawati, Head, Serology 56 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM FAO Dr. Moujahed Achouri, FAO Representative in Egypt Dr. Yilma Jobre, Team Leader, ECTAD Egypt Dr. Amira Abdel Nabi, National Technical Consultant, ECTAD Egypt Dr. AbdelHakim Ali, National Technical Consultant, ECTAD Egypt Mr. Toni Ettel, Operations Officer, ECTAD Egypt Dr. Philippe Ankers, Head, Livestock Production, Rome Dr. Gwenaëlle Dauphin, Laboratory Expert, ECTAD Rome Dr. Ahmed Saad, National Technical Consultant Dr. Ihab ElMasry, National Technical Consultant WHO Dr. Naema ElGasseer, WHO Country Representative Dr. Nasr El Tantwy, Infectious Diseases Adviser UNICEF Dr. Magdy Elsanady, Senior Health Officer Dr. Sahar Hegazi, Communication for Development Specialist NAMRU-3 Dr. Maha Salat, Director, Infection Control Unit Dr. Emily Robinson, Head, Epidemiology Dr. Ann Gainer, Head, Virology STOP AI Dr. Farid Hosny, Project Coordinator, Egypt CHL Ms. Marwa Kamel, BCC Activities for API Mr. Ron Hess, (now in Kampala, Uganda) Mr. Douglas Storey, JHU/CHL GHARBIA GOVERNORATE Dr. Wafik Negm, Chief Veterinary Officer Dr. Azmi Abdel Hamid Hisham, Head, Preventive Medicine Dr. Baher Atef El Sokary, Head, Poultry Department Mr. Mohsen El Shazly, Poultry farmer USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 57 FAYOUM GOVERNORATE Dr. Makhlouf Badr, Head, Preventive Medicine Dr. Eatedal Abdelsamad, Head, Poultry Department Dr. Hoda Saad, Head, Epidemiology Dr. Samira Tantawy, Head, Poultry Section, Fayoum District Dr. Samia Kamal, District Epidemiology Unit, Fayoum District Mr. Samy Youssef, Farm owner OTHER Dr. Hassan Hameida, former chairman, NSC Dr. Abdel Rahman Fawzy, former member, NSC Gavin MacGregor-Skinner, USAID (now in Pennsylvania) POULTRY FARMER FGD PARTICPANTS Mr. Mohsen El Shazly, Gharbia Mr. Abdelattief Abdelfatah, Giza Mr. Ahmed Hamdy Kenawy, Menofia Mr. Ashraf Elmaadawy, Gharbia Mr. Baha Eldin Ahmed, Qalyubia Mr. Baheg Abdelmoniem Fadel, Fayoum Mr. Bakr El Maadawy, Gharbia Dr. Ragab Mahmoud Alshawarby, Qalyubia Elsayed Elosia, Gharbia Emad Eldin Mohamed, Giza Hamdy Menshawy, Menofia Ibrahim Attia Ahmed, Qalyubia Ibrahim Korany Mohamed, Fayoum Mr. Ibrahim Mahoud Hashim, Giza Plus 7 others 58 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 59 APPENDIX C. REFERENCES Abdel-Hakim, M.A. 2011. Mapping HPAI Risk Pathways. Cairo: FAO-ECTAD Egypt. Abdel-Hakim, Ali, and Yilma Jobre. 2011. Biosecurity Training Impact Evaluation and Mapping of Poultry Farms. Cairo: Emergency Centre for Transboundary Animal Diseases (ECTAD) UN FAO. CHL 2006 and 2008. Egyptian Health Communication Survey. CHL. 2007–2009. Quarterly and Annual Reports. CHL, MOHP, and MOALR. 2007. Detailed Implementation Plan for AI and PI Communication, October 2007–September 2008. El-Zanaty Associates. 2007. “Knowledge Attitudes and Practices Survey Household Survey related to Avian Influenza. Prepared for UNICEF. FAO. 2010. Second Real-Time Evaluation of FAO’s Work on the Highly Pathogenic Avian Influenza. Evaluation Report. Rome: FAO Office of Evaluation. FAO. 2007– 09. Quarterly, Annual, and Project Reports 2007–2009. FAO. N.D. Training Course Reports (Surveillance, On-the-Job, Epidemiology). Fasina, F.O., et al. 2012. “The Cost-Benefit of Biosecurity Measures on Infectious Diseases in Egyptian Household Poultry.” Preventive Veterinary Medicine 103: 178–91. Fasina, F.O, A. M. Ali, J. M, Yilma, O. Thieme, and P. Ankers. N.D. Technical and Financial Feasibilities of Improved Biosecurity implementation in Egyptian Household Poultry. (undated). Fauci, A. 2012. “Research on Highly Pathogenic H5N1 Influenza Virus: the Way Forward.” mBio 3 (5): e00359-12. Geerlings, E. 2011 Household Producer Value Chain Assessment. Rome; Food and Agriculture Organization of the United Nations. Government of Egypt. 2006. Avian Influenza Communication Strategy and Plan. Cairo. ——. 2007a. “Pandemic Influenza Communication Strategy.” Draft, November 2007. Cairo: ——. 2007b. Integrated National Plan for Avian and Human Influenza, 2007–2008. Cairo.: ——. 2008. Egyptian Demographic Health Survey. Cairo. GOVS. 2009. Knowledge Attitudes and Practices HPAI Bird-to-Bird: Qualitative and Quantitative Research Reports. Cairo: General Organization for Veterinary Services. ——. 2010a. Results of Evaluation of SAIDR Project Field Activities. Cairo: General Organization for Veterinary Services. ——. 2010b A Simplified Guide of Minimum Biosecurity Practices for Improved Poultry Production in Egypt. Cairo: USAID, FAO and STOP AI. Herfst, S, et al. 2012. “Airborne Transmission of Influenza A/H5N1 Virus Between Ferrets.” Science 336: 1534–41. 60 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Hess, R. ND. “Egypt Avian Influenza Communication.” Power Point Presentation prepared for SAIDR. Cairo: Johns Hopkins University, Center for Communication Program, Communication Healthy Living Project (CHL). Hosny, Farid. 2012a. HPAI Egypt Predicament. Prepared for USAID/STOP AI. ——. (2012b). Concept Paper: Reducing Socio-economic HPAI Negative Implications. Prepared for USAID STOP AI. Imai, M. et al. 2012. “Experimental Adaptation of an Influenza H5 HA Confers Respiratory Droplet Transmission to a Reassortant H5 HA/H1N1 Virus in Ferrets.” Nature 486: 420–28. Lambert, Virginia, and Hanan Radwan, 2012. USAID, Gender Assessment of Avian Influenza in Egypt. Washington: USAID. Lohiniva, A.-L., E. Dueger, M. Talaat, et al. 2012. “Poultry Rearing and Slaughtering Practices in Rural Egypt: An Exploration of Risk Factors for H5N1 Virus Human Transmission. Influenza and other Respiratory Viruses doi: 10.1111/irv.12023. MOALR. 2010. Integrated National Plan for Avian and Human Influenza. Animal Health and Livelihood Sustainability Strategy. Revised 2010. Cairo: Arab Republic of Egypt. Ministry of Health and Population. 2008. Preparedness Plan for Pandemic Influenza. Cairo: MOHP, Arab Republic of Egypt. ——. ND. “Avian H5N1 Influenza Situation and Best Practice, Egypt 2006-2010.” PowerPoint presentation by Dr. Samir Refaey and CHL..Cairo: Ministry of Health. ——. Quarterly and Annual Reports submitted for SAIDR Project. Cairo: MOHP. Neumann, G., et al. 2012. “Egyptian H5N1 Influenza Viruses—Cause for Concern?” PLoS pathogens 8(11) e1002932. soi:10.1371/journal.ppat.1002932. Peyre, M. et al. 2009. “Avian Influenza Vaccination in Egypt: Limitations of the Current Strategy.” Journal of Molecular Genetic Medicine 3 (2): 000-000. Refaey, Samir. ND. Influenza Situation and Best Practice, Egypt 2006-2010. Cairo: Ministry of Health, Government of Egypt. Safir, Basil. 2008. “Mobilizing the Egyptian Community for Avian Influenza Bio-Security Preparedness.” Power Point presentation prepared for Global Health Council Conference May 2008. Sahar, F., et al. ND. “Could Avian Influenza Be Contained in Rural Egypt through a Community Outreach Behavior Change Program?” Power Point Presentation. SAIDR Project. Various dates. Communication materials. Developed with Ministry of Health and Population, Government of Egypt and Communication for Healthy Living. Sedyaningsih, E., S. Isfandra, T. Soendor, and S. Supari 2008. “Toward Mutual Trust, Transparency and Equity in Virus Sharing Mechansims: The Avian Influenza Case of Indonesia.” Annals of the Academy of Medicine, Singapore 37: 482–88. SPAN Consultants. 2010 Evaluation Summary of Avian Influenza Community Education Interventions in Rural Egypt. Prepared for UNICEF by SPAN Consultants. STOP AI (2009a). API Quarterly Reports. ——. 2009b. Egypt Work Plan. USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 61 Storey, D., R. Hess, Jennifer Kreslake, and Basil Safi. 2013. “Hitting a Moving Target: Risk Volatility and Egypt’s Avian Influenza Campaigns, 2006–2009.” Presentation at the Health Communication Division, International Health Communication Association Annual Conference, London, June 2013. Swayne, D. 2011. Avian Influenza Vaccine Efficacy Project (AIVEP). Project final report. UN. 2010. Joint United Nations Assessment of Government of Egypt H5N1 Control Efforts, 6-16 December 2009.” Final Report. 9 February. New York: United Nations. UNICEF/Egypt. 2012. “Avian Influenza Power Point Presentation.” Cairo: UNICEF/Egypt. ——. ND. Communication and education materials related to avian and pandemic influenzas. USAID/DELIVER. ND. Shipping, inventory, and delivery documents related to personal protective equipment. van Engelen, A. 2011. A Preliminary study into Egypt’s Poultry Value Chains, Their Oorganisation and the Role They Can Play in the Control of HPAI. Cairo: FAO/AGAH and FAO-ECTAD. Vergne, T., V. Grosbois, Y. Jobre, et al. 2012 “Avian Influenza Vaccination of Poultry and Passive Case Reporting, Egypt.” Emergency Infectious Diseases. Wilsmore, T. 2008. Report on an Avian Influenza ECTAD Mission in Egypt, 14 May to 4 July 2008. ——. ND. Government of Egypt—Strengthening HPAI Detection and Response Project. Surveillance Proposal. 62 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 63 APPENDIX D. DATA COLLECTION AND ANALYSIS DATA COLLECTION METHODOLOGY Three evaluation instruments were used to conduct this evaluation: 1. Literature review (quantitative, and qualitative) 2. In-Depth interviews (qualitative) 3. Focus group discussions (qualitative). Literature Review. Because of the life span of these six projects, there was an abundance of documentation, including project documents, quarterly and final reports, consultant and other special reports, scientific publications, communication materials, and surveys. Monitoring and evaluation documentation included initial PMPs or logical framework matrices, KAP surveys, training assessments, back-to-office reports of FAO headquarters staff, and program reviews and evaluations. Much of the literature was provided electronically for the team to review prior to travelling to Egypt. Where gaps were identified, further documentation was sought and/or specific technical questions were scheduled, to obtain particular information from interviewees. Not all of the documentation sought was obtainable in the time available and efforts in pursuing this were based on the importance of particular information. For example, there was almost no information available for the DELIVER Project, apart from one PPE delivery report. However, since this globally implemented project represented a very small component of the Egypt Program, and since inquiries satisfied the evaluation team that the materials supplied met personnel needs, the lack of project documentation was considered of low priority for assessing program performance. In-Depth Interviews(IDI). Interviews were conducted with 16 key players who managed and implemented the API activities. A general questionnaire covering the five overarching evaluation issues was used in each IDI and a series of technical questions were asked of specific technical interviewees, for example, communication-focused questions for UNICEF and CHL technical experts and diagnostic laboratory questions for the NLQP experts. Most of the key informants were identified by the USAID Mission and meetings scheduled for the team. Other meetings were arranged by the Mission at the request of the team or scheduled directly by team members where professional contact had already been made. Two field visits were made, one to Gharbeya and the other to Fayoum governorates, to conduct IDIs but also to gain a first-hand impression of the circumstances in which the program had been implemented and some of the constraints on achieving successful outcomes. The visits included discussions with governorate and district veterinary staff, visits to local poultry markets, and visits and discussions with commercial and household poultry producers. Focus Group Discussions (FGD). To evaluate the practical impact and application of the API program on real beneficiaries, two FGDs were conducted, one with commercial poultry producers who participated in program activities and the other with animal health care 64 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM providers who received training within the program. With the commercial poultry producers, their opinions were sought on the value of the information provided by program resources and the benefits they realized as a result of participating in workshops and on-farm activities, particularly with respect to improving biosecurity for poultry production. With the animal health care trainees, their opinions were sought on the quality of training provided in specific technical areas and their ability to apply that training in the course of their daily activities. DATA COLLECTION INSTRUMENTS Questionnaire for In-Depth Interviews (IDI) For the IDI a two part questionnaire was developed. The first part comprised nine questions that were designed to capture the five evaluation questions described in the Scope of Work. The second part was made up of technical questions for experts in the various technical areas. For example, a series of questions was developed specific to the animal and human health and communication respondents. The questions were open-ended, seeking opinion and elaboration rather than yes or no answers. Following are the questions for the IDI. The nine questions in Part 1 address evaluation Questions #1, #2, #4 and #5. Question #3 is captured in Part II, the technical section. Part 1. All Audiences 1. Are you familiar with the USAID Avian and Pandemic Influenza (API) program? 2. If you participated in the program, what was your role in one or more specific projects? 3. A major overall objective of the API program was to increase awareness and prevent the virus from spreading in Egypt. From your experience with the program, what would you identify as the successes in meeting this objective and what would you identify as issues that hindered this objective from being met? Probe: Was the API program sustainable? 4. From your experience, what were some of the overall highlights of the API program, and what were the highlights of your specific work? Probe: How effective was the management structure? 5. From your experience, what were the gaps in the overall API program? Probe: What helped or hindered your ability to do your work given the API program’s design and objectives? 6. Were the partners you worked with—partners being identified as the national or local government entities, civil society, and private sector—receptive to the API program? If so, how? If not, why not? 7. Was your work specific to men, women, or both genders? Why? What is the approximate proportion of men and women in your profession or area of work? 8. If the program were continuing, or a new program started now, what would be the top three recommendations you would make for the next Avian and Pandemic Influenza Program? 9. Is there anything else you would like to tell us about your work or overall about the API Program? USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 65 Part 2. Technical Audiences Behavior Change and Communication I.R. 3.1 and I.R. 3.2 Research 1. Behaviors: How were the behaviors determined? 2. Research: Was there a baseline established? 3. Impact: How was impact measured for increase in awareness? 4. Impact: How was impact measured for changes in behaviors? 5. Who were target audiences? Probe: stakeholders, at-risk audiences, general populations 6. Media channels/Sources of information: What were the determinants for communication dissemination? 7. Pre-testing: Were materials and messages pre-tested? If so, how and by whom? Coordination 1. How did you coordinate your activities with the government, other donors, the UN, i.e., UNICEF, FAO, WHO communication? Probe: Did this include tasks or assignments? 2. Was there a task force or steering committees in Egypt to manage the API response? Did JHU/CHL participate in meetings or have a seat on the committee? 3. Was this effective? Probe: timelines, approvals, strategy and direction 4. Could it have been improved? 5. How large was the project staff? Planning and Implementation 1. Did Save the Children implement the activities? Is this true for both H5N1 and H1N1 influenzas? 2. What geographic areas were chosen in which to implement? Why those communities? 3. Were there local partners that implemented? Probe: Production companies, TV stations, research firms, local NGOs 4. Was training conducted for the awareness and behavior change components? Probe: Media training, health providers, social mobilizers? 5. For H1N1 pandemic influenza, was there a rapid response requirement? Did the government and appropriate officials assist with the rapid response? 6. Was there involvement with local resources to design and implement the project (objective to transfer capabilities to the country)? Long View 1. What are the successes? 2. Where are opportunities for changes? 66 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Technical Questions for Animal Health Evaluation Question 1. To what extent has the API program achieved intended goals? Subgoal 1: Prevent H5N1 infections in birds 1. Has the incidence or spatial distribution of outbreaks of HPAI been reduced over the program period? 2. What would be the expected current HPAI situation in Egypt now if the program had not been implemented? 3. Are quantitative data available to measure the impact of the program? Sub-goal 2: Prevent human exposure to H5N1 virus 1. What is the trend in incidence of human cases of H5N1 infection? 2. Is available data of sufficient consistency to enable analysis of any trend? 3. Have the risk factors for human exposure been clearly identified? Goal: Prevent H5N1 infections in humans 1. Is there evidence that the program has been effective in reducing the incidence of infection in humans? 2. Has the program assisted in providing facilities and procedures that will minimize future human H5N1 infection and mortality? Evaluation Question 3. To what extent were the technical components and approaches on the API program effective? 3.1 SAIDR Project Sub-Goal 1–Prevent H5N1 infections in birds 1. Has the program been successful in reducing outbreaks of HPAI? 2. If yes, what have been the main elements of assistance that have contributed to success? 3. If no, what were the main factors contributing to lack of success and how could the program have performed better? IR 1.1 Preparedness and planning 1. What was the general approach to HPAI control and was it appropriate? 2. Were program recommendations for strategy consistent with government policy and were the policies appropriate, e.g., compensation? 3. Was there sufficient epidemiological information available to inform an appropriate strategy? 4. Was there adequate capacity within veterinary services to undertake HPAI control, and was this appropriately considered in the planning? IR 1.2 Preventing future outbreaks 1. How were improved biosecurity measures for backyard poultry promoted and implemented? 2. Were the measures realistic in terms of ability and willingness of producers to implement them? USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 67 3. Was training of staff in communication skills and dissemination of knowledge to producers and other industry participants effective, and how was this measured? 4. How was the backyard sector vaccination campaign monitored and what was the contribution of the USAID program to developing vaccination strategies? 5. How successful was the program in identifying virus transmission risk along the value chain and promoting attention to this with GOVS? 6. Were measures for improving market hygiene adopted by stakeholders? IR 1.3 Detection of H5N1 in poultry and wild birds 1. How successful was the planning and implementation of active surveillance? How were data collected, analyzed, and used? 2. Was passive surveillance (suspect disease reporting or participatory disease search) effectively implemented? 3. Was wild bird surveillance informative? 4. Was design of a highly secure laboratory undertaken and was it appropriate? IR 1.4 Improved outbreak containment measures 1. How was the effectiveness of outbreak response measured? 2. Was the program successful in assisting to improve outbreak response? 3. How was the issue of compensation addressed? Is there now an acceptable and equitable policy in place? 4. Is there evidence of improved compliance of communities as a result of communication efforts to limit spread of virus and exposure of humans? 5. Are SOPs for outbreak response available for review? IR 1.5 Limit exposure of personnel to H5N1 virus 1. Were PPE kits well-distributed and generally available to personnel undertaking outbreak control? 2. Was the proper use of PPE monitored? 3. How were producer families counseled in outbreak circumstances and what was the communication linkage with public health authorities? 3.2 AIVEP Output 1: Identification of circulating H5N1 virus strains 1. Is isolate collection and screening still being undertaken? 2. If so, how are candidate viruses selected for cartography? Output 2: Challenge testing of representative antigenic variants and vaccine strains selection 1. What was the outcome of demonstrating vaccine-resistant strains? 2. Capacity building was not a documented project output but appears to have been a valuable consequence. How well-resourced is the laboratory now to continue its work? 68 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 3. Is collaboration still continuing with SEPRL? 3.3 Improved biosecurity and hygiene Output 1: Target poultry farms identified and prioritized 1. What was the outcome of the biosecurity evaluation of farms? 2. How was a decision made to use either Virkon or citric acid as a disinfectant? What is the cost difference? Output 2: Farms cleaning and disinfection guidelines and protocols developed and agreed upon with veterinary officials and local 1. Were draft guidelines (as in final project report) finalized? Output 3: Sufficient staff and workers adequately trained to undertake cleaning and/or disinfection activities in targeted project areas 1. Can we see the database on prevailing risk behaviours? 2. Can we have a focus group discussion with trainees to assess their level of knowledge? Output 4: Cleaning and disinfection operations implemented and monitored in all targeted project sites 1. May we see the qualitative checklist for hygiene improvement? 2. Has composting been adopted for carcase and waste disposal? 3. Did farm gate operations concentrate on bio-exclusion (materials coming in) or bio￾containment (materials going out)? 3.4 Developing and maintaining PPP Output 1: Roles of the stakeholders of public and private sectors involved in the poultry production and marketing chains assessed, clearly defined, distributed, and agreed in regard to the control of HPAI and other animal diseases 1. Is the baseline survey available for review? Output 2: Capacity of public veterinary services strengthened to lead the development and management of the animal health systems to prevent, detect, and control HPAI and other animal diseases 1. Who was trained in the public sector to develop and lead public-private alliances in animal disease control? Output 3: Lessons learned from cases of PPP in key areas of disease control (vaccination, compensation, biosecurity) are available for replication 1. How did the project contribute to the development of compensation policy and what is the current status? Output 4: Forums and networks of public-private stakeholders in place for enhanced communication, education, information dissemination, and awareness for HPAI and other animal diseases prevention, detection, and control. 2. Are the Supreme and Technical Committees on HPAI still in place? 3.5 STOP AI Objective 1: Establish a PPP with the Poultry Union, individual Poultry Union members, or other commercial partners to provide technical assistance in biosecurity upgrading USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 69 and other services for Sector 2 and 3 broiler grow-out, layer, hatchery, and breeder farm managers and owners 1. Has there been any assessment of improvement in biosecurity in Sector 2 and 3 farms in the project areas? 2. How successful were attempts to extend advice to poultry farmers in other governorates? Was the biosecurity video produced and distributed? 3. Are there any quantitative data indicating improved biosecurity of Sector 2 and 3 farms or of reduced HPAI incidence in target farms or governorates? Objective 2: Provide coordinated support for FAO’s proposed decontamination program 1. Please confirm that this activity was not conducted due to changed FAO activities. How was the budget re-allocated? Objective 3: Provide training materials for use by FAO and GOVS to provide biosecurity and outbreak response training to GOVS veterinarians 1. Is GOVS committed to emphasizing farm and market chain biosecurity measures as a priority for HPAI control? How is it rated in comparison to outbreak response as a control measure? Sample Technical Questions for Human Health Interviewees In addition to the SAIDR specific goals (4), objectives (15) and intermediate results (IR), sample focus questions for the human health side (MOHP) include 1. What was the impact of the H5N1 avian influenza-pandemic preparedness efforts between winter 2006 and on the spring-summer 2009 pandemic influenza H1N1 in Egypt? a. Availability of personal protective equipment (PPE): were more PPEs available? b. Availability of anti-influenza medications, such as oseltamivir: was more available? c. Better infection prevention and control policies in place in hospitals and clinics? d. Better clinical evaluation and treatment protocols (guidelines) pre-pandemic H1N1 in 2009 as a result of API 2006–2009 in Egypt? 2. How were the funds reallocated by USAID in 2009 from H5N1 API to pandemic H1N1 influenza used, and what impact did they have on the pandemic response in Egypt? 3. How did the influenza H1N1 pandemic that started in Egypt in June 2009 impact subsequent (2010–today) overall avian influenza H5N1 preparedness and response? a. For example, did the perception of the threat of HPAI H5N1 cause a pandemic decrease after 2009? b. Or did the perceived threat of a future influenza pandemic increase due to the potential of a recombinant (or “hybrid”) H5N1 and pandemic H1N1 virus that was readily transmissible from person-to-person? (Note: In May 2012 the in vitro laboratory creation of such a recombinant virus between H5N1 [Vietnam clade 1] and pandemic H1N1 was published by a laboratory in the U.S.) End of Interview 70 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Focus Group Discussions (FGD) FGD were designed to gauge the current situation and practices of the beneficiaries of the API program. Interviews with two target audiences, one of which had been recipients of specific training or workshops to strengthen their capabilities in a specific technical areas, i.e., surveillance, evaluation, bio-security; and the other an at-risk audience that was targeted to adopt best practices to reduce the risk of introduction of HPAI. Following are the FGD Guides for the focus groups. Focus Group Discussion (FGD) Guides Commercial Poultry Producers (Sectors 2 and 3) The learning objective of the FGD is to determine the level of awareness of poultry farmers of avian influenza in poultry and human infection with the virus and changes to their behavior that can prevent infection in their flocks and avoid transmission of the disease to humans— addressing IR 3.1 and 3.2 of the SAIDR Project. Introduction to Focus Group Respondents Hello, my name is (name of facilitator) and I am a (occupation). I and my colleagues are part of an independent evaluation team that is conducting an assessment of the USAID/Egypt Avian and Pandemic Influenza Program. The program was comprised of six projects or activities that were designed to prevention infection in birds and to avoid the transmission of the disease to humans. Because you are poultry producers and a key audience for the Avian and Pandemic Influenza program, we wanted to ask you about your awareness and poultry-raising practices that you follow to prevent the disease. I/We appreciate your participation in this assessment and it is very important that we hear from you about your familiarity with the prevention and practices that were promoted. We’d like to learn about you. So if each of you would introduce yourself, where you live, what type/kind of poultry you raise, how many birds, and what type of poultry production you have. Respondents introduce themselves 1. Tell us about your farm—where it is, how many birds you have, whether they are breeders, broilers or layers, where you market your produce . . . . 2. Are you familiar with bird flu? How did you learn about it? Have you had outbreaks in your poultry? 3. What were your primary concerns about the disease? Probe: economic, health, safety, profession 4. How would you rate bird flu in comparison with Newcastle disease in importance? 5. Do you vaccinate your birds for bird flu? 6. Did you have visits from animal health workers to discuss the disease? Have you been invited to any meetings on the disease? 7. Do you know how the virus can be transmitted to humans? 8. Do you know about the signs of the disease in humans and how it is transmitted? USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM 71 9. Do you know how to prevent the disease being transmitted to humans? 10. Do you consider the disease to be a threat to your poultry production or to the health of your family? End of FGD Animal Health Training for Animal Health Care Workers The objective is to undertake a discussion with a group of animal health professionals who received training under part of the program, to assess their knowledge of key aspects of HPAI prevention and control. This is addressing elements of IRs 1.2, 1.3, and 1.4. The facilitator will introduce the discussion, explaining the context of the evaluation and emphasizing that the questions are directed at assessing the quality of training, not the knowledge of particular individuals. 1. Introductory question: job title and duties of each person, training they received (where, when, under which project). 2. What are four key behaviors to prevent HPAI introduction into farms? Probe: all-in/all-out production—no introduction of birds; decontamination of workers on entry; uncontaminated food and water; protection from wild birds 3. How can you prevent movement of H5N1 virus along the market chain, i.e., through markets and transport vehicles? Probe: inspection of source flocks and birds before going to market; decontamination of vehicles after unloading birds or before loading birds; disinfection of marketplaces; no birds to be returned to farms from markets 4. What are the key elements of an outbreak response? Probe: reduce viral load by slaughter of infected birds; quarantine and movement control to prevent birds being disseminated; disease search around outbreak to detect source and spread; cooperation of community to report disease and respect movement control 5. How would you rank the following in importance for long-term HPAI reduction in Egypt: disease surveillance; biosecurity; outbreak response? End of FGD DATA ANALYSIS The great majority of the information collected was qualitative. From review of program documentation, the team gained insights into the activities that were planned and the extent to which the planned activities had been successfully implemented. IDIs and FGDs corroborated or otherwise the information coming from document reviews, and discussions with a broad range of stakeholders enabled the team to make evidence-based judgments on program performance, asking key questions of multiple informants and allowing for potential bias of informants, depending on their particular stake in program outcomes. Quantitative data were particularly related to the extent to which activities had been implemented or improved practices adopted. Information was generally not subjected to statistical analysis as the numbers involved were too small or the measurement was not critical. For example, a finding that 64% of farmers who received assistance in improving their farm biosecurity successfully applied the information gained was an outcome that stood as an indicator of satisfactory performance of that program activity. 72 USAID/ EGYPT: END OF PROJECT PERFORMANCE EVALUATION OF AVIAN AND PANDEMIC INFLUENZA PROGRAM Of particular note is that it was not possible to make a quantitative assessment of whether there has been real progress in reducing the incidence of HPAI. The bulk of the data available was from passive and active disease surveillance activities, the intensity of which varied over the course of the program. Changes in reported numbers of HPAI cases were therefore not a reliable indicator of changes in incidence of the disease. Document review and IDIs provided information to address each of the five evaluation questions. Particular focus was given in IDIs to addressing issues of program performance and monitoring and also lessons learned. The FGDs were particularly useful for addressing Evaluation Question 3 (technical components and approaches) and Question 5 (lessons learned). For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com