EVALUATION FUNDING FACILITY FOR STABILIZATION MIDTERM PERFORMANCE EVALUATION REPORT USAID/IRAQ PERFORMANCE MANAGEMENT AND SUPPORT PROGRAM FINAL REPORT SUBMITTED APRIL 6, 2020 Cover Page Photo: A class in session in October 2017 at Al Thibyania Primary School in Nimrud, recently renovated by UNDP’s Funding Facility for Stabilization. Photo Credit: UNDP. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development. This publication was produced at the request of the United States Agency for International Development. It was prepared independently by IBTCI. ABSTRACT A midterm performance evaluation by the Iraq Performance Management and Support Program identified factors that promoted or impeded outcomes from Funding Facility for Stabilization (FFS) interventions from July 2015 through June 2018. The Evaluation Team used seven quantitative and qualitative data collection methods, sampling beneficiary communities, 10 stakeholders representing 10 intervention areas, and stakeholders from local government partners and the United States Agency for International Development (USAID). Investigation of four Evaluation Questions resulted in four Key Conclusions: 1. How have FFS activities affected access to services in target communities? FFS positively affected access to services in all targeted communities. Targeted water, electricity, education, and health services were restored to pre-ISIS levels with limited sanitation services. 2. How have FFS activities affected the quality of services in target communities? FFS activities restored the quality of basic services to pre-2015 levels. Furthermore, to improve quality of service, education and health services required additional human capacity and technical knowledge. 3. How have FFS activities affected IDPs in target communities? FFS addressed factors that influenced decisions to return or stay (respondents who never left) for up to 86 percent of the population, although not all services are equally important and none were cited by a majority of respondents as the critical reason for return. 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Various factors contributed to or impeded local authorities’ capacity to administer FFS services depending on the province or sector, including local authorities’ low technical capacity to administer education services in all three provinces, and across all Ninewa sectors. Evaluation Team: Robert Grossman-Vermaas, Team Leader Elizabeth Cunningham, Subject Matter Specialist Samuel Malmberg, Senior Data Analyst Wesley Thompson, Data Analyst Field Team: Ghazanfar Hoti, Field Team Leader Willie Kachaka, Methodologist Sahib Haq, Livelihoods Specialist i TABLE OF CONTENTS EXECUTIVE SUMMARY ix INTRODUCTION ix METHODOLOGY x FUNDAMENTAL OBSERVATIONS – WHAT HAS FFS ACHIEVED FROM 2015 TO 2018? xii CONCLUSIONS AND RECOMMENDATIONS xiii QUESTION 1: HOW HAVE FFS ACTIVITIES AFFECTED ACCESS TO SERVICES IN TARGET COMMUNITIES? xiii QUESTION 2: HOW HAVE FFS ACTIVITIES AFFECTED THE QUALITY OF SERVICES IN TARGET COMMUNITIES? xv QUESTION 3: HOW HAVE FFS ACTIVITIES AFFECTED IDPs IN TARGET COMMUNITIES? xvi QUESTION 4: WHAT FACTORS ARE CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES TARGETED BY FFS? xvii INTRODUCTION 1 1. PURPOSE AND OBJECTIVES 1 EVALUATION QUESTIONS 3 2. BACKGROUND 4 FFS DEVELOPMENT HYPOTHESIS 5 EVALUATION DESIGN 5 3. METHODOLOGY 6 3.1 Document Review 6 3.2 Key Informant Interviews 7 3.3 Focus Group Discussions (FGDs) 7 3.4 Population-Based Survey (PBS) 8 3.4.1 Survey Areas 8 3.4.2 Primary Sampling Units 9 3.4.3 Sample Size 10 3.4.4 Selection of Households and Respondents 11 3.4.5 Sampling Weights 12 3.4.6 Analysis 12 3.5 Livelihoods Survey (LS) 12 3.5.1 Sample Size 13 3.5.2 Selection Method of Respondents 13 3.5.3 Analysis 14 3.6 On-site Observations 14 3.7 Capacity Assessment of Local Authorities 14 3.8 Data Analysis Methods 14 3.9 Data Treatment Protocols 15 ii LIMITATIONS 16 Technical 16 Operational 17 FUNDAMENTAL OBSERVATIONS—WHAT HAS FFS ACHIEVED FROM 2015 TO 2018? 18 OBSERVATION 1 18 OBSERVATION 2 21 QUESTION 1: HOW HAVE FFS ACTIVITIES AFFECTED ACCESS TO SERVICES IN TARGET COMMUNITIES? 28 1A. CHANGES TO SERVICES AVAILABILITY IN TARGET COMMUNITIES RESULTING FROM FFS ACTIVITIES ACROSS ALL FFS TARGET SECTORS. 28 1B. IS ACCESS TO SERVICES EQUITABLE ACROSS VULNERABLE GROUPS? 31 1C. WHAT SIGNIFICANT BARRIERS, IF ANY, PREVENT MEMBERS OF TARGET COMMUNITIES FROM ACCESS TO SERVICES? 34 QUESTION 2: HOW HAVE FFS ACTIVITIES AFFECTED THE QUALITY OF SERVICES IN TARGET COMMUNITIES? 38 2A. SECTOR-SPECIFIC PARAMETERS OF QUALITY OF SERVICES TARGETED BY FFS 38 2B. PERCEPTIONS OF THE QUALITY OF SERVICES AMONG MEMBERS OF TARGET COMMUNITIES 39 2C. PERCEPTIONS OF THE EFFECTIVENESS OF GOVERNMENT ADMINISTRATION OF SERVICES AMONG MEMBERS 41 QUESTION 3: HOW HAVE FFS ACTIVITIES AFFECTED IDPs IN TARGET COMMUNITIES? 44 3A. TO WHAT EXTENT HAVE FFS ACTIVITIES INFLUENCED IDPS’ DECISIONS TO RETURN TO THEIR COMMUNITIES OR INFLUENCED OTHERS TO REMAIN IN THEIR COMMUNITIES RELATIVE TO OTHER FACTORS? 44 3B. IN WHAT WAYS HAVE FFS ACTIVITIES AFFECTED IDPs’ LIVES SINCE THEIR RETURN TO THEIR COMMUNITIES? 61 QUESTION 4: WHAT FACTORS ARE CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES TARGETED BY FFS? 67 4A. SOURCES AND AVAILABILITY OF OPERATIONS AND MAINTENANCE BUDGETS FOR REHABILITATED SERVICES 67 4B. CAPACITY TO COLLECT REVENUE 68 4C. ORGANIZATIONAL CAPACITY OF LOCAL ADMINISTRATIVE OFFICES 68 4D. TECHNICAL AND MANAGEMENT CAPACITY OF STAFF 70 ANNEX 1: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS MATRIX 74 ANNEX 2: SCOPE OF WORK 84 ANNEX 3: EVALUATION DESIGN 98 ANNEX 4: LIST OF KEY INFORMANT INTERVIEWS 218 ANNEX 5: DATA COLLECTION TOOLS 220 KEY INFORMANT INTERVIEW GUIDE 220 FOCUS GROUP DISCUSSION FORM 237 POPULATION BASED SURVEY 248 iii LIVELIHOODS QUANTITATIVE SURVEY 268 ANNEX 6: LANDSAN FILES 300 ANNEX 7: SIGNED CONFLICT OF INTEREST FORMS 301 ANNEX 8: STATEMENT OF DIFFERENCE 309 ANNEX 9: INFOGRAPHIC 318 iv TABLE OF TABLES Table 1: Overview of Methodology .............................................................................................................................. x Table 2: Evaluation Questions........................................................................................................................................ 3 Table 3: Proposed Data Collection Methods for the Evaluation Questions....................................................... 6 Table 4: Number and Distribution of FGDs............................................................................................................... 8 Table 5: Distribution of Facilities in the Provinces and Intervention Areas ........................................................ 9 Table 6: Assumptions for Sample Size Calculation..................................................................................................10 Table 7: Proposed Sample Allocation in the Provinces and Intervention Areas..............................................11 Table 8: Summary of Livelihoods’ Program Beneficiaries ......................................................................................13 Table 9: Assumptions for Sample Size Calculation..................................................................................................13 Table 10: Changes to Education Access pre-2015 vs. 2019 – Average Responses across PBS Respondents......................................................................................................................................................................30 Table 11: List of On-Site Observations......................................................................................................................38 Table 12: PBS Respondents’ Satisfaction with Administration of Services........................................................42 Table 13: Top Five Factors Influencing Return by Intervention Area, in the Year(s) of Highest Returns . 46 Table 14: Percentage of Returnees by Year and Intervention Area....................................................................47 Table 15: Comparison of factors influencing return as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. ................................................................................................................................................................................48 Table 16: Comparison of factors influencing return in Falluja as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors.................................................................................................................................................50 Table 17: Comparison of factors influencing return in Qayara as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors.................................................................................................................................................52 Table 18: Comparison of factors influencing return in Karma as reported by surveyed returnees who returned in 2016 and 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors........................................................................................................................................53 Table 19: Comparison of factors influencing return in Ninewa as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors.................................................................................................................................................54 Table 20: Comparison of factors influencing return in Ramadi as reported by surveyed returnees who returned in 2016 and 2017 (the highest years of returns) and number of FFS projects started that same year relating to those factors........................................................................................................................................56 Table 21: Comparison of factors influencing return in Shirqat as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors.................................................................................................................................................57 Table 22: Comparison of factors influencing return in Tikrit as reported by surveyed returnees who returned in 2015 and 2016 (the highest years of returns) and number of FFS projects started that same year relating to those factors........................................................................................................................................59 Table 23: Comparison of factors influencing return in West Mosul as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. ................................................................................................................................................60 Table 24: Ministry Budgets and Expenditures, 2018 ...............................................................................................67 Table 25: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Anbar. .........................................................................................................71 Table 26: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Salah al-Din................................................................................................72 Table 27: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Ninewa. ......................................................................................................73 v TABLE OF FIGURES Figure 1: Restoration of Destroyed Basic Services by intervention Area – Percentage of PBS Respondents.....................................................................................................................................................................xiv Figure 2: Critical factors and other considerations in decisions to return or stay across full population xvi Figure 3: FFS Activity and Military Defeat in Iraq....................................................................................................... 2 Figure 4: Example of grid cells in Landscan™ of Ramadi .......................................................................................10 Figure 5: Percentage of Households by Intervention Area that Perceived Destroyed Services to be Restored ............................................................................................................................................................................28 Figure 6: Percent of Households by Intervention Area that Perceived Destroyed Education and Health Services to be Restored.................................................................................................................................................29 Figure 7: Average Distance (km) Household Members Must Travel to Nearest Functioning Government Health Facility 2019 vs. Pre-2015 – Percentage of PBS Respondents................................................................. 31 Figure 8: Access to Basic Services by Religious Affiliation.....................................................................................32 Figure 9: Access to Basic Services by Respondents’ Status – Percentage of PBS Respondents...................32 Figure 10: Access to Basic Services by Ethnic Origin – Percentage of PBS Respondents..............................32 Figure 11: Percentage of Households with School-aged Children Attending School by Religion, Ethnicity, and Status ..........................................................................................................................................................................33 Figure 12: Average Distance in Kilometers Households Must Travel to Nearest Functioning Government Health Center by Ethnicity, Religion, and Status......................................................................................................33 Figure 13: Percentage of Households' Hours of Connectivity to Electricity in a Given Day 2019 v pre￾2015 ....................................................................................................................................................................................40 Figure 14: Perception of Water Quality in 2019 Compared to Pre-2015 - Based on Percentage of Households that Perceived the Taste and Smell to be Good and the Clarity to be Clear...........................41 Figure 15: Percentage of Households that Perceive the Quantity of Potable Water is Adequate for Daily Needs Today.....................................................................................................................................................................41 Figure 16: Drivers of return as reported by surveyed returnees and addressed by FFS during year(s) of highest return by intervention area.............................................................................................................................45 Figure 17: Number of FFS projects started and number of surveyed returnees, by year, for East Mosul 49 Figure 18: Number of FFS projects started and number of surveyed returnees, by year, for Falluja.........51 Figure 19: Number of FFS projects started and number of surveyed returnees, by year, for Qayara.......52 Figure 20: Number of FFS projects started and number of surveyed returnees, by year, for Karma........54 Figure 21: Number of FFS projects started and number of surveyed returnees, by year, for Ninewa......55 Figure 22: Number of FFS projects started and number of surveyed returnees, by year, for Ramadi.......56 Figure 23: Number of FFS projects started and number of surveyed returnees, by year, for Shirqat.......58 Figure 24: Number of FFS projects started and number of surveyed returnees, by year, for Tikrit..........59 Figure 25: Number of FFS projects started and number of surveyed returnees, by year, for West Mosul ..............................................................................................................................................................................................60 Figure 26: LS responses to “Is overall life improving for people who returned?” ...........................................61 Figure 27: Percentage of LS responses to “Was the money you received from C4W adequate for your family’s needs?”................................................................................................................................................................. 62 Figure 28: Percentage of LS responses to “Overall has life improved since your participation in the C4W activity?” .............................................................................................................................................................................62 Figure 29: LS responses to: “How would you compare your household income to the period before the C4W support?”................................................................................................................................................................63 Figure 30: Mean amount of money received from the C4W for the period enrolled by sex ......................63 Figure 31: Mean amount of money received from the C4W activity for the period enrolled by sex and province. ............................................................................................................................................................................63 Figure 32: Mean amount of money received from the C4W activity for the period enrolled by ethnic origin...................................................................................................................................................................................64 Figure 33: Mean amount of money received from the C4W activity for the period enrolled by ethnic origin and province..........................................................................................................................................................64 Figure 34: Mean amount of money received from the C4W activity for the period enrolled by religion.65 vi Figure 35: Mean amount of money received from the C4W activity for the period enrolled by religion and province......................................................................................................................................................................65 Figure 36: Critical factors and other considerations in decisions to return or stay across full population ..............................................................................................................................................................................................66 Figure 37: Revenue Collection Capacity by Directorate and Province..............................................................68 Figure 38: Procurement Capacity by Directorate and Province..........................................................................68 Figure 39: Strategic Planning Capacity by Directorate and Province..................................................................68 Figure 40: Work Planning Capacity by Directorate and Province .......................................................................69 Figure 41: HR Management Capacity by Directorate and Province....................................................................69 Figure 42: M&E Capacity by Directorate and Province..........................................................................................69 Figure 43: IT Capacity by Directorate and Province...............................................................................................70 Figure 44: Technical Capacity across Provinces.......................................................................................................70 vii ACRONYMS LIST ADS 201 Automated Directive System 201 AI Appreciative Inquiry AO Agreement Officer C4W Cash for Work EQ Evaluation Question EU European Union FCAS Fragile and Conflict-Affected States FFES Funding Facility for Expanded Stabilization FFIS Funding Facility for Immediate Stabilization FFS Funding Facility for Stabilization FGD Focus Group Discussion G2A Getting to Answers GOI Government of Iraq IBTCI International Business and Technical Consultants Inc. IDP Internally Displaced Person IFM Iraqi Field Monitor IGPA Iraq Governance and Performance Accountability Project IIACSS Independent Institute and Administration Civil Society Studies IOM International Organization for Migration IP Implementing Partner ISIS Islamic State of Iraq and Syria KII Key Informant Interview LS Livelihoods Survey M&E Monitoring and Evaluation MSC Most Significant Change NGO Non-Governmental Organization OCA Organizational Capacity Assessment OCHA United Nations Office for the Coordination of Human Affairs O&M Operations and Maintenance PBS Population-Based Survey PIO Public International Organization PMSP Performance Management and Support Program viii REM Religious and Ethnic Minority RI Refugees International SOW Scope of Work UNDP United Nations Development Programme USAID United States Agency for International Development USG United States Government USIP United States Institute of Peace ix EXECUTIVE SUMMARY INTRODUCTION Under contract AID-267-TO-17-00001, International Business and Technical Consultants, Inc. (IBTCI), through its Performance Management and Support Program for Iraq (PMSP), has conducted a midterm performance evaluation to identify factors that promoted or impeded positive outcomes from Funding Facility for Stabilization (FFS) stabilization interventions from July 2015 through June 2018. The evaluation is meant to inform United States Agency for International Development (USAID)/Iraq stabilization and development programming, and other international donor-funded activities. The report consists of sections devoted to (a) fundamental observations the Evaluation Team identified through a holistic data analysis, and in particular a data analysis triangulated with primary and secondary sources relating to necessary and sufficient stabilization factors in fragile and conflict-affected states (FCAS); (b) answers to key Evaluation Questions (EQs) asked by USAID; and (c) actionable and utilization-focused recommendations derived from an analysis of the data from the evaluation’s seven data collection methods. The report opens with a description of the purpose and objectives of the evaluation, the methodology used to answer the Evaluation Questions, and a summary of the Findings, Conclusions, and Recommendations. It should be noted that the Evaluation Team reviewed and analyzed FFS interventions between its inception in 2015 and June 2018, so some EQs use a narrower time frame to allow for more granular analysis of correlation (specified where applicable in the report). PURPOSE AND OBJECTIVES The purpose of the evaluation is to identify factors that promoted or impeded positive outcomes from FFS stabilization interventions from July 2015 through June 2018 to inform ongoing and new stabilization activities and other USAID/Iraq programs. This evaluation was designed to evaluate the current FFS program and inform future USAID stabilization programming in Iraq and other complex, insecure environments. The main audiences of the evaluation include U.S. Mission Iraq, USAID/Washington Middle East Bureau, stabilization stakeholders and partners, and the Department of State. The objectives of this evaluation are threefold: 1. Evaluate the United Nations Development Programme (UNDP) FFS program, including components funded by USAID under the Funding Facility for Immediate Stabilization (FFIS) and the Funding Facility for Expanded Stabilization (FFES) (collectively referred to as FFS) in accordance with all applicable USAID Automated Directive System (ADS) regulations regarding midterm evaluations. 2. Identify and characterize contextual and programmatic factors associated with FFS achievement of results. 3. Provide specific, programmatic recommendations for corrective action if needed. The following were the Evaluation Questions that USAID sought to answer through the evaluation: 1. How have FFS activities affected access to services in target communities? 2. How have FFS activities affected the quality of services in target communities? 3. How have FFS activities affected internally displaced persons (IDPs) in target communities? 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? x METHODOLOGY The evaluation used a mixed-methods approach comprising seven data collection methods, as summarized in Table 1 below. Table 1: Overview of Methodology Data Collection Method Data Source Sampling Approach Rationale and Applicability Population-based survey 2,735 residents (returnees, IDPs in host communities, and those who never left) living in 10 FFS intervention areas Stage 1: Three FFS provinces identified per evaluation scope of work Stage 2: FFS intervention areas1 selected within three provinces based on number of eligible facilities2 Stage 3: Cluster-based population￾proportional sampling approach, based on population data from Landscan™ Representative sample of residents in intervention areas with highest number of eligible facilities to assess how FFS interventions have affected access to services (EQ1) and quality of services (EQ2), and addressed drivers of return (EQ3a) Livelihoods survey 730 Cash for Work (C4W) beneficiaries representing those for whom UNDP had contact information A stratified random sampling technique was used to calculate the sample size; respondents were then selected via systematic random sampling within each district Representative sample of a subset of C4W beneficiaries to assess how FFS activities have affected IDPs’ lives since their return to their communities (EQ3b) Key informant interviews (KIIs) 30 local authorities in directorates and municipalities related to FFS interventions, donor representatives, and IP staff Purposive based on involvement in FFS and/or familiarity with status of services in a given sector Used to contextualize or expand on survey findings for all evaluation questions Focus group discussions (FGDs) 12 homogeneous groups of nine to 12 participants in Anbar, Ninewa, and Salah al￾Din Purposive and convenience sampling with focus groups established by gender, returnee or IDP status, and religious and ethnic identity Used to contextualize or expand on survey findings related to how FFS interventions have affected access to services (EQ1), quality of services (EQ2), and IDPs in target communities (EQ3) 1 Selected intervention areas by province were Anbar (Ramadi, Falluja, and Karma), Ninewa (Qayara, East Mosul, West Mosul, and Ninewa Plains), and Salah al-Din (Tikrit, Mekeshifa, and Shirqat). 2 Eligible facilities include education, electricity, health, sewage, and water facilities supported by FFS interventions funded by USAID and other donors. xi Observation 24 facility-infrastructure sites Facility sites located within the cells selected for PBS data collection Used to assess how FFS interventions have affected the quality of electricity and water/sanitation services (EQ2) Document review FFS quarterly and annual reports, 467 PMSP third-party monitoring reports, reports from international nongovernmental organizations and government organizations, and Ministry of Finance data N/A Used to contextualize findings from primary data sources for all evaluation questions Rapid capacity assessment Provincial directorates of Water, Sewer Services, Municipality, Health, Education, and Electricity in Anbar, Ninewa, and Salah al-Din Purposive sampling of at least one senior representative from each of the seven core governance and technical departments for each directorate Used to identify factors contributing to or impeding local authorities’ capacity to administer services targeted by FFS (EQ4) xii FUNDAMENTAL OBSERVATIONS – WHAT HAS FFS ACHIEVED FROM 2015 TO 2018? FFS is the Stabilization Response of the anti-Islamic State of Iraq and Syria (ISIS) coalition’s liberation effort implemented by UNDP and governed by a multi-donor Steering Committee. FFS consists of two functional sequenced initiatives, FFIS and FFES. The U.S. Government (USG), through USAID, was one of the first donors to support the United Nations Development Programme (UNDP) Funding Facility for Immediate Stabilization (FFIS). As cities and territory occupied by ISIS were liberated, this multi-donor funded program provided critical support to the Government of Iraq (GOI) to help facilitate the safe and voluntary return of displaced populations, primarily in Anbar, Ninewa, and Salah al-Din. USAID supports two components of the FFIS. The first component supports public works and light infrastructure rehabilitation, such as health clinics, water infrastructure and treatment facilities, power grids and generators, schools, other government buildings, and access roads. The second component supports livelihood activities to jumpstart the local economy and generate income for local households, for example through grants to small businesses with high community impact (bakeries, electrical shops, agriculture, food kiosks, and micro-stores) and short-term employment through public work schemes, i.e. cash-for-work (C4W) opportunities. The report concentrates on the specific conclusions, findings, and recommendations of the respective Evaluation Questions, but the Evaluation Team also presents an aggregate of the conclusions, findings, and recommendations. This analysis is based both on the data and on the FFS objectives, and it suggests two fundamental observations, which are important to understanding FFS’s overall outputs and outcomes, and indeed its relative accountable successes, as evidenced in the evaluation findings and PMSP Iraqi Field Monitor data derived from third-party monitoring Site Visit Reports. Evaluation data support clear and evident perceptions of noticeable positive change at the province and intervention area levels from Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), and the Population-Based Survey (PBS) and Livelihood Survey (LS) respondents. The vast majority of self￾reported indicator data from FFS shows targets being consistently achieved. This is supported by findings from FFS outputs in annual and quarterly reports, with some qualitative data and studies and analysis undertaken on the impact of FFS, including those led by other donors and FFS in-house evaluation exercises which contain data sets for thousands of beneficiaries reached in both Central and Northern liberated areas. For example DFID supported the “Lessons Learned” exercise and the independent lessons learned exercise on Civil/Military coordination focused on Mosul. As such, the Evaluation Team suggests that for the remainder of the current FFS period of performance, USAID and FFS consider incorporating more meaningful monitoring and evaluation (M&E) criteria and normative projections of outcomes. The Evaluation Team suggests USAID and FFS better and more comprehensively track outcomes over time and during implementation rather than just at midterm or end-term of implementation, to better inform the refinement of FFS throughout the process cycle. The Evaluation Team understands that FFS monitors progress of outputs but suggests complementary and systematic monitoring of outcome- and process-based metrics. For example, for a more human￾centric, normative projection of outcomes, the Evaluation Team also proposes using two qualitative Observation 1. FFS is a coherently sound—and instrumental—formative activity to assuring stabilization in the conflict-affected regions of Anbar, Ninewa, and Salah al-Din. Assuming security is ensured, basic services delivery is a critical “push” and “pull” factor for returnees. Observation 2. FFS has had a demonstrable positive outcome on the lives of thousands of returnees. FFS activities have resulted in a measurable positive change in the delivery of essential services in key geographical areas since 2015. These can be evidenced with key “Success Stories.” xiii analysis techniques to better identify positive change contributed by FFS during its implementation: Most Significant Change (MSC) and Appreciative Inquiry (AI). Both techniques are participatory and should be conducted in a joint setting with a facilitator familiar with USAID and the affected regions of Iraq, and ideally with stabilization principles and M&E techniques. Unlike a midterm or final performance evaluation, both MSC and AI are conducted at regular intervals within a period of performance (e.g., quarterly), with purposively sampled groupings of the intervention population describing perceived changes to their lives. The regular periodicity of the perceptions allows for systematic qualitative, emotive data, which in turn can result in outcome harvesting to better identify successes and/or to identify needs for course correction. Finally, because measuring outcomes in stabilization contexts is challenging, USAID and FFS might also consider more meaningful techniques to gather, measure, and disseminate metrics of success. These criteria can include practical steps on how to monitor and measure progress toward the achievement of FFS objectives and means to learn from the monitoring and measurement findings and to adapt and include more relevant techniques such as process-based metrics so as to focus on the processes that lead to desired stabilization outcomes. Process benchmarks may also be more conducive to sharing across the U.S. interagency so as to describe how well the stabilization activity is working (e.g., as a function of programmatic cycle; specific sub-activity implementation time, completeness, efficiency, capacity, and cost-effectiveness). They can also be more easily digestible for policy and public audiences as a means to ensure transparency and accountability. CONCLUSIONS AND RECOMMENDATIONS QUESTION 1: HOW HAVE FFS ACTIVITIES AFFECTED ACCESS TO SERVICES IN TARGET COMMUNITIES? The Evaluation Team found that activities implemented by FFS positively affected access to services in all targeted communities within the intervention areas. Respondents from the PBS reported that water, electricity, education, and health services in FFS intervention areas surveyed by PMSP in Qayara, East Mosul, West Mosul, Ninewa Plains, Ramadi, Falluja, Karma, Tikrit, Mekeshifa, and Shirqat were restored to pre-ISIS levels. However, respondents currently residing in the intervention areas sampled perceived themselves to have limited access to sanitation services in relation to other services. This is illustrated in figure 1. The majority of respondents who stated that sanitation systems had been destroyed during ISIS occupation in Karma, Mekeshifa, and Shirqat reported that they perceived that those systems were not yet restored in their communities. Additionally, as described below in regards to barriers to access 54% of respondents reported not being connected to a public sewage system. Measurement of Services Restored The Evaluation Team asked respondents whether water, electricity, sanitation, education, and health systems, respectively, existed prior to ISIS occupation in their communities, and, if so, whether those systems were damaged. Those that responded yes were subsequently asked if they perceived that those services had been restored. xiv Low levels of restoration to sanitation systems in these areas could be attributed to the level of sanitation activities implemented. While FFS did support sanitation activities in some areas, there was a correlation between areas with low rates of positive responses and low numbers of FFS sanitation interventions. Very few respondents reported barriers to accessing services and those who did reported reasons that were outside the scope of the FFS intervention, including the following: ● Electricity: While only 7 percent of PBS respondents were not connected to the public grid, those who were not cited the cost of the connection fee and/or monthly subscription, living in dwellings that did not meet standards for connections, and delayed applications. ● Water: The 15 percent of PBS respondents who did not have water pumped into their own dwelling cited damaged/vandalized infrastructure, a dried water source, poor system design, and lack of maintenance at the community access point as barriers to accessing public water services. ● Sanitation: Fifty-four percent of PBS respondents stated that they were not connected to a public sewage/water drainage system (covered or uncovered canal); however, among those, 47 percent stated they did have access to a septic tank/sub-tank and eight percent were either connected to other means of waste disposal were not connected to any means of waste disposal. Those not connected to a public system cited the following barriers to access: lack of permission to be connected to the sewer system (reasons were not provided), no regulation mandating connections, connection costs, lack of awareness of the public system, and poor operation and maintenance of the sewage system. ● Health: Twenty-three percent of PBS respondents reported that they were not assisted at their nearest health facility when seeking services and cited the following barriers: the health center did not offer the service sought, the facility was closed, long wait times, and the cost was prohibitive. Figure 1: Restoration of Destroyed Basic Services by intervention Area – Percentage of PBS Respondents 99% 89%99% 99% 69% 97% 96% 86%89%93%94% 90%97% 91% 64% 92% 87% 85% 74% 93% 70% 61% 56% 46% 29% 69% 58% 31% 55% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% East Mosul Falluja Qayara Karma Mekeshifa N. Plains Ramadi Shirqat Tikrit West Mosul Electricity Services Restored Water Services Restored Sanitation Services Restored xv ● Education: Among the 63 percent of PBS respondents who had a school-age child in the household, 3 percent stated that their child/children was/were not attending school, citing the following barriers: distance to the school, the school was in a restricted zone, insecurity and personal safety issues, the cost was prohibitive, poor quality of the schools, the child’s illness or disability, and school teachings conflicted with their beliefs. RECOMMENDATION 1.1 According to PBS respondents in the 10 surveyed FFS intervention areas, 97 percent of returnees across all surveyed areas have access to schools for their children. Given this, supporting education infrastructure (not in areas of quality, content, or capacity) may be reaching a state of diminishing returns for the purposes of stabilization in the 10 surveyed intervention areas, although not necessarily for longer-term transition or development in other intervention sites. USAID and FFS may wish to consider reviewing its programming to focus activities and resources (a) tactically on all sectors in more newly liberated areas, or (b) tactically on other less-entrenched sectors for returnees in selected areas where there are incongruous signs of output and outcome achievement. Additionally, the donor community or GOI may want to consider a strategic shift from a stabilization to development approach in the education sector in the 10 surveyed intervention areas, including, for example, increasing community management capacity functioning and local governance capacity functioning. QUESTION 2: HOW HAVE FFS ACTIVITIES AFFECTED THE QUALITY OF SERVICES IN TARGET COMMUNITIES? Because of the difficult environment in which FFS was implemented, a baseline assessment to determine service quality among beneficiaries prior to the FFS intervention was not possible. While the lack of a baseline makes it difficult to produce concrete findings demonstrating progress towards achieving improved service quality among beneficiaries, the Evaluation Team was able to determine perceptions of change in quality via the PBS survey. Beneficiaries were asked retrospective questions about their access to services prior to 2015 and their access to services in 2019. These responses were analyzed along with a thorough review of UNDP progress reports and other related documents, and KII and FGD transcripts to determine changes to service availability in target communities. The Evaluation Team determined that FFS activities restored the quality of electricity, water, and sanitation services to pre￾2015 levels. Given the nature of FFS interventions—namely that FFS did not support improvements in quality of services in the education and health sectors, such as training service delivery providers like teachers or clinicians (outside of equipment use), but rather focused on infrastructure—emphasis was given to water, sanitation, and electricity facilities in areas sampled in the PBS. The Evaluation Team visited electricity service sites in the intervention areas sampled under the PBS to determine whether services met sector-specific parameters of quality and found that at all visited power stations, electrical substations, and generators were functioning properly and no substantial technical issues were identified. The Evaluation Team also found that water and sanitation standards were met, per the following metrics: • All (100 percent) PBS respondents reported using at least one bucket (18L) of water per household per day. Depending on the household size, this would meet the SPHERE3 standard that requires that a minimum of 15L/person/day be provided to meet basic human needs. 3 SPHERE standards are a set of principles and minimum humanitarian standards in four technical areas of humanitarian response: Water supply, sanitation and hygiene promotion (WASH) Food security and nutrition. Shelter and settlement xvi • SPHERE-required water quality tests were being performed on a regular basis at all treatment plants visited except for a treatment plant in Mekeshifa, where no tests were being performed. • All treatment plants visited used a form of chlorine (chlorine gas or calcium hypochlorite) to disinfect water. Related to the quality of services was the administration of those services, and PBS respondents felt mixed about local authorities’ abilities and/or capacities to administer services across implementation areas. When asked about their level of satisfaction with local authorities’ administration of various services, respondents’ replies ranged from “somewhat satisfied” (2) to “very dissatisfied” (4). Respondents particularly disapproved of the administration of sanitation. Across all implementation areas, the average rating of administration of sanitation services was “somewhat dissatisfied” (3) and in Mekeshifa “very dissatisfied” (4). RECOMMENDATION 2.1 Based on respondents’ low satisfaction with local administration of sanitation services, FFS should consider working with local authorities to build capacity and improve administration of sanitation services (Window Three) to ensure improved service provision to beneficiaries and improve local governance through service management. Should this recommendation not fall within FFS’s scope of work under Window Three, the Evaluation Team recommends that USAID or other donors consider this in the design of future programming. QUESTION 3: HOW HAVE FFS ACTIVITIES AFFECTED IDPS IN TARGET COMMUNITIES? Through its efforts to restore access to core services, FFS addressed—and continues to address—factors that influence decisions to return or to stay (for those who never left) for up to 86 percent of the population, although not all services are equally important and none were cited by a majority of PBS respondents as critical to decisions to return across the full population. The figure to the right shows that for roughly a third of the population, restoration of electricity, water and sanitation, health, and education services are critical factors in decisions to stay or return, and a consideration—though not a critical one—for an additional 30 percent to 50 percent of the population, depending on the sector. These numbers are even higher when looking at factors influencing return by intervention area only among returnees (excluding those who never left or current IDPs) in the year(s) the majority of those surveyed returned to their homes. These data, detailed in the following section, present further evidence that FFS interventions responded to key concerns of IDPs vis-à-vis their decisions to return to their homes at the time they are making those decisions. However, the extent varies by intervention area and intervention type and does not hold true for one intervention area (Mekeshifa). This analysis includes all intervention types, including roads and bridges, municipality support, police stations, markets, and residential rehabilitation. 0% 20% 40% 60% 80% 100% Electricity Water and Sanitation Health Education Critical factor Consideration Figure 2: Critical factors and other considerations in decisions to return or stay across full population xvii Furthermore, life is improving for returnees. The Cash for Work (C4W) activities were successful, unsurprisingly, in improving the income of participants. Nearly 50 percent of respondents noted that their lives, and their income, have improved since the C4W activities. Despite these successes, there is clear indication that not all ethno-religious groups in Iraq benefited the same from this specific intervention. As further delineated in the full body of the report (Figures 29–35), which includes disaggregation by province, Shia, Sunnis, and Yazidis earned less than Christians and those who identified as “Other” in their C4W payments, while Shabaks fared better than other ethnic groups in the cash received. Whether this is due to discrimination or other factors is unclear in the data. The data suggest that life is improving for people who returned, with 34 percent reporting life has mostly improved and 17 percent that life has completely improved, with 32 percent reporting not very much improved and 16 percent reporting that life has not improved at all for those who have returned, according to the livelihood survey. These findings are expected following recovery from the loss of income and property, and distortion of livelihoods during the crisis. It should be noted that more than half of the returnees (51 percent) are reporting improved living conditions, implying that the returnees have been set on a good pathway and demonstrating gains toward stable livelihoods. RECOMMENDATIONS 3.1 If all interventions are meant to encourage returns, as documented in the FFS development hypothesis, then based on factors driving return,4 USAID and UNDP should reconsider projects related to municipality support and roads and bridges in all intervention areas, and livelihoods support in certain intervention areas. However, if these interventions are meant to support other objectives important to FFS goals—for example, if the cash infusion into villages from livelihoods activities is an objective in itself and is not linked to encouraging returns—the Steering Committee should revisit the FFS development hypothesis to clarify these objectives. 3.2 FFS should consider the impacts of C4W activities on each community, noting the differences in the intervention on each religious and ethnic group. A perception of favoritism, whether implicit or explicit, will likely lead to communal and national tension. QUESTION 4: WHAT FACTORS ARE CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES TARGETED BY FFS? Several factors contribute to or impede local authorities’ capacity to administer services targeted by FFS, depending on the province and sector. We discuss these variations in detail in the body of the report but a summary of factors contributing to or impeding local authorities’ capacity to administer services is listed below by province based on rapid capacity assessment scores by province, sector, and function in which a score of one indicates low capacity; two, basic capacity; three, moderate capacity; and four, strong capacity. Tables detailing these results are in the main body of this evaluation report. In Anbar, factors impeding local authorities’ capacity to administer services include the following: • Budgeting5 in the sewage, electricity, and health directorates • Revenue collection6 in the sewage, water, education, and electricity directorates • IT and communications in the sewage, water, and health directorates Factors contributing to local authorities’ capacity to administer services include the following: 4 As reported by returnee respondents to the population-based survey and cross-referenced with the timing of returns and FFS interventions in each intervention area per the project list provided by UNDP. 5 This includes the existence of a realistic, clear, and well-documented directorate budget developed without external assistance; the existence of a realistic budget for operations and maintenance (O&M); and whether locally generated revenue is sufficient to cover O&M costs. 6 This includes the existence of a system to collect revenue and enforce penalties for nonpayment. xviii • Procurement7 in the education and health directorates • Strategic planning8 in the education, electricity, and health directorates and the municipality In Salah al-Din, factors impeding local authorities’ capacity to administer services include the following: • Budgeting in the sewage and electricity directorates and the municipality • Revenue collection in the sewage and education directorates • M&E in the education directorate • IT and communications in the municipality • Technical capacity in the education directorate9 The evaluation did not identify any factors strongly contributing to local authorities’ capacity to administer services in Salah al-Din. In Ninewa, factors impeding local authorities’ capacity to administer services include the following: • Budgeting in the education directorate and municipality • Revenue collection in the sewage, water, education, and health directorates • M&E in the municipality Factors contributing to local authorities’ capacity to administer services relate to procurement in the sewage, water, and electricity directorates. RECOMMENDATIONS 4.1 Although outside the scope of FFS, improving provincial authorities’ capacity for revenue collection and budgeting may need special attention from the donor community and the GOI. The Evaluation Team suggests the lack of capacity in this domain be considered in the design of future relevant programming. 4.2 Donors should consider using these findings to tailor support within each province by sector and function (technical vs. management and sub-functions within management). 7 This includes the existence of, adherence to, and monitoring of a procurement policy. 8 This includes the existence, regular revision, and use of a strategic plan for the entity. 9 This includes the existence of and adherence to an approved education sector plan and guidelines that guide quality delivery and monitoring standards for education in line the with the national education policy; the existence of established community- and district-based structures and processes that facilitate effective education delivery in the province; and documented improvements over time in the number of classrooms and number of schools with access to clean water sources and sanitation facilities. 1 INTRODUCTION Under contract AID-267-TO-17-00001, International Business and Technical Consultants, Inc. (IBTCI), through its Performance Management and Support Program for Iraq (PMSP), has conducted a midterm performance evaluation to identify factors that promoted or impeded positive outcomes from Funding Facility for Stabilization (FFS) stabilization interventions from July 2015 through June 2018. The evaluation is meant to inform the United States Agency for International Development (USAID)/Iraq stabilization and development programming, and other international donor-funded activities. The report consists of sections devoted to (a) fundamental observations the Evaluation Team identified through a holistic data analysis, and in particular a data analysis when triangulated with primary and secondary sources relating to necessary and sufficient stabilization factors in fragile and conflict-affected states (FCAS); (b) answers to key Evaluation Questions asked by USAID; (c) actionable and utilization￾focused recommendations derived from an analysis of the data from the evaluation’s seven data collection methods. The report opens with a description of the purpose and objectives of the evaluation, and the methodology used to answer the Evaluation Questions. The evaluation focused primarily at the intervention area-level in the following 10 geographical intervention areas: East Mosul, West Mosul, Ninewa Plain, Falluja, Ramadi, Qayara, Karma, Shirqat, Tikrit, and Mekeshifa. 1. PURPOSE AND OBJECTIVES This evaluation was designed to evaluate the current FFS program and inform future USAID stabilization programming in Iraq and other complex, insecure environments. The main audiences of the evaluation include U.S. Mission Iraq, USAID/Washington Middle East Bureau, stabilization stakeholders and partners, and the Department of State. The objectives of this evaluation are threefold: 1. Evaluate the United Nations Development Program (UNDP) FFS program, including components funded by USAID under the Funding Facility for Immediate Stabilization (FFIS) and the Funding Facility for Expanded Stabilization (FFES) (collectively referred to as FFS) in accordance with all applicable USAID ADS regulations regarding midterm evaluations. 2. Identify and characterize contextual and programmatic factors associated with FFS achievement of results. 3. Provide specific, programmatic recommendations for corrective action if needed. The Evaluation Team reviewed and analyzed FFS interventions between its inception in 2015 and June 2018, but the reader will see that some Evaluation Questions (e.g., 3a) use a narrower time frame to allow for more granular analysis of correlation. The evaluation looked at both completed and ongoing activities as of June 2018. The following timeline provides some key context to relevant events in FFS intervention areas, highlighting the commencement of FFS activities by area, relative to ISIS defeat and liberation (Figure 1). 2 Figure 3: FFS Activity and Military Defeat in Iraq 3 EVALUATION QUESTIONS The Evaluation Questions listed in Table 2 formed the basis for the evaluation, as outlined in the Scope of Work (Annex 2). Table 2: Evaluation Questions QUESTION 1. How have FFS activities affected access to services in target communities? Evaluators will assess the following: 1a. Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including, but not limited to, health, education, water, electricity, and other municipal services). 1b. Whether access to services is equitable across vulnerable groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, politically affiliated groups, and youth and children. 1c. What significant barriers, if any, prevent members of target communities from accessing services? QUESTION 2. How have FFS activities affected the quality of services in target communities? Evaluators will assess the following: 2a. Sector-specific parameters of quality of services targeted by FFS (including, but not limited to, health, water, electricity, and other municipal services). Note changes in performance indicators, application of performance standards (e.g., potable water with a free chlorine level of 1 milligram per liter), application of performance plan (e.g., periodic testing of chlorine in potable water), hours per day of availability of water/electric services, availability of services in FFS-supported hospitals and health clinics, and completion and use by students of rehabilitated school buildings. 2b. Perceptions of the quality of services among members of target communities. 2c. Perceptions of the effectiveness of government administration of services among members of target communities. QUESTION 3. How have FFS activities affected IDPs in target communities? Evaluators will assess the following: 3a. To what extent have FFS activities influenced IDPs’ decision to return to their communities or influenced others to remain in their communities relative to other factors? 3b. In what ways have FFS activities affected IDPs’ lives since their return to their communities? QUESTION 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Evaluators will assess the following: 4a. Sources and availability of operations and maintenance budgets for rehabilitated services. 4b. Capacity to collect revenue. 4c. Organizational capacity of local administrative offices. 4d. Technical and management capacity of staff. 4 2. BACKGROUND The Islamic State of Iraq and Syria (ISIS) formed in 2006 as an offshoot of Al Qaeda in Iraq. In 2013 it was disavowed by Al Qaeda after in-fighting, and in June 2014 ISIS began a rapid advance across Iraq, seizing control of Iraq’s second largest city, Mosul, as well as Tikrit, four other Iraqi towns, and the Baiji oil refinery. By May 2015, ISIS had also taken control of Ramadi, controlling two large lines of territory to the north and the west advancing toward Baghdad. ISIS eventually occupied between 30 and 40 percent of Iraqi territory. After over three years of struggle to liberate occupied territory, on December 10, 2017, the Prime Minister declared victory in liberating Iraqi territory previously occupied by ISIS. FFS is the Stabilization Response of the anti-ISIS coalition’s liberation effort implemented by UNDP and governed by a multi-donor Steering Committee. FFS consists of two functional, sequenced initiatives, FFIS and FFES. The U.S. Government, through USAID, was one of the first donors to support the United Nations Development Programme (UNDP) Funding Facility for Immediate Stabilization (FFIS). As cities and territory occupied by ISIS were liberated, this multi-donor funded program provides critical support to the Government of Iraq (GOI) to help facilitate the safe and voluntary return of displaced populations, primarily in Anbar, Ninewa, and Salah al-Din. USAID support includes public works and light infrastructure rehabilitation, such as health clinics, water infrastructure and treatment facilities, power grids and generators, schools, other government buildings, and access roads; livelihood activities to jumpstart the local economy and generate income for local households, for example through grants to small businesses with high community impact (bakeries, electrical shops, agriculture, food kiosks, and micro-stores) and short-term employment through public work schemes (cash-for-work opportunities); and the rehabilitation of municipal and other government buildings and infrastructure. The global support toward stabilization of Iraq is organized into three main governing bodies: the Stabilization Working Group, the Stabilization Task Force, and the FFS Steering Committee. Each of these organizations plays a different role in supporting the GOI in its stabilization efforts as well as in organizing the international community’s support for stabilization. Together these three bodies comprise the governance framework for global support toward stabilization of Iraq. From the beginning, the UNDP established the FFS Steering Committee, whose mandate it was to oversee and provide strategic guidance. The Steering Committee consists of Iraqi government officials, donor partners, UNDP, and international financial institutions. The GOI Prime Minister’s Office guides overall implementation, with the Steering Committee approving locations and prioritizing activities based on need and any specific requirements for location and activity set by individual donors. Core criteria for the selection of stabilization activities in Iraq include the following: 1. Any support funded under any of the components must adhere to the general principles of international human rights and humanitarian law (especially nondiscrimination and impartiality), and promote reconciliation and social cohesion. 2. Activities shall employ accountability and transparency mechanisms that, to the greatest extent possible, reinforce existing good governance practices and systems for local development. 3. Areas receiving support must have established mechanisms for the safe and voluntary return of IDPs. 4. Legitimate security arrangements fully under GOI control should be reestablished. 5. Gender dimensions and minority rights should receive particular attention. USAID’s agreement with UNDP stipulates that USAID may direct which activities/locations to fund and implement with USG resources. On decision from the FFS Steering Committee, UNDP has agreed that donors may earmark their funds at any level. In accordance with the terms of the agreement, UNDP must obtain approval from the Agreement Officer (AO) prior to using any USAID funds for any project 5 or activity. To obtain approval, UNDP is required to submit a proposal with an estimated budget to the AO for initial review. USAID’s project selection is conducted in close consultation with the U.S. Embassy leadership. To ensure that the Mission is in compliance with all applicable laws, regulations, and policies governing the use of USAID funds, each proposal submitted by UNDP is reviewed and cleared by the Technical Office and the Resident Legal Officer, with the AO providing the final review and approval. UNDP is responsible for carrying out regular monitoring of FFS activities and reporting results to USAID. This work is in accordance with USAID policy in ADS 308 for grants to Category 1 Public International Organizations (PIOs), such as UNDP. USAID also relies on weekly activity updates and formal quarterly reporting from UNDP to monitor FFS progress and results. Three points are critical to note, and indeed are imperative to framing the evaluation within the proper geographical, programming, and evaluability context: 1. FFES activities were never envisaged as a substitute for larger-scale reconstruction efforts, including those being implemented by the World Bank and the German Development Bank. 2. FFS consists of implementation of large or medium sized infrastructure projects across four “windows:” (a) light infrastructure rehabilitation, (b) immediate livelihoods support, (c) capacity support to municipalities, and (d) community reconciliation and social cohesion. Collectively these windows contribute iteratively, and in an inextricably intertwined way, toward stabilization, yet FFES by default concentrates primarily on essential services and infrastructure rehabilitation and livelihoods, and only obliquely if at all to capacity support. To date it has not supported community cohesion. 3. FFS is pool-funded by 28 donors, including the USG, making confident attribution claims in an evaluation challenging. As of the end of 2018, FFS had funded $1.029 billion. USAID was the first donor to contribute to FFIS in July 2015 and FFES in August 2016. As of the end of the period of performance for the evaluation (June 30, 2018), USAID had obligated $190.3 million.10 4. As of the end of 2018, FFS was operating in 31 intervention areas across five provinces: Ninewa (1,883 projects, or 61% of the portfolio), Anbar (749, 24%), Salah al-Din (312, 10%), Kirkuk (137, 4%), and Diyala (21, 1%).11 FFS DEVELOPMENT HYPOTHESIS According to FFS documentation, the purpose of the FFS is to encourage the return of IDPs and to promote reconciliation by demonstrating that the GOI is committed to providing citizens with essential basic services in newly liberated areas. If immediate delivery of critical services (for example, health, water, power, and education) and employment and livelihood opportunities are provided by the GOI to encourage people to return to their homes, then communities in liberated areas will stabilize and recover more quickly and the trust of displaced Iraqis in the GOI will be restored. EVALUATION DESIGN The valuation design (Annex 4) includes the following: a) A detailed evaluation design matrix that aligns methodologically the Evaluation Questions in the Evaluation Statement of Work (SOW) to relevant data sources, methods, and the data analysis plan 10 At the time of this report’s submission the U.S. government had obligated $363.3 million to FFS. 11 FFS Annual Report 2018 6 b) Draft questionnaires and other data collection instruments or their main features c) The list of potential interviewees and sites to be visited, and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology) d) Known limitations to the evaluation design e) A dissemination plan A detailed evaluation design matrix linking evaluation questions to data sources, methods, and analysis plan is attached as Getting to Answers (G2A) matrix (Annex 1 of the Evaluation Design). A proposed evaluation methodology along with a sampling framework and limitations to the evaluation design is presented in the section below; proposed data collection tools are provided in Annex 5. 3. METHODOLOGY The evaluation employed a mixed-methods approach to data collection. The team designed and employed both quantitative and qualitative methods to derive answers to the Evaluation Questions. Quantitative data collection included a Population Based Survey (PBS) and a Livelihoods Survey (LS). Qualitative data collection included Focus Group Discussions (FGDs), Key Informant Interviews (KIIs), and Onsite Observations. In addition, the team conducted rapid capacity assessments using a tool adapted from USAID Organizational Capacity Assessment (OCA) tool. A description of data collection methods for each evaluation question is presented in Annex 3. All data was collected between July and October 2019. Table 3: Proposed Data Collection Methods for the Evaluation Questions Data Collection Methods Evaluation Questions Desk Review 1, 2, 3, 4 Key Informant Interviews 1, 2, 3, 4 Population-Based Surveys 1, 2, 3 Focus Group Discussions 1, 2, 3 Field Observation and Site Visits 2 Livelihood Beneficiary Survey 3 Capacity Assessment 4 3.1 DOCUMENT REVIEW Before commencing primary field data collection, the team reviewed project documents and existing secondary data. The team concentrated on three12 years of FFS intervention from July 2015 through June 2018. Below is a list of documents reviewed: a) Existing program data, including targets and output indicators 12 It should be noted that not all areas had three years of intervention. For example, FFS interventions in Tikrit started in 2015, followed by Fallujah and Ramadi. Mosul followed in fall 2017 and activities in Western Anbar started in February 2018. 7 b) Activity monthly, quarterly, and annual reporting and activity documentation (such as donor updates from steering committee meetings, annual work plans, etc.) c) Grant agreements between USAID and UNDP, with all relevant modifications and deliverables d) Implementing Partner (IP) Monitoring and Evaluation (M&E) data sets and indicators e) Field monitoring reports from FFS f) Activity monitoring and verification data from over 1,000 UNDP and PMSP third-party monitors g) Donor, host country, and analytical reports on Iraq stabilization efforts 3.2 KEY INFORMANT INTERVIEWS KIIs with key stakeholders with precise knowledge of the target intervention areas and provinces, and in particular government officials, provided overall background information and helped to contextualize the overall evaluation framework. Key informants included 30 local authorities in directorates and municipalities related to FFS interventions, donor representatives, and IP staff selected using purposive sampling based on their involvement in FFS and/or familiarity with the status of relevant services in a given sector of FFS interest. While the quantitative data from the PBS and LS provided perception-based, micro-level data, the qualitative data from the KII respondents provided primary context. KIIs were used to understand higher-level issues related to access to and quality of basic services in the targeted areas (EQs 1 and 2). In particular KIIs provided perspectives of key stakeholders on accomplishments, best practices and lessons learned, and other factors that may have inhibited or promoted success of FFS (EQ 4). KIIs were also instrumental in assessing the capacity of government entities to administer FFS service delivery (EQs 1, 2, and 4). The Evaluation Team used an inductive approach to the analysis of all qualitative data to summarize findings from KIIs and FGDs and identify patterns within the data. The Team mapped all relevant qualitative data to an evaluation sub-question. The KII list is in Annex 4, and includes the following individuals: 1. USAID mission staff and other FFS donor staff 2. GOI counterparts at the national, provincial, and local levels 3. Senior IP staff, as well as other key technical staff a. Representatives from civil society and community The nature, relationship to the activity, and potential for bias of KIIs was weighed in interpreting responses and acknowledged in the presentation of Findings. 3.3 FOCUS GROUP DISCUSSIONS (FGDs) FGDs were conducted with FFS direct and indirect beneficiaries. The evaluation employed purposive and convenience sampling technique for selection of FGD participants. FGDs were conducted with homogeneous groups of nine to 12 participants in the three provinces (Anbar, Ninewa, and Salah al￾Din). The Evaluation Team conducted 12 FGDs. Separate FGDs were conducted for males and females. Six FGDs (three for each gender) were conducted with participants from the returnee communities in Ninewa, Anbar, and Salah al-Din; two FGDs (one for each gender) were conducted with participants from the IDP community in Salah al-Din; while four FGDs (two for each gender) were conducted with Religious and Ethnic Minorities (REMs) from the IDP and returnee communities in Ninewa. 8 FGD participants from the returnee communities were selected from the catchment areas of FFS￾rehabilitated facilities. REMs selected from returnee and IDP communities. Table 4 below shows the distribution of FGDs among the different identity groups in the three provinces. Table 4: Number and Distribution of FGDs Identity Group Province Ninewa Anbar Salah al-Din Total Male Female Male Female Male Female Returnees 1 1 1 1 1 1 6 REM Returnees 1 1 - - - - 2 IDPs - - - - 1 1 2 REM IDPs 1 1 - - - - 2 TOTAL 3 3 1 1 2 2 12 While the PBS and LS provided quantitative data for statistical analysis, FGDs helped to contextualize the evaluation findings and conclusions, considering plausible explanations for correlations and patterns observed, and triangulating with quantitative response data. Specifically, FGD responses were used to contextualize or expand on survey findings related to how FFS interventions have affected access to services (EQ1), quality of services (EQ2), and IDPs in target communities (EQ3). FGD participants were asked about the reasons for or explanations about perceived changes in their community relating to the access and quality of health care, education, electricity, and other infrastructure services implemented by the FFS. 3.4 POPULATION-BASED SURVEY (PBS) An extensive PBS was designed and administered so as to elicit sample population perceptions of services in areas rehabilitated by FFS. The PBS provides the evaluation findings with generalizable quantitative data responses relative to EQs 1, 2, and 3. The design below was proposed to the Evaluation Team by USAID and verified by IBTCI. The sample was informed by population data results using Landscan.TM 3.4.1 SURVEY AREAS The survey was administered at the intervention area level in Anbar, Ninewa, and Salah al-Din provinces to assess perceptions of services in catchment areas served by facilities rehabilitated by FFS. Eligible facilities include education, electricity, health, sewage, and water facilities supported by FFS interventions funded by USAID and other donors. In addition, FFS implements housing, bridges and roads, municipal equipment, and livelihoods programming that includes cash grants, small business grants, and cash-for￾work. These implementation areas are addressed in the overall midterm evaluation of FFS but are not included in the population survey because these interventions do not address basic services. Within each province, intervention areas that have a high concentration of FFS activities were selected for data collection. The total number of eligible facilities was tallied for each intervention area, including those that were under implementation or completed by June 6, 2018. The 10 intervention areas selected ensured coverage of at least 83 percent of eligible FFS facilities in each province. Table 5 below presents the distribution of FFS-sponsored facilities in the intervention areas and provinces. Data on the locations of facilities was provided by UNDP, with the intervention area identified and the GPS coordinate listed, for the intervention areas. Based on the information provided by UNDP about the FFS 9 facilities and their locations, the Evaluation Team determined the selected intervention area, assuming more than 80 percent of affected areas to be a strong sample. These are represented in the yellow highlights. Table 5: Distribution of Facilities in the Provinces and Intervention Areas Ninewa Anbar Salah al-Din FFS Intervention Area Eligible facilities (#) Eligible facilities (%) FFS Intervention Area Eligible facilities (#) Eligible facilities (%) FFS Intervention Area Eligible facilities (#) Eligible facilities (%) Qayara 55 9% Ramadi 137 46% Tikrit 44 51% East Mosul 198 34% Falluja 69 23% Mekeshifa 9 10% West Mosul 109 19% Karma 41 14% Shirqat 20 23% Ninewa Plains 167 28% Haditha 27 9% Al Dour 6 7% Sinuni 22 4% Heet 0 0% Baiji 7 8% Makhmur 0 0% Rutba 1 0% Yathrib 0 0% Sinjar 17 3% Rawa 6 2% Tuz Khurmatu 0 0% Rabia 19 3% Qaim 7 2% Suleiman Bek 0 0% Telafar 0 0% Anah 8 3% Ba’aj 0 0% Total 90% Total 83% Total 84% 3.4.2 PRIMARY SAMPLING UNITS The Evaluation Team applied a cluster-based population-proportional sampling approach, based on population data from Landscan™, which USAID proposed in the PBS design dated June 21, 2018. In the absence of official population data, Landscan™ provides a community standard for gridded global population data. Landscan™ combines models tailored to data conditions in each country and region to disaggregate best-available demographic data across a grid of approximately one-square-kilometer of spatial resolution. In each of the 10 selected intervention areas, the team mapped eligible facilities using geo-coordinates provided by UNDP and classified then by facility type. The team determined an appropriate geographic buffer, representing a population potentially served by the facility; the radius of this buffer differed between facility types. Grid cells were then coded for all types of facilities by which they could potentially be served. This process only approximated the actual service area of any given facility, as delineating exact service areas on a facility-by-facility basis was deemed to be prohibitively time￾consuming. The Evaluation Team’s approach, however, was based on a reasonable supposable reach of a given facility, based on its function. Grid cells that did not fall within the buffer for any eligible facility were omitted from the sampling pool. Cells were stratified by type and saturation of facilities (e.g., categorized by number and type of facilities serving the cell), and then selected randomly, proportionate to population. See example in Figure 4, as well as Annex 6 for Landscan™ of all 114 cells. 10 Figure 4: Example of grid cells in Landscan™ of Ramadi 3.4.3 SAMPLE SIZE The sample size for each province was calculated using the following assumptions: Table 6: Assumptions for Sample Size Calculation Margin of error 5% Confidence interval 95% Precision 5% Power 80% Proportion (est.) 50% Target population 7,096,889 Design effect 2 Nonresponse rate 15% N-unadjusted 768 N-adjusted for nonresponse 904 Using the targeted population for the three provinces from the 2018 Population and Housing census from Central Statistics Organization for Iraq projection estimated at 7,096,889 in 2018, the proposed minimum sample size per-province is 768. After adjusting for a nonresponse rate of 15 percent an additional 136 respondents per province are considered. The minimum sample size per province was 11 therefore n = 904.13 The sample size of 904 was the minimum n to ensure representativeness with a reasonable 5 percent margin of error, a confidence interval of 95 percent, and a nonresponse rate of 15 percent, all of which are standard. In each of the selected cells the Evaluation Team surveyed 24 respondents (12 men and 12 women).14 In each province, 3815 cells were selected, for a total of 114. Therefore, the final sample size for the PBS was population = 2,73616 respondents (see Table 7). Sample population weights were used to account for population size distribution. Cells that were determined to be inaccessible due to security concerns were replaced by the nearest accessible cell. The sample allocation for cells across the 10 selected intervention areas was derived based on the concentration of facilities in the intervention area varying proportional to size. Table 7: Proposed Sample Allocation in the Provinces and Intervention Areas Province Selected Intervention Areas Number of Eligible Facilities Number of Sampled Cells Targeted Sample Respondents Anbar Ramadi 137 21 504 Anbar Fallujah 69 11 264 Anbar Karma 41 6 144 Salah al-Din Tikrit 44 23 552 Salah al-Din Shirqat 20 10 240 Salah al-Din Mekeshifa 9 5 120 Ninewa East Mosul 198 14 336 Ninewa N. Plain 167 12 288 Ninewa West Mosul 109 8 192 Ninewa Qayara 55 4 96 N-total sample 849 114 2,736 3.4.4 SELECTION OF HOUSEHOLDS AND RESPONDENTS Households were selected using a transect walk approach. Surveyors began at the center point of each selected cell, and randomly selected a direction to follow away from the facility. Households were selected following a given interval (e.g., every third household from the first), with a left-hand principle (turning left if the forward path is blocked). Each enumerator was assigned to select either only female or only male respondents. Household selection proceeded until 24 interviews were completed in the cell. The appropriate household interval was selected to maximize coverage of a grid cell. 13 The adjustment for nonresponse is nonlinear given that nonresponse will vary across primary sampling unit. 14 The number of 24 respondents per cluster resonates well with other typical gold standard household surveys. Most Demographic Health Surveys (DHS) use a fixed take of households per cluster of about 25-30 households, determining the number of clusters to be selected (Guide to DHS Statistics, 2018). 15 Rounded off after dividing 904 (sample size per province) by 24 (number of respondents per cell). 16 The number is slightly higher due to rounding off the number of total cells included in the survey. 12 Once a household was selected, the enumerator asked to speak with either the primary male household head or the primary female household head, as assigned.17 If the male/female household head was not available, they moved on to the next house. If more than one male/female household head resided in the household (as in a multifamily home) the male/female household head was selected at random using a Kish grid. 3.4.5 SAMPLING WEIGHTS Sampling weights were determined using Landscan™ population estimates where these were deemed adequate. Additional localized population data sources including facility catchment populations will be sought, especially for areas affected by recent population movements due to conflict, and these sources will be triangulated to determine the most appropriate sampling weight. Adjusted populations for selected cells were accomplished using appropriate correction factors. 3.4.6 ANALYSIS The Evaluation Team analyzed PBS data using frequency analyses and cross-tabulations for Evaluation Questions 1, 2, and 3. Hypothesis testing was then conducted using t-tests. For EQ1, the Evaluation Team reviewed the frequency and percentage frequency of responses to various questions18 by dividing the frequency of a given response by the total number of participants that responded to the respective question. Additionally, the Evaluation Team calculated the mean response to questions such as distance to health centers and schools. The Evaluation Team also reviewed the frequency and percentage frequency for these questions by demographic subsets including ethnicity, religious affiliation, and geographic location of the respondent at the time of survey - the team reviewed both district and governorate subsets. For EQ2, the Evaluation Team used the same analytical approach but for a different set of questions19. For Evaluation Sub-question 3a, the Evaluation Team ran cross-tabulations of responses to PBS question G6 (What are the most critical factors that influenced you to return (if returnee) or to stay (if never left) in this community?) by respondent status (returnee, IDP, never left), and intervention area. The ET identified the year or years that the majority of surveyed returnees reported returning to their homes and ranked reported drivers of return by intervention area for those years. These results were then compared to FFS interventions in that area through the given time frame, using the project list provided by UNDP in June 2019. 3.5 LIVELIHOODS SURVEY (LS) Three types of livelihood activities have been implemented under the FFS program: Cash for Work (C4W), small grant support to female-headed households and small business grants. According to 17 In some cases, the main head of the household may not match the gender assigned to the data enumerator to interview. The enumerator should then clarify that they are looking for the person of the assigned gender who most closely mirrors the main household head (e.g., if a man is identified as the main household head, but the enumerator has been assigned to interview women, they should identify the woman living in the household who makes most household decisions). In this example, that could be the household head’s wife, or his mother or sister, depending on the configuration of the household. The data collection tools will include fields to record exactly how each respondent was selected. 18 C1, C2, C3, C4, C12, D1_1, D1_2, D1_3, D1_4, D1_5, D1_6, D1_7, D1_8, D1_9, D1_10, D1_15, D1_16, D2_1, D2_2, D2_3, D2_4, D2_5, E1, E2, E3, E4, E8, E9, F1, F2, F3, F5, F6, F7, F8_A – E, F12 19 C13, C14, C15, D1_17, D1_18, D1_19, D1_20, D1_22, D1_23, D1_24, D2_10, D2_11, E10, E11, F13, F14, F15, F16, F17 13 UNDP, beneficiary data was available for only 8,023 C4W20 participants. The evaluation therefore did not include beneficiaries of small grants support to female-headed households and small business grants. According to FFS Annual Report 2018, 34,844 beneficiaries received support from one of the three livelihood activities as shown in Table 8 below. Table 8: Summary of Livelihoods’ Program Beneficiaries21 Cash for work Beneficiaries 26,505 Small grant for female headed households 4,858 Small business grants for women 3,481 Total 34,844 3.5.1 SAMPLE SIZE Keeping in view the disproportionate spread over geographical area, a stratified random sampling technique was used to calculate the sample size. Table 9 below outlines the assumptions used to determine the overall sample size. Based on the assumptions, 856 beneficiaries were selected for surveying, with 727 with 15 percent nonresponse, being the final sample. Table 9: Assumptions for Sample Size Calculation Margin of Error 5% Confidence interval 95% Proportion (est.) 50% Precision 5% Statistical power 80% Beneficiaries population 8,023 Design effect 2 Refusal rate 15% N-unadjusted 727 N (min. sample size) 856 3.5.2 SELECTION METHOD OF RESPONDENTS A systematic random sampling technique was used to select beneficiaries.22 UNDP provided a coded list of beneficiaries for each activity along with their other details such as gender, age, year of enrollment, location, etc., without divulging personal identifying information. Given by the following notation: n = D[(Z^2(1-P)P)/µ^2], where n = sample size, D = design effect = 2, Z = 95 percent confidence interval 20 According to the FFS Annual Report 2018, the total number of Cash for Work beneficiaries was 26,505, of which contact information was available for 8,023 with UNDP. 21 The figures in this table are taken from FFS Annual Report, 2018. 22 The sample size was overestimated by 15 percent to cover for refusals or missing respondents. A 20 percent additional contingency sample of beneficiaries has been set aside for replacement of refusals. 14 (1.96), P = proportion of the outcome variable = 0.5, µ = margin of error = 5 percent, and a nonresponse adjustment = 15 percent, this is a representative sample. 3.5.3 ANALYSIS For EQ 3B, The Evaluation Team ran cross-tabulations of various Livelihood Survey questions23 by religion, ethnicity, and sex. 3.6 ON-SITE OBSERVATIONS The Evaluation Team analyzed over 500 FFS third-party monitoring site visit reports from the PMSP team to better understand onsite observations of access, quality, efficacy, efficiency, utility, and compliance of FFS delivery by intervention area site. The Evaluation Team also made site visits to assess the public works rehabilitation and light infrastructure activities implemented by FFS from 2015 to 2018. Given the nature of FFS interventions,24 emphasis for EQ 2 (quality) was given to water, sanitation, and electricity facilities in areas sampled in the PBS. These sites were selected to align with locations selected for the PBS. Facility sites located within the cells selected for PBS data collection were included in the visit plan. Site visit checklists were used to assess the quality of public services provided and the role of infrastructure in the delivery of quality basic services in response to EQ 2a. A total of 24 facility-infrastructure sites were observed, eight in each province representing 20 percent of the selected PBS sites. The selected sites included facilities for electricity (generators and substations), sewerage, and water treatment. The field teams visiting the facilities captured GPS-tagged photos of each facility visited. 3.7 CAPACITY ASSESSMENT OF LOCAL AUTHORITIES The Evaluation Team adapted the USAID Organization Capacity Approach (OCA)25 to assess capacity of provincial directorates in budget, revenue collection, management and administration,26 and technical capacity for operations and maintenance to answer EQ 4.27 The Evaluation Team conducted rapid capacity assessments of provincial directorates of Water, Sewer Services, Municipality, Health, Education, and Electricity in Anbar, Ninewa, and Salah al-Din. In addition to assessing capacity in the core governance functions of strategic planning, budgeting, procurement, monitoring and evaluation, information and communication technology, and human resource management, the team assessed the technical capacity of local authorities to deliver services in line with their core mandate. The assessments were conducted through facilitated discussions with at least one senior representative from each of the seven core governance and technical departments. In total, 18 facilitated discussions (six in each province) were conducted with provincial directorates in the three provinces. When analyzing the respective operations and maintenance budgets, the Evaluation Team relied on KII responses and a document review of sources from the Ministry of Finance as sources. The Evaluation Team calculated the mean scores for each functional and technical area in the rapid capacity assessment tool by sector and province, with all questions weighted equally. Mean scores were then compared by province and sector. 3.8 DATA ANALYSIS METHODS 23 1.6, 1.11, 1.12, 1.13, 1.14, 1.19, 1.20 24 FFS primarily supports service delivery infrastructure. 25 See https://usaidlearninglab.org/library/organizational-capacity-assessment 26 Includes planning, human resource management, procurement, monitoring and evaluation, and information and communication technology 27 A full-scale organizational capacity assessment is beyond the scope of this evaluation and this tool is not intended to capture that. Typically, a full-scale OCA is carried out over several days and includes a facilitated workshop and developing an action plan with the intent that reassessments will be carried out over time. 15 The evaluation’s activities were conducted in accordance with the 2011 USAID Evaluation Policy and its 2016 revision.28 This policy stipulates that evaluations ensure that reliable data will be produced to support evidence-based findings, and will be a sound basis for analysis that will lead to conclusions and recommendations that are both useful for assessing the progress to date and capable of providing actionable recommendations relevant to future programming. In line with the policy, the evaluation combined mixed data collection methods to allow for the verification of findings through triangulation. The team understands triangulation as a method used during the evaluation process to check and establish validity of conclusions by analyzing the responses to EQs from multiple perspectives, such as when conducting the desk review, interviews, FGDs, etc. Triangulation of data will enable the Evaluation Team to ensure validity of conclusions, i.e., that the findings are true (and accurately reflect the situation) and certain (supported by the evidence). While not a data collection method, analytical triangulation approaches were employed to verify and validate the findings from different methods, data sources and/or team members, and to identify correlations between findings to determine programmatic impact. These triangulation approaches included the following: ● Methodological triangulation – Up to seven data collection methods, but no fewer than two for data collection will be used to support each of the EQs. It should be noted that the document review was extensive and included primary and secondary sources. ● Data source triangulation – The team collected data about FFS through its own primary and secondary sources, but also from snowballed KIIs and other relevant sources. ● Investigator triangulation – The team had several core members and enumerators from a local research partner [Independent Institute and Administration Civil Society Studies (IIACSS)] and so there were complimentary experiences and skill sets at play to analyze data and findings. This allowed for analysis of the data from different—and richer— perspectives. 3.9 DATA TREATMENT PROTOCOLS The data cleaning and validation approach was built in the data capturing system. Generally, there were very few variables that required further data cleaning as the data collection sub-contractor submitted a clean preliminary dataset. Cleaning and validation protocols included: a) Identification of duplicate cases – no duplicates were observed; b) Ensuring data for each household were complete data and included requisite identification details; c) Filtering questions included skip rules to ensure those not eligible should not respond follow up questions. Therefore any N/A responses were recoded to missing; d) Mandatory questions to be answered by all including YES/NO filters have to be complete without any missing responses. Missing responses were edited to NO response. E.g. G3. The cleaning of illegal responses which are not applicable were edited to missing, year of return is 28 The assessment meets or exceeds the USAID 2011 Evaluation Policy, USAID’s Performance Monitoring & Evaluation TIPS: Data Quality Standards, relevant chapters of the USAID Automated Directive System (ADS), the performance standards outlined in the RFTOP Task Order Scope of Work, as well as with the Federal Policy for the Protection of Human Subjects or “Common Rule” [ADS Chapter 200, http://www.usaid.gov/policy/ads/200/200mbe.pdf]. The IBTCI team is also apprised of the July 2012, supplement to ADS 203 on current good practice in preparing evaluation reports and the main deliverables expected in evaluation reports. https://www.usaid.gov/evaluation 16 not applicable to IDPs hence edited to missing (14 observations affected). Missing year of return (31 cases) edited using the mode year of return 2017. LIMITATIONS The evaluation had some technical and operational limitations. These limitations did not affect the veracity of the data, but did affect timelines and operational security. TECHNICAL 1. The lack of a baseline significantly limited before-and-after comparisons to assess change due to FFS interventions. The Evaluation Team addressed this by using recall questions in certain tools; however, this introduced a risk of recall bias into findings. 2. The administration of a PBS allowed for findings representative of the population at the time of the survey; however, by definition it did not allow us to capture feedback from a key population of interest—IDPs who left areas of FFS interventions and have not returned. The comparison group was all of FFS (i.e., there was no control group). 3. UNDP does not have a full beneficiary list for its livelihoods activities. Of the over 26,000 livelihoods beneficiaries across three interventions, UNDP only had contact information for just over 8,000 C4W beneficiaries and no grantees for the reporting period in the intervention areas. This necessarily limits the representativeness of our sample to these 8,000 beneficiaries, not the total beneficiary population. 4. There were sampling and quantitative data collection limitations. a. The lack of current population data at the lowest geographical level posed a significant challenge for the PBS survey. There have also been significant population and migration shifts within the FFS beneficiary intervention areas. The team anticipated that at least some of the estimates from Landscan™ that were used for cell selection were out-of￾date, because of population movements. The Evaluation Team sought to address this by assessing population movements in each of the selected cells. b. In absence of baseline data, the evaluation primarily relied on retrospective perceptions of respondents, which can potentially lead to recall bias. Recall bias occurs when respondents do not accurately remember previous events or experiences. To address this, data collection tools were shared with and vetted by USAID staff, translated into local languages, and piloted in the field prior to fieldwork. All data was collected in the respondents’ preferred language. Data collection teams were trained in administering each instrument. And finally, data were collected by local enumerators conversant in the local languages to avoid errors from mistranslation. c. Because much of the qualitative data collection for this evaluation relied on purposive sampling—meaning that respondents were selected for their particular roles, responsibilities, and knowledge—it did not produce statistically representative samples. As a result, although these respondents typically represent a variety of views on a topic, they may not represent all relevant views, and selection bias may have emerged. d. Respondents for the livelihood survey included only those beneficiaries for whom phone contact information was available, which created a selection bias because only individuals with access to phone and phone networks were selected. e. In the absence of data for the two livelihood activities i.e., small grants for businesses and grants for female-headed households, the evaluation only assessed cash-for-work activities under the livelihood support. Likewise, because the data for C4W is available 17 for only 8,023 beneficiaries out of over 22,000 beneficiaries, this may have excluded views from those not included in the sample and may have led to skewed results. f. Collecting quantitative and qualitative data concurrently due to time constraints presented a limitation to the evaluation because it did not allow the team to use qualitative data to fill in gaps that emerge after analysis of the quantitative data. To the extent possible, the Evaluation Team addressed this issue by following up with key informants when the need for further clarification arose. OPERATIONAL a. Protests in the country during the data collection period resulted in increased nonresponse rates from the PBS and LS. Protests prevented some of the Baghdad-based enumerators from deploying in a timely manner to the intervention areas; they mitigated this by conducting some telephone interviews in place of face-to-face interviews. b. Internet outages caused pauses and breaks in data collection; security considerations throughout the evaluation period of performance affected collection veracity and therefore purity. As above, some LS interviews were conducted by telephone. Studies have shown that telephone surveys are marginally less effective than face-to-face interviews because they are more anonymous (and so respondents may be less forthcoming) and may also yield lower overall response rates. Face-to-face surveys deliver the most representative results. Telephone surveys may provide a good alternative, but they are not the ideal. 18 FUNDAMENTAL OBSERVATIONS—WHAT HAS FFS ACHIEVED FROM 2015 TO 2018? Below are two fundamental evaluation observations that are derived from an aggregate analysis of the Evaluation’s themes, buttressed against a sample of secondary literature findings. The FFS Development Hypothesis suggests the following: delivery of much-needed core essential services + employment and livelihoods opportunities from GOI will encourage IDPs to return AND THEN communities in liberated areas will stabilize and recover. The Development Hypothesis is a logic chain of necessary and sufficient means to a desired end, the means in the case of FFS being rehabilitation and recovery for populations of returnees, and the ends being stability. The FFS Development Hypothesis is also both contextually dependent and deterministic, meaning that (a) the security situation must be sufficiently robust in Anbar, Ninewa, and Salah al-Din to be conducive to the implementation of essential services delivery, and (b) essential services delivery is a necessary and sufficient precursor to stabilization and recovery. The Evaluation Team acknowledges these assumptions, and these assumptions are supported by the peer-reviewed and gray literature as well as the primary data analyzed through the course of the evaluation. The report will turn to the specific Conclusions, Findings, and Recommendations of the respective Evaluation Questions, but before this an analytical aggregate of the Conclusions, Findings, and Recommendations is below. This analysis is based both on the primary and secondary data, as well as on an examination of the FFS objectives, and it suggests two Fundamental Observations, which are important to understanding FFS’s overall outputs and outcomes, and indeed its relative, accountable successes. OBSERVATION 1 FFS is an instrumental, formative activity to assuring stabilization in the conflict-affected regions of Anbar, Ninewa, and Salah al-Din. Assuming security is ensured, basic services delivery is a critical “push” and “pull” factor for returnees. According to the literature, there is no single mechanism to ensure refugees or displaced populations (a) return to place of origin (home) and (b) stay returned. The secondary research conducted in support of this evaluation highlights the complexities involved in population movements with respect to stabilization efforts, and largely validates the research conducted by the Evaluation Team through both the qualitative and quantitative data. The literature frames the motivations for return in stabilization environments in two factors, both of which are evident in the FFS Development Hypothesis and its implementation. The pull factors – A recent International Organization for Migration (IOM) and European Union (EU) Report29 notes that when security is guaranteed, it is the most important pull factor for IDPs. Beyond security, in another IOM report, returnees gave four major pull factors: missing home (95 percent), return of other people (76 percent) and family members (56 percent), improved safety situation in the location of origin (59 percent) and saving on living costs associated to returning to their habitual residence (51 percent). IDPs who own property return at faster rates.30 USIP notes that in addition to these factors, IDPs require more than financial compensation as an incentive to return: sustainable livelihoods; finding peaceful living conditions; having access to health care, education, and employment opportunities; and enjoying full rights of citizenship.31 Equally, a lack of security or those who experienced discrimination in their area of displacement is important driver for returnees. A bulletin 29 Obstacles to Return – IOM and EU Report 2017. 30 IOM in Iraq: Displacement and Return, page 6. 31 Fagen, Patricia Weiss. “Refugees and IDPs after Conflict: Why They Do Not Go Home.” 2011, USIP, p. 1. 19 from the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) reveals that “preferential treatment [was] given to host communities by employers in some provinces, like Anbar and Diyala, according to emergency livelihoods partners. This creates competition between displaced, return and host communities, exacerbating tensions.”32 Another pull factor is financial, with OCHA stating that “households staying in camps or with host families have returned at slower rates than those who rent accommodation.”33 In some cases host accommodations may in fact be seen as disincentives to return. FGD data collected as a part of this evaluation also reflects this finding in the literature, with many individuals reporting that they returned because they simply could not afford to live in a displaced area anymore due to the high cost of rent. Non-security pull factors identified by the Evaluation Team aligned completely with Refugees International’s (RI’s) reporting, for example, with an emphasis on the restoration of the most essential services being an immediate factor for return and remain. According to RI, “Most often, a functioning water system, an operating electricity grid, and the availability of medical care were the basic services that people felt were necessary for them to start living a normal life.” In addition, RI visited Iraq in the middle of 2017 and many people mentioned that the ability to enroll their children in school—back in their home town—at the beginning of the coming academic year was of great importance to them, rather than returning in the middle of the school year and further disrupting children’s studies. While many children had access to education even while displaced, families were eager to go home and start life at the beginning of the academic year.”34 The push factors – As the United States Institute of Peace (USIP) notes, most environments and in particular those in FCAS lack the security, economic opportunities, proper governance, and general services to encourage IDPs to return to their homes. Moreover, damaged/destroyed/ taken property and land discourages IDP return to their homes. In Iraq, land is contested in almost every province, and the government is not fully equipped to account for these disputes.35 According to the Brookings Institution, property can be especially difficult to recover due to a lack of reconstruction while minorities also suffer doubly due to the tendency to have informal titles to property. As validated in this evaluation’s research, if basic health and education services are not met, IDPs are more likely to remain displaced. Also, and in alignment with the FFS Development Hypothesis, to turn from IDP to returnees, Iraqis need forms of livelihood opportunities (absent of pre-conflict ethnic, gender or political discrimination).36 Kyra Luchtenberg, writing for The London School of Economics and Political Science, doubles down on these assertions, stating that the decision to return “is influenced by the level of property damage, the access to basic services, and the prospect of livelihood opportunities in their area of origin. Yet, the decision to return predominantly depends on the security actor in control and the perceived risk of being subjected to reprisal attacks, harassment or discrimination.”37 It is important to note that in late 2017/early 2018 (during the FFS evaluation POP) four majority minority groups (Christians, Shabak Shia, Turkmen Shia, and Yazidi) indicated that they remain displaced first due to “better safety in displacement,” second due to “better services in displacement” and third due to “children attending school.”38 Altogether, “for the 85 percent of IDPs who intend to stay in displacement in the short term, better security, public services, and economic opportunities were the primary drivers.”39 32 OCHA Humanitarian Bulletin Iraq, July 2017. 33 IOM. Understanding Ethnoreligious Groups in Iraq: Displacement and Return, page 6. 34 Grisgraber, Daryl. “Too Much Too Soon: Displaced Iraqis and the Push to Return Home.” Refugees International, September 25, 2017, page 9. 35 Fagen, pages 8 and 11. 36 Koser, Khalid. “The Return of Refugees and IDPs and Sustainable Peace.” Brookings Institution. July 29, 2016. 37 Luchtenberg, Kyra. “The Politics of Return in post-ISIS Iraq.” London School of Economics. October 23, 2018. 38 IOM Understanding Ethnoreligious Groups in Iraq: Displacement and Return, page 18. 39 IOM Understanding Ethnoreligious Groups in Iraq: Displacement and Return, page 35. 20 As Luchtenberg notes, IDPs with perceived ISIS ties fear returning because of discriminatory vetting by the state, militias, or the local populations.40 This discriminatory vetting can be teased through the PBS, as almost two-thirds of all returnee respondents noted that since they had returned they encountered a lack of security, with nearly 10 percent of Arabs polled in the PBS stating that they had encountered discrimination since returning.41 A further 25 percent of “Others” reported that they encountered discrimination since returning.42 Chasing this larger point, another USIP report argues that Iraq must remove sectarian militias that intimidate IDPs who wish to return home, while replacing that ad hoc security apparatus with a professional national army that reflects a multi-ethnoreligious Iraq.43 Lastly, one of the oft-overlooked impediments to return is the lack of documentation that IDPs possess, either being left behind, destroyed, or lost. As RI notes, “Numerous IDPs told [us] about their struggles to obtain the documentation needed in order to access services as IDPs or make the trip home. Many IDPs either left their personal documentation behind when they fled, lost it en route, or had it destroyed by ISIS. It is possible to have the papers replaced but the process is usually slow and often has to be initiated in Baghdad, which is out of reach for many. Without personal documentation, IDPs are generally not eligible for services from NGOs, cannot register for their monthly food distributions, cannot get permission to travel, and cannot register births and marriages, to name a few challenges.”44 Ultimately, the end of hostilities is not a strong enough pull factor on its own to facilitate return.45 Pull factors include the broad spectrum of essential basic services that all people need: a livelihood, a sense of security from both external groups and inter-communal discrimination, and access to quality education, water, and electricity. The secondary literature confirms much of the research done by the Evaluation Team and provides a level of confidence in triangulation of this evaluation’s findings and conclusions in reference to FFS’s discrete activities in Ninewa, Salah al-Din, and Anbar provinces. Stabilization Metrics – Stabilization is a relatively new phenomenon to the development space, and measuring its efficacy is still mired in debate. Furthermore, there is no consensus on whether outputs from development programs can be used to measure stabilization or if there is a correlation. USIP notes that there is a difference between success of implementation and the success of stabilization, and most implementers “are inclined to report their success at implementing programs rather than their impact on stabilization.”46 Furthermore, USIP reports, “Progress should be measured in terms of outcomes. Indicators should focus on measuring outcomes or processes leading directly to these outcomes, as opposed to assessing the number of products and services delivered or the amount of resources consumed.”47 The risk here is that FFS activities, as shown in this report, have been successful in implementation, and that their outputs are sound, but the attribution to stability may be less evident. In a paper for the Canadian Defense and Foreign Affairs Institute, Sarah Meharg underlines that “there are few available systems of measurement that allow for the quantitative and qualitative aspects of reconstruction and stabilization to be measured in a way that says something important about effectiveness.”48 Measuring the effectiveness of stabilization programming remains frustratingly illusive. 40 Luchtenberg. 41 FFS Evaluation PBS Question G13 “Since you have returned have you encountered a lack of security?” 42 FFS Evaluation PBS Question G14 “Since you have returned have you encountered discrimination?” 43 Strass, Fred. “How Iraq’s Minorities Can Secure a Future after ISIS.” USIP. September 7, 2017. 44 Grisgraber, page 8. 45 Costantini, Irene. “Displacement-Emigration-Return: Understanding Uncertainty in the Context of Iraq (why do IDPs emigrate as opposed to return home?).” Middle East Research Institute. 2018. 46 Cohen, Craig. “Measuring Progress in Stabilization and Reconstruction.” United States Institute of Peace. March 1, 2006. Page 1. 47 Ibid, page 2. 48 Maherg, Sarah Jane. “Measuring the Effectiveness of Reconstruction and Stabilization Activities.” In Measures of Effectiveness: Peace Operations and Beyond, 1st ed., 10:29–44. The Pearson Papers, n.d., page 42. 21 That said, Roberto Belloni is optimistic about FFS’s emphasis on returnees in its implementation. “The UNDP Stabilization Programme has set the return of IDPs to areas liberated from the Islamic State as its main performance criterion […] Contrary to standard realist worldviews […] the return and re-mixing of the population does not increase the possibility of violence, but rather diminishes it.”49 Thus, the output metrics identified and reported by FFS and evaluated here, at least in the eyes of some, should not be so easily dismissed. OBSERVATION 2 FFS has had demonstrable positive outcomes on the lives of thousands of returnees. USAID-funded activities under FFS have resulted in a measurable, positive change in the delivery of essential services in key geographical areas since 2015. These can be evidenced with key “Success Stories.” As will be seen in greater detail when the report turns to the four Evaluation Questions below, FFS has had immediate successes in contributing to stabilization in the key intervention areas of Anbar, Ninewa, and Salah al-Din. These are evidenced in the evaluation findings derived from the KIIs, FGDs, and PBS responses, but also in the 509 third-party monitoring site visits conducted via the PMSP Iraqi Field Monitors (IFMs) to July 2018. Of the 509 PMSP IFM site visits, the vast majority received overwhelmingly positive scores for services implementation based on level of completion and schedule; quality of delivery; quality of equipment and supplies; utilization; and, access. Three success stories represent FFS’s immediate positive outcomes, and are indicative of a cumulative positive outcomes and perceived impacts on the lives of returnees that FFS has had to date. 49 Belloni, Roberto, and Irene Costantini. “From Liberal Statebuilding to Counterinsurgency and Stabilization: The International Intervention in Iraq.” Ethnopolitics, 18, no. 5 (July 23, 2019): 509–25. https://doi.org/10.1080/17449057.2019.1640964. Page 519. 22 23 24 25 As previously mentioned, the evaluation data below supports the evidence of such “success stories” and perceptions of noticeable, positive, change at the province and intervention area levels from KII, FGD, PBS, and LS respondents. Also, the vast majority of self-reported indicator data from FFS shows targets being consistently achieved. This is supported by findings from FFS outputs in annual and quarterly reports. However, much of these outcomes are supported by only numerical outputs. As supported by other relevant literature, including the “U.K. Government’s Approach to Stabilisation,”50 the Evaluation Team therefore recommends two options for richer, contributory— and more accountable—measures of success, and one alternative method for tracking progress. This recommendation does not preclude the need for post hoc evaluations, and instead suggests USAID and FFS better and more comprehensively track progress over time and during implementation. This includes developing and implementing a more robust Activity, Monitoring, Evaluation, and Learning Plan (AMELP) and/or a Developmental Evaluation, both to better inform the refinement of FFS. For a more human-centric, normative projection of outcomes, the Evaluation Team also proposes using two qualitative analysis techniques to better identify positive change contributed by FFS during its implementation: Most Significant Change (MSC) and Appreciative Inquiry (AI). Both techniques are participatory, and should be conducted in a joint setting, with a facilitator familiar with USAID and the affected regions of Iraq, and ideally with stabilization principles and monitoring & evaluation techniques. Unlike a midterm or final performance evaluation, both MSC and AI are conducted at regular intervals within a period of performance (e.g., quarterly), with purposively sampled groupings of the intervention population describing perceived changes to their lives. The regular periodicity of the perceptions allows for systematic qualitative, emotive, data, which in turn can result in outcome harvesting to better identify successes and/or to identify needs for course correction. Finally, because measuring outcomes in stabilization contexts is challenging, USAID and FFS might also consider additional techniques to gather, measure, and disseminate metrics of success for complex, stabilization activities. These criteria can include practical steps on how to monitor and measure progress toward the achievement of FFS objectives and means to learn from the monitoring and measurement findings and to adapt. One way of doing this is to consider using process-based metrics so as to focus on the processes that lead to desired stabilization outcomes rather than just the outcomes themselves. Described below, process benchmarks may also be more conducive to sharing across the U.S. interagency so as to describe how well the stabilization activity is working. They can also be more easily digestible for policy and public audiences as a means to ensure transparency and accountability Most Significant Change The Most Significant Change (MSC) model is an approach to developmental evaluations that involve assessing the changes and, in turn, the impacts that people experience as a result of an intervention from the perspective of program participants. Performance and program evaluations can be informed by the MSC technique since it focuses on impact monitoring by supplying information at impact and outcome levels rather than activity or indicator levels thus allowing the review of program performance as a whole. This also implies that MSC is a useful methodology for evaluations of interventions that have vague, ambiguous, transitory, or undefined outcomes or face a shortage in monitoring data due to obstacles to data collection, such as the FFS operating environment. Furthermore, the very structure of the MSC approach directly contributes to capacity building efforts in several ways. MSC can help identify unexpected changes—both positive and negative—and deliver a more intricate and rich picture of what is happening, since the pool of stories from stakeholders is as wide as the experiences that the respondents have had throughout the duration of the intervention. Through the compilation of experiences and storytelling, the approach engages participants in the analysis of information, because 50 “The UK Government’s Approach to Stabilisation: A Guide for Policy Makers and Practitioners.” 26 there is need to provide a compelling argument when developing criteria for significance.51 The process involves stakeholders discussing stories that revolve around the perceived changes that have taken place as a result of the intervention, and then selecting the stories they consider most significant on the basis of a series of criteria defined by the group during discussions. The MSC methodology is sometimes referred to as monitoring-without-indicators, because it does not make use of indicators that can be tracked, or as the ‘story’ approach, given that the data collected is in the format of stories that describe the change that occurred, when it occurred, and the reasoning behind it.52 The MSC technique is not meant to be a stand-alone methodology, and as such the Evaluation Team suggests it be used in combination with the other evaluation methods to inform the normative projection of outcomes in very fragile environments. Appreciative Inquiry To augment the FFS output-based indicators, the Evaluation Team also suggests that the MSC technique include some Appreciative Inquiry (AI) approaches. AI is an organizational development tool and change process technique that inquires into, identifies, and further develops the best of “what is” in order to create a better future, and can be utilized as a means of addressing issues, challenges, changes, and concerns in ways that build on the successful experiences of members or participants.53 In a participatory session using an appreciative framework, the first question to purposive groups of beneficiaries, stakeholders, and FFS implementers would focus on stories of best practices, positive moments, successful and, processes. The AI approach to evaluation is “learning-oriented and views inquiry as ongoing, iterative, and integrated into community life.”54 Rather than ask participants to list problems that they are facing in relation to the intervention at hand, they are asked to explain what is going well, why it is going well, and what they want more of, essentially reframing problem statements into a focus on strengths and successes. The AI tools for evaluation work extremely well in contexts where previous evaluation efforts have failed, within hostile or volatile environments such as northwestern Iraq, where there is fear of or skepticism about evaluation, when change needs to be accelerated, and when relationships among individuals and groups have deteriorated. AI is beneficial in programs that are highly complex, where the technique can serve to reinstate what is valuable, useful, and important. Finally, AI is particularly useful in program evaluations by helping to identify and make explicit areas of good performance and to communicate and institutionalize what is already known about good performance so that it is furthered and replicated in future programming. Similar to the MSC approach, AI offers the opportunity for capacity-building because it fosters an optimistic co-construction of the future and emphasizes building on past successes and valuing inquiry, dialogue, and reflection. Process-based Metrics USAID and FFS should consider incorporating more meaningful process-based measurements of success. Stabilization programming is inherently output-based, with the near-term objectives of providing tangible, concrete, and immediate results to communities in an effort to appease, neutralize, and acquiesce, and to stabilize quickly and efficiently. That said, output-based metrics, while quantifiable and indeed necessary to measure immediate progress, do not sufficiently measure the ability to measure stabilization. Process-based metrics can be used to better illustrate progress toward the achievement of processes such as stabilization, which is not an end in itself but rather a series of steps contributing to an end. Process-based metrics are also more easily digestible for policy, congressional, and public audiences to understand programs and to ensure transparency and accountability to the U.S. Government and 51 Further, the MSC process can be used as a tool for community empowerment through the creation of mechanisms for dialoguing about diverse ideas, fostering a shared vision, and conceptualizing impact. 52 Davies, R. and Dart, J. (2005). “The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use.” 53 Fraser, D. and Rogers, P. (2003). “Appreciating Appreciative Inquiry.” New Directions for Evaluation. 54 Preskill, H. and Tzavaras Castambas, T. (2006). “Introducing Appreciative Inquiry.” Reframing Evaluation through Appreciative Inquiry. 27 other international donors. While process-driven metrics can appear counterintuitive to USAID, they can strike a balance between setting output-based targets and benchmarks and offering flexibility in the course of achieving it within the continuum of a process cycle. They may also be more conducive to sharing across donors to assess how well the overall stabilization approach is working (e.g., as a function of programmatic cycle, specific sub-activity implementation time, completeness, efficiency, capacity, and cost-effectiveness). 28 QUESTION 1: HOW HAVE FFS ACTIVITIES AFFECTED ACCESS TO SERVICES IN TARGET COMMUNITIES? FINDINGS 1A. CHANGES TO SERVICES AVAILABILITY IN TARGET COMMUNITIES RESULTING FROM FFS ACTIVITIES ACROSS ALL FFS TARGET SECTORS. Because of the difficult environment in which FFS was implemented, a baseline assessment to determine service access among beneficiaries prior to the FFS intervention was not possible. While the lack of a baseline makes it difficult to produce concrete findings demonstrating progress towards achieving improved access among beneficiaries, the Evaluation Team was able to determine perceptions of change in access via the PBS survey. Beneficiaries were asked retrospective questions about their access to services prior to 2015 and their access to services in 2019. These responses were analyzed along with a thorough review of UNDP progress reports and other related documents to determine changes to service availability in target communities. As of June 2019, FFS completed 849 activities in electricity (79), water (77), sewage (81), health (164), education (448), and municipality assistance (184). When asked whether respondents perceived services to have been restored after liberation, a large majority of respondents stated that they did believe that electricity, water, education, and health services were restored after liberation in their respective targeted intervention area (as illustrated in figures 5 and 6). Figure 5: Percentage of Households by Intervention Area that Perceived Destroyed Services to be Restored Restoration of sanitation systems in intervention areas sampled by the ET was limited in relation to other services. And in Mekeshifa, Shirqat, and Karma less than 50 percent of respondents perceived that 99% 89%99% 99% 69% 97% 96% 86%89%93%94% 90%97% 91% 64% 92% 87% 85% 74% 93% 70% 61% 56% 46% 29% 69% 58% 31% 55% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% East Mosul Falluja Qayara Karma Mekeshifa N. Plains Ramadi Shirqat Tikrit West Mosul Electricity Services Restored Water Services Restored Sanitation Services Restored DATA SOURCES, TERMINOLOGY, AND LIMITATIONS Unless otherwise noted, all cited data for this evaluation question refer to findings from the population￾based survey (PBS) as well as KIIs and FGDs with beneficiaries. The Evaluation Team utilized frequencies as well as measures of central tendencies to analyze PBS data. In Mosul, we expanded the water project of the old left side and Al￾Sairoon project. We have also done maintenance to a lot of projects in Telafar, and we have also dug wells in some areas. – KII with Male Member of Local Government in Ninewa 29 their sanitation services were restored. It is possible that this was due to the targeting of sewage/sanitation activities by FFS. The majority of the 77 sewage projects took place in East Mosul (43) and West Mosul (21). FFS conducted no activities relating to sanitation in Falluja, Tikrit, Mekeshifa, or Qayara, and only one sanitation activity was implemented in both Karma and Shirqat. The below charts illustrate data collected from both IDPs and Returnees alike that were located in the respective intervention areas at the time of the PBS survey. Figure 6: Percent of Households by Intervention Area that Perceived Destroyed Education and Health Services to be Restored The Evaluation Team analyzed measures of central tendency to identify the average change as reported by respondents in education access in 2019 compared to pre-2015 (illustrated in Table 10). The majority of PBS respondents acknowledged that 2019 standards were similar to or better than pre-2015 levels. Specifically, schools across intervention areas were perceived to be open for longer periods of time (both daily and weekly), and five intervention areas reported decreases in disruptions to the school year compared to before 2015. In Mekeshifa the average response indicated that the number of disruptions to the school year decreased by over a quarter when compared to pre-2015 levels. With the exception of Ramadi, where disruptions increased by over 25 percent compared to pre-2015, other intervention areas listed only minor (less than 10 percent) increases in disruptions in 2019 compared to 2015. 82%89% 81% 74% 53% 90% 77% 77% 74%81%86%95% 92% 90% 68% 89%95% 82% 75% 85% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% East Mosul Falluja Qayara Karma Mekeshifa N. Plains Ramadi Shirqat Tikrit West Mosul Health Services Restored after Liberation Education Services Restored after Liberation 30 Table 10: Changes to Education Access pre-2015 vs. 2019 – Average Responses across PBS Respondents Based on average response from respondents in a given intervention areas Metric E. Mosul Falluja Qayara Karma Meke￾shifa N. Plains Ramadi Shirqat Tikrit W. Mosul Days Open per Week 0% 4% 1% 1% -1% 0% 15% 8% -1% 0% Hours Open per Day 0% 4% 0% 0% -3% -1% 18% 9% 3% -1% Disruptions per Year -12% 8% -13% 4% -26% -16% 26% 9% 9% -15% Positive Change from pre-2015 to 2019 No Change 1 to 10% 11 to 20% 21 to 30% 31 to 40% Negative Change from pre-2015 to 2019 No Change 1 to 10% 11 to 20% 21 to 30 % 31 to 40% 31 Health services were also perceived to be more accessible across all intervention areas by respondents. In every targeted intervention area, respondents found they were slightly closer to a functioning government health facility in 2019 than they were prior to 2015. 1B. IS ACCESS TO SERVICES EQUITABLE ACROSS VULNERABLE GROUPS? Iraq has a rich history of ethnic and religious diversity, which has become a source of conflict and inequality. Following the 2005–07 sectarian war, invisible ethnic and religious barriers separated various groups and prevented many from straying far from their communities.55 This conflict has led to more pronounced spatial divergences, not only in population makeup of regions and communities but also in the delivery of services.56 Some literature also points at the difficulty that minorities face receiving national identity documents, which can in turn limit their ability to access services such as education and health.57 Yet, despite these constraints, the Evaluation Team found that while some differentiation existed across ethnicities and religious affiliations, the majority of vulnerable populations, including those living with disabled family members, and IDPs reported that they had access to water and electricity, while sanitation was not as accessible by the majority of Sunni Muslims and Arabs. As stated in 1a, this latter finding is possibly due to the lack of sanitation interventions by FFS in some areas that are largely inhabited by Sunni and Arab populations. 55 Iraq Systematic Country Diagnostic. (February 3, 2017). Retrieved January 14, 2020, from http://documents.worldbank.org/curated/en/542811487277729890/pdf/IRAQ-SCD-FINAL-cleared-02132017.pdf 56 Iraq 2018 International Religious Freedom Report. (n.d.). Retrieved from https://www.state.gov/wp￾content/uploads/2019/05/IRAQ-2018-INTERNATIONAL-RELIGIOUS-FREEDOM-REPORT.pdf 57 Country Guidance: Iraq. (June 2019). Retrieved January 14, 2020, from https://www.easo.europa.eu/sites/default/files/Country_Guidance_Iraq_2019.pdf Km to Nearest Gov. Health Facility 0 1 2 3 4 5 6 7 8 East Mosul Falluja Qayara Karma Mekeshifa N. Plain Ramadi Shirqat Tikrit West Mosul Intervention Areas 2019 Pre-2015 Figure 7: Average Distance (km) Household Members Must Travel to Nearest Functioning Government Health Facility 2019 vs. Pre-2015 – Percentage of PBS Respondents 32 Figure 8: Access to Basic Services by Religious Affiliation Figure 9: Access to Basic Services by Respondents’ Status – Percentage of PBS Respondents Figure 10: Access to Basic Services by Ethnic Origin – Percentage of PBS Respondents 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Electricity Service Meets Needs Potable Water Qty. Meets Daily Needs Connected to Public Sewer (Covered or Uncovered) Muslim Shia Muslim Sunni Christian Yazidi Other 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Electricity Service Meets Needs Potable Water Qty. Meets Daily Needs Connected to Public Sewer (Covered or Uncovered) IDP Returnee Never Left Home 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Electricity Service Meets Needs Potable Water Qty. Meets Daily Needs Connected to Public Sewer (Covered or Uncovered) Arab Assyrian Kurd Turkmen Shabak Other 33 Access to education was also equal across populations; almost 100 percent of respondents, including vulnerable groups, reported that their school-aged children were attending school (see Figure 11). Similarly, all respondents stated that they had a government health center accessible to them within walking distance (1–6.13 km), as shown in Figure 12. Figure 11: Percentage of Households with School-aged Children Attending School by Religion, Ethnicity, and Status Figure 12: Average Distance in Kilometers Households Must Travel to Nearest Functioning Government Health Center by Ethnicity, Religion, and Status 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Arab Assyrian Kurd Turkmen Shabak Other Muslim Shia Muslim Sunni Christian Yazidi Other IDP Returnee Never Left Home Ethnicity Religion Status 3.77 1.59 2.85 3.06 2.35 1.64 4.79 3.65 4.1 1.61 2.33 6.13 3.31 4.16 0 1 2 3 4 5 6 7 Arab Assyrian Kurd Turkmen Shabak Other Muslim Shia Muslim Sunni Christian Yazidi Other IDPs Returnees Never Left Ethnicity Religion Status 34 1C. WHAT SIGNIFICANT BARRIERS, IF ANY, PREVENT MEMBERS OF TARGET COMMUNITIES FROM ACCESS TO SERVICES? Very few respondents reported barriers to access. Those who did have difficulty mentioned the following: ● Electricity: Preferred use of private/shared generators over connection to the public service o This is a low occurrence: 181 out of 2,735 respondents (7 percent) identified that they were not connected to power even though there was an active electric grid in their neighborhood. Of the 181 respondents, when asked to mark all reasons that applied to their situation, 34 stated that the connection fee was unaffordable, 20 said their structure/dwelling did not meet standards, 27 stated that they could not afford the monthly subscription, 8 said the application was delayed, 82 preferred private or shared generators, and 26 responded “other.” ● Water: Little to no maintenance was provided at the access point o Of 2,735 respondents, when asked to mark all that apply, 410 (15 percent) said they did not have water pumped into their own dwelling. Of those, when asked to mark all barriers to water access that applied to their situation, 98 identified damaged/vandalized infrastructure, 38 said the water source had dried, 132 replied that the system was of poor design, 186 said there was a lack of maintenance at the water point they used, and responded “other.” ● Sanitation: There was no regulation requiring homes to have connection to sanitation systems o While this does not appear to be a hard obstacle, no requirement for sewage connections at the domicile could result in a lack of motivation to connect one’s home to the public system which could be considered as a challenge for some individuals. o 1,485 out of 2,735 respondents (54 percent) stated that they were not connected to a public sewage/water drainage system (covered or uncovered canal), however among those, 1,278 (47 percent) stated they did have access to a septic tank/sub tank. 207, or eight percent were either connected to other means of waste disposal were not connected to any means of waste disposal. When asked to mark all reasons that applied as to why they were not connected to a public system, 255 stated they were not permitted to be connect to the sewer system (reasons were not provided), 692 said there was no regulation to do so, 114 were not willing to pay the full cost to connect to the sewage system, 105 relayed that they did not know enough or were not aware of the public system, 269 perceived the standards of sewage system construction were an obstacle, 221 identified poor operation and maintenance of the sewage system, and 222 responded “other.” ● Health: Facilities did not offer specific services needed o Of 2,734 respondents, 640 (23 percent) identified that they were not assisted at their nearest health facility when seeking services. Of these, 315 related that the health center did not offer the service they sought, 35 said the facility was closed, 135 mentioned that the wait was too long, 89 stated that the cost was prohibitive, and 46 responded “other.” ● Education: Distance to school o Of 2,735 respondents, 1,713 (63 percent) had a member of their household of school age. Among those, 54 (3 percent) stated that their child/children was/were not attending school. When asked to mark all reasons that applied as to why their child/children 35 was/were not in school, 27 said the school was too far away, 5 identified that the school was in a restricted zone, 8 stated insecurity and personal safety issues, 14 relayed that the cost was prohibitive, 11 mentioned the poor quality of the schools, 2 stated their child had an illness or disability, 2 said the school teachings conflicted with their beliefs, and 9 responded “other.” o The FFS team understands that pointing out “distance to schools” as a main barrier may appear to contradict the above finding that the average distance to schools is only 6 km. However, the Evaluation Team would like to point out that the majority of respondents did not find any barriers at all to their children attending school, and that distance was only an issue for the majority of those who were not able to put their children in school (27 respondents out of 54). CONCLUSIONS The Evaluation Team found that activities implemented by FFS positively affected access to services in all targeted communities. Respondents from the PBS reported that water, electricity, education, and health services in FFS intervention areas surveyed by PMSP in Qayara, East Mosul, West Mosul, Ninewa Plains, Ramadi, Falluja, Karma, Tikrit, Mekeshifa, and Shirqat were restored to pre-ISIS levels. However, respondents currently residing in the areas sampled perceived themselves to have limited access to sanitation services in relation to other services. The majority of respondents that stated sanitation systems had been destroyed during ISIS occupation in Karma, Mekeshifa, and Shirqat, reported that they perceived that those systems were not yet restored in their communities. Low levels of restoration to sanitation systems in these areas could be attributed to the level of sanitation activities implemented. While FFS did support sanitation activities in some areas, there was a correlation between areas with low rates of positive responses and low numbers of FFS sanitation interventions. Very few respondents reported barriers to access to services and those who did reported reasons that were outside of the scope of the FFS intervention. The following obstacles were identified by sector: ● Electricity: Preferred use of private/shared generators over connection to the public service o This is a low occurrence: 181 out of 2,735 respondents (7 percent) identified that they were not connected to power even though there was an active electric grid in their neighborhood. Out of the 181 respondents, when asked to mark all reasons that applied to their situation, 34 stated that the connection fee was unaffordable, 20 said their structure/dwelling did not meet standards, 27 stated that they could not afford the monthly subscription, 8 said the application was delayed, 82 preferred private or shared generators, and 26 responded “other.” ● Water: Little to no maintenance was provided at the access point o Of 2,735 respondents, 410 (15 percent) said they did not have water pumped into their own dwelling. Of those respondents, when asked to mark all barriers to water access that applied to their situation, 98 identified damaged/vandalized infrastructure, 38 said Measurement of Services Restored The Evaluation Team asked respondents whether water, electricity, sanitation, education, and health systems, respectively, existed prior to ISIS occupation in their communities, and, if so, whether those systems were damaged. Those who responded yes were subsequently asked if they perceived that those services had been restored. 36 the water source had dried, 132 replied that the system was of poor design, 186 said there was a lack of maintenance at the water point they used, and, responded “other.” ● Sanitation: There was no regulation requiring homes to have connection to sanitation systems o While this does not appear to be a hard obstacle, no requirement for sewage connections at the domicile could result in a lack of motivation to connect one’s home to the public system, which could be considered as a challenge for some individuals. o Of 2,735 respondents, 1,485 (54 percent) stated that they were not connected to a public sewage/water drainage system (covered or uncovered canal); however, among those, 1,278 (47 percent) stated they did have access to a septic tank/sub- tank. Two hundred seven (8 percent) of respondents were connected to other means of waste disposal or none. When asked to mark all reasons that applied as to why they were not connected to a public system, 255 stated they were not permitted to connect to the sewer system (reasons were not provided), 692 said there was no regulation to do so, 114 were not willing to pay the full cost to connect to the sewage system, 105 relayed that they did not know enough or were not aware of the public system, 269 perceived the standards of sewage system construction were an obstacle, 221 identified poor operation and maintenance of the sewage system, and 222 responded “other.” ● Health: Facilities did not offer specific services needed o Of 2,734 respondents, 640 (23 percent) identified that they were not assisted at their nearest health facility when seeking services. Of these, 315 related that the health center did not offer the service they sought, 35 said the facility was closed, 135 mentioned that the wait was too long, 89 stated that the cost was prohibitive, and 46 responded “other.” ● Education: Distance to school o Of 2,735 respondents, 1,713 (63 percent) had a member of their household of school age. Among those, 54 (3 percent) stated that their child/children was/were not attending school. When asked to mark all reasons that applied as to why their child/children was/were not in school, 27 said the school was too far away, 5 identified that the school was in a restricted zone, 8 stated insecurity and personal safety issues, 14 relayed that the cost was prohibitive, 11 mentioned the poor quality of the schools, 2 stated their child had an illness or disability, 2 said the school teachings conflicted with their beliefs, and, 9 responded “other.” The Evaluation Team did not identify or encounter respondents who identified their ethnicity, religious affiliation, gender, or other vulnerable factors as a barrier to access to services. RECOMMENDATION 1.1 According to PBS respondents in the 10 surveyed FFS intervention areas, 97 percent of returnees across all surveyed areas have access to schools for their children. Given this, supporting education infrastructure (not quality, content, or capacity) may be reaching a state of diminishing returns for the purposes of stabilization in the 10 surveyed intervention areas, although not necessarily for longer-term transition or development. USAID and FFS may wish to consider reviewing its programming to focus activities and resources (a) tactically on all sectors in more newly liberated areas, or (b) tactically on other less entrenched sectors for returnees in selected areas where there are incongruous signs of output and outcome achievement. Additionally, the donor community or GOI may want to consider a strategic shift from a stabilization to development approach in the education sector in the 10 surveyed intervention 37 areas, including, for example, increasing community management capacity functioning and local governance capacity functioning. 38 QUESTION 2: HOW HAVE FFS ACTIVITIES AFFECTED THE QUALITY OF SERVICES IN TARGET COMMUNITIES? FINDINGS 2A. SECTOR-SPECIFIC PARAMETERS OF QUALITY OF SERVICES TARGETED BY FFS To establish quality of services for sector-specific parameters, the Evaluation Team visited 24 facility￾infrastructure sites for observation. It selected eight in each province, representing 20 percent of the selected PBS sites. The observed sites (see Table 11) included facilities for electricity (generators and substations), and sewage/water treatment facilities. Table 11: List of On-Site Observations On-Site Observational Visits Location Sector Project (Case) Name 1-1-Ramadi Electricity Generator in Tameem, Ramadi 1-1-Ramadi Electricity Mobile Substations for Anbar province Ramadi Electricity Rehabilitation of Secondary Electrical Power Station No.5 Close to the Arts College in Anbar University, Ramadi Ramadi Electricity Diesel Generator (250 kVA) in Ramadi Tikrit Electricity 33/11KV Tikrit MSS for 1x16MVA for Salahdeen Electricity Directorate in Alam Tikrit Electricity 33/11KV Mobile Sub-Station for Tikrit University in Tikrit Shirqat Electricity Provision of Recommended Equipment of (132/33/11 KV) Sub￾station of Al Shirqat in Shirqat Mosul-West Electricity Sinae Ayman 33/11 substation N. Plain Electricity Substation (33/11 KV) (2x31.5 MVA) in Bashiqa, Ninewa Plain Mosul-East Electricity Mobile Sub-Station (25 MVA) for Ninewa, Mosul-East Mosul-East Electricity Mobile Sub-Station (25 MVA) for Ninewa, Mosul-East Ramadi Sewage Waste Water Treatment Plant in 7 Kilo, Ramadi Karma Sewage Street Sewage Lift Station in Karma Shirqat Sewage Lifting pump stations in Shirqat Mosul-East Sewage Sewerage system in Hay Al Samah (Lot 4) Ninewa Plain Sewage Generator 50 KVA to Talkaif Sewerage Plant Fallujah Water Al Azragiyah Water Complex (2,100 m3/h) in Fallujah Karma Water Karma Main Water Treatment Plant (2,000 m3/h) in Karma Tikrit Water Tikrit Central Water Project in Tikrit Mekeshifa Water Mekeshifa Water Project (250 m3/hr) in Mekeshifa Shirqat Water Al-Faris Water Treatment Plant Compact Unit 200 m3/HR in Shirqat Tikrit Water Al Qadisia 2 Water Project Near Tikrit University in Tikrit Qayara Water Qayara Water Treatment Plant in Qayara N. Plain Water Hama Alaleel Water Treatment Plant in Ninewa Plain DATA SOURCES, TERMINOLOGY, AND LIMITATIONS Unless otherwise noted, all cited data for this evaluation question refer to findings from the population￾based survey (PBS), KIIs and FGDs with beneficiaries, and on-site observational data collected by the FFS Evaluation Team. 39 Electricity: Nine FFS sponsored substations and three generators were visited by the electricity subject matter specialist. Overall, the electrical substations and generators visited were functioning properly and no substantial technical issues were reported, except for one substation where a circuit breaker was out of order. All eight substations and the three generators visited for on-site observations were operating at the required capacity. One substation was operating at less than full capacity due to routine maintenance by the electricity department. The majority of substations and the two generators visited were providing electricity from 12 to 18 hours and only two were providing electricity for 24 hours, which is consistent with what was reported by key informants and PBS data. Water/Sanitation: The Evaluation Team observed quality control laboratory and chemical engineers for testing the influent in producing water quality. In all three provinces the average quality of water was rated at 80 percent. Tests were conducted either daily or weekly. The following tests were conducted: pH, color test, turbidity (NTU), residual chlorine Cl, hardness in terms of CaCO3, fecal coliform (FC), total coliform count (TC), and heavy metals. In most sites there were filtration units that were not working. During the days of visits the water quality test results were all positive and of acceptable range. 2B. PERCEPTIONS OF THE QUALITY OF SERVICES AMONG MEMBERS OF TARGET COMMUNITIES Because of the difficult environment in which FFS was implemented, a baseline assessment to determine service quality among beneficiaries prior to the FFS intervention was not possible. While the lack of a baseline makes it difficult to produce concrete findings demonstrating progress towards achieving improved service quality among beneficiaries, the Evaluation Team was able to determine perceptions of change in quality via the PBS survey. Beneficiaries were asked retrospective questions about their access to services prior to 2015 and their access to services in 2019. These responses were analyzed along with a thorough review of UNDP progress reports and other related documents, and KII and FGD transcripts to determine changes to service availability in target communities. The Evaluation Team determined that FFS activities restored the quality of electricity, water, and sanitation services to pre￾2015 levels. Electricity: Connectivity to electricity exceeded pre-2015 levels according to PBS data. Generally, respondents reported a more consistent supply of electricity in 2019 than prior to 2015. Among participants of the survey, 0 percent reported receiving no electricity in 2019 compared to 4 percent prior to 2015. Additionally, the number of respondents who reported that they received electricity 24 hours a day in 2019 increased by 4 percent compared to those who received electricity 24 hours per day in 2015. Increases in electricity connectivity are illustrated in Figure 13 below. 40 Figure 13: Percentage of Households' Hours of Connectivity to Electricity in a Given Day 2019 v pre-2015 Water: Water quality was also restored to pre-2015 levels. Across intervention areas, respondents stated that the daily quantity of potable water received was adequate to meet their needs. The Evaluation Team also found that respondents were using water in 2019 at pre-2015 levels, e.g., an equal number of respondents who used 76–100 buckets of water per day in 2015 were able to retrieve the same amount in 2019. Respondents also perceived the quality of water to be slightly better in 2019. Sixty percent of respondents found its taste to be good, 64 percent found the smell to be good, and about 58 percent of respondents’ perceived water to be of good clarity. These percentages are slightly improved from perceptions of water quality prior to 2015, as illustrated in Figure 14, in which a lower percentage of respondents perceived the water clarity to be good. Additionally, as shown in Figure 15, the majority of respondents in all intervention areas perceived that, since liberation, they have sufficient potable water to meet their needs. 5% 17% 39% 35% 4% 12% 39% 39% 11% 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 24 hours/day 13-23 hours/day 4-12 hours/day 1-3 hours/day None 24 hours/day 13-23 hours/day 4-12 hours/day 1-3 hours/day None Pre-2015 2019 41 Figure 14: Perception of Water Quality in 2019 Compared to Pre-2015 - Based on Percentage of Households that Perceived the Taste and Smell to be Good and the Clarity to be Clear Sanitation: Respondents reported little issue with the quality of sanitation systems that are currently functioning in their communities. Almost 72 percent of the users had experienced no or infrequent problems with sanitation services, while only 26 percent reported frequent sewer breakdowns. KII and FGD participants noted that the sewer systems were operating normally because most of the households used their own septic tanks and the municipality was collecting the waste. 2C. PERCEPTIONS OF THE EFFECTIVENESS OF GOVERNMENT ADMINISTRATION OF SERVICES AMONG MEMBERS Across intervention areas and sectors, the average responses from participants of the PBS survey reported they were either “somewhat satisfied” or “somewhat dissatisfied” with local authorities’ and/or utility companies’ administration of services (see Table 12). Respondents perceived sanitation to have been particularly mishandled by local authorities and/or utility companies. 50% 55% 60% 65% Taste Smell Clarity 2019 Pre-2015 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% East Mosul Falluja Qayara Karma Mekeshifa N. Plain Ramadi Shirqat Tikrit West Mosul Figure 15: Percentage of Households that Perceive the Quantity of Potable Water is Adequate for Daily Needs Today 42 Table 12: PBS Respondents’ Satisfaction with Administration of Services Based on average response from respondents in a given intervention area Service E. Mosul Falluja Qayara Karma Meke￾shifa N. Plains Ramadi Shirqat Tikrit W. Mosul Electricity 2 3 2 2 3 2 3 2 3 2 Water 2 3 2 2 3 2 3 3 3 2 Sanitation 3 3 3 3 4 3 3 3 3 3 Health 3 2 2 2 3 2 3 2 2 3 Education 2 2 2 2 3 2 2 2 3 3 Satisfaction with Administration of Services 1 = Very Satisfied 2 = Somewhat Satisfied 3 = Somewhat Dissatisfied 4= Very Dissatisfied Fifty-seven percent of the PBS respondents had positive views (“somewhat satisfied” or “very satisfied”) about the local authority’s ability to maintain health infrastructure, 53 percent in its ability to respond to outbreaks, and 56 percent in its ability to provide the required number of health workers. Overall, there were no significant differences across various groups on perception about the effectiveness of government to administer health services. Overall, almost two-thirds of the respondents were either “very satisfied” or “somewhat satisfied” with the local authority’s ability to maintain education infrastructure. Fifty-three percent were “satisfied” with the authority’s ability to provide teaching and learning materials, and 58 percent were “satisfied” with the ability to conduct school inspections. With the exception of Mekeshifa, Tikrit, and West Mosul, respondents of most implementation areas were generally satisfied with the administration of education services. KII consultations with the Directorates of Education revealed that the inadequate number of teachers due to displacement affected their efficacy in dealing with the needs of the communities served; in addition, schools were rehabilitated irrespective of the catchment areas populations providing education services. Overall, about half of the respondents had positive views (“somewhat satisfied” or “very satisfied”) on the local authority’s ability to administer electricity in terms of processing new applications, checking meters, providing customer services, billing, and responding to service interruptions. About two-thirds of the respondents had positive views on the local authority’s maintenance of existing infrastructure. Significant variations were noted among the provinces. Whereas more than two-thirds of the respondents had positive views in Ninewa, fewer than half of the respondents in Anbar and Salah al-Din had positive views. At the intervention area level, East Mosul had the highest number of respondents (more than 90 percent) with positive views of administration and Mekeshifa had the lowest number of respondents (less than one-third) with positive views. It should be noted that 56 out of 79 FFS activities that supported electricity projects were in Ninewa, of which 34 were implemented in East Mosul, compared to seven electricity projects being implemented in the Mekeshifa intervention area of Salah al￾Din. This may account for the low positive response in the Ninewa province. Sixty-six percent of the respondents were either “very satisfied” (28 percent) or “somewhat satisfied” (38 percent) with the local authority’s ability to maintain the water utility infrastructure. Fifty-nine percent were “satisfied” with the authority’s efficient response to repair service interruptions, and 56 percent were “satisfied” with the regularity of billing services. However, some higher levels of 43 dissatisfaction were reported on the regularity of checking water bills (43 percent). Respondents also lamented that there was poor customer service and complaint handling (43 percent). The level of satisfaction followed the same pattern across all the project intervention areas and segments of population. Only 41 percent were satisfied with the ability of the local authorities to maintain and rehabilitate sanitation infrastructure. Thirty-four percent reported that the utility providers were responsive to non￾functioning and dysfunctional drainages, and 45 percent were satisfied with the municipalities’ collection of solid waste. Thirty-six percent reported councils emptied waste for households not connected to sanitation systems. CONCLUSIONS Utilizing information from KIIs, FGDs, the PBS perception survey, and on-site observations, the Evaluation Team determined that FFS activities did restore the quality of basic services— including electricity, water, and sanitation—to pre-2015 levels. While FFS supported infrastructure development, some services, such as education and health, required technical knowledge and human capacity to improve quality of service. According to KII and FGD respondents, much of the human capital (trained health and education professionals) who resided in implementation areas fled during the ISIS occupation. As of this report, many had not yet returned. Because training of health and education personnel was not included as part of the FFS scope, many of the health centers and schools continue to have an employment gap. As illustrated in Table 12, respondents felt mixed about local authorities’ administration of services across implementation areas. When asked via the PBS survey their level of satisfaction with local authorities’ administration of various services, respondents’ replies ranged from “somewhat satisfied” (2) to “very dissatisfied” (4). Respondents particularly disapproved of the administration of sanitation. Across all implementation areas, the average rating of administration of sanitation services was “somewhat dissatisfied” (4) and, in Mekeshifa, “very dissatisfied” (4). RECOMMENDATIONS 2.1 Based on respondents’ low satisfaction with local administration of sanitation services, FFS should consider working with local authorities to build capacity and improve administration of sanitation services (Window Three) to ensure improved service provision to beneficiaries and improve local governance through service management. If this does not fall within FFS’s scope of work under Window Three, the Evaluation Team recommends that USAID or other donors consider this in the design of future programming. 44 QUESTION 3: HOW HAVE FFS ACTIVITIES AFFECTED IDPS IN TARGET COMMUNITIES? FINDINGS 3A. TO WHAT EXTENT HAVE FFS ACTIVITIES INFLUENCED IDPS’ DECISIONS TO RETURN TO THEIR COMMUNITIES OR INFLUENCED OTHERS TO REMAIN IN THEIR COMMUNITIES RELATIVE TO OTHER FACTORS? • Drivers of return addressed by FFS— primarily restoration of essential services, livelihoods support, and rehabilitation of residences—were not the primary factor influencing return in any intervention area in the year(s) the majority of IDPs returned. In all intervention areas save one, security was the primary driver of return as reported by 77 percent to 99 percent of surveyed respondents who returned during the specific year(s) of interest. • In all but one intervention area, essential services and residential rehabilitation were among the top five drivers of return for between 19 percent and 69 percent of the population, depending on the intervention area and intervention type (see Table 13). Livelihoods support was not among the top five factors influencing return in any intervention area; it was cited by between 3 percent and 24 percent of returnees depending on the intervention area. • When considering drivers of return as reported by returnees, annual trends in returns, and the timing and type of FFS projects by intervention area, the data suggest the following: • Between 77 percent and 96 percent of FFS projects in each intervention area responded to drivers of return for at least 10 percent of returnees who returned to their homes during the year or years the highest number of surveyed returnees reported returning (see blue bars in Figure 16). In each intervention area, the number of drivers of return not addressed by FFS during the year(s) of highest returns, but within FFS’s mandate, varied from zero in Ramadi to four in Qayara (noted for each intervention area in Figure 16). This suggests that FFS effectively identified and responded to drivers of return. • However, data from three intervention areas suggest the support may not have been timely. In Karma, Tikrit, and West Mosul the number of returnees declined as the volume of FFS projects increased. Therefore, while responding to factors influencing returns, FFS activities likely did not influence returns. This is not surprising given that in Data Sources, Terminology, and Limitations Unless otherwise noted, all cited data for this evaluation sub-question refer to findings from the population-based survey (PBS). The Evaluation Team surveyed a representative sample of the population in FFS intervention areas as of June 2018 in Salah al-Din, Anbar, and Ninewa. Respondents included 2,158 returnees (former IDPs or refugees who returned home at any time prior to the survey), 278 IDPs (those displaced at the time of the survey and living in host communities, not in camps), and 299 residents who were never displaced from their home communities. Data referring to ISIS military defeats are drawn from FFS reports. The analysis that follows focuses on responses from returnees who returned during the year or years the majority of respondents reported returning to a given intervention area to identify which factors influenced their decisions to do so. Due to the use of gridded population data to develop the sample for the population-based survey, representativeness of disaggregated groups (e.g., returnees, IDPs, etc.) is not possible at the intervention area level. Therefore, the intervention area-level findings below are not representative of all returnees in a given intervention area. They do however provide meaningful evidence of drivers of return in each intervention area. 45 all intervention areas, surveyed returnees reported security as a bigger driver of return than any factors addressed by FFS. • Mekeshifa is excluded from the intervention area-level analysis because the majority of surveyed returnees (61 percent) returned to the intervention area in 2015, prior to the start of FFS implementation in 2016, suggesting that FFS activities could not have significantly influenced decisions to return. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Shirqat(3) East Mosul (3) Ninewa Plains (1) Qayara (4) Ramadi (0) Falluja (3) Karma (2) Tikrit (2) West Mosul (3) % of FFS projects through year of highest returns responding to drivers of return cited by 10% or more of returnees that year % of FFS projects through year of highest returns responding to drivers of return cited by less 10% of returnees that year Figure 16: Drivers of return as reported by surveyed returnees and addressed by FFS during year(s) of highest return by intervention area 46 Table 13 shows the top five factors influencing returns by intervention area. Those in dark orange indicate factors cited by a majority of surveyed returnees who reported returning during the year or years the majority of surveyed returnees returned. Those in medium orange indicate factors cited by 25 percent to 49 percent of surveyed returnees, and those in light orange, less than 25 percent. Table 13: Top Five Factors Influencing Return by Intervention Area, in the Year(s) of Highest Returns Intervention area Factor #1 Factor #2 Factor #3 Factor #4 Factor #5 East Mosul Security (77%) Electricity (40%) Property ownership (36%) Water and sanitation (32%) Rehabilitation of residences58 and education (31%) Falluja Security (90%) Health and education (49%) N/A Electricity (48%) Water and sanitation (44%) Qayara Security (82 percent) Electricity (69%) Education (61%) Health and water and sanitation (57%) N/A Karma Security (98%) Education (52%) Electricity (52%) Health (48%) Water and sanitation (37%) Mekeshifa None (51%) Security (49%) Police stations (32%) Markets (30%) Business grants (3%) Ninewa Plains Security (81%) Electricity (53%) Water and sanitation (52%) Health (47%) Rehab of residences and education (43%) Ramadi Security (99%) Education services (27%) Health (26%) Rehabilitation of residences (20%) Electricity (19%) Shirqat Security (91%) Electricity (51%) Health and education (41%) N/A Police stations (26%) Tikrit Security (94%) Health (46%) Education (42%) Electricity (42%) Water and sanitation (31%) West Mosul Security (79%) Health (36%) Property ownership (36%) Education and water and sanitation (34%) N/A Table 14 shows the year the majority of surveyed returnees reported returning (highlighted in yellow), and therefore the time frame of analysis for each of the intervention areas, as discussed below. 58 In 2017, the year of highest returns among surveyed returnees in East Mosul, FFS implemented education projects but did not rehabilitate any residences. 47 Table 14: Percentage of Returnees by Year and Intervention Area 2012 2013 2014 2015 2016 2017 2018 2019 East Mosul 0% 0% 3% 3% 3% 64% 23% 2% Falluja 1% 0% 0% 0% 40% 51% 7% 1% Qayara 0% 0% 1% 3% 11% 73% 9% 0% Karma 0% 0% 1% 3% 11% 73% 9% 0% Mekeshifa 0% 0% 4% 66% 21% 4% 4% 0% Ninewa Plains 0% 0% 1% 1% 4% 76% 14% 2% Ramadi 0% 0% 2% 9% 38% 38% 10% 1% Shirqat 0% 0% 0% 0% 7% 58% 28% 6% Tikrit 0% 0% 0% 39% 38% 19% 2% 1% West Mosul 0% 0% 1% 1% 3% 51% 39% 4% East Mosul Table 15 compares the percentage of selected surveyed returnees who report a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (64 percent) returned to East Mosul. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees cited seven factors of return that FFS addressed during the period of their return. In descending order of importance, they are, as highlighted in yellow above, electricity, water and sanitation, education, health, employment and livelihoods, police stations, and roads and bridges. There are three factors, highlighted in orange in the table within FFS’s mandate but which FFS did not address in East Mosul prior to 201859—rehabilitation of residences (cited by 31 percent of returnees in 2017 as a consideration in their decision to return), and availability of business grants and access to markets (18 percent each). FFS supported 31 projects cited by 5 percent or less of 2017 returnees (highlighted in blue), suggesting a low return on investment of these projects. 59 A reminder to the reader that the analysis for all intervention areas is focused on the year the majority of surveyed returnees reported returning to their homes, facilitating an analysis of any correlation between FFS interventions and decisions to return (using factors of return cited by returnees who responded to the PBS as a proxy). Therefore, FFS interventions that were implemented after the majority of surveyed returnees returned are not considered in the analysis. 48 Table 15: Comparison of factors influencing return as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects60 Security 77% N/A Access to electricity 40% 28 Property ownership 36% N/A Access to water and sanitation services 32% 36 Rehabilitation of residences 31% 0 Access to education 31% 67 Access to health services 30% 14 Employment and livelihoods opportunities 23% 15 (all CFW) Presence of family 21% N/A Home community 20% N/A Availability of business grants 18% 0 Availability of markets 18% 0 Police stations 11% 2 Roads and bridges 5% 2 Municipality support 0% 29 60 Data source: project list provided by USAID to IBTCI in July 2019. 49 There were a number of factors important to decisions to return that fell outside FFS’s mandate or manageable interest. In descending order of importance these include security, property ownership, the presence of family, and a desire to return to their home community. Figure 17 shows that the launch of the 256 FFS projects started between 2015 and June 2018 coincided with the number of surveyed respondents who reported returning between 2015 and 2018. Both the highest number of projects started (193) and highest number of returns to home communities (141) in East Mosul happened in 2017, followed by 63 FFS projects started in the first half of 2018 and 52 returnees that year. This data, coupled with the crosswalk of factors of return and number of FFS projects started in 2017, as shown in Table 15, suggests a correlation between FFS activities and decisions to return in East Mosul, with the caveat that decisions to return were also, as reported by returnees, largely driven by security considerations which would have been impacted by the military defeat of ISIS at the end of 2016. Falluja Table 16 compares the percentage of selected surveyed returnees who reported that a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (51 percent) returned to Falluja. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Falluja cited seven factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) education, health services, electricity, water and sanitation services, rehabilitation of residences, and employment and livelihoods opportunities. There are three factors, highlighted in orange in the table, within FFS’s mandate but which FFS did not address in Falluja prior to 2018—availability of markets (cited by 22 percent of returnees in 2017 as a consideration in their decision to return) and availability of business grants and police stations (19 percent each). However, FFS supported 12 projects related to factors cited by 2 percent or less of 2017 returnees as drivers of return, suggesting a low return on investment of these projects (highlighted in blue). Figure 17: Number of FFS projects started and number of surveyed returnees, by year, for East Mosul 0 20 40 60 80 100 120 140 160 180 200 2015 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military Defeat of ISIS:Q4 2016 50 Table 16: Comparison of factors influencing return in Falluja as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 90% N/A Access to education 49% 51 Access to health services 49% 7 Access to electricity 48% 1 Access to water and sanitation services 44% 7 Rehabilitation of residences 30% 9 Presence of family 27% N/A Availability of markets 22% 0 Availability of business grants 19% 0 Police stations 19% 0 Property ownership 18% N/A Employment and livelihoods opportunities 15% 30 Home community 14% N/A Roads and bridges 2% 3 Municipality support 0% 8 Access to other grants 0% 1 51 There were a number of factors important to decisions to return but outside FFS’s mandate or manageable interest. In descending order of importance these include security, the presence of family, property ownership, and a desire to return to their home community. Figure 18 shows that the launch of the 126 FFS projects started between August 2016 and June 2018 coincided with the number of surveyed respondents who reported returning between 2016 and 2018. Both the highest number of projects started (97) and highest number of returns to home communities (105) in Falluja happened in 2017, followed by 20 FFS projects started in 2016 and 83 returnees that year. This data, coupled with the crosswalk of factors of return and number of FFS projects started in 2017, as shown in Table 16, suggests a correlation between FFS activities and decisions to return in Falluja, with the caveat that decisions to return were also, as reported by returnees, largely driven by security considerations which would have been impacted by the military defeat of ISIS at the end of 2016. Qayara Table 17 compares the percentage of selected surveyed returnees who report a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (73 percent) returned to Qayara. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Qayara cited five factors of return that FFS addressed during the period of their return. In descending order of importance, they are, as highlighted in yellow above, electricity, education, health services, water and sanitation services, and employment and livelihoods opportunities. However, with the exception of education, FFS implemented very few projects to address these concerns during the height of returns, with only one to two projects in each sector. There are four factors, highlighted in orange in the table, within FFS’s mandate that FFS did not address in Qayara prior to 2018— rehabilitation of residences (cited by 49 percent of returnees in 2017 as a consideration in their decision to return), availability of business grants (35 percent), availability of markets (22 percent), and police stations (16 percent). FFS supported nine municipality projects that did not relate to any factors influencing decisions to return (highlighted in blue). Figure 18: Number of FFS projects started and number of surveyed returnees, by year, for Falluja 0 20 40 60 80 100 120 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q2 2016 52 Table 17: Comparison of factors influencing return in Qayara as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 82% N/A Access to electricity 69% 1 Access to education 61% 42 Access to health services 57% 1 Access to water and sanitation services 57% 2 Property ownership 53% N/A Rehabilitation of residences 49% 0 Presence of family 35% N/A Availability of business grants 35% 0 Home community 29% N/A Availability of markets 22% 0 Employment and livelihoods opportunities 22% 1 Support system 20% N/A Police stations 16% 0 Municipality support 0% 9 There were a number of factors important to decisions to return but outside FFS’s mandate or manageable interest. In descending order of importance these include security, property ownership, the presence of family, a desire to return to their home community, and the presence of a support system. Figure 19 shows that the launch of the 67 FFS projects started between August 2016 and June 2018 coincided with the number of surveyed respondents who reported returning between 2016 and 2018. Both the highest number of projects started (55) and highest number of returns to home communities (51) in Qayara happened in 2017. This data, coupled with the crosswalk of factors of return and number of FFS projects started in 2017, as shown in Table Figure 19: Number of FFS projects started and number of surveyed returnees, by year, for Qayara 0 10 20 30 40 50 60 # of FFS projects and # of surveyed 2016 2017 2018 returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q3 2016 53 17, suggests a correlation between FFS activities and decisions to return in Qayara. However we note two significant caveats to this suggested correlation: (1) decisions to return, as reported by returnees, were also largely driven by security considerations that would have been impacted by the military defeat of ISIS in mid-2016, and (2) FFS implemented very few projects related to three return factors reported as important decision points by a majority of returnees. Karma Table 18 compares the percentage of selected surveyed returnees who reported that a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2016 and 2017, the years the majority of surveyed returnees (91 percent) returned to Karma. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Karma cited seven factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) electricity, education, health services, water and sanitation services, availability of markets, police stations, and employment and livelihoods opportunities. There are two factors, highlighted in orange in the table, within FFS’s mandate that FFS did not address in Karma prior to 2018—rehabilitation of residences (cited by 25 percent of returnees in 2017 as a consideration in their decision to return) and availability of business grants (15 percent). FFS supported four municipality projects that did not relate to any factors influencing decisions to return (highlighted in blue). Table 18: Comparison of factors influencing return in Karma as reported by surveyed returnees who returned in 2016 and 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 98% N/A Access to electricity 52% 1 Access to education 52% 19 Access to health services 48% 7 Access to water and sanitation services 37% 7 Presence of family 26% N/A Rehabilitation of residences 25% 0 Availability of markets 25% 1 Police stations 23% 1 Home community 22% N/A Property ownership 19% N/A Availability of business grants 15% 0 Employment and livelihoods opportunities 15% 3 Municipality support 0% 3 54 There were a number of factors important to decisions to return but outside FFS’s mandate or manageable interest. In descending order of importance these include security, the presence of family, a desire to return to their home community, and property ownership. Although Table 18 indicates a strong alignment between FFS activities and factors influencing return, Figure 20 suggests that other factors may have been more influential in driving returns, as number of returns decreased as FFS implementation in the intervention area ramped up. This may not be surprising given that 98 percent of surveyed returnees who returned in 2016 and 2017 reported security as a factor influencing return. Ninewa Plains Table 19 compares the percentage of selected surveyed returnees who report a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (76 percent) returned to Ninewa. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Ninewa cited seven factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) electricity, water and sanitation services, health services, education, rehabilitation of residences, employment and livelihoods opportunities, and availability of markets. There is one factor, highlighted in orange in the table, within FFS’s mandate that FFS did not address in Ninewa prior to 2018—availability of business grants (cited by 26 percent of returnees in 2017 as a consideration in their decision to return). FFS supported 25 municipality projects that did not relate to any factors influencing decisions to return (highlighted in blue). Table 19: Comparison of factors influencing return in Ninewa as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 82% N/A Access to electricity 53% 11 Access to water and sanitation services 52% 11 Access to health services 46% 36 Access to education 43% 66 Rehabilitation of residences 43% 2 Property ownership 34% N/A Availability of business grants 26% 0 Employment and livelihoods opportunities 24% 2 Availability of markets 20% 2 Figure 20: Number of FFS projects started and number of surveyed returnees, by year, for Karma 0 10 20 30 40 50 60 70 80 2015 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q2 2015 55 Presence of family 17% N/A Home community 17% N/A Humanitarian support 10% N/A Municipality support 0% 25 There were a number of factors important to decisions to return but outside FFS’s mandate or manageable interest. In descending order of importance these include security, property ownership, the presence of family, a desire to return to their home community, and humanitarian support. Figure 21 shows that the launch of the 239 FFS projects started between October 2016 and June 2018 coincided with the number of surveyed respondents who reported returning between 2016 and 2018. Both the highest number of projects started (149) and highest number of returns to home communities (2017) in Ninewa happened in 2017, followed by 84 FFS projects started in 2018 and 23 returnees that year. This data, coupled with the crosswalk of factors of return and number of FFS projects started in 2017, as shown in Table 19, suggests a correlation between FFS activities and decisions to return in Ninewa, with the caveat that decisions to return were also, as reported by returnees, largely driven by security considerations, which would have been impacted by the military defeat of ISIS at the end of 2016. Ramadi Table 20 compares the percentage of selected surveyed returnees who report a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2016 and 2017, the years the majority of surveyed returnees (76 percent) returned to Ramadi. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Ramadi cited five factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) education, health services, rehabilitation of residences, electricity, and water and sanitation services. Of note, however, only between 17 percent and 27 percent of 2017 returnees cited these factors as influencing their decisions to return. There were no factors cited by at least 10 percent of returnees which FFS did not address. However, FFS supported 24 projects related to factors cited by less than 5 percent of 2017 returnees as influencing decisions to return, suggesting a low return on investment of these projects (highlighted in blue). Figure 21: Number of FFS projects started and number of surveyed returnees, by year, for Ninewa 0 20 40 60 80 100 120 140 160 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q4 2016 56 Table 20: Comparison of factors influencing return in Ramadi as reported by surveyed returnees who returned in 2016 and 2017 (the highest years of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 99% N/A Access to education 27% 54 Access to health services 26% 29 Rehabilitation of residences 20% 10 Access to electricity 19% 12 Access to water and sanitation services 17% 35 Availability of business grants 4% 2 Employment and livelihoods opportunities 3% 10 Police stations 3% 3 Roads and bridges 1% 3 Municipality support 0% 6 Security was the only factor reported by 2017 returnees as influencing decisions to return but outside FFS’s mandate or manageable interest. However, this was cited by 99 percent of returnees. Figure 22 shows that the launch of the 178 FFS projects started between January 2016 and May 2018 coincided with the number of surveyed respondents who reported returning between 2016 and 2018. 2016 and 2017 saw the highest numbers of returns (188 each year) as well as the largest volume of projects, 70 and 94, respectively. This data would suggest a correlation between FFS activities and decisions to return in Ramadi; however, this is undermined by the low percentages of respondents citing factors supported by FFS as influencing their decisions to return coupled by the fact that 99 percent of returnees reported security as influencing their decisions to return, which would have been impacted by the military defeat of ISIS in 2015. Figure 22: Number of FFS projects started and number of surveyed returnees, by year, for Ramadi 0 50 100 150 200 2015 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: 2015 57 Shirqat Table 21 compares the percentage of selected surveyed returnees who reported that a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (58 percent) returned to Shirqat. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in Shirqat cited four factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) electricity, education, health services, and police stations. There were three factors cited by at least 10 percent of returnees that were within FFS’s mandate but which FFS did not address. In descending order of importance to returnees, these are availability of markets (cited by 24 percent of returnees in 2017 as a consideration in their decision to return), availability of business grants (22 percent), and rehabilitation of residences (16 percent). FFS supported six projects related to factors cited by 8 percent or less of 2017 returnees as drivers of return, suggesting a low return on investment of these projects (highlighted in blue). Table 21: Comparison of factors influencing return in Shirqat as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 91% N/A Access to electricity 51% 2 Access to education 41% 7 Access to health services 41% 8 Police stations 26% 1 Availability of markets 24% 0 Availability of business grants 22% 0 Rehabilitation of residences 16% 0 Access to water and sanitation services 10% 2 Employment and livelihoods opportunities 8% 1 Municipality support 0% 5 Security was the only factor reported by 2017 returnees as influencing decisions to return but outside FFS’s mandate or manageable interest. However, this was cited by 91 percent of returnees. Figure 23 shows that the launch of the 27 FFS projects started between April 2017 and April 2018 coincided with the number of surveyed respondents who reported returning between 2016 and 2018. 2017 saw the highest numbers of returns (87) as well as the largest volume of projects (26). This data, coupled with the crosswalk of factors of return and number of FFS projects started in 2017, as shown in Table 21, suggests a correlation between FFS activities and decisions to return in Shirqat, with the caveat that decisions to return were also, as reported by returnees, largely driven by security considerations, which would have been impacted by the military defeat of ISIS in mid-2016. 58 Tikrit Table 22 compares the percentage of selected surveyed returnees who report a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2015 and 2016, the years the majority of surveyed returnees (77 percent) returned to Tikrit. Therefore, the number of projects reflects those ongoing or completed as of December 2016. Returnees in Tikrit cited five factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow above) health services, electricity, education, water and sanitation, and police stations. There were two factors cited by at least 10 percent of returnees that were within FFS’s mandate that FFS did not address. In descending order of importance to returnees, these are rehabilitation of residences (cited by 24 percent of returnees in 2016 and 2017 as a consideration in their decision to return), and availability of markets (14 percent). FFS supported two projects cited by only 1 percent of returnees as drivers of return (highlighted in blue), suggesting a low return on investment for these projects. Figure 23: Number of FFS projects started and number of surveyed returnees, by year, for Shirqat 0 20 40 60 80 100 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q3 2016 59 Table 22: Comparison of factors influencing return in Tikrit as reported by surveyed returnees who returned in 2015 and 2016 (the highest years of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 94% N/A Access to health services 46% 9 Access to electricity 42% 3 Access to education 42% 8 Access to water and sanitation services 31% 5 Police stations 28% 1 Rehabilitation of residences 24% 0 Availability of markets 14% 0 Presence of family 11% N/A Availability of business grants 0% 1 Employment and livelihoods opportunities 0% 1 There were two drivers of return for at least 10 percent of returnees in 2016 and 2017 but outside FFS’s mandate or manageable interest: security (cited by 94 percent of returnees) and the presence of family in their home communities (11 percent). Although Table 22 indicates a correlation between FFS activities and factors influencing return, Figure 24 suggests that other factors may have been more influential in driving returns, as number of returns decreased as FFS implementation in the intervention area ramped up, albeit by a small amount each year. This may not be surprising given that 94 percent of surveyed returnees who returned in 2015 and 2016 reported security as a factor influencing return. West Mosul Table 23 compares the percentage of selected surveyed returnees who reported that a given factor informed their decision to return with the number of projects implemented by FFS related to that factor. Responses reflect those who surveyed household representatives who returned in 2017, the year the majority of surveyed returnees (51 percent) returned to West Mosul. Therefore, the number of projects reflects those ongoing or completed as of December 2017. Returnees in West Mosul cited five factors of return that FFS addressed during the period of their return. In descending order of importance, they are (as highlighted in yellow below) health services, electricity, education, water and sanitation, and livelihoods opportunities. There were three factors cited by at least 10 percent of returnees that were within FFS’s mandate but which FFS did not address. In descending order of importance to returnees, these are rehabilitation of residences (cited by 23 percent of returnees in 2017 as a consideration in their decision to return), availability of business grants (13 Figure 24: Number of FFS projects started and number of surveyed returnees, by year, for Tikrit 0 20 40 60 80 100 120 140 160 180 2015 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q2 2015 60 percent), and availability of markets (10 percent). FFS supported two roads and bridges projects, a factor not cited by any returnees as a driver of return (highlighted in blue), suggesting a low return on investment for these projects. Table 23: Comparison of factors influencing return in West Mosul as reported by surveyed returnees who returned in 2017 (the highest year of returns) and number of FFS projects started that same year relating to those factors. Factor for Return Percent of Returnees Reporting Factor as Consideration for Return No. of Projects Security 79% N/A Property ownership 36% N/A Access to health services 36% 8 Access to electricity 34% 2 Access to education 34% 21 Access to water and sanitation services 34% 11 Rehabilitation of residences 23% 0 Presence of family 17% N/A Employment and livelihoods opportunities 15% 5 Availability of business grants 13% 0 Availability of markets 10% 0 Roads and bridges 0% 2 There were three drivers of return for at least 10 percent of returnees in 2017 outside FFS’s mandate: security (cited by 79 percent of returnees), property ownership (36 percent), and the presence of family in their home communities (17 percent). Although Table 23 indicates a correlation between FFS activities and factors influencing return, Figure 25 suggests that other factors may have been more influential in driving returns in West Mosul, as number of returns decreased as the volume of FFS projects in the intervention area increased. This is further supported by the fact that other drivers of return were cited by approximately only a third of returnees. This may not be surprising given that 79 percent of surveyed returnees who returned in 2017 reported security as a factor influencing return. Figure 25: Number of FFS projects started and number of surveyed returnees, by year, for West Mosul 0 20 40 60 80 100 120 140 2016 2017 2018 # of FFS projects and # of surveyed returnees # of FFS projects by start date # of surveyed returnees, by year of return Military defeat of ISIS: Q4 2016 61 FINDINGS 3B. IN WHAT WAYS HAVE FFS ACTIVITIES AFFECTED IDPS’ LIVES SINCE THEIR RETURN TO THEIR COMMUNITIES? Unless otherwise noted, all cited data for this evaluation sub-question refer to findings from the population-based survey (PBS) or Livelihoods Survey (LS). As noted in the Question 2 findings, FFS interventions have provided an increase in service provisions to pre-ISIS levels, but this does not provide the full story of interventions and its effect on the lives of returnees. The data outlined below from the LS illustrate some of the successes, and cautions, of livelihood interventions on income though C4W programming. Furthermore, life is improving for returnees. The C4W activities were successful, unsurprisingly, in improving the income of participants. Nearly 50 percent of respondents (Figure 25) noted that their lives, and their income, have improved since the C4W activities. Despite these successes, there is clear indication that not all ethno-religious groups in Iraq benefited the same from this specific intervention. As seen in the data, while all groups benefitted from C4W activities, Shia, Sunnis, and Yazidis earned less than Christians and those who identified as “Other” in their C4W payments, while Shabaks earned more than other ethnic groups. Whether this is due to discrimination or other factors is unclear in the data. The data suggest that life is improving for people who returned (Figure 25)—about a third (34 percent) reported that they feel life has mostly improved, 17 percent reported that life has completely improved, with (32 percent) reporting not very much improved, and 16 percent reporting life has not improved at all. These findings are expected following recovery from the loss of income and property and distortion of livelihoods during the crisis. It should be noted that more than half of the returnees (51 percent) are reporting improved living conditions, implying that the returnees have been set on a good pathway demonstrating gains toward stable livelihoods. Cash for Work and Household Income 0% 5% 10% 15% 20% 25% 30% 35% 40% Not at all Not very much Mostly improved Completely improved Not Sure Figure 26: LS responses to “Is overall life improving for people who returned?” 62 Cash for Work activities have had some success in improving lives. For example, a majority of C4W respondents said that the income they received was adequate to support their family’s needs (Figure 26), while a further majority of participants said it was their largest source of income. Most C4W beneficiaries (Figure 31) noted that their income levels improved or were static after the work, which led many to lament the ending of the C4W activities, as noted through the KIIs and FGDs. Furthermore, a near majority (Figure 31) of beneficiaries noted that life improved since their participation. Using measures of central tendency (mean) the Team looked at the demographics of C4W beneficiaries. Women, Christians, and Shabaks, (Figures 29–35) by a wide margin, earned the most from the C4W activities. For example, Christians earned a mean of IQD 2,250,000 over the full duration of their participation, while women earned nearly IQD 1,400,000. By ethnic grouping, Shabaks earned just over IQD 2,000,000. Comparison of monthly household incomes indicates that overall household income levels have declined after returning from displacement. About 63 percent of the returnees had an average monthly income of over IQD 300,000, as compared to 68 percent before displacement, according to respondents. Most returnees show their incomes declining into the lower income groups (i.e., IQD150,000 – 300,000 monthly), which increased from 15 percent before returning to 19 percent and IQD160,000–150,000, which increased from 9 percent to 12 percent. The proportion of the lowest income group of less than IQD 60,000 monthly increased from 8 percent to 9 percent. The findings imply that the mean household income has significantly dropped compared to before displacement at 1 percent significant level (p<0.0115). The main sources of household income among the returnees are salaried employment and small business enterprises (40 percent), casual labor (34 percent), and daily wage earners and retirement pensions (6 percent each). 0% 10% 20% 30% 40% 50% 60% Yes No Not sure Figure 27: Percentage of LS responses to “Was the money you received from C4W adequate for your family’s needs?” 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Improved Same as before Worse Figure 28: Percentage of LS responses to “Overall has life improved since your participation in the C4W activity?” “Many people returned for the Cash for Work program but [this lasted] for a month or two only so the people felt disappointed afterwards.” – Local Government Employee KII in Ninewa 63 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Improved Same as before Worse Figure 29: LS responses to: “How would you compare your household income to the period before the C4W support?” IQD - IQD 200,000 IQD 400,000 IQD 600,000 IQD 800,000 IQD 1,000,000 IQD 1,200,000 IQD 1,400,000 IQD 1,600,000 IQD 1,800,000 IQD 2,000,000 Anbar Salah al Din Ninewa Anbar Salah al Din Ninewa Male Female Figure 31: Mean amount of money received from the C4W activity for the period enrolled by sex and province. IQD - IQD 200,000.00 IQD 400,000.00 IQD 600,000.00 IQD 800,000.00 IQD 1,000,000.00 IQD 1,200,000.00 IQD 1,400,000.00 Male Female Figure 30: Mean amount of money received from the C4W for the period enrolled by sex 64 IQD - IQD 500,000 IQD 1,000,000 IQD 1,500,000 IQD 2,000,000 IQD 2,500,000 Arab Assyrian Kurd Turkmen Shabak Others: Arab Assyrian Kurd Turkmen Shabak Others: Arab Assyrian Kurd Turkmen Shabak Others: Anbar Salah al Din Ninewa Figure 33: Mean amount of money received from the C4W activity for the period enrolled by ethnic origin and province. IQD - IQD 500,000.00 IQD 1,000,000.00 IQD 1,500,000.00 IQD 2,000,000.00 IQD 2,500,000.00 Arab Kurd Turkmen Shabak Others: Figure 32: Mean amount of money received from the C4W activity for the period enrolled by ethnic origin. 65 As Figure 32 above indicates, there were no female beneficiaries in Salah al-Din in the sample. However, females did earn more than males, on average, in Ninewa. Arabs were the only beneficiaries found in the sample for Anbar and Salah al-Din, but on average did not make as much as those in Ninewa, while Assyrians were absent for all three provinces. While Shabaks were only sampled in Ninewa, they were the highest earners across all ethnic groups. Similarly to the Iraqi-Arab demographic, only Sunnis were found in this sample, but those in Salah al-Din were the lowest earners across all three provinces and across all the religious groups. Christians in Ninewa were by far the highest earners. IQD - IQD 500,000 IQD 1,000,000 IQD 1,500,000 IQD 2,000,000 IQD 2,500,000 Muslim-Shia Muslim-Sunni Christian Yazidi Kakai Others: Muslim-Shia Muslim-Sunni Christian Yazidi Kakai Others: Muslim-Shia Muslim-Sunni Christian Yazidi Kakai Others: Anbar Salah al Din Ninewa Figure 35: Mean amount of money received from the C4W activity for the period enrolled by religion and province IQD - IQD 500,000.00 IQD 1,000,000.00 IQD 1,500,000.00 IQD 2,000,000.00 IQD 2,500,000.00 Muslim-Shia Muslim-Sunni Christian Yazidi Others: Figure 34: Mean amount of money received from the C4W activity for the period enrolled by religion. 66 CONCLUSIONS According to PBS data, through its efforts to restore access to core services, FFS addresses factors that influence decisions to return or stay (for those who never left) for up to 86 percent of the population, although not all services are equally important and none were cited by a majority of respondents as critical to decisions to return across the full population. Figure 14 shows that for roughly a third of the population, restoration of electricity, water and sanitation, health, and education services are critical factors in decisions to stay or return, and a consideration—though not a critical one—for an additional 30 to 50 percent of the population, depending on the sector. These numbers are even higher when looking at factors influencing return by intervention area only among returnees (excluding those who never left or current IDPs) in the year(s) the majority of those surveyed returned to their homes. These data, detailed in the following section, present further evidence that FFS interventions respond to key concerns of IDPs vis-à-vis their decisions to return to their homes at the time they are making those decisions. However the extent varies by intervention area and intervention type and does not hold true for one intervention area (Mekeshifa). Furthermore, life is improving for returnees based upon data from the livelihoods survey. The C4W activities were successful in improving the income of participants. Nearly 50 percent of respondents noted that their lives (Figure 26) and their income (Figure 27) had improved since the C4W activities. Despite these successes, there is clear indication that not all ethno-religious groups in Iraq benefited the same from this specific intervention. Christians, women, and Shabaks earned significantly more from C4W than other groups (Figures 31-35). RECOMMENDATIONS 3.1 If all interventions are meant to encourage returns, as documented in the FFS development hypothesis, then based on factors driving return as reported by returnee respondents to the population-based survey and cross-referenced with the timing of returns, also as reported by returnee respondents to the population-based survey, and FFS interventions in each intervention area, USAID and UNDP should reconsider projects related to municipality support and roads and bridges in all intervention areas, and livelihoods support in certain intervention areas. However, if these interventions are meant to support other objectives important to FFS goals—for example, if the cash infusion into villages from livelihoods activities is an objective in itself and is not linked to encouraging returns—the Steering Committee should revisit the FFS development hypothesis to clarify these objectives. 3.2 FFS should consider the impacts of C4W activities on each community, noting the differences in the intervention on each religious and ethnic group. A perception of favoritism, whether implicit or explicit, will likely lead to communal and national tension. 0% 20% 40% 60% 80% 100% Electricity Water and Sanitation Health Education Critical factor Consideration Figure 36: Critical factors and other considerations in decisions to return or stay across full population 67 QUESTION 4: WHAT FACTORS ARE CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES TARGETED BY FFS? FINDINGS 4A. SOURCES AND AVAILABILITY OF OPERATIONS AND MAINTENANCE BUDGETS FOR REHABILITATED SERVICES Ministries allocate operations and maintenance budgets (O&M) for the directorates, as confirmed in key informant interviews with local authorities. The Ministries of Health, Education, and Electricity establish and allocate O&M budgets for health, education, and electricity directorates, respectively. The Ministry of Housing and Construction, Municipalities, and Public Works establishes and allocates O&M budgets for water, sewer, and municipality directorates. Interviewees reported the allocated budgets were barely sufficient for the paying staff salaries and were not sufficient for the O&M needs of the directorates. An analysis of the Ministry of Finance data61 revealed that a significant portion of the overall budget was spent on employee salaries for three ministries, as high as 95 percent for education. In contrast to this, the proportion of expenditure on asset maintenance was reported to be as low as 0.2 percent for municipality and education sectors. However, the Evaluation Team could not confirm whether salaries included resources for operations and maintenance. Table 24 summarizes the 2018 budget for the relevant ministries and the percent of expenditure on salaries, asset maintenance, and capital for each.62 Table 24: Ministry Budgets and Expenditures, 2018 Ministry Total Expenditures (Iraqi Dinar) % Salaries % Asset Maintenance % Capital Electricity 3,462,192,440,450 81 0.1 0 Health 1,317,937,457,987 69 1.3 0.5 Education 1,783,995,566,495 95 0.2 0.1 Housing and Construction, Municipalities, and Public Works 760,389,703,710 17 0.2 0.1 Using the rapid capacity assessment tool, the Evaluation Team assessed the extent to which directorates’ budgets are realistic, clear, and well documented without external assistance; the existence and source of an O&M budget and its sufficiency in terms of meeting O&M requirements; and the sufficiency of locally generated revenue to meet O&M costs. Results are shown in Figure 37, with scores ranging from one (low capacity) to four (strong). Budget capacity is highest in the Anbar Municipal Services Directorate (3.7), and lowest in the Sewage and Electricity Directorates in Salah al-Din, the Electricity Directorate in Anbar, and the Municipal Services Directorate in Ninewa (1.0 for each). By sector, we observe the highest scores across provinces in the water sector (2.6) and lowest scores 61 The source for this information can be found at http://www.mof.gov.iq/obs/en/Pages/WDMGChart.aspx 62 Other categories of expenditures include Commitments and Foreign Aid, Grants and Subsidies, and Debt Service, Intermediate Goods, Investment Projects, Social Welfare, and Supplies Service. 68 among sewage and electricity directorates (1.4); by province scores across all sectors were slightly higher in Ninewa (2.2) than Anbar and Salah al-Din (2.0 each). 4B. CAPACITY TO COLLECT REVENUE The Evaluation Team assessed the capacity of the six departments in each of the three provinces to collect revenue, specifically assessing whether there is a system in place to collect revenue as well as systems to enforce payment, and whether collected revenue meets the department’s O&M needs. Results are shown in Figure 37. Revenue collection capacity is highest in the Ninewa Municipal Services Directorate (3.7) and lowest in the Anbar Sewage Directorate (1.0). By sector, we observed the highest score across provinces in municipal services (2.9) and lowest score among sewage and education directorates (1.4); by province scores across all sectors were higher in Ninewa and Salah al-Din (2.1) than Anbar (1.7). 4C. ORGANIZATIONAL CAPACITY OF LOCAL ADMINISTRATIVE OFFICES The Evaluation Team assessed the capacity of local administrative offices related to procurement, planning, human resource management, monitoring and evaluation, and information technology. Procurement capacity was assessed based on the existence of and adherence to procurement policies and guidelines, as well as their monitoring for compliance. Results are shown in Figure 38. Procurement capacity is highest in the Sewage, Water, and Electricity Directorates in Ninewa and the Education and Health Directorates in Anbar (4.0), and lowest in the Salah al￾Din Health Directorate (1.3). By sector, we observed the highest score across provinces in water (3.4) and lowest in health (2.4); by province scores across all sectors were higher in Ninewa and Anbar (3.3) than Salah al-Din (2.3). Strategic planning capacity was assessed based on the existence of a strategic plan with clear priorities, measurable objectives, and targets, which is regularly reviewed and/or revised, and consistently used in guiding management decisions and operational planning. Results are shown in Figure 39. Strategic planning capacity is highest in the Education, Electricity, Health, and Municipal Services Directorates in Anbar (4.0), and lowest in the Health Directorate in Ninewa (2.0). By sector, we observed the highest score across provinces in electricity and education (3.3) and lowest in health (2.8); by province scores across all sectors were highest in Anbar (3.8) followed by Salah al-Din (2.9) and Ninewa (2.5). 0.0 1.0 2.0 3.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 37: Revenue Collection Capacity by Directorate and Province 0.0 2.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 38: Procurement Capacity by Directorate and Province 0.0 1.0 2.0 3.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 39: Strategic Planning Capacity by Directorate and Province 69 Work planning capacity was assessed based on how the local authority prepared its annual work plans, who developed them, alignment of the timing of their development with government planning cycles, whether they are linked to the local authority’s budget, and how they are used for management decisions, operational planning, and monitoring and evaluation. Results are shown in Figure 40. Work planning capacity is highest in the Health Directorate in Anbar (3.8), and lowest in the water and municipality directorates in Salah al-Din, the municipality directorate in Ninewa, and the electricity directorate in Anbar (2.2 each). By sector, the scores are highest across provinces in water (2.9) and lowest in electricity (2.5); by province, scores across all sectors were highest in Anbar (3.2) followed by Salah al-Din and Ninewa (each at 2.7). Human resources management capacity was assessed based on the existence of and adherence to policies, procedures, and clear job descriptions for staff; full staffing; open and transparent recruitment and hiring processes; and the staffing of key management and administrative positions, as well as other positions, with qualified personnel. Results are shown in Figure 41. Human resources management capacity is highest in the Education and Municipal Services Directorates in Anbar and the Water and Municipal Services Directorates in Salah al-Din (3.4), and lowest in the Sewage and Health Directorates in Ninewa (2.0). By sector, we observed the highest score across provinces in municipal services (3.3) and lowest in health (2.6); by province scores across all sectors were highest in Anbar (3.2), followed by Salah al-Din (3.1) and Ninewa (2.5). M&E capacity was assessed based on the existence of and adherence to policies and procedures that allow the organization to have a good sense of how it is doing and what is accomplished; whether the organization has indicators and realistic targets; whether the staff have sufficient expertise in data collection and monitoring; and the reliability, timeliness, and use of monitoring data. Results are shown in Figure 42. M&E capacity is highest in the Electricity Directorate in Ninewa (3.7), and lowest in the Education Directorate in Salah al-Din (1.5). By sector, we observed the highest score across provinces in electricity and health (3.1) and lowest in education (2.2); by province scores across all sectors were higher in Ninewa and Salah al-Din (2.6) than Anbar (2.5). 0.0 1.0 2.0 3.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 40: Work Planning Capacity by Directorate and Province 0.0 1.0 2.0 3.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 41: HR Management Capacity by Directorate and Province 0.0 2.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 42: M&E Capacity by Directorate and Province 70 IT capacity was assessed based on the existence of and adherence to communications and IT policies and procedures; whether IT systems are adequate vis-à-vis staff size, networked, and backed up; risks to the IT system; and the skills of IT staff and users. Results are shown in Figure 43. IT capacity is highest in the Health Directorate in Salah al-Din (3.3), and lowest in the Water Directorate in Anbar (1.3). By sector, we observed the highest score across provinces in electricity (2.7) and lowest in water (1.9); by province scores across all sectors were highest in Ninewa (2.8), followed by Salah al-Din (2.4) and Anbar (1.8). 4D. TECHNICAL AND MANAGEMENT CAPACITY OF STAFF Figure 44 shows technical capacity across provinces. On average, technical capacity is highest across provinces in electricity (3.4), including the overall highest score (3.7) in Salah al-Din, and lowest in education (2.1), including the overall lowest score (1.5) in Salah al-Din. There was less variation across provinces than within them, with technical capacity across all sectors highest in Anbar (2.9) followed by Salah al-Din (2.8) and Ninewa (2.6). By contrast, on average across provinces, management capacity is lowest in the electricity sector (2.4) and highest in education (2.7), although there is much less variation across sectors and provinces for management capacity as compared to technical capacity. A comparison with citizen perceptions of service delivery (see Figure 5 in the Question 1 findings), reveals both convergence and divergence with local authorities’ self-assessments. ● Management capacity is highest in education and, on average across provinces, respondents report highest satisfaction with service delivery in the education sector vis-à-vis electricity, water, sanitation, and health. ● Citizens overwhelmingly report lowest satisfaction with services in the sanitation sector, whereas local authorities in that sector assess their management and technical capacity to be at or only slightly below those of their counterparts in other sectors. ● On average across sectors, citizens were least satisfied with service delivery in Salah al-Din; however, local authorities in that province generally assessed themselves on par with their peers in Anbar and Ninewa. CONCLUSIONS Various factors contribute to or impede local authorities’ capacity to administer services targeted by FFS, depending on the province and sector. We discuss these variations in detail in the following section, but a summary of factors contributing to or impeding local authorities’ capacity to administer services is listed below by province based on rapid capacity assessment scores by province, sector, and function in 0.0 2.0 4.0 Sewer Water Education Electricity Health Municipal Services Ninewa Anbar Salah-al-Din Figure 43: IT Capacity by Directorate and Province 0.0 1.0 2.0 3.0 4.0 Electricity Water Municipal Services Sewage Health Education Ninewa Anbar Salah-al-Din Figure 44: Technical Capacity across Provinces 71 which a score of one indicates low capacity; two, basic capacity; three, moderate capacity; and four, strong capacity. Unless otherwise noted, all findings are supported by the rapid capacity assessment the Evaluation Team conducted with provincial directorates of Water, Sewer Services, Municipality, Health, Education and Electricity in Anbar, Ninewa, and Salah al-Din. Participants were selected from core governance functions of strategic planning, budgeting, procurement, monitoring and evaluation, information, communication technology, and human resource management, as well as technical departments responsible for operations and maintenance. The representatives of each directorate discussed each of the functions addressed in the capacity assessment tool and reached a consensus on scores on a scale63 of 1 to 4, with 4 representing the highest capacity. The exercise assessed the directorates’ capacity in core functions, relying on self-assessment of directorate staff. In Anbar, factors impeding local authorities’ capacity to administer services include the following: • Budgeting64 in the sewage, electricity, and health directorates • Revenue collection65 in the sewage, water, education, and electricity directorates • IT and communications in the sewage, water, and health directorates Factors contributing to local authorities’ capacity to administer services include the following: • Procurement66 in the education and health directorates Strategic planning67 in the education, electricity, and health directorates and the municipality Table 25: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Anbar. Sewage Water Educ. Elec. Health Muni. Budget 1.3 2.3 2.3 1.0 1.3 3.7 Revenue Collection 1.0 1.7 1.3 1.3 2.7 2.3 Procurement 2.7 3.0 4.0 2.3 4.0 3.7 Strategic Planning 3.3 3.8 4.0 4.0 4.0 4.0 Work Planning 2.6 3.4 3.4 2.2 3.8 3.6 HR Management 3.2 3.0 3.4 3.0 3.0 3.4 M&E 2.2 2.2 2.0 2.5 3.2 2.8 IT & Comm. 1.4 1.3 2.0 2.1 1.7 2.1 Technical Capacity 2.6 2.1 2.3 3.2 3.4 3.6 63 1 = Low, 2 = Basic, 3 = Moderate, 4 = Strong 64 This includes the existence of a realistic, clear, and well documented directorate budget developed without external assistance, the existence of a realistic budget for operations and maintenance (O&M), and whether locally generated revenue is sufficient to cover O&M costs. 65 This includes the existence of a system to collect revenue and enforce penalties for nonpayment. 66 This includes the existence of, adherence to, and monitoring of a procurement policy. 67 This includes the existence, regular revision, and use of a strategic plan for the entity. 72 In Salah al-Din, factors impeding local authorities’ capacity to administer services include the following: • Budgeting in the sewage and electricity directorates, and the municipality • Revenue collection in the sewage and education directorates • M&E in the education directorate • IT and communications in the municipality • Technical capacity in the education directorate68 As shown in Table 26, the evaluation did not identify any factors strongly contributing to local authorities’ capacity to administer services in Salah al-Din. Table 26: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Salah al-Din. Sewage Water Educ. Elec. Health Muni. Budget 1.0 2.3 3.0 1.0 3.0 1.7 Revenue Collection 1.7 2.0 1.7 2.3 2.3 2.7 Procurement 2.7 3.3 2.3 2.0 1.3 2.0 Strategic Planning 3.8 2.8 2.5 3.0 2.3 3.0 Work Planning 2.8 2.2 3.2 2.6 3.0 2.2 HR Management 3.0 3.4 3.2 2.6 2.8 3.4 M&E 2.7 2.5 1.5 3.2 3.5 2.5 IT & Comm. 2.0 2.1 2.6 2.7 3.3 1.9 Technical Capacity 2.9 3.1 1.5 3.7 3.0 2.7 In Ninewa, factors impeding local authorities’ capacity to administer services include the following: • Budgeting in the education directorate and municipality • Revenue collection in the sewage, water, education, and health directorates • M&E in the municipality As shown in Table 27, factors contributing to local authorities’ capacity to administer services relate to procurement in the sewage, water, and electricity directorates. 68 This includes the existence of and adherence to an approved education sector plan and guidelines that guide quality delivery and monitoring standards for education in line the with the national education policy; the existence of established community- and district-based structures, and processes that facilitate effective education delivery in the province; and documented improvements over time in the number of classrooms and number of schools with access to clean water sources and sanitation facilities. 73 Table 27: Rapid capacity assessment scores identifying factors contributing to (in green) or impeding (in red) capacity to administer services in Ninewa. Sewage Water Educ. Elec. Health Muni. Budget 2.0 3.0 1.7 2.3 3.0 1.0 Revenue Collection 1.7 1.7 1.3 2.7 1.7 3.7 Procurement 4.0 4.0 2.3 4.0 2.0 3.3 Strategic Planning 2.5 2.3 3.5 2.8 2.0 2.3 Work Planning 3.0 3.0 2.6 2.6 2.6 2.2 HR Management 2.0 2.6 2.6 2.6 2.0 3.2 M&E 2.0 2.7 3.0 3.7 2.5 1.7 IT & Comm. 2.4 2.3 3.0 3.1 2.6 3.1 Technical Capacity 2.3 2.9 2.5 3.4 2.6 2.1 RECOMMENDATIONS 4.1 Although outside the scope of FFS, improving provincial authorities’ capacity for revenue collection and budgeting may need special attention from the donor community and the Government of Iraq. The Evaluation Team suggests the lack of capacity in this domain be considered in the design of future relevant programming. 4.2 Donors should consider using these findings to tailor support within each province by sector and function (technical vs. management and sub-functions within management). 74 ANNEX 1: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS MATRIX EQ1: How have FFS activities affected access to services in target communities? 1a. Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including, but not limited to, health, education, water, electricity, and other municipal services). 1b. Whether access to services is equitable across vulnerable groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, politically affiliated groups, and youth and children. 1c. What significant barriers, if any, prevent members of target communities from accessing services? 75 Conclusion Findings Recommendations Activities implemented by FFS positively affected access to services in all targeted communities. Respondents from the PBS reported that water, electricity, education, and health services in FFS intervention areas surveyed by PMSP in Qayara, East Mosul, West Mosul, Ninewa Plains, Ramadi, Falluja, Karma, Tikrit, Mekeshifa, and Shirqat were restored to pre￾ISIS levels. However, respondents currently residing in the areas sampled perceived themselves to have limited access to sanitation services in relation to other services. The majority of respondents that stated sanitation systems had been destroyed during ISIS occupation in Karma, Mekeshifa, and Shirqat, reported that they perceived that those systems were not yet restored in their communities. • The majority of vulnerable populations, including those living with disabled family members, and IDPs reported that they had access to water and electricity, while sanitation was not as accessible by the majority of Sunni Muslims and Arabs. • Access to education was also equal across populations; almost 100% of respondents, including vulnerable groups reported that their schoolaged children were attending school. Similarly, 100% of respondents stated that they had a government health center accessible to them within walking distance (1-6 km). • Very few respondents reported barriers to accessing services and those that did, reported reasons that were outside of the scope of the FFS intervention, including: Electricity: Cost of the connection fee and/or monthly subscription, living in dwellings that did not meet standards for connections, and delayed applications. Water: damaged or vandalized infrastructure; a dried water source, poor system design, and lack of maintenance at the community access point. Sanitation: lack of permission to be connected to the sewer system (reasons were not provided), no regulation mandating connections, connection costs, lack of awareness of the public system, and poor operation and maintenance of the sewage system. Health: the health center did not offer the service sought; the facility was closed; long wait times; and the cost was prohibitive. Education: distance to the school, the school was in a restricted zone, insecurity and personal safety issues, the cost was prohibitive, poor quality of the schools, the child’s illness or disability, and school teachings conflicted with their beliefs. According to PBS respondents in the ten surveyed FFS intervention areas, 97% of returnees across all surveyed areas have access to schools for their children. Given this, supporting education infrastructure (not quality, content, or capacity) may be reaching a state of diminishing returns for the purposes of stabilization in the ten surveyed intervention areas, although not necessarily for longer-term transition or development. USAID and FFS may wish to consider reviewing its programming to focus activities and resources a) tactically on all sectors in more newly liberated areas; or b) tactically on other less entrenched sectors for returnees in selected areas where there are incongruous signs of output and outcome achievement. Additionally, the donor community or GOI may want to consider a strategic shift from a stabilization to development approach in the education sector in the ten surveyed intervention areas, including for example increasing community management capacity functioning and local governance capacity functioning. 76 EQ2: How have FFS activities affected the quality of services in target communities? 2a. Sector-specific parameters of quality of services targeted by FFS (including, but not limited to, health, education, water, electricity, and other municipal services). 2b. Perceptions of the quality of services among members of target communities. 2c. Perceptions of the effectiveness of government administration of services among members of target communities. Conclusion Findings Recommendations FFS activities restored the quality of electricity, water, and sanitation services to pre-2015 levels. The Evaluation Team visited electricity and water/sanitation service sites in areas sampled in the PBS to determine if services met sector-specific parameters of quality and found that at all visited power stations in Ninewa, Anbar, and Salah al Din electrical substations and generators were functioning properly and no substantial technical issues were identified. The Evaluation Team also found that water and sanitation standards are being met. • Electrical substations and generators visited were functioning properly, except for one substation where a circuit breaker was out of order. All eight substations and the three generators operated at the required capacity. • The majority of substations and the two generators visited provided electricity from 12 to 18 hours and only two were providing electricity for 24 hours. • Required tests (pH, color test, turbidity (NTU), residual chlorine Cl, hardness in terms of CaCO3, fecal coliform, total coliform count, and heavy metals) were either conducted daily or weekly. During the days of visits the water quality test results were all positive and of acceptable range. • PBS respondents perceived that the quality of electricity and water services returned to pre-2015 levels. • Connectivity to electricity exceeded pre-2015 levels according to PBS data. • The number of respondents reporting that they received electricity 24 hours a day in 2019 increased by 4% compared to 2015. • 60% of respondents responded positively vis-à-vis water taste; 64%, smell; and 58%, clarity. These are slight improvements from perceptions prior to 2015. • 72% of respondents experienced no or infrequent problems with sanitation services, while only 26 percent reported frequent breakdown of the sewer. Based on respondents’ low satisfaction with local administration of sanitation services, FFS should consider working with local authorities to build capacity and improve administration of sanitation services (Window 3) to ensure improved service provision to beneficiaries and improve local governance through service management. If this does not fall within FFS’s scope of work under Window 3, the Evaluation Team recommends that USAID or other donors consider this in the design of future programming. 77 Related to the quality of services was the administration of those services. Respondents felt mixed about local authorities’ administration of services across implementation areas. When asked via the PBS survey their level of satisfaction with local authorities’ administration of various services, respondents’ replies ranged from “somewhat satisfied” to “very dissatisfied”. Respondents particularly disapproved of the administration of sanitation. Across all implementation areas, the average rating of administration of sanitation services was “somewhat dissatisfied” and, in Mekeshifa, “very dissatisfied”. • 57% of respondents had positive views about the local authority’s ability to maintain health infrastructure; 53%, its ability to respond to outbreaks; and 56%, its ability to provide required number of health workers. Overall, there were no significant differences across various groups on perception about the effectiveness of government to administer health services. • Two-thirds of respondents were either “very satisfied” or “somewhat satisfied” with the local authority ability to maintain education infrastructure. • With the exception of Mekeshifa, Tikrit, and West Mosul, respondents of most implementation areas were satisfied with the administration of education services. • About half of the respondents had positive views on the local electricity authority’s ability to process new applications, check meters, provide customer services, billing and responding to service interruptions. Two-thirds of respondents had positive views on the local authority’s maintenance of existing electrical infrastructure, with significant variations among the provinces. • 66% of the respondents were either “very satisfied” or “somewhat satisfied” with the local authority’s ability to maintain the water utility infrastructure. • 59% were satisfied with response times to repair service interruptions, and 56% with the regularity of billing. • 43% of respondents lamented that there was poor customer service and complaints handling. • 41% were satisfied with the ability of the local authorities to maintain and rehabilitate sanitation infrastructure. • 34% reported that the utility providers were responsive to dysfunctional drainages, and 45% were satisfied with the municipalities’ collection of solid waste. 78 EQ3: How have FFS activities affected IDPs in target communities? 3a. To what extent FFS activities have influenced IDPs’ decision to return to their communities or influenced others to remain in their communities relative to other factors. 3b. In what ways have FFS activities affected IDPs’ lives since their return to their communities. 79 Conclusion Findings Recommendations Through its efforts to restore access to core services, FFS addresses factors that influence decisions to return or stay (for those who never left) for up to 86% of the population, although not all services are equally important and none were cited by a majority of PBS respondents as critical to decisions to return across the full population. For roughly a third of the population, restoration of electricity, water and sanitation, health, and education services are critical factors in decisions to stay or return, and a consideration – though not a critical one – for an additional 30 to 50% of the population, depending on the sector. These numbers are even higher when looking at factors influencing return by intervention area only among returnees in the year(s) the majority of those surveyed returned to their homes, presenting further • Drivers of return addressed by FFS – primarily restoration of essential services, livelihoods support, and rehabilitation of residences – were not the primary factor influencing return in any intervention area in the year(s) the majority of IDPs returned. In all intervention areas save one, security was the primary driver of return as reported by 77% to 99% of surveyed respondents who returned during the specific year(s) of interest. • In all but one intervention area, essential services and residential rehabilitation were among the top five drivers of return for between 19% and 69% of the population, depending on the intervention area and intervention type. Livelihoods support was not among the top five factors influencing return in any intervention area; it was cited by between 3% and 24% of returnees depending on the intervention area. • Between 77% and 96% of FFS projects in each intervention area responded to drivers of return for at least 10% of returnees who returned to their homes during the year or years the highest number of surveyed returnees reported returning. In each intervention area, the number of drivers of return not addressed by FFS during the year(s) of highest returns, but within FFS’s mandate, varied from zero in Ramadi to four in Qayara. This suggests that FFS effectively identified and responded to drivers of return. However, data from three intervention areas suggest the support may not have been timely. In Karma, Tikrit, and West Mosul the number of returnees declined as the volume of FFS projects increased. Therefore, while responding to factors influencing returns, FFS activities likely did not influence returns. This is not surprising given that in all intervention areas, surveyed returnees reported security as a bigger driver of return than any factors addressed by FFS. • Livelihoods survey data show that life is improving for those surveyed – 34% reported that they feel life has mostly improved; 17%, completely improved; 32%, not very much improved, and 16% not improved at all. If all interventions are meant to encourage hypothesis, then based on factors driving re population-based survey and cross-reference returnee respondents to the population intervention area, USAID and UNDP show support and roads and bridges in all intervention areas. If however, these interventions are important to FFS goals – for example, if the c is an objective in itself, and is not linked to should revisit the FFS development hypothesis. FFS should consider the impacts of C4W activities on each community, noting the differences in the intervention on each religious and ethnic group. A perception of favoritism, whether implicit or explicit will likely lead to communal and national tension. 80 evidence that FFS interventions respond to key concerns of IDPs vis-à-vis their decisions to return to their homes at the time they made those decisions. However the extent varies by intervention area and intervention type and does not hold true for one intervention area. The Cash for Work, C4W, activities were successful in raising the income of participants. Nearly 50% of respondents noted that their lives and income had improved since the C4W activities, although not all ethno-religious groups benefitted equally. • A majority of C4W respondents said that the income they received was adequate to support their families’ needs, while a further majority of participants said it was their largest source of income. • Women, Christians, and Shabaks, earned the most from C4W activities. • Overall household income levels have declined after returning from displacement. 63% of surveyed returnees had an average monthly income of over IQD 300,000, as compared to 68% before displacement. • 15% of respondents earned a monthly income of IQD150,000- 300,000 (low income bracket) before displacement compared to 19% after return • Mean household income significantly dropped among those surveyed compared to before displacement. 81 EQ4: What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? 4a. Sources and availability of operations and maintenance budgets for rehabilitated services. 4b. Capacity to collect revenue. 4c. Organizational capacity of local administrative offices. 4d. Technical and management capacity of staff. Conclusion Findings Recommendations Various factors contribute to or impede local authorities’ capacity to administer services targeted by FFS, depending on the province and sector. In Anbar, factors impeding local authorities’ capacity to administer services include: budgeting in the sewage, electricity, and health directorates; revenue collection in the sewage, water, education, and electricity directorates; and IT and communications in the sewage, water, and health directorates. Factors contributing to local authorities’ capacity to administer services include: procurement in the education and health directorates; and strategic planning in the education, electricity, and • Budget capacity is highest in the Anbar Municipal Services Directorate and lowest in the Sewage and Electricity Directorates in Salah-al-Din, the Electricity Directorate in Anbar, and the Municipal Services Directorate in Ninewa. By sector, we observe the highest score across provinces in the water sector and lowest scores among sewage and electricity directorates; by provinces scores across all sectors were slightly higher in Ninewa than Anbar and Salah-al-Din. • Revenue collection capacity is highest in the Ninewa Municipal Services Directorate, and lowest in the Anbar Sewage Directorate. By sector, we observed the highest score across provinces in municipal services and lowest scores among sewage and education directorates; by provinces scores across all sectors were higher in Ninewa and Salah-al-Din than Anbar. • Procurement capacity is highest in the Sewage, Water, and Electricity • Directorates in Ninewa and the Education and Health Directorates in Anbar, and lowest in the Salah-al-Din Health Directorate. By sector, we observed the highest score across provinces in water and lowest in health; by provinces scores across all sectors were higher in Ninewa and Anbar than Salah-al￾Din. • Strategic planning capacity is highest in the Education, Electricity, Health, and Municipal Services Directorates in Anbar, and lowest in the Health Directorate in Ninewa. By sector, we observed the Although outside the scope of FFS, improving provincial authorities’ capacity for revenue collection and budgeting may need special attention from the donor community and the Government of Iraq. Donors should consider using these findings to tailor support within each province by sector and function (technical vs. management and sub￾functions within management). 82 health directorates and the municipality. In Salah al-Din, factors impeding local authorities’ capacity to administer services include: budgeting in the sewage and electricity directorates, and the municipality; revenue collection in the sewage and education directorates; M&E in the education directorate; IT and communications in the municipality; and technical capacity in the education directorate. The evaluation did not identify any factors strongly contributing to local authorities’ capacity to administer services in Salah al-Din. In Ninewa, factors impeding local authorities’ capacity to administer services include: budgeting in the education directorate and municipality; revenue collection in the sewage, water, education, and health directorates; and M&E in the municipality. Factors contributing to local authorities’ capacity to administer services relate to procurement in the highest score across provinces in electricity and education and lowest in health; by provinces scores across all sectors were highest in Anbar followed by Salah-al-Din and Ninewa. • Work planning capacity is highest in the Health Directorate in Anbar, and lowest in the water and municipality directorates in Salah-al-Din, the municipality directorate in Ninewa, and the electricity directorate in Anbar. By sector, the scores are highest across provinces in water and lowest in electricity; by provinces, scores across all sectors were highest in Anbar followed by Salah￾al-Din and Ninewa. • Human resources management capacity is highest in the Education and Municipal Services Directorates in Anbar and the Water and Municipal Services Directorates in Salah-al-Din, and lowest in the Sewage and Health Directorates in Ninewa. By sector, we observed the highest score across provinces in municipal services and lowest in health; by provinces scores across all sectors were highest in Anbar, followed by Salah-al-Din and Ninewa. • M&E capacity is highest in the Electricity Directorate in Ninewa, and lowest in the Education Directorate in Salah-al-Din. By sector, we observed the highest score across provinces in electricity and health and lowest in education; by provinces scores across all sectors were higher in Ninewa and Salah-al-Din than Anbar. • IT capacity is highest in the Health Directorate in Salah-al-Din, and lowest in the Water Directorate in Anbar. By sector, we observed the highest score across provinces in electricity and lowest in water; by provinces scores across all sectors were highest in Ninewa, followed by Salah-alDin and Anbar. • On average, technical capacity is highest across provinces in electricity, including the overall highest score in Salah-al-Din, and lowest in education, including the overall lowest score in Salah￾al-Din. There was less variation across provinces than within them, with technical capacity across all sectors highest in Anbar 83 sewage, water, and electricity directorates. followed by Salah-al-Din and Ninewa. By contrast, on average across provinces, management capacity is lowest in the electricity sector and highest in education, although there is much less variation across sectors and provinces for management capacity as compared to technical capacity. 84 ANNEX 2: SCOPE OF WORK I.PURPOSE OF THE EVALUATION The purpose of the evaluation is to identify factors that promoted or impeded positive outcomes from FFS stabilization interventions from July 2015 through December 2017 to inform ongoing and new stabilization activities and other USAID/Iraq programs. This midterm evaluation will be designed to evaluate the current FFS program and inform future USAID stabilization programming in Iraq and other complex, insecure environments. The main audiences of the evaluation include U.S. Mission Iraq, USAID/Washington Middle East Bureau, stabilization stakeholders and partners, and the Department of State. The objectives of this evaluation are three-fold: 1. Evaluate the United Nations Development Program (UNDP) FFS program, including components funded by USAID under the Funding Facility for Immediate Stabilization (FFIS) and the Funding Facility for Expanded Stabilization (FFES) (collectively referred to as FFS) in accordance with all applicable USAID ADS regulations regarding midterm evaluations. 2. Identify and characterize contextual and programmatic factors associated with FFS achievement of results. 3. Provide specific, programmatic recommendations for corrective action if needed. II. SUMMARY INFORMATION Strategy/Project/ Activity Name FFS, including FFIS and FFES components Implementer United Nations Development Program Public International Organization Grant # AID-267-IO-15-00001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $197 million as of December 31, 2017 Life of Strategy, Project, or Activity 2015-2020 Active Geographic Regions Newly liberated areas of Ninewa, Anbar, and Salah al-Din Development Objective(s) (DOs) Strengthen Stabilization of Areas Liberated from Da’esh USAID Office USAID/Iraq – Office of Governance and Economic Opportunity (GEO) III. BACKGROUND The Islamic State of Iraq and Syria (ISIS) formed in 2006 as an offshoot of Al Qaeda in Iraq. In 2013 it was disavowed by Al Qaeda after in-fighting, and in June 2014 ISIS began a lightning advance across Iraq, 85 seizing control of Iraq's second largest city, Mosul, as well as Tikrit, four other Iraqi towns, and the Baiji oil refinery. By May 2015, ISIS had also taken control of Ramadi, controlling two large lines of territory to the north and the west advancing toward Baghdad. ISIS eventually occupied between 30-40% of Iraqi territory. After over three years of struggle to liberate occupied territory, on December 10, 2017, the Prime Minister declared victory in liberating Iraqi territory previously occupied by ISIS. The U.S. Government, through USAID, was one of the first donors to support the United Nations Development Program (UNDP) Funding Facility for Immediate Stabilization (FFIS). As cities and territory occupied by ISIS were liberated, this multi-donor funded program provides critical support to the Government of Iraq (GOI) to help facilitate the safe and voluntary return of displaced populations, primarily in Anbar, Ninewa, and Salah ad Din. USAID supports two components of the FFIS. The first component supports public works and light infrastructure rehabilitation, such as health clinics, water infrastructure and treatment facilities, power grids and generators, schools, other government buildings, and access roads. The second component supports livelihood activities to jumpstart the local economy and generates income for local households, for example through grants to small businesses with high community impact (bakeries, electrical shops, agriculture, food kiosks, and micro-stores) and short-term employment through public work schemes (cash-for-work opportunities). When the FFIS was formed, the working assumption was that enhanced stabilization projects would be undertaken by the Iraqi Government, using public revenues, as soon as FFIS left a city or intervention area. This sequencing of post-liberation efforts did not materialize because of the drastic drop in oil revenue. Concerned that military gains were at risk and that progress made during immediate stabilization might be reversed by the slower pace of reform and reconstruction, the Iraqi government, UNDP, and the donor community approved the Funding Facility for Expanded Stabilization (FFES). The goal of the FFES was to support high-impact, medium-scale projects with priority given to projects that help stabilize safe corridors between liberated cities, and projects that generate large numbers of jobs and otherwise create incentives for displaced persons to return. FFES activities were never envisaged as a substitute for larger-scale reconstruction efforts, including those being implemented by the World Bank and the German Development Bank. USAID was the first donor to contribute to the FFIS in July 2015 and the FFES in August 2016. As of December 2017, USAID has obligated $115.3 million. From the beginning, the UNDP established a FFS steering committee whose mandate is to oversee and provide strategic guidance. The Steering Committee consists of Iraqi government officials, donor partners, UNDP, and international financial institutions. The GOI Prime Minister’s Office guides overall implementation, with the Steering Committee approving locations and prioritizing activities based on need and any specific requirements for location and activity set by individual donors. Core criteria for stabilization activities in Iraq include: 1. any support funded under any of the components must adhere to the general principles of international human rights and humanitarian law (especially non-discrimination and impartiality), and promote reconciliation and social cohesion; 2. activities shall employ accountability and transparency mechanisms that, to the greatest extent possible, reinforce existing good governance practices and systems for local development; 3. areas receiving support must have established mechanisms for the safe and voluntary return of IDPs; 4. legitimate security arrangements fully under Government of Iraq control should be reestablished; and, 5. gender dimensions and minority rights should receive particular attention. 86 USAID’s agreement with UNDP stipulates that USAID may direct where and which activities to fund and implement with USG resources. UNDP has agreed generally that donors may earmark their funds at any level. In accordance with the terms of the agreement, UNDP must obtain approval from the Agreement Officer (AO) prior to using any USAID funds for any project or activity. To obtain approval, UNDP is required to submit a proposal with an estimated budget to the AO for initial review. USAID’s project selection is conducted in close consultation with the U.S. Embassy leadership. To ensure that the Mission is in compliance with all applicable laws, regulations, and policies governing the use of USAID funds, each proposal submitted by UNDP is reviewed and cleared by the Technical Office and the Resident Legal Officer, with the AO providing the final review and approval. UNDP is responsible for carrying out regular monitoring of FFS activities and reporting results to USAID. This work is in accordance with USAID policy in ADS 308 for grants to Category 1 PIOs, such as UNDP. USAID also relies on weekly activity updates and formal quarterly reporting from UNDP to monitor FFS progress and results. Development Hypothesis The purpose of the FFS is to encourage the return of IDPs and to promote reconciliation by demonstrating that the GOI is committed to providing citizens with essential basic services in newly liberated areas. If immediate delivery of critical services (for example for health, water, power, and education) and employment and livelihood opportunities are provided by the GOI to encourage people to return to their homes, then communities in liberated areas will stabilize and recover more quickly and the trust of displaced Iraqis in the GOI will be restored. Critical Assumptions USAID/Iraq identified a number of critical assumptions that relate to both the implementing environment in Iraq as well as the U.S. Government environment for funding and operating in Iraq: Host Country Critical Assumptions: • Implementation opportunities and political will continue to allow for planned activities. • Security conditions permit program operations. • Iraqi Security Forces (ISF) continue to take back territory from ISIS. • Mosul Dam holds. USG-USAID Critical Assumptions: • USAID/Iraq staffing levels remain at least where they are currently. • Funding and all resources for planned programming are secured on an annual basis. IV. EVALUATION QUESTIONS The evaluation will consider the primary programmatic areas addressed by stabilization activities from 2015-2017. The evaluation questions are designed to result in the identification of factors that promoted or impeded positive outcomes from FFS stabilization interventions. To achieve this, the sub-questions are designed to elicit, in part, characterizations of inclusivity and level of engagement of government officials and civil society in design and implementation, including access to services. With this information, the Evaluation Team will then identify contextual and programmatic factors that promoted or impeded positive outcomes, including inclusive access to services. The evaluation questions below are the specific areas and questions that the Evaluation Team must address. 87 These evaluation questions apply to all USAID-funded and, to the extent practical related other donorfunded, rehabilitation of public works and other critical infrastructure (all sectors) and livelihoods (jobs and small business grants) activities. The main overall focus of the evaluation will be to identify what factors are associated with the successful identification, prioritization, and rehabilitation of the most critical infrastructure needs. 1. How have FFS activities affected access to services in target communities? Evaluators will assess: • Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including, but not limited to, health, education, water, electricity, and other municipal services). • Whether access to services is equitable across vulnerable groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, politically affiliated groups, and youth and children. • What significant barriers, if any, prevent members of target communities from accessing services? 2. How have FFS activities affected the quality of services in target communities? Evaluators will assess: • Sector-specific parameters of quality of services targeted by FFS (including, but not limited to, health, education, water, electricity, and other municipal services). Note changes in performance indicators; application of performance standards (e.g., potable water with a free chlorine level of 1 milligrams per liter); application of performance plan (e.g., periodic testing of chlorine in potable water); hours per day of availability of water/electric services; availability of services in FFS-supported hospitals and health clinics; and completion and use by students of rehabilitated schools buildings. • Perceptions of the quality of services among members of target communities. • Perceptions of the effectiveness of government administration of services among members of target communities. 3. How have FFS activities affected IDPs in target communities? Evaluators will assess: • To what extent FFS activities have influenced IDPs’ decision to return to their communities or influenced others to remain in their communities relative to other factors. • In what ways have FFS activities affected IDPs’ lives since their return to their communities. 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Evaluators will assess: • Sources and availability of operations and maintenance budgets for rehabilitated services. • Capacity to collect revenue. • Organizational capacity of local administrative offices. 88 • Technical and management capacity of staff. V. EVALUATION DESIGN AND METHODOLOGY Desk Study Before fieldwork begins, the Evaluation Team will conduct a desk study of relevant documentation relating to FFS activities. Some of the documents that USAID will share with the Evaluation Team are as follows: • Regular reporting and activity documentation (such as quarterly reports, donor updates from Steering Committee meetings). • Relevant sections of the grant agreement between USAID and UNDP, with all relevant modifications and deliverables. • Field monitoring reports on FFS collected by third-party monitors (USAID-funded primarily and, if available, reports from UNDP and other donor third-party monitors). While in the field, the team will collect records and documentation from local administrations and public service facilities, as available. Key informant interviews The Evaluation Team will conduct key informant interviews (KIIs) with USAID mission staff; government counterparts at the national, provincial, and local levels; and senior implementing partner staff, as well as other key technical staff, and representatives from civil society and community representatives. The nature, relationship to the activity, and potential for bias of KIIs will be weighed in interpreting responses and acknowledged in the presentation of findings and in the evaluation limitations. Population-based surveys The Evaluation Team will conduct a population-based survey(s) to provide evidence of high-level results in response to Evaluation Questions. The Evaluation Team will stratify sampling by gender to ensure an equal number of male and female respondents, with appropriately assigned male and female enumerators. The Evaluation Team will propose a sampling approach (e.g., cluster-based) that is rigorous and proportionately representative of youth, ethno-religious groups, religious and ethnic minorities, people with disabilities, and other marginalized groups. The Evaluation Team must consult with USAID/Iraq on the appropriate means to engage and sample such minority and marginalized groups. The Evaluation Team will use this survey to assess availability of relevant basic services addressed by FFS, as well as quality of services and inclusive access to them. Because there are no baseline data for the FFS activity, the survey will need to be designed to elicit retrospective perceptions. The survey will be integrated methodologically with direct observation (described below). Wherever possible, direct observations of services and service infrastructure will be made in communities in which the survey is administered. The Evaluation Team may also draw upon situation reports and other monitoring data from the communities being surveyed, collected by the USAID Iraq Performance Management and Support Program (PMSP) or others on behalf of USAID. Livelihoods program beneficiary surveys To provide evidence of high-level results in response to the evaluation questions, the Evaluation Team will conduct a rigorous survey of beneficiaries who participated in livelihoods programming. The 89 Evaluation Team will work with the implementing partner and/or implementing NGOs to obtain lists of participants and sample appropriately. Qualitative data collection Qualitative data collection (e.g. semi-structured or group interviews, Focus Group Discussions) will be used to gather qualitative information from community members and representatives to supplement KIIs and provide context for quantitative findings. Analysis of qualitative data will be approached systematically and described in evaluation report annexes. The Evaluation Team will provide justification for the selection of individual respondents (in addition to the stakeholder groups to which they belong) and seek to ensure that diverse perspectives are represented, including those of women, religious and ethnic minorities, and youth. On-site observations To the extent possible, the Evaluation Team is expected to make site visits to assess the public works rehabilitation and light infrastructure activities implemented by FFS. A description of proposed site visits will be included in the evaluation work plan. These will be selected to align with other data collection methods, particularly sites selected for surveys. Site visit checklists will be used to assess the quality of public services provided and the role of infrastructure in the delivery of quality basic services. Gender Gender considerations must be taken into account in this performance evaluation. In addition to gender disaggregation and differences in trends for all questions, the evaluation partner also will briefly summarize UNDP assistance promoting gender equality and the empowerment of women among GOI partners. The evaluation will consider the extent to which central and provincial governments took gender equality and women’s empowerment issues and concerns into consideration in delivering services. VI. EVALUATION TEAM COMPOSITION Evaluation Team members must provide a written disclosure of any conflicts of interest (COI), and require key personnel to submit their COI disclosure with the proposal. The Evaluation Team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID ADS 200. The contractor shall employ an Evaluation Team with appropriate qualifications and relevant experience in post-conflict stabilization activities and program evaluation. Collectively, the Evaluation Team must have the following qualifications: • Demonstrated technical expertise in public sector service delivery, including familiarity with: • Implementation in post-conflict and insecure states • Demonstrated expertise in monitoring and evaluation methodology, including in: • Post-conflict, high-risk security environments • Relevant participatory evaluation methodology • Population-based perceptions survey design • Knowledge of stabilization principles and experience assessing • Appropriate language skills: • Fluency in Arabic • Fluency in English language writing, editing, and communication skills 90 The majority of materials available for review to the Evaluation Team will be in English or Arabic. Interviews, surveys, and participatory evaluation sessions likely will be conducted in Arabic. Therefore, it is expected that multiple team members will need to have fluency in Arabic and English. All team members must be able and willing to travel to Iraq. The contractor is highly encouraged to ensure strong female representation on the team. At least one member of the Evaluation Team (not including data collectors) will be local. Data collectors will include a sufficient number of female staff to ensure that female data collectors are able to conduct all interviews with female survey respondents. The Evaluation Team will include all of the skills listed above. It is the responsibility of PMSP to ensure that the overall composition of the team provides expertise and skills needed to answer all evaluation questions provided in this SOW. VII. CRITERIA TO ENSURE QUALITY OF THE EVALUATION REPORT Per ADS 201 maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). If the Evaluation Team wishes to provide information on challenges to data collection, they should do so in a separate section. • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. For each evaluation subquestion, findings should be clearly linked to analysis; conclusions should be clearly linked to findings, and any recommendations made should be clearly linked to the conclusions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. For this evaluation, recommendations should be grounded in the analysis of factors that contribute to or impede success, and provide information about the potential replicability of those factors by USAID or other donors in future programming. VIII. OTHER REQUIREMENTS 91 All quantitative data collected by the Evaluation Team must be provided in machine-readable, nonproprietary formats as required by USAID’s Open Data policy (see ADS 579). The data will be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, Evaluation Team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions will be updated in the SOW that is included as an annex to the Evaluation Report. IX. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation work plan: A draft work plan for the evaluation shall be completed by the lead evaluator and presented to the COR. The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the Evaluation Team, delineated by roles and responsibilities. The work plan will include the draft schedule for briefings, data collection, analysis, and submission of deliverables. 2. Evaluation design: Within three weeks of approval of the work plan, the Evaluation Team must submit to the COR an evaluation design (which will become an annex to the Evaluation Report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited, and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID offices and relevant stakeholders are asked to take no more than 10 business days to review and consolidate comments through the COR. Once the Evaluation Team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within four business days. 3. In-briefing: Within two days of arrival in Baghdad, the Evaluation Team will have an in-briefing with USAID for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the SOW, if necessary. If the team arrives prior to conducting the primary desk review, then a second in-briefing will take place after the review has taken place. Note: The core PMSP staff will meet regularly with USAID to discuss progress; Evaluation Team members will join when needed. In lieu of an inception report, the weekly updates will include electronic copies of materials presented during the regular briefings. 4. Midterm briefing: The Evaluation Team is expected to hold at least one midterm briefing with the USAID Program Office on the status of the evaluation, including potential challenges, emerging 92 opportunities, and preliminary findings. The timing of the midterm briefing(s) will depend upon the work plan and will be agreed through approval of the evaluation design. 5. Exit briefing: The Evaluation Team is expected to hold a final exit briefing prior to leaving the country to discuss the status of data analyses and preliminary findings. This will include a presentation of key findings and recommendations that the Evaluation Team will also give to other stakeholders. The draft power-point slide deck will be shared with USAID for review at least two days prior to the exit briefing. 6. Exit presentation: The Evaluation Team will give a final presentation of findings and recommendations to USAID and other stakeholders, potentially including implementing partners of this or related activities, key government stakeholders, or other donors, at USAID’s discretion. This presentation will be scheduled in the work plan and agreed through the evaluation design. 7. Draft evaluation report: The draft evaluation report will be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the Evaluation Team considers to have a bearing on the objectives of the evaluation. Any additions or changes to the report that were not included in the approved evaluation design will be made only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID will have two calendar weeks in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the Evaluation Team. The Evaluation Team will then be asked to submit a revised final draft report within one calendar week, and again the USAID will review and send comments on this final draft report within one calendar week of its submission. 8. Final evaluation report: The Evaluation Team will be asked to take no more than one calendar week to respond/incorporate the final comments from USAID. The Evaluation Team leader will then submit the final report to the COR. All project data and records, including a list of the documents reviewed, will be submitted in full and will be in electronic form in an easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. The evaluation final report will include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. The executive summary will be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. The Evaluation Team will clearly describe all sampling methodologies, including for group discussions. This includes describing the sampling frame, the sampling method, and all sample sizes with justification. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation 93 methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The final report will also include the final evaluation schedule for briefings, data collection, analysis, and submission of deliverables. The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the Evaluation Team; • An updated evaluation matrix summarizing key findings to the sources and methods used for each evaluation question and sub-question; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; • Signed disclosure of conflict of interest forms for all Evaluation Team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest; and • Summary information about Evaluation Team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the USAID Development Experience Clearinghouse within three months of the evaluation’s conclusion, unless otherwise directed by USAID. USAID may request PMSP to redact sensitive material before posting. XII. SUBMISSION REQUESTS Evaluation matrix The Evaluation Team will utilize a design matrix following the general format provided below in their initial evaluation design. The information in the matrix will be specific at the sub-question level. At the submission of the evaluation design, the team will include an updated version of this matrix, identifying the tools (including which sections/questions are relevant) that were developed to provide data for each sub-question. The Evaluation Team will submit a matrix of the same format identifying key high-level findings as an annex to the evaluation report. Questions Suggested Data Sources Suggested Data Collection Methods Data Analysis Methods 1. How have FFS activities Sources including: Monitoring data, site To be determined 94 affected access to services in target communities? performance monitoring data, previous evaluations, national statistics, activity staff, stakeholders, expert knowledge, beneficiaries, facility records visit data, and direct observation/Key Informant Interviews (KIIs), and qualitative methods with community members, e.g. Focus Group Discussions (FGDs), and population based surveys by Evaluation Team. Requested level of disaggregation— gender, ethnicity, location (Intervention area, province). Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including but not limited to, health, education, water, electricity, and other municipal services). Activity staff, records and reports, Facility staff and records, Activity contractors, Third-party monitoring records and reports Secondary analysis of monitoring data and site visit data, Direct observation, Key Informant Interviews (KIIs) Whether access to services is equitable across vulnerable Beneficiaries Population-based surveys groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, political affiliation, and youth and children. What significant barriers, if any, prevent members of target communities from accessing services? Beneficiaries KIIs, Qualitative beneficiary interviews 2. How have FFS activities affected the quality of services in target communities? Sources including: performance monitoring data, previous evaluations, national statistics, project staff, stakeholders, expert knowledge, beneficiaries Secondary analysis of monitoring data and facility records, KIIs, Direct observation Population-based surveys To be determined by Evaluation Team. Requested level of disaggregation— gender, ethnicity, location (Intervention area, province). 95 Sector-specific parameters of quality for services targeted by FFS (including but not limited to, health, education, water, electricity, and other municipal services). Note changes in performance indicators; application of performance standards (e.g., potable water with a free chlorine level of 1 milligrams per liter); application of performance plan (e.g., periodic testing of chlorine in potable water); hours per day of availability of water/electric services; availability of services in FFS-supported hospitals and health clinics; and completion and use by students of rehabilitated schools buildings. Activity staff, records and reports, Facility staff and records, Activity contractors, Third-party monitoring records and reports Secondary analysis of monitoring data and facility records, KIIs, Direct observation Perceptions of the quality of services among members of target communities. Beneficiaries Population-based surveys Perceptions of the effectiveness of government administration of services among members of target communities. Beneficiaries Population-based surveys 3. How have FFS activities affected IDPs in target communities? Sources including: performance monitoring data, previous evaluations, national statistics, project staff, stakeholders, expert knowledge, beneficiaries Population based survey and qualitative methods To be determined by Evaluation Team. Requested level of disaggregation— gender, ethnicity, location (Intervention area, province). 96 To what extent FFS activities have influenced IDPs’ decision to return to or to remain in their communities, relative to other factors Beneficiaries population-based survey In what ways have FFS activities affected IDPs’ lives since the transition back to their communities Population-based survey Qualitative beneficiary interviews 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Sources including: performance monitoring data, previous evaluations, national statistics, project staff, stakeholders, expert knowledge, beneficiaries KIIs and organizational capacity assessment To be determined by Evaluation Team. Requested level of disaggregation— gender, ethnicity, location (Intervention area, province). Sources and availability of operations and maintenance budgets of rehabilitated services Local authorities and administrative staff, Activity staff records and reports, Facility staff and records KIIs Capacity to collect revenue Local authorities and administrative staff, Activity staff records and reports Facility staff and records KIIs Organizational capacity of local administrative offices Local authorities and administrative staff Organizational capacity assessment Technical and management capacity of staff Local authorities and administrative KIIs staff, Activity staff records and reports, Facility staff and records 97 CLIN 0008 PAYMENT SCHEDULE Deliverable Document Required for Payment. Payment Amount Final Evaluation Design Accepted by COR Final Evaluation Design 25% Draft Evaluation Report Acceptance by COR of the Draft Evaluation Report 25% Exit Presentation Email from COR accepting the presentation as acceptable 10% Final Evaluation Report Acceptance by COR Final Evaluation Report 40% 98 ANNEX 3: EVALUATION DESIGN Introduction 1.1 Brief Summary of the Program and Evaluation USAID tasked International Business and Technical Associates, Inc. (IBTCI), through its Performance Management and Support Program for Iraq (PMSP), to conduct a mid-term performance evaluation to identify factors that promoted or impeded positive outcomes from Funding Facility for Stabilization (FFS) stabilization interventions from July 2015 through June 2018 to inform ongoing and new stabilization activities and other United States Agency for International Development (USAID)/Iraq programs. This midterm performance evaluation is designed to evaluate the current FFS program and inform future USAID stabilization programming in Iraq and other complex, insecure environments. The main audiences of the evaluation include USAID/Iraq, USAID/Washington Middle East Bureau, stabilization stakeholders and partners, and the Department of State. The objectives of this evaluation are three-fold: Evaluate the United Nations Development Program (UNDP) FFS program, including components funded by USAID and other donors under FFS in accordance with all applicable USAID Automated Directives System (ADS) regulations regarding midterm evaluations. Identify and characterize contextual and programmatic factors associated with FFS achievement of results. Provide specific, programmatic recommendations for corrective action if needed. 1.2 List of Evaluation Questions The evaluation questions are listed as follows in Table 1: Table 1: Evaluation Questions Questions QUESTION 1. How have FFS activities affected access to services in target communities? Evaluators will assess: 1a. Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including, but not limited to, health, education, water, electricity, and other municipal services). 1b. Whether access to services is equitable across vulnerable groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, politically affiliated groups, and youth and children. 1c. What significant barriers, if any, prevent members of target communities from accessing services? 99 QUESTION 2. How have FFS activities affected the quality of services in target communities? Evaluators will assess: 2a. Sector-specific parameters of quality of services targeted by FFS (including, but not limited to, health, water, electricity, and other municipal services). Note changes in performance indicators; application of performance standards (e.g., potable water with a free chlorine level of 1 milligrams per liter); application of performance plan (e.g., periodic testing of chlorine in potable water); hours per day of availability of water/electric services; availability of services in FFS-supported hospitals and health clinics; and completion and use by students of rehabilitated school buildings. 2b. Perceptions of the quality of services among members of target communities. 2c. Perceptions of the effectiveness of government administration of services among members of target communities. QUESTION 3. How have FFS activities affected IDPs in target communities? Evaluators will assess: 3a. To what extent FFS activities have influenced IDPs’ decision to return to their communities or influenced others to remain in their communities relative to other factors. 3b. In what ways have FFS activities affected Internally Displaced Persons’ (IDP)’ lives since their return to their communities. QUESTION 4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Evaluators will assess: 4a. Sources and availability of operations and maintenance budgets for rehabilitated services. 4b. Capacity to collect revenue. 4c. Organizational capacity of local administrative offices. 4d. Technical and management capacity of staff. 2.0 Evaluation Design This evaluation design includes: A detailed evaluation design matrix that aligns methodologically the Evaluation Questions in the Evaluation Statement Of Work (SOW) to relevant data sources, methods, and the data analysis plan; 100 Draft questionnaires and other data collection instruments or their main features; The list of potential interviewees and sites to be visited, and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); Known limitations to the evaluation design; and, A dissemination plan. A detailed evaluation design matrix linking evaluation questions to data sources, methods and analysis plan is attached as Getting to Answers (G2A) matrix (Annex I). A proposed evaluation methodology along with a sampling framework and limitations to the evaluation design is presented in the section below; proposed data collection tools are provided in Annexes IV to IX. 3.0 Methodology The evaluation will employ a mixed-methods approach to data collection. The team will use both quantitative and qualitative methods to support answers to the evaluation questions. Quantitative data collection will include a Population Based Survey and a Livelihood Survey. Qualitative data collection will include Focus Group Discussions (FGD), Key Informant Interviews (KII) and Onsite Observations. In addition, the team will conduct rapid capacity assessments using a tool adapted from USAID Organizational Capacity Assessment (OCA) tool. A description of data collection methods for each evaluation question is presented in Annex I. Table 2: Proposed data collection methods for the evaluation questions Data Collection Methods Evaluation Questions Desk Review 1, 2, 3, 4 Key Informant Interviews 1, 2, 3, 4 Population Based Surveys 1, 2, 3 Focus Group Discussions 1, 2, 3 Field Observation and Site Visits 2 Livelihood Beneficiary Survey 3 Capacity Assessment 4 3.1 Document Review Before commencement of the field data collection, the team will review project documents and existing secondary data. The team will address the three69 years of FFS intervention from July 2015 through June 2018. Below is an illustrative list of documents to be reviewed: 69 It should be noted that not all areas had three years of intervention. For example FFS interventions in Tikrit started in 2015, followed by Fallujah and Ramadi. Mosul followed in fall 2017 and activities in Western Anbar started in February 2018. 101 Existing program data; Activity monthly, quarterly, and annual reporting and activity documentation (such as donor updates from steering committee meetings, annual work-plans, etc.); Grant agreements between USAID and UNDP, with all relevant modifications and deliverables; Implementing Partner (IP) Monitoring and Evaluation (M&E) data; Field monitoring reports on FFS collected by IP or third-party monitors (USAID-funded primarily and, if available, reports from UNDP and other donor third-party monitors); and, Donor reports, host country data, other FFS evaluations (if available) and analysis reports on Iraq stabilization efforts when available. 3.2 Key Informant Interviews KIIs with key stakeholders with in-depth knowledge on what is going on in target intervention areas and provinces, particularly government officials, are expected to provide overall background information and help contextualize the overall evaluation framework. While the quantitative data from surveys will provide micro-level data from households (PBS) and individuals (livelihoods survey) the qualitative data from key informant interviews will provide context for overall intervention area/provinces. KIIs will be used to understand higher-level issues related to access to and quality of basic services (EQ 1 and 2) in the targeted areas. In particular KIIs will provide perspectives of key stakeholders on accomplishments, best practices and lessons learned, and other factors that may have inhibited or promoted success of FFS (EQ 4). KIIs will also be instrumental in assessing capacity of government entities to administer FFS service delivery. The team expects to conduct a total of 35 KIIs (Annex IX). The Evaluation Team will conduct KIIs with: USAID mission staff and other FFS donor staff; Government counterparts at the national, provincial, and local levels; Senior implementing partner staff, as well as other key technical staff; and. Representatives from civil society and community. The nature, relationship to the activity, and potential for bias of KIIs will be weighed in interpreting responses and acknowledged in the presentation of findings and in the evaluation limitations. The list of key informant interviews will be developed after the Evaluation Team’s initial consultations with USAID. 3.3 Focus Group Discussions FGDs will be conducted with beneficiaries of FFS. The evaluation will employ purposive (and/or convenience) sampling technique for selection of FGD participants. FGDs will be conducted with homogeneous groups of nine to 12 participants in the three provinces (Anbar, Ninewa, and Salah alDin). The Evaluation Team proposes a total of 12 FGDs; for each identity group, separate FGDs will be conducted for males and females for cultural reasons. Six FGDs, three for each of the gender group, will 102 be conducted with participants from the returnee communities in Ninewa, Anbar and Salah al-Din; two FGDs, one for each of the gender group, will be conducted with participants from the IDP community in Salah al-Din; while four FGDs, two for each of the gender group, will be conducted with Religious and Ethnic Minorities (REMs) from the IDP and returnee communities in Ninewa. FGD participants from the returnee communities will be selected from the catchment areas of FFS rehabilitated facilities. Religious and ethnic minorities will be selected from returnee and IDP communities. Table 3 below shows the distribution of FGDs among the different identity groups in the three provinces. Table 3: Proposed Number and Distribution of FGD Identity Group Province Total Ninewa Anbar Salah al-Din Male Female Male Female Male Female Returnees 1 1 1 1 1 1 6 REM Returnees 1 1 - - - - 2 IDPs - - - - 1 1 2 REM IDPs 1 1 - - - - 2 TOTAL 3 3 1- 1 2 2 12 While the surveys will provide quantitative data for statistical analysis, FGDs will help in contextualizing the evaluation framework, considering plausible explanations for correlations and patterns observed, and triangulating with quantitative data. FGD participants will be asked about the reasons and/or explanations for whether or not there were positive or negative changes in their community related to various issues around access and quality of healthcare, education, electricity and other infrastructure implemented by the FFS. 3.4 Population Based Survey (PBS) The PBS will be conducted to assess perceptions of services in areas serviced by a facility rehabilitated by FFS. The PBS will provide generalizable quantitative data in response to evaluation questions number 1, 2 and 3. The design below was proposed by USAID and reflects revisions proposed by PMSP, specifically regarding the use of LandscanTM. 3.4.1 Survey areas The survey will be conducted in Anbar, Ninewa, and Salah al-Din provinces to assess perceptions of services in areas served by facilities rehabilitated by FFS. Eligible facilities include education, electricity, health, sewage, and water facilities supported by FFS interventions funded by USAID and other donors. In addition, FFS implements housing, bridges and roads, municipal equipment, and cash-for-work programming. These implementation areas are addressed in the overall mid-term evaluation of FFS, but are not included in the population survey, because these interventions do not address basic services. 103 Within each province, intervention areas that have a high concentration of FFS sponsored activities have been selected for data collection. The total number of eligible facilities was tallied for each intervention area, including those that were under implementation or completed by June 6, 2018. The ten intervention areas selected ensure coverage of at least 83% of eligible FFS facilities in each province. Table 4 below presents the distribution of FFS sponsored facilities in the intervention areas and provinces. Data on the locations of facilities will be provided by UNDP, with the intervention area identified and the GPS coordinate listed, for the intervention areas in Table 4. Based on the information provided by UNDP about the FFS facilities and their locations, the Evaluation Team will determine the catchment population. Table 4: Distribution of Facilities in the Provinces and Intervention areas70 Anbar Salah al-Din Ninewa Interventi on area Eligibl e faciliti es % of Eligibl e Faciliti es Interventi on area Eligibl e faciliti es % of Eligibl e Faciliti es Interventi on area Eligibl e faciliti es % of Eligibl e Faciliti es Ramadi 137 46% Tikrit 44 51% East Mosul 198 34% Fallujah 69 23% Shirqat 20 23% N. Plain 167 28% Karma 41 14% Mekeshifa 9 10% West Mosul 109 19% Haditha 27 9% Baiji 7 8% Qayara 55 9% Anah 8 3% Al Dour 6 7% Sinuni 22 4% Qaim 7 2% Rabia 19 3% Rawa 6 2% Sinjar 17 3% Heet 3 1% Rutba 1 0% Total eligible facilities in Anbar 299 Total eligible facilities in Salah alDin 86 Total eligible facilities in Ninewa 587 Facilities in selected intervention areas 247 Facilities in selected intervention areas 73 Facilities in selected intervention areas 529 70 Adapted from the draft evaluation design for FFS PBS from USAID to PMSP of June 21, 2018 104 Coverage in three intervention areas 83% Coverage in three intervention areas 85% Coverage in four intervention areas 90% 3.4.2 Primary Sampling Units A cluster-based population-proportional sampling approach will be used, based on population data from Landscan™, which was suggested by USAID in the proposed design for PBS dated June 21, 2018. In absence of official population data, Landscan provides a community standard for gridded global population data. Landscan™ combines models tailored to data conditions in each country and region to disaggregate best-available demographic data across a grid of approximately one-square-kilometer spatial resolution. In each of the ten selected intervention areas, eligible facilities will be mapped using geo-coordinates provided by UNDP, and classified by facility type. An appropriate geographic buffer71 will be determined, representing a population potentially served by the facility; the radius of this buffer will differ between facility types. Grid cells will be coded accordingly, for all types of facilities by which they could potentially be served. This process will only approximate the actual service area of any given facility, as delineating exact service areas on a facility-by-facility basis would be prohibitively time￾consuming. The approach proposed here, however, will be based on a reasonable supposable reach of a given facility, based on its function. The Evaluation Team may consider some exceptional cases on a case-by-case basis for individual coding of the geographic buffer. Grid cells that do not fall within the buffer for any completed facility will be omitted from the sampling pool. Cells will be stratified by type and saturation of facilities (e.g. categorized by number and type of facilities serving the cell), and then selected randomly, proportional to population. 3.4.3 Sample size The sample size for each province was calculated using the following assumptions: Table 5: Assumptions for Sample Size Calculation Margin of error 5% Confidence interval 95% Precision 5% Power 80% Proportion (est.) 50% Target Population 7,096,889 Design effect 2 Non-response rate 15% 71 The geographic buffer will be determined once updated LandScan data is available to the Evaluation Team. 105 N- unadjusted 768 N- adjusted for non-response 904 Using the targeted population for the 3 provinces from the 2018 Population and Housing census from Central Statistics Organization for Iraq projection estimated at 7,096,889 in 2018 , the proposed minimum sample size per-province is 768, after adjusting for non-response rate of 15% an additional 136 respondents per province are considered. The minimum sample size per province would then be n=9044 . If the non–response rate becomes higher than planned, there will be sample weights adjustments for the respondents to compensate for the non-response during analysis. In each of the selected cells the Evaluation Team will survey 24 respondents (12 men and 12 women)72. In each province, 3873 cells will be selected, for a total of 114. Therefore, the final sample size for population based survey is n=2,73674 respondents (see Table 6). Sample population weights will be used to account for population size distribution. Cells that are determined to be inaccessible due to security concerns will be replaced by the nearest accessible cell. The sample allocation for cells across the 10 selected intervention areas is derived based on the concentration of facilities in the intervention area varying proportional to size. Table 6: Proposed Sample Allocation in the Provinces and Intervention areas 4 The adjustment for non-response is non-linear given that non-response will vary across clusters Province Selected Intervention areas Number of Eligible Facilities No. of Sampled Cells Targeted Sample Respondents Anbar Ramadi 137 21 504 Anbar Fallujah 69 11 264 Anbar Karma 41 6 144 Salah al-Din Tikrit 44 23 552 Salah al-Din Shirqat 20 10 240 Salah al-Din Mekeshifa 9 5 120 Ninewa East Mosul 198 14 336 Ninewa N. Plain 167 12 288 72 The number of 24 respondents per cluster resonates well with other typical gold standard household surveys. Most Demographic Health Surveys (DHS) use a fixed take of households per cluster of about 25-30 households, determining the number of clusters to be selected (Guide to DHS Statistics, 2018). 73 Rounded off after dividing 904 (sample size per province) by 24 (number of respondents per cell). 74 Number slightly higher due to rounding off the number of total cells included in the survey. 106 Ninewa West Mosul 109 8 192 Ninewa Qayara 55 4 96 Total 849 114 2736 3.4.4 Selection of Households and Respondents Households will be selected using a transect walk approach. Surveyors will begin at the center-point of each selected cell, and will randomly select a direction to follow away from the facility. Households will be selected following a given interval (e.g. every third household from the first), with a left-hand principle (turning left if the forward path is blocked). Each enumerator will be assigned to select either only female or only male respondents. Household selection will proceed until 24 interviews have been completed in the cell. The appropriate household interval will be selected to maximize coverage of a grid cell. Once a household is selected, the enumerator will ask to speak with either the primary male household head or the primary female household head, as assigned75. If the male/female household head is not available, they will move on to the next house. If more than one male/female household head resides in the household (as in a multi-family home) the male/female household head will be selected at random using a Kish grid. 3.4.5 Sampling Weights Sampling weights will be determined using Landscan™ population estimates where these are deemed adequate. Additional localized population data sources including facility catchment populations will be sought, especially for areas affected by recent population movements due to conflict and these sources will be triangulated to determine the most appropriate sampling weight. Adjusted populations for selected cells will be accomplished using appropriate correction factors. 3.5 Livelihoods Survey Three types of livelihood activities have been implemented under the FFS program, namely, Cash for Work, small grant support to female headed households and small business grants. According to UNDP, beneficiary data is available for 8,023 Cash for Work76 participants only. The evaluation will therefore not include beneficiaries of small grants support to female headed households and small business grants. According to FFS Annual Report 2018, 34,844 beneficiaries received support from one of the three livelihood activities as shown in Table 7 below. Table 7: Summary of Livelihoods’ Program Beneficiaries77 75 In some cases, the main head of the household may not match the gender assigned to the data enumerator to interview. The enumerator should then clarify that they are looking for the person of the assigned gender who most closely mirrors the main household head; e.g., if a man is identified as the main household head, but the enumerator has been assigned to interview women, they should identify the woman living in the household who makes most household decisions. In this example, that could be the HoH’s wife, or his mother, or sister, depending on the configuration of the household. The data collection tools will include fields to record exactly how each respondent was selected. 76 According to the FFS Annual Report, 2018, the total number of Cash for Work beneficiaries was 26,505 of which contact information was available for 8,023 with UNDP. 77 The figures in this table are taken from FFS Annual Report, 2018. 107 Type of Intervention No. of Beneficiaries Cash for work 26,505 Small grant for female headed households 4,858 Small business grants for women 3,481 Total 34,844 3.5.1 Sample Size Keeping in view the disproportional spread over geographical area, a stratified random sampling technique was originally proposed to be used to calculate the sample size, based on the following assumptions: Table 8: Assumptions for Sample Size Calculation Margin of error 5% Confidence interval 95% Proportion (est.) 50% Precision 5% Statistical Power 80% Beneficiaries population 8,023 Design effect 2 Refusal rate 15% N (min. sample size) 856 Based on the above assumptions, a total of 730 beneficiaries were to be selected for surveying. 3.5.2 Selection method of respondents A systematic random sampling technique will be used to select beneficiaries.78 UNDP will provide a coded list of beneficiaries for each activity along with their other details such as gender, age, year of enrolment, location, etc., without divulging personal identifying information. After selection of beneficiaries, the list of proposed beneficiaries will be shared with UNDP. Prior to commencing the 78 The sample size is over estimated by 15% to cover for refusals or missing respondents. A 20% additional contingency sample of beneficiaries has been set aside for replacement of refusals. 108 survey, UNDP will contact the selected respondents to obtain their consent for participation. Once the respondents agree to participate, enumerators will conduct the interviews in the field. 3.6 On-site Observations To the extent possible, the Evaluation Team will make site visits to assess the public works rehabilitation and light infrastructure activities implemented by FFS. Given the nature of FFS interventions79, emphasis will be given to water, sanitation, and electricity facilities in areas sampled in the PBS. Based on the security situation, the Evaluation Team’s subject matter experts will travel to the sites for technical observation of the operation of the facilities. These sites will be selected to align with locations selected for the PBS. Facility sites located within the cells selected for PBS data collection will be included. Site visit checklists will be used to assess the quality of public services provided and the role of infrastructure in the delivery of quality basic services in response to evaluation question #2a. A total of 24 facility-infrastructure sites will be observed, eight in each province representing 20% of the selected PBS sites. The selected sites will include facilities for electricity (generators and substations), sewerage and water treatment. The field teams visiting the facilities will capture GPS-tagged photos of each facility visited. 3.7 Capacity Assessment80 of Local Authorities The evaluation will adapt the USAID OCA approach81 to assess capacity of provincial directorates in budget, revenue collection, management and administration82, and technical capacity for operations and maintenance to answer EQ 4. The Evaluation Team will conduct rapid capacity assessments of provincial directorates of Water, Sewer Services, Municipality, Health, Education and Electricity in Anbar, Ninewa, and Salah al-Din. In addition to assessing capacity in the core governance functions of strategic planning, budgeting, procurement, monitoring and evaluation, information and communication technology, and human resource management, the team will assess technical capacity of local authorities to deliver services in line with their core mandate. The assessments will be conducted through facilitated discussions with atleast one senior representative from each of the seven core governance and technical departments. In total, 18 facilitated discussions (six in each province) will be conducted with-provincial directorates in the three provinces. 3.8 Pretesting of Data Collection Tools Both qualitative and quantitivative data collection tools will be pretested prior to data collection. The tools will be adjusted based on feedback from the enumerators. The final tools will be shared with USAID for approval before data collection is commenced. 4.0 Data Analysis Methods 79 FFS primarily supports service delivery infrastructure. 80 A full scale organizational capacity assessment is beyond the scope of this evaluation and this tool is not intended to capture that. Typically, a full scale OCA is carried out over several days and includes a facilitated workshop and developing an action plan with the intent that reassessments will be carried out over time. 81 See https://usaidlearninglab.org/library/organizational-capacity-assessment 82 Includes planning, human resource management, procurement, monitoring and evaluation, and information and communitaion technology 109 The evaluation’s activities will be conducted in accordance with the 2011 USAID Evaluation Policy83 to ensure that reliable data will be produced to support evidence-based findings, and will be a sound basis for analysis that will lead to conclusions and recommendations that are both useful for assessing the progress to date and capable of providing actionable recommendations relevant to FFS future programming. In line with the policy, the evaluation will combine mixed data collection methods to allow for the verification of findings through triangulation. The team understands triangulation as a method used during the evaluation process to check and establish validity of conclusions by analyzing the responses to Evaluation Questions from multiple perspectives, such as when conducting the desk review, interviews, FGDs, etc. Triangulation of data will enable the Evaluation Team to ensure validity of conclusions, i.e. that the findings are true (and accurately reflecting the situation) and certain (supported by the evidence). While not a data collection method, analytical triangulation approaches will be employed to verify and validate the findings from different methods, data sources and/or team members, and to identify correlations between findings to determine programmatic impact. These triangulation approaches will include: Methodological triangulation – Up to seven data collection methods, but no less than two for data collection will be used to support each of the Evaluation Questions. It should be noted that the document review will be extensive and include primary and secondary sources. Data source triangulation – The team will collect data about FFS through its own primary and secondary sources, but also from snow-balled KIIs and other relevant sources. Investigator triangulation – The team had several core members and enumerators from a local research partner – IIACSS – and so there will be complimentary experiences and skill-sets. This will allow for analysis of the data from different – richer - perspectives. 4.1 Quantitative Data Analysis The Evaluation Team will use STATA or SPSS to analyze quantitative data collected via PBS and the Livelihoods survey. After beneficiary community survey data are checked for accuracy and completeness, the data will be analyzed using descriptive statistics, including cross-tabulations and frequencies.84 Cross￾tabulation of data will allow for identifying relationships among multiple variables of interest85. Quantitative data will be triangulated with qualitative data, to lend context to findings and identify any inconsistencies in responses that may need to be addressed. When possible, appropriate, and applicable, all data will be analyzed by gender, age, religion, ethnicity and displacement status (whether IDP or not). Additionally, capacity assessments scores for each functional area will be derived from the average scores of the specific capacity elements selected for assessment. Comparisons will be drawn across directorates and provinces, and core functions. 83 The assessment meets or exceeds the USAID 2011 Evaluation Policy, USAID’s Performance Monitoring & Evaluation TIPS: Data Quality Standards, relevant chapters of the USAID Automated Directive System (ADS), the performance standards outlined in the RFTOP Task Order SOW, as well as with the Federal Policy for the Protection of Human Subjects or "Common Rule" [ADS Chapter 200 - http://www.usaid.gov/policy/ads/200/200mbe.pdf]. The IBTCI team is also apprised of the July 2012 supplement to ADS 203 on current good practice in preparing evaluation reports and the main deliverables expected in evaluation reports. 84 If secondary time series data are available, trend analysis for the period covering implementation of FFS will also be conducted. 85 The Evaluation Team may also run models on factors that determine return when advised by USAID. 110 4.2 Qualitative Data Analysis Qualitative data in this evaluation consist of data collected through Key Informant Interviews, Focus Group Discussions and Onsite Observations. Qualitative data will be analyzed by systematically looking for themes and patterns. The Evaluation Team will employ appropriate analytical methods to analyze the qualitative data that are collected. For the Key Informant Interviews and Focus Group Discussions the Evaluation Team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. The team will then code the data using qualitative data analysis software such as Atlas.ti to identify themes and build a narrative that responds to the evaluation questions. When possible, the team will conduct comparative analysis across various groups of stakeholders to ensure triangulation of data sources. Qualitative data analysis will help in contextualizing the evaluation framework, considering plausible explanations for correlations and triangulating with quantitative data. 5.0 Limitations to the Methodology Lack of current population data at the lowest geographical level poses a significant challenge for this survey. There have also been significant population and migration shifts within the FFS beneficiary intervention areas. We anticipate that at least some of the estimates from Landscan™ that are used for cell selection will be found to be out-of-date, because of population movements. The Evaluation Team will seek to address this by assessing population movements in each of the selected cells. The process of approximating the service areas of individual facilities may result in the sampling of people who do not have ready access to the facility. The Evaluation Team will consider an analytic approach that maps accessibility reported by respondents, which will contribute both to post-hoc validation of the sampling approach, but also to potentially useful information for UNDP with respect to the reach of supported facilities, in particular with respect to different types of intervention. Any given grid cell in the ten selected intervention areas may be served by one or more facilities, including facilities of multiple types. On one hand, selecting respondents who are only served by a single type of facility may lower the survey’s statistical power, because only a subset of responses will apply to sector-specific questions. On the other hand, prioritizing cells that are served by multiple types of facilities to guarantee completeness of responses would potentially bias selection toward respondents with more favorable perceptions. The stratification of cells by type and saturation will help ensure that respondents with multiple experiences of improved services are included in the survey. Once cells have been coded by facility type, the Evaluation Team will consider oversampling those served by only a single facility type. This oversampling will be corrected in aggregate reporting, and will help address the loss of power due to missing data in these responses. In absence of baseline data, the evaluation will primary rely on retrospective perceptions of respondents, which can potentially lead to recall bias. Recall bias occurs when respondents do not accurately remember previous events or experiences. To address this, data collection tools will be shared with and vetted by USAID staff, translated into local languages, and piloted in the field prior to fieldwork. All data will be collected in the respondents’ preferred language. Data collection teams will be trained in administering each instrument. And finally data will be collected by local enumerators conversant in the local languages to avoid errors from mistranslation. Because much of the qualitative data collection for this evaluation relies on purposive sampling, meaning that respondents are selected for their particular roles, responsibilities, and knowledge, it will not produce 111 statistically representative samples. As a result, although these respondents typically represent a variety of views on a topic, they may not represent all relevant views, and selection bias can emerge. Respondents for the livelihood survey will include only those beneficiaries for whom phone contact information is available, which will create a selection bias as only individuals with access to phone and phone networks will be selected. In absence of data for the two livelihood activities i.e., small grants for businesses and grants for women headed households, the evaluation will only assess cash for work activity under the livelihood support. Likewise, the data for cash for work is available for 8,023 beneficiaries out of over 22,000 beneficiaries this will exclude views of those not included in the sample and may lead to skewed results. Since quantitative and qualitative data will be collected concurrently due to time constraints. This presents a limitation to the evaluation as it does not allow the team to use qualitative data to fill in gaps emerging after analysis of the quantitative data. To the extent possible, the evaluation will address this issue by following up with key informants should any need for further clarification arises. 6.0 Dissemination Plan PMSP will share the draft and final versions of the evaluation report with USAID. Within three months after the report is approved by USAID, the report will be lodged on the Development Experience Clearinghouse (DEC). The primary audience of the report will include USAID/Iraq, Government of Iraq, UNDP, other donor organizations and the development community at large, 112 Annex I: Getting To Answers Evaluation Questions Type of Answer/Evidence Needed (Check one or more, as appropriate) Methods for Data Collection86, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey87 Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, CrossTabulations, Content Analysis Data Source(s) Method EQ1: How have FFS activities affected access to services in target communities? Evaluators will assess: Changes to service availability in target communities resulting from FFS activities, across all FFS target sectors (including, but not limited to, health, education, water, electricity, and other municipal services). Whether access to services is equitable across vulnerable groups, including, but not limited to, women, IDPs, persons with disabilities, religious and ethnic minorities, politically Yes/No List of Documents: Existing program data Activity monthly, quarterly, and annual reporting and activity documentation (such as donor updates from Steering Committee meetings, annual work-plans, etc.) Literature review Quantitative Method: Population Based Survey Qualitative Methods: Key Informant Interview Focus Group Discussion Stratified random sampling for Population Based Survey (PBS) to select a random sample of 2712 households in ten intervention areas across the three provinces. Purposive sampling for KIIs. Interviews will be conducted with relevant USAID/Iraq officials and key FFS staff. Additionally, interviews with relevant officials of FFS sectors (health, education, water, electricity, and other municipal services) in the three provinces will be conducted. Quantitative data analysis will include frequency distribution and cross tabulations. To the extent possible data will be disaggregated by gender, area (intervention area/provinces), ethnicity, religion etc. Given absence of any baseline data for FFS, the evaluation will rely on respondent’s retrospective perceptions. For qualitative data (key informant interviews and focus group discussions) the Evaluation Team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and Yes Description Yes Comparison88 Yes Explanation89 86 Data will be collected in there provinces namely Anbar, Ninewa and Salah ad Din. 87 Data from evaluations are a deliverable and methods should indicated how data will be captured, i.e., for focus groups USAID requires a transcript. 88 Comparison – before and after based on retrospective perception. 89 Explanation – for questions that ask “why” or about the attribution of an effect to a specific intervention (causality) 113 affiliated groups, and youth and children. What significant barriers, if any, prevent members of target communities from accessing services? Purposive (and/or convenience sampling) for selection of FGD participants. One male and one female FGD with returnees in each province (six total); one male and female FGD with religious and ethnic minorities among the returnees; one male and female FGD with IDPs; and one male and female FGD with religious and ethnic minorities among the IDPs. reporting frequencies and other quantitative summaries of responses when possible and appropriate. Evaluation Questions Type of Answer/Evidence Needed (Check one or more, as appropriate) Methods for Data Collection19, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey20 Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, CrossTabulations, Content Analysis Data Source(s) Method EQ2. How have FFS activities affected the quality of services in target communities? Yes/No List of Documents: Existing program data Literature review Stratified random sampling for Population Based Survey (PBS) to select a random sample of 2712 households in ten Quantitative data analysis will include frequency distribution and cross tabulations. To the extent possible data will be Yes Description Yes Comparison 114 Evaluators will assess: Sector-specific parameters of quality of services targeted by FFS (including, but not limited to, accessibility / availability in education health, water, electricity, and other municipal services). Note changes in performance indicators; application of performance standards (e.g., potable water with a free chlorine level of 1 milligrams per liter); application of performance plan (e.g., periodic testing of chlorine in potable water); hours per day of availability of water/electric services; availability of services in FFS supported hospitals and health clinics; and completion and use by students of rehabilitated school buildings. Perceptions of the quality of services among members of target communities. Yes Explanation Activity monthly, quarterly, and annual reporting and activity documentation (such as donor updates from Steering Committee meetings, annual work-plans, etc.) Implementing partner M&E data Quantitative Method: Population Based Survey Qualitative Methods Key Informant Interviews intervention areas across the three provinces. Purposive sampling for KIIs. Interviews will be conducted with relevant USAID/Iraq officials and key FFS staff. Additionally, interviews with relevant officials of FFS sectors (health, education, water, electricity, and other municipal services) in the three provinces will be conducted. Purposive (and/or convenience sampling) for selection of FGD participants. One male and one female FGD with returnees in each province (six total); one male and female FGD with religious and ethnic minorities among the returnees; one male and female FGD with IDPs; and one male and female FGD with religious and ethnic disaggregated by gender, area (intervention area/provinces), ethnicity, religion etc. In absence of any baseline data for FFS, the evaluation will rely on respondent’s retrospective perceptions. For qualitative data (key informant interviews and focus group discussions) the Evaluation Team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. Field monitoring reports on FFS collected by IPs or third-party monitors (USAID-funded primarily and, if available, reports from UNDP and other donor third-party monitors) Records and documentation from local administrations and public service facilities, as available Key informant interviews (KIIs): Government counterparts at the Focus Group Discussion Site Visits 115 Perceptions of the effectiveness of government administration of services among members of target communities. national, provincial, and local levels; Senior implementing partner staff and other key technical staff; and Representatives from civil society and community representatives. Focus Group Discussion Beneficiaries: FFS beneficiaries in three provinces. FFS Facilities: Water, Sewerage and electricity. At least two facilities will be selected for onsite observation to include water, electricity, and sewerage facilities in each of the three provinces. Evaluation Questions Type of Answer/Evidence Needed (Check one or more, as appropriate) Methods for Data Collection19, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey20 Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, CrossTabulations, Content Analysis Data Source(s) Method 116 EQ3. How have FFS activities affected IDPs in target communities? Evaluators will assess: To what extent FFS activities have influenced IDPs’ decision to return to their communities or influenced others to remain in their communities relative to other factors. In what ways have FFS activities affected IDPs’ lives since their return to their communities? Yes/No Key informant interviews (KIIs): Government counterparts at the national, provincial, and local levels; Senior implementing partner staff and other key technical staff; and Representatives from civil society and community representatives. Focus Group Discussion Beneficiaries: FFS beneficiaries in three provinces having access to FFS sponsored health, education, water, Literature review Quantitative Methods: Population Based Survey Livelihoods survey Qualitative Methods: Key Informant Interview Focus Group Discussion Stratified random sampling for Population Based Survey (PBS) to select a random sample of 2712 households in ten intervention areas across the three provinces. Systematic sampling technique will be used to calculate the sample size for the livelihoods survey. Based on the available beneficiary information, a total 856 beneficiaries will be selected for survey. Purposive sampling for KIIs. Interviews will be conducted with relevant USAID/Iraq officials and key FFS staff. Additionally, interviews with relevant officials of FFS sectors (health, education, water, Quantitative data analysis will include frequency distribution and cross tabulations. To the extent possible data will be disaggregated by gender, area (intervention area/provinces), ethnicity, religion etc. In absence of any baseline data for FFS, the evaluation will rely on respondent’s retrospective perceptions. For qualitative data (key informant interviews and focus group discussions) the Evaluation Team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and Yes Description Yes Comparison Yes Explanation 117 electricity, and other municipal services. Key informant interviews (KIIs): Government counterparts at the national, provincial, and local levels; Senior implementing partner staff and other key technical staff; and Representatives from civil society and community representatives. Focus Group Discussion Beneficiaries: FFS beneficiaries in three provinces having access to FFS sponsored health, education, water, electricity, and other municipal services. FFS beneficiaries who participated in cash for work program and those who received cash grants for small businesses. electricity, and other municipal services) in the three provinces will be conducted. Purposive (and/or convenience sampling) for selection of FGD participants. One male and one female FGD with returnees in each province (six total); one male and female FGD with religious and ethnic minorities among the returnees; one male and female FGD with IDPs; and one male and female FGD with religious and ethnic. reporting frequencies and other quantitative summaries of responses when possible and appropriate. 118 Evaluation Questions Type of Answer/Evidence Needed (Check one or more, as appropriate) Methods for Data Collection19, e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey20 Sampling or Selection Approach, (if one is needed) Data Analysis Methods, e.g., Frequency Distributions, Trend Analysis, CrossTabulations, Content Data Source(s) Method Analysis EQ4. What factors are contributing to or impeding local authorities’ capacity to administer services targeted by FFS? Evaluators will assess: Sources and availability of operations and maintenance budgets for rehabilitated services. Capacity to collect revenue. Organizational capacity of local administrative offices. Technical and management capacity of staff. Yes/No Key informant interviews (KIIs): Government counterparts at the national, provincial, and local levels; Senior implementing partner staff and other key technical staff; and Stabilization advisors, area coordinators, field engineers, national engineers etc. (Embedded advisors through FFS capacity building support). FFS Facilities: Health, education, water, electricity, and other municipal services. Qualitative Method: Key Informant Interview Capacity Assessment Purposive sampling for KIIs. Interviews will be conducted with relevant USAID/Iraq officials and key FFS staff. Additionally, interviews with relevant government officials of FFS sectors (health, education, water, electricity, and other municipal services) will be conducted in the three provinces. For capacity assessment, respondents will be selected from the relevant departments within the directorates of electricity, education, health, water and sanitation, and municipality in three provinces. For qualitative data (key informant interviews and focus group discussions) the Evaluation Team will employ a structured approach to analysis by identifying key themes, coding responses according to these themes, and reporting frequencies and other quantitative summaries of responses when possible and appropriate. The evaluation will conduct capacity assessments of the selected directorates. The evaluation does not intend to conduct an organizational assessment like an audit firm and the analysis will be limited to answering the evaluation questions. Yes Description No Comparison Yes Explanation 119 Annex ii: Roles and Responsibilities Evaluation Analysis and Reporting Team Members: Robert Grossman-Vermaas, Team Leader, is a conflict, conflict assessment, stabilization, operations, and monitoring & evaluation specialist with over 15 years of experience working in conflict, post-conflict, fragile, and transitioning environments for clients including USAID, DoS, DoD, DfID, SIDA, and the US intelligence community. He has led large-scale conflict assessment, resilience, CVE, and stabilization/transition assessment programs, including acting as Chief of Party or Team Leader on USAID programs (incl. OTI, OFDA, FFP); leading a large USG 1207-funded evaluation of a stabilization program in Haiti; serving as Technical Director for the Measuring Progress in Conflict Environments (MPICE) program, an innovative interagency stabilization program sponsored by DoS, USAID and DoD (and operating in Iraq, Afghanistan, Sudan); leading program evaluations for experimental USG counterterrorism programs; leading conflict assessments, evaluations and verification activities in Syria, Afghanistan, Colombia, Somalia, Iraq, Sudan, Ethiopia, Yemen; and, acting as Senior Technical Advisor for monitoring programs in Syria, Somalia, Yemen. He has extensive experience designing strategies and accompanying metrics, indicators, results frameworks, logframes and M&E/outcomes/results-based plans for DoD, DoS, USAID, DfID, and in operationalizing these frameworks to identify outcome, impact and actionable program recommendations. He has published over two dozen articles on stability operations, resilience in East Africa, M&E, conflict assessment, inter-agency planning, civil-military relations, and post-conflict stabilization and reconstruction. He worked long- and short-term in several conflict affected environments. He has recently led TPM activities performance evaluations in East Africa, Somalia, DRC, CAR, Syria, and Iraq. The TL will take the lead in preparing, delivering, and disseminating all evaluation technical deliverables to USAID/Iraq. Elizabeth Cunningham, Subject Matter Specialist, is an experienced development professional with a decade of experience implementing, monitoring, and evaluating stabilization, reconstruction, and governance programming in fragile and conflict-affected environments. She has led or overseen the design and revision of various assessment, monitoring, and evaluation instruments and protocols as well as qualitative and quantitative data analysis for USAID Missions, including USAID/Iraq, and other donors. As the former Project Director and Acting Chief of Party for PMSP, she brings to the evaluation a deep understanding of USAID/Iraq programming and objectives, and in particular those of FFS. Beth will support the analysis of the qualitative data and support the Team Leader in synthesizing all data to respond to the evaluation questions. She will draw upon her expertise in stabilization programming and familiarity with FFS programming to ensure that findings, conclusions, and recommendations are appropriately contextualized and of maximum use and value to USAID/Iraq. She will be a lead analyst and co-writer of the report. Samuel Malmberg, Senior Data Analyst, has seven years of experience in international development. Currently, as a Program Manager at IBTCI (2017-present), Mr. Malmberg manages quality control of technical deliverables, operations, FAR/AIDAR compliance, budgets, and procurement for a portfolio of six (6) multi-million dollar programs totaling $109.5 M across the Crisis, Conflict, & Governance (CCG) practice. From 2016 through 2018, he supported the USAID/OFDA Syria Humanitarian Monitoring Program (SHMP). In this role, Mr. Malmberg participated in every step of the monthly monitoring report (MMR) cycle, including the procurement of third party monitors (TPMs), development of site visit tools, and review of TPM reports and data collected through KIIs, FGDs, and HH surveys. He also contributed to, and reviewed, individual site visit reports and monthly monitoring reports for quality assurance. as well Recently - June-October 2019 - he served as the Data Manager for an Assessment of the Conflict and Early Warning CEWARN mechanism in East Africa, where he collected and analyzed qualitative data and contributed to the final assessment report. Mr. Malmberg was also involved in the USAID/OFDA Ebola Response Evaluation from 2016-2017. During this activity, he oversaw the quality 120 and accuracy of data collection, data analysis, and technical deliverables including data collection tools, evaluation methodology, and final reports. He will support the quantitative and qualitative analysis and visualization for the FFS evaluation. Wesley Thompson, Data Analyst, serves as a Senior Program Associate at IBTCI. He supports multiple M&E MEL Platforms for USAID, including PMSP/Iraq, MECC/DRC, and SPSS/East Africa. He has also provided support to M&E awards for Pakistan and Syria. He has contributed to multiple evaluations for USAID, including Evaluation Reports for USAID/Pakistan, USAID/Iraq and USAID/East Africa/Kenya. Wes will support the literature analysis and qualitative analysis for the FFS evaluation. Reach-back Team and Data Collection Leads: Ghazanfar Hoti, former Team Leader (TL) of the FFS evaluation. Willie Kachaka, Evaluation Methodologist, will take the lead in designing the sampling methodology and will work closely with the local data collection partner to ensure rigor in the evaluation design and tools. He will also work with the Subject Matter Experts to analyze quantitative data from the surveys and qualitative data from FGDs and KIIs. Sahib Haq, Livelihood Expert: The Livelihoods Specialist will take the lead in designing the livelihoods survey and will work closely with the local data collection partner in overseeing its administration. He will analyze findings related to evaluation question #3 and draft the relevant sections of the evaluation report. Other Subject Matter Experts (SMEs): The Evaluation Team will also include Iraqi consultants with expertise in water/sanitation, electrification, city public service delivery as well as in organizational capacity. The SMEs will provide technical inputs to the evaluation design, analysis plan, and data collection tools to ensure that the evaluation appropriately assesses the quality of service delivery in these sectors. The SMEs will also participate in data collection and corresponding analysis of findings related to the quality of services. Management of the evaluation will be led by the Chief of Party (COP) and the Senior Monitoring and Evaluation Specialist. The evaluation will be supported by Mr. Mohammed Talal, the PMSP Evaluation Coordinator. Mr. Talal, an Iraqi national with native Arabic proficiency will coordinate the team’s activities and interaction with the subcontractor, translators, and security provider. ANNEX iii: KEY INFORMANT INTERVIEW FORM KEY INFORMANT INTERVIEW FORM- GOVERNMENT OFFICIALS (ELECTRICITY) TARGET GROUP: DIRECTORATE OF ELECTRICITY (DIRECTOR OR REPRESENTATIVE) Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 121 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS activities with regard to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine how FFS activities influenced their decision to return and/or remain in their communities. The findings will assist decision makers to understand outcomes of FFS activities and may be used to inform future programming. The interview will last approximately 50 minutes to an hour, and your responses will be included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 3 Consent Do I have your consent to proceed with the interview? YES NO 122 SECTION 1: ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is the current status of electricity supply in your area? What is the source of electricity supplied in the area (e.g. generator, hydro, other etc.? What is the distribution capacity of electricity? Is the supply adequate to meet the demand? If not what are underlying reasons for the undersupply? How has FFS contributed to the supply of electricity in your area? Have there been any unintended outcomes, positive or negative? Please explain. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? SECTION 2: QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Can you please tell us about the factors affecting quality of electricity supply in your area? In terms of reliability i.e., the number of electricity hours/day? If not reliable, what are the reasons? How is this being addressed? Whether voltage is stable or not? If not what are the underlying reasons? How is this being addressed? SECTION 3: CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) Do you have adequate technical staff (number of engineers/technicians) in place to respond to connections, power failures and routine maintenance of equipment? Do they have the appropriate skill set and qualifications to do their current job? Please explain. Do you have adequate administrative staff (number of procurement, human resources and finance) in place to perform administrative responsibilities of your office? Do they have the appropriate skill set and qualifications to do their current job? Please explain. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? If yes, does your institution have adequate capacity in terms of: How revenue is collected Number of staff to collect revenue Enforcement mechanisms to collect revenue (i.e. controlling Illegal connections) What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? How are requests for funding made? Does that involve input from your office? How much of the allocated budget is received and utilized for O&M? 123 Are you generating enough revenue to meet O&M costs (as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 124 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 3 Consent Do I have your consent to proceed with the interview? YES NO 125 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is the current status of potable water supply (in terms of number and type of water facilities) in your area? Is the portable water supply adequate to meet the demand? If not what are underlying reasons for the undersupply? What steps has your institution taken to address the potable water supply gaps if any? Please explain. How has FFS contributed to the development of infrastructure for potable water supply in your area? What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Can you please tell us about the factors affecting reliability of potable water supply (in terms of number of hours/day, quantity/pressure) in your service area? Have there been significant changes in access to potable water as a result of the FFS intervention? Have there been any unintended outcomes, positive or negative? Please explain. In your opinion what are some of the barriers to accessing potable water in your area? Are there initiatives in place to address these barriers? How does the directorate ensure equal access between men and women, among different ethnic groups, for IDPs, etc.? Please explain. SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Have there been any significant changes to quality of water infrastructure due to FFS interventions in your area since FFS interventions? Please explain. To what extent have the FFS activities contributed to the quality of water and infrastructure in this area since FFS interventions? Are there any unintended outcomes? Please give examples. How have these changes in water infrastructure affected quality of water in your area? Have these changes been consistent across different groups (e.g. minorities, IDPs, and other vulnerable groups)? How have these changes in quality affected IDP returns to their communities in this area? In your opinion does the potable water supplied meet acceptable quality standards? Please provide examples in terms of parameters of quality of water supplied (e.g. number of water treatment plants, chlorine concentration level, turbidity, hardness, (pH), total dissolved salts (tds)) and other parameters and the recommended targets). Who is responsible for testing water safety and how often is the testing done? What are test results showing today? Can we get copies? SECTION 3. CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) Do you have adequate technical and support staff (e.g., in terms of number of engineers/technicians) in place to respond to distribution losses and routine maintenance of equipment? Do they have the appropriate skill set and qualification to do their current jobs well? Please explain. Apart from budget allocations from the central government, does the directorate have any other avenues for generating revenue? What are these sources? 126 If yes, does your institution have adequate capacity in terms of: How revenue is collected Number of staff to collect revenue Enforcement mechanisms to collect revenue (i.e. controlling Illegal connections) What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? How are requests for funding made? Does that involve input from your office? How much of the allocated budget is received and utilized for O&M? Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 127 KEY INFORMANT INTERVIEW-GOVERNMENT OFFICIALS (SANITATION) TARGET GROUP: DIRECTOROF SANITATION OR REPRESENTATIVE Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 50 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 128 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 3 Consent Do I have your consent to proceed with the interview? YES NO 129 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is the current status of sewerage infrastructure in your area? Is the sewerage infrastructure adequate to meet the demand? If not what are underlying reasons for the undersupply? What steps has your institution taken to address the sewerage infrastructure gaps if any? Please explain. How has FFS contributed to the development of sewerage infrastructure in your area? Have there been any unintended outcomes, positive or negative? Please explain. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? In your opinion what are some of the barriers to accessing sewage infrastructure in your area? Are there initiatives in place to address these barriers? If sewerage system is not available in this area, what services are available for waste water disposal and treatment? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Have there been any significant changes in quality of sewage infrastructure and equipment due to FFS interventions in your area since FFS interventions? Please explain. To what extent has the FFS activities contributed to the quality of sewage infrastructure and equipment in this area since FFS interventions? Please give examples. Have changes in quality of sewerage infrastructure and equipment affected the welfare of surrounding communities? If so, how? Have these effects been the same along ethnic / tribal lines? Have these changes in quality influenced IDP decisions to return to their communities in this area? SECTION 3. CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) Do you have adequate technical and support staff (e.g., in terms of number of engineers/technicians) in place to respond to new connections, blockages of pipes and routine maintenance of equipment? Do they have the appropriate skill set and qualification to do their current jobs well? Please explain. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? If yes, does your institution have adequate capacity in terms of: How revenue is collected Number of staff to collect revenue Enforcement mechanisms to collect revenue (i.e. controlling Illegal connections) What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? 130 Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 131 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is the current status of education infrastructure (i.e. in terms of number and type of schools; number of classrooms per schools; availability 132 of basic facilities in schools etc.) in your area? Is the available infrastructure adequate to meet the demand for education services in this area? If not, what are underlying reasons? If there are any infrastructure gaps (such as schools that have not been rehabilitated), what are underlying reasons? And what steps has your institution taken to address these, if any? Please explain. How has FFS contributed to the development of education infrastructure in your area? Have there been any unintended outcomes, positive or negative? Please Explain. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. In your opinion what are some of the barriers to going to school in your area? Are there any government programs or non-government programs (i.e. run by NGOs or religious groups) in place to address these barriers? How does the directorate ensure equal access between boys and girls, among different ethnic groups, for IDPs, etc.? Please explain. What policies does the Ministry/ Directorate of Education have in place to ensure inclusive education? How would you describe the trends of enrollment, retention and completion rates for boys and girls in primary and secondary schools? (to request provincial official school statistics from 2015 to 2018) How would you describe cultural norms about girls going to school? Are there separate male/female bathrooms/sanitary facilities at schools? Whether commuting/walking to school is considered safe? Do you think access to new/improved school influenced the decisions of many IDPs to return to their communities in this area? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Have there been any significant changes to quality of school environment (in terms of classroom space, facilities and equipment, basic amenities such as water and electricity, separate bathrooms for boys and girls, safety at school etc.) in your area since FFS interventions? Please explain? To what extent have the FFS activities contributed to the quality of school buildings in this area since FFS interventions? Please give examples. SECTION 3: CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) Do you have adequate human resources (e.g., in terms of number of teachers) in place in the rehabilitated schools? Do they have the appropriate skill set and qualification to do their current jobs well? At what capacity (i.e., full/half capacity, or underutilized etc.) are rehabilitated schools/ facilities working? Please explain. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? How are budget shortfalls addressed? Are there any family or teacher contributions? 133 If yes, does your institution have adequate capacity in terms of: How revenue is collected Number of staff to collect revenue Enforcement mechanisms to collect revenue What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? How are requests for funding made? Does that involve input from your office? How much of the allocated budget is received and utilized for O&M? Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 134 KEY INFORMANT INTERVIEW FORM- GOVERNMENT OFFICIALS (HEALTH) TARGET GROUP: DIRECTOR OF HEALTH OR REPRESENTATIVE Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 50 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 135 3 Consent Do I have your consent to proceed with the interview? YES NO 136 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is the current status of health infrastructure (i.e., in terms of number and type of health facilities) and equipment in your area? Is the available health infrastructure adequate to serve the population? If not, what are underlying reasons and what steps has your institution taken to address these gaps? Please explain. How has FFS contributed to provision of health infrastructure and equipment in your area? What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. Have the changes in provision of health infrastructure and equipment affected delivery of health services (e.g. are previously unavailable services now available). Are more people able to access those services? Have there been any unintended outcomes, positive or negative? Please explain. What are the primary barriers preventing residents from accessing health services? Are there initiatives in place to address these barriers? How does the directorate ensure equal access between men and women, among different ethnic groups, for IDPs (living in the targeted areas), etc.? Please explain. Are there any differences in the utilization of health services between men and women? Are there any differences in utilization of health services among different identity groups (e.g., IDPs, minorities, returnees, disabled persons etc.)? Are there any official statistics on utilization to share? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Have there been any significant changes to quality of health infrastructure and equipment due to FFS interventions in your area since FFS interventions? Please explain. To what extent has the FFS activities contributed to changes in improved quality of health infrastructure and equipment in this area since FFS interventions? Please give examples? Have these changes in health infrastructure and equipment affected quality of health care in your area? If so, have these changes been consistent across different groups (e.g. minorities, IDPs, women, and other vulnerable groups? How have these changes in quality affected IDP returns to their communities in this area? SECTION 3: CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) Do you have adequate number of staff in place to run and maintain the rehabilitated health facilities? Do they have the appropriate skill set and qualification to do their current jobs well? At what capacity (i.e., full/half capacity, or underutilized etc.) are rehabilitated health facilities working? Please explain. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? (Examples include user fees, health insurance, etc.) If yes, does your institution have adequate capacity in terms of: 137 How revenue is collected Number of staff to collect revenue Enforcement mechanisms to collect revenue What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? How are requests for funding made? Does that involve input from your office? How much of the allocated budget is received and utilized for O&M? Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 138 KEY INFORMANT INTERVIEW-GOVERNMENT REPRESENTATIVES (MAYOR’S OFFICE/LOCAL COUNCIL) TARGET GROUP: MAYOR OR HEAD OF LOCAL COUNCIL Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 30 mins, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 139 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 3 Consent Do I have your consent to proceed with the interview? YES NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What are the major accomplishments of FFS in terms of encouraging IDPs to return to their place of origin in the newly liberated areas since FFS interventions? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. (FFS provided public infrastructure support in education, health, water, sanitation, electricity and other municipal services etc.) What have been some of the major challenges (political, economic, social, technological, environmental, legal, etc.) faced in implementing the FFS? How were these challenges addressed? Please explain. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. How much municipal government input was sought and utilized? What mechanisms were put in place to ensure equitable distribution of infrastructural projects for basic services among the IDPs and returnees? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Has there been any significant change in quality of service delivery (FFS priority sectors) since FFS interventions in the targeted areas? How did FFS contribute to these changes? Have these changes in quality affected IDP returns to their communities in this area? Please explain? SECTION 3. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EVALUATION QUESTION 3) What trends have you observed in terms of IDPs returning to their place of origin in the since FFS intervention? Please explain and provide examples. In your opinion in what ways has FFS infrastructure rehabilitation influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? In your opinion, which sectors (if any) supported through FFS infrastructure rehabilitation initiatives have been more successful in attracting IDPs to return than others and if so, why? (i.e., electricity, education health, water and sanitation)? Please explain. In your opinion in what ways have FFS livelihood activities influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? Please explain. 140 Are there any other ongoing or completed donor-funded programs that complement FFS activities in terms of stability of returns among IDPs? If yes, what are they? How are activities complemented? SECTION 4. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) How effective is the coordination between the UNDP and the mayor’s office/city council? Effectiveness of provincial government’s contribution and commitment? The level of engagement of civil society in design and implementation of services? Is there anything else you’d like to share that we haven’t asked in this interview? 141 KEY INFORMANT INTERVIEW FORM-DEVELOPMENT PARTNERS TARGET AUDIENCE: UNDP, USAID, OTHER DONORS Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 30 mins, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 142 3 Consent Do I have your consent to proceed with the interview? YES NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) What is your perception of the major accomplishments of FFS in terms of encouraging IDPs to return to their place of origin in the newly liberated areas since FFS interventions? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. What are some of the major challenges faced in implementing the FFS? How were these challenges addressed? Please explain. How supportive and cooperative was the GOI in the design, implementation, and monitoring of FFS interventions? How much provincial and municipal government input was sought and utilized? Please explain. What mechanisms were put in place to ensure equitable distribution of infrastructural projects for basic services among the IDPs and returnees? SECTION 2. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EVALUATION QUESTION 3) In your opinion in what ways has FFS infrastructure rehabilitation influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? In your opinion, which sectors (if any) supported through FFS infrastructure rehabilitation initiatives have been more successful in attracting IDPs to return than others and if so, why? (i.e., electricity, education health, water and sanitation)? Please explain. In your opinion in what ways have FFS livelihood activities influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? Please explain. What trends have you observed in terms of IDPs returning to their place of origin since FFS intervention? Please explain and provide examples. Are your aware of any other ongoing or completed donor-funded programs that complement FFS activities to ensure stability of returns among IDPs? If yes, what are they? How are activities coordinated? SECTION 3. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) How effective is the coordination between the implementing partner and government? Effectiveness of national/provincial government’s contribution and commitment? The level of engagement of civil society in design and implementation of services? In your view, have you observed any changes, positive or negative, in the capacity of the local authorities to sustainably operate and maintain the public infrastructure? Do they have the resources and training needed to operate and maintain the public infrastructure? Please explain. 143 Is there anything else you’d like to share that we haven’t asked in this interview? KEY INFORMANT INTERVIEW FORM TARGET AUDIENCE: CIVIL SOCIETY AND OTHER INTERNATIONAL NGOS Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex Male Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. 144 The interview will last approximately 30 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? YES NO 3 Consent Do I have your consent to proceed with the interview? YES NO 145 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) In your opinion what are some of the major accomplishments of FFS in terms of encouraging IDPs to return to their place of origin in the newly liberated areas since FFS intervention? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. Do they think project was appropriately designed and targeted? Was it complementary to other stabilization efforts underway? Was there duplication of any efforts? Was it well-managed? Was FFS team interested in outside input? Did they seek it? What have been some of the major challenges faced in implementing FFS? How were these challenges addressed? Do you think their efforts were satisfactory? Please explain. How supportive and cooperative was the GOI in the design, implementation, and monitoring of FFS interventions? How much CSO input was sought and utilized? Please explain. What mechanisms were put in place to ensure equitable access to infrastructure projects for basic services among the IDPs and returnees? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) Has there been any significant change in quality of public infrastructure (FFS priority sectors) in since FFS intervention in the targeted areas? How did FFS contribute to these changes? How have these changes in quality affected IDP returns to their communities in this area? Please explain? SECTION 3. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EQ3) What trends have you observed in terms of IDPs returning to their place of origin since FFS intervention? Please explain and provide examples. What other programs and interventions are there to complement the FFS activities and ensure stability of returns among IDPs? Are there interrelationships between those programs and FFS? SECTION 4. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EQ4) How effective is the coordination among the civil society organizations, implementing partners and the government? The level of engagement of civil society in design and implementation of services? In your opinion , have you observed any changes positive or negative related in the capacity of local authorities to sustainably operate and maintain the public infrastructure? Please explain. Is there anything else you’d like to share that we haven’t asked in this interview? 146 THANKS ANNEX iV: FOCUS GROUP DISCUSSION FORM FOCUS GROUP DISCUSSION FORM TARGET GROUP: IDPS Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. IDPs), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? YES NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity 147 Has there been any change in access to electricity for households in your community since FFS intervention. Please explain. (EQ1, 3) What is the difference for you between a day with electricity and a day without? How does it change your routine? How would availability of electricity affect your decision to return to place of origin? Would its absence factor into your decision to return? Education Has there been any change in access to schools for school-going children in your community in the since FFS intervention? Please explain. (EQ1, 3) Do all the children living in this community have the opportunity to attend school? What are the challenges? How is your children’s education affected if there is no school nearby? How would availability of schools in your place of origin affect your decision to return? Health Has there been any change in access to health services for your community in the since FFS intervention? Please explain. (EQ1, 3). What do you do when you or any member of your community gets sick? What do you do when the treatment you seek is not available in nearby hospital or other health facility? What are the main barriers faced in accessing health services? (For women) Do you need approval of your husband/father to access health services? Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? How would availability of health services in your place of origin your affect your decision to return? Water Supply Has there been any change in access to What do you do when potable water is not available in potable water for your community since nearby area? FFS intervention? Please explain. (EQ1, 3) How would availability of potable water in your place of origin your affect your decision to return? Questions Prompts Sanitation 148 Has there been any change in access to sewerage services in your community since FFS intervention? Please explain. (EQ1, 3) Are the sewerage services adequate for your community needs? Please explain. How would availability of sanitation services in your place or origin affect your decision to return? Local Authorities’ Ability to Manage Public Infrastructure What is your perception of local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? What would you suggest that the local authorities can do better to improve service delivery amongst IDPs? General Question In future, would you like stay in your current location or move to your place of origin? Please explain. FOCUS GROUP DISCUSSION FORM TARGET GROUP: RELIGIOUS AND ETHNIC MINORITIES (IDPs) Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION 149 Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. IDPs), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? YES NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basics services available in your area. Electricity Has there been any change in access to electricity for households in your community since FFS intervention? Please explain. (EQ1, 3) What is the difference for you between a day with electricity and a day without? How does it change your routine? How would availability of electricity affect your decision to return to your place of origin? Would its absence factor into your decision to return? Education 150 Has there been any change in access to schools for school-going children in your community since FFS intervention? Please explain. (EQ1, 3) Do all the children living in this community have the opportunity to attend school? What are the challenges? How is your children’s education affected if there is no school in nearby? How would availability of schools in your place of origin affect your decision to return? Health Has there been any change in access to What do you do when you or any member of your health services for your community since community gets sick? What do you do when the FFS intervention? Please explain. (EQ1, 3). treatment you seek is not available in a nearby hospital or other health facility? What are the main barriers faced in accessing health services? (For women) Do you need approval of your husband/father to access health services? Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? How would availability of health services in your place of origin affect your decision to return? Water Supply Has there been any change in access to What do you do when potable water is not available in potable water for your community since your area? FFS intervention? Please explain. (EQ1, 3) How would availability of potable water in your place of Questions Prompts origin affect your decision to return? Sanitation Has there been any change in access to sewerage services for your community since FFS intervention? Please explain. (EQ1, 3) Are the sewerage services adequate for your community needs? Please explain. How would availability of sanitation services in your place or origin affect your decision to return? Local Authorities’ Ability to Manage Public Infrastructure 151 What is your perception of the local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? What would you suggest that the local authorities can do better to improve service delivery amongst IDPs? General Question In future, would you like stay in your current location or move to your place of origin? Please explain. Would you feel safe going back to the communities? Please explain. FOCUS GROUP DISCUSSION FORM TARGET GROUP: RETURNEES Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. Returnees), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? YES NO [Allow only those who have verbally consented to proceed with the focus group discussions.] 152 Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity Has there been any change in access to What is the difference for you between a day with electricity for households in your electricity and a day without? How does it change your community since/due to FFS routine? interventions? Please explain. (EQ1, 3) What is the average of total electricity provision per day? Is it constant or vary depend on the season? Tell us about the process you had to go through to get your house connected to the electricity grid? Are there any barriers preventing households from this community from accessing electricity? Please explain. How did availability of electricity affect your decision to return? Would its absence factor into your decision to leave? Please explain. Do you depend on other sources of electricity providers, if so, what are they and for how many hours? Education Has there been any change in access to schools for school-going children in your community since FFS interventions? Please explain. (EQ1, 3) Do all the children living in this community have the opportunity to attend school? How is your children’s education affected if there is no school in nearby? What are the challenges? How did availability of schools affect your decision to return? Would the absence of schools in your current location factor into future decision to leave? Health 153 Has there been any change in access to health services for your community since FFS interventions? Please explain. (EQ1, 3). What do you do when you or any member of your community gets sick? What do you do when the treatment you seek is not available in nearby hospital or other health facility? (For women) Do you need approval of your husband/father to access health services? Have you observed any changes in the type of services offered in the hospitals or other health facilities since you returned to your area? How did availability of health services affect your decision to return? Would its absence factor into future decisions Questions Prompts to leave? 154 Water Supply Has there been any change in access to potable water in your community since FFS interventions? Please explain. (EQ1, 3) What do you do when potable water is not available in nearby? How did availability of potable water affect your decision to return? Would its absence factor into future decision to leave? Please explain. What is the average number of hours of water supply per day? Sanitation Has there been any change in access to Are the sewerage services adequate for your community sewerage services in your community needs? Please explain. after your return? Please explain. (EQ1, 3) How did availability of sanitation services affect your decision to return? Would its absence in the future factor into future decision to leave? Please explain. Local Authorities’ Ability to Manage Public Infrastructure What is your perception of the local What is your opinion of the local authority’s capacity to authority’s ability to manage the public provide basic services? Do the services provided by your infrastructure and municipal services local authority meet your basic needs? Has this changed (electricity, health, education, water and over time? sanitation) in your community? Please What would you suggest that the local authorities can do explain. (EQ4) better to improve service delivery amongst returnees? General Question What are the main reasons why you returned to this area? What are the main reasons why you decided to remain in your homes or why were you displaced after 2014? Please explain. What are your thoughts around your future location of residence if you think about the options of staying here or moving somewhere else? Please explain. 155 FOCUS GROUP DISCUSSION FORM TARGET GROUP: RELIGIOUS AND ETHNIC MINORITIES90 (RETURNEES) Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. Returnees), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? YES NO [Allow only those who have verbally consented to proceed with the focus group discussions.] 90 Upon local expert’s advice using political affiliated groups renders itself to be a very sensitive selection criteria and would raise insecurity fears/concerns among potential FGD participants. 156 Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity Has there been any change in access to What is the difference for you between a day with electricity for households in your electricity and a day without? How does it change your community since FFS interventions. Please routine? explain. (EQ1, 3) What is the average of total electricity provision per day? Is it constant or vary depend on the season? Tell us about the process you had to go through to get your house connected to the electricity grid. Are there any barriers preventing households from this community from accessing electricity? Please explain. How did availability of electricity affect your decision to return or remain in this community? Would its absence factor into your decision to leave? Please explain. Do you depend on other sources of electricity providers, if so, what are they and for how many hours? Education 157 Has there been any change in access to schools for school-going children in your community since FFS interventions? Please explain. (EQ1, 3) Do all the children living in this community have the opportunity to attend school? How is your children’s education affected if there is no school in nearby? What are the challenges? How did availability of schools affect your decision to return or remain in this community? Would the absence of schools factor into future decision to leave? Please explain. Health Has there been any change in access to health services for your community since FFS interventions? Please explain. (EQ1, 3). What do you do when you or any member of your community gets sick? What do you do when the treatment you seek is not available in nearby hospital or other health facility? (For women) Do you need approval of your husband/father to access health services? Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? How did availability of health services affect your decision to return or remain in this community? Would its Questions Prompts absence factor into future decision to leave? Please explain. 158 Water Supply Has there been any change in access to potable water in your community since FFS interventions? Please explain. (EQ1, 3) What do you do when potable water is not available in nearby? How did availability of potable water supply affect your decision to return or remain in your community? Would its absence factor into your future decisions to leave? What is the average number of hours of water supply per day? Sanitation Has there been any change in access to Are the sewerage services adequate for your community sewerage services in your community needs? Please explain. since FFS interventions? Please explain. (EQ1, 3) How did availability of sewerage and sanitation services affect your decision to return or remain in this community? Would its absence in the future factor into future decisions to leave? Please explain. Local Authorities’ Ability to Manage Public Infrastructure What is your perception of the local What is your opinion of the local authority’s capacity to authority’s ability to manage the public provide basic services? Do the services provided by your infrastructure and municipal services local authority meet your basic needs? Has this changed (electricity, health, education, water and over time? sanitation) in your community? Please What would you suggest that the local authorities can do explain. (EQ4) better to improve service delivery amongst returnees? General Question What are the main reasons why you returned to this area? What are the main reasons why you decided to remain in your homes or why were you displaced after 2014? What are your thoughts around your future location of residence if you think about the options of staying here or moving somewhere else? Do you feel safe living in this community? Please explain. 159 Midterm Performance Evaluation of USAID/Iraq FFS Programs * NOTES TO THE ENUMERATOR CODING Enumerator ID Enter text here Supervisor Enter text here Date DD/MM/YYYY province Anbar Salah al-Din Ninewa Intervention area Enter text here Name of Location/ Neighborhood/Block Enter text here Grid Cell No Enter text here HH No in Cell (1-24) Enter text here Enter text here A10a HH GPS (Longitude) Enter text here A10b HH GPS (Latitude) Enter text here INFORMED CONSENT 1 Introductory scripts Assalam-o-Alaikum. My name is ______________________________, and I am working with ------. I am conducting a survey for IBTCI under the auspices of USAID to evaluate the stabilization related services in Anbar, Ninewa, and Salah al-Din provinces to support Iraqis return and resettle back in their homes and hometowns. We value your feedback and would appreciate your participation in this survey. This information will help the Government of Iraq and its development partners to further improve the stabilization programming for returning families. The survey will take about 30-45 minutes to complete. Whatever information you provide will be kept strictly confidential and anonymous. Participation in this survey is voluntary and you can choose not to answer any individual question or all of the questions. You can stop the survey at any time. However, we hope that you will participate in this survey since your views are important. Your responses are valuable to us and shall remain confidential. 160 2 Consent Do I have your consent to proceed with the survey? YES NO SECTION B: IDENTIFICATION CAPI ANONYMIZE NUMBERS Sex of Head of HH Male Female Respondent No. (1-24 in each grid cell) Sex Male Female Age Relationship of respondent to head of HH Self Spouse Parent Child Sibling Other: Total number of persons living in this household Total Male Female Married Unmarried Divorced Widowed Others: Marital Status Ethnic Origin Arab Assyrian Kurd Turkmen Shabak Others: Languages Spoken Arabic Kurdish Shabaki Others: Religion MuslimShia Muslim - Sunni Christian Yazidi Kakai Others: Highest Education None Elementary Junior High Secondary Higher Religious 161 Current Status IDP Returnee Never Left Home Family Status 91 Nuclear Joint Anyone in HH living with Disability Yes If Yes, How Many: No Type of Shelter Rented house Religious building School building Unfinished/abandoned building Informal settlement Other SECTION C: ELECTRICITY Changes to access and availability of electricity supply (EQ1.1) What is the source of electricity connected to your home? If not Option 1 (Public network) skip >>C8) [Related to access] Public electricity network Private Generator Shared Generator Other None 91 Note: Nuclear Family = Family that consists of a mother, father and their children. Does not include grandparents, uncles, and aunts. Joint Family = Family that consists of parents, and close relatives/multiple generations 162 If Option 1(Public electricity) in C1, when did you get connected to the public electricity network? (Please choose one only) 2019 2018 2017 2016 2015 Before 2015 Don’t know If the public electricity services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? Yes No Don’t Know Has the public electricity currently in use been restored/repaired after the liberation? Yes No Don’t Know Did availability of electricity affect your decision to return (If respondents are returnees)? Yes No Don’t Know Would availability of electricity at your place of origin affect your decision to return (If respondents are IDPs)? Yes No Don’t Know Would the absence of electricity in the future factor into your decision to leave your community? ASK ALL Yes No Don’t Know Barriers to accessing electricity (EQ1.2) Although you said you don’t have or are not connected to public electricity network in your home (in C1), is there public electricity supply/grid in your neighborhood/area? Yes No Don’t Know 163 If yes to C8, what is the main reason your home does not have electricity? MARK ALL THAT APPLY [Refers to Access Barriers] Connection fee is unaffordable The structure/dwelling does not meet the standards suitable to have electricity connection Could not afford electricity monthly subscription Application for connection is delaying Prefer private or shared generator Other (Specify) What best describes the process of getting connected to electricity? (Please choose one only) Very difficult Somewhat difficult Easy Very Easy Don’t know What best describes the cost of electricity? (Please choose one only) Affordable Not affordable Don’t know On the whole, do you agree or disagree with the following statement: Provision of electricity meets the needs of your household? (Please choose one only) Strongly agree Agree Neutral Disagree Strongly disagree Perceptions of the quality of electricity supply (EQ2) For how many hours per day is/was electricity available at your home? (Please choose one only) Now Prior to FFS intervention (Before 2015) 164 All day (24 hours per day) Most of the day (13-23 hours per day) Some of the day (4-12 hours per day) Rarely (1-3 hours per day) Never (0 hours per day) All day (24 hours per day) Most of the day (13-23 hours per day) Some of the day (4-12 hours per day) Rarely (1-3 hours per day) Never (0 hours per day) How many days a week is/was electricity available at your home? Now Prior to FFS intervention (Before 2015) (Please choose one only) Every day (7 days a week) Most days (4-6 days per week) Some days (2-3 days per week) Rarely (1 day per week) Never (0 days per week) Every day (7 days a week) Most days (4-6 days per week) Some days (2-3 days per week) Rarely (1 day per week) Never (0 days per week) Perceptions of the effectiveness of administration of electricity supply (EQ4) 165 How satisfied are you with the local authority’s or utility company’s administration of electricity supply services in the following areas? Maintaining infrastructure Response to power failures Customer billing Responsiveness to complaints Checking meters on a regular basis Ability to process new applications for connection [ENUMERATOR READ OUT] For each mark one whether Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Don’t know N/A SECTION D1: POTABLE WATER SUPPLY Changes to access and availability of portable water (EQ1.1) What is the main source of potable water for members of your household? (Please choose one only) Now Prior to FFS intervention (Before 2015) 166 (Refers to access) Piped into dwelling Piped into compound Community/neighborhood pipes Hand-pump/borehole Protected well Open unprotected well River/stream/lake Water tanker/truck/ bowser Bottled water Other Piped into dwelling Piped into compound Community/neighbo rhood pipes Handpump/borehole Protected well Open unprotected well River/stream/lake Water tanker/truck/ bowser Bottled water Other If source is piped water, is the water metered? (applies to options 1-3 in D1.1) Yes No Don’t Know Since when has the source of potable water been available for your household use? (Please choose one only) (Related to period service was accessed vs baseline) 2019 2018 2017 2016 2015 Before 2015 Don’t know 167 If the water supply services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? Yes No Don’t Know Have the water supply services currently in use been restored/repaired after the liberation? Yes No Don’t Know Where is the water source located? (Please choose one only) Into own dwelling skip >>A3- 1.8 Into own yard/plot skip >>A3-1.8 Elsewhere How long (in minutes) does it take you to walk from your home to reach this main source of safe drinking water [ONEWAY] [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] Minutes:……………. On average, how many hours per day is water supply available to your household from this source? [ Relates to continuity of supply/availability or non￾availability]] No of hours: ……………..(0-24 hours) How many pails/buckets of water do you use in a day in your home? (1 bucket=18 Liters) [LOCAL NAME FOR BUCKET] [ Relates to adequacy to consumption needs] # of pails/buckets/day……………….. Is the amount of potable water you get on a daily basis adequate for your household needs? (Please choose one only) [ Relates to relevance to consumption needs] Not adequate at all Not adequate, but we can manage Adequate Not sure Did availability of potable water affect your decision to return to your community (if respondents are returnees)? Yes No Don’t Know 168 Would availability of potable water at your place of origin affect your decision to return (If respondents are IDPs)? Yes No Don’t Know Would the absence of portable water in future factor into your decisions to leave? ASK ALL Yes No Don’t Know Barriers to access (EQ 1.2) Who usually goes to collect water for your household? [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] (Please choose one only) Adult woman (15+ years) Adult man (15+ years) Female child <15yrs Male child <15yrs How would you rate the difficulty of obtaining potable water? (Likert scale) ENUMERATOR : These questions should be asked from the person responsible for getting water] [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] (Please choose one only) Very difficult Somewhat difficult Not difficult Very Easy Don’t know What are the reasons for the unavailability? [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] MARK ALL THAT APPLY Damaged/vandalism of infrastructure Drying of water source Poor design No maintenance of water point Other Perceptions of the quality of water (EQ2) How would you describe the taste of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) 169 Good Good Salty/saline Salty/saline Greasy Greasy Slippery Slippery Sour Sour Metallic Metallic Chemical Chemical Not sure Not sure How would you describe the smell of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) Good Foul smell Not sure Good Foul smell Not sure How would you describe the clarity/color of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) Clear Turbid Not sure Clear Turbid Not sure During the last one month have you or any member of your household been sick due to the drinking water? Yes No Don’t Know If you or members of your family have suffered from any waterborne diseases in the last 12 months, do you know which of the following you or your family members had? Now (Last 12 months) Prior to FFS intervention (Before 2015) 170 Can you identify any you had before the FFS intervention? MARK ALL THAT APPLY DO NOT READ LIST None Diarrhea Bilharzia Cholera Dysentery DK Other (specify) None Diarrhea Bilharzia Cholera Dysentery DK Other (specify) Do you treat your water to make it safe to drink? If No skip >>D1.24 Yes No Don’t Know If yes, what do you usually do to treat water to make it safe to drink? (Please choose one only) Boil Add bleach/chlorine Strain it through cloth Use water filter, Filtration (ceramic, sand, etc.) Purification tablets Solar disinfection Other (specify) Perceptions of the effectiveness of administration of water supply (EQ4) How satisfied are you with the local authority’s or utility company’s administration of water supply services in the following areas? Maintaining infrastructure Response to service interruptions Regular billing Availability of customer service and complaints hotline Checking meters on a regular basis Ability to process new applications for connection [ENUMERATOR READ OUT] For each mark one whether Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Don’t know N/A 171 SECTION D2: SANITATION Changes to access and availability of sanitation services (EQ1.1) How does your household dispose of solid waste materials/garbage? (Please choose one only] Household refuse bins Rubbish pits Collection from dwelling by municipality Communal refuse bins or pits Waste disposed in the open Burying Other (specify) None What type of waste water/sewerage system is your household connected to? (Please choose one only) Public sewer network with covered canal Public sewer network with uncovered canal Septic tank Other (Specify) None Since which year have the sanitation services (sewerage) been available for your household use in this area? (Related to period service was accessed vs baseline) (Please choose one only) 2019 2018 2017 2016 2015 Before 2015 Don’t know If the sewerage services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? Yes No Don’t Know 172 Has the sewerage system been restored/repaired after the liberation? Yes No Don’t Know Did availability of sanitation services affect your decision to return (if respondents are returnees)? Yes No Don’t Know Would availability of sanitation services at your place of origin affect your decision to return (If respondents are IDPs)? Yes No Don’t Know Would the absence of sanitation services in the future factor into your decisions to leave? ASK ALL Yes No Don’t Know Perceptions of the quality of sanitation services (EQ2) How frequent are your complaints on the non￾functionality of the wastewater/sewerage system? Frequent Not frequent Never Don’t Know Perceptions of the effectiveness of administration of sanitation services (EQ4) How satisfied are you with the local authority’s or utility company’s administration of sanitation services regarding the following areas? Maintaining sanitation infrastructure Connecting households to sewer Responsiveness to non-functioning/ blockages and other disruptions on the sewer Collection of solid waste Collection of human waste if not connected to sewer [ENUMERATOR READ OUT] For each mark one whether Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Don’t know N/A SECTION E: HEALTH 173 Changes to access and availability of health services (EQ1.1) How far in kilometers (Km) is the nearest government health facility to your household compared to before the FFS period? Now Prior to FFS intervention (Before 2015) Km…….. Km…….. Since when or in which year have the health services been available for your household use in this area? (Related to period service was accessed vs baseline) (Please choose one only) 2019 2018 2017 2016 2015 Before 2015 Don’t know If the health services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? Yes No Don’t Know Were the health services restored/repaired after the liberation? Yes No Don’t Know Did availability of health services affect your decision to return (if respondents are returnees)? Yes No Don’t Know Would availability of health services at your place of origin affect your decision to return (If respondents are IDPs)? Yes No Don’t Know Would the absence of health services in the future factor into your decisions to leave? ASK ALL Yes No Don’t Know Barriers to access (EQ 1.2) 174 Have you sought health services at the nearest facility and have not been assisted? If Yes Skip >>A4.8 Now Prior to FFS intervention (Before 2015) Yes No Don’t Know Yes No Don’t Know If not, what are the reasons for not being assisted? MARK ALL THAT APPLY (Refers to barriers to access and utilization) Now Prior to FFS intervention (Before 2015) Health facility did not offer the services Facility was closed, The wait was too long, It was cost-prohibitive Others: __________ Health facility did not offer the services Facility was closed, The wait was too long, It was cost prohibitive Others: __________ Perceptions of the quality of health services (EQ2) How satisfied are you with the following at your nearest public health facility? [ENUMERATOR READ OUT] Now Prior to FFS intervention (Before 2015) 175 For each one mention whether Quality of pediatric/child health services Quality of surgical services Quality of maternal health services Quality of general outpatient services Quality of emergency services Quality of infectious diseases treatment Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied No opinion because I did not receive the service No opinion because the service is not offered. Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied No opinion because I did not receive the service Quality of mental health care Waiting time to be attended to at the facility General cleanliness of the facility Don’t Know No opinion because the service is not offered. Don’t Know Perceptions of the effectiveness of administration of Health Services (EQ4) How satisfied are you with the local authority’s administration of health services in the following areas? Ability to maintain health infrastructure Ability to respond to outbreaks Ability to provide the required number of health workers [ENUMERATOR READ OUT] For each mark one whether Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Don’t know N/A SECTION F: EDUCATION Changes to access and availability of education services (EQ1.1) Is there any member of your households who is of school￾age? If NO skip>> G1 Yes No Don’t Know Are the school- aged children attending school? If NO skip>> F9 Yes No Don’t Know 176 If yes, how many children in your household are attending school? …………Male ………… Female What type of school do the children attend? ENUMEATOR ENTER FOR EACH TYPE AND LEVEL Level Type Number of children Primary Government Private non￾religious Islamic school Other religious Other Secondary Government Private non￾religious Islamic school Other religious Other Tertiary Government Private non￾religious Islamic school Other religious Other Since when or in which year have the school services been available to your household in this area? (Please choose one only) 2019 2018 2017 2016 2015 Before 2015 Don’t know 177 In general, how would you compare the following access dimensions for education between now and prior to FFS interventions? Now Prior to FFS intervention (Before 2015) Number of days per week school is open Number of hours per day school is open Distance in Kilometers from home Km:………………….. Km:……………… Frequency of disruptions during the academic calendar Frequent Not frequent Never Don’t Know Frequent Not frequent Never Don’t Know Prolonged school closures due to insecurity Frequent Not frequent Never Don’t Know Frequent Not frequent Never Don’t Know Did availability of education services affect your decision to return (if respondents are returnees)? Yes No Don’t Know Would availability of education services at your place of origin affect your decision to return (If respondents are IDPs)? Yes No Don’t Know Would the absence of education services in the future factor into your decisions to leave? ASK ALL Yes No Don’t Know Barriers to access (EQ 1.2) 178 If no (A5-2), what are the reasons for not attending school (barriers)? It is far away School in restricted area Insecurity and personal safety issues Cost-prohibitive Poor quality of schools Illness or disability of student Illness or disability of family member School conflict with beliefs Others: ________________________ Perceptions of the quality of education services (EQ2) Does the school the children attend have desks where every child can sit? Yes No Don’t Know Does the school the children attend have a sufficient number of classrooms? Yes No Don’t Know Does the school the children attend have water and sanitation facilities? Yes No Don’t Know Does the school the children attend have electricity? Yes No Don’t Know Perceptions of the effectiveness of administration of Education Services (EQ4) How satisfied are you with the local authority’s administration of education services with regard to the following areas? Ability to maintain school infrastructure Ability to provide teaching and learning materials Ability to conduct school inspections ENUMERATOR READ OUT] For each mark one whether Very satisfied Somewhat satisfied Somewhat dissatisfied Very dissatisfied Don’t know N/A 179 Section B: IDPS or Returnees Have you always lived in this area since birth? Yes No Don’t Know If no, how long have you lived in this area? No of years In which year did you move away from this area the last time? (Please choose one only) (refers to period of displacement) 2019 2018 2017 2016 2015 Before 2015 Don’t know N/A In which year did year you return to this area? (Please choose one only) (refers to period of resettlement) 2019 2018 2017 2016 2015 Before 2015 Don’t know Never left Are you planning to stay here permanently? Yes No Don’t Know 180 What are/were the main reasons for your displacement? [ENUMERATOR : THIS APPLIES TO RETURNEES AND IDPs] MARK ALL THAT APPLY Lack of basic services (e.g. water, electricity, health, education, sewerage services etc.) in area of origin Other Specify Ongoing fight/lack of security in area of origin Security in area of displacement House/property destroyed/damaged/demolished Lack of jobs back home Fear of security actors in area of origin Lack of services back home Fear of reprisal acts/violence back home Fear of harassment/discrimination back home Better services in place of displacement Fear of ISIS returning to area of origin Availability of jobs in place of displacement 181 What are the most critical factors that influenced you to return? [ENUMERATOR : THIS APPLIES TO RETURNEES AND IDPs] MARK ALL THAT APPLY Improvements in security Rehabilitation of residences and buildings/shelter Availability of health services Availability of education services Availability of water and sanitation services Availability of electricity Availability of business grants Employment and livelihood opportunities Availability of markets Police stations Manufacturing enterprises Roads and Bridge projects Humanitarian support This is the community where I grew up This is where my family lives I have a strong support system here I own property here None Other (specify What is your main source of household income MARK ALL THAT APPLY Street vending Casual labor Salaried employment /small business enterprise Large business enterprise Remittances Sale of Assets Agriculture Assets Other (Please specify) 182 Are your most important needs met by the basic services that are provided by the local authority? ASK ALL-RETUNEES AND IDPS [ONLY ONE MENTION] Not at all Not very much Mostly met Completely met Not Sure How would you compare your HH average Monthly income now to the period before returning/coming to this area? [ONLY ONE MENTION] Now Before Returning/Displa cement IQD300,000 Don’t Know N/A IQD300,00 Don’t Know N/A Overall is life improving for people who returned? [ENUMERATOR : THIS APPLIES TO RETURNEES] MARK ALL THAT APPLY Not at all Not very much Mostly met Completely met Not Sure Social Cohesion and Security Were you afraid to return to your community? [ENUMERATOR : THIS APPLIES TO RETURNEES] Yes No Not sure 183 If yes (B1-14), which of these were you worried about? MARK ALL THAT APPLY Discrimination Retaliation Tribal/Ethnic tensions A lack of security Explosive hazards Personal/family distress Reintegrating would be too difficult Other Since you returned have you encountered any of these? [ENUMERATOR : THIS APPLIES TO RETURNEES] Discrimination Yes No Not sure Retaliation Yes No Not sure Tribal/Ethnic tensions Yes No Not sure A lack of security Yes No Not sure Explosive hazards Yes No Not sure Personal/family distress Yes No Not sure Reintegrating would be too difficult Yes No Not sure Other Yes No Not sure Thank you very much for your time. The information you have provided will be very useful in evaluating the program and improve future programming 184 Midterm Performance Evaluation of USAID/Iraq FFS Programs (June 2019) Form Number TO BE ASSIGNED NOTES TO THE ENUMERATOR CODING 1 Interviewe r Code Enter text here 2 Supervisor Code Enter text here 3 Date Enter text here 4 province Enter text here 5 Interventio n area Enter text here 6 Areas Urban Rural INFORMED CONSENT 1 Introduce yourself Hello. My name is ______________________________, and I am working with ______. I am conducting a livelihood survey for IBTCI under the auspices of USAID to evaluate stabilization related services in Anbar, Ninewa, and Salah al-Din provinces to support Iraqis return and resettle back in their homes and hometowns. We value your feedback and would appreciate your participation in this survey. This information will help the Government of Iraq and its development partners to further improve stabilization programming for the returning families. The survey usually takes 30 minutes to complete. Whatever information you provide will be kept strictly confidential. Participation in this survey is voluntary and you can choose not to answer any individual question or all of the questions. You can stop the survey at any time. However, we hope that you will participate in this survey since your views are important. Your responses are valuable to us and shall remain confidential. 2 Consent Do I have your consent to proceed with the survey? [In the case of No, call the next individual] YES NO IDENTIFICATION (Respondent) 1 Sex Male Female 185 2 Age Enter number of full years here 3 Head of HH Sex Male Female 4 Relationship of respondent to HH Head Self Parent Child Sibling Other: 5 Number of persons living in this household Total Male Female Married Unmarried Divorced widowed Others 6 Marital Status 7 Ethnic Origin Arab Assyrian Kurd Turkmen Shabak Others: 8 Languages Spoken Arabic Kurdish Shabak Others: 9 Religion Muslim￾Shia MuslimSunn i Christian Yazidi Kakai Others: 10 Highest Education None Elementary Junior high Secondary Tertiary Religiou s 11 Status before receiving livelihood support IDP Returnee Refugee Never Left Home 12 Current Status IDP Returnee Refugee Never Left Home 13 Family Status 25 Nuclear Joint 14 HH Average Monthly income IQD 300,001 Don’t Know 186 15 Anyone in HH living with Disability Yes If Yes, How Many: ___ No 25 Note: Nuclear Family = Family that consists of a mother, father and their children. Does not include grandparents, uncles, and aunts. Joint Family = Family that consists of parents, and close relatives/multiple generations 16 If yes, is the livelihood beneficiary disabled Yes No 17 Living in the area for years months SECTION 1: LIVELIHOODS INTERVENTIONS -Cash for Work When you were first enrolled in the Cash for Work activity? Year 2015 ______ Year 2016 ______ Year 2017 ______ Year 2018 ______ How did you know about Cash for Work activity? Tick one option Informed by friends/relatives Informed by the Government Contacted by the contractors Public announcement Other, Specify ____________ How many times have you participated in a Cash for Work activity since 2015? Once Twice Three times and more Did you complete the last Cash for Work activity you were enrolled in?(If no skip to 1.5) Yes No Not sure 187 If yes, when did you complete the last Cash for Work activity you were enrolled in? Year 2015 ______ Year 2016 ______ Year 2017 ______ Year 2018 ______ How much money did you receive from the Cash for Work activity? IQD:____________ How long did you work for? Less than 1 week 1-4 weeks 5-8 weeks 9-12 weeks More than 12 weeks Did you receive the payment from the Cash for Work activity on time according to agreement schedule? Yes No Did you have any other sources of income when you received Cash for Work support? (If no, skip to 1.9) Yes No If yes, what were these other sources of income? Please mention up to three sources. 1. 2. 3. Was the money you received from the Cash for Work activity adequate to support your family’s needs? Yes No Not sure 188 How did you spend the money from the Cash for Work activity? Tick all that apply Spent on household basic needs Bought assets Started business Saved the money Other How much money did you spend last month on your household’s basic needs? Total spent on basic need: __________IQD Of which, spent on food: __________IQD What was the contribution of your income earned from Cash for Work activity to your household needs? (ENUMERATOR TO READ OUT THE RESPONSES) Primary source of household expenditure (contribution greater than 50%) Significant contributor but not primary source of household expenditure (contribution greater that 20% but less than 50%) Partial source of household expenditure (contribution less than 20%) Not a source of household expenditure Not sure Did you face any challenges when participating in the Cash for Work activity? Yes No 189 If yes, can you tell us what type of challenges you faced? (Check all that apply) Delay in payment Insecurity High transportation costs Physical harassment Sexual exploitation Money extortion Nepotism/favoritism Others: (specify____) Did the Cash for Work support influence your decision to return to this community? Yes No Not sure NA Did the Cash for Work support influence your decision to stay in this community? (For returnees only) Yes No Not sure NA How would you compare your household income now to the period before the Cash for Work support? Improved Same as before Worse How has your participation in the Cash for Work activity changed your power to make decisions at home? (For female) Improved Same as before Worse Not sure 190 (For returnees) What are the major reasons that influenced your decision to return to this community? Mark all that apply. Security situation is stable Livelihood opportunities are available Emotional obligation to return Family/community members have returned To secure personal housing, land or property Basic services are available Difficult conditions in area of displacement Other (Specify) ________ Thank you very much for your time. The information you have provided will be very useful in evaluating the program and improve future programming CHECKLIST FOR ON-SITE OBSERVATION FOR ELECTRICITY FACILITIES- GENERATOR Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 3 Date visited 4 Enumerator Visiting System Items Satisfactory Remarks (if there is an issue) No Yes 191 Generator and Controlling System Control Panel & Wiring Breaker Assembly Strip Heaters Remote Annunciators/Alarms Rotating Rectifier Main Rotor/Stator Phase Bar/Isolators Regulator/Wiring Starting System Batteries Load Test Battery Charger Float Rate VDC Starting Motor Alternator Output VDC Ignition Satisfactory Remarks (if there is an issue) 192 System Items Yes No Solenoids Eng. Wiring Harness/Sensors Air Intake & Exhaust System Air Filters Air Intake Assembly Charge Air Cooler Exhaust Assembly Turbo Assembly Silencer/Muffler Intercooler Lubrication System Oil MTU Oil Analysis Date of Sample Oil Filters Oil Pressure PSI 193 Crankcase Ventilation Lube System Gasket Seals Fuel System Fuel Line Connections Fuel Filters Primary/Secondary Fuel Pressure PSI Governor/Connections Set Tank/Day Tank Regulator/NG PSI Amps:L1_____ +L2_____ +L3______ =______ Amps ÷ 3 =______ Avg. Amps 194 Avg. Amps:_______ x Volts:______ x 1.732 (for 3ph) ÷ 1000 =________ KVA If final KVA calculation is greater than 30% of name plate value = “Pass” If final KVA calculation is less than 30% of name plate value = “Fail” CHECKLIST FOR ON-SITE OBSERVATION FOR ELECTRICITY FACILITIES- SUBSTATION Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 4 Date visited 4 Engineer Visiting Substation: Item to be checked Normal Remarks 400kV Overhead Lines Voltage reading and current reading ES NO Y Power Transformer The voltage and current readings of (400kV) of the main transformers YES NO The voltage and current readings of (132kV) of the main transformers YES NO Any noise from the transformers ES NO Y Winding temperature indicator reading ES NO Y 195 Oil temperature indicator reading ES NO Y On Load Tap Changer (OLTC) is working correctly or not ES NO Y OLTC reading ES NO Y Is there oil leak from the transformers ES NO Y Dehydrating Breather Status ES NO 132kV Overhead Lines & Underground Cable (UGC) Voltage reading and current reading Overhead Line (OHL) feeders ES NO Y Voltage reading and current reading UGC feeders ES NO Y Voltage reading and Status of Busbars/ Bus Couplers / Bus sections ES NO Y Battery Chargers and Inverters The voltage and current readings for (110V DC) Battery and Charger ES NO Y The voltage and current readings for (48V DC) Battery and Charger ES NO Y The voltage and current readings for inverters (220VAC UPS ) ES NO Y Building Services Lighting and Small power working conditions (Problems if any) ES NO Y 196 Air conditioner working conditions (Problems if any) ES NO Y Fire Alarm system working conditions (Problems if any) ES NO CHECKLIST FOR ON-SITE OBSERVATION TO SEWERAGE (WASTEWATER TREATMENT PLANT “WWTP”AND LIFT STATIONS) FACILTIES Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 3 Date visited 4 Engineer Visiting ON-SITE OBSERVATION FOR SANITATION FACILITIES Are there any rehabilitation activities underway or all work has been completed for the intended facility? YES NO What is the design capacity of the WWTP/lift station and what is the working capacity now? Design Capacity Working capacity 197 If the WWTP/lift station is working less than its design capacity, what are the reasons for this? Rehabilitation activities not completed yet Non-continuous power supply to the plant Staff is less than required Intermittent supplying of chemicals (coagulants, chlorine, etc.) Some equipment need spare parts Other (specify____) How many villages, intervention areas, population does the WWTP/lift station served? Is there any laboratory for testing the influent and the treated water quality? YES NO If yes, what is frequency of water quality measurements? Daily Weekly Monthly What are the physical, chemical and biological tests that are measured at the WWTP (pH, Color, test, TS, TDS, COD, BOD5, TOC, Fecal coliform FC, Total coliform count TC, heavy metals, Nitrate NO3, Phosphate PO4, etc.)? pH Color test TS TDS COD, BOD5, TOC Fecal Coliform (FC) Total Coliform count (TC) Heavy metals (Pb, Cd, Cr, et.) Nitrate NO3 Phosphate PO4 Other tests (to be mentioned) Are these tests done on schedule? YES NO Are there any units in the WWTP/lift station that don’t work (Lift pumps, distribution pumps, connection YES NO 198 pipes, valves, air blowers, mixers, screenings, grit chambers, Primary and secondary sedimentation tanks, aeration tanks, chlorination tanks, digesters, drying beds, odor control system, etc.)? Please check those that are not working from the list. Lift Pumps Distribution Pumps Connection Pipes Valves Air blowers Screenings Grit chambers Primary and secondary sedimentation tanks Aeration tanks Drying beds Odor control system Other Units (to be mentioned) In case of any unit that doesn’t work what is the reason for this? (Please mark for each checked “not working” from the list in Q9) Need spare parts Need more time to complete rehabilitation work Need qualification staff and training Other Specify____________ What is the type of coagulants/flocculants used in the WTP and in what quantity/concentration (for example Alum, Ferric Chloride, etc.)? Type Quant. Conc. Acids Alkalis Others What is the type of disinfectant used and in what is the quantity/concentration (for example chlorine gas, sodium or calcium hypochlorite, etc.)? Type Quant. Conc. Chlorine gas Sodium Hypochlorite Calcium Hypochlorite 199 Is there any lack in supplying the chemicals used in water purification? YES NO If so why? Is there any schedule for maintenance the WWTP units/lift station? YES NO If yes, what is schedule? If no, what are the reasons? How many tanks are working from each unit in its full capacity? How many tanks are present for each unit? (e.g., two sedimentation tanks, three filtration tanks, etc.)? Type of Tank No. Capacit y Sediment Filtration Others (to be mentioned) 1. 2. 3. 4. 5. How many hours the WWTP/lift station is operated per day? How often the treatments units cleaned from sludge? Weekly Every two weeks Monthly At certain Turbidity limits (mentioned) What is the plan for the sludge collected Disposed at landfill Reused as compost Is there a contingency plan in case of accidents, fire, injuries, etc.? YES NO 200 Is there any Occupational Health and Safety (OHS) plan in the WWTP/lift station? YES NO Is there a proper storage of chemicals used in wastewater treatment processes? YES NO Is there a continuous supplying for electricity to the WWTP/lift station? YES NO If no, for how many hours in day? Does the WWTP/lift station has its own electrical generator in case of national electrical grids is shut down? YES NO How was the technical and management capacity of staff for the intended facility? CHECKLIST FOR ON-SITE OBSERVATION TO WATER TREATMENT PLANTS FACILTIES (WTP) Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 5 Date visited 6 Engineer Visiting ON-SITE OBSERVATION FOR SANITATION FACILITIES Are there any rehabilitation activities underway or all work has been completed for the intended facility? YES NO What is the design capacity of the WTP and what is the working capacity now? Design Capacity Working capacity 201 If the WTP is working less than its design capacity, what are the reasons for this? Rehabilitation activities not completed yet Non-continuous power supply to the plant Staff is less than required Intermittent supplying of chemicals (coagulants, chlorine, etc.) Some equipment need spare parts Other (specify______) How many villages, intervention areas, population does the WTP served? Is there any laboratory for testing the influent and the producing water quality? YES NO If yes, what is frequency of water quality measurements (every day, every week, every month, etc.)? Daily Weekly Monthly What are the physical, chemical and biological tests that are measured at the WTP (pH, Color, test, Turbidity (NTU), Residual chlorine Cl- , hardness in term of CaCO3, Fecal coliform FC, Total coliform count TC, heavy metals, etc.)? pH Color test Turbidity (NTU) Residual Chlorine Cl￾Hardness in terms of CaCO3 Fecal Coliform (FC) Total Coliform count (TC) Heavy metals Are there any units in the WTP doesn’t work (Lift pumps, distribution pumps, connection pipes, valves, YES NO 202 mixers, screenings, water intakes, coagulation unit, flocculation unit, sedimentation unit, filtration unit, disinfection unit)? If yes, which ones? Lift Pumps Distribution Pumps Connection Pipes Valves Mixers Screenings Water Intakes Coagulation Unit Flocculation Unit Sedimentation Unit Filtration Unit Disinfection Unit In case of any unit doesn’t work what is the reason for this? Need spare parts Need more time to complete rehabilitation work Need qualification staff and training Other (specify____) What is the type of coagulants/flocculants/disinfectants used in the WTP and in what quantity/concentration (for example Alum, Ferric Chloride, etc.)? Type Quan t. Conc. Alum Ferric Chloride What is the type of disinfectant used and in what the quantity/concentration (for example chlorine gas, sodium or calcium hypochlorite, etc.)? Type Quan t. Conc. Chlorine gas Sodium Hypochlorite Calcium Hypochlorite 203 Is the chemical (disinfectant) readily available when needed for water treatment? YES NO If no why? Is there any schedule for maintenance the WTP units? YES NO If yes, what is schedule? If no, what are the reasons? How many tanks are working from each unit in its full capacity? How many tanks are present for each unit? (e.g., two sedimentation tanks, three filtration tanks, etc.)? Type of Tank No. Capacity Sediment Filtration Others (to be mentioned) 1. 2. 3. 4. 5. How many hours is the WTP operated per day? How often the coagulation/flocculation/ sedimentation tanks are cleaned from sludge? Weekly Every two weeks Monthly At certain Turbidity limits (to be mentioned) How often the filtration units are backwashed At certain Turbidity limits (to be mentioned) At certain pressure drop (to be mentioned) At regular schedule (to be mentioned) 204 Is there a contingency plan in case of accidents, fire, injuries, etc.? YES NO Is there any Occupational Health and Safety (OHS) plan in the WTP? YES NO Is there a proper storage of chemicals used in water purification processes? YES NO Is there a continuous supplying for electricity to the WTP? YES NO If no, for how many hours in day? Does the WTP has its own electrical generator in case of national electrical grids is shut down? YES NO How was the technical and management capacity of staff for the intended facility? 205 26 A full scale organizational capacity assessment is beyond the scope of this evaluation and this tool is not intended to capture that. technical capacity of local authorities to deliver services in line with their core mandate. 206 The capacity assessments will be conducted through facilitated discussions with at least one senior representative from each of the seven core governance and technical functions in each directorate. In total, 18 facilitated discussions will be conducted with six provincial directorates in each of the three provinces. List of Participants Local Authority Directorate # Name of Participant Gender Department Designation 1 2 3 4 5 6 7 8 9 10 Capacity Assessment of Provincial Directorates In this tool capacity is assessed on a scale of 1 to 4, where: 1 = Low, 2 = Basic, 3 = Moderate, 4 = Strong Functional area Questions Scores27 Notes 1 2 3 4 Budget Capacity Budget Are directorate budget realistic, clear, and well documented without external assistance? 207 Is there a budget for Operations and Maintenance (O&M) and what is the source? Is the budget realistic in term of meeting the O&M requirements? Is the locally generated revenue sufficient to meet O&M costs? Revenue Collection Capacity Revenue Collection Is there a system in place to collect revenue? (i.e., revenue staff, bills collection points, IT system etc.) Do you have systems in place for enforcement of fines or penalties for those who don’t pay? Does the revenue generated meets the operations and maintenance needs? Are multi-year revenue projections and expenditures reported annually? • 27 The questions are meant to be guidelines for the moderators conducting the assessment workshops. The ET will define responses for each of scores for all questions and the details will be provided in facilitator’s guide. For example for Question 3 under Budgets “Are project budgets realistic, clear, and well documented without external assistance?”, a score of 1=Project budgets are not realistic, clear, and well documented; 2=Project budgets are realistic, clear, and well documented only with external assistance; 3=Project budgets are realistic, clear, and well documented only with external assistance; and, 4=Project budgets are realistic, clear, and well documented without external assistance. Functional area Questions Scores27 Notes 1 2 3 4 Management and Administration Capacity 208 Procurement Does the directorate have a procurement policy and guidelines? Are these procurement guidelines adhered to? Are the procurement procedures monitored for compliance? How does the directorate handle donor procurement requirements? Planning92 Strategic Planning (Medium to Long term ) Does the organization have a strategic plan (business plan)? Is the strategic plan clear and specific on priorities, measurable objectives, and targets? Is the strategic plan regularly reviewed? When was it last revised? Is the strategic plan consistently used in guiding Management decisions and operational planning? Work-plan Development How does the local authority prepare its annual work-plans? Who is involved in developing annual workplans? 92 This tool has been adapted from USAID’s Organizational Capacity Assessment (OCA) Facilitator’s Guide Version With NUPAS Items, revised July 27, 2015. This document can be found on the URL https://usaidlearninglab.org/library/organizational-capacity-assessment 209 Are the annual work￾plans developed according to the planning cycle of the central government? Are the work-plans linked to the budget of the local authority? How are annual workplans used for management decisions, operational planning, and monitoring and evaluation? Human resource management Are there written policies and procedures for staff? Does staff have clear job descriptions? Is the staffing complete? Are the policies and procedures on staffing and job descriptions followed? Is the recruitment and hiring process open and transparent? How are positions announced or advertised? Are the key management and administration positions currently filled by people with qualified and skilled staff? Do most staff have the capacity in terms of their qualification and skills set to do their current jobs? Functional area Questions Scores27 Notes 1 2 3 4 210 Monitoring & Evaluation Does the local authority have policies and procedures for the organization to have a good sense of how it is doing and what it is accomplishing? Are the policies and procedures followed? Does the organization consistently set realistic targets and quantitative and qualitative performance indicators? Do staff have sufficient expertise in data collection and monitoring? Are project monitoring data consistently timely and reliable? Are monitoring data used to design effective remediation measures? Information and Communication Technology? Does the directorate have policies and procedures for communication and information technology? Are the policies and procedures adhered to by all staff? Are IT systems adequate for the current staff size, including expected growth over the next year? Are IT systems networked? Are IT systems backed up? What risks are there to the IT system (e.g. unreliable electricity, malware and viruses)? 211 Do IT staff and users have the skills to use the systems effectively? What additional IT skills are needed? Is training available on IT equipment use, software, and security? Technical Capacity (Facilitator Use Section that Applies to the Relevant Directorate) Electricity Does the directorate have an O&M manual? Is it being followed? Does the directorate have an annual O&M plan? Is it being followed? Are the electrical installation inspected regularity, are these inspections in line with the O&M plan? Are the defective parts repaired promptly? Are the required parts easily accessible in Iraq? Do most technical staff have the capacity in terms of their qualification and skills set to do their current jobs? Has the technical staff been trained? If so do they think training was adequate and prepared them for their jobs? Do directorates have the ability to provide timely service for new connections, respond to emergency situations (i.e. power failures) and conduct routine maintenance of equipment? 212 Is there a policy on how quickly complaints must be resolved or customer provided information? Is it followed? Water Does the directorate have an O&M manual? Is it followed? Does the directorate have an annual O&M plan? Is it followed? Are the water pipelines inspected regularly? Are the inspections in line with the O&M plan? Are the defective parts repaired promptly? Are the required parts easily accessible in Iraq? Do most technical staff have the capacity in terms of their qualification and skills set to do their current jobs? Has the technical staff been trained? If so do they think training was adequate and prepared them for their jobs? Is water quality being regularly tested? Does it meet required standards? Functional area Questions Scores27 Notes 1 2 3 4 Sewer (Waste Water Management)/Municipal Services Does the directorate have an O&M manual? Is it followed? 213 Does the directorate have an annual O&M plan? Is it followed? Is the solid waste (garbage) collected regularly by the municipality? Is it in line with the operation plan for garbage collection? Is there a system in place to maintenance of equipment and machinery (e.g., garbage containers, pressers, tractors etc.? Are the defective parts repaired promptly? Are the required parts easily accessible in Iraq? Do most technical staff has the capacity in terms of their qualification and skills set to do their current jobs? Are the sewerage lines inspected regularly? Are the inspections in line with the O&M plan? Are the quality standards consistently monitored? Is there a policy on how quickly complaints must be resolved or customer provided information? Is it followed? Health Does the directorate have sufficient numbers of health workers according to staff established positions? Functional area Questions Scores27 Notes 214 1 2 3 4 What are the vacancy rates for main cadre of doctors and nurses in the province? Does the directorate recruit and retain staff according to Human Resources for Health strategy staff requirements? Does the directorate develop and distribute guidelines and protocols for standard health practice? Does the directorate provide supportive supervision in the intervention areas and in the use delivery? Does the directorate have an effective supply chain management system-to effectively store and account for health commodities? Does the directorate have an O&M manual? Is it followed? Does the directorate have an annual O&M plan? Is it followed? Education Does the directorate have an approved education sector plan and guidelines that guide quality delivery & monitoring standards for education in line the with the national education policy? Is it being followed? 215 Does the directorate have established community and intervention area based structures, and processes Functional area Questions Scores27 Notes 1 2 3 4 that facilitate effective education delivery in the province? Has the directorate registered any improvement in number of classrooms? Has the directorate registered any improvement in number schools with access to clean water sources and sanitation facilities? 216 Name Organization Department TBD USAID Head of Stabilization Team TBD USAID Stabilization Program Specialist TBD USAID/GEO USAID GEO Office Director TBD OFDA DART OFDA Dart Team Leader TBD DOS DOS Religious Minority Groups Lead TBD DOS DOS Political Head TBD UNDP TBD TBD UNDP Head of Monitoring & Evaluation TBD UK/DFID UK/DFID-First Secretary Development TBD FDA France/FDA-Project Officer TBD New Zealand Embassy in Iraq New Zealand-Head of Development Cooperation TBD Embassy of Canada in Iraq Canada Head of Development Cooperation TBD Embassy of Germany Germany Head of Development Cooperation TBD Directorate of Electricity￾Ninewa Director (or representative) TBD Directorate of Water￾Ninewa Director (or representative) TBD Directorate of Sewer￾Ninewa Director (or representative) TBD Directorate of Municipality￾Ninewa Director (or representative) 217 TBD Directorate of Education￾Ninewa Director (or representative) TBD Directorate of Health Director (or representative) Name Organization Department TBD Mayor’s Office/Local Council- Ninewa Mayor/Head of Local Council TBD Directorate of Electricity￾Anbar Director (or representative) TBD Directorate of WaterAnbar Director (or representative) TBD Directorate of SewerAnbar Director (or representative) TBD Directorate of MunicipalityAnbar Director (or representative) TBD Directorate of EducationAnbar Director (or representative) TBD Directorate of HealthAnbar Director (or representative) TBD Mayor’s Office/Local Council-Anbar Mayor/Head of Local Council TBD Directorate of Electricity￾Salah al-Din Director (or representative) TBD Directorate of Water￾Salah al-Din Director (or representative) TBD Directorate of Sewer￾Salah al-Din Director (or representative) TBD Directorate of Municipality- Salah al-Din Director (or representative) 218 TBD Directorate of Education￾Salah al-Din Director (or representative) TBD Directorate of Health￾Salah al-Din Director (or representative) TBD Mayor’s Office/Local Council- Salah al-Din Mayor/Head of Local Council ANNEX 4: LIST OF KEY INFORMANT INTERVIEWS LIST OF KEY INFORMANT INTERVIEWEES # Name Area Name of Organization Title 1 Mustafa Saadoun Anbar Sewerage Deputy Director 2 Abdusattar Ali Hamad Anbar Education Deputy Director 3 Monther Mahdi Saleh Anbar Electricity Director, Planning Department 4 Mahmoud Salim Mahdi Anbar Water Deputy Director 5 Ahmed Mahdi Ibrahim Anbar Municipality Assistant Director 6 Mohamed Khalaf Hamad Anbar Health Deputy Director 7 Hikmat Al-Baldawi Anbar Council Technical Representative for Salah Al-Din Governor 219 8 Dana Mansuri Baghdad USAID Mission Director 9 Paul Bruning Baghdad USAID USAID GEO Office Director 10 Ahmed Mohammed Agha Baghdad USAID Stabilization Team 11 Jodi Rosenstein Baghdad USAID Acting Head of Stabilization 12 Joey Hood Baghdad Department of State Former Deputy Chief of Mission 13 Hilary Murphy Baghdad UNDP Head of Monitoring and Evaluation 14 William Evans Baghdad British Embassy Governance and Conflict Adviser 15 Jane Reid Baghdad UNDP Reporting Officer 16 Harald Herrmann Baghdad German Embassy First Secretary 17 Muhammad Mudawi Baghdad UNDP Former Head of Stabilization 18 Mahmoud Younis Ninawa Sewerage Head of Engineers, Department of Operations at the Directorate of Sanitation 19 Nima Zainal Ninawa Education Deputy Director 20 Asmaa Salim Ninawa Electricity Director of Technical Affairs 21 Nielsen Philps Khoshaba Ninawa Water Director of Water Directory 220 22 Engineer Mohammed Ninawa Municipality Head of Planning Department 23 AbdulSattar Ahmed Mohammed Ninawa Health Coordinator of Organization 24 Sami Al-Faisal Ninawa Council Coordinator of USAID Projects 25 Ahmed Ayed Mohammed Tikrit Sewerage Execution Officer 26 Mohammed Hussain Kadhum Tikrit Electricity Associate Director of Electricity Distribution 27 Engineer Mumtaz Tikrit Water Administrative Assistant of Salah Al-Din Directorate of Water. 28 Engineer Eissa Ibrahim Tikrit Municipality Associate Director of Services 29 Omar Khair El-Dein Tikrit Health Director of Inspection Department, Coordinator of the Work of International Organizations 30 Ammar Hekmat Al Balgawi Tikrit Municipality Deputy Director ANNEX 5: DATA COLLECTION TOOLS KEY INFORMANT INTERVIEW GUIDE KEY INFORMANT INTERVIEW FORM- GOVERNMENT OFFICIALS (ELECTRICITY) TARGET GROUP: DIRECTORATE OF ELECTRICITY (DIRECTOR OR REPRESENTATIVE) Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 221 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS activities with regard to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine how FFS activities influenced their decision to return and/or remain in their communities. The findings will assist decision makers to understand outcomes of FFS activities and may be used to inform future programming. 2 Ascription The interview will last approximately 50 minutes to an hour, and your responses will be included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential.” Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO 222 SECTION 1: ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. What is the current status of electricity supply in your area? a) What is the source of electricity supplied in the area (e.g. generator, hydro, other etc.? What is the distribution capacity of electricity? b) Is the supply adequate to meet the demand? If not what are underlying reasons for the undersupply? 2. How has FFS contributed to the supply of electricity in your area? Have there been any unintended outcomes, positive or negative? Please explain. 3. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? SECTION 2: QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Can you please tell us about the factors affecting quality of electricity supply in your area? a. In terms of reliability i.e., the number of electricity hours/day? If not reliable, what are the reasons? How is this being addressed? b. Whether voltage is stable or not? If not what are the underlying reasons? How is this being addressed? SECTION 3. CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. Do you have adequate technical and support staff (e.g., in terms of number of engineers/technicians) in place to respond to distribution losses and routine maintenance of equipment? Do they have the appropriate skill set and qualification to do their current jobs well? Please explain. 2. Apart from budget allocations from the central government, does the directorate have any other avenues for generating revenue? What are these sources? 3. If yes, does your institution have adequate capacity in terms of: a) How revenue is collected b) Number of staff to collect revenue c) Enforcement mechanisms to collect revenue (i.e. controlling Illegal connections) 4. What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? a) How are requests for funding made? Does that involve input from your office? b) How much of the allocated budget is received and utilized for O&M? 5. Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 223 KEY INFORMANT INTERVIEW-GOVERNMENT OFFICIALS (SANITATION) TARGET GROUP: DIRECTOROF SANITATION OR REPRESENTATIVE Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 50 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO 224 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. What is the current status of sewerage infrastructure in your area? a) Is the sewerage infrastructure adequate to meet the demand? If not what are underlying reasons for the undersupply? b) What steps has your institution taken to address the sewerage infrastructure gaps if any? Please explain. 2. How has FFS contributed to the development of sewerage infrastructure in your area? Have there been any unintended outcomes, positive or negative? Please explain. 3. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? 4. In your opinion what are some of the barriers to accessing sewage infrastructure in your area? Are there initiatives in place to address these barriers? 5. If sewerage system is not available in this area, what services are available for waste water disposal and treatment? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Have there been any significant changes in quality of sewage infrastructure and equipment due to FFS interventions in your area since FFS interventions? Please explain. 2. To what extent has the FFS activities contributed to the quality of sewage infrastructure and equipment in this area since FFS interventions? Please give examples. 3. Have changes in quality of sewerage infrastructure and equipment affected the welfare of surrounding communities? If so, how? Have these effects been the same along ethnic / tribal lines? Have these changes in quality influenced IDP decisions to return to their communities in this area? SECTION 3. CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. Do you have adequate technical and support staff (e.g., in terms of number of engineers/technicians) in place to respond to new connections, blockages of pipes and routine maintenance of equipment? Do they have the appropriate skill set and qualification to do their current jobs well? Please explain. 2. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? 3. If yes, does your institution have adequate capacity in terms of: a) How revenue is collected b) Number of staff to collect revenue c) Enforcement mechanisms to collect revenue (i.e. controlling Illegal connections) 4. What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? 5. Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 225 KEY INFORMANT INTERVIEW FORM- GOVERNMENT OFFICIALS (EDUCATION) TARGET GROUP: DIRECTORS OF EDUCATION OR REPRESENTATIVE MIDTERM PERFORMANCE EVALUATION OF USAID/IRAQ FFS PROGRAM RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 50 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 226 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. What is the current status of education infrastructure (i.e. in terms of number and type of schools; number of classrooms per schools; availability of basic facilities in schools etc.) in your area? Is the available infrastructure adequate to meet the demand for education services in this area? If not, what are underlying reasons? 2. If there are any infrastructure gaps (such as schools that have not been rehabilitated), what are underlying reasons? And what steps has your institution taken to address these, if any? Please explain. 3. How has FFS contributed to the development of education infrastructure in your area? Have there been any unintended outcomes, positive or negative? Please Explain. 4. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. 5. In your opinion what are some of the barriers to going to school in your area? Are there any government programs or non-government programs (i.e. run by NGOs or religious groups) in place to address these barriers? 6. How does the directorate ensure equal access between boys and girls, among different ethnic groups, for IDPs, etc.? Please explain. o What policies does the Ministry/ Directorate of Education have in place to ensure inclusive education? o How would you describe the trends of enrollment, retention and completion rates for boys and girls in primary and secondary schools? (to request provincial official school statistics from 2015 to 2018) o How would you describe cultural norms about girls going to school? o Are there separate male/female bathrooms/sanitary facilities at schools? o Whether commuting/walking to school is considered safe? 7. Do you think access to new/improved school influenced the decisions of many IDPs to return to their communities in this area? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Have there been any significant changes to quality of school environment (in terms of classroom space, facilities and equipment, basic amenities such as water and electricity, separate bathrooms for boys and girls, safety at school etc.) in your area since FFS interventions? Please explain? 2. To what extent have the FFS activities contributed to the quality of school buildings in this area since FFS interventions? Please give examples. SECTION 3: CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. Do you have adequate human resources (e.g., in terms of number of teachers) in place in the rehabilitated schools? Do they have the appropriate skill set and qualification to do their current jobs well? At what capacity (i.e., full/half capacity, or underutilized etc.) are rehabilitated schools/ facilities working? Please explain. 227 2. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? a. How are budget shortfalls addressed? Are there any family or teacher contributions? 3. If yes, does your institution have adequate capacity in terms of: a. How revenue is collected b. Number of staff to collect revenue c. Enforcement mechanisms to collect revenue 4. What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? a. How are requests for funding made? Does that involve input from your office? b. How much of the allocated budget is received and utilized for O&M? 5. Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? KEY INFORMANT INTERVIEW FORM- GOVERNMENT OFFICIALS (HEALTH) TARGET GROUP: DIRECTOR OF HEALTH OR REPRESENTATIVE Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 228 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 50 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO 229 SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. What is the current status of health infrastructure (i.e., in terms of number and type of health facilities) and equipment in your area? 2. Is the available health infrastructure adequate to serve the population? If not, what are underlying reasons and what steps has your institution taken to address these gaps? Please explain. 3. How has FFS contributed to provision of health infrastructure and equipment in your area? 4. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. 5. Have the changes in provision of health infrastructure and equipment affected delivery of health services (e.g. are previously unavailable services now available). Are more people able to access those services? Have there been any unintended outcomes, positive or negative? Please explain. 6. What are the primary barriers preventing residents from accessing health services? Are there initiatives in place to address these barriers? 7. How does the directorate ensure equal access between men and women, among different ethnic groups, for IDPs (living in the targeted areas), etc.? Please explain. o Are there any differences in the utilization of health services between men and women? o Are there any differences in utilization of health services among different identity groups (e.g., IDPs, minorities, returnees, disabled persons etc.)? o Are there any official statistics on utilization to share? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Have there been any significant changes to quality of health infrastructure and equipment due to FFS interventions in your area since FFS interventions? Please explain. 2. To what extent has the FFS activities contributed to changes in improved quality of health infrastructure and equipment in this area since FFS interventions? Please give examples? 3. Have these changes in health infrastructure and equipment affected quality of health care in your area? If so, have these changes been consistent across different groups (e.g. minorities, IDPs, women, and other vulnerable groups? How have these changes in quality affected IDP returns to their communities in this area? SECTION 3: CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. Do you have adequate number of staff in place to run and maintain the rehabilitated health facilities? Do they have the appropriate skill set and qualification to do their current jobs well? At what capacity (i.e., full/half capacity, or underutilized etc.) are rehabilitated health facilities working? Please explain. 2. Apart from budget allocations from central government, does the directorate have any other avenues for generating revenue? What are these sources? (Examples include user fees, health insurance, etc.) 3. If yes, does your institution have adequate capacity in terms of: a) How revenue is collected b) Number of staff to collect revenue c) Enforcement mechanisms to collect revenue 4. What is the source of your Operational and Maintenance (O&M) budget? Is the budget adequate to meet the O&M needs? Please explain? 230 a) How are requests for funding made? Does that involve input from your office? b) How much of the allocated budget is received and utilized for O&M? 5. Are you generating enough revenue to meet O&M costs (e.g., as a percentage of expected revenue collected)? If no, why? Is there anything else you’d like to share that we haven’t asked in this interview? 231 KEY INFORMANT INTERVIEW-GOVERNMENT REPRESENTATIVES (MAYOR’S OFFICE/LOCAL COUNCIL) TARGET GROUP: MAYOR OR HEAD OF LOCAL COUNCIL Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 30 mins, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 232 1. What are the major accomplishments of FFS in terms of encouraging IDPs to return to their place of origin in the newly liberated areas since FFS interventions? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. (FFS provided public infrastructure support in education, health, water, sanitation, electricity and other municipal services etc.) 2. What have been some of the major challenges (political, economic, social, technological, environmental, legal, etc.) faced in implementing the FFS? How were these challenges addressed? Please explain. 3. What was your office’s involvement in FFS interventions? What was your involvement in site selection, type and scope of support, and oversight and monitoring of implementation? Please explain. 4. How much municipal government input was sought and utilized? 5. What mechanisms were put in place to ensure equitable distribution of infrastructural projects for basic services among the IDPs and returnees? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Has there been any significant change in quality of service delivery (FFS priority sectors) since FFS interventions in the targeted areas? How did FFS contribute to these changes? Have these changes in quality affected IDP returns to their communities in this area? Please explain? SECTION 3. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EVALUATION QUESTION 3) 1. What trends have you observed in terms of IDPs returning to their place of origin in the since FFS intervention? Please explain and provide examples. 2. In your opinion in what ways has FFS infrastructure rehabilitation influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? 3. In your opinion, which sectors (if any) supported through FFS infrastructure rehabilitation initiatives have been more successful in attracting IDPs to return than others and if so, why? (i.e., electricity, education health, water and sanitation)? Please explain. 4. In your opinion in what ways have FFS livelihood activities influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? Please explain. 5. Are there any other ongoing or completed donor-funded programs that complement FFS activities in terms of stability of returns among IDPs? If yes, what are they? How are activities complemented? SECTION 4. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. How effective is the coordination between the UNDP and the mayor’s office/city council? a. Effectiveness of provincial government’s contribution and commitment? b. The level of engagement of civil society in design and implementation of services? Is there anything else you’d like to share that we haven’t asked in this interview? 233 KEY INFORMANT INTERVIEW FORM-DEVELOPMENT PARTNERS TARGET AUDIENCE: UNDP, USAID, OTHER DONORS Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 30 mins, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. What is your perception of the major accomplishments of FFS in terms of encouraging 234 IDPs to return to their place of origin in the newly liberated areas since FFS interventions? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. 2. What are some of the major challenges faced in implementing the FFS? How were these challenges addressed? Please explain. 3. How supportive and cooperative was the GOI in the design, implementation, and monitoring of FFS interventions? How much provincial and municipal government input was sought and utilized? Please explain. 4. What mechanisms were put in place to ensure equitable distribution of infrastructural projects for basic services among the IDPs and returnees? SECTION 2. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EVALUATION QUESTION 3) 1. In your opinion in what ways has FFS infrastructure rehabilitation influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? In your opinion, which sectors (if any) supported through FFS infrastructure rehabilitation initiatives have been more successful in attracting IDPs to return than others and if so, why? (i.e., electricity, education health, water and sanitation)? Please explain. 2. In your opinion in what ways have FFS livelihood activities influenced IDPs decision to return and settle in their communities in the targeted areas? Were the program’s expectations in ensuring stable returns met? Please explain. 3. What trends have you observed in terms of IDPs returning to their place of origin since FFS intervention? Please explain and provide examples. 4. Are your aware of any other ongoing or completed donor-funded programs that complement FFS activities to ensure stability of returns among IDPs? If yes, what are they? How are activities coordinated? SECTION 3. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EVALUATION QUESTION 4) 1. How effective is the coordination between the implementing partner and government? a. Effectiveness of national/provincial government’s contribution and commitment? b. The level of engagement of civil society in design and implementation of services? 2. In your view, have you observed any changes, positive or negative, in the capacity of the local authorities to sustainably operate and maintain the public infrastructure? Do they have the resources and training needed to operate and maintain the public infrastructure? Please explain. Is there anything else you’d like to share that we haven’t asked in this interview? KEY INFORMANT INTERVIEW FORM TARGET AUDIENCE: CIVIL SOCIETY AND OTHER INTERNATIONAL NGOS Midterm Performance Evaluation of USAID/Iraq FFS Program RESPONDENT’S INFORMATION 1 Interview Date Click Here to Enter Text. 2 Interviewer Name Click Here to Enter Text. 235 3 Name of Respondent Click Here to Enter Text. 4 Sex ☐ Male ☐ Female 5 Name of Org. Name & Acronym Click Here to Enter Text. 6 Designation/Title Click Here to Enter Text. 7 Contact Information Click Here to Enter Text. 8 Sector & Areas of Responsibilities Click Here to Enter Text. 9 province/ Intervention area/ Community Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT 1 Introductory scripts “I am a representative of ------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FFS) in Iraq on behalf of USAID. FFS aims to create the conditions for return of displaced communities through the restoration of essential services areas liberated from ISIS. The purpose of this interview is to better understand FFS Programs activities with regards to access and quality of services and contributing or impeding factors to program delivery in the targeted areas. Our interview also focuses on the affected population and beneficiaries (i.e., IDPs/Returnees) to determine their decision to return and/or remain in their communities. The findings will assist decision makers to form lessons and good practices in the field and will be used to inform future programming. The interview will last approximately 30 mins to an hour, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential if you wish.” 2 Ascription Can the Evaluation Team directly attribute your designation/title in the body of the report (e.g. organization and title)? ☐ YES ☐ NO 3 Consent Do I have your consent to proceed with the interview? ☐ YES ☐ NO SECTION 1. ACCESS TO SERVICES (LINKED TO EVALUATION QUESTION 1) 1. In your opinion what are some of the major accomplishments of FFS in terms of encouraging IDPs to return to their place of origin in the newly liberated areas since FFS intervention? What are the major lessons learned? Have there been any unintended outcomes, positive or negative? Please explain. o Do they think project was appropriately designed and targeted? o Was it complementary to other stabilization efforts underway? Was there duplication of any efforts? o Was it well-managed? o Was FFS team interested in outside input? Did they seek it? 236 2. What have been some of the major challenges faced in implementing FFS? How were these challenges addressed? Do you think their efforts were satisfactory? Please explain. 3. How supportive and cooperative was the GOI in the design, implementation, and monitoring of FFS interventions? How much CSO input was sought and utilized? Please explain. 4. What mechanisms were put in place to ensure equitable access to infrastructure projects for basic services among the IDPs and returnees? SECTION 2. QUALITY OF SERVICES (LINKED TO EVALUATION QUESTION 2) 1. Has there been any significant change in quality of public infrastructure (FFS priority sectors) in since FFS intervention in the targeted areas? How did FFS contribute to these changes? How have these changes in quality affected IDP returns to their communities in this area? Please explain? SECTION 3. IDPS/RETURNEES PUSH AND PULL FACTORS (LINKED TO EQ3) 1. What trends have you observed in terms of IDPs returning to their place of origin since FFS intervention? Please explain and provide examples. 2. What other programs and interventions are there to complement the FFS activities and ensure stability of returns among IDPs? Are there interrelationships between those programs and FFS? SECTION 4. FACTORS CONTRIBUTING TO OR IMPEDING LOCAL AUTHORITIES’ CAPACITY TO ADMINISTER SERVICES (LINKED TO EQ4) 1. How effective is the coordination among the civil society organizations, implementing partners and the government? a. The level of engagement of civil society in design and implementation of services? 2. In your opinion , have you observed any changes positive or negative related in the capacity of local authorities to sustainably operate and maintain the public infrastructure? Please explain. Is there anything else you’d like to share that we haven’t asked in this interview? THANKS 237 FOCUS GROUP DISCUSSION FORM FOCUS GROUP DISCUSSION FORM TARGET GROUP: IDPS Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. IDPs), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? ☐ YES ☐ NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity 1. Has there been any change in a) What is the difference for you between a day with access to electricity for households electricity and a day without? How does it change in your community since FFS your routine? intervention. Please explain. (EQ1, 3) b) How would availability of electricity affect your decision to return to place of origin? Would its absence factor into your decision to return? 238 Education 2. Has there been any change in a) Do all the children living in this community have access to schools for school-going the opportunity to attend school? What are the children in your community in the challenges? since FFS intervention? Please b) How is your children’s education affected if there explain. (EQ1, 3) is no school nearby? c) How would availability of schools in your place of origin affect your decision to return? Health 3. Has there been any change in access to health services for your community since FFS intervention? Please explain. (EQ1, 3). a) What do you do when you or any member of your community gets sick? What do you do when the treatment you seek is not available in nearby hospital or other health facility? b) What are the main barriers faced in accessing health services? c) (For women) Do you need approval of your husband/father to access health services? d) Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? e) How would availability of health services in your place of origin your affect your decision to return? Water Supply 4. Has there been any change in postable water available in your your community since FFS intervention? Please explain. (EQ1, 3). a) What do you do when potable water is not accessible in the nearby area? b) How would availability of potable water in your place of origin affect your decision to return? Sanitation 5. Has there been any change in access to sewerage services in your community since FFS intervention? Please explain. (EQ1, 3) a) Are the sewerage services adequate for your community needs? Please explain. b) How would availability of sanitation services in your place or origin affect your decision to return? Local Authorities’ Ability to Manage Public Infrastructure 6. What is your perception of local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) a) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? b) What would you suggest that the local authorities can do better to improve service delivery amongst IDPs? 239 General Question 7. In future, would you like stay in your current location or move to your place of origin? Please explain. 240 FOCUS GROUP DISCUSSION FORM TARGET GROUP: RELIGIOUS AND ETHNIC MINORITIES (IDPs) Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. IDPs), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? ☐ YES ☐ NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basics services available in your area. Electricity 1. Has there been any change in access to electricity for households in your community since FFS intervention? Please explain (EQ1, b). a) a) What is the difference for you between a day with electricity and a day without? How does it change in routine? b) b) How would availability of electricity affect your return to your place of origin? Would its absence factor into your decision to return? Education 241 2. Has there been any change in access to schools for school-going children in your community since FFS intervention? Please explain. (EQ1, 3) c) Do all the children living in this community have the opportunity to attend school? What are the challenges? d) How is your children’s education affected if there is no school in nearby? e) How would availability of schools in your place of origin affect your decision to return? Health 3. Has there been any change in access to health services for your community since FFS intervention? Please explain. (EQ1, 3). a) What do you do when you or any member of your community gets sick? What do you do when the treatment you seek is not available in a nearby hospital or other health facility? b) What are the main barriers faced in accessing health services? c) (For women) Do you need approval of your husband/father to access health services? d) Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? e) How would availability of health services in your place of origin affect your decision to return? Water Supply 4. Has there been any change in access to potable water for your community since FFS intervention? Please explain. (EQ1, 3) a) What do you do when potable water is not available in your area? b) How would availability of potable water in your place of origin affect your decision to return? Sanitation 5. Has there been any change in access to sewerage services for your community since FFS intervention? Please explain. (EQ1, 3). a) Are the sewerage services adequate for your community needs? Please explain. b) How would availability of sanitation services in your place or origin affect your decision to return? Local Authorities’ Ability to Manage Public Infrastructure 6. What is your perception of the local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) a) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? b) What would you suggest that the local authorities can do better to improve service delivery amongst IDPs? General Question 242 7. In future, would you like stay in your current location or move to your place of origin? Please explain. Would you feel safe going back to the communities? Please explain. 243 FOCUS GROUP DISCUSSION FORM TARGET GROUP: RETURNEES Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. Returnees), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? ☐ YES ☐ NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity 244 1. Has there been any change in access to electricity for households in your community since/due to FFS interventions? Please explain. (EQ1, 3) a) What is the difference for you between a day with electricity and a day without? How does it change your routine? b) What is the average of total electricity provision per day? Is it constant or vary depend on the season? c) Tell us about the process you had to go through to get your house connected to the electricity grid? d) Are there any barriers preventing households from this community from accessing electricity? Please explain. e) How did availability of electricity affect your decision to return? Would its absence factor into your decision to leave? Please explain. f) Do you depend on other sources of electricity providers, if so, what are they and for how many hours? Education 2. Has there been any change in access to schools for school-going children in your community since interventions? Please explain. (EQ1, 3). a) Do all the children living in this community have the opportunity to attend school? b) How is your children’s education affected if there is no school nearby? What are the challenges? c) How did the availability of schools affect your decision to return? Would the absence of schools in your current location factor into future decision to leave? Health 3. Has there been any change in access to health services for your community since FFS interventions? Please explain. (EQ1, 3). a) What do you do when you or any member of your community gets sick? b) What do you do when the treatment you seek is not available in nearby hospital or other health facility? c) (For women) Do you need approval of your husband/father to access health services? d) Have you observed any changes in the type of services offered in the hospitals or other health facilities since you returned to your area? e) How did availability of health services affect your decision to return? Would its absence factor into future decisions to leave? Water Supply 4. Has there been any change in a) What do you do when potable water is not access to potable water in your available in nearby? community since FFS b) How did availability of potable water affect your interventions? Please explain. decision to return? Would its absence factor into 245 (EQ1, 3) future decision to leave? Please explain. c) What is the average number of hours of water supply per day? Sanitation 5. Has there been any change in a) Are the sewerage services adequate for your access to sewerage services in community needs? Please explain. your community after your return? b) How did availability of sanitation services affect Please explain. (EQ1, 3) your decision to return? Would its absence in the future factor into future decision to leave? Please explain. Local Authorities’ Ability to Manage Public Infrastructure 6. What is your perception of the local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) a) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? b) What would you suggest that the local authorities can do better to improve service delivery amongst returnees? General Question 7. What are the main reasons why you returned to this area? What are the main reasons why you decided to remain in your homes or why were you displaced after 2014? Please explain. 8. What are your thoughts around your future location of residence if you think about the options of staying here or moving somewhere else? Please explain. FOCUS GROUP DISCUSSION FORM TARGET GROUP: RELIGIOUS AND ETHNIC MINORITIES93 (RETURNEES) Midterm Performance Evaluation of USAID/Iraq FFS Program INTRODUCTION 93 29 Upon local expert’s advice using political affiliated groups renders itself to be a very sensitive selection criteria and would raise insecurity fears/concerns among potential FGD participants. 246 Script I am a representative of -------, an independent consulting firm commissioned to collect data for Midterm Performance Evaluation of the USAID/Iraq Funding Facility for Stabilization Programs (FFS) in Iraq by USAID. The purpose of this discussion is to better understand FFS activities with regards to access and quality of services and factors contributing or impeding program delivery in the targeted areas. The discussion focuses on the affected population and beneficiaries (i.e. Returnees), to understand how they arrived at the decision to return and/or remain in their communities. The findings will assist decision makers to draw lessons from the IDP/returnee experience and this knowledge will be used to inform future programming. The discussion will last approximately (1) hour (30) minutes and your responses will included in the midterm evaluation report. Your responses are valuable to us and shall remain confidential. Consent If you wish not to stay for the entire focus group discussion, you may leave at any time. Do I have your consent to proceed with the interview? ☐ YES ☐ NO [Allow only those who have verbally consented to proceed with the focus group discussions.] Questions Prompts General Introduction: I will now ask you some questions about the basic services available in your area. Electricity 1. Has there been any change in access to electricity for households in your community since FFS interventions? Please explain. (EQ1, b). a) What is the difference for you between a day with electricity and a day without? How does it change in routine? b) What is the average of total electricity provision per day? Is it constant or vary depend on the season? c) Tell us about the process you had to go through to get your house connected to the electricity grid. d) Are there any barriers preventing households from this community from accessing electricity? Please explain. e) How did availability of electricity affect your decision to return or remain in this community? 247 Would its absence factor into your decision to leave? Please explain. f) Do you depend on other sources of electricity providers, if so, what are they and for how many hours? Education 2. Has there been any change in access to schools for school-going children in your community since FFS interventions? Please explain. (EQ1, 3) a) Do all the children living in this community have the opportunity to attend school? b) How is your children’s education affected if there is no school in nearby? What are the challenges? c) How did availability of schools affect your decision to return or remain in this community? Would the absence of schools factor into future decision to leave? Please explain. Health 3. Has there been any change in access to health services for your community since FFS interventions? Please explain. (EQ1, 3). a) What do you do when you or any member of your community gets sick? b) What do you do when the treatment you seek is not available in nearby hospital or other health facility? c) (For women) Do you need approval of your husband/father to access health services? d) Have you observed any changes in the type of services offered in the hospitals or other health facilities in this area? e) How did availability of health services affect your decision to return or remain in this community? Would its absence factor into future decision to leave? Please explain. Water Supply 4. Has there been any change in access to potable water in your community since FFS interventions? Please explain. (EQ1, 3) a) What do you do when potable water is not available in nearby? b) How did availability of potable water supply affect your decision to return or remain in your community? Would its absence factor into your future decisions to leave? c) What is the average number of hours of water supply per day? Sanitation 5. Has there been any change in a) Are the sewerage services adequate for your access to sewerage services in community needs? Please explain. your community since FFS b) How did availability of sewerage and sanitation interventions? Please explain. services affect your decision to return or remain in (EQ1, 3) this community? Would its absence in the future factor into future decisions to leave? Please explain. Local Authorities’ Ability to Manage Public Infrastructure 248 6. What is your perception of the local authority’s ability to manage the public infrastructure and municipal services (electricity, health, education, water and sanitation) in your community? Please explain. (EQ4) a) What is your opinion of the local authority’s capacity to provide basic services? Do the services provided by your local authority meet your basic needs? Has this changed over time? b) What would you suggest that the local authorities can do better to improve service delivery amongst returnees? General Question 7. What are the main reasons why you returned to this area? What are the main reasons why you decided to remain in your homes or why were you displaced after 2014? 8. What are your thoughts around your future location of residence if you think about the options of staying here or moving somewhere else? Do you feel safe living in this community? Please explain. POPULATION BASED SURVEY Midterm Performance Evaluation of USAID/Iraq FFS Programs * NOTES TO THE ENUMERATOR CODING A1 Enumerator ID Enter text here A2 Supervisor Enter text here A3 Date DD/MM/YYYY A4 Province ☐ Anbar ☐ Salah al-Din ☐ Ninewa A5 Intervention area Enter text here A6 Name of Location/ Neighborhood/Block Enter text here A7 Grid Cell No Enter text here A8 HH No in Cell (1-24) Enter text here A9 Enter text here A10a HH GPS (Longitude) Enter text here 249 A10b HH GPS (Latitude) Enter text here INFORMED CONSENT 1 Introductory scripts Assalam-o-Alaikum. My name is ______________________________, and I am working with ------. I am conducting a survey for IBTCI under the auspices of USAID to evaluate the stabilization related services in Anbar, Ninewa, and Salah al-Din provinces to support Iraqis return and resettle back in their homes and hometowns. We value your feedback and would appreciate your participation in this survey. This information will help the Government of Iraq and its development partners to further improve the stabilization programming for returning families. The survey will take about 30-45 minutes to complete. Whatever information you provide will be kept strictly confidential and anonymous. Participation in this survey is voluntary and you can choose not to answer any individual question or all of the questions. You can stop the survey at any time. However, we hope that you will participate in this survey since your views are important. Your responses are valuable to us and shall remain confidential. 2 Consent Do I have your consent to proceed with the survey? ☐ YES ☐ NO SECTION B: IDENTIFICATION CAPI ANONYMIZE NUMBERS B1 Sex of Head of HH ☐ Male ☐ Female B2 Respondent No. (1-24 in each grid cell) B3 Sex ☐ Male ☐ Female B4 Age B5 Relationship of respondent to head of HH ☐ Self ☐ Spouse ☐ Parent ☐ Child ☐ Sibling ☐ Other: B6 Total number of persons living in this household Total Male Female B7 Marital Status ☐ Married ☐ Unmarried ☐ Divorced ☐ Widowed ☐ Others: 250 B8 Ethnic Origin ☐ Arab ☐ Assyrian ☐ Kurd ☐ Turkmen ☐ Shabak ☐ Others: B9 Languages Spoken ☐ Arabic ☐ Kurdish ☐ Shabaki ☐ Others: B10 Religion ☐ Muslim￾Shia ☐ Muslim - Sunni ☐ Christian ☐ Yazidi ☐ Kakai ☐ Others: B11 Highest Education ☐ None ☐ Elementary ☐ Junior High ☐ Secondary ☐ Higher ☐ Religious B12 Current Status ☐ IDP ☐ Returnee ☐ Never Left Home B13 Family Status 94 ☐ Nuclear ☐ Joint B14 Anyone in HH living with Disability ☐ Yes ☐ If Yes, How Many: ☐ No B15 Type of Shelter ☐ Rented house ☐ Religious building ☐ School building ☐ Unfinished/abandoned building ☐ Informal settlement ☐ Other SECTION C: ELECTRICITY Changes to access and availability of electricity supply (EQ1.1) C1 What is the source of electricity connected to your home? If not Option 1 (Public network) skip >>C8) [Related to access] ☐Public electricity network ☐Private Generator ☐Shared Generator ☐Other ☐None 94 Note: Nuclear Family = Family that consists of a mother, father and their children. Does not include grandparents, uncles, and aunts. Joint Family = Family that consists of parents, and close relatives/multiple generations 251 C2 If Option 1(Public electricity) in C1, when did you get connected to the public electricity network? (Please choose one only) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know C3 If the public electricity services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? ☐Yes ☐No ☐Don’t Know C4 Has the public electricity currently in use been restored/repaired after the liberation? ☐Yes ☐No ☐Don’t Know C5 Did availability of electricity affect your decision to return (If respondents are returnees)? ☐Yes ☐No ☐Don’t Know C6 Would availability of electricity at your place of origin affect your decision to return (If respondents are IDPs)? ☐Yes ☐No ☐Don’t Know C7 Would the absence of electricity in the future factor into your decision to leave your community? ASK ALL ☐Yes ☐No ☐Don’t Know Barriers to accessing electricity (EQ1.2) C8 Although you said you don’t have or are not connected to public electricity network in your home (in C1), is there public electricity supply/grid in your neighborhood/area? ☐Yes ☐No ☐Don’t Know C9 If yes to C8, what is the main reason your home does not have electricity? MARK ALL THAT APPLY [Refers to Access Barriers] ☐ Connection fee is unaffordable ☐ The structure/dwelling does not meet the standards suitable to have electricity connection ☐ Could not afford electricity monthly subscription ☐ Application for connection is delaying ☐Prefer private or shared generator ☐ Other (Specify) 252 C10 What best describes the process of getting connected to electricity? (Please choose one only) ☐Very difficult ☐Somewhat difficult ☐Easy ☐Very Easy ☐Don’t know C11 What best describes the cost of electricity? (Please choose one only) ☐Affordable ☐Not affordable ☐Don’t know C12 On the whole, do you agree or disagree with the following statement: Provision of electricity meets the needs of your household? (Please choose one only) ☐Strongly agree ☐Agree ☐Neutral ☐Disagree ☐Strongly disagree Perceptions of the quality of electricity supply (EQ2) C13 For how many hours per day is/was electricity available at your home? (Please choose one only) Now Prior to FFS intervention (Before 2015) ☐All day (24 hours per day) ☐Most of the day (13-23 hours per day) ☐Some of the day (4-12 hours per day) ☐Rarely (1-3 hours per day) ☐Never (0 hours per day) ☐All day (24 hours per day) ☐Most of the day (13-23 hours per day) ☐Some of the day (4-12 hours per day) ☐Rarely (1-3 hours per day) ☐Never (0 hours per day) C14 How many days a week is/was electricity available at your home? Now Prior to FFS intervention (Before 2015) 253 (Please choose one only) ☐Every day (7 days a week) ☐Most days (4-6 days per week) ☐Some days (2-3 days per week) ☐Rarely (1 day per week) ☐Never (0 days per week) ☐Every day (7 days a week) ☐Most days (4-6 days per week) ☐Some days (23 days per week) ☐Rarely (1 day per week) ☐Never (0 days per week) Perceptions of the effectiveness of administration of electricity supply (EQ4) C15 How satisfied are you with the local authority’s or utility company’s administration of electricity supply services in the following areas? Maintaining infrastructure Response to power failures Customer billing Responsiveness to complaints Checking meters on a regular basis Ability to process new applications for connection [ENUMERATOR READ OUT] For each mark one whether ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐Don’t know ☐N/A SECTION D1: POTABLE WATER SUPPLY Changes to access and availability of portable water (EQ1.1) D1.1 What is the main source of potable water for members of your household? (Please choose one only) Now Prior to FFS intervention (Before 2015) 254 (Refers to access) ☐Piped into dwelling ☐Piped into compound ☐Community/neighborhood pipes ☐Hand-pump/borehole ☐Protected well ☐Open unprotected well ☐River/stream/lake ☐Water tanker/truck/ bowser ☐Bottled water ☐Other ☐Piped into dwelling ☐Piped into compound ☐Community/neighborh ood pipes ☐Hand￾pump/borehole ☐Protected well ☐Open unprotected well ☐River/stream/lake ☐Water tanker/truck/ bowser ☐Bottled water ☐Other D1.2 If source is piped water, is the water metered? (applies to options 1-3 in D1.1) ☐Yes ☐No ☐Don’t Know D1.3 Since when has the source of potable water been available for your household use? (Please choose one only) (Related to period service was accessed vs baseline) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know D1.4 If the water supply services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? ☐Yes ☐No ☐Don’t Know D1.5 Have the water supply services currently in use been restored/repaired after the liberation? ☐Yes ☐No ☐Don’t Know D1.6 Where is the water source located? (Please choose one only) ☐Into own dwelling skip >>A3-1.8 ☐Into own yard/plot skip >>A3-1.8 ☐Elsewhere D1.7 How long (in minutes) does it take you to walk from your home to reach this main source of safe drinking water [ONEWAY] [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] Minutes:……………. 255 D1.8 On average, how many hours per day is water supply available to your household from this source? [ Relates to continuity of supply/availability or nonavailability]] No of hours: ……………..(0-24 hours) D1.9 How many pails/buckets of water do you use in a day in your home? (1 bucket=18 Liters) [LOCAL NAME FOR BUCKET] [ Relates to adequacy to consumption needs] # of pails/buckets/day……………….. D1.10 Is the amount of potable water you get on a daily basis adequate for your household needs? (Please choose one only) [ Relates to relevance to consumption needs] ☐Not adequate at all ☐Not adequate, but we can manage ☐Adequate ☐Not sure D1.11 Did availability of potable water affect your decision to return to your community (if respondents are returnees)? ☐Yes ☐No ☐Don’t Know D1.12 Would availability of potable water at your place of origin affect your decision to return (If respondents are IDPs)? ☐Yes ☐No ☐Don’t Know D1.13 Would the absence of portable water in future factor into your decisions to leave? ASK ALL ☐Yes ☐No ☐Don’t Know Barriers to access (EQ 1.2) D1.14 Who usually goes to collect water for your household? [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] (Please choose one only) ☐Adult woman (15+ years) ☐Adult man (15+ years) ☐Female child <15yrs ☐Male child <15yrs D1.15 How would you rate the difficulty of obtaining potable water? (Likert scale) ENUMERATOR : These questions should be asked from the person responsible for getting water] [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] (Please choose one only) ☐Very difficult ☐Somewhat difficult ☐Not difficult ☐Very Easy ☐Don’t know 256 D1.16 What are the reasons for the unavailability? [ONLY THOSE WHO ANSWERED –ELSEWHERE IN [D1.6] MARK ALL THAT APPLY ☐Damaged/vandalism of infrastructure ☐Drying of water source ☐Poor design ☐No maintenance of water point ☐Other Perceptions of the quality of water (EQ2) D1.17 How would you describe the taste of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) ☐Good ☐Good ☐Salty/saline ☐Salty/saline ☐Greasy ☐Greasy ☐Slippery ☐Slippery ☐Sour ☐Sour ☐Metallic ☐Metallic ☐Chemical ☐Chemical ☐Not sure ☐Not sure D1.18 How would you describe the smell of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) ☐Good ☐Foul smell ☐Not sure ☐Good ☐Foul smell ☐Not sure D1.19 How would you describe the clarity/color of water accessible to you? (Please choose one only) Now Prior to FFS intervention (Before 2015) ☐Clear ☐Turbid ☐Not sure ☐Clear ☐Turbid ☐Not sure D1.20 During the last one month have you or any member of your household been sick due to the drinking water? ☐Yes ☐No ☐Don’t Know D1.21 If you or members of your family have suffered from any waterborne diseases in the last 12 months, do you know which of the following you or your family Now (Last 12 months) Prior to FFS intervention (Before 2015) 257 members had? Can you identify any you had before the FFS intervention? MARK ALL THAT APPLY DO NOT READ LIST ☐None ☐Diarrhea ☐Bilharzia ☐Cholera ☐Dysentery ☐DK ☐Other (specify) ☐None ☐Diarrhea ☐Bilharzia ☐Cholera ☐Dysentery ☐DK ☐Other (specify) D1.22 Do you treat your water to make it safe to drink? If No skip >>D1.24 ☐Yes ☐No ☐Don’t Know D1.23 If yes, what do you usually do to treat water to make it safe to drink? (Please choose one only) ☐Boil ☐Add bleach/chlorine ☐Strain it through cloth ☐Use water filter, Filtration (ceramic, sand, etc.) ☐Purification tablets ☐Solar disinfection ☐Other (specify) Perceptions of the effectiveness of administration of water supply (EQ4) D1.24 How satisfied are you with the local authority’s or utility company’s administration of water supply services in the following areas? Maintaining infrastructure Response to service interruptions Regular billing Availability of customer service and complaints hotline Checking meters on a regular basis Ability to process new applications for connection [ENUMERATOR READ OUT] For each mark one whether ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐Don’t know ☐N/A SECTION D2: SANITATION Changes to access and availability of sanitation services (EQ1.1) 258 D2.1 How does your household dispose of solid waste materials/garbage? (Please choose one only] ☐Household refuse bins ☐Rubbish pits ☐Collection from dwelling by municipality ☐Communal refuse bins or pits ☐Waste disposed in the open ☐Burying ☐Other (specify) ☐None D2.2 What type of waste water/sewerage system is your household connected to? (Please choose one only) ☐Public sewer network with covered canal ☐Public sewer network with uncovered canal ☐Septic tank ☐Other (Specify) ☐None D2.3 Since which year have the sanitation services (sewerage) been available for your household use in this area? (Related to period service was accessed vs baseline) (Please choose one only) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know D2.4 If the sewerage services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? ☐Yes ☐No ☐Don’t Know D2.5 Has the sewerage system been restored/repaired after the liberation? ☐Yes ☐No ☐Don’t Know D2.6 Did availability of sanitation services affect your decision to return (if respondents are returnees)? ☐Yes ☐No ☐Don’t Know D2.7 Would availability of sanitation services at your place of origin affect your decision to return (If respondents are IDPs)? ☐Yes ☐No ☐Don’t Know D2.8 Would the absence of sanitation services in the future factor into your decisions to leave? ASK ALL ☐Yes ☐No ☐Don’t Know Perceptions of the quality of sanitation services (EQ2) 259 D2.9 How frequent are your complaints on the nonfunctionality of the wastewater/sewerage system? ☐Frequent ☐Not frequent ☐Never ☐Don’t Know Perceptions of the effectiveness of administration of sanitation services (EQ4) D2.10 How satisfied are you with the local authority’s or utility company’s administration of sanitation services regarding the following areas? Maintaining sanitation infrastructure Connecting households to sewer Responsiveness to non-functioning/ blockages and other disruptions on the sewer Collection of solid waste Collection of human waste if not connected to sewer [ENUMERATOR READ OUT] For each mark one whether ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐Don’t know ☐N/A SECTION E: HEALTH Changes to access and availability of health services (EQ1.1) E1 How far in kilometers (Km) is the nearest government health facility to your household compared to before the FFS period? Now Prior to FFS intervention (Before 2015) Km…….. Km…….. E2 Since when or in which year have the health services been available for your household use in this area? (Related to period service was accessed vs baseline) (Please choose one only) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know E3 If the health services were available prior to FFS interventions (before 2015), were they damaged during the ISIS occupation? ☐Yes ☐No ☐Don’t Know E4 Were the health services restored/repaired after the liberation? ☐Yes ☐No ☐Don’t Know 260 E5 Did availability of health services affect your decision to return (if respondents are returnees)? ☐Yes ☐No ☐Don’t Know E6 Would availability of health services at your place of origin affect your decision to return (If respondents are IDPs)? ☐Yes ☐No ☐Don’t Know E7 Would the absence of health services in the future factor into your decisions to leave? ASK ALL ☐Yes ☐No ☐Don’t Know Barriers to access (EQ 1.2) E8 Have you sought health services at the nearest facility and have not been assisted? Now Prior to FFS intervention (Before 2015) If Yes Skip >>A4.8 ☐Yes ☐No ☐Don’t Know ☐Yes ☐No ☐Don’t Know E9 If not, what are the reasons for not being assisted? MARK ALL THAT APPLY (Refers to barriers to access and utilization) Now Prior to FFS intervention (Before 2015) ☐Health facility did not offer the services ☐Facility was closed, ☐The wait was too long, ☐It was cost-prohibitive ☐Others: __________ ☐Health facility did not offer the services ☐Facility was closed, ☐The wait was too long, ☐It was cost prohibitive ☐Others: __________ Perceptions of the quality of health services (EQ2) E10 How satisfied are you with the following at your nearest public health facility? [ENUMERATOR READ OUT] Now Prior to FFS intervention (Before 2015) 261 For each one mention whether i. Quality of pediatric/child health services ii. Quality of surgical services iii. Quality of maternal health services iv. Quality of general outpatient services v. Quality of emergency services vi. Quality of infectious diseases treatment vii. Quality of mental health care viii. Waiting time to be attended to at the facility ix. General cleanliness of the facility ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐ No opinion because I did not receive the service ☐No opinion because the service is not offered. ☐ Don’t Know ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐No opinion because I did not receive the service ☐No opinion because the service is not offered. ☐ Don’t Know Perceptions of the effectiveness of administration of Health Services (EQ4) E11 How satisfied are you with the local authority’s administration of health services in the following areas? Ability to maintain health infrastructure [ENUMERATOR READ OUT] For each mark one whether ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied Ability to respond to outbreaks Ability to provide the required number of health workers ☐Very dissatisfied ☐Don’t know ☐N/A SECTION F: EDUCATION Changes to access and availability of education services (EQ1.1) F1 Is there any member of your households who is of school-age? If NO skip>> G1 ☐Yes ☐No ☐Don’t Know F2 Are the school- aged children attending school? If NO skip>> F9 ☐Yes ☐No ☐Don’t Know F3 If yes, how many children in your household are attending school? …………Male ………… Female F4 What type of school do the children attend? ENUMEATOR ENTER FOR EACH TYPE AND LEVEL Level Type Number of children 262 Primary ☐Government ☐Private non-religious ☐Islamic school ☐Other religious ☐Other Secondary ☐Government ☐Private non-religious ☐Islamic school ☐Other religious ☐Other Tertiary ☐Government ☐Private non-religious ☐Islamic school ☐Other religious ☐Other F5 Since when or in which year have the school services been available to your household in this area? (Please choose one only) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know F6 In general, how would you compare the following access dimensions for education between now and prior to FFS interventions? Now Prior to FFS intervention (Before 2015) a. Number of days per week school is open b. Number of hours per day school is open c. Distance in Kilometers from home Km:………………….. Km:…………… … d. Frequency of disruptions during the academic calendar ☐Frequent ☐Not frequent ☐Never ☐Don’t Know ☐Frequent ☐Not frequent ☐Never ☐Don’t Know e. Prolonged school closures due to insecurity ☐Frequent ☐Not frequent ☐Never ☐Frequent ☐Not frequent ☐Never 263 ☐Don’t Know ☐Don’t Know F7 Did availability of education services affect your decision to return (if respondents are returnees)? ☐Yes ☐No ☐Don’t Know F8 Would availability of education services at your place of origin affect your decision to return (If respondents are IDPs)? ☐Yes ☐No ☐Don’t Know F9 Would the absence of education services in the future factor into your decisions to leave? ASK ALL ☐Yes ☐No ☐Don’t Know Barriers to access (EQ 1.2) F10 If no (A5-2), what are the reasons for not attending school (barriers)? ☐It is far away ☐School in restricted area ☐Insecurity and personal safety issues ☐Cost-prohibitive ☐Poor quality of schools ☐Illness or disability of student ☐Illness or disability of family member ☐School conflict with beliefs ☐Others: ________________________ Perceptions of the quality of education services (EQ2) F11 Does the school the children attend have desks where every child can sit? ☐Yes ☐No ☐Don’t Know F12 Does the school the children attend have a sufficient number of classrooms? ☐Yes ☐No ☐Don’t Know F13 Does the school the children attend have water and sanitation facilities? ☐Yes ☐No ☐Don’t Know F14 Does the school the children attend have electricity? ☐Yes ☐No ☐Don’t Know Perceptions of the effectiveness of administration of Education Services (EQ4) 264 F15 How satisfied are you with the local authority’s administration of education services with regard to the following areas? Ability to maintain school infrastructure Ability to provide teaching and learning materials Ability to conduct school inspections ENUMERATOR READ OUT] For each mark one whether ☐Very satisfied ☐Somewhat satisfied ☐Somewhat dissatisfied ☐Very dissatisfied ☐Don’t know ☐N/A Section B: IDPS or Returnees G1 Have you always lived in this area since birth? ☐Yes ☐No ☐Don’t Know G2 If no, how long have you lived in this area? No of years G3 In which year did you move away from this area the last time? (Please choose one only) (refers to period of displacement) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know ☐ N/A G4 In which year did year you return to this area? (Please choose one only) (refers to period of resettlement) ☐ 2019 ☐ 2018 ☐ 2017 ☐ 2016 ☐ 2015 ☐ Before 2015 ☐ Don’t know ☐ Never left G5 Are you planning to stay here permanently? ☐Yes ☐No ☐Don’t Know 265 G6 What are/were the main reasons for your displacement? [ENUMERATOR : THIS APPLIES TO RETURNEES AND IDPs] MARK ALL THAT APPLY ☐Lack of basic services (e.g. water, electricity, health, education, sewerage services etc.) in area of origin ☐Other Specify ☐Ongoing fight/lack of security in area of origin ☐Security in area of displacement ☐House/property destroyed/damaged/demolished ☐Lack of jobs back home ☐Fear of security actors in area of origin ☐Lack of services back home ☐Fear of reprisal acts/violence back home ☐Fear of harassment/discrimination back home ☐Better services in place of displacement ☐Fear of ISIS returning to area of origin ☐Availability of jobs in place of displacement G7 What are the most critical factors that influenced you to return? [ENUMERATOR : THIS APPLIES TO RETURNEES AND IDPs] MARK ALL THAT APPLY ☐Improvements in security ☐Rehabilitation of residences and buildings/shelter ☐Availability of health services ☐Availability of education services ☐Availability of water and sanitation services ☐Availability of electricity ☐Availability of business grants ☐Employment and livelihood opportunities ☐Availability of markets ☐Police stations ☐Manufacturing enterprises ☐Roads and Bridge projects ☐Humanitarian support ☐This is the community where I grew up ☐This is where my family lives ☐I have a strong support system here ☐I own property here ☐None ☐Other (specify 266 G8 What is your main source of household income MARK ALL THAT APPLY ☐Street vending ☐Casual labor ☐Salaried employment /small business enterprise ☐Large business enterprise ☐Remittances ☐Sale of Assets ☐Agriculture Assets ☐ Other (Please specify) G9 Are your most important needs met by the basic services that are provided by the local authority? ASK ALL-RETUNEES AND IDPS [ONLY ONE MENTION] ☐Not at all ☐Not very much ☐Mostly met ☐Completely met ☐ Not Sure G10 How would you compare your HH average Monthly income now to the period before returning/coming to this area? [ONLY ONE MENTION] Now Before Returning/Displacement ☐ IQD300,000 ☐ IQD300,00 ☐ Don’t Know ☐ N/A ☐ Don’t Know ☐ N/A G11 Overall is life improving for people who returned? [ENUMERATOR : THIS APPLIES TO RETURNEES] MARK ALL THAT APPLY ☐Not at all ☐Not very much ☐Mostly met ☐Completely met ☐Not Sure G12 Social Cohesion and Security Were you afraid to return to your community? [ENUMERATOR : THIS APPLIES TO RETURNEES] ☐Yes ☐No ☐ Not sure 267 G13 If yes (B1-14), which of these were you worried about? MARK ALL THAT APPLY ☐Discrimination ☐Retaliation ☐Tribal/Ethnic tensions ☐A lack of security ☐Explosive hazards ☐Personal/family distress ☐Reintegrating would be too difficult ☐Other G14 Since you returned have you encountered any of these? [ENUMERATOR : THIS APPLIES TO RETURNEES] Discrimination ☐Yes ☐No ☐ Not sure Retaliation ☐Yes ☐No ☐ Not sure Tribal/Ethnic tensions ☐Yes ☐No ☐ Not sure A lack of security ☐Yes ☐No ☐ Not sure Explosive hazards ☐Yes ☐No ☐ Not sure Personal/family distress ☐Yes ☐No ☐ Not sure Reintegrating would be too difficult ☐Yes ☐No ☐ Not sure Other ☐Yes ☐No ☐ Not sure Thank you very much for your time. The information you have provided will be very useful in evaluating the program and improve future programming 268 LIVELIHOODS QUANTITATIVE SURVEY Midterm Performance Evaluation of USAID/Iraq FFS Programs (June 2019) Form Number TO BE ASSIGNED NOTES TO THE ENUMERATOR CODING 1 Interviewer Code Enter text here 2 Supervisor Code Enter text here 3 Date Enter text here 4 province Enter text here 5 Intervention area Enter text here 6 Areas ☐ Urban ☐ Rural INFORMED CONSENT 1 Introduce yourself Hello. My name is ______________________________, and I am working with ______. I am conducting a livelihood survey for IBTCI under the auspices of USAID to evaluate stabilization related services in Anbar, Ninewa, and Salah al-Din provinces to support Iraqis return and resettle back in their homes and hometowns. We value your feedback and would appreciate your participation in this survey. This information will help the Government of Iraq and its development partners to further improve stabilization programming for the returning families. The survey usually takes 30 minutes to complete. Whatever information you provide will be kept strictly confidential. Participation in this survey is voluntary and you can choose not to answer any individual question or all of the questions. You can stop the survey at any time. However, we hope that you will participate in this survey since your views are important. Your responses are valuable to us and shall remain confidential. 2 Consent Do I have your consent to proceed with the survey? [In the case of No, call the next individual] ☐ YES ☐ NO IDENTIFICATION (Respondent) 269 1 Sex ☐ Male ☐ Female 2 Age Enter number of full years here 3 Head of HH Sex ☐ Male ☐ Female 4 Relationship of respondent to HH Head ☐ Self ☐ Parent ☐ Child ☐ Sibling ☐ Other: 5 Number of persons living in this household Total Male Female 6 Marital Status ☐ Married ☐ Unmarried ☐ Divorced ☐ widowed ☐ Others 7 Ethnic Origin ☐ Arab ☐ Assyrian ☐ Kurd ☐ Turkmen ☐Shabak ☐ Others: 8 Languages Spoken ☐ Arabic ☐ Kurdish ☐ Shabak ☐ Others: 9 Religion ☐ Muslim￾Shia ☐ Muslim￾Sunni ☐ Christian ☐ Yazidi ☐ Kakai ☐ Others: 10 Highest Education ☐ None ☐ Elementary ☐ Junior high ☐ Secondar y ☐ Tertiary ☐ Religious 11 Status before receiving livelihood support ☐ IDP ☐ Returnee ☐ Refugee ☐ Never Left Home 12 Current Status ☐ IDP ☐ Returnee ☐ Refugee ☐ Never Left Home 270 13 Family Status 31 ☐ Nuclear ☐ Joint 14 HH Average Monthly income ☐ IQD 300,001 ☐ Don’t Know 31 Note: Nuclear Family = Family that consists of a mother, father and their children. Does not include grandparents, uncles, and aunts. Joint Family = Family that consists of parents, and close relatives/multiple generations 15 Anyone in HH living with Disability ☐ Yes ☐ If Yes, How Many: ___ ☐ No 16 If yes, is the livelihood beneficiary disabled ☐ Yes ☐ No 17 Living in the area for years months SECTION 1: LIVELIHOODS INTERVENTIONS -Cash for Work 1.1. When you were first enrolled in the Cash for Work activity? Year 2015 ______ Year 2016 ______ Year 2017 ______ Year 2018 ______ 1.2. How did you know about Cash for Work activity? Tick one option ☐Informed by friends/relatives ☐Informed by the Government ☐Contacted by the contractors ☐Public announcement ☐Other, Specify ____________ 1.3. How many times have you participated in a Cash for Work activity since 2015? ☐ Once ☐ Twice ☐ Three times and more 1.4. Did you complete the last Cash for Work activity you were enrolled in?(If no skip to 1.5) ☐ Yes ☐ No ☐ Not sure 271 1.5. If yes, when did you complete the last Cash for Work activity you were enrolled in? Year 2015 ______ Year 2016 ______ Year 2017 ______ Year 2018 ______ 1.6. How much money did you receive from the Cash for Work activity? IQD:____________ 1.7. How long did you work for? ☐ Less than 1 week ☐ 1-4 weeks ☐5-8 weeks ☐9-12 weeks ☐ More than 12 weeks 1.8. Did you receive the payment from the Cash for Work activity on time according to agreement schedule? ☐ Yes ☐ No 1.9. Did you have any other sources of income when you received Cash for Work support? (If no, skip to 1.9) ☐ Yes ☐ No 1.10. If yes, what were these other sources of income? Please mention up to three sources. 1. 2. 3. 1.11. Was the money you received from the Cash for Work activity adequate to support your family’s needs? ☐ Yes ☐ No ☐ Not sure 1.12. How did you spend the money from the Cash for Work activity? Tick all that apply ☐Spent on household basic needs ☐Bought assets ☐Started business ☐Saved the money ☐Other 1.13. How much money did you spend last month on your household’s basic needs? Total spent on basic need: __________IQD Of which, spent on food: __________IQD 272 1.14. What was the contribution of your income earned from Cash for Work activity to your household needs? (ENUMERATOR TO READ OUT THE RESPONSES) ☐Primary source of household expenditure (contribution greater than 50%) ☐ Significant contributor but not primary source of household expenditure (contribution greater that 20% but less than 50%) ☐Partial source of household expenditure (contribution less than 20%) ☐Not a source of household expenditure ☐Not sure 1.15. Did you face any challenges when participating in the Cash for Work activity? ☐ Yes ☐ No 1.16. If yes, can you tell us what type of challenges you faced? (Check all that apply) ☐Delay in payment ☐ Insecurity ☐High transportation costs ☐Physical harassment ☐Sexual exploitation ☐Money extortion ☐Nepotism/favoritism ☐Others: (specify____) 1.17. Did the Cash for Work support influence your decision to return to this community? ☐ Yes ☐ No ☐ Not sure☐ NA 1.18. Did the Cash for Work support influence your decision to stay in this community? (For returnees only) ☐ Yes ☐ No ☐ Not sure☐ NA 1.19. How would you compare your household income now to the period before the Cash for Work support? ☐ Improved ☐Same as before ☐Worse 1.20. How has your participation in the Cash for Work activity changed your power to make decisions at home? (For female) ☐ Improved ☐ Same as before ☐ Worse ☐ Not sure 273 1.21. (For returnees) What are the major reasons that influenced your decision to return to this community? Mark all that apply. ⎯ ⎯ Security situation is stable Livelihood opportunities are available ⎯ Emotional obligation to return ⎯ Family/community members have returned ⎯ To secure personal housing, land or property ⎯ Basic services are available ⎯ Difficult conditions in area of displacement ⎯ Other (Specify) ________ Thank you very much for your time. The information you have provided will be very useful in evaluating the program and improve future programming ONSITE OBSERVATION CHECKLISTS CHECKLIST FOR ON-SITE OBSERVATION FOR ELECTRICITY FACILITIES- GENERATOR Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 3 Date visited 4 Enumerator Visiting System Items Satisfactory Yes No Remarks (if there is an issue) 274 Generator and Controlling System Control Panel & Wiring Breaker Assembly Strip Heaters Remote Annunciators/Alarm s Rotating Rectifier Main Rotor/Stator Phase Bar/Isolators Regulator/Wiring Starting System Batteries Load Test Battery Charger Float Rate VDC Starting Motor Alternator Output VDC Ignition Solenoids Eng. Wiring Harness/Sensors Air Intake & Exhaust System Air Filters Air Intake Assembly 275 Charge Air Cooler Exhaust Assembly Turbo Assembly Silencer/Muffler Intercooler Lubrication System Oil MTU Oil Analysis Date of Sample Oil Filters Oil Pressure PS I Crankcase Ventilation Lube System Gasket Seals Fuel System Fuel Line Connections Fuel Filters Primary/Secondary Fuel Pressure PS I Governor/Connections Set Tank/Day Tank Regulator/NG PSI Coolant System Radiator 276 Heat Exchanger Coolant Last Coolant Sample Gaskets/Seals Hoses & Connections Water Jacket Heater Temperature Fan Drive & Pulley Assembly Fan Belts Load Calculation (NFPA110) Name Plate KVA:____________ x .3 = ___________________ (30% Rating) Amps:L1_____ +L2_____ +L3______ =______ Amps ÷ 3 =______ Avg. Amps Avg. Amps:_______ x Volts:______ x 1.732 (for 3ph) ÷ 1000 =________ KVA If final KVA calculation is greater than 30% of name plate value = “Pass” If final KVA calculation is less than 30% of name plate value = “Fail” CHECKLIST FOR ON-SITE OBSERVATION FOR ELECTRICITY FACILITIES- SUBSTATION Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 4 Date visited 4 Engineer Visiting Substation: Item to be checked Normal Remarks 277 A) 400kV Overhead Lines 1) Voltage reading and current reading ☐ YES ☐NO B) Power Transformer 1) The voltage and current readings of (400kV) of the main transformers ☐ YES ☐NO 2) The voltage and current readings of (132kV) of the main transformers ☐ YES ☐NO 3) Any noise from the transformers ☐ YES ☐NO 4) Winding temperature indicator reading ☐ YES ☐NO 5) Oil temperature indicator reading ☐ YES ☐NO 6) On Load Tap Changer (OLTC) is working correctly or not ☐ YES ☐NO 7) OLTC reading ☐ YES ☐NO 8) Is there oil leak from the transformers ☐ YES ☐NO 9) Dehydrating Breather Status ☐ YES ☐NO C) 132kV Overhead Lines & Underground Cable (UGC) 1) Voltage reading and current reading Overhead Line (OHL) feeders ☐ YES ☐NO 2) Voltage reading and current reading UGC feeders ☐ YES ☐NO 3) Voltage reading and Status of Busbars/ Bus Couplers / Bus sections ☐ YES ☐NO D) Battery Chargers and Inverters 278 1) The voltage and current readings for (110V DC) Battery and Charger ☐ YES ☐NO 2) The voltage and current readings for (48V DC) Battery and Charger ☐ YES ☐NO 3) The voltage and current readings for inverters (220VAC UPS ) ☐ YES ☐NO E) Building Services 1) Lighting and Small power working conditions (Problems if any) ☐ YES ☐NO 2) Air conditioner working conditions (Problems if any) ☐ YES ☐NO 3) Fire Alarm system working conditions (Problems if any) ☐ YES ☐NO CHECKLIST FOR ON-SITE OBSERVATION TO SEWERAGE (WASTEWATER TREATMENT PLANT “WWTP”AND LIFT STATIONS) FACILTIES Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 3 Date visited 4 Engineer Visiting ON-SITE OBSERVATION FOR SANITATION FACILITIES 1. Are there any rehabilitation activities underway or all work has been completed for the intended facility? ☐ YES ☐ NO 2. What is the design capacity of the WWTP/lift station and what is the working capacity now? Design Capacity Working capacity 279 3. If the WWTP/lift station is working less than its design capacity, what are the reasons for this? ☐ Rehabilitation activities not completed yet ☐ Non-continuous power supply to the plant ☐ Staff is less than required ☐ Intermittent supplying of chemicals (coagulants, chlorine, etc.) ☐ Some equipment need spare parts ☐ Other (specify____) 4. How many villages, intervention areas, population does the WWTP/lift station served? 5. Is there any laboratory for testing the influent and the treated water quality? ☐ YES ☐ NO 6. If yes, what is frequency of water quality measurements? Daily Weekly Monthly 280 7. What are the physical, chemical and biological tests that are measured at the WWTP (pH, Color, test, TS, TDS, COD, BOD5, TOC, Fecal coliform FC, Total coliform count TC, heavy metals, Nitrate NO3, Phosphate PO4, etc.)? ☐pH ☐Color test ☐ TS ☐ TDS ☐ COD, BOD5, TOC ☐Fecal Coliform (FC) ☐Total Coliform count (TC) ☐Heavy metals (Pb, Cd, Cr, etc.) ☐ Nitrate NO3 ☐ Phosphate PO4 ☐ Other tests (to be mentioned) 8. Are these tests done on schedule? ☐ YES ☐ NO 9. Are there any units in the WWTP/lift station that don’t work (Lift pumps, distribution pumps, connection pipes, valves, air blowers, mixers, screenings, grit chambers, Primary and secondary sedimentation tanks, aeration tanks, chlorination tanks, digesters, drying beds, odor control system, etc.)? Please check those that are not working from the list. ☐ YES ☐ NO ☐Lift Pumps ☐Distribution Pumps ☐Connection Pipes ☐Valves ☐Air blowers ☐Screenings ☐Grit chambers 281 ☐Primary and secondary sedimentation tanks ☐Aeration tanks ☐Drying beds ☐Odor control system ☐Other Units (to be mentioned) 10. In case of any unit that doesn’t work what is the reason for this? (Please mark for each checked “not working” from the list in Q9) ☐Need spare parts ☐Need more time to complete rehabilitation work ☐Need qualification staff and training ☐ Other Specify____________ 11. What is the type of coagulants/flocculants used in the WTP and in what quantity/concentration (for example Alum, Ferric Chloride, etc.)? Type Quant. Conc. Acids Alkalis Others 12. What is the type of disinfectant used and in what is the quantity/concentration (for example chlorine gas, sodium or calcium hypochlorite, etc.)? Type Quant. Conc. Chlorine gas Sodium Hypochlorite Calcium Hypochlorite 13. Is there any lack in supplying the chemicals used in water purification? ☐ YES ☐ NO 14. If so why? 15. Is there any schedule for maintenance the WWTP units/lift station? ☐ YES ☐ NO 282 If yes, what is schedule? If no, what are the reasons? 16. How many tanks are working from each unit in its full capacity? How many tanks are present for each unit? (e.g., two sedimentation tanks, three filtration tanks, etc.)? Type of Tank No. Capacity Sediment Filtration Others (to be mentioned) 1. 2. 3. 4. 5. 17. How many hours the WWTP/lift station is operated per day? 18. How often the treatments units cleaned from sludge? ☐Weekly ☐Every two weeks ☐Monthly ☐At certain Turbidity limits (mentioned) 19. What is the plan for the sludge collected ☐ Disposed at landfill ☐ Reused as compost 20. Is there a contingency plan in case of accidents, fire, injuries, etc.? ☐ YES ☐ NO 283 21. Is there any Occupational Health and Safety (OHS) plan in the WWTP/lift station? ☐ YES ☐ NO 22. Is there a proper storage of chemicals used in wastewater treatment processes? ☐ YES ☐ NO 23. Is there a continuous supplying for electricity to the WWTP/lift station? ☐ YES ☐ NO If no, for how many hours in day? 24. Does the WWTP/lift station has its own electrical generator in case of national electrical grids is shut down? ☐ YES ☐ NO 25. How was the technical and management capacity of staff for the intended facility? 284 CHECKLIST FOR ON-SITE OBSERVATION TO WATER TREATMENT PLANTS FACILTIES (WTP) Midterm Performance Evaluation of USAID/Iraq FFS Programs ON -SITE OBSERVATION OF FFS COMMUNITY INFRASTRUCTURE To be completed by the data collectors following the site visit 1 Name of the facility 2 Location of the facility 5 Date visited 6 Engineer Visiting ON-SITE OBSERVATION FOR SANITATION FACILITIES 26. Are there any rehabilitation activities underway or all work has been completed for the intended facility? ☐ YES ☐ NO 27. What is the design capacity of the WTP and what is the working capacity now? Design Capacity Working capacity 28. If the WTP is working less than its design capacity, what are the reasons for this? ☐ Rehabilitation activities not completed yet ☐ Non-continuous power supply to the plant ☐ Staff is less than required ☐ Intermittent supplying of chemicals (coagulants, chlorine, etc.) ☐ Some equipment need spare parts ☐ Other (specify______) 29. How many villages, intervention areas, population does the WTP served? 30. Is there any laboratory for testing the influent and the producing water quality? ☐ YES ☐ NO 285 31. If yes, what is frequency of water quality measurements (every day, every week, every month, etc.)? Daily Weekly Monthly 32. What are the physical, chemical and biological tests that are measured at the WTP (pH, Color, test, Turbidity (NTU), Residual chlorine Cl- , hardness in term of CaCO3, Fecal coliform FC, Total coliform count TC, heavy metals, etc.)? ☐pH ☐Color test ☐Turbidity (NTU) ☐Residual Chlorine Cl- ☐Hardness in terms of CaCO3 ☐Fecal Coliform (FC) ☐Total Coliform count (TC) ☐Heavy metals 33. Are there any units in the WTP doesn’t work (Lift pumps, distribution pumps, connection pipes, valves, ☐ YES ☐ NO 286 mixers, screenings, water intakes, coagulation unit, flocculation unit, sedimentation unit, filtration unit, disinfection unit)? If yes, which ones? ☐Lift Pumps ☐Distribution Pumps ☐Connection Pipes ☐Valves ☐Mixers ☐Screenings ☐Water Intakes ☐Coagulation Unit ☐Flocculation Unit ☐Sedimentation Unit ☐Filtration Unit ☐Disinfection Unit 34. In case of any unit doesn’t work what is the reason for this? ☐Need spare parts ☐Need more time to complete rehabilitation work ☐Need qualification staff and training ☐ Other (specify____) 35. What is the type of coagulants/flocculants/disinfectants used in the WTP and in what quantity/concentration (for example Alum, Ferric Chloride, etc.)? Type Quant. Conc. Alum Ferric Chloride 36. Type Quant. Conc. 287 What is the type of disinfectant used and in what the quantity/concentration (for example chlorine gas, sodium or calcium hypochlorite, etc.)? Chlorine gas Sodium Hypochlorite Calcium Hypochlorite 37. Is the chemical (disinfectant) readily available when needed for water treatment? ☐ YES ☐ NO 38. If no why? 39. Is there any schedule for maintenance the WTP units? ☐ YES ☐ NO If yes, what is schedule? If no, what are the reasons? 40. How many tanks are working from each unit in its full capacity? How many tanks are present for each unit? (e.g., two sedimentation tanks, three filtration tanks, etc.)? Type of Tank No. Capacity Sediment Filtration Others (to be mentioned) 1. 2. 3. 4. 5. 41. How many hours is the WTP operated per day? 288 42. How often the coagulation/flocculation/ sedimentation tanks are cleaned from sludge? ☐Weekly ☐Every two weeks ☐Monthly ☐At certain Turbidity limits (to be mentioned) 43. How often the filtration units are backwashed ☐ At certain Turbidity limits (to be mentioned) ☐ At certain pressure drop (to be mentioned) ☐At regular schedule (to be mentioned) 44. Is there a contingency plan in case of accidents, fire, injuries, etc.? ☐ YES ☐ NO 45. Is there any Occupational Health and Safety (OHS) plan in the WTP? ☐ YES ☐ NO 46. Is there a proper storage of chemicals used in water purification processes? ☐ YES ☐ NO 47. Is there a continuous supplying for electricity to the WTP? ☐ YES ☐ NO If no, for how many hours in day? 48. Does the WTP has its own electrical generator in case of national electrical grids is shut down? ☐ YES ☐ NO 49. How was the technical and management capacity of staff for the intended facility? CAPACITY ASSESSMENT TOOL CAPACITY ASSESSMENT95 (LOCAL GOVERNMENT AUTHORITIES) Midterm Performance Evaluation of USAID/Iraq FFS Program 95 A full scale organizational capacity assessment is beyond the scope of this evaluation and this tool is not intended to capture that. 289 GENERAL INFORMATION Interview Date Click Here to Enter Text. Moderator Name Click Here to Enter Text. Name of Org. Name & Acronym Click Here to Enter Text. Facilitator Name Click Here to Enter Text. Directorate/Sector ☐ Directorate of Municipality ☐ Directorate of Education ☐Directorate of Health ☐ Directorate of Water ☐ Directorate of Sewer ☐ Directorate of Energy province/ Province Click Here to Enter Text. Click Here To Enter Text. INFORMED CONSENT Introductory “I am a representative of ------, an independent consulting firm statement commissioned to collect data for the Midterm Performance Evaluation of the UNDP Funding Facility of Stabilization Programs (FSS) in Iraq. The purpose of this discussion is to assess the existing capacity of provincial directorates to administer services provided by the FFS. The findings will assist decision makers to identify lessons learnt and good practices, and inform future programming. The workshop will last approximately three to four hours, and your response will be used for the midterm evaluation report. Your responses are valuable to us and shall remain confidential.” Background In keeping with the requirements of EQ4, this tool is designed for rapid capacity assessment of provincial directorates of Water, Sewer Services, Municipality, Health, Education and Electricity in the local authorities of Anbar, Ninewa, and Salah al-Din. In addition to assessing capacity in the core governance functions of strategic planning, budgeting, procurement, monitoring and evaluation, information and communication technology, and human resource management, the tool will assess technical capacity of local authorities to deliver services in line with their core mandate. The capacity assessments will be conducted through facilitated discussions with at least one senior representative from each of the seven core governance and technical functions in each directorate. In total, 18 facilitated discussions will be conducted with six provincial directorates in each of the three provinces. 290 List of Participants Local Authority Directorate # Name of Participant Gender Department Designation 1 2 3 4 5 6 7 8 9 1 0 Capacity Assessment of Provincial Directorates In this tool capacity is assessed on a scale of 1 to 4, where: 1 = Low, 2 = Basic, 3 = Moderate, 4 = Strong Functional area Questions Scores33 Notes 1 2 3 4 Budget Capacity 1. Budget 1. Are directorate budget realistic, clear, and well documented without external assistance? 291 2. Is there a budget for Operations and Maintenance (O&M) and what is the source? Is the budget realistic in term of meeting the O&M requirements? 3. Is the locally generated revenue sufficient to meet O&M costs? Revenue Collection Capacity 2. Revenue Collection 1. Is there a system in place to collect revenue? (i.e., revenue staff, bills collection points, IT system etc.) 2. Do you have systems in place for enforcement of fines ● 33 The questions are meant to be guidelines for the moderators conducting the assessment workshops. The ET will define responses for each of scores for all questions and the details will be provided in facilitator’s guide. For example for Question 3 under Budgets “Are project budgets realistic, clear, and well documented without external assistance?”, a score of 1=Project budgets are not realistic, clear, and well documented; 2=Project budgets are realistic, clear, and well documented only with external assistance; 3=Project budgets are realistic, clear, and well documented only with external assistance; and, 4=Project budgets are realistic, clear, and well documented without external assistance. or penalties for those who don’t pay? 3. Does the revenue generated meets the operations and maintenance needs? 4. Are multi-year revenue projections and expenditures reported annually? Management and Administration Capacity 3. Procurement 1. Does the directorate have a procurement policy and guidelines? 292 2. Are these procurement guidelines adhered to? 3. Are the procurement procedures monitored for compliance? 4. How does the directorate handle donor procurement requirements? 4. Planning96 Strategic Planning (Medium to Long term ) 1. Does the organization have a strategic plan (business plan)? 2. Is the strategic plan clear and specific on priorities, measurable objectives, and targets? 3. Is the strategic plan regularly reviewed? When was it last revised? 4. Is the strategic plan consistently used in guiding management decisions and operational planning? Work-plan Development 1. How does the local authority prepare its annual work-plans? 2. Who is involved in developing annual work-plans? 96 This tool has been adapted from USAID’s Organizational Capacity Assessment (OCA) Facilitator’s Guide Version With NUPAS Items, revised July 27, 2015. This document can be found on the URL https://usaidlearninglab.org/library/organizational-capacity-assessment 293 3. Are the annual workplans developed according to the planning cycle of the central government? 4. Are the work-plans linked to the budget of the local authority? 5. How are annual workplans used for management decisions, operational planning, and monitoring and evaluation? 5. Human resource management 1. Are there written policies and procedures for staff? Does staff have clear job descriptions? Is the staffing complete? 2. Are the policies and procedures on staffing and job descriptions followed? 3. Is the recruitment and hiring process open and transparent? How are positions announced or advertised? 4. Are the key management and administration positions currently filled by people with qualified and skilled staff? 5. Do most staff have the capacity in terms of their qualification and skills set to do their current jobs? 294 6. Monitoring & Evaluation 1. Does the local authority have policies and procedures for the organization to have a good sense of how it is doing and what it is accomplishing? 2. Are the policies and procedures followed? 3. Does the organization consistently set realistic targets and quantitative and qualitative performance indicators? 4. Do staff have sufficient expertise in data collection and monitoring? 5. Are project monitoring data consistently timely and reliable? 6. Are monitoring data used to design effective remediation measures? 7. Information and Communication Technology? 1. Does the directorate have policies and procedures for communication and information technology? 2. Are the policies and procedures adhered to by all staff? 3. Are IT systems adequate for the current staff size, including expected growth over the next year? 4. Are IT systems networked? 295 5. Are IT systems backed up? 6. What risks are there to the IT system (e.g. unreliable electricity, malware and viruses)? 7. Do IT staff and users have the skills to use the systems effectively? What additional IT skills are needed? Is training available on IT equipment use, software, and security? Technical Capacity (Facilitator Use Section that Applies to the Relevant Directorate) 8.1 Electricity 1. Does the directorate have an O&M manual? Is it being followed? 2. Does the directorate have an annual O&M plan? Is it being followed? 3. Are the electrical installation inspected regularity, are these inspections in line with the O&M plan? 4. Are the defective parts repaired promptly? 5. Are the required parts easily accessible in Iraq? 6. Do most technical staff have the capacity in terms of their qualification and skills set to do their current jobs? 7. Has the technical staff been trained? If so do they think training was adequate and 296 prepared them for their jobs? 8. Do directorates have the ability to provide timely service for new connections, respond to emergency situations (i.e. power failures) and conduct routine maintenance of equipment? 9. Is there a policy on how quickly complaints must be resolved or customer provided information? Is it followed? 8.2 Water 1. Does the directorate have an O&M manual? Is it followed? 2. Does the directorate have an annual O&M plan? Is it followed? 3. Are the water pipelines inspected regularly? Are the inspections in line with the O&M plan? 4. Are the defective parts repaired promptly? 5. Are the required parts easily accessible in Iraq? 6. Do most technical staff have the capacity in terms of their qualification and skills set to do their current jobs? 7. Has the technical staff been trained? If so do they think training was adequate and prepared them for their jobs? 297 8. Is water quality being regularly tested? Does it meet required standards? 8.3 Sewer (Waste Water Management)/Munici pal Services 1. Does the directorate have an O&M manual? Is it followed? 2. Does the directorate have an annual O&M plan? Is it followed? 3. Is the solid waste (garbage) collected regularly by the municipality? Is it in line with the operation plan for garbage collection? 4. Is there a system in place to maintenance of equipment and machinery (e.g., garbage containers, pressers, tractors etc.? 5. Are the defective parts repaired promptly? 6. Are the required parts easily accessible in Iraq? 7. Do most technical staff has the capacity in terms of their qualification and skills set to do their current jobs? 8. Are the sewerage lines inspected regularly? Are the inspections in line with the O&M plan? 9. Are the quality standards consistently monitored? 298 10. Is there a policy on how quickly complaints must be resolved or customer provided information? Is it followed? 8.4 Health 1. Does the directorate have sufficient numbers of health workers according to staff established positions? What are the vacancy rates for main cadre of doctors and nurses in the province? 2. Does the directorate recruit and retain staff according to Human Resources for Health strategy staff requirements? 3. Does the directorate develop and distribute guidelines and protocols for standard health practice? 4. Does the directorate provide supportive supervision in the intervention areas and in the use delivery? 5. Does the directorate have an effective supply chain management system￾to effectively store and account for health commodities? 6. Does the directorate have an O&M manual? Is it followed? 299 7. Does the directorate have an annual O&M plan? Is it followed? 8.5 Education 1. Does the directorate have an approved education sector plan and guidelines that guide quality delivery & monitoring standards for education in line the with the national education policy? Is it being followed? 2. Does the directorate have established community and intervention area based structures, and processes that facilitate effective education delivery in the province? 3. Has the directorate registered any improvement in number of classrooms? 4. Has the directorate registered any improvement in number schools with access to clean water sources and sanitation facilities? 300 ANNEX 6: LANDSAN FILES https://drive.google.com/drive/u/1/folders/1oXMLHH1PcB2EujRioAJ2UbdFb3sFi4NY 301 ANNEX 7: SIGNED CONFLICT OF INTEREST FORMS DISCLOSURE OF CONFLICT OF INTEREST FORM I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Date: Jan, 22, 2020 Signature: Name: Robert Grossman-Vermaas Title: Vice President, Technical Services Organization: International Business and Technical Consultants, Inc. (IBTCI) Evaluation Position: Team Leader Evaluation Award Number: AID-267-TO17-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s), and award number(s), if applicable) Project Name: Funding Facility for Stabilization (FFS); Implementing Organization: UNDP Award/agreement Number: AID-267-IO-15-00001 I Have real or potential conflicts of interest to disclose (Yes/No): No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct of significant through indirect experience with the project(s) being evaluated, including the involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. DISCLOSURE OF CONFLICT OF INTEREST FORM I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Date: January 22, 2020 Signature: Name: Ghazanfar Ali Khan Hoti Title: Consultant Organization: International Business and Technical Consultants, Inc. (IBTCI) Evaluation Position: Team Leader Evaluation Award Number: AID-267-TO17-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s), and award number(s), if applicable) Project Name: Funding Facility for Stabilization (FFS); Implementing Organization: UNDP Award/agreement Number: AID-267-IO-15-00001 I Have real or potential conflicts of interest to disclose (Yes/No): No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct of significant through indirect experience with the project(s) being evaluated, including the involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. DISCLOSURE OF CONFLICT OF INTEREST FORM I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Date: 1/21/20 Signature: Name: Beth Cunningham Title: Senior Director Organization: International Business and Technical Consultants, Inc. (IBTCI) Evaluation Position: Subject Matter Specialist Evaluation Award Number: AID-267-TO17-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s), and award number(s), if applicable) Project Name: Funding Facility for Stabilization (FFS); Implementing Organization: UNDP Award/agreement Number: AID-267-IO-15-00001 I Have real or potential conflicts of interest to disclose (Yes/No): No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct of significant through indirect experience with the project(s) being evaluated, including the involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. DISCLOSURE OF CONFLICT OF INTEREST FORM I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Date: 21-01-2020 Signature: Name: Sahib Haq Title: Consultant Organization: International Business and Technical Consultants, Inc. (IBTCI) Evaluation Position: Livelihood Specialist Evaluation Award Number: AID-267-TO17-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s), and award number(s), if applicable) Project Name: Funding Facility for Stabilization (FFS); Implementing Organization: UNDP Award/agreement Number: AID-267-IO-15-00001 I Have real or potential conflicts of interest to disclose (Yes/No): No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct of significant through indirect experience with the project(s) being evaluated, including the involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. DISCLOSURE OFCONFLICTOFINTERESTFORM Name: Willie Kachaka Title: Mr Organization: IBTCI Evaluation Position: Evaluation Methodologist Evaluation Award Number (contract or another instrument, if AID-267-IO-17-00001PMSP USAID Project(s)Evaluated (Include project name(s), Implementer name(s)and award number(s), if applicable) FFS, including FFIS and FFES components United Nations Development Program Contract Number: AID-267-TO-17-00001 I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Signature: Date: 09 July, 2019 DISCLOSURE OF CONFLICT OF INTEREST FORM Name: Sam Malmberg Title: Manager Organization: International Business and Technical Consultants, Inc. (IBTCI) Evaluation Position: Research Analyst Evaluation Award Number: AID-267-TO17-00001 USAID Project(s) Evaluated (Include project name(s), implementer name(s), and award number(s), if applicable) Project Name: Funding Facility for Stabilization (FFS); Implementing Organization: UNDP Award/agreement Number: AID-267-IO-15-00001 I Have real or potential conflicts of interest to disclose (Yes/No): No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct of significant through indirect experience with the project(s) being evaluated, including the involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. Date: 1/21/2020 Signature: 308 309 ANNEX 8: STATEMENT OF DIFFERENCE 310 Funding Facility for Stabilization (FFS) USAID Mid-Term Performance Evaluation July 2015 – June 2018 Statement of Differences August 2020 Project Overview In the summer of 2014, fighters, led by the Islamic State in Iraq and the Levant (ISIL), swept across northern and western Iraq, assuming control over a population of several million. The fall of Mosul, and the rapid advance of ISIL and its allies south toward Baghdad, plunged Iraq into one of the most significant political, social and security crises of recent decades. It also led to the internal displacement of approximately six million individuals. Iraqi Security Forces, supported by an international coalition of partners, liberated all territories formally held by ISIL in Iraq. Western Anbar wasthe last to be liberated, in late 2017. The Government of Iraq (GOI) declared victory over ISIL on 9 December 2017, and the Global Coalition to Defeat ISIL announced ISIL’s territorial defeat in March 2019, when the last remaining stronghold of ISIL, Barghouz (Syria), was liberated. In March 2015, the United Nations Development Programme (UNDP) was tasked to provide stabilization support to areas liberated from ISIL. Following the endorsement by the Prime Minister and Member States at the Coalition’s Working Group on Stabilization in April 2015, UNDP established a Funding Facility for Immediate Stabilization (FFIS). FFIS became operational in June 2015. The main objective of the FFIS through its early years of operation was to rapidly create favourable conditions that would allow internally displaced persons (IDPs) to return to their places of origin and begin to rebuild their lives. Following the establishment of FFIS, the Government of Iraq faced an acute financial crisis as global oil prices collapsed, reducing available revenue to support reconstruction in the liberated areas. Recognising the implications of thisfor facilitating the return of displaced populations, the Government of Iraq requested UNDP in April 2016 to open a new channel for ‘Expanded Stabilization’ (Funding Facility for Expanded Stabilization, FFES). The purpose of the FFES was to build upon immediate stabilization gains by supporting the rehabilitation of complex, more extensive infrastructure, while also re-establishing functional, large public institutions such as hospitals and educational facilities. These FFES projects were critical to revitalising the social sector and served as key employment hubs in the impacted areas. Since then, the efforts of FFIS and FFES have become collectively known as the Funding Facility for Stabilization (FFS). The FFS is organised to support four ‘Windows’ of activity. Specific projects are based upon needs assessments and a prioritisation process in agreement with Provincial Command Cells (PCCs) chaired by the Governor and endorsed by the Steering Committee. Selected projects are initially identified at the local level through consultations with various actors in the five governorates. The FFS processes are based on a set of partnership principles, which deliberately position local stakeholders at the heart of the needs assessments and prioritisation processes. According to the 2019 UNDP Independent Country Programme Evaluation, “the FFS has been hailed as a considerable success, delivering very significant results in a highly risky environment to a high standard and to a very large population, 311 achieving its intended political objectives in large measure by facilitating the spontaneous return of displaced populations.”1 Introduction As the first donor to support the FFS, the United States Agency for International Development (USAID) has provided intrinsic financial support to the establishment and scale-up of stabilization activities across the liberated territories of Iraq. USAID is the largest donor to the FFS, and its leadership among international partners has set the tone for providing regular and reliable funding to support critical response activities in the liberated areas. Other partners followed suit, which resulted in the FFS effectively contributing to the return of approximately 4.6 million Iraqis and in revitalising heavily destroyed areas. Since the initial contribution made by USAID in 2015, the donor community supporting FFS has expanded to include 28 donor countries, including Iraq. UNDP takes this opportunity to express sincere gratitude to USAID for its confidence and continuous support to the FFS. UNDP is also extremely appreciative to USAID for its third-party monitoring and for undertaking this important evaluation exercise, which affords UNDP and partners alike a vital perspective on the achievements and impact of stabilization activities in what continues to be a highly volatile operational context. The findings and recommendations as presented in the Mid-Term Evaluation Report allow UNDP to reflect on the progress and achievements of the stabilization effort, consider the insights presented in making course corrections as necessary and account for the valuable beneficiary feedback collected from the process within our daily programming. The opportunity to submit comments and clarifications on the assumptions, findings and recommendations contained in the Evaluation Report is also highly appreciated. We hope this feedback proves useful to USAID and the wider audience of this report. The below sections summarise the key observations made by the FFS team while reviewing the Evaluation Report. A more detailed orientation to specific sections where UNDP has concerns regarding ambiguities, misrepresentations and/or factual inaccuracies is outlined in the following sections. UNDP’s primary observation relates to the limited recognition of the scope and objectives of the FFS as a stabilization response supported by the Global Coalition to Defeat ISIL and the GOI in the immediate post-conflict setting. The fundamental parameters of the project in respect to its position within the broader GOI-led stabilization effort also have a bearing on the full applicability of the report’s recommendations. Recommendations based on the evaluation findings that imply a shift or redirection of the scope and parameters of stabilization activities suggest that the metrics of success applied to the evaluation did not completely factor in the specific objectives of the FFS stabilization response, and instead addressed perceived gaps or shortcomings appropriate to a wider development/recovery lens and timeframe. UNDP highlights this observation as intrinsic to the merit of the evaluation, given that the findings apply only to the period from July 2015 to June 2018, when FFS was guided by its approved Project Document, which was specifically designed to account for the immediate post-conflict need in Iraq at that time. In addition, it is unclear how the sampling and targeting of specific sectors and geographic regions was designed to ensure the evaluation findings are adequately reflective of the entire scope of the FFS programme. There is ambiguity related to the ‘eligibility’ of specific sectors of work and the locations of interventions referred to within the report. Further examination and clarification to understand how 1 https://erc.undp.org/evaluation/evaluations/detail/12276 312 the insights were extrapolated is needed to be adequately reflective of the entire FFS portfolio, as was the ambition of the evaluation exercise. These two more ‘general’ concerns are elaborated below: 1) FFS scope and objective The evaluation lacks a clear distinction between the limitations deriving from the nature of stabilization programmes and the limitations of FFS as a programme (in its design and execution, etc.). Overall, there is a lack of clarity regarding what UNDP Iraq’s FFS was established to achieve, in particular, what role UNDP FFS was designed to play in the post-ISIL context as one component of the broader stabilization process led by the Government of Iraq. FFS interventions are localised and operate within the immediate post-conflict space, with the clear objective of facilitating the return of IDPs and, in turn, supporting the extension of legitimate political authority in Iraq. These parameters, as well as the processes that underpin FFS in terms of project identification, prioritisation and development, are critical in understanding the scope and impact of the programme. These are distinct characteristics that set the FFS apart from other more conventional programmes for peacebuilding and recovery, which are designed for and aim to address pre-existing structural or root causes of conflict for a more extended period. In short, this distinction is critical as the Evaluation Report does not seem consistent when assessing and describing the achievements of FFS despite its clearly set objective, parameters and measurement for results. The success of the FFS as a programme cannot be effectively evaluated against objectives outside of the scope of the FFS and/or longer-term development objectives that FFS was not designed to achieve. 2) Evaluation methodology and final report In terms of the evaluation approach and methodology, UNDP notes that the Evaluation Report does not fully account for the factors that set the FFS apart from more traditional development programmes. Hence the evaluation findings are not comprehensive. For instance, the report would have benefitted from clarifying or considering the below points: a) (agreement on) definition of key concepts and measurement of performance; b) importance of paying more attention to the process and its rationale/justification, as well as the outcomes; c) importance of contextual detail to better assess the success of the process as well as to better determine transferability of lessons learned; and d) participatory monitoring and evaluation methods to ensure fair representation of stakeholder views. Points (a) and (b) are related to the specificities of stabilization programmes/FFS, as described above under section (1). As the report confounds the measures for longer-term development activities with the FFS’ activities and tries to evaluate the impact of FFS against the longer timeline, the findings of the report are not wholly applicable. 313 Observations by UNDP Concerning the Executive Summary The methodological overview provided in Table I references “eligible facilities” that were considered in the evaluation. This includes facilities within the sectors of education, electricity, health, etc. It is noted that here, and throughout the document, there are inconsistencies within the evaluation narrative in suggesting the entirety of FFS has been evaluated, while just ‘eligible’ sectors and locations have been assessed. The absence of a coherent justification for the basis of either the sectoral or geographical selection, and how observations are then applied to the full FFS portfolio, challenges the findings and recommendations. It is recognised that further detail is provided in Section 3.4.1 ‘Survey Areas’ (also discussed below), but this also falls short of providing a solid rationale or definition of eligibility, which are fundamental to the evaluation exercise. In consideration of this, it should be noted that FFS processes are designed to facilitate operating at-scale. In practice, this means delivering across all four Windows of activity (sequenced as nuancing to context may require) in all mandated liberated areas, which present a need for stabilization support. Operating at scale is at the heart of FFS delivery and, as such, represents the cornerstone of the FFS approach. In that regard, considering a holistic and integrated picture of the stabilization effort is fundamental in measuring the achievements or shortcomings of the programme, or at a minimum, ensuring adequate consideration of the cumulative effort and interrelation of the various aspects of FFS activities within the evaluation exercise. Concerning Fundamental Observations – What has FFS achieved from 2015-2018? The above concern is again reflected in this section under ‘Observation 1’ pg. xiv. The effectiveness of FFS as a ‘pull’ factor for returnees is noted as an observation, but only in reference to Anbar, Ninewa and Salah al-Din, excluding FFS impact in Diyala and Kirkuk. It is not clear if this key observation is reflective of the entirety of the FFS programme, as was the indicated goal of the evaluation exercise (outlined in the Purpose and Objectives of the Executive Summary) or what the justification for the omission of sampling (evaluative evidence) FFS interventions in these Governorates was based upon. Concerning Conclusions and Recommendations Here, the aforementioned observations also apply. Little attention is given to the importance of outlining the parameters of FFS programming. The text reflects limitations in terms of understanding and outlining the FFS scope and objectives. The adopted parameters do not fully cover UNDP’s support to stabilizing the conflict-affected areas of Iraq, acknowledge the specificities of UNDP’s mandate in the post-conflict context, or the responsibilities of the Government of Iraq and the positioning of FFS within the broader UN Country Team. In that regard, it does not appear that the limitations of stabilization programming or where FFS fits into the broader stabilization effort and/or vis-à-vis other humanitarian and development portfolios is factored into the evaluation’s recommendations. Examples of this can be explicitly seen in relation to two targeted questions of the survey: − The narrative relating to “Question 1: How have FFS Activities Affected Access to Services in Target Communities” pg. xv gives much attention to reported access issues in relation to sanitation. Although FFS does implement activities in the area of sewerage and wastewater infrastructure rehabilitation, it does not work in the area of sanitation per se. FFS rehabilitation activities in the sewerage sector address ISIL-damaged sewerage networks and supportive infrastructure, not connecting individual houses to the public sewerage system (particularly if they were never connected, even pre-ISIL). This is an example of where the report seems to be assessing FFS impact against structural problems that are beyond the scope of the FFS. Without an indication of what types of work were considered within the context of ‘sanitation’ for the evaluation, it is difficult to understand the applicability of the recommendations to FFS 314 activities and raises concerns that the evaluation team was not clear on FFS project identification and prioritization processes (in collaboration with Provincial Command Cells and local stakeholders) and criteria (ISIL-inflicted damages, limitation of scope to rehabilitation, etc.), as well as the specific context of the locations selected. These processes are fundamental considerations of the FFS approach and delivery mechanisms and are often what set FFS apart from other programmes operating in the rehabilitation/reconstruction sphere. − Under “Question 2: How have FFS Activities Affected the Quality of Services in Target Communities” on pg. xvii it is not clear why or how the quality of service was considered. The report does acknowledge that FFS “did not support improvements in quality of services” (i.e. capacity building). It contradicts this acknowledgement with an exploration into the perception change in the quality of services through a population-based survey. This is subsequently confused by a frequent interchange of the terms ‘access’ to services and ‘quality’ of services, as can be seen, for example, in the statement “…the Evaluation Team was able to determine perceptions of change in quality via the PBS survey. Beneficiaries were asked retrospective questions about their access to services prior to 2015 and their access to services in 2019”. It is not clear whether this examination relates to a measurement of impact relating to FFS interventions (access) or issues of quality of service prior to 2015, which fall outside the scope of FFS. Concerning Background On pg. 5 in identifying critical points to note, Point 2 contains factual inaccuracies. FFS includes a full spectrum of infrastructure work covering ‘light’ infrastructure (FFIS) through to medium-scale (FFES) infrastructure projects. There are no infrastructure projects implemented within Window 2 Livelihood Support. Furthermore, the same point indicates that FFES projects contribute to infrastructure and livelihoods. It is important to note that FFES projects are implemented under Window 1, which relates to infrastructure rehabilitation only. Although many FFES projects (hospitals, universities, etc.) are indeed hubs of employment, they are not categorised as livelihood support projects and very much fall into the infrastructure rehabilitation categorisation. FFES projects are distinguished from FFIS projects in terms of the scale of rehabilitation due to size of infrastructure and consequently the typical duration for implementation. Point 3 on the same page refers to FFS Window 4 as “community reconciliation and social cohesion”. UNDP takes this opportunity to clarify that social cohesion activities are implemented under FFS Window 4. FFS implements no reconciliation (meaning financial compensation) activities. Concerning Methodology – Section 3.1 Document Review Footnote 12 indicates an initiation of activities in Mosul in the fall of 2017. This is inaccurate. FFS activities were underway in East Mosul from early 2017, and much pre-positioning work in anticipation of Mosul’s liberation was undertaken in the last quarter of 2016. Concerning Methodology – Section 3.4.1 Survey Areas Limiting the evaluation to sectors of education, electricity, health, sewerage and water, while leaving out housing, roads and bridges, municipal support and livelihoods support is justified on pg. 8 by the following statement “because these interventions do not address basic services”. It is not stated or explained comprehensively why the evaluation was limited to sectors of basic services. The same section indicates a geographical limitation to Anbar, Ninewa and Salah al-Din, without providing a justification or reasoning for the exclusion of Diyala and Kirkuk. As noted concerning the Executive Summary, the justification for the selection of particular sectors and particular geographic locations to 315 inform this evaluation is neither transparent nor inclusive and raises questions as to the applicability of the findings to the programme at large. Concerning Methodology – Section 3.5 Livelihood Survey (LS) On pg. 13 it is stated that “beneficiary data was available for only 8,023 CFW participants.” UNDP would like to clarify that beneficiary data is collected for all cash-for-work (CfW) participants. However, there are instances where participants’ contact details (phone numbers) are not available. Furthermore, UNDP would like to clarify that contact details are indeed available for all beneficiaries of grants to women-headed households via the Ministry of Labour and Social Affairs (MOLSA). This data was not requested for the evaluation exercise. Concerning Methodology – Table 8: Summary of Livelihoods’ Programme Beneficiaries. There is an inaccuracy in the labelling of CfW beneficiaries in Table 8 on pg. 13. UNDP would like to clarify that, while many of the beneficiaries of Small Business Grants were women, the activity did not target women only. Grants were provided to disadvantaged small business owners (both men and women) who met the selection criteria. Concerning Methodology – 3.6 On-Site Observations It is mentioned on pg. 14 that the evaluation team utilised quality assessments in sectors of “water, sanitation and electricity facilities in the areas sampled”. As outlined earlier, although FFS does work in the area of sewerage and wastewater infrastructure rehabilitation, it does not work in the area of sanitation. Without an indication of what was considered as ‘sanitation’ for the evaluation, it is difficult to understand the applicability of the findings and recommendations. Concerning Limitations – Point 3 and 4(e) UNDP would consider the most critical limitations of the report to relate to the fact that not all sectors of work, windows of operation or geographical locations of FFS activity were included in the evaluation. Projects delivered within the governorates of Kirkuk and Diyala were omitted in their entirety. Similarly, activities implemented under Windows 3 (Capacity Support to Municipalities) and 4 (Social Cohesion) were not evaluated, while sectors under Window 1 such as housing rehabilitation and the rehabilitation of municipal structures (e.g. Mayoral and Directorate facilities, police stations) were also not considered as part of the evaluation. Finally, the evaluation was limited to projects that were complete or under implementation from the June 2015 to July 2018 period. Concerning Limitation 1 and Limitation 4(b) which relates to the lack of baselines, this is typical of the operational context for immediate stabilization work and arguably should have been considered within the scope of the evaluation exercise. While projects set in the development sphere are usually afforded conditions favourable to establishing solid baselines, stabilization projects operate in the emergency context and are designed for rapid implementation ensuring responsiveness to the most urgent of emerging needs. Lack of baselines should not hinder assessment of results for implemented stabilization activities. In relation to Limitation 2 and Limitation 4(e) as detailed in earlier points, contact information is available for recipients of grants to women-headed households from MOLSA. This data was not requested from UNDP for this evaluation. 316 Concerning Fundamental Observations – Observation 1 As detailed above in relation to the outline of Fundamental Observations on pg. xiv, the Governorates of Kirkuk and Diyala are omitted from the statement “FFS is an instrumental, formative activity to assuring stabilization in the conflict-affected regions of Anbar, Ninewa and Salah al-Din.” Concerning Findings – 3B. In what ways have FFS Activities Affected IDPs Lives Since Their Return to Their Communities? The findings outlined on pg. 61 that particular ethnoreligious groups earned more income from CfW interventions than others imply, within the presentation of the findings, a difference in salary scale applied to different groups. It is important to note that this is not the case. UNDP, in consideration of the CfW Working Group guidance, applies standardised daily wages for workers and supervisors of CfW projects in all areas. Depending on the size of the local population and the number of CfW interventions implemented over time, interviewed beneficiaries may have engaged in more than one CfW activity and therefore earned a higher cumulative wage. CfW activities are implemented in keeping with FFS project identification and prioritisation process. The FFS CfW interventions are designed to support the most vulnerable populations in approved locations with exigent needs for cash liquidity. As outlined in the ICPE, “UNDP does not distinguish/discriminate beneficiaries on ethnic or religious basis. Public announcements and advertisements are issued locally in the targeted geographic areas for workers to apply for cash-for-work. Applicants must be at least 18 years old, should be unemployed and should not be on the list of civil servants or be part-time workers with private companies. Only one family member is given the chance to work on each project to ensure that as many families as possible can benefit. Cash-for-work projects which target the hiring of women have been specifically designed to safely and comfortably accommodate women workers while also encouraging women to move into non￾traditional work areas.” In closing, UNDP highly appreciates the opportunity to respond to the Mid-Term Evaluation of the UNDP Funding Facility for Stabilization programme. UNDP looks forward to maintaining its solid partnership with USAID in support of a stable and prosperous Iraq. for Zena Ali-Ahmad Resident Representative UNDP Iraq Baghdad, 27 August 2020 317 November 30, 2020 To: United States Agency for International Development (USAID)/Iraq From: International Business and Technical Consultants, Inc. (IBTCI), as the implementer of the Performance Management and Support Program for Iraq (PMSP) Subject: Statement of Difference (SOD) Response Letter (To Be Included in the Evaluation Report Annex) Re: UNDP’s Statement of Difference Letter on the Funding Facility for Stabilization (FFS) USAID Mid-Term Performance Evaluation Report, July 2015 – June 2018 IBTCI appreciates UNDP’s comments on the FFS Evaluation Report, approved by USAID on April 22, 2020. Although not required by ADS 201, IBTCI has responded to UNDP’s comments. Please see: https://usaidlearninglab.org/sites/default/files/resource/files/evaluation_statements_of_difference s_aug_2017.pdf The FFS evaluation report is the culmination of evidence generated from a population-based survey of 2,735 residents; a beneficiary survey of 730 respondents; interviews with 30 local authorities; and, 12 focus group discussions all in line with the evaluation design approved by USAID on July 15, 2019. The final report reflects an analysis of these data, enriched by relevant literature and close review of the primary activity documents. IBTCI’s Evaluation Team stands by the findings, conclusions, and recommendations of the report, which responded to the specific questions and parameters elucidated in the statement of work (SOW) developed by USAID. UNDP was involved in the review process for the report, presumably through and by interactions and outreach to USAID. Based on the data, we therefore do not believe any revision to the report is necessary or appropriate. --End-- 318 ANNEX 9: INFOGRAPHIC 319 320 321 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov