August 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Jennifer Kuiper and Ethan Arnheim, on behalf of the QED Group, LLC, under Iraq PERFORM contract number 267- M-00-09-00513. EVALUATION USAID/Iraq Community Action Program III End of Project Performance Evaluation CHF International I Legislative Strengthening Program for the Iraq Council of Representatives Report USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION August 2012 This publication was produced for review by the United States Agency for International Development. It was prepared on behalf of the QED Group, LLC, under Iraq PERFORM contract number 267-M-00-09-00513. USAID/IRAQ Community Action Program III END OF PROJECT PERFORMANCE EVALUATION The QED GROUP, LLC 1250 Eye Street, NW Suite 1100 Washington, DC 20005 Phone: 202-521-1900 Fax: 202-521-1901 www.qedgroupllc.com Report authors: Jennifer Kuiper and Ethan Arnheim Disclaimer The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION i TABLE OF CONTENTS ACRONYMS ..................................................................................................................................iii ACKNOWLEDGEMENTS ............................................................................................................ v EXECUTIVE SUMMARY.............................................................................................................vii I. INTRODUCTION ...................................................................................................................... 1 II. BACKGROUND ....................................................................................................................... 3 A. The CAP III Program and Its Operating Environment .................................................... 3 III. METHODOLOGY ................................................................................................................... 7 IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS............................................ 9 A. Research Question 1............................................................................................................. 9 B. Research Question 2 ........................................................................................................... 18 C. Research Question 3 ........................................................................................................... 25 D. Research Question 4 .......................................................................................................... 30 V. LESSONS LEARNED AND RESEARCH QUESTION 5 ..................................................... 39 A. Community Engagement/Participation............................................................................ 39 B. Training Methodology ........................................................................................................ 40 C. Securing Government Buy-in and Participation .............................................................. 41 VI. ISSUES .................................................................................................................................... 43 ANNEX A. SCOPE OF WORK ................................................................................................. 45 Annex A-1. Illustrative Evaluation Questions and Evaluation Criteria to Ensure Report Quality ...................................................................................................................................... 61 Annex A-2. USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report ....................................................................................................................................... 61 ANNEX B. GLOSSARY OF TERMS ......................................................................................... 65 ANNEX C. EVALUATION METHODS.................................................................................... 67 Annex C-1. Sample Field Visit Agenda with IP Central Office ........................................... 79 Annex C-2. Sample Field Visit Agenda without IP Central Office ..................................... 80 Annex C-3. Summary of Estimated Number of Responses by Data Collection Instrument and Respondent Type .............................................................................................................. 81 ANNEX D. MATRIX OF RESEARCH QUESTIONS AND SUB-QUESTIONS ................... 83 ANNEX E. EVALUATION ACTIVITIES AND SITE VISIT SCHEDULE .............................. 93 ANNEX F. SAMPLE AGENDAS FOR FIELD VISITS (CENTRAL OFFICE, OTHER PROVINCES) ............................................................................................................................... 98 ANNEX G. DATA COLLECTION TOOLS ........................................................................... 100 ANNEX H. LIST OF VISITED PROJECTS ............................................................................. 148 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION ii ANNEX I. TEAM BIOGRAPHIES ........................................................................................... 155 ANNEX J. DISCLOSURE OF CONFLICT OF INTEREST FORM ....................................... 157 ANNEX K. BIBLIOGRAPHY ................................................................................................... 159 LIST OF FIGURES AND TABLES Figure 1: Map of Iraq with Selected Projects for Field Visits .................................................... 8 Table 1: Governmental Sources for Meeting Community Needs .......................................... 10 Table 2: Nongovernmental Sources for Meeting Community Needs .................................. 11 Table 3: Reasons War Victims Do Not Access the Marla Fund ............................................. 32 Table 4: Alternative Sources of Support for Marla Fund Beneficiaries ................................. 34 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION iii ACRONYMS A/V ACDI/VOCA AoR Area of responsibility AQI Al Qaeda in Iraq BPCS USAID/Iraq Broadening Participation Through Civil Society CAG Community action group CAP III Community Action Program III CHF CHF International CM Community mobilizer COP Chief of party COR Contracting Officer’s Representative COR Council of Representatives CSO Civil society organization DGO Democracy and Governance Office GoI Government of Iraq GSP USAID/Iraq Governance Strengthening Project HHO Haraa Humanitarian Organization IASD Iraq Association of Securities Dealers IDP Internally displaced person IHAO Iraq Health Aid Organization INGO International nongovernmental organization IP Implementing partner IRD ` International Relief and Development IT Information technology KRG Kurdistan Regional Government LGP USAID/Iraq Local Government Project LNGO Local nongovernmental organization MC Mercy Corps MOE Margin of error MOU Memorandum of understanding MRIWVF Marla Ruzicka Iraqi War Victims Fund USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION iv NGO Nongovernmental organization PMP Performance monitoring plan PRT Provincial Reconstruction Team USAID U.S. Agency for International Development USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION v ACKNOWLEDGEMENTS The team would like to express its deep appreciation to the USAID Mission in Iraq for its strong support throughout this performance evaluation. Special thanks are due to the Democracy and Governance Office staff, in particular CAP program managers Erin Epstein and Varghese Jacob, who provided important programmatic documentation and data on all three phases of the Community Action Program. In Washington, we would like to thank Dr. Gavin Helf, Senior Democracy and Governance Officer with USAID/Asia and Middle East, as well as Christopher Jennings, Senior Democracy and Governance Specialist with USAID/Middle East, Office of Iraq and Arabian Peninsula Affairs. Both provided essential historical background on the early years of Community Action Program implementation in Iraq. The evaluation would not have been possible without the generous and responsive support of USAID’s four implementing partners: ACDI/VOCA, CHF International, IRD, and Mercy Corps. Without the expert facilitation of each Chief of Party, the evaluation team would not have been able to conduct the ambitious evaluation work plan within the limited time available for field work. Further, the four IP staffs expertly and efficiently arranged over 100 meetings with CAP program participants and beneficiaries as well as Iraqi government officials. They tirelessly accompanied our team to project sites, some of which were in remote rural areas, and made themselves available for interviews, follow-up conversations, and other data-collection activities. Most important, we are indebted to the many Iraqi respondents throughout the country who took time out of their schedules to meet and help us understand the achievements and challenges of the CAP III program. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION vi USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION vii EXECUTIVE SUMMARY USAID/Iraq’s Community Action Program (CAP) is the Mission’s longest-running development program. Begun in May 2003, it was designed to develop the ability of Iraqi citizens to effectively address pressing community needs through organized democratic processes. From October 2008 to September 2012, CAP’s third phase (CAP III) has focused on building skills and cooperation between constituencies and their local representative and executive governments. This report provides an independent end-of-project performance evaluation conducted in the third quarter of CAP III’s final year of implementation. CAP III Program Description and Operating Environment The CAP III program fits under the USAID/Baghdad Mission’s results framework for USAID Strategic Objective 9: Effective Local Government Established, as defined at the time of CAP’s project design, implementation, and evaluation activities. The CAP program had three objectives: Objective 1: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. Objective 2: Local executive and representative government in CAP communities better meet the articulated needs of the community. Objective 3: Civilian victims of conflict assisted CAP III is implemented by four implementing partners (IPs), which are each responsible for a designated geographic area of responsibility (AoR), together covering 15 of Iraq’s 18 governorates: Mercy Corps: Basrah, Dhi Qar, Maysan, Muthanna International Relief and Development (IRD): Baghdad CHF International (CHF): Anbar, Babil, Karbala, Najaf, Qadissiyah, Wasit ACDI-VOCA : Diyala, Kirkuk, Ninawa, Salah ad Din Although each partner had variation in program design, all provided some form of technical assistance, training, and targeted funding for community-initiated CAP projects. By design, the CAP III project acted as an incentive for a given community to engage in a process of effectively prioritizing local service needs in communities through an organized democratic process, and then advocating for those needs with local governments. Each IP helped to select, form, and train groups of community-based citizen representatives, generally referred to as community action groups (CAGs). CAP projects initiated by the CAGs vary by type but typically provide infrastructure support in areas of education, health, transportation, electricity, sanitation, water, youth, and recreation. The IPs were also tasked with development of in-kind projects to assist and provide support to Iraqi who were victims of war involving U.S. Forces/Iraq through the Marla Ruzicka Iraqi War Victims Fund (MRIWVF or Marla Fund). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION viii Evaluation Methodology The evaluation was conducted by a 12-person team, including 2 U.S. citizens and 10 Iraqi nationals. The team implemented the evaluation over a three-month period, including six weeks of data collection and analysis in Iraq. The performance evaluation applies a mixed methods design, which uses both quantitative and qualitative data collection and analysis. Data collection activities included field visits to all 15 participating provinces with structured stakeholder interviews of implementing partner staff members, Provincial Councils, line ministries, and Marla Fund local nongovernmental organizations (LNGOs); focus groups with CAGs, Local Council members, CAP III beneficiaries, and Marla Fund beneficiaries; mini-surveys of CAP III beneficiaries, Marla Fund beneficiaries, and IP community mobilizers; and project site visits of 63 projects in 15 provinces. Findings, Conclusions, and Recommendations The report provides specific observations and recommendations on the first four research questions, followed by lessons learned in response to the fifth research question. Research Question 1: Did the CAP Program contribute to increased participation in collective community actions? The evaluation team found that: 1. Iraqis who participated in or benefited from CAP III turn to a variety of governmental and nongovernmental institutions to meet community needs. Surveyed respondents most frequently cited sources of support as being Local Councils and CAG members, with limited reliance on Iraqi and international NGOs. 2. Although CAP III participants place primary value on development projects ahead of process objectives, they also demonstrate a commitment to the CAP model of citizen participation. These two program benefits – projects and participatory processes – were interconnected, with the projects providing a strong motivation for CAP III participants to build capacity and apply CAP’s citizen engagement approach. 3. CAP contributed to important clarifications of the roles of the citizen advocate and government official, with CAG members best suited for identifying, prioritizing, and advocating on behalf of community needs: Local Councils provide forums for community input and discussion of needs as part of national and provincial budget-planning processes; Provincial Councils prioritize competing community needs at the provincial level and allocate resources; and line ministries/directorates provide technical expertise, project approvals for authorized projects, and project implementation and monitoring. 4. CAP-initiated processes at the provincial level have the potential to be sustained beyond the program period, especially with regard to community-based engagement in provincial budget-planning processes. Based on these findings, the evaluation team concludes that: 1. An engaged Iraqi public can hold its government leaders accountable for responding to community needs. 2. CAP III provided a model of effective citizen engagement in which diverse stakeholders came together through their identification of a common need with clear constituencies and tangible mutual benefits. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION ix 3. Gaining a seat at the table as a representative of community needs depends on the aspiring representative’s (e.g., CAG or Local Council member) ability to add perceived value for those who allocate resources (e.g., Provincial Councils, ministries). For example, respondents defined appropriate roles as the following: citizen advocates provide a direct link to the community and identification of needs; Local Council members bridge the gap between the community and official decision makers; Provincial Councils bring resources; and ministry staff provide technical expertise and approval authority. 4. Provincial budget-planning processes provide institutional and procedural opportunities for impactful citizen participation. Building on CAP’s achievements and lessons, the evaluation team makes the following recommendations: 1. Link USAID-sponsored capacity-building activities with participant-identified priorities that visibly and transparently meet community needs. 2. Train civil society leaders to build their capacity to provide an effective bridge between their community and government institutions, focusing on community assessment techniques, use of data for determining and validating needs, interest-based negotiation strategies, and communication and media strategies. 3. Facilitate citizen participation in the provincial budget-planning process by disseminating information on the process, required documentation, and timeline. 4. Encourage elected officials’ appreciation for citizen input by sponsoring joint planning conferences and workshops that feature community expertise, identifying mutual priorities for public investment, and providing opportunities for relationship-building. 5. Provide consultative services to line ministry staff to establish mechanisms for ongoing citizen access, such as citizen input on infrastructure needs, processing of community￾initiated requests for projects, and a customer service approach to public engagement. Research Question 2: Are CAP communities better able to articulate their needs and mobilize resources within and outside the community to solve common problems? If not, why not? The evaluation team found that: 1. The majority of interviewed CAGs demonstrated an ability to determine community needs and mobilize resources, with frequent use of public meetings as forums for community input and most resources provided as in-kind contributions by the Government of Iraq. 2. CAGs expressed a mixed sense of ability to advocate on behalf of their communities, including challenges presented by their dependency on the IP throughout the project development process, lack of established role and status in Iraqi society, the difficulty of operating in remote locations, and missed opportunities for promoting greater awareness of their activities. 3. CAG membership composition had varying levels of representation of the community they served (including participation of targeted vulnerable groups), depending on a range of factors including the process for CAG member selection, the influence of the IP community mobilizer, existing power dynamics (including political parties), local traditions/customs (especially related to participation of women and youth), and attitudes toward volunteerism and community service. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION x 4. Though it is unlikely that CAGs will continue in their current format after the CAP program ends, there are several ways in which CAG activities can be sustained, including continued work by individual leaders whose capacity was developed through CAP, CAG members forming local NGOs to continue work on community priorities, and future engagement of CAP IP staff who hold a deep understanding of the CAP process. Based on these findings, the evaluation team concludes that: 1. Attribution of credit for providing benefit to the community has significant implications, including inspiration for future community-led initiatives and electoral gains for those recognized for leadership in addressing community needs. 2. Having notables on CAGs can improve access to key decision makers and facilitate project implementation but can also stifle new leadership development. 3. Voluntary, or unpaid, leadership attracts a more altruistically motivated participant. 4. Including vulnerable groups requires intentional strategies that account for local conditions and promote equal status and authority of representatives of vulnerable groups. The evaluation team makes the following recommendations: 1. Promote public outreach throughout project implementation to make the power of citizen action transparent and inspirational. 2. Explore a variety of participant selection options that promote equal opportunity for access to the most appropriate individuals by providing training for community organizers in facilitation strategies that encourage participation of marginalized or less confident participants, and promoting selection processes that avoid selection bias favoring dominant leaders. 3. Work in communities without elected representation where the community voice is the only channel for advocating needs. Research Question 3: Does local executive and representative government in CAP communities better meet the articulated needs of the community? If not, why not? The evaluation team found that: 1. Constraints in accessing resources are significant across all levels of the Iraqi government due to bureaucratic constraints, making community mobilization of resources challenging. 2. CAP III provided training in a range of tools and approaches to help local government officials understand community needs and properly conduct project procurement and budgeting. These officials desire higher levels of advanced training in addition to what they have already received. 3. CAP III provided a forum for engagement between the Local Council and other national and sub-national government officials, especially with Provincial Councils and line ministries. 4. The most frequently mentioned CAP III contribution highlighted by government program participants (e.g., line ministries, Provincial Councils, Local Councils) were the CAP￾provided infrastructure projects; however, Local Councils also described increased engagement with their constituencies as an important result of CAP III. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION xi Based on these findings, the evaluation team concludes that: 1. Although their status in Iraqi society has been diminished by the state’s failure to hold local elections in all provinces, Local Councils are becoming increasingly relevant as a bridge between communities and the national and sub-national hierarchy. 2. Iraqi communities that have participated in Local Council elections are taking community service into account when casting their vote. 3. Local Councils are more active and better equipped through their participation in CAP to prioritize and respond to citizens’ needs. The evaluation team makes the following recommendations: 1. Encourage Local Councils to convene and facilitate public meetings for community input, as well as to develop mechanisms for reporting back responses to the community. 2. Continue to train Local Councils in project management, advocacy, and budgeting in order to be more effective in the provincial-budgeting process. 3. Encourage continued capacity building and engagement of Local Councils through support for the IRD-initiated Local Council Association. Research Question 4: Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the Fund’s defined purpose? If not, why not? The evaluation team found that: 1. There are uncounted additional war victims who have not received support from the Marla Fund, primarily because they were not harmed by Coalition forces and therefore not eligible, or because they had difficulties providing the required eligibility documentation. 2. IP community mobilizers and Marla Fund beneficiaries rank both NGOs and the Iraqi government above traditional social networks as replacement sources of support to the Marla Fund. 3. Iraqi local nongovernmental organizations (LNGOs) are ill-equipped to replace implementing partners in the administration of the Marla Program without support, especially in terms of fiscal management. Based on these findings, the evaluation team concludes that: 1. The likely number of eligible war victims outstrips the funding allocation that would be needed to provide meaningful support to all eligible individual cases. 2. Current documentation requirements are a significant barrier to accessing support, especially verification requirements based on police and medical reports and court documents. Because legitimate documentation can be difficult to obtain, IP staff expressed concern that false documents are being used not only by actual victims but also in submissions by fraudulent applicants. 3. Small business start-up requires intensive training and ongoing support; accessing employment with an established employer may be more feasible for Marla Fund beneficiaries who face multiple hardships, such as disabilities, illiteracy, and poverty. 4. Community-based projects have the potential to reach a broader public, eliminate challenges of individual eligibility determination, and better meet war victims’ needs. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION xii The evaluation team makes the following recommendations: 1. Conduct an intentional transition from CAP to BPCS that captures data and maintains relationships with CAP-era partners by establishing a transition committee of key stakeholders from CAP’s Marla Fund implementation and obtaining person-level data of Marla Fund applicants and recipients to initiate a national database of prior applicants. 2. Improve support to individual beneficiaries by providing for a more uniform program implementation that emphasizes transparency and consistency of the eligibility verification process and better customizes projects to individual needs and priorities. 3. Take steps to invest in more community projects in areas of continuing need, by conducting a national assessment of need, providing ongoing medical services, and creating jobs accessible to persons with disabilities through established employers. 4. Establish clear selection criteria and performance responsibilities for continuing and new LNGOs that will be providing services to Marla Fund beneficiaries in the future. 5. Support the Iraqi government in implementing Law No. 20/2009 Compensating the Victims of Military Operations, Military Mistakes and Terrorist Actions. Lessons Learned Research Question 5: Provide lessons learned that could be applied to future USAID work with Iraqi communities and local governments, specifically on approaches to: community engagement/ participation, training methodology, and securing government buy￾in/participation. (1) Community engagement/participation When beginning implementation of any USAID initiative, engage the highest jurisdictional level to promote transparency and political support, especially with Provincial Councils and governors Seek out “neutral” venues for meeting and events to encourage participation of vulnerable groups and demonstrate the program’s impartiality to competing political and social factions Use the diversity of staff to model cooperation and provide “relatable” partners for participants Integrate new programming into existing political and societal systems to support incremental change and strengthen democratic institutions Accommodate local language (e.g., Arabic and Kurdish) speakers to facilitate participation of credible and influential community-based leaders who do not speak English Work with champions, role models, and trail blazers within vulnerable groups to provide inspiration and encouragement to others that identify with members of their identity group (2) Training methodology Whenever feasible, use experiential learning approaches to training Enable relationship-building and cross-sector understanding of roles through joint training between government and non-government as well as national and sub-national leadership Integrate peer review or learning opportunities that include site visit exchanges Provide easily accessed and updated materials for post-visit follow-up and references USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION xiii Use a diverse (majority/minority, gender, age) training team to model cooperation across “differences” Use creative and context-sensitive strategies to include women in training opportunities When conducting inter-generational trainings, adapt training pace and topics to have cross￾generational appeal (3) Securing government buy-in/participation Maintain focus on trust-building and transparency throughout the process of program implementation, not just at the beginning and end of a program Connect community-based work with provincial-level planning to promote better understanding of larger systems, establish meaningful relationships that are useful beyond a one-off effort, and promote equitable and efficient distribution of resources Seek common understanding among program participants of each other’s roles, responsibilities, and expectations with respect shown for existing hierarchy and status Bring something meaningful to the table that adds perceived value for those who allocate resources, such as funding, expertise, data, valued partnership, prestige, access to influential people, or validation. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION xiv USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 1 I. INTRODUCTION USAID/Iraq’s Community Action Program (CAP) is the Mission’s longest-running development program. Begun in May 2003, it was designed to provide support for Iraq’s communities to “exercise true grassroots democracy by implementing projects on their own where necessary, and in partnership with local government where possible, to meet community needs.”1 Beginning in October 2008, CAP’s third phase (CAP III) has focused on building skills and cooperation between constituencies and their local representative and executive governments. This report provides an independent end-of-project performance evaluation2 conducted in the third quarter of CAP III’s final year of implementation. 3 As defined by USAID in the statement of work (Annex A), this end-of-project performance evaluation has the following two purposes: 1. Accountability: To assess to what extent CAP III’s objectives and goals have been achieved, within the context of USAID’s Strategic Objective 9 Effective Local Government Strengthened:4 CAP III Objective 1: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. Sub-IR 9.2.1. Communities better able to mobilize resources within the community to meet their articulated needs. Sub-IR 9.3.1. Communities better articulate their needs. CAP III Objective 2: Local executive and representative Government in CAP communities better meet the articulated needs of the community. Sub-IR 9.3.2. Qadaa and nahiyaa better articulate needs of their communities. CAP III Objective 3: Civilian victims of conflict assisted. Sub-IR 9.3.3: Civilian victims of conflict assisted by the Marla Ruzicka Iraqi War Victims Fund (MRIWVF or the Marla Fund). 2. Learning: To inform the implementation of future projects, especially those working at the community level to link Iraqi citizens with their government for improved development results. In particular, evaluation results should be useful to USAID’s Governance Strengthening Project (GSP) and Broadening Participation through Civil Society (BPCS). 1 Statement of Work, USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation (April 11, 2012). 2 As explained in the statement of work, “USAID/Iraq does not believe it is possible at this stage in the program to apply a methodology [for an impact evaluation] that is based on a credible and rigorously defined counterfactual to control for factors other than CAP that might account for any observed changes reported by the evaluation team.” Further, the team is not serving as auditors or monitors as such assessments are outside the scope of this evaluation. In its January 2011 evaluation policy, USAID defines performance evaluations as those that “focus on descriptive and normative questions: what a particular project or program has achieved; how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making.” “USAID Evaluation Policy: Learning from Experience.” USAID. Washington, DC (January 2011). 3 The estimated cost of the evaluation as documented in the scope of work (April 11, 2012) is $417,215.56. The actual expenditure is anticipated to be much less. 4 USAID’s SO 9 and intermediate results as applied to CAP III and this evaluation are diagramed in the scope of work, page 5. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 2 The principal audience for the evaluation is USAID’s Iraq Mission (USAID/Iraq), specifically the Democracy and Governance Office and the Capacity Building Office. Additionally, the evaluation should inform the U.S. Congress and American public regarding CAP’s contribution in support of Iraq’s transition toward democracy. To this end, this report provides programmatic and contextual background of CAP III’s operating environment, which is found in Section II. Section III describes the five research questions that guided the evaluation design and methodology used to conduct the evaluation. Section IV provides the summary of the key findings, conclusions, and recommendations organized according to the first four research questions, followed by Section V on lessons learned as highlighted in the fifth research question. Section VI provides declarative statements pertinent to potential evaluation team conflicts of interest and statements of differences in the findings, conclusions, or recommendations of the evaluation team members; in both cases, there were none. The main report is followed by annexes, including a statement of work and additional detail on methodology and data-collection instruments used for the evaluation. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 3 II. BACKGROUND The Iraq Community Action Program was launched in May 2003 to address the immediate challenges facing the country in the wake of Operation Iraqi Freedom. The first phase, known as Iraq Community Action Program, or CAP I, started three weeks after the fall of Baghdad to Coalition forces and lasted until March 2007. CAP I aimed to support ethnic and tribal reconciliation, increase the speed of development assistance, mitigate ethnic/sectarian conflict, and encourage citizen engagement. Of these goals, the speed of development assistance was emphasized in the aftermath of the former regime. As the situation on the ground changed, the program adjusted. CAP’s second phase, which lasted from September 2006 to December 2008, placed greater emphasis on developing a democratic process of community decision making. Under CAP III, implemented from October 2008 to September 2012, the program has focused on building the capacity of local government to perform its role as a service provider, including providing community needs assessment and prioritization, project design, resource mobilization, and project implementation. CAP III shifted to encourage greater government responsiveness to citizen requests, while maintaining the objective of increasing community mobilization and citizen engagement. The goal of CAP III is “to increase the local government’s ability to identify, articulate and better meet the needs of its constituency.”5 USAID has invested approximately $675 million across all three phases of the CAP program.6 At the core of the CAP model for all three phases are community action groups (CAGs), community committees or boards charged with identifying and prioritizing community needs. Their role was appropriate for the Iraqi context and transition from an authoritarian regime in which citizens had little input with regard to development and community issues. CAGs differentiated the CAP model from other development assistance programming, as projects served as a means to achieve the larger ends of community development and citizen participation. A. THE CAP III PROGRAM AND ITS OPERATING ENVIRONMENT CAP III is implemented by four USAID implementing partners (IPs), each responsible for a designated geographic area of responsibility (AoR), together covering 15 of Iraq’s 18 governorates: Mercy Corps: Basrah, Dhi Qar, Maysan, Muthanna International Relief and Development (IRD): Baghdad CHF International (CHF): Anbar, Babil, Karbala, Najaf, Qadissiyah, Wasit ACDI/VOCA: Diyala, Kirkuk, Ninawa, Salah ad Din While each partner has an individual program design, all provide some form of technical assistance, training, and targeted funding for community-initiated projects. CAP projects vary but typically provide infrastructure support in areas of education, health, transportation, electricity, sanitation, water, youth, and recreation. In addition, another set of CAP activities 5 Community Action Program (CAP III) Performance Management Plan (PMP). USAID. March 2009. 6 “The Iraq Community Action Program: USAID’s Agreement with CHF Meet Goals, But Greater Oversight Is Needed.” Office of the Inspector General for Iraq Reconstruction. April 2011. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 4 serve victims of war through the Marla Ruzicka Iraqi War Victims Fund (MRIWVF or Marla Fund). The CAP program has three objectives. The first two objectives are most closely related to the program’s aim of increasing local government’s ability to identify, articulate, and better meet the needs of its constituency. The program’s third objective involves the implementation of the Marla Ruzicka War Victims Fund and uses CAGs to help identify war victims for support. Objective 1: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. Objective 2: Local executive and representative government in CAP communities better meet the articulated needs of the community. Objective 3: Civilian victims of conflict assisted. The CAP III program fit under what was the USAID/Iraq Mission’s results framework for Strategic Objective 9: Effective Local Government Established, as defined at the time of CAP’s project design, implementation, and evaluation activities. CAP III is based on the theory of change that an active partnership and investment among all stakeholders – citizens, local government, and business and social leaders – to identify community priorities and design interventions will integrate marginalized population segments (such as internally displaced persons, or IDPs) and strengthen and expand the connections between community and government that increase civic functioning. CAP III provides the inputs of high-quality technical assistance, training, and targeted funding. These inputs are the basis for the higher-level outcome of a stronger partnership between communities and local government that better provides for citizen needs. In order for training, projects, funding, and other inputs to lead to higher-level results, the program assumed that Local Councils would receive a stronger mandate through national elections and increased funding, which was not the case. In September 2010, the IPs began to track the program’s inclusion of and benefits to IDPs in accordance with modifications to all four cooperative agreements. In fall 2011, USAID sent a letter to IPs requesting that they also collect data to track their engagement with other vulnerable populations, including female-headed households, religious/ethnic minorities, and youth. Additional performance data was developed for each partner with common indicators articulated in individual performance management plans (PMPs) and reported to USAID in quarterly reports. CAP III was launched amid expectations that local government would increase in importance in providing for citizen needs. CAP III planned to support the Provincial Powers Law, which established the legal and practical division of governance among the national, provincial, and local levels. As part of CAP III’s design, the main government partner was the Local Council, (qadaa, nahiyaa, and neighborhood councils), which was expected to increase in significance and legitimacy following planned elections. When district and sub-district elections did not occur in the fall of 2011, the existing Local Councils were not replaced with elected members and never gained the anticipated financial authority. Other CAP III program assumptions were not invalidated by the failure to hold local elections and other political developments, but did require adjustment. The program expected that “other USAID projects – Local Governance Project (LGP) and Tatweer – [would] facilitate USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 5 access to provincial and national budget processes and cycles,” which did not regularly occur. Another assumption, that “GoI budget processes for the provincial capital development budget and the ministerial capital development and [Operations & Maintenance] budgets will facilitate cost-sharing at the sub-national level” was not the case for sharing budgets at the Local Council level. Finally, the program assumed that “national and provincial budget execution will improve over time,” but over the course of CAP III there were only minor changes to this process.7 The program's third objective, support for civilian victims of war, relates to U.S. Public Law 108-11, the "Emergency Wartime Supplemental Appropriations Act, 2003," which calls for “assistance for families of innocent Iraqi civilians who suffer losses as a result of military operations.” Named the Marla Ruzicka Iraqi War Victims Fund, or Marla Fund, after a young activist who was killed in Iraq while advocating for greater assistance for civilian victims, the Marla Fund provides in-kind medical, income generation, property repair, and community project-based support. Marla Fund assistance is limited to Iraqi civilian war victims who suffered harm due to U.S. or Coalition forces. CAGs established under CAP frequently identified cases in their community. Under CAP III, there were 1,941 Marla Fund projects, 66% of which funded income-generation activities.8 CAP III IPs used the same basic model that centered on communities identifying priorities through CAGs. However, there were some programmatic features that were not common to all IPs. The IPs used different priorities for “theme” or issue-based CAGs to coalescence individuals around a common idea, in addition to CAGs based on geography. ACDI/VOCA had more vocational, apprenticeship, and job-creation programs. Mercy Corps placed an emphasis on services to women and people with disabilities. Working exclusively in Baghdad, IRD developed a strategy to work in each of the city’s 115 neighborhoods. When CAP III was launched in 2008, the country was more stable than in earlier phases. Overall, the insurgency had waned, though in many areas the situation remained tenuous and interethnic and sectarian attacks were not uncommon. CAP III program implementation spanned critical developments including the November 2008 status of forces agreement through the December 15, 2011, withdrawal of American troops. Despite these challenges, CAP III managed to implement activities throughout its lifespan in each of the 15 governorates. The program was forced to make programmatic adjustments for security reasons in some areas, usually based on specific circumstances within each AoR. Among these changes, CAP III activities were suspended in parts of Diyala province from 2009 to 2010 due to instability, although ACDI/VOCA continued work in 16 communities. In Ninawa governorate, the program focused on disputed areas where the Kurdistan Regional Government (KRG) had a strong and competitive presence and provided greater stability, while other areas such as Mosul’s city center were considered too risky to implement projects. The security situation in Salah ad Din had improved and CAP activities continued apace, though Al Qaeda in Iraq (AQI) killed one Provincial Council member who had worked with CAP. In Dhi Qar governorate, the program avoided some particularly dangerous districts known for robberies. 7 “Section 3: Critical Assumptions.” Community Action Program (CAP III) Performance Management Plan (PMP). USAID. March 2009. 8 USAID’s “Roll-Up” of Community Action Program (CAP III) Report, as of March 31, 2012. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 6 There were significant local variations in the political makeup within and between governorates, often reflecting the ethnic and sectarian composition of those areas. At the governorate level there were major differences within partners’ AoRs. CHF’s AoR included Anbar, which is almost entirely Sunni Arab, as well as Karbala, which is approximately 90% Shiite. In Ninawa, the heterogeneous makeup of ethnic Arab, Kurdish, Armenian Orthodox, Yazidi, and Christian population meant that ACDI/VOCA had to operate with extreme sensitivity to local dynamics. The CAP III program was implemented in this context of ethnic, regional, and socioeconomic variation, political tumult, and frequent insecurity. IPs implemented projects in each of the 15 provinces and adjusted their activities to respond to circumstances while retaining the core features of the CAP model. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 7 III. METHODOLOGY The key questions to which this evaluation will respond, as outlined in the statement of work (dated April 11, 2012), are as follows: 1. Did CAP III contribute to increased participation in collective community actions? 2. Are CAP communities better able to articulate their needs and mobilize resources within and outside the community to solve common problems? If not, why not? 3. Does local executive and representative government (defined as governorate-level ministry representatives, governorate councils, Local Councils, and the Council of Representatives [COR]) in CAP communities better meet the articulated needs of the community? If not, why not? 4. Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the fund’s defined purpose? If not, why not? 5. Provide lessons learned that could be applied to future USAID work with Iraqi communities and local governments, specifically on approaches for: (1) community engagement/participation, (2) training methodology, and (3) securing government buy-in/ participation. Cross-cutting themes across all research questions include: (1) prospects for sustainability of program results beyond the program period, and (2) outreach to and engagement of vulnerable populations, specifically female-headed households, ethnic and religious minorities, IDPs, and youth under the age of 24. The evaluation was conducted by a 12-person team, including 2 U.S. citizens and 10 Iraqi nationals, contracted through USAID’s PERFORM/Iraq contract with The QED Group LLC. The team implemented the evaluation over a three-month period, which included two weeks of document review and interviews of IP program managers in Washington, DC, and six weeks of data collection and analysis in Iraq. The evaluation covered the CAP III performance period from October 2008 until the third quarter of its final year of implementation (June 2012). CAP III ends in September 2012. The evaluation applied a mixed methods design, which uses both quantitative and qualitative data collection and analysis. Data-collection activities included field visits to all 15 participating provinces. Field work in Iraq included: Structured stakeholder interviews: Structured stakeholder interviews: Implementing partner staff members (4 chiefs of party, 13 senior staff and CAP III program managers, 5 M&E specialists, 42 community mobilizers, 11 CAP III trainers, 10 CAP III engineers, 5 Marla Fund managers); Provincial Council members (18); line ministry staff (23); Marla Fund local nongovernmental organization partners (10 NGOs), additional Iraqi government officials (3) Focus groups: Community action groups (57 sessions, 210 CAG members); Local Council members (46 sessions, 164 Local Council members); CAP III beneficiaries (13 sessions, 57 beneficiaries); and Marla Fund beneficiaries (12 sessions, 108 beneficiaries) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 8 Mini-surveys: CAP III beneficiaries (252 surveyed); Marla Fund beneficiaries (108 surveyed); IP community mobilizers (101 surveyed) Project site visits: a purposeful sample of two to five projects in each of the 15 provinces, 63 in total (See Figure 1: Map of Iraq with Selected Projects for Field Visits).9 Additional explanation regarding evaluation design, team training and experience, and data limitations is provided in the annexes. Figure 1: Map of Iraq with Selected Projects for Field Visits 9 Projects were selected to provide insight into a variety of project, community, and CAG characteristics, including: urban/rural, open/closed, geographic/theme/cluster CAGs, high government cost share, variety of project types (e.g., education, electricity, youth, health), benefit to vulnerable populations, and security profile. A list of the selected projects is found in Annex H, List of Visited Projects. Annex C, Evaluation Methods, lays out the criteria used to select projects. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 9 IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS CAP’s overall vision is to develop the ability of Iraqi citizens to effectively address pressing community needs through organized, democratic processes. This section describes the specific observations of the evaluation team in terms of CAP III’s contributions toward shifts in collective community action in Iraq (findings); analysis of what these observations imply regarding CAP III achievements and challenges (conclusions); and a description of how USAID can build on the CAP III experience for future programming (recommendations). Below are findings, conclusions, and recommendations related to the first four research questions. Section V on lessons learned examines the fifth research question. A. RESEARCH QUESTION 1: Did the CAP Program contribute to increased participation in collective community actions? Based on interviews, surveys, and field visits, government and CAG program participants indicated that CAP III did contribute to important changes in knowledge, attitudes, and behaviors among key stakeholders involved in community-based development. These changes can be summarized as follows: Knowledge: Both local government leaders and citizen CAG members described a deeper understanding of each other’s roles and responsibilities as well as the development of skills to better perform these roles. Attitude: Citizen CAG members expressed an increased expectation that their local government leaders would provide services that adequately address community needs, while local government officials expressed an increased sense of accountability. Behavior: Government and CAG members who participated in the CAP III program described changes in procedures to enable greater citizen input in assessing and prioritizing community needs, including in some procedures connecting directly to the provincial budget-planning process. In reflecting on the changes brought about through CAP III, an implementing partner COP summarized the cultural barriers to citizen engagement and CAP’s contribution to overcoming these obstacles: “You may not see the results now or even in the next two to three years. But it is about breaking a lock. It was unheard of for a citizen to interact with local issues. It is all top down instructions in Iraq. You take orders from Baghdad. Citizens and communities have no say at all as to what is happening. You cannot imagine it until you see it. You want to change a doorknob but you have to talk to the Ministry. People think: ‘the government has to do everything for me; I can only complain.’ We have unlocked this. Now citizens know they can approach the council members and work with them.” Below the evaluation team discusses its specific observations regarding CAP III’s contributions and the continuing challenges. Findings Iraqis who participated in or benefited from CAP III turn to a variety of governmental and nongovernmental institutions to meet community needs. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 10 With the end of the CAP III program in sight, the evaluation team identified governmental and nongovernmental entities that are relied on by community members as partners and providers in meeting community needs. Understanding these perspectives is useful for USAID’s future programming in relation to community outreach and potential development partners. The team’s analysis is based on surveys of the following: beneficiaries of visited CAP III projects (which provided a view of the general Iraqi community exposed to CAP III’s community-based benefits); all IP community mobilizers (which provided a view of Iraqis with intensive exposure to the CAP approach and skill-building); and Marla Fund beneficiaries (which provided a view of those deriving direct individual benefits from CAP III services). Survey results are summarized Table 1: Governmental Sources for Meeting Community Needs Source CAP III Project Beneficiaries1 IP Community Mobilizers2 Marla Fund Beneficiaries3 Local Councils Many times 26% A few times 30% Never 28% No response 16% Many times 80% A few times 13% Never 0% No response 7% 20% (1st choice) Provincial Councils Many times 16% A few times 12% Never 52% No response 20% Many times 15% A few times 64% Never 8% No response 13% n/a Line Ministries and Directorates Many times 16% A few times 13% Never 52% No response 19% Many times 16% A few times 34% Never 41% No response 9% 4% Governors Many times 6% A few times 8% Never 63% No response 23% Many times 10% A few times 51% Never 27% No response 12% n/a Parliament Many times 5% A few times 8% Never 63% No response 24% Many times 5% A few times 45% Never 42% No response 8% n/a 1 Evaluation team surveyed in-person 252 beneficiaries of CAP III projects, asking: “How often have I gone to these people for help in meeting a community need?” Response options were: Many times, A few times, Never. 2 Evaluation team conducted an online survey of all IP community mobilizers, receiving 101 responses to: “How often do Iraqi communities go to the following people for help in meeting their communities’ needs?” Response options were the same as with CAP beneficiaries. 3 Evaluation team surveyed in-person 108 beneficiaries of the Marla Fund asking, “Where would you go to replace the support you are receiving if the program stopped providing services?” USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 11 in Table 1: Governmental Sources for Meeting Community Needs, and Table 2: Nongovernmental Sources for Meeting Community Needs. Governmental Bodies Although the Iraqis surveyed did not generally indicate that they engage their government leaders for community support, there was a notable difference when it came to Local Councils as compared with other officials. When asked, “How often have I gone to these people for help in meeting a community need?” 56% of surveyed CAP III beneficiaries responded that they went to Local Councils “many times” (26%) or “a few times” (30%), with only 28% indicating “never.” When asked, “Where would you go to replace the support you are receiving if the program stopped providing services?” 20% of Marla Fund beneficiaries indicated Local Councils as their most frequently selected option.10 Interestingly, IPs’ community mobilizers responded to the question “How often do Iraqi communities go to the following people for help in meeting their communities’ needs?” by ranking Local Councils at an even higher level of 93% (80% responded “many times,” 13% responded “a few times,” 0% responded “never”). Over half of CAP III beneficiaries do not go to other government bodies for community support. Nearly 30% approach Provincial Council members and line ministry staff but less than 15% go to governors or Council of Representative members. Although lower than with Local Council members, 79% of community mobilizers indicated that communities seek support from Provincial Council members (15% “many times” and 64% “a few times”) and 61% from governors.11 Despite COR members being elected officials responsible for representing constituent needs, community mobilizers share the same low assessment with CAP III beneficiaries, with 42% indicating that communities “never” go to COR members for community assistance. Only 4% of Marla Fund beneficiaries indicated that ministries/directorates are seen as alternative sources of support. Nongovernmental Bodies Despite CAG’s lack of the legal status as government bodies or registered nongovernmental organizations, respondents indicated that communities turn to CAGs for community support at a frequency comparable to Local Councils. CAP beneficiaries tend to turn to CAGs with more frequency (35% “many times,” 17% “a few times”) than community mobilizers’ estimates of community behavior (15% “many times,” 65% “a few times”). Perhaps most relevant for USAID’s work with Iraqi civil society, IP community mobilizers and CAP III beneficiaries hold divergent views toward Iraqi civil society organizations and non￾governmental organizations. Nearly 60% of CAP III beneficiaries indicated they have “never” sought support from Iraqi civil society organizations (CSOs) and NGOs, as compared to 9% of IP community mobilizers who think community members “never” seek support from these sources. However, IP community mobilizers do not indicate an overwhelming endorsement for CSOs/NGOs, given that 54% responded that Iraqis go to this source only “a few times.” Marla 10 Other options on the survey included: ministries/directorates, religious organizations, Iraqi NGOs or CSOs, family, and international organization. 11 The fact that community mobilizers work closely with Provincial Councils in their programmatic role in building relationships with government bodies to facilitate project approvals and cost share may explain the difference in perspectives on community behavior. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 12 beneficiaries – many of whom are receiving benefits through the newly engaged local NGO partners – selected CSOs/NGOs as their second most likely source of alternative support. This pattern repeats with assessment of international NGOs, with most CAP III beneficiaries never seeking support from INGOs (60%), nearly all IP community mobilizers viewing the community as seeking this support (54% “many times,” 38% “a few times”), and Marla Fund beneficiaries listing it as the third most likely source of alternative support. Table 2: Nongovernmental Sources for Meeting Community Needs Source CAP III Project Beneficiaries1 IP Community Mobilizers2 Marla Fund Beneficiaries3 CAG Members Many times 35% A few times 17% Never 32% No response 16% Many times 15% A few times 64% Never 8% No response 13% n/a Iraqi CSOs and NGOs Many times 13% A few times 10% Never 58% No response 19% Many times 29% A few times 54% Never 9% No response 8% 19% (2nd choice) International NGOs/Donors Many times 11% A few times 17% Never 60% No response 12% Many times 54% A few times 38% Never 0% No response 8% 17% (3rd choice) 1 Evaluation team surveyed in-person 252 beneficiaries of CAP III projects, asking: “How often have I gone to these people for help in meeting a community need?” Response options were: Many times, A few times, Never. 2 Evaluation team conducted an online survey of all IP community mobilizers, receiving 101 responses to: “How often do Iraqi communities go to the following people for help in meeting their communities’ needs?” Response options were the same as with CAP beneficiaries. 3 Evaluation team surveyed in-person 108 beneficiaries of the Marla Fund, asking “Where would you go to replace the support you are receiving if the program stopped providing services?” Although CAP participants place primary value on development projects ahead of process objectives, they also demonstrate a commitment to the CAP model of citizen participation. CAP participants – including IP staff, CAG members, and Local Council members – valued the results of the process (i.e., the development project) more than the process itself (i.e., citizen participation). When asked in focus groups with CAGs and Local Councils “What were CAP III’s most important successes,” 71% of the 42 responding Local Council focus groups and 40% of the 50 responding CAG focus groups indicated that the project was what was most important. Another 17% of Local Council focus groups and 44% of the CAG focus groups stated both the project and process was most important. Local Council and CAG members emphasized that projects responded to vital community needs and filled gaps in infrastructure (e.g., roads, bridges, school buildings, electricity, and drinking water) that the Iraqi government had failed to provide. Even 71% of the surveyed IP community mobilizers – who are tasked with facilitating USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 13 participatory processes – indicated that projects were the most important “community contribution.” Implementing partner staff emphasized that there is an important link between projects and the promotion of collective community action. In an environment that has grown cynical from failed provision of services from the Iraqi government as well as international donors, both Iraqi and international IP staff noted that Iraqis are less inclined to participate in a training program or capacity-building activity that lacks a direct, transparent benefit to their community or themselves. CAP III offered the project as the “bait”12 to engage citizen volunteers and government officials. As one Chief of Party stated: “CAP wouldn’t have worked without bringing money to the table, or maybe only after 10 years, because you can’t get people engaged. People need to see immediate results to believe because they have become skeptical of development efforts since 2003.” One IP trainer said, “We use the projects as a vehicle in which people can ride around and see what is going on.” Other IP staff members emphasized that the projects provided evidence to the Iraqi community that the citizen engagement model works. These respondents saw the project as more of a byproduct than the goal itself. An IP COP emphasized this point: “CAP was heavily process￾oriented but it was important to have the projects because it was the evidence that [CAP participants] had learned the process; the projects provided an outlet through which people can reinforce for themselves that they succeeded.” Local Council and CAG focus groups verified that they gained new understanding and approaches for citizen participation through the experience of developing and implementing projects. In responding to questions regarding how Local Councils and CAGs assessed and prioritized community needs, Local Councils and CAG members indicated their understanding is rooted in their own experience and networks of personal relationships, as they are closely tied to the communities in which they live. However, the vast majority of focus groups across all IPs and provinces indicated that they do not rely exclusively on their own personal views. They also look toward community input in one form or another to identify and prioritize projects to meet community needs (98% of the 43 responding Local Council focus groups, 94% of 51 responding CAG focus groups). Strategies highlighted by Local Council respondents included consultation with CAG members, individual meetings with community member advocates, public meetings with constituencies, site visits to communities, and demographic and needs assessments from ministries. CAG members relied heavily on the format of public community meetings and a voting process to select priorities. Local Council and CAG focus group participants repeatedly expressed the value they place on citizen participation and achieving a sense of empowerment. “[The CAGs] were a good idea so that the ideas come from the bottom to the top.” (Kirkuk Local Council member) “As the Local Council, we might be able to know the general needs for the communities but sometimes there are special needs we do not recognize. For example, at one village we thought that the major need was for paving the roads. But through the CAG, we recognized that their urgent need is the expansion of electrical power.” (Muthanna Local Council member) 12 Implementing partner staff interview. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 14 “CAP encouraged people in the community to be more effective and raise their voices to connect the government and let them know about their community needs.” (Baghdad CAG member) “The project improved our relationship with the government. The officials didn’t used to listen to us and we didn’t use to go to them often. With CHF’s help that was changed.” (Qadissiyah CAG member) “We created channels of communication and mutual trust between the citizens and local governments.” (Ninawa CAG member) A recent activity of a CAG in Bartala (Ninawa) provides an example of a CAG that has internalized the process of community consultation. Following his May cabinet meeting conducted in Ninawa province, Prime Minister Maliki recommended that the Iraqi Government re-issue Ninawa’s provincial development funds from the prior year that had gone unspent due to the inability to implement projects under difficult security conditions. Recognizing the possibility of unallocated funds becoming available, the Bartala CAG convened an open community meeting to discuss community needs and priorities. Based on the discussions, the CAG plans to prepare proposals to advocate on behalf of its community, as members learned to do through the CAP program. CAP contributed to important clarifications of the respective roles of citizen advocate and government official. CAP III clarified and reinforced distinct roles of key stakeholders in the community development process. Against a backdrop of evolving democratic institutions and practices as well as ill-defined policies on decentralization, CAP participants described individuated responsibilities that help define expectations as well as responsibilities. CAG members and government officials interviewed for the evaluation saw this clarity as an important contribution, with one Babil Local Council member saying that “the community started to be more understanding of the government role.” A consensus view among all respondent types regarding the role of each of the major players emerged during discussions on the project development process: 1. CAG members: Provide a bridge between the community and local government, especially the Local Council. Identify community needs, prioritize those that are most important to the community, and advocate on behalf of the community before government decision makers. 2. Local Council members: Provide forums to invite community input. Represent community needs before the ministries and Provincial Councils. Work to mobilize resources through the budget-planning process and obtaining of project approvals. 3. Provincial Council members: Receive input on community priorities from Local Council members. Prioritize communities according to the need for equitable distribution and inclusion in provincial budget plans. Provide pressure, when needed, to obtain support, approvals, and resources from ministries. 4. Line ministries/directorates: Provide technical expertise and code specifications. Grant approvals for authorized projects. Implement, monitor, and/or maintain approved projects. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 15 CAP-initiated processes at the provincial level have the potential to be sustained beyond the program period. Beyond the improved capacity of individual CAP III participants and clarifications of roles, several CAP-initiated practices have the potential to be of value and possibly be sustained beyond the CAP program period. The examples below involve engaging citizen input on a provincial scale, which was not an explicit objective of CAP III. Mercy Corps has timed its project development cycle to complement the provincial budget￾planning process. Because CAP III participants prepare project proposals to correspond to when Provincial Councils make their budget recommendations to governors, this opportune timing has increased the likelihood that a community project can be integrated into provincial plans. An example is Mercy Corps’ community mobilizer’s convening of a three-day workshop that includes: (1) CAG members, who understand community needs; (2) the relevant line ministries’ staff, who provide guidance on ministry priorities and technical expertise on project design; and (3) Local Council members, who are responsible for presenting community projects and advocating on their behalf before the Provincial Councils. This approach directly addresses one of the biggest challenges and sources of discouragement to CAGs that fail to get the necessary approvals or cost sharing for their projects: Resources have already been allocated through the previous year’s budget process. Despite self-reported high rates of success in obtaining approvals for projects through this process in 2010, Mercy Corps staff acknowledged that the failure to hold Local Council elections has undermined public perception of Local Council legitimacy and, therefore, limited their effectiveness in advocating on behalf of communities. Although the timing is still right, future efforts may require an alternative champion that is seen as more credible. Another example of institutionalizing citizen input at the provincial level can be found in Kirkuk and Diyala. As a result of ACDI/VOCA’s Annual Provincial Planning Conferences, Kirkuk and Diyala Provincial Councils have passed resolutions this year that require Local Councils to convene public consultation meetings to validate the submission of any project to the provincial-budgeting process. In the case of Diyala, the Provincial Council has even indicated it will cover the costs of these public meetings. Salah ad Din and Ninawa Provincial Councils have indicated they are considering similar requirements. This development is a direct result of ACDI/VOCA’s active engagement of Provincial Council members, keeping them well-informed of program activities and including them in joint training sessions. The annual conferences include CAG members as citizen representatives, along with the governor and key staff, Provincial Council chair and members, Local Council members, COR members, and ministry officials. ACDI/VOCA created space to place CAG members on equal footing, thereby providing an opportunity for them to demonstrate their value. Conclusions An engaged Iraqi public can hold government leaders accountable for responding to community needs. As stated above, the evaluation team found that participants value CAP-initiated processes providing for citizen input. Additionally, through their experiences with the development process, CAP III participants have clarified roles in which the citizen advocate is best positioned to identify and prioritize needs while government officials are responsible for delivering on USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 16 these needs. The team also found examples demonstrating that elected Provincial Council members recognize the importance of listening to their constituents and have set up formal systems for constituent participation in the budget-planning process. Through the project development process, citizen advocates (i.e., CAG members) demonstrated that collective community action can gain the attention of Iraqi government officials. CAGs succeeded through the implementation of the prioritized projects in overcoming years of neglect to gain vital infrastructure for their communities. Such success elevates the credibility of those who are seen as responsible for meeting these needs, whether government officials or informal community-based leaders. This recognition for providing benefit to the community is not lost on elected government leaders, who will face a referendum on their performance in the next elections. And even without elections to validate their status, Local Council members have the potential to maintain credibility among their constituencies by exhibiting their commitment to meeting community needs. This conclusion is further explored in the sections below, which discuss the sustainability of CAP activities, especially examples of CAG members who are elected to Local Councils (in areas with such electoral possibilities, such as Najaf and Karbala) and to Provincial Councils based on demonstrated commitment to addressing community priorities. A common need with a clear constituency and tangible benefits works to motivate engagement and define success. The evaluation team found that CAP III provided a strong motivation for engagement through its focus on the opportunity for participants to meet important community-based needs. The strategy of focusing on development projects enabled CAP III to successfully secure the joint participation of community-based leaders and government officials to promote grassroots democratic processes. The possibility of meeting community needs was so compelling that CAP III participants worked without any compensation other than the satisfaction of meeting vital community needs (and perhaps the political benefit of public recognition, as noted in the first conclusion). This dynamic is especially significant given the still foreign concept of volunteerism in the Iraqi culture. The high level of CAG, CAP III beneficiary, and Local Council appreciation for the development projects indicate that similar capacity-building efforts should clearly articulate the anticipated benefit beyond skill development of individual participants. In communities that still lack basic necessities (such as water, electricity, school facilities, roads, etc.), USAID can anticipate that development projects will continue to be a strong draw for citizen engagement and capacity-building opportunities. Additionally, other incentives might also be persuasive, such as common threats (e.g., carcinogenic pollutants), aspirations (e.g., state-of￾the-art surgical facility), or rights (e.g., child protection campaigns). Gaining a seat at the table depends on ability to add value according to defined roles. CAP III’s clarification of roles and its emphasis on how demonstrated benefit earns credibility has implications for how USAID can structure programs and recruit key stakeholders to create platforms for public participation. CAP III demonstrated that there are limits to what can be expected of citizen action that is not linked to the ladders of political hierarchy. The best trained and most highly organized and motivated CAGs will still face barriers to achieving objectives if they are unable to gain access to and cooperation from Iraqi officials. As highlighted in the above finding regarding roles, community-based leaders are best positioned to understand constituent priorities and provide compelling testimony of the need. However, USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 17 unless the citizen group has appropriate expertise (e.g., engineering, architectural, construction), it is less equipped to provide the technical detail to formulate a feasible project design. This responsibility is within the realm of line ministries and directorates. Accessing the required approvals to mobilize the requisite resources also depends on CAGs’ forming a government partnership to carry forward proposals. While the most accessible ally is the Local Council, some CAGs have successfully appealed to Provincial Councils, governors, and, to a lesser degree, directly to ministries. Citizen leaders who are better able to provide compelling evidence of community needs and can skillfully negotiate with power brokers will be the most effective in demonstrating their value to their communities as well as to government leadership. The provincial budget-planning process has openings for impactful citizen participation. Although CAP III’s focus was at the community level and on promoting cooperation between Local Councils and their constituents, the evaluation team found important institutional openings for citizen engagement at the provincial level. With CAP III’s emphasis on securing government cost sharing for project implementation, implementing partners recognized that submission of CAP-initiated proposals for bigger infrastructure projects needed to align with the provincial budget-planning process in order to most readily obtain necessary approvals and resources. Therefore, IPs provided training and support to facilitate CAG and Local Council understanding of these processes. CAGs are empowered by understanding Provincial Council budget-planning considerations, such as use of ministerial data for census estimates and asset gaps, existing commitments for central government infrastructure investments, and political party considerations based on political support and opposition. Recent national developments have further opened Provincial Councils to constituent input. With the recent shift in provincial development funds’ allocations from sectors to population, Provincial Councils should be able to understand needs across the entire governorate, not just in urban centers or places in which individual Provincial Council members reside. This confluence of the increased capacity of communities and Local Councils to articulate needs and increased receptivity of Provincial Councils to receive proposals creates a tremendous opportunity for impactful citizen participation. Recommendations In general, USAID is well-positioned to promote continued collective community action through both its Broadening Participation through Civil Society and Governance Strengthening programs. Building on CAP’s achievements and lessons, the evaluation team makes the following recommendations: Link USAID-sponsored capacity-building activities with participant-identified priorities that meet community benefit. This approach acknowledges that participants will be more actively engaged and more effective in applying new capacity if there is an immediate application to demonstrate as well as visible value. Train civil society leaders to build their capacity to provide an effective bridge between their community and government institutions. The following skills were found to be especially important for citizen leaders to be successful in understanding and representing community needs and fulfilling their appropriate role: Community assessment techniques Use of data for determining and validating needs USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 18 Interest-based negotiation strategies Communication and media strategies to promote transparency and accountability Facilitate citizen participation in the provincial budget-planning process by disseminating information on the process, required documentation, and timeline. The national and provincial budget process is not well understood by most Iraqis and is not consistently applied across provinces or from year to year. Given that the budget process is a strong arena for community input and influence, USAID can make an important contribution by providing to citizen advocacy groups and Local Councils reliable, accurate, and current information on the process, timeframe, and deadlines at the national and provincial levels. Encourage elected officials’ appreciation for citizen input through sponsoring joint planning conferences and workshops. Although CAP III participants gained an understanding and appreciation for each other’s appropriate contributions, it is not clear that this view is broadly accepted throughout Iraq. To widen the opening CAP III has created, USAID capacity-building activities should be structured to: Feature community expertise Identify mutual priorities for public investment Provide opportunities for relationship-building Provide consultative services to line ministry staff to establish mechanisms for ongoing citizen access. As seen in the survey of CAP III beneficiaries, IP community mobilizers, and Marla Fund beneficiaries, these CAP III participants were less likely to turn to ministries for support in meeting community needs; in focus groups, CAG members described ministry officials as more remote and less accessible. This view is consistent with an appropriate ministerial role that is focused on implementing – as opposed to determining – government plans and budgets. However, CAP III has demonstrated that citizen input can support prioritization of approved projects, with the potential of improving community satisfaction with ministry performance. Useful mechanisms for citizen participation could focus on the following: Citizen input on infrastructure needs Processing community-initiated requests for projects, especially those that are not included in established budgets Customer service approach to public engagement B. RESEARCH QUESTION 2: Are CAP communities better able to articulate their needs and mobilize resources within and outside the community to solve common problems? If not, why not? This question corresponds to the first objective of the CAP III program: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. This objective is integral to the CAP program’s theory of change. Through the experience of designating and implementing individual CAP III projects as well as other training and resources provided to CAG members, the program expects that Iraqi communities will be better able to identify and prioritize their common needs and then access the resources to provide for their USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 19 needs through collective action. The objective’s emphasis is more on the grassroots mobilization process for addressing concerns and less on the project itself. Findings The majority of interviewed CAGs demonstrated an ability to determine community needs and mobilize resources. CAGs used a range of approaches to identify community needs. The approaches mentioned in CAG focus groups included site visits, community meetings, and some of the tools provided by IPs during trainings (listed below). Of these approaches, community meetings were the most frequently cited. Over half of the CAG focus groups that discussed this question (54% of 51 focus groups) stated that an open meeting was used to identify community needs. While community meetings were an important part of identifying local needs, the IP also played a role in project selection. In focus groups, 61% of 51 responding CAGs mentioned the IP’s involvement in terms of funding limitations, training and/or guidance when answering the question “How did the CAG decide what were the community needs?” The IP’s role varied in terms of work with CAGs. In 12% of 51 responding focus groups, CAGs said that the IP presented a number of projects and encouraged the CAG to select one. This did not appear tied to the mobilization strategy of any one implementer. In other instances, IP support was focused on new mechanisms for identifying community needs. These mechanisms, or tools, varied by IP but were similar in the participatory approach used: IRD: Community-based planning and action CHF: Participatory rapid assessment Mercy Corps: Joint action planning sessions ACDI/VOCA: Annual provincial planning conference and community project plans Community-based planning and action activities were centered around participatory workshops that resulted in detailed community needs assessments and actions plans. CHF’s participatory rapid assessment involved training CAG members and Local Council officials on determining community development priorities. Mercy Corps’ Joint Action Planning Sessions and ACDI/VOCA/s Annual Provincial Planning Conference brought CAG members, local government representatives, and other stakeholders together to discuss their respective priorities and community needs. Specifically, ACDI/VOCA trained CAG and Local Council members to build community project plans to identify and prioritize community needs as developed through the community meeting process. These approaches were not used as widely as community meetings, which were common across all IPs, but were still a part of the CAG process of identifying local needs. Out of 51 focus groups, 22% mentioned using one of these tools to help identify community needs or to determine which project was the most appropriate. Though IPs provided training to CAGs on these methodologies, 18% of 51 CAGs indicated in focus groups that living in and being part of their community enabled them to best understand community needs. This view might be best embodied by a Maysan CAG member who said, “Since we are part of the community and we are in direct connection with them, we know the needs. In addition, we make some visits to our friends.” USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 20 The community’s contribution represented an average of 2% of the total project value compared to 59% provided by the Iraqi government and 39% provided by USAID.13 Given this, CAGs needed to build outside support and resources in order to implement their activities. Among the range of actors CAGs engaged to secure contributions were local government officials, including the Local Council, Provincial Council, and line ministries; local businesses; and local NGOs and community organizations. Contributions in the form of land constituted the majority of the support to CAGs from the government, although CAGs also received in-kind assistance in the form of equipment. When the CAG or the community members contributed to the project, it was typically in the form of labor, protection of materials from theft, or monitoring of project implementation. 14 IP community mobilizers supported CAG resource mobilization. After the CAG was formed, its members needed access to key government stakeholders who would help secure the government’s contribution for the project. Community mobilizers leveraged their status as representatives of an international NGO to provide access to key stakeholders that might not otherwise have been available. Community mobilizers helped broker meetings for the CAG with the Local Council and also provided introductions to the Provincial Council, line ministries, and other influential entities that could be helpful in project implementation. CAGs relied heavily on Local Council members taking forward their proposals to both line ministries and Provincial Councils, which were critical in ensuring the government’s contribution. Although the government provided the majority of funding to the CAG, resource mobilization was not limited to government actors. CAGs also worked with international NGOs. One of the projects was implemented by Al-Amia CAG in Baghdad in partnership with Women for Women, an international NGO that provided vocational training for women. In another instance, the Kuwait Embassy gave 2,000 wheelchairs to a CAG project. One CAG linked IDPs to justice services. The Qadissiyah Farmer’s Association, an agricultural cooperative to which several CAG members belonged, volunteered its land and labor to support implementation. These partnerships involved a mix of CAG and IP initiative and were unlikely to occur without the active involvement of either group. In addition some CAGs in Basra province have established partnerships with oil companies, which have provided funding for implementing projects in the community. Despite these successful efforts at resource mobilization, CAGs’ lack of a formal legal organizational status was an impediment to greater success in their resource mobilization efforts. The degree of formalization varied among CAGs, with the most organized CAGs having a governing board and bylaws. Yet even well-established CAGs were unable to accept funds themselves, as an NGO or private business would be able to do. This obstacle was apparent when IRD organized a fair of international donors so CAGs could meet potential funders for their work. But CAGs’ lack of legal status made receiving grants prohibitively difficult for the potential donors. In interviews with IP staff as well as CAG focus groups, the process of 13 Community Action Program (CAP III) Performance Management Plan (PMP). USAID. March 2009. 14 Each IP followed a USAID-approved policy for valuing in-kind contributions from the community and the GoI. There were some differences among IPs, with one specifying the exact hourly rate to value a community’s donated labor. Typically, land, which constituted the largest in-kind contribution, was valued based on the total cost in relation to comparable properties. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 21 organizing and registering as an Iraqi NGO as required by Iraqi government regulation was described as arduous, expensive, and outside the capacity of most CAGs. CAG memberships expressed a mixed sense of agency and empowerment to advocate on behalf of their communities. When advocating on behalf of their communities, CAGs were confronted with multiple challenges but still registered notable advocacy successes. Among the challenges faced were dependency on the IP, lack of established role/status in Iraqi society, difficulty of operating in remote locations, and missed opportunities to promote greater awareness of their activities. Among CAGs’ successes were ensuring that CAG project materials and supplies were protected, the ability of some to replicate the CAP III process after successful implementation of the initial project, and recognition of and satisfaction with their own ability to serve their communities. A noteworthy challenge faced by many CAGs was their dependency on the IP for many of the steps within the project implementation process. The IP, the principal community mobilizer, provided introductions, forums for engagement, technical expertise, credibility, and funding. For many CAGs, the absence of IP support would mean the CAG would be disbanded or would have a diminished ability to advocate for community needs. Among CAG focus groups that answered how they expected the end of the CAP program to change their advocacy, 56% of 55 focus groups said that they expected the work to continue as it had before. On the other hand, CAG members in 15% of 55 focus groups said that they would not be able to advocate without the IP’s support. The remaining six focus groups expressed a view that certain aspects of the CAG would continue, although its activities would be significantly changed. Survey results of IP community mobilizers corroborated these findings. When asked “What three reasons best explain why a CAG successfully implements a project?” USAID funding and Good relations between the CAG and Local Council received 89% and 6%, respectively, of the 101 respondents to this question. The next most selected option, Training of CAG members in important skills, received only 40%, while Good relations between CAG and line ministries, Good relations between CAG and Provincial Council, were each selected by less than 10% of respondents Another obstacle to CAG advocacy was the absence of an official role within the community. In focus groups asked about the barriers to CAG project implementation, CAGs made the point that they are not employers and have no recognized leadership role in the political hierarchy. In 9% of 46 focus groups, CAGs recommended using an identification badge to establish some status and credibility. Community mobilizers echoed this suggestion, saying that such a designation would have been helpful for CAGs in gaining access to government buildings. For at least one CAG, another obstacle to advocacy was the remote location of the community and lack of any official government representation. This Babil-based CAG has no communication with its Local Council and no members of the Local Council are from their village. In the focus group, members also made the point that travel costs to reach the Local Council were prohibitive. The CAG asserted that its inability to connect effectively to their Local Council was a significant obstacle to advocating on behalf of their community and gaining the resource support needed to meet community needs. Among CAG advocacy successes, one CAG member described their CAP III experience as “breaking the fear factor” of calling the government into account. In Qadissiyah, a CAG aimed USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 22 to expand electricity access in a small village. After a delay in project implementation because the operational budget was not approved, the CAG was concerned that a powerful Local Council member would use the electrical equipment bought with CAP III funds to serve his own area, as had happened previously. The CAG devised a system of recording serial numbers on materials, which were tracked by the CAG chair to protect their investment against the corrupt official. IPs took steps to encourage the development of a sense of community responsibility and empowerment to ensure that CAGs understood their role as community advocates. Among the different efforts, ACDI/VOCA included a resource-mapping strategy that helped the CAG members identify their existing assets. The goal was to overcome the CAGs’ initial perspective that they lacked valued resources or ability at the community level. This process of positive inquiry aimed to help communities see themselves in a new light and appreciate local capacity and resources. Finally, CAGs were clearly effective at identifying projects that addressed important community needs. Based on the survey of CAP III beneficiaries, 84% of 240 respondents indicated that the project was Very Important to the community. The evaluation team visits to the project sites also verified the importance of the projects to the communities. Only 4% said Not Very Important (2%) or Not Important (2%). CAGs had varying levels of representation in the community they served. The CAGs’ composition varied, depending on a range of factors including the process for selection, the influence of the IP community mobilizer, existing power dynamics (including political parties), local traditions/customs, and attitudes toward volunteerism and community service. The varied approaches to membership selection resulted in a range of characteristics, including size of the group, number of women and vulnerable groups represented, presence of youth, and number of Local Council members included on CAGs. CAGs’ self-reported size typically ranged from 7 to 12 members (in some cases, the number was much higher), although the active members may be no more than two or three dedicated individuals, according to experiences reported in CAG and Local Council focus groups as well as interviews with IP community mobilizers. In terms of gender representation, women had a presence on most CAGs. In focus groups 80% of 55 CAGs mentioned that women were represented on the CAG or could describe a concerted effort to involve women. However, in all but a few CAGs females were a distinct minority, constituting only one or two members of the total CAG membership. Some women CAG members who participated in CAG focus groups asserted that women often lacked an equal voice, while some male CAG members were dismissive of the need for a more inclusive representation. When asked how he would know what women want without a female representative on the CAG, a Dhi Qar CAG member explained that “Women can come speak to my wife, and then I’ll know.” Though women were underrepresented on CAGs, CAP did create a platform for certain enterprising women to use the position as an opportunity for leadership. This opening is significant in an Iraqi society that has limited space for female organizing, meetings, and events. From focus groups with CAG members come a few examples of women who became active through their participation on a CAG. In Maysan, one of the female CAG members said, “I was USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 23 shy to say anything in front of men before I began working on the CAG.” During focus group discussions with the other CAG members, however, she was observed to participate actively in the conversation. Similarly, in Baghdad, an IDP woman CAG member had to speak at a community event, although she described herself as previously being too shy to talk in a public setting. There were indications, however, that other forms of community engagement may have provided more opportunities for women. Under CAP II, theme- and issue-based CAGs were a part of the program design. According to an IP community mobilizer who had worked on both CAP II and CAP III, theme CAGs were a more effective way of securing women’s involvement. Although there were instances of CAP III providing a platform for women’s leadership, according to some community mobilizers, CAP III showed a net decrease in women’s involvement. In terms of IDPs, another marginalized group that was targeted later in the implementation of the CAP III program, there were challenges with directing assistance or even accurately tracking involvement. In interviews, community mobilizers explained that IDPs are disinclined to self-identify as such. IDP inclusion on CAGs was usually coincidental (when the CAG member happened to be an IDP), unreported (such as when an IDP preferred not to reveal his/her status to the IP), or a result of the CAG serving a community of IDPs. Though it is unlikely that CAGs will continue in their current format after the CAP program ends, there are several ways in which CAG activities can be sustained. While CAGs have varying degrees of dependence on IPs, there are several ways in which their activities will be continued after the CAP program finishes. This includes individual leadership development, NGO formation, an alumni network of former CAG members, and CAP IP staff who hold a deep understanding of the CAP philosophy and process. In focus groups, CAG members consistently mentioned their increased willingness to speak on behalf of their community and its needs and their increased skill set in being able to do so more effectively. This attitudinal and knowledge shift is evidenced by these self-reported changes among the participants. As CAG members frequently had other roles, working on behalf of their communities, such as sheikh, mukhtar, civil servant, or Local Council member, it is likely that these leadership skills will assist the former CAG members in better serving their communities through these responsibilities. CAG members may sustain their community advocacy efforts through other channels. Several CAG members now serve on Local Councils and others have been elected to Provincial Councils. As service on the CAG provides an opportunity for leadership growth, this is a natural and encouraging development for CAG members. Although IPs do not formally track these developments, focus groups reported at least two instances where former CAG members now serve as Provincial Council members. Another possible avenue of sustaining the CAG structure is for members to work for an NGO or to create such an organization themselves. This transition is likely easier for issue-based CAGs, which are working on issues that have broader resonance than CAGs that are geographically limited. However, as noted earlier, there are significant obstacles to CAGs forming NGOs, including the costs of registration and the loss of credibility in the community, USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 24 given the continuing skeptical public perception of NGOs. Despite these obstacles, the IP COPs mentioned this possibility for program continuation. Sustaining CAG activities is also possible through an alumni group of CAG members. This idea was mentioned by several interviewees and has the potential to support networking between CAG members and allow participants to share their expertise. For USAID, an alumni group could function as a platform for maintaining contact with those who benefited from CAP III involvement as well as tracking the longer-term contributions of former CAP participants. Another important asset and resource for maintaining CAP contributions is the CAP III IP staff, especially the more than 100 community mobilizers. Staff members have a unique skill set that is especially valuable for other efforts to engage local government and community-based leaders. This valuable experience comes from working with a wide range of government stakeholders and from navigating the bureaucratic process and is not easily gained. These staff members embody values and hold a deep understanding of the community mobilization and grassroots democratic processes promoted by CAP. When surveyed, 98% of IP community mobilizers said they feel better able to advocate on behalf of their own communities as a result of their work with CAP. Conclusions Attribution of credit for meeting community needs has significant implications. Successful implementation of CAP III projects can have a demonstration effect that encourages and inspires others to participate or even lead subsequent efforts. Surveyed CAP III beneficiary recognized CAP projects as valuable within the community. As CAP III beneficiary and CAG focus groups indicated, CAGs conducted outreach to the community at varying points of the project cycle, beginning with needs assessments and prioritization, resource mobilization, project implementation, and maintenance efforts. Greater public outreach promotes individuals and groups that support community services and raises their public profile and credibility. This recognition is especially important given the upcoming elections. In Muthanna, a Local Council focus group member stated that CAP has to announce its achievements through the media or someone else will take credit for the contributions and use it for political benefit. Having notables on CAGs can improve access and facilitate project implementation but can also stifle new leadership development. As CAG focus groups indicated, Local Council members, Provincial Council officials, and tribal leaders who served on CAGs were important in securing project approvals. At the same time, the minority and vulnerable groups that were represented on CAGs struggled to compete with these members who had already had greater status in the community. Voluntary, or unpaid, leadership attracts a more altruistically motivated participant. In focus groups, CAG members consistently conveyed their commitment to developing projects that served the public interest and evinced pride in the project after its completion. If CAG service were financially rewarded, this attitude within and toward CAG members could change. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 25 Including vulnerable groups in CAGs and community-based activities requires intentional strategies that account for local conditions. With each of the vulnerable and minority groups targeted by the CAP program, a tailored approach that accounts for the sensitivities of working with these groups is needed. Although CAG membership is an important step, it should be part of a larger plan to ensure that vulnerable groups receive equal status and authority, or the needs of the communities they purport to represent may not be addressed by the leadership group’s decisions. While many CAGs did include the participation of women, IDPs, and youth, these groups were rarely the most vocal members and their representation did not ensure the development of programs that served their interests. Recommendations Promote public outreach throughout project implementation to make the power of citizen action transparent and inspirational. The potential for a demonstration effect is increased with greater publicity of CAP activities. Greater public outreach is needed throughout the project cycle, not just at the beginning and end. This process must be transparent and understandable to observers who may seek to replicate the approach. Although the value of partnership with the government is better learned through actual work on a project, if results are apparent to other community members, they too may be inspired to pursue a CAP-like partnership with government officials. Through ribbon-cuttings, invitations to walk-throughs of the project sites, and other forms of publicity, CAP III projects can increase their inspirational impact. Explore a variety of participant selection options that promote equal opportunity for access to the most appropriate individuals by: Providing training for community organizers in facilitation strategies that encourage participation of marginalized or less confident participants. Promoting selection processes that avoid selection bias that favor dominant leaders and that create space for new faces, such as secret ballot or instant electronic voting, as has been experimented with by ACDI/VOCA. Work in communities without elected representation where the community voice is the only channel for advocating needs. CAP III program helped communities identify their priorities and advocate for services. This contribution is more significant in areas where there are fewer existing channels of communication between the community and the government. USAID activities should continue to prioritize working with community-based leadership in these underserved areas. C. RESEARCH QUESTION 3: Does local executive and representative government15 in CAP communities better meet the articulated needs of the community? If not, why not? This question corresponds to the second objective of the CAP III program: Local executive and representative government in CAP communities better meet the articulated needs of the community. 15 USAID defines “local executive and representative Government” as provincial-level ministry representatives, Provincial Councils, Local Councils, and Parliament (if applicable). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 26 While the first program objective existed in earlier phases of the CAP program, this objective is unique to CAP III. This transition was understandable, as Local Councils did not exist in the first phase of CAP program and there was more of a programmatic focus on providing for immediate needs, not on helping the government address these needs. In the course of the evaluation, local executive and representative government was broadly understood, to include the Local Council, line ministry, and Provincial Council. However, the focus of field research was on the Local Council, because it was the level of government most directly involved in CAP programming. Findings Constraints on accessing resources are significant across all levels of the Iraqi government. Local government in Iraq is actively involved in developing and implementing projects that serve basic needs. Though a participant in the CAP III programming, local government was hampered by its difficulties accessing government resources despite the fact that Iraq is a relatively wealthy country. Although limited in amount, USAID-provided funding was much easier for CAGs to access. After the program finishes, it is unclear how local government will replace the support provided by USAID funds for development projects that were available through the CAP program. Local Councils are critical to the CAP project implementation process and serve as the main point of contact for CAG efforts to mobilize government resources. In response to the IP community mobilizer survey question “When beginning the process to form a CAG, whom in the community and/or province should your organization contact first in order to be most successful?” 82% of 100 responding community mobilizers named the Local Council. The other options, including the sheikh/mukhtar, Provincial Council, and line ministry, only registered a combined total of 18% of respondents. Despite their importance in project implementation, Local Councils were dependent on others for actual allocation of project funding. In focus groups, Local Councils were asked, “How do you get the funding to do the project?” Of the 45 groups that responded, the IP was the most frequently mentioned as a source of support, identified by 84% of the Local Councils. Just over half of those responding, or 23 focus groups, described the role that the line ministries played, both in providing approvals but also through their contribution typically of land, labor, or equipment needed to complete the project. The 9% of Local Council focus groups that said they provided support described the assistance in terms of helping to seek approvals or monitoring project implementation, but none had budgets to provide financial support. As Local Councils were dependent on others for resources, support, and approvals, the process of project implementation required navigating a range of stakeholders. This process of managing the bureaucracy was judged the most challenging part of implementing a CAP project. Of the 33 Local Council focus groups that discussed the most difficult part of implementing a CAP project, 39% specifically identified the challenges of obtaining approvals as the hardest part of the process. The same number of focus groups did not specifically mention approvals, but highlighted the challenges of coordinating among the various stakeholders involved in project implementation, such as line ministries, community members, and Provincial Councils. Other USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 27 Local Councils (18% of 33 focus groups) said that the hardest aspect was designing a program that satisfied community needs within the limited funding available. No Local Council representatives anticipated that they would be able to immediately supplant the USAID support provided by the CAP III program. In focus group discussions, "support' was understood to mean the financial assistance to complete the project, as well as the connection with the community provided by the CAG and the training assistance offered by the IP. The most widely held view, expressed among 42% of the 33 CAGs, was that the support, regardless of how the question was interpreted, would not be replaced after the end of CAP III. Some 27% of 33 CAGs said that the Local Councils would turn to foreign donors or NGOs, though respondents were not optimistic that this assistance would offset the loss of CAP support. An equal number said they would look to the Iraqi government, including Provincial Councils, line ministries, and the central government. CAP provided training in a range of tools and approaches to help local government officials understand community needs. IPs provided all Local Councils with a range of training relevant to their work. At least one Local Council member in each of the Local Council focus groups said that he/she received some training from IPs. Of the 35 focus groups that discussed the type of training, 69% said they received some form of program management training, which typically includes procurement and budgeting. Some of the program management trainings were more specialized, such as Mercy Corps’ training on project advocacy, which was hailed by three different focus groups and its trainer singled out for his effectiveness. More specific trainings that were highlighted as most useful were the trainings on advocacy, leadership, and Law 21, which provides governors and provincial legislators with more decision-making authority. Local Council focus group participants requested more courses. A recurring point, raised by 24% of the 25 focus groups that addressed the question of which trainings would be helpful in the future, was a request for training abroad, irrespective of content. An equal number requested more training on government operations. In several Local Council focus groups, there were requests for other trainings that varied dramatically from the offerings provided by CAP partners, such as English language and computer instruction. Though several focus groups highlighted the training on understanding citizen's needs as the most valued, this was not requested as a topic for additional training. When discussing changes in the relationship between the government and the community, many of the Local Councils felt that it was easier to meet and receive requests from their community. Of the 48 focus groups that addressed the question “How do people in the community usually contact you and other government leaders to let them know what projects they need? How successful are they in getting their needs met from the government? And has this process gotten easier or harder in the past four years?” 40% explained that the process of receiving and addressing community concerns was easier. The other respondents did not comment on whether the process was easier or harder but described receiving concerns through a mix of informal and formal processes including complaint boxes, official letters of correspondence, and formal and informal meetings. None of the focus groups referred to the training when discussing how they received community concerns. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 28 The experience of CAP project implementation complemented the training and helped Local Councils better identify community needs. When asked “How did your involvement in the CAP program change the way you advocate on behalf of your community’s needs to the Ministries, Provincial Councils and Parliament?” only 9% of the 41 focus groups that answered this question said that there was no change. Of the 37 focus groups that said their advocacy changed, 62% or 23 offered an explanation. For 52% of these 23 focus groups, CAP helped their ability to advocate for community needs by providing improved advocacy tools. One CAG focus group reached a consensus that "In the past, the response to community needs was random. But the CAP program made us change our way of advocating on behalf of our communities by listening to the community and preparing the list of needs and then prioritizing these needs according to the resources we have. We can say we are more organized." CAP provided a forum for engagement between the Local Councils and other national and sub-national government officials. Interactions required by CAP project implementation forced Local Councils to seek support from other parts of the national and sub-national government. This interaction was required as part of helping CAGs secure project approvals. When Local Councils were asked in focus groups “How were you involved in the selection and design of this project for the community?” only 13% of the 40 Local Council focus groups that addressed this question discussed a relationship exclusive to project selection. Much more frequently mentioned, discussed in 46% of the 37 groups, was how the project designated by the CAG required the Local Council to work with the Provincial Council and the line ministry for project approval. The coordination between government stakeholders had implications for sound project implementation and maintenance. Among the projects that were assessed in Project Observation Forms, 74% of the 58 were judged to be in “Good” or “Excellent” condition. According to both CAG and Local Council focus groups, line ministries were generally responsible for project maintenance and operations. Without line ministry coordination and prior approval, IP staff members did not think projects would have been implemented with the same level of quality. There are indications that CAP III supported Provincial Council members’ understanding of local needs. In interviews with Provincial Council members, they described CAGs as a valued albeit informal mechanism for obtaining important information on community necessities. In some cases, Provincial Council members did attend CAG-organized community meetings, which some found worthwhile. One Provincial Council member said, “We consult CAGs almost on any problem in any area. CAP had helped me a lot and lifted a big burden off our shoulders through defining community needs. It was also quicker than the government in implementing projects.” Another explained, “The CAP program increased my knowledge and information and helped me develop my ways of knowing community needs.” In the area of service delivery, CAP III served niche roles that the Provincial Councils could not. For example, a Provincial Council member remarked “CAP III was very active in the disputed territories where we were not able to work because these areas were controlled by the Kurdish leadership. So the CAP program helped in responding to the needs of the people in these areas and covered in our absence. CAP III was the peace pigeon and we appreciate what they did.” USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 29 CAP-provided infrastructure projects were often most appreciated by the community and the government, though this did not overshadow the appreciation for CAP’s participatory process of collaboration and partnership. Although CAP projects are designed to serve as the means for community mobilization and engagement outcomes, Local Councils most appreciated the projects themselves. However, there was an acknowledgement of the importance of the CAP process and some recognition that this did change the way these officials interact with their communities. Local Council members shared the perception of CAG members that the most significant contribution of CAP III was the implementation of the project itself. Of the 41 Local Council focus groups that discussed the question of “What do you consider to be your greatest success?” 80% pointed to the project that was implemented in their community. Five percent of the 41 responding Local Council focus groups spoke exclusively of the process as being their biggest accomplishment, saying that it "enabled the community to understand the councils’ responsibilities and increase their confidence in the Local Councils" and that the Local Council held "public seminars and communication with citizens." Another 17% of the 41 focus groups described the significance of both the project and the process. There are indications that the Local Council members recognized that the community might have changed in the past four years in terms of the way constituents interact with Local Councils. In focus groups, Local Council members were asked “How do people in the community usually contact you and other government leaders to let them know what projects they need? How successful are they in getting their needs met from the government? And has this process gotten easier or harder in the past four years?” All of the 19 focus groups that discussed how the process had changed described it as easier than four years ago. Of these 19 focus groups, 42% specifically mentioned the CAG as a new channel of communication with the community. A member of the Basra Local Council pointed out that "Through holding joint meetings between the government and the communities, CAP eliminated the fear or reluctance of citizens to speak openly to government officials. The fear barrier that was present before is eliminated now with the help of CAP." Conclusions Local Councils are becoming increasingly relevant. Though still unelected and without their own budget (with notable exceptions in Karbala and Najaf), Local Councils played an important role in the CAP III process and play an increasingly significant role in the sub-national system of governance in Iraq. As described in focus groups, Local Council members’ relationships with Provincial Councils are becoming more defined, especially around the critical issue of submitting input to the provincial budgets. However, many Local Councils are still struggling for status when engaging certain Provincial Councils and ministries. Local Councils are still dependent on line ministries for service delivery. Communities view Local Councils as their “nearest” access to the government. As surveys of IP community mobilizers and focus groups with CAGs and Local Councils revealed, Local Councils were the main governmental point of contact for the CAG program and the appropriate level of engagement for basic community needs. Although there were some remote communities that lacked Local Council representation, Local Councils were widely recognized as the first entity to contact for a basic infrastructure project. Local Council members emphasized the multiple channels open for community members to share requests and spoke positively of the USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 30 trainings designed to help them channel requests and better understand government operations and program management. Iraqi communities that have participated in Local Council elections are taking community service into account when casting their vote. Although Local Council elections have been held in only a few locations, the initial feedback is that voters recognize which officials have been more receptive to community needs and able to address citizen concerns. Focus group comments underscored that there is political capital to be won when projects are completed. Local Councils are more active and better equipped to prioritize and respond to citizen’s needs. This point was recurrent among focus groups with Local Councils, surveys of community mobilizers, and CAG members. Through more experience navigating the system, improved tools to manage and prioritize community requests and CAP training, Local Councils are better able to respond to citizens needs than they were four years ago. Recommendations Encourage Local Councils to convene and facilitate public meetings for community input, as well as mechanisms for reporting back responses to the community. Some channels for community input to the Local Council existed before CAG activities, though these were expanded under the CAP program. The public meetings that brought community members and Local Councils together should continue, even if CAGs do not continue their existence in their current form. Continue to train Local Councils in project management, advocacy, and budgeting in order to achieve greater effectiveness in the provincial budgeting process. As the budgeting process is revised to allow more input at the local level and in population￾based budget planning, it is important that Local Councils be prepared to contribute. When the new budgeting process takes effect, there will be a significantly greater burden placed on local and Provincial Councils to provide improved services and increased responsiveness. Connecting skill-building exercises to this impending change will help Local Councils meet this challenge. Encourage continued Local Council capacity building and engagement through support for the IRD-initiated Local Council Association. IRD provided training to Local Council members on how to form associations and created the Local Council Association, an NGO that is in the final stages of its registration process. This new organization will help Local Councils share best practices on program management, advocacy, and budgeting. It addresses an unmet need, given that the Local Government Association, which could serve this purpose, does not accept Local Councils because their members are not elected. D. RESEARCH QUESTION 4: Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the Fund’s defined purpose? If not, why not? Although the Marla Ruzicka Iraqi War Victims Fund has been active since 2004 and U.S. Coalition forces withdrew from Iraq at the end of 2011, eligible war victims are still in need of USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 31 benefits provided by the Marla Fund. However, these victims face barriers to accessing support through the fund. Marla Fund beneficiaries do not see traditional social networks as a reliable form of alternative support should the Marla Fund end. Based on interviews with local NGO and implementing partner staff, the evaluation team does not currently advise sustaining service provision beyond the CAP III period through local NGOs. While the Iraqi Government has mechanisms for addressing the needs of Iraqi war victims, potential beneficiaries do not find these mechanisms to be effective and assert that such programs would require political and technical support to activate. Findings There are uncounted additional eligible victims that have not received support from the Marla Fund. Based on interviews with implementing partner’s Marla Fund program managers, local NGO partners, and Marla Fund beneficiaries, there are many more victims of U.S. Coalition forces who have not accessed support.16 However, respondents were unable to provide a count of the unmet need. In an effort to quantify the gap between those potentially eligible and those who have already been served, the evaluation team reviewed literature on estimates of Iraqi war victims. The Journal of Humanitarian Assistance (2009) reviews direct and indirect estimates, identifies the range of factors influencing counts, and notes that "reports range from 128,000 to 1,033,000.”17 Estimating the number of non-fatal injuries is especially problematic. The most widely cited attempt is an ABC News/USA Today/BBC/ARD Poll, which found that "53 percent of Iraqis say a close friend or immediate family member has been hurt in the current violence. That ranges from 3 in 10 in the Kurdish provinces to, in Baghdad, nearly 8 in 10. (The size of extended families in Iraq likely contributes to this result.)"18 With such wide-ranging estimates of deaths and imprecise measures of wounded, the evaluation team was not able to quantify the level of unmet need; therefore, this assessment relies on Marla Fund stakeholder testimony. Surveyed Marla Fund beneficiaries and IP community mobilizers were in general agreement regarding explanations for why some Iraqi war victims were not helped by the Marla Fund (see Table 3: Reasons War Victims Do Not Access the Marla Fund). Respondent cited as the most common reason was that U.S. Coalition forces had not harmed the victims applying for Marla Fund support. In these cases, the victims are ineligible for the Marla Fund, which does not provide support for those harmed by terrorist and insurgency groups. Only about 6% of respondents overall thought people did not access the Marla Fund because they already had all the support they needed. Both of these points are further explored in Finding 2 regarding alternative sources of support. Aside from ineligibility, by far the most frequently cited reason for not accessing Marla Fund support is victims’ inability to provide adequate documentation verifying eligibility. Respondents explained that providing a death certificate or medical report was the most problematic. In 16 According to self-reported IP data, CAP III implemented nearly 2,000 projects through the Marla Fund, benefiting nearly 178,000 Iraqis. (USAID’s “Roll-Up” of Community Action Program (CAP III) Report, as of March 31, 2012.). 17 Karagiozakis, Maria. “Counting excess civilian casualties of the Iraq War: Science or politics?” The Journal of Humanitarian Assistance (Feinstein International Center). June 22, 2009. 18 “Ebbing Hope in a Landscape of Loss Marks a National Survey of Iraq.” ABC News/USA Today/BBC/ARD Poll. Embargoed for release after March 19, 2007. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 32 Table 3: Reasons War Victims Do Not Access the Marla Fund Implementing Partner Community Mobilizers Marla ACDI/ Beneficiaries VOCA CHF IRD Mercy Corps Total They were not harmed by U.S. Coalition forces 84% 93% 90% 82% 88% 90% They could not provide verification to prove eligibility 87% 87% 88% 64% 85% 89% They had not heard of the Marla Fund 13% 27% 29% 45% 25% 24% Marla Fund did not provide the support they needed 3% 7% 10% 9% 7% 9% They did not want to accept support from the U.S. 3% 20% 10% 0% 8% 7% They had all the support they needed 3% 20% 5% 0% 6% 6% Number of survey respondents: 31 ACDI/VOCA, 17 CHF, 42 IRD, 11 Mercy Corps, 101 total IP community mobilizers, 108 Marla Beneficiaries. Percentages do not total 100% because some respondents provided multiple responses to the question: What are the reasons war victims do not access the Marla Fund (check all that apply)? particular, many of these documents did not explicitly indicate that the cause of death or injury was Coalition forces. Being unaware of the need for such specificity, family members did not seek clarification at the time of the incident and were unable to obtain more accurate documentation long after. Also, with so much chaos and dysfunctional bureaucratic processes at the height of the war, respondents explained that many documents simply indicated “terrorist attack,” a designation that automatically disqualifies Marla Fund applicants. Although IP staff indicated concern that eligible victims were not able to provide required documentation in stakeholder interviews, they also warned against loosening the criteria. IP community mobilizers who were opposed to changing criteria explained that there were insufficient funds to meet the current level of need. In the case of two COPs, they warned that the Marla Fund had been around long enough that Iraqis know how to “game the system” and that the availability of fraudulent documents means that more scrutiny, and not less, was needed. One warned that the next Marla Fund implementer should follow a practice to “trust but verify” to avoid abuse. Despite concerted efforts to publicize the availability of support through the Marla Fund, about 25% of IP community mobilizer and beneficiary survey respondents indicated that victims were still unaware of its availability. Outreach efforts included radio announcements, consultation with community leaders, and outreach through CAGs. In one case, a community mobilizer explained that she worked through staff at the provincial office for the Ministry of Labor and Social Affairs to access lists of potential victims that might overlap with those seeking widow or orphan support from the Iraqi Government. Although less than 10% of IP community mobilizers and beneficiaries indicated that the Marla Fund did not provide the support needed, this issue of Marla Fund support not matching USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 33 victims’ needs was a strong theme during the Marla Fund beneficiary focus group. The most common unaddressed needs were serious medical issues (e.g., amputated limbs, burns, and surgical corrections). One beneficiary explained that their family member was in need of three surgeries but only received one. Another’s granddaughter has ongoing psychosomatic symptoms, being despondent and unable to speak since the death of her parents and her own serious injuries years before. Also, some IP community mobilizers involved with the Marla Fund as well as local NGO staff note the following concerns with the income-generation projects that provided start-up supplies for small businesses: (1) many beneficiaries are illiterate and lack the basic skills to run a business, despite being provided training and support; (2) the total value of the support provided is inadequate to sustain a viable business; (3) those with severe physical injuries are ill￾equipped to manage a business, especially without specialized equipment to accommodate their disability; and (4) some women beneficiaries (especially those in conservative areas) are vulnerable to losing the business to male family members. Less than 10% of the respondents thought that eligible victims did not access the Marla Fund because they did not wish to accept support from the U.S. Instead, beneficiaries indicated that it was appropriate that the U.S. provide compensation in some form, although most preferred the cash support that had been available through the U.S. military and U.S Provincial Reconstruction Teams (PRTs) over the in-kind projects provided through the Marla Fund. One beneficiary explained that this practice of compensation was consistent with the tribal practice of blood money payments. In fact, 63% of surveyed beneficiaries indicated that the Marla Fund changed their opinion of the U.S. by either “significantly improving” (36%) or “improving a little” (27%) their view of the United States. Only 6% indicated that their opinion had “worsened” and 31% indicated there was no change as a result of the program. IP community mobilizers and Marla Fund beneficiaries rank both NGOs and the Iraqi government above traditional social networks as replacement sources of support for the Marla Fund. Both IP community mobilizers and Marla Fund beneficiaries look to the Iraqi Government as the most important and likely alternative source of support to victims of war and their families. (See Table 4: Alternative Sources of Support for Marla Fund Beneficiaries.) This view is understandable and appropriate, given the Iraqi Council of Representatives passed legislation in 2009 mandating support for all victims of war, regardless of the cause of harm.19 Similar to the Marla Fund, the law allows for compensation for death, missing persons, disability, injury, damage to property, and damage related to loss of job or schooling. It further establishes formulas for compensation as well as procedures and institutions for implementation. However, this law has yet to be executed. Other probable government entities that could potentially address the needs of Iraq’s victims of war are the Ministry of Labor and Social Affairs (which administers programs to support widows and orphans) and the Independent Committee for Marla Fund, the law allows for compensation for death, missing persons, disability, injury, damage to property, and damage related to loss of job or schooling. It further establishes formulas for compensation as well as procedures and institutions for implementation. However, 19 Law No. 20/2009 Compensating the Victims of Military Operations, Military Mistakes and Terrorist Actions. Dec. 28, 2009. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 34 Table 4: Alternative Sources of Support for Marla Fund Beneficiaries Implementing Partner Community Mobilizers Marla ACDI/ Beneficiaries VOCA CHF IRD Mercy Corps Total Iraqi Government 35% 41% 10% 45% 27% 24% Iraqi NGOs/CSOs 23% 0% 12% 0% 12% 19% International NGOs 23% 35% 52% 27% 38% International 17% governments 0% 0% 7% 0% 3% Religious institutions 3% 6% 2% 0% 3% 3% Tribes/family networks 16% 18% 17% 27% 18% 1% Number of survey respondents: 31 ACDI/VOCA, 17 CHF, 42 IRD, 11 Mercy Corps, 101 total IP community mobilizers), 108 Marla beneficiaries. Percentages do not total to 100% because some respondents provided multiple responses. Question to community mobilizers: “In addition to the Marla Ruzicka Iraqi War Victims Fund, who within Iraqi society is the most important provider of support to victims of war and their families?” Question to Marla Fund beneficiaries: “Where would you go to replace the support you are receiving if the program stopped providing services?” Marla beneficiaries’ responses included as “Iraqi Government” are “Local Councils” (20%) and “Ministry/Directorate” (4%). Marla Fund beneficiaries’ response option was “International organization;” CM response options were either “International NGOs” or “International governments.” The sole option for Marla beneficiaries for Tribes/Family networks was “Family.” this law has yet to be executed. Other government entities that could potentially address the needs of Iraq’s victims of war are the Ministry of Labor and Social Affairs (which administers programs to support widows and orphans) and the Independent Committee for Victims of Terrorism (which reportedly has over 6,000 registered cases without any reported progress in addressing the need). At the time of the preparation of this report, neither of these institutions has implemented systematic support for victims. Further, there is no comprehensive database of victims, which would be an initial starting point for verification of need and budgeting purposes. The second most cited source of alternative support are Iraqi and international NGOs. Turning to Iraqi NGOs is especially interesting, given the general skepticism within Iraqi society regarding civil society organizations.20 For victims seeking medical support, surgeries, management of disabilities, or trauma healing and psychosocial support, NGOs may be a more apt provider of services than the in-kind support currently available through the Marla Fund, which under CAP III has focused more on income generation (with the notable exception of Mercy Corps and IRD’s continued provision of medical support).21 Iraqi local nongovernmental organizations (LNGOs) are ill-equipped to continue administration of the Marla Program without support 20 Barbour, Josephine, Melissa Brill, and Jeffrey Swedberg. “Iraq Civil Society Assessment.” The QED Group, LLC. February 2012. 21 According to self-reported IP data, CAP III implemented nearly 2,000 projects through the Marla Fund, with 66% in income generation followed by 25% for medical support, 9% for property repair, and only 2% for community projects. (USAID’s “Roll-Up” of Community Action Program (CAP III) Report, as of March 31, 2012.) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 35 Based on implementing partner interviews, their LNGO partners are not prepared to take on sole responsibility for administering the Marla Fund following the end of CAP. The common concern among IP staff is that the LNGOs are unable to meet auditable standards for financial management. Additionally, the LNGOs are not immune from the general suspicion within Iraqi society regarding their commitment to serving the Iraqi community. Despite these shortcomings, IP staff commented that LNGOs that have established relationships in their respective provinces might be well-positioned to identify eligible victims. If they are organizations with an established track record of providing related services or working with relevant populations (e.g., business development services, medical assistance, work with people with disabilities, assistance to widows, etc.), they may be successful in handling the non-financial aspects of service delivery. One IP COP defined them as adequate “task order drivers” but assessed them to be unprepared to manage administrative functions. There is some variation among the LNGOs in terms of readiness to step into the role of carrying on the Marla Fund program. Mercy Corps and IRD LNGO partners have had more experience, including programmatic work additional to what is funded through the Marla Fund, and so are likely to have better systems in place.22 With the notable exception of IRD’s partners, open Marla Fund projects for all LNGO partners are low, ranging from 5 to 36 projects per organization. IRD’s two partners manage about 100 projects each. The least prepared are CHF’s partners handling cases in south-central provinces, which just began their involvement with the program in recent months and are even further behind in their capacity to take on this responsibility.23 Conclusions Level of eligibility outstrips allocation of funds to provide meaningful support to all eligible individual cases. Although the precise number of war victims is unknown, by all counts it outstrips the capacity of Marla Fund to provide needed benefits to all those eligible for support. All IP Marla Fund managers and LNGO partners cited lack of sufficient funds to meet the current caseload. They also indicated that the established per capita limit per project is insufficient to provide the most valued support, such as expensive surgeries or sustained support for successful business start￾up. In some cases, IP and LNGO staffs indicate that they limit their public outreach efforts in recent years because the program’s funds are insufficient to meet the true demand resulting from increased awareness of the Marla Fund’s availability. Perhaps in an effort to ration 22 The evaluation team did not conduct a thorough audit of LNGO practices, facilities, or staffing so a determination of capacity is dependent upon IP staff commentary and LNGO self-assessment. 23 Mawtini Organization for Youth Development and Iraqi Human Rights Watch Society signed their contracts with CHF on February 16, 2012, to cover south-central provinces. Each are currently handling only five projects each. Al-Muna Humanist Organization has worked in Anbar with CHF since 2010 (11 projects). Mercy Corps began its work with Bahja Al-Fouad in 2010; its current caseload is 36 projects. IRD has worked with Iraq Health Aid Organization (IHAO) (medical, 98 patients) since May 2011 and Iraq Association of Securities Dealers (IASD) (income generation, 102 projects) since 2010. ACDI/VOCA began its work in Ninawa with Ammal Al-Watan Center for Development in 2011 (income generation, 6 projects) and Haraa Humanitarian Organization (HHO) in Diyala two years ago (income generation, 11 projects). The evaluation team did not interview staff from Afkar Association for Development and Relief who worked with CHF in Anbar, or Al Murshed Center for Economic Development and Investment (Kirkuk) and Al Malwiva Relief Foundation for Development (Salah ad Din), which worked with ACDI/VOCA. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 36 allocations, IPs have introduced eligibility guidelines beyond those legislatively mandated, such as age of beneficiary, maximum length of time since harm occurred, and minimum level of economic hardship. In other words, if an applicant is too old or young, was injured several years ago, or is not poor, he or she is unlikely to receive support. Further, eligibility varies by IP so there is not consistent eligibility across the country. With such a sense of imbalance between supply and demand and variation in eligibility criteria, it is difficult for IPs to achieve equitable and consistent administration of the program. Current documentation requirements are a significant barrier to accessing support. Current documentation requirements to establish cause of harm are unrealistic for many potentially eligible Iraqi war victims. Interviews with IP Marla Fund managers and LNGO partners as well as IP community mobilizers involved in the Marla Fund describe a pattern of rejecting applications based on failure to establish whether U.S. Coalition forces are the cause of harm. Respondents frequently mentioned inconsistency between police and medical reports and court documents as cause for rejection. Respondents described careful efforts to identify fraud through triangulation of information, leading some to speculate whether verification tactics were both leading eligible applicants to rely on phony documentation and whether ineligible applicants were becoming more skilled at working the system. Small business start-up requires intensive training and ongoing support. According to Marla Fund beneficiary focus groups and IP and LNGO staffs, small business start￾up requires more investment of capacity building and longer-term support than the current projects provide. Respondents described Marla Fund beneficiaries as low-skilled and in many cases illiterate; the evaluation team saw this characterization further evidenced when the Marla Fund beneficiaries required assistance to read and complete the evaluation team’s surveys. Accessing employment with an established employer as opposed to becoming a business owner may be more feasible for the Marla Fund beneficiaries who face multiple hardships, such as disabilities, poverty, and isolation. Community-based projects have potential to reach a broader public. Community-based projects reach a larger pool of beneficiaries and do not require the same complications of determining individual eligibility. Several infrastructure projects would provide services highlighted by Marla beneficiaries as important for their ongoing recovery. Examples of these include: physical therapy services, rebuilding of damaged community infrastructure supportive of economic development (e.g., electricity, roads, marketplaces), surgical services provided by visiting specialists (e.g., plastic surgeons, orthopedists, neurologists), or reengineering of workspaces and factories to accommodate employees with physical disabilities. Such efforts would reach a larger population with reduced exclusion of eligible victims. Recommendations The ending of the CAP III program provides an opportunity for USAID to rethink its method of assisting civilian Iraqi victims of conflict in terms of (1) trade-offs between individual and community projects, and (2) support of the Iraqi Government to better meet the needs of its population harmed by years of conflict, regardless of the source and type of injury. A revised approach would involve both BPCS as well as GSP. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 37 Conduct an intentional transition that captures data and maintains relationships with CAP-era partners by: Establishing a transition committee of key stakeholders from CAP’s Marla Fund implementation, including IP Marla Fund program managers, community mobilizers involved with the Marla Fund, LNGO personnel, and beneficiaries. Ideally, ad hoc committee members would be contracted for their services. Obtaining person-level data of Marla Fund applicants and recipients from all four CAP III implementing partners to initiate a national database of prior applicants. To improve provision of support to individual beneficiaries, USAID can take steps to provide a more uniform program that emphasizes transparency, consistency, and suitability by: Establishing a working group of CAP III participants familiar with the challenges of eligibility verification to develop a process that minimizes the possibility of excluding eligible victims, minimizes bureaucratic procedures, and ensures against fraud (e.g., LNGO case workers, IP community mobilizers, Marla Fund program managers, Local Council members). Standardizing eligibility criteria for all applicants to promote equitable assessment and access. Instituting a uniform system for rationing support that acknowledges and addresses the inadequate funding level. Providing more intensive preparation and longer-term coaching to support small business start-ups; this preparation is even more relevant for beneficiaries with low educational attainment or who are illiterate. Customizing projects to individual needs and priorities, including the amount of the grant and type of service provided. Adopting a supportive services and case management model in which the service provider establishes developmental goals, a work plan with benchmarks and beneficiary responsibilities, and an established point of “graduation.” Adding trauma healing and psychosocial support services to the suite of available support, especially for children and young people. Exploring the feasibility of sweat equity programs for reconstruction of homes and businesses, with on-the-job training and support. To increase the reach of the Marla Fund to a larger population of war victims and reduce the complications of individual eligibility determination and the fact that the Marla Fund funding allocation is insufficient to meet the needs of all eligible war victims, USAID should take steps to invest in more community projects in areas of continuing need: Conduct a national assessment to identify locations that have enduring infrastructure damage as well as higher concentrations of impacted communities; prioritize these areas. Develop a community-based model for providing ongoing medical needs (e.g., national surgical center for treatment of war victims, rotations of international medical specialists USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 38 such as plastic surgeons, local capacity to provide physical therapy, factories to manufacture high-quality prosthetics). Create jobs with established employers as an alternative option to small business start￾ups, working with employers to develop work environments that can accommodate employees with disabilities and with potential employees to ensure they are properly trained. Establish clear selection criteria and performance responsibilities for LNGO implementation partners. Request that IP Marla Program managers provide an assessment of their LNGO partners to determine if collaboration should continue. For continuing as well as new LNGO partners, develop a capacity-building plan and mentoring program that indicates long-term commitment and a graduated transition of responsibility based on meeting performance benchmarks. Support the Iraqi government in implementing Law 20/2009 and building a sustainable means for providing support to Iraqi victims of war, regardless of cause of harm: Provide technical support for the development of a national database to facilitate tracking of victims, reduce potential duplication of support, and manage eligibility determination. Although the Ministry of Labor and Social Affairs may be an appropriate partner for administering the program, an alternative means for disbursement of U.S.-provided funds would be required until anti-corruption measures are fully implemented. Explore whether the Ministry of Interior’s Victims of Terrorism Committee is an appropriate partner for this effort. Consider a public-private partnership model in which civil society provides the needed services but the government provides the funding as well as referrals. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 39 V. LESSONS LEARNED AND RESEARCH LESSONS Provide lessons learned that could be applied to future USAID work with Iraqi communities and local governments specifically on approaches to community engagement/participation, training methodology, and securing government buy￾in/participation. A. COMMUNITY ENGAGEMENT/PARTICIPATION When beginning implementation of any USAID initiative, engage the highest jurisdictional level to promote transparency and political support. As CAGs lacked any official government mandate or recognition, their status was sometimes questioned within the community as well as at higher levels of formal leadership. To facilitate community-based informal leadership, future USAID programming needs to follow the CAP IP examples of gaining the support and validation of established leadership. Even in the case of community-based projects, it is essential that USAID’s partners reach out to provincial-level officials. CAP III IPs’ outreach took a variety of forms, including memoranda of understanding, letters of support, or partnership agreements. Ideally these understandings should be in writing in both local languages and English. Seek out “neutral” venues for meeting and events. CAG projects did pay political dividends for many of the parties involved, especially at project completion. In instances where there were clear socioeconomic or ethnic and cultural divides, it is critical to be as impartial as possible in the delivery of services and to take precautions against appearing to favor one side or another. Choosing impartial or neutral venues for meetings and events related to the project can support this. Use the diversity of staff to model cooperation and provide “relatable” partners. IP staff members, in particular community mobilizers, were frequently from the community where they were working. This improved community mobilizers’ access to the community and increased the community’s receptivity when first introduced to the CAG concept. More staff diversity can help community mobilizers better model the same type of cooperation they seek to encourage when working in communities. Integrate new programming into existing political and societal systems to support incremental change and strengthen democratic institutions. The creation of a parallel form of service delivery like the CAG model was critical in the period immediately after Operation Iraqi Freedom. With Iraq now fully sovereign, it is important that new programming target existing formalized systems and work to support incremental change. Accommodate non-English speakers. Community programs must not exclude important stakeholders who are unable to communicate in English. Trainings, written materials, public meetings, and other program events should be conducted in local languages, or at a minimum with translation Work with champions, role models, and trail blazers within vulnerable groups. Choosing leaders who will share key messages following their engagement in the program can expand CAP’s work with vulnerable groups. CAP did engage a range of vulnerable groups but a deliberate attempt to pick future leaders in these groups could yield significant dividends in the future. They provide inspiration and encouragement to others that identify with members of their USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 40 identify group, whether they be other women, young people, ethnic or religious minorities, or IDPs. B. TRAINING METHODOLOGY Whenever feasible, use experiential learning approaches to training. Both IP trainers as well as training participants (i.e., focus groups of CAGs and Local Councils) emphasized the value of training that was immediately relevant to their lives and integrated with the task at hand. Learning by doing and on-the-job training were by far the preferred methodology. This strategy includes “shadowing,” with CAG members accompanying IP community mobilizers to government offices to observe how to conduct advocacy meetings. Other examples included learning various assessment mechanisms by conducting an assessment of their own communities and learning proposal writing through preparing and presenting their own project proposals to government officials. Enable relationship-building and cross-sector understanding of roles through joint training between government and non-government as well as national and sub-national leadership. Repeatedly, CAP III participants attributed joint training between CAG members and government officials – especially Local Council members but also Provincial Council members and line ministry staff – as the most impactful mechanism for improving mutual understanding and appreciation of each other’s roles as well as improved collaboration. In some cases, government officials highlighted training as helping them to overcome initial skepticism and suspicion of CAG intentions. This effect can be potentially explained by Contact Hypothesis (i.e., under appropriate conditions, interpersonal contact is one of the most effective ways to reduce prejudice between rival groups24) as well as exposure to each other’s demonstrated value. Integrate peer review or learning opportunities that include, when appropriate, site visit exchanges. An IP community mobilizer in Dhi Qar has initiated use of this strategy. She has started to bring CAGs and Local Council members to visit “successful” projects to meet with beneficiaries and project organizers. She described the visits as “inspirational,” especially for newer CAG members who were still learning how to approach their responsibilities. These peer reviews and site visits also provided an opportunity to switch roles, enabling the “student” to become the teacher. Provide easily accessed and updated materials for post-visit follow-up and references. Several training participants complained that their training sessions did not include take-away materials to use as references as they were conducting their work. They suggested hard copies be made available, especially in communities where Internet access is less reliable. For those with connectivity, trainees preferred online resources through a clearinghouse format in which the most current versions were readily available. All materials should be available in Arabic, Kurdish, or the language relevant to the community. Use a diverse (majority/minority, gender, age) training team to model cooperation across “differences.” ACDI/VOCA relied on this strategy heavily, especially when working in divided 24 Appropriate conditions according to hypothesis theory or intergroup contact theory include: (1) equal status between/among groups, (2) common goals in which both groups work on a shared problem/task or “subordinate goal,” (3) acquaintance potential in which participants have the opportunity to socialize and cultivate familiarity or friendship, and (4) support of authorities, law, or customs that support the interaction. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 41 communities and disputed territories. Having a diverse representation – whether in terms of gender or ethnicity or age – ensures that training participants will have at least one person with whom they feel they have a shared background. It also demonstrates healthy and productive relationships across identity divides. Use creative and context-sensitive strategies to include women in training opportunities. There are advantages and trade-offs to including men and women together in the same trainings as oppose to implementing a “separate but equal” approach to training. The surveyed IP community mobilizers (both men and women) far preferred conducting joint trainings. Joint training offer all the advantages highlighted above regarding Contact Hypothesis. Additionally, they demonstrate that men and women are treated equally within the program. IPs were quite creative in opening space where little existed for women’s full participation. One such example was a CHF training in which both men and women were trained in the same room but with a partition down the center to create a visual barrier between the men and women. In front of the women trainees was a female facilitator and in front of the men was a male. In another training, the women were required to sit outside the meeting room but a loudspeaker was set up so that they could listen and a microphone was provided so that they could speak. However, in some traditional communities even these measures are unacceptable. To enable women’s participation it is useful to conduct separate all-women training with women trainers and to conduct in a home setting or a women’s center easily accessed by female participants. Even when co-ed trainings are possible, some women may prefer to conduct separate activities so as to have more of an independent voice without having to compete against a male-dominated hierarchy. When conducting inter-generational trainings, adapt training pace and topics to have cross￾generational appeal. Interviewed trainers see the advantages of intergeneration training in terms of promoting mutual appreciation and understanding. However, these trainings can also fall apart when young people, who tend to learn new concepts more quickly, want to move at a faster rate. Similarly, interests between the generations can be disparate, so trainers must be sure to provide examples and experiences with broad appeal. The trainer must also carefully manage power and hierarchy dynamics, given traditional patterns of social behavior that leave little space for youth voice or leadership. C. SECURING GOVERNMENT BUY-IN AND PARTICIPATION Maintain focus on trust-building and transparency throughout the implementation process. It is important to not “bookend” the engagement between the IP, the community, and the government by encouraging inclusion at the beginning and celebration at the end but leaving out substantive involvement throughout the process. Trust and mutual respect comes from intentional engagement during the planning phase, as well as implementation and conclusion. Initial buy-in can be easily lost by the end of an effort if the relationships are not nurtured throughout the lifetime of the project. Connect community-based work with provincial-level planning. By creating the linkages from bottom to top, program participants are able to better understand larger systems, establish meaningful relationships that are useful beyond a one-off effort, and promote equitable and efficient distribution of resources. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 42 Seek common understanding among program participants of roles, responsibilities, and expectations with respect for existing hierarchy and status. A successful relationship between individuals of different status – such as the CAG and the Local Council, or the IP community mobilizer and the Provincial Council – requires that both parties understand the other’s status and role in the hierarchy. CAP was able to engage the range of stakeholders when it was cognizant of this dynamic. As the roles of the Local Council, Provincial Council, and voluntary community leadership evolve with Iraq’s election cycles and new budgeting procedures, it is important to remain aware of these different roles and make sure they are properly respected. Bring something meaningful to the table. Though CAP program-provided training, tools, and experienced IP staff, the program was able to provide funding to implement community projects. Absent this support, community engagement would have been much more difficult to cultivate. This is not to say that the IP only provided funding – rather, the possibility of concretely meeting community needs through CAP III projects and the benefits projects would provide encouraged many communities to be engaged, and was the part of the CAP program that communities appreciated most. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 43 VI. ISSUES Disclosure of Conflict of Interest All U.S. and Iraqi evaluation team members provided a signed statement attesting to a lack of conflict of interest relative to the project being evaluated. No member of the team had a conflict, as indicated by their signed statements submitted at the time QED contracted each for services. QED staff provided a written and verbal explanation of conflicts of interest at the time the form was signed. The Evaluation Team Leader and QED Operations Manager again reviewed qualifying conflicts at the evaluation team training in the field. See Annex J for a blank copy of the Disclosure of Conflict of Interest form. Statement of Differences There were no significant and/or unresolved differences of opinion presented by USAID at the time of the outbriefing (conducted on June 18, 2012) or among the evaluation team members. USAID’s implementing partners have not been provided with a pre-release draft of the report; therefore, they have not had the opportunity for comment. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 44 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 45 ANNEX A. SCOPE OF WORK STATEMENT OF WORK USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation A. BACKGROUND INFORMATION The Community Action Program (CAP) was first introduced to Iraq in 2003, shortly following military action by Coalition forces that removed Saddam Hussein from power. The program was modeled on similar programs in post-conflict environments, such as Lebanon and the Balkans. In areas where conflict and violence have eroded the services available to a country’s citizens, this CAP model builds on the underlying concept of community-centered development. The first Community Action Program ran from 2003-2007. Due to no-cost extensions, the project continued to wind down even after CAP II started up in October 2006. A similar overlap occurred between CAP II and CAP III, which ended in December 2008 and started in October 2008, respectively. CAP I and II focused primarily on building the capacity of community action groups (CAGs) to fill the gap of local government and exercise true grassroots democracy by implementing projects on their own where necessary, and in partnership with local government where possible, to meet community needs. CAP III continued to work at the neighborhood level with CAGs, while adding a component to build the capacity of district and sub-district councils to respond to local development needs. 1. Identifying Information 1. Project: Community Action Program (CAP), Phase III 2. Award Numbers: 267-A-00-08-00503-00, 267-A-00-08-00504-00, 267-A-00-08-00505-00, 267-A-00-08-00506-00 3. Award Dates: October 2008 – September 2012 4. Funding: $322,960,000 5. Implementing Organizations: ACDI/VOCA, CHF, IRD, Mercy Corps 6. Agreement Officer’s Technical Representatives (AOTRs): Erin Epstein, Varghese Jacob 7. USAID Mission & Office: USAID/Iraq/Democracy and Governance Office Below is the breakdown of project information for all phases of the Community Action Program in Iraq. While the main thrust of this evaluation will focus on CAP III, it is important to consider the sum of CAP efforts in Iraq since 2003 as part of the evaluation. The following table shows program period and allocated budget by implementing partners in each program phase. 2. Development Context Problem or Opportunity Addressed When CAP III began, Iraq was emerging from a six-year struggle with political instability and sectarian violence. Years of conflict have taken a substantial toll and the country’s capacity for governance and economic production has been severely weakened and unevenly developed, as were its institutional capacity and infrastructure. Many Iraqis with education and skills have fled, which exacerbates all of the foregoing problems. The conflict has disrupted employment and USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 46 CAP I Budget Start Date End Date ACDI/VOCA $49,578,089 16-May-03 31-Mar-07 CHF $55,306,225 16-May-03 31-Mar-07 IRD $65,060,482 16-May-03 31-Mar-07 Mercy Corps $56,575,496 16-May-03 31-Mar-07 Save the Children $43,043,954 16-May-03 14-Jul-06 Total CAP I $269,564,246 16-May-03 31-Mar-07 CAP II Budget Start Date End Date CHF $147,013,258 1-Oct-06 31-Dec-08 Total CAP II $147,013,258 1-Oct-06 31-Dec-08 CAP III Budget Start Date End Date ACDI/VOCA $87,322,000 1-Oct-08 30-Sep-12 CHF $88,621,000 1-Oct-08 30-Sep-12 IRD $91,011,000 1-Oct-08 30-Sep-12 Mercy Corps $56,006,000 1-Oct-08 30-Sep-12 Total CAP III $322,960,000 1-Oct-08 30-Sep-12 Total CAP I, II, III $739,537,504 16-May-03 30-Sep-12 social services delivery and the Iraqi government has struggled at all levels to develop, finance and implement programs and projects designed to meet local needs. Nonetheless, Iraq has taken several key steps towards a more stable future. These include the passage of the Provincial Powers Law, which codified the legal and practical division of governance between the national, provincial, and local levels. CAP III has continued work in this context to build the skills of communities and local governments to ensure development needs are being met. 3. Target Areas and Groups CAP III was designed to empower Local Councils and citizens to work in partnership to meet the needs of the community. The specific goals of CAP III are to: 1. Improve the capacity of communities to better identify their needs, articulate their role, and mobilize resources; 2. Improve capacity of district and sub-district councils to meet the articulated needs of the community and mobilize resources;25 and 3. Increase assistance to civilian victims of conflict. 25 This was amended in 2010 to reflect that fact that local councils below the provincial level never got budget authority, as expected when the project was first designed in 2008. New language varies slightly by partner, but basically reads: “Local executive and representative Government in CAP communities better meet the articulated needs of the community.” USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 47 Four U.S.-based organizations – ACDI/VOCA, CHF, International Relief and Development (IRD), and Mercy Corps – implement CAP III in 15 of Iraq’s 18 provinces. CAP III operates at the neighborhood level in 15 of Iraq’s 18 provinces: Ninawa, At Tamim/Kirkuk, Salah ad Din, Diyala, Anbar, Karbala, Babil, Wasit, Najaf, Qadissiyah/Diwaniya, Muthanna, Dhi Qar, Basrah, Maysan, and Baghdad.26 The four CAP III implementing partners were assigned to areas of responsibility (AoRs) roughly in line with the areas each covered under CAP II. Figure A-1 shows a map of Iraq where each partner operates at the start of CAP III. Baquba was later moved from IRD’s AoR to ACDI/VOCA’s. Each CAP implementing partner developed a set of criteria to form community action groups (CAGs), which formed the basis of the project. The implementing partners worked with CAGs to develop their abilities to identify, prioritize, and seek solutions to development needs in their neighborhoods. While the overall project works broadly with populations in need, project was asked to ensure their programs were reaching vulnerable populations. In 2010, the project was asked to track its impact on internally displaced persons (IDPs). 4. Intended Results CAP III’s program and intended results are clearly described in its development hypothesis and the accompanying results framework. The CAP III development hypothesis posited that local community needs are best met by ensuring the active partnership of all community groups: citizens, local government, and business and social leaders, in identifying priorities and designing interventions. CAP I and II focused primarily on building the capacity of CAGs to fill the gap of local government and exercise true grassroots democracy by implementing projects on their own where necessary and in partnership with local government where possible to meet community needs. CAP III was designed to focus on furthering the evolution of community-centered development by building the capacity of local government to take on its proper governance role as the locus of community needs assessment, prioritization, project design, funding, and implementation. By providing high-quality technical assistance, training, and targeted funding with both and CAGs and local government, CAP III aimed to advance Iraqi democracy to the next level of organizational development and bring much needed local development to Iraq’s citizenry. 26 The three northern-most provinces of the country – Dahuk, Erbil, and Sulaymaniya – have a different development context and are not covered under the Community Action Program. (Note: CAP PowerPoint stated these areas not subject to provincial law.) Figure A-1: Map of CAP Partners AoRs (2008) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 48 PERFORMANCE GOAL: A STABLE, DEMOCRATIC AND PROSPEROUS IRAQ Defeat the Insurgency Legitimate and Effective Government Institutions Established Foundation for a Market-Based Economy Established Strategic Objective 9 Effective Local Government Strengthened IR 9.2 Capacity of Sub-National Government to Perform Its Core Functions is Improved IR 9.3 Outreach Mechanisms & Capacity for Citizen Participation in Decision-Making & Local Development are Institutionalized IR 9.1 Establishment of legal, regulatory and policy framework for decentralized local government is facilitated Passage of the provincial powers act except in KRG Sub-IR 9.3.1 Communities better articulate their needs Sub-IR 9.2.1 Communities better able to mobilize resources within the community to meet articulated needs Sub-IR 9.3.2 Qada and Nahiya better articulate community needs Sub-IR 9.3.3 Civilian victims of conflict assisted by the Marla Ruzicka Innocent Victims of War Fund Figure A-2: CAP III Results Framework in the Context of SO 9 5. Approach and Implementation In the original design, CAP III’s goal was “to increase the local government’s ability to identify, articulate and better meet the needs of its constituency.” As key assumptions about the authorities of district and sub-district councils failed to materialize (such as the absence of elections scheduled in 2009 for lower level councils), the program adapted and trained lower level councils and community action groups to develop priorities and communicate their needs up to Provincial Councils and government ministries that control resources and provide services. To achieve this goal, CAP III partners implement programming under the following three objectives and associated sub-intermediate results (sub-IRs), which fit into the results framework pictured in Figure A-2: Objective 1: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. Sub-IR 9.2.1. Communities better able to mobilize resources within the community to meet their articulated needs. Sub-IR 9.3.1. Communities better articulate their needs. Objective 2: Local executive and representative Government in CAP communities better meet the articulated needs of the community. Sub-IR 9.3.2. Qada and Nahiya better articulate needs. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 49 Objective 3: Civilian Victims of Conflict Assisted. Sub-IR 9.3.3: Civilian victims of conflict assisted by the Marla Ruzicka Iraqi War Victims Fund. 6. Existing Data The evaluation should build on, rather than duplicate, existing performance information on CAP III. Therefore, USAID will provide the evaluation team with a full package of briefing materials. The team should familiarize themselves with these documents before arrival in Iraq to a full understanding of the program. These documents will include at a minimum: Statement of work for CAP III and subsequent mini-RFAs Four (4) Cooperative Agreements and subsequent modifications Four (4) sets of project quarterly reports and work plans Four (4) Performance Management Plans (PMPs) Performance and financial audits finalized to date, including April 2008 OIG audit on Marla27 , April 2011 SIGIR audit for CHF, Nov 2011 OIG audit for IRD, Additionally, the team should be intimately familiar with the methodologies and findings from previous phases of CAP III. USAID will provide the following documents for the team to examine during the desk review phase: CAP I evaluation narrative report and associated annexes CAP II evaluation narrative report and associated annexes Documentation on Marla guidance provided to CAP partners Finally, the team must adhere to the standards outlined by USAID’s latest guidance related to evaluations, as outlined in the following documents: USAID Evaluation Policy as of January 2011 (http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) Checklist for Assessing USAID Evaluation Reports, V1.0 (http://www.usaid.gov/policy/evalweb/documents/EvaluationSOW-GoodPracticeExamples.pdf) TIPS #17 – Constructing an Evaluation Report, dated 2010 (http://www.usaid.gov/policy/evalweb/documents/TIPS-ConstructinganEvaluationReport.pdf) The evaluation team also may find it useful to consult a broad range of other background documents related to community centered democracy programs in countries such as in Lebanon, Serbia, and Montenegro in addition to other program documents to be provided by USAID/Iraq. 27 While this OIG audit on Marla pre-dates the CAP III program, the document is important for understanding some of the thinking that went into design of the third phase of CAP. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 50 B. EVALUATION RATIONALE 1. Evaluation Purpose CAP III represents the third phase of USAID/Iraq’s longest running development. An end-of￾project performance evaluation28 as CAP III enters its final year is important to provide insight into the long-term effects of the program. An evaluation at this stage is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals—at all result levels— have been achieved, and (2) to inform the implementation of future projects aiming to work directly with Iraqi citizens. It is expected that approximately two-thirds of the evaluation team’s effort will be devoted to an evaluation of CAP III activities’ results from 2008-2012, and one￾third of the evaluation team’s effort will be devoted to lessons learned from CAP’s specific efforts to link citizens and government. USAID/Iraq’s Democracy and Governance Office (DGO) expects that such an evaluation will be useful to the Mission in the following ways: To determine whether CAP III accomplished the intended results To determine the effectiveness of the current approach to community and local democratic governance and development To identify program approaches that have the greatest potential for improving civil society and governance sector 2. Audience and Intended Use The audience of the evaluation will be the USAID/Iraq Mission, specifically the Democracy and Governance Office, to serve the purposes outlined above. In addition, CAP in all its phases represents a significant investment of U.S. government resources. A rigorous and credible evaluation of CAP’s impact in Iraq offers evidence to inform Congress and the American taxpayer about the difference our assistance has made. This program evaluation will assess where Iraqi communities are in terms of the transition to participatory democracy after eight years of CAP programming in all its phases. It will also set a baseline for measuring the impact of USAID’s new civil society and governance investments moving forward. Finally, a strong evaluation of CAP also benefits USAID as an Agency by examining the success of the community-based development model in Iraq – an approach that USAID has applied in countries such as Serbia, Montenegro, and Lebanon. As such, the evaluation can and should provide useful lessons to other USAID Missions considering similar programs. 3. Evaluation Questions The performance evaluation will be based on a series of specific questions that aim to elicit the long-term effects of CAP III in Iraq. The descriptive and normative questions outlined below should incorporate before-after comparisons to determine what CAP has achieved. Specific 28 While this evaluation will examine CAP III’s impact, it will focus on performance rather than serve as an “impact evaluation” in USAID’s use of the term. USAID/Iraq does not believe it is possible at this stage in the program to apply a methodology that is based on a credible and rigorously defined counterfactual to control for factors other than CAP that might account for any observed changes reported by the evaluation team. Distinction between a performance and an impact evaluation is based on Management Systems International’s “Evaluation Statements of Work: Good Practice Examples.” Washington, DC: U.S. Agency for International Development, 2011. http://www.usaid.gov/policy/evalweb/documents/EvaluationSOW-GoodPracticeExamples.pdf USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 51 question areas are broken into categories to guide the structure of the evaluation report. The report must provide evidenced-based answers to each of the questions below. The questions below respond to the first section of the evaluation, highlighting the degree to which CAP III achieved the highest-level outcome in a chain of results, based on the program’s results framework. The evaluation must address sustainability under each question, i.e. whether program results are likely to endure. The evaluators should specifically note particular issue areas and/or stakeholders that present opportunities or partnerships for future USAID projects. Each question must also be answered with specific regard to gender and vulnerable populations in order to explicate the degree to which CAP III was able to reach these marginalized groups in Iraqi society. I. Did the CAP Program contribute to increased participation in collective community actions? II. Are CAP communities better able to articulate their needs and mobilize resources within and outside the community to solve common problems? If not, why not? III. Does local executive and representative Government29 in CAP communities better meet the articulated needs of the community? If not, why not? IV. Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the Fund’s defined purpose? If not, why not? V. Provide lessons learned that could be applied to future USAID work with Iraqi communities and local governments specifically on approaches to: a. Community engagement/participation b. Training methodology c. Securing government buy-in/participation The evaluation is expected to answer the above research questions with regard to the sustainability of program accomplishments, detailing any contributing factors for sustainability or lack thereof. The evaluation must also consider, for each question, the effect of project activities on women and vulnerable populations such as internally displaced persons (IDPs), youth, female-headed households, and ethnic/religious minorities. The questions above are a mandatory part of the evaluation report. During the research process, however, the team may elaborate on other observations or highlight unexpected findings of note. Some illustrative questions for inclusion in evaluation instruments are included in Annex A-1. These are merely illustrative and not exhaustive. C. EVALUATION DESIGN AND METHODOLOGY 1. Evaluation Design Methodology: This is a performance evaluation that will be led and facilitated by an independent consultant team. The evaluation will investigate and document the results of CAP III from the project’s start date on October 1, 2008, through April 30, 2012. 29Provincial-level ministry representatives, Provincial Councils, Local Councils, and Parliament (if applicable). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 52 The Mission is looking for new, creative suggestions regarding designing this evaluation. Based on the pre-competed mechanism to meet USAID/Iraq’s monitoring and evaluation needs, the implementer is expected to propose a more detailed evaluation design in its implementation plan, send that plan and design to USAID/Iraq prior to the start of travel to Iraq, and present the evaluation design methodology to the USAID/DGO and Program Office representatives upon arrival in Baghdad. The Team should come up with limitations to the methods identified. The evaluation team shall outline and further break up the evaluation questions to explain for each identified question and sub-questions, including explanations of measures or indicators, targets, baseline data (if any), data sources, sample sizes, data collection instruments and data analysis. Due to data limitations this performance evaluation will only incorporate before-after comparisons, and will not be able to apply a rigorous counterfactual approach. 2. Data Collection Methods The Team should employ both quantitative and qualitative methods to analyze and answer the evaluation questions. The following illustrative data collection and analysis methods may be used to address the evaluation questions as appropriate: Key informant interviews with selected local leaders, government officials and CAG members. Focus group discussions with CAGs, with districts and sub districts elected officials. Mini survey using structured questionnaire (communities involved with CAP activities). Possible survey to collect community participation information for question I. Scorecard for Community Action Groups, adapted from versions used by CAP implementers. Joint workshop of CAG and local government leaders. This will be a facilitated workshop to reflect on past experience, successes and lessons learned; suggest secure location for ease of expatriate participation, such as Erbil or Kirkuk. Site visits to selected community projects sites. Interviews with community mobilizers to gain insight into CAP processes, challenges, and relationships with both CAGs and local government leaders. Other tools to be determined through discussions with USAID and the implementer. 3. Data Analysis Methods Prior to the start of field work, the evaluation team will develop and present, for USAID review and approval, their evaluation design which will include a data analysis plan that details how focus groups and other interviews will be transcribed and analyzed; what procedures will be used to analyze qualitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with available quantitative data to respond to evaluation questions. Should the evaluation team suggest collecting further quantitative data, this must be presented to USAID for review and approval as well. It is likely that some CAP III projects may have affected the different social strata differently. Thus, the Team shall collect and analyze sex and other social variable disaggregated USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 53 data to see how projects are affecting the different social strata including women, religious groups, IDPs and other variables. D. EVALUATION PRODUCTS 1. Deliverables The implementer shall provide the following deliverables to USAID/Iraq: 1. Draft Work Plan and Evaluation Design: Upon completing the initial desk review in Washington, DC, the team will develop and submit for approval to USAID a work plan for conducting the CAP III final evaluation. They will submit this plan to USAID via email before ending the desk review period and beginning travel to Iraq. The work plan shall contain the evaluation design matrix as discussed in Subsection C.3 above, as well as an implementation timeline. The Program Office will organize evaluation in-brief to discuss on the work plan and implementation of the evaluation. 2. Oral Briefings (three): The implementer will provide three briefings for USAID, including: (a) an in-brief – within two (2 days) days of arrival in country. During the in-brief , the team will present its work plan for the field work, data analysis, and report writing submitted earlier and discuss with the USAID/DGO and Program Office teams. The team will have two days to submit its final work plan incorporating comments from USAID before the field work started is due (b) a mid-brief – within thirteen (13) working days, the team will present its preliminary findings and conclusions to ensure that sufficient evidence is being gathered to answer the evaluation questions; and (c) an exit briefing – one (1) day prior to departure, the team will present an outline (PowerPoint format recommended) of the evaluation report with general findings, conclusions, and anticipated recommendations. 3. Final Work Plan and Evaluation Design: The team will submit a final work plan, integrating USAID comments received at the in-brief, before commencement of field work. 4. Outline of Evaluation Report and Anticipated Findings, Conclusions, and Recommendations: The team will submit and present a report outline at the exit briefing mentioned above. A draft of this document should be provided for USAID comment two (2) working days previous to the exit briefing. 5. Draft Report: The team shall submit a draft report within six working days upon arriving Washington, DC. This document should explicitly respond to the requirements of the SOW, be logically structured, and adhere to the standards of the USAID Evaluation Policy and checklist for assessing evaluation reports. The Mission Evaluation POC at Program Office will ensure that the draft report meets standards as stated the Agency’s evaluation policy. 6. Final Report: The evaluation team shall incorporate USAID’s comments and submit the final report to QED in electronic format (Microsoft Word and Adobe PDF), within six (6) working days following receipt of comments on the draft report. QED will copy edit the final version and submit it electronically to USAID/Iraq within four working days of receipt of the final draft form the consultant team. Once the final version is approved by Democracy and Governance Office, the Evaluation POC at Program Office should upload the report to the Agency’s Development Experience Clearinghouse (DEC.) This version must be clean of any identifiable private information that may expose individuals and organizations to security risk and should be made compliant with Section 508. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 54 7. Raw Data and Evaluation Instruments: Once the evaluation is completed and the evaluation is uploaded onto the Agency’s Development Evaluation Clearinghouse, the contractor shall provide all raw data and data collection instruments in electronic copy to Mission’s Evaluation Point of Contact at the Program Office. 2. Reporting Guidelines The evaluation report should adhere to the Agency’s evaluation policy which explains criteria for ensuring quality of evaluation reports (see Annex A-2). The format for the evaluation report is as follows: 1. Executive Summary: Concisely state the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and any lessons learned; should be able to sufficiently detailed, yet brief, to serve a stand-alone product (3-5 pg); 2. Table of Contents: List section headings and page numbers, as well as any figures or tables (1 pg); 3. Introduction: State the purpose, audience, and outline of the evaluation (1 pg); 4. Background: Provide a brief overview of project, USAID project strategy and activities implemented in response to the problem and purpose of the evaluation (2–3 pg); 5. Methodology: Provide an overview of evaluation methods, including constraints and gaps; greater detail should be included in the appendices (1 pg); 6. Findings/Conclusions/Recommendations: Explicitly answer each question in the required evaluation categories: impact, sustainability and gender/vulnerable populations; the report should distinguish between findings (the facts), conclusions (interpretation of the facts), and recommendations (judgments related to possible future programming) (17–20 pg); 7. Issues: Provide a list of key technical and/or administrative issues, if any; may include Disclosure of Conflict of Interest: statement to the effect that, as an external evaluations, all evaluation team members provided a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated, and Statement of Differences: When applicable, evaluation reports should include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team. (1–2 pg); 8. Lessons Learned (2–3 pg); 9. References – including bibliographical documentation, meetings, interviews and focus group discussions; 10. Annexes: Annexes should include this statement of work, a glossary of terms, and a clear documentation of evaluation methods, schedules, interview lists and tables, and any focus group scripts or questionnaires used; the presentation should be succinct, pertinent and readable. The evaluation report expresses an independent view of the Evaluation Team. The contractor should solicit and attach any differences in opinion there may be on the findings, conclusion or the recommendations to the report. The report format should be presented in Microsoft Word and use 12-point type font throughout the body of the report, using page margins 1” top/bottom and left/right. The body of the report should ideally be within 20, but up to 40 pages maximum will be allowed given the need to address performance of four distinct implementing partners. This page limit does not USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 55 include the executive summary, table of contents, references, and annexes. Annexes can be used to provide evidence or graphic displays of information summarized in the body of the report. E. TEAM COMPOSITION The team leader and an expatriate consultant will lead a team of 8 locally-hired researchers. The local research team will be responsible for conducting field interviews, translating gathered data into English and producing summary reports using standard reporting format. Local team members will be trained by the expatriate consultant on how to conduct interviews and summarize results using standard format. All attempts should be made for the team to be comprised of an equal number of male and female members, especially as field staff may be called upon to interview groups of both genders. The expatriate consultants will be responsible for data analysis, presentation of preliminary findings, and compiling the final evaluation report. The entire team should become familiar with and apply the principles outlined in USAID’s January 2011 Evaluation Policy. The team shall include: Team Leader (International) – This person should have a professional background in international development work and skills in leading and implementing development program impact evaluations. He/she should have a minimum of ten (10) years of related experience in community driven development in post conflict societies. Experience in good governance and civil society strengthening is desirable. Knowledge of USAID democratic governance programming is required. Ability to conduct interviews and discussions, experience in training of local researchers on the use of evaluation tools and to write in English is required. Knowledge of the host country language(s) is a plus, but not required, as long as another team member is fluent (written and spoken). General knowledge of the USAID monitoring and evaluation process is strongly preferred. Previous overseas experience and background knowledge in Iraq or the Middle East region is preferable. Team Member (International) – This person should have a minimum of five (5) years of related experience in Middle East, at least some of which would preferably include Iraq. This member should have substantial demonstrated experience in designing, implementing, assessing and evaluating donor projects in democracy and governance, with a strong understanding of data collection and analysis methodologies as well as experience in training of local researchers on the use of evaluation tools is required. Experience in one or more of the following would be desirable: community development, local government, or civil society development. Regional experience and/or country knowledge is required. Ability to write in English is required. Knowledge of host country language(s) is a plus, but not required, as long as another team member is fluent (written and spoken). General knowledge of USAID programming and procedures is preferable. Local Researcher (up to eight persons) – These researchers should have a minimum of three (3) years of relevant experience in community driven development, local government, civil society, political processes, possessing strong background knowledge of Iraq and experience in the implementation, evaluation and/or monitoring of foreign assistance programs. Previous experience with USAID assistance projects is highly desirable. Knowledge of Arabic and English is required. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 56 The contractor will get a written non-disclosure and conflict of interest agreement from each of the Evaluation Team members to ensure that they have no conflicts of interest in engaging this evaluation and will not disclose procurement-sensitive information. The final evaluation needs to be carried out in a participatory fashion, forming a team that, in various places and times, includes a range of managers, implementers, community leaders, partner agency staff, and stakeholders. Such coordination can take place in Washington and in Iraq. F. EVALUATION MANAGEMENT 1. Logistics Restrictions on Dissemination. The team’s final report belongs to USAID and may not be shared with any organizations or individuals outside of USAID. The final approved report of a public version will be uploaded on the Agency’s Development Evaluation Clearinghouse. The report shall follow USAID branding and marking requirements. Period of Performance. The work called for this scope will start in late April and will be completed approximately 12 weeks later. The team is projected to arrive in country May 6. Commencement of the field work is targeted for May 13, 2012. The first draft report shall be submitted to USAID no later than June 29, 2012. The final draft report must be satisfactorily completed within ten (10) working days of receiving USAID’s comments on the draft report. Logistical Support. Logistic support to be provided by the implementer includes: international travel, transportation, secretarial and office support, interpretation, report printing and communication, as appropriate. All logistical support will be provided by the implementer including travel, transportation, secretarial and office support, interpretations, report printing and communication, CAC card or USF-I badges, life and security support. Staff from CAP implementers, specifically the monitoring and evaluation staff and community mobilizers, may be contacted to assist in setting up of interviews, organizing logistics for workshops and coordinating the work of the field research team. Work Week. A 6-day work week is authorized in Iraq with no premium pay. Friday and Saturday are weekend; the team may choose one of these two days as an “off” day. Technical Direction. Technical direction during the performance of this evaluation will be provided by USAID’s Office of Democracy & Governance. The head of the DG Office and other members of the DGO Team will provide the evaluation team with relevant USAID documentation to review and a list of suggested contacts to interview. The evaluation team is also expected to propose review of additional documents for desk research as needed, and to recommend relevant Iraqi experts for interviews to supplement the key contacts list, if necessary. The team shall conduct interviews with appropriate USAID staff in Washington and with appropriate persons at CHF, IRD, ACDI/VOCA and Mercy Corp office in Washington. Approximately ten (10) workdays of US-based preparation before field work commences. Background information on the program will be provided for desk review, which may be performed wherever the consultants reside. No more than five (5) working days will be required in Washington, D.C., to meet with USAID/Middle East Bureau and implementers’ staff. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 57 2. Scheduling As stated under the period of performance, the work called for in this statement of work will start in late April and be completed approximately 12 weeks later. The expected start date for field work (data collection) is May 13, 2012. All work necessary for a fully realized first draft must be completed no later than June 29, 2012. The timeline for the evaluation is as follows. Weeks 1-3: Desk Review and Preparation Phase (US and Iraq): The first phase of the activity will involve a desk review of relevant materials and key documents. A pre-trip meeting with QED, USAID/Washington D.C. staff and CHF, IRD, ACDI/VOCA and Mercy Corp headquarters staff should be conducted by the expatriate team members during the preparation phase. On the basis of this information, the team will develop an evaluation/design methodology that includes research questions and interview protocols which it will share via email with the USAID/DGO and Program Office representatives. The team should also prepare a schedule of interviews for the subsequent field work stage. Fifteen (15) working days per expatriate team member are authorized for the preparation phase including desk review, DC meetings, and in-country preparation. At least one Iraqi local team should be working at this stage to become familiar with the documents and be setting up the preliminary schedule and team members. Training of the Iraqi teams in the background, purpose and organization of CAP III will commence in the third week. Upon arrival in Iraq, the Team Leader and expatriate team members will present their evaluation design and methodology to USAID/Iraq. In addition, the expatriate members of the team will meet with the Iraqi members of the team and will integrate them into the process, briefing on what they learned from the desk review and U.S. meetings and conduct training of local researchers on the use of the evaluation tools. This training and team building phase is critical to the success of the field work, as security and logistics will demand that Iraqi staff members are empowered to conduct many of the interviews. The team will submit a preliminary work plan upon completing the desk review and defend the proposed implementation plan second day of the Team’s arrival in-country. Week 4-8 Field Work (Data Collection) Phase: The team will spend 39working days in Iraq (45 days in country, including arrival/departure days and rest days) conducting field research, including gathering of additional documents. It will also involve the conduct of structured interviews with key informants (and focus groups, if appropriate) and project beneficiaries, including indirect beneficiaries (such as community members, civil society and Local Government and Provincial Government representatives). Week 8-9: Data consolidation and analysis: The team shall have time for consolidating field data analysis and filling missing information. Week 10. Submission of the Draft Report: The implementer will submit a draft evaluation report that responds explicitly to the required evaluation questions and adheres to the standards outlined in the January 2011 USAID Evaluation Policy and associated checklist for assessing evaluation reports. Week 11: Period for feedback and revisions including USAID Washington review. Week 13: Submission of the Evaluation Final Report, fully integrating requested edits (within 10 working days of receipt of USAID comments) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 58 3. LoE Estimates and Deliverables Due Dates (Note that dates may be changed, with the permission of USAID, in the implementation work plan once the team arrives in Iraq.) Description Date Expat Days Total Local Days Total Desk review of relevant documents Apr. 23- 27 2 5 10 0 0 0 Work plan and consultation (DC and Baghdad) Apr. 30, May 1-4 2 5 10 2 5 10 Travel to Baghdad May 5-6 2 2 4 0 0 0 In-brief, work plan refinement and planning of training May 7-9 2 3 6 2 3 6 Team planning and training of data collectors May 10- 12 2 2 4 8 2 16 Field work- I May 13- 27 2 13 26 8 12 96 Mid-term briefing May 28 2 1 2 8 1 8 Field work- II May 29- June 5 2 7 14 8 7 56 Field work- III/ data analysis (to be determined in team work plan) June 6-11 2 5 10 8 5 40 Post field work analysis and consolidations June 12- 17 2 5 10 4 5 20 Exit briefing June 18 2 1 2 0 0 0 Travel to DC June 19- 20 2 2 4 0 0 0 Draft Report June 21- 29 2 6 12 0 0 0 USAID Feedback July 8 0 0 0 0 0 0 Final Report Preparation July 9-13 2 6 12 0 0 0 Final report copyediting and formatting (QED HQ) July 16- 19 0 0 0 0 0 0 TOTAL 63 126 40 252 Deliverable / Approval Due Date Submit Draft Implementation Plan from DC May 4 In-briefing May 8 Submit Final Implementation Plan May 12 Mid-Term Briefing May 28 Draft Outline of Report June 14 Final Briefing June 18 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 59 Deliverable / Approval Due Date Draft Report June 29 USAID Response Due July 8 Final Report Due July 20 4. Budget A draft budget has been reviewed and will be sent back for comment. The total preliminary estimated budget is $417,215.56 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 60 Item Daily Rate/Unit Cost Units Estimated Total 1. Labor Costs 126 $ 125,748.00 STTA $ 125,748.00 Team Leader (Jennifer Kuiper) - Senior Associate - Evaluations $ 998 63 $ 62,874.00 Team Member (TBD) - Senior Associate - Evaluations $ 998 63 $ 62,874.00 2. DIRECT COSTS $ 142,502.24 Other Direct Costs $ 142,502.24 R/T Airfare (Home of Record - DC) $ 1,450 2 $ 2,900.00 Lodging - DC $ 224 15 $ 3,360.00 M&IE - DC $ 77 15 $ 1,155.00 Ground Transportation - DC $ 350 2 $ 700.00 R/T Airfare (Home of Record - Baghdad) $ 3,700 2 $ 7,400.00 Ground Transportation (Home of Record - Baghdad) $ 150 2 $ 300.00 Lodging - Dubai $ 362 4 $ 1,448.00 M&IE - Dubai @ 75% $ 107 4 $ 429.00 M&IE - Dubai $ 143 4 $ 572.00 Overflow Lodging - Baghdad $ 150 72 $ 10,800.00 M&IE (Baghdad) $ 20 90 $ 1,800.00 Local Consultants $ 160 252 $ 40,320.00 Local staff air fare/travel $ 258 16 $ 4,120.00 Local Consultant Lodging (Erbil, Dohuk & Sulaimaniyah Provinces) $ 107 160 $ 17,139.20 Local Consultant M&IE (Erbil, Dohuk & Sulaimaniyah Provinces) $ 92 160 $ 14,739.71 DBA (2%) 2% $ 76,292 $ 1,525.85 MEDEX $ 360 4 $ 1,440.00 Medical exams and inoculations $ 333 2 $ 666.29 Visas $ 200.00 2 $ 400.00 Danger Pay 35% $ 31,789 $ 11,125.98 Post Differential 35% $ 31,789 $ 11,125.98 M&E Trainings $ 1,071.00 $ 1 $ 1,071.00 Catering/Refreshments (Food) $ 1,071.00 $ 4 $ 4,284.00 Translation / Document and Map Production $ 237 8 $ 1,894.80 Translation - Interpretation service $ 161 8 $ 1,285.44 Publication/Communication Product Production $ 500 1 $ 500.00 3. Subcontracts $ 112,500.00 RONCO Secure Transport $ 112,500.00 Dedicated Team for portion of fieldwork $ 4,500.00 25 $ 112,500.00 4. Indirect Costs $ 36,465.32 G&A (14.3%) $ 36,465.32 TOTAL $ 417,215.56 The QED Group, LLC Iraq PERFORM Order No. 267-M-00-09-00513 CAP III Final Evaluation Activity Cost Estimate USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 61 ANNEX A-1: ILLUSTRATIVE EVALUATION QUESTIONS AND EVALUATION CRITERIA TO ENSURE REPORT QUALITY Illustrative Evaluation Questions The questions below are illustrative supporting questions related to the mandatory evaluation questions listed in Subsection B.3 of Annex A. These may be adapted and used in the informant interviews, surveys, or focus groups. These are only illustrative and not exhaustive. In examining Objective 3 under the impact section, please examine the success of the program in identifying war victims. Is there a sense that most victims of Coalition military operations have been reached? Is there evidence to support any changes to the definition of the criteria used to identify the beneficiaries? To what extent has CAP III succeeded in building productive relationships between Community Action Groups and elected local government leaders? And how effective are these relationships in terms of building trust between CAGs and local leaders, institutionalizing accountability, and strengthening capacity of both CAGs and local leaders in order to respond effectively to community priority needs? Given current Iraqi context (political, social, economic, etc.), what are the factors that promote strong partnerships and factors that may work against it? In which regions CAP program has been successful in building stronger ties with district and sub-district councils and why? Community projects are intended to serve as a tool to promote good governance by providing common platforms for local leaders (at districts and sub-districts) and community representatives (CAGs) to work together in assessing and prioritizing community needs, designing appropriate interventions, mobilizing resources from within and outside community to meet those needs, soliciting regular community inputs, and giving feedback on progress to the larger community. How effective are those projects in achieving their intended goal? To the extent possible, describe/measure increase in the ability of communities to improve service delivery in their neighborhoods. How effective was the CAP Program in assisting victims of war and communities affected by the war? What was the impact of local NGOs in managing Marla program, and was their capacity significantly improved? How effective were the trainings that have been provided to the CAGs? How does the implementer design the trainings for the community? How effectively did the training design anticipate cultural specificities and particular needs of communities? What are the regional differences in program implementation, and in which regions was CAP program the most effective? How effective and sustainable are the current mechanisms put in place by CAP implementers to promote participatory decision making process at the local level? How did the implementing mechanism and structure of four distinct partners impact program results? How does CAP implementation vary by each partner? How is CAP perceived and valued? Have communities been empowered? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 62 Are local governments more responsive? What elements of the program could be relevant to future projects with citizens and local governments? Are there lessons to be learned from CAP implementation challenges that can be avoided in future projects? Identify issues areas (specifically topics that resonate regardless of region or identity group) or stakeholders (such as strong community action group leaders or local government partners) that could be relevant for future USAID programming with citizens and government? Questions related to Relevance, Effectiveness, and Efficiency: To what extent are the objectives of the program still valid? Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? Are the activities and outputs of the program consistent with the intended impacts and effects? To what extent were the objectives achieved /are likely to be achieved? What were the major factors influencing the achievement or non-achievement of the objectives? Were activities cost-efficient? Were objectives achieved on time? Was the program or project implemented in the most efficient way compared to alternatives? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 63 ANNEX A-2. USAID EVALUATION POLICY – CRITIERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical officer. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 64 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 65 ANNEX B. GLOSSARY OF TERMS Advocacy The aggregation of citizen interests and representation of those interests to government or other decision-making bodies by citizens or on behalf of citizens Assessment A synonym for evaluation. Assumptions A proposition that is taken for granted, as if it were true. For project management, assumptions are hypotheses about causal linkages or factors that could affect the progress or success of an intervention. Conclusion A judgment based on a synthesis of empirical findings and factual statements. Contact Hypothesis A development hypothesis that if key actors from belligerent groups are given the opportunity to interact, they will better understand and appreciate one another, be better able to work with one another, and prefer to resolve conflicts peacefully. Evaluation A systematic and objective assessment of an on-going or completed project, program, or policy. Evaluations are undertaken to (a) improve the performance of existing interventions or policies; (b) assess their effects and impacts; and (c) inform decisions about future programming. Evaluations are formal analytical endeavors involving systematic collection and analysis of qualitative and quantitative information. Findings Factual statements about a project or program based on empirical evidence. Findings include statements and visual representations of the data, but not interpretations, judgments, or conclusions about what the findings may mean or imply. Focus group A group of people convened for the purpose of obtaining perceptions or opinions, suggesting ideas, or recommending actions. A focus group is a method of collecting information for the evaluation process that relies on the particular dynamic of group settings. Impact Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended, or unintended – inter alia, impacts may be economic, institutional, technological, environmental, sociocultural, or gender-related; measurement of extent of impacts (if possible, a cost-benefit analysis should be undertaken). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 66 Lessons learned Generalizations based on evaluation experiences with activities, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons learned highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact. Performance Management Plan A tool used by USAID Missions, Offices, and assistance objective teams to plan and manage the process of assessing and reporting progress toward achieving an assistance objective. Objective A statement of the condition or state one expects to achieve. Project A discrete activity (or “development intervention”) implemented by a defined set of implementers and designed to achieve specific objectives within specified resources and implementation schedules. A set of projects make up the portfolio of a program. Sustainability The continuation of benefits from a development intervention after major development assistance has been completed: sustainability of benefits (technological, social, environmental, gender); sustainability of institutional capacity; and maintenance of future recurrent budget (financial sustainability). Theory of Change An outgrowth of repeated successful proofs of a hypothesis; the process by which an intervention or a series of interventions changes a situation from one condition to another. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 67 ANNEX C. EVALUATION METHODS USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation Evaluation Work Plan (as o15 May 2012) PROGRAM BACKGROUND USAID’s Community Action Program (CAP) is its longest running development program in Iraq. Begun in May 2003, it was designed to support Iraq’s communities to “exercise true grassroots democracy by implementing projects on their own where necessary, and in partnership with local government where possible, to meet community needs.”30 Beginning in October 2008, CAP’s third phase (CAP III) has focused on building skills and cooperation between constituencies and their local representative and executive governments. CAP III is implemented in each governorate in Iraq by four USAID implementing partners (IPs), which are responsible for a designated area of responsibility (AoR), together covering 15 of Iraq’s 18 governorates: Mercy Corps: Basrah, Dhi Qar, Maysan, Muthanna International Relief and Development (IRD): Baghdad CHF International (CHF): Anbar, Babil, Karbala, Najaf, Qadisiyyah, Wasit ACDI/VOCA: Diyala, Kirkuk, Ninawa, Salah ad Din Although each partner had some variation in program design, all provided some form of technical assistance, training, and targeted funding for community-initiated projects. CAP projects vary by type but typically provide infrastructure support in areas of education, health, transportation, electricity, sanitation, water, youth, and recreation. The IPs worked with both community leaders (in the form of community action groups, or CAGs) and local government councils (Qadaa, Nahiyaa, and neighborhoods). In September 2010, the IPs began to track the program’s impact on internally displaced persons (IDPs) in accordance with modifications to all four cooperative agreements. In fall 2011, USAID sent a letter to IPs requesting that they also collect data to track their impact on other vulnerable populations, including female-headed households, religious/ethnic minorities, and youth. Additional performance data was developed for each partner with common indicators articulated in the individual performance management plans (PMPs) and reported to USAID in quarterly reports. Additionally, the four CAP III partners are responsible for administering the Marla Ruzicka Iraqi War Victims Fund (Marla Fund). This congressionally authorized fund provides individual and community support for victims of U.S. Coalition forces and victims’ families. The implementing partners or their sub-contracted local organizational partners identify war victims and develop 30 Statement of Work for USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation (April 11, 2012). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 68 appropriately tailored support projects, including medical treatment, livelihood opportunities, rehabilitation, and home repair. EVALUATION PURPOSE The evaluation team will conduct an end-of-project performance evaluation.31 As explained in the statement of work, “USAID/Iraq does not believe it is possible at this stage in the program to apply a methodology [for an impact evaluation] that is based on a credible and rigorously defined counterfactual to control for factors other than CAP that might account for any observed changes reported by the evaluation team.”32 Further, the team is not serving as auditors or monitors as such assessments are outside the scope of this evaluation. As presented in the statement of work, this end-of-project performance evaluation has several purposes: (1) Accountability: To assess to what extent the project’s objectives and goals have been achieved,33 according to established objectives in USAID’s Results Framework. Specifically: CAP III Objective 1: Communities better articulate their needs and mobilize resources within and outside the community to solve common problems. i. Sub-IR 9.2.1. Communities better able to mobilize resources within the community to meet their articulated needs. ii. Sub-IR 9.3.1. Communities better articulate their needs. CAP III Objective 2: Local executive and representative Government in CAP communities better meet the articulated needs of the community. i. Sub-IR 9.3.2. Qadaa and Nahiyaa better articulate needs of their communities. CAP III Objective 3: Civilian victims of conflict assisted iii. Sub-IR 9.3.3: Civilian victims of conflict assisted by the Marla Ruzicka Iraqi War Victims Fund (MRIWV or Marla Fund). (2) Learning: To inform the implementation of future projects, especially those working at the community level to link Iraqi citizens with their government for improved development results. In particular, evaluation results should be useful to USAID’s 31 In its January 2011 evaluation policy, USAID defines performance evaluations as those that “focus on descriptive and normative questions: what a particular project or program has achieved; how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making.” “USAID Evaluation Policy: Learning from Experience.” USAID, Washington, DC (January 2011). 32 Impact evaluations, as defined by USAID, are “based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured.” “USAID Evaluation Policy: Learning from Experience.” USAID, Washington, DC (January 2011). 33 QED will explore the possibility of developing an appropriate scale to characterize the extent to which objectives are met. Accompanying this scale will be objective criteria to be used for classification. In the course of the research, the team may find that different aspects of each objective have achieved greater or lesser levels of achievement or there may be geographic or circumstantial variation. However, if the findings lend themselves to this categorization, the team will provide these conclusions in the evaluation report. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 69 Governance Strengthening Project (GSP) and Broadening Participation through Civil Society (BPCS). The principal audience for the evaluation is USAID’s Iraq Mission (USAID/Iraq), specifically the Democracy and Governance Office. Additionally, the evaluation should inform the U.S. Congress and American public regarding the contribution CAP has made in support of Iraq’s transition toward democracy. EVALUATION QUESTIONS There are several key questions to which this evaluation will respond, as outlined in the statement of work (dated April 11, 2012). Each of these questions also indicates sub-questions that will be addressed through the evaluation process (see Annex A for additional research questions to guide all evaluation activities): I. Did CAP III contribute to increased participation in collective community actions? II. Are CAP communities better able to articulate their needs and mobilize resources within and outside the community to solve common problems? If not, why not? III. Does local executive and representative government (defined as governorate level ministry representatives, governorate councils, Local Councils, and the Council of Representatives) in CAP communities better meet the articulated needs of the community? If not, why not? IV. Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the fund’s defined purpose? If not, why not? V. Provide lessons learned that could be applied to future USAID work with Iraqi communities and local governments specifically on approaches for: (1) community engagement/participation, (2) training methodology, and (3) securing government buy-in/ participation. METHODOLOGY Evaluation Team The evaluation team is comprised of a contract team of two U.S. consultants, nine Iraqi field team researchers (seven under the CAP III contract and two from the PERFORM contract), one Iraqi data management expert (under the PERFORM contract, also provided field work assistance), and one database design expert (under the PERFORM contract).34 The QED Group LLC (QED) manages the team and provides operational and logistical support from out of its Baghdad compound. QED’s Project Manager, USAID/PERFORM, is based in Baghdad and oversees all project activities. Evaluation Design, Validity, and Generalizability The performance evaluation applies a mixed methods design, which uses both quantitative and qualitative data collection and analysis. The design follows a mixed method model that is concurrent (both quantitative and qualitative data are collected at the same time, as opposed to 34 The CAP II program benefits from the participation of three PERFORM staff, providing cost savings on the CAP III budget, which provides for up to eight field researchers. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 70 sequential) with equal status (both quantitative and qualitative data will factor into the analysis with equal priority). In the case of the CAP III evaluation, the team’s rationale for using a mixed methods approach is as follows:35 (1) Triangulation: Provides corroboration of responses to research questions from multiple perspectives (e.g., different respondents) as well as via different modes of input (e.g., open-ended and closed ended information, observation). (2) Complementarity: Provides elaboration and clarification of results between the two methods (e.g., qualitative data describes implementation variation that can help to explain quantitative results data). (3) Expansion: Provides greater number of respondents through qualitative mini-survey with close-ended questions to complement fewer but more detailed information from respondents included in qualitative discussions (e.g., focus groups, individual interviews). Throughout the evaluation process, the evaluation team monitors potential compromises to validity of data and whenever possible mitigates effects. One such possible challenge to validity is potential evaluation team conflicts of interest. When there were clear conflicts of interest, the conflicts were managed in the hiring, scheduling, and training process so that each member of the team confirmed any possible areas in which they have an interest in the outcome of the evaluation. Another possible challenge is reflexivity of our evaluation team, which arises from potential bias and/or pre-disposition of a researcher toward the CAP III activities, stakeholders, or results. This was managed during our team training and ongoing meeting process as well as during the analysis phase. Reflexivity bias is addressed through raising each researcher’s self-awareness and critical self-reflection on potential biases and predispositions toward a particular conclusion. Team meetings include a process to review team assumptions going into field visits, which may inappropriately bias findings. The evaluation tools also prompt for information in which findings will be based on evidence and not exclusively on researcher interpretation. Additionally, the analysis phase will include a group discussion with all team members in order to detect differences in findings based on individual interpretation versus verifiable information. The analysis phase will also emphasize low-inference reporting that minimizes individual interpretation, such as direct quotations from respondents and quantitative survey results. Triangulation of data sources will also help to mitigate this effect. The team will also track validity of the evaluation findings and conclusions to the extent to which the evaluation results can be assumed to apply across the general population (e.g., men/women, young/old, ethno-sectarian groups), settings (e.g., urban/rural, resource rich/poor), and time (e.g., changes in contextual factors may change project outcomes/appropriateness). In the methodology section of the research report, the team will note the confounding variables that are likely to impact the independent variables and the generalizability of the conclusions based on evaluation findings. Some of these confounding variables might be: Variation in IP program design across IPs as well as over time within the same IP (i.e., treatment/intervention variation validity). 35 Adapted from Jennifer Greene, Valerie Caracelli, and Wendy Graham. “Toward a Conceptual Framework for Mixed Method Evaluation Design.” Educational Evaluation and Policy Analysis. Fall 1989. Vol. 11. No. 3. Cornell University (1989). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 71 Shifts in USAID priorities such as emphasis on internally displaced persons and other vulnerable groups. Contextual factors such as levels of violence impacting accessibility of communities and project sites or increasing the number of victims or IDPs in a community. Some aspect of our field team member (e.g., ethno-sectarian identity, sex, age, place of residence, prior experience with project) may influence behavior or answers of respondents (i.e., experimental effects). Preparation for Field Work The evaluation includes approximately six weeks of research in Iraq, including field visits to implementing partners and CAP III projects. Prior to these visits, the team will undergo the following preparation activities (see section below on the timeline of evaluation activities: (1) USAID CAP Document Review (April 23 start, ongoing): The evaluation will focus on documentation of CAP III (project period October 2008 to September 2012) but also take into account the implementation history beginning with the initial project implementation of ICAP/CAP I (project period May 2003 to March 2007) and its continuation through CAP II (project period October 2006 to December 2008). Documents to be reviewed are: implementing partner plans’ reports to USAID, including statements of work (with modifications); cooperative agreements and mini￾RFAs; and quarterly reports and work plans based on performance management plans (PMPs), which include PMP indicators. The evaluation team will also review publicly available evaluations of CAP programs and evaluations of programs that were implemented in other countries, such as Lebanon, Serbia, and Montenegro. (2) Implementing Partner Interviews, Washington, DC (April 30-May 4): The U.S.-based researchers conducted individual interviews with each implementing partner’s program officer responsible for its CAP III activities. The purpose of these interviews was to understand design and implementation variation across partners and identify possible challenges to implementation and perceived CAP III accomplishments. (3) USAID Democracy and Governance (DG) Officers (Washington, DC) (April 27, May 4): The U.S.-based researchers conducted individual interviews with key USAID-DG staff familiar with CAP and the Marla Fund. The purpose of these interviews was to understand the intended variation in program design between CAP II and CAP III (as reflected in the initial RFA), refine research questions to areas of DG priority, and clarify USAID modifications in CAP III priorities over the project period. (4) Development of data collection tools (May 7-12): Based on IP and USAID interviews in DC and the document review process, the evaluation team developed data collection tools (described below) prior to field visits. Iraqi team members contributed to tool development in order to ensure culturally appropriate processes for data gathering, locally understandable verbiage in question formation, and verification of appropriateness of questions (i.e., the correct questions asked to the appropriate respondent under proper conditions). The team translated all tools into Arabic.36 36 Because of limited time available for field data collection, the evaluation team did not pre-test the data collection tools. Instead, an initial version of the tools was used in the first week of data collection. Based on feedback from field team researchers, the evaluation team leader modified the tools, which were used in all subsequent data USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 72 (5) Translation of surveys: Iraqi CAPIII beneficiaries and Marla Fund beneficiaries will complete paper surveys and IP Community Mobilizers will complete online surveys. In order to include Iraqis that do not read and write in English (anticipated to be a sizeable proportion of our survey respondents, if not the majority), all survey questionnaires have been translated into Arabic. Translators on the Iraqi evaluation team translated from English to Arabic; another bilingual colleague translated the Arabic back into English to ensure that the intended meaning of each question was not lost in translation. (6) Scheduling of field visits with field team: On April 23, USAID reminded the implementing partners of the upcoming evaluation. The evaluation team lead contacted the implementing partners’ chief of parties (COPs) on May 4 to outline field visit activities and to advise them of the scheduling process. The Iraqi scheduling team (3 persons) met May 5 in order to: Agree on field-team composition (2 to 3 field visitors/team)37 Assign teams to each of the implementing partners to cover the corresponding area of responsibility, which includes 15 governorates38 Identify cities, towns, and neighborhoods (based on an urban/rural balance) to be visited as part of project site visits (3 to 5 projects/governorate) Schedule tentative dates for visits (final dates to be coordinated with implementing partners) (7) Scheduling of field team visits with implementing partners: Based on the tentative schedule developed during the field team meeting, the team lead followed up on the May 4 email to discuss dates of visits with the implementing partners and COPs (and in the case of CHF International in Anbar, the Deputy COP). The evaluation team anticipates scheduling to be an iterative process, as the COP may need to confirm availability of key staff on the proposed dates. (8) Training field team: The U.S. evaluation team members developed a one-day training of all field team visitors, held on May 13. The training included: Team building, as many team members have not worked previously together collection activities. Original versions were cross-walked to match the final versions and information determined to lack comparability was not used in the final analysis. 37 QED’s Deputy Chief of Party in Baghdad initially vetted all team members. Bios were developed and submitted to USAID. The three-person field teams were formed collaboratively (i.e., no one was coerced to work with anyone or to go to any location) based on the following considerations: Skill level – a lead was identified for each team in order to provide added support and judgment to team members when in the field. Ethno-sectarian sensitivities in assigned governorates – some locations indicate different levels of comfort based on field team members’ identify and background. Availability – the team responsible for visits to Mercy Corps’ southern governorates will be traveling away from their homes for nearly 12 consecutive days. IPs are uncomfortable having private security contractors in the projects’ communities; therefore expat team members are limited to IP-based data collection activities and phone interviews. 38 Team skill-level, ethno-sectarian characteristics, and availability were all taken into account in assigning teams to implementing partners. No team member had worked for any of the IPs before so there were not issues in terms of prior relationships that might bias data gathering. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 73 Overview of CAP III objectives, partner implementation variation, documented performance, and programmatic modifications, including changes from CAP I and CAP II activities Presentation of evaluation objectives and key research questions Review of field visit schedule, on-site agendas, and key activities Data collection tools, including surveys, key stakeholder interview protocol, observation checklist, focus group guides Administrative requirements (e.g., timesheets, travel expense reports) The evaluation team lead is in daily contact with field teams to identify any difficulties with evaluation plan implementation. She disseminates guidance and clarifications to entire team, as needed. Additionally, the evaluation team will meet on a weekly basis in order to review any clarifications needed on evaluation process or data collection, beginning May 19. Field Visit Data Collection Beginning on May 15, the field team conducts visits to each of the 15 governorates participating in the CAP III program. To distribute the workload and travel considerations, a two- to three￾person team is assigned to each governorate. The team will spend two to three days in each governorate, depending upon their ability to schedule all planned activities in the allowed time. The team lead works with each of the IP’s COP/POC to develop governorate-specific agendas.39 To assist in the scheduling process, the team lead provides COPs with a sample agenda. In governorates that include an IP central office (e.g., where the COP and other key IP staff work), the agenda will include IP staff interviews. These governorates include Anbar, Babil, Baghdad, Basrah, and Kirkuk. In governorates without a central office (e.g., small program offices out of which IPs’ Community Mobilizers operate), the agendas will not include activities with the central office staff. The expat team members will interview the COPs instead of the Iraqi field team. Additionally, Iraqi staff advised that non-Iraqi COPs often feel more comfortable providing sensitive information to expat members of the team. This approach also enables easier communication as none of the COPs are native Arabic speakers and are more comfortable conversing in English. The expat team will also interview the lead M&E Specialist for each IP in order to understand the variation in program data collection processes and definitions of key variables. Survey data will be input and uploaded to a central database designed by the database designer (as mentioned previously, under the PERFORM contract) and managed by the team’s Iraqi data management specialist (working under PERFORM contract). The latter will input all survey date to ensure consistency of data entry across governorates and surveys. Each field team member will be responsible for preparing written notes for all interviews and focus groups in English. Teams are expected to complete notes within one week of the field visit. 39 Due to security concerns expressed by evaluation research team members, only one evaluator traveled to Diyala for a two-day visit. The research agenda was consequently abbreviated, conducting only two project visits and eliminating interviews of a Provincial Council member and Marla Fund focus group/surveys. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 74 Data Collection Tools and Analysis The evaluation field team will gather data during the field visits through mixed methods. Data gathered will be used for both quantitative and qualitative analysis, as described in the next section. The data collection tools and analysis methods for each are described below. (1) Individual stakeholder interviews: Individual interviews will be conducted in a semi￾structured format using a questionnaire. Each interview protocol will include: Introduction of the purpose of the interview Explanation of confidentiality Set of common questions specific to CAP III asked across respondent types Individually tailored questions relevant to the specific background and experience of the respondent type Follow-up probes for further clarification in the course of the interview The guides will be open-ended questions, although post field visit analysis will be able to convert qualitative data into quantitative data of frequency of specific themes, when appropriate. Interviewers will use a standard interview format for summarizing notes to facilitate cross-respondent and cross-site analysis. (2) Group Interviews: As appropriate, several people from the same organization or agency will participate together in an interview. The research team members will conduct these group interviews using the same semi-structured interview approach used in individual interviews. For example, if there is more than one trainer or engineer on staff, these people will be interviewed together. (3) Focus Group Discussion: Each focus group should not include more than 10 people in order to allow adequate time for each person’s active participation. Discussions will focus on participants’ actual experience with the CAP III program, lessons learned from their participation, sustainability of CAP III activities, and recommendations regarding citizen engagement and community-driven development for the future. The CAG and Local Council focus groups will be especially helpful in observing group dynamics and interactions (e.g., dominant leader, level of agreement/differences in perceptions, common/divergent understanding of community priorities). CAG and Marla Fund beneficiaries’ focus groups will assist in identifying benefits accrued to target populations and the sustainability of those benefits. (4) Mini-survey (Questionnaires): Mini-surveys with primarily closed-ended questions will provide quantitative data of key CAP III program participants and stakeholders to supplement qualitative data and provide a snapshot of CAP III’s end-of-project contributions. The short questionnaires will be a self-report and self-assessment instrument. As part of the project visits, field team members will administer the surveys to CAP III beneficiaries at each of the project sites. At a location suitable to the comfort of Marla Fund project beneficiaries, a field team member will administer a survey of these beneficiaries. The questionnaires for beneficiaries will be paper surveys in Arabic. Additionally, all IPs’ CAP Community Mobilizers/facilitators/developers (mobilizers)40 will be asked to complete an online survey, assuming administering electronic surveys 40 Implementing partners use different titles to refer to the staff member that works most closely with CAGs. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 75 are technically feasible. The evaluation team designed all surveys to be completed in approximately 15 with primarily close-ended questions. Iraqi field team members will be consulted on development of surveys in order to ensure language is understandable and familiar. (5) Project Observation Guide and checklist (checklist): At each project site, the field team member will complete a one-page form41 to document observed project characteristics and utilization. The checklist requires field team members to comment on project utilization, physical condition, ongoing operation and maintenance, and relevancy to community needs. Project Visits and Selection Criteria The purpose of the project site visits is to understand the various steps of project development (i.e., CAG formation, needs assessment/prioritization, project design, mobilization of resources, implementation, maintenance). The visits will provide a ground-truthing of individual interviews and focus groups and provide examples of different types of projects. Data collection will focus on the processes for community driven development, citizen engagement, and local government contribution/leadership. These visits are not part of a "monitoring" or "audit" process but an effort to identify lessons learned from these on-the-ground experiences. To form a complete understanding of the various roles of key stakeholders and to determine to what extent projects provide opportunities for citizen/government engagement and cooperation, the evaluation team will conduct the following activities at each project site (with the minimum number of expected respondents): CAP Community Mobilizer associated with the project: Individual interview and escort throughout the day. (1 Community Mobilizer) CAG members: Focus group of CAG members involved with the project. Local Council: Focus group of Local Council members (qadaa, nahiyaa, neighborhood) from the area in which project is located. CAP III project beneficiaries: Discussion with people who are using the facility or benefiting from the service. This discussion can take a variety of formats depending upon the project, including focus groups, informal discussions with those on-site at time of visit, "town hall" style meeting. Project Observation Guide: Completed by field team member to document observed project characteristics and utilization. The team will prepare a purposeful sample of 6542 projects to ensure inclusion of projects with a variety of relevant characteristics to understanding citizen and government involvement in the 41 The Project Observation Guide and Checklist has been modeled on the USAID DGO/Field Monitoring Reporting Form. 42 Visiting 65 projects provides a margin of error (MOE) of 11.97%. Increasing this number to 80 (representing an additional 15 days of field work) would bring the MOE to 10.75%, which does not represent a significant added value. To bring the MOE to 5% would require a sample size of 325 projects. However, the field visit design for the 65 projects provides a rich level of detail on a variety of projects that will provide detailed insight into the processes by which these projects were undertaken, including involvement by CAGs and local government councils as well as community beneficiaries. This level of detail would not be possible with a sample size as large as 325 projects without unrealistic expenditure of time and money. Calculations of MOE can be made through this online tool: http://www.americanresearchgroup.com/moe.html. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 76 process of meeting community needs. Independent variables the evaluation team will use for project selection are the following: Projects located in both urban and rural communities Projects that are “active” (in the process of being implemented) and others that have been closed for at least 6 months (to consider project sustainability) Projects implemented by location-based CAGs as well as thematic and cluster CAGs Projects with an above average GOI contribution or cost share for that governorate. Different project types/sectors (e.g., education, health, transportation, parks, recreation, essential services, sanitation, electricity, income generation, etc.) Projects designed to assist targeted populations: IDPs, female heads of households, youth, religious/ethnic minorities Projects of CAGs with a variety of members representing communities of interest: women, IDPs, female heads of households, youth, religious/ethnic minorities Security considerations in terms of access to sites to avoid putting CAP III participants or field team members at risk43 Key Stakeholders The evaluation team will include the following categories of key stakeholders in the data collection activities: (1) Implementing partner staff and program partners: The evaluation team can conduct these interviews in the IP headquarters or at the IP’s AoR hub(s) or sub-program offices, whichever are more convenient for the IP. [Note: Per the above section on project visits and selection criteria, the Community Mobilizers affiliated with the visited projects will accompany the evaluation team for the day at the project site.] Although the actual position title may vary, those performing the following duties should be interviewed for each IP: Chief of Party (and Deputy, if applicable) CAP III Program Manager Marla Fund Program Manager M&E Specialist CAP Community Mobilizers (all): Both individual interviews for those associated with visited projects and a mini-survey for all on staff. Trainers [both CAG and Local Council training]: Group interview if more than one. Engineers/technical specialists: At least one IP has specialists on their staff to support CAGs in the design of their projects as well as to review CAG project proposals. (approximately 2 persons) Marla Fund Local NGOs: Iraqi NGO staff partnered for implementation of Marla Fund activities 43 The field team’s final decisions as to adequate security levels will be based on input from IP-provided context assessments, QED contracted security teams, and evaluation team personal knowledge and levels of comfort. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 77 (2) CAG members: The CAP program has made a significant investment of resources and aspirations in cultivating the CAGs as effective citizen advocates. The degree to which CAG members understand their role reflects a sense of leadership on behalf of the common good and expresses empowerment to take action will provide an important measure of the current capacity of communities to participate in the local development process. The field team members will conduct focus groups and surveys of CAG members associated with the visited projects. (3-5 members/focus group) (3) Relevant government officials: A key part of the CAP program is encouraging local government officials to understand and respond to constituent priorities. Although the initial priority was fostering engagement between communities and Local Councils (i.e., qadaa, nahiyaa, neighborhood councils), authority over service provision and allocation of government resources remains primarily at the ministries and with Provincial Councils. Interviewing these multiple levels of government can support USAID understanding whether CAP contributed to opening formal and/or informal channels for citizen advocacy to these various decision makers, with a particular focus on underlying reasons for challenges and successes. The following types and number of respondents will be interviewed at each project site: District/sub-district/neighborhood councils of visited projects - Focus group of those council members from the visited project’s area (3-5 persons/project) Line ministry involved with the projects that will be visited - Individual interview (e.g., planning, education, transportation, health, youth) (1 person/province) Provincial council member who has been active in the CAP program (1 person/province) (4) CAP project beneficiaries: The purpose of these interviews is to understand the contributions CAP has made to individuals and communities who potentially benefit from the projects. Working with the IP, the team will aim to gather beneficiaries at the project site or another location that is convenient and comfortable for the group (e.g., IP office, community facility) in order to: Conduct a short mini-survey OR Convene a group discussion facilitated by an evaluation team member (5-10 persons/project) (5) Marla Fund beneficiaries: The purpose of these interviews is to understand the contributions the Marla Fund has made to individuals and communities impacted by Coalition forces, and how these contributions can be sustained. To help this assessment, the team would like to gather Marla Fund recipients at a location that is convenient and comfortable to the group (for example, IP office, community facility, youth center, or women’s center) in order to: Conduct a short mini-survey AND Convene a group discussion facilitated by an evaluation team member (approximately 5-10 persons/province) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 78 TIMELINE OF EVALUATION ACTIVITIES The CAP III performance evaluation is to be completed during a three-month period (April 23 to July 24, 2012). The major phases and timeline of the evaluation are as follows: 1. Desk Review and Preparation Phase (April 23 to May 14): a. Review of CAP documentation b. DC-based interviews of USAID and IP staff c. Development of evaluation work plan: Draft (May 4), Final (May 15) d. Development of data collection tools (translation into Arabic) e. Scheduling of field visits f. Training of field team g. In-brief: initial conference call (May 7), In-brief at Mission (May 14) 2. Field Work/Data Collection (May 14 to June 11): a. Field visits (May 14 to June 11) b. Data input (ongoing) c. Evaluation team meetings (weekly) d. Mid-brief presentation (May 28) 3. Field Work – Consolidation and Analysis (June 11 to 18): a. Final input of all data b. Data analysis c. Submit draft report outline to USAID (June 14) d. Exit briefing (June 18) 4. Reporting (June 21 to July 24): a. Draft report (June 29) b. USAID submits comments (July 11) c. Final report to QED for editing and formatting (July 16) d. Final report to USAID (July 24) e. Report 508 compliant and submitted to DEC (July 27) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 79 ANNEX C-1. SAMPLE FIELD VISIT AGENDA WITH IP CENTRAL OFFICE USAID/Iraq CAP III Performance Evaluation DAY 1 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:30am CAP III Program Manager Marla Fund Program Manager 11:00am-12:30pm Provincial Council member(s) Line ministry/ministries Marla Fund NGO Partner(s) 12:30-1:00pm Lunch 1:00-2:00pm CAG and local government trainers interview Meeting and survey with Marla Fund recipients (approx. 10) at IP office or central/location 2:00-3:00pm CAP engineers or technical experts 3:00-4:00pm Community Facilitator for Project #1 Community Facilitator for Project #2 Community Facilitator for Project #3 DAY 2 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:30am Focus group with CAG for Project #1 (survey) Focus group with CAG for Project #2 (survey) Focus group with CAG for Project #3 (survey) 11:00am-12:00pm Focus group with local government for Project #1 (survey) Focus group with local government for Project #2 (survey) Focus group with local government for Project #3 (survey) 12:00-1:00pm Lunch 1:00-3:00pm Site Visit to Project #1 Meeting with beneficiaries (survey) Site visit to Project #2 Meeting with beneficiaries (survey) Site Visit to Project #3 Meeting with beneficiaries (survey) 3:30-4:00pm Debrief with Community Facilitator for Project #1 Debrief with Community Facilitator for Project #2 Debrief with Community Facilitator for Project #3 Additional Field Visit Activities: 1. Distribute community facilitator survey (web link or paper survey) 2. Identify relevant Council of Representative member (Baghdad interview) 3. Collect any relevant documentation: community outreach materials, training agendas, program resources Interviews by Phone/Skype (conducted by expat team) 1. Implementing Partner Chief of Party 2. Implementing Partner M&E Specialist USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 80 ANNEX C-2. SAMPLE FIELD VISIT AGENDA WITHOUT IP CENTRAL OFFICE USAID/Iraq CAP III Performance Evaluation DAY 1 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3* 9:00-10:00am Community Facilitator for Project #1 Community Facilitator for Project #2 Community Facilitator for Project #3 10:00-11:00am Focus group with CAG for Project #1 (survey) Focus group with CAG for Project #2 (survey) Focus group with CAG for Project #3 (survey) 11:30am-12:30pm Focus group with local government for Project #1 (survey) Focus group with local government for Project #2 (survey) Focus group with local government for Project #3(survey) 12:30-1:30pm Lunch 1:30-3:30pm Site Visit to Project #1 Meeting with beneficiaries (survey) Site Visit to Project #2 Meeting with beneficiaries (survey) Site Visit to Project #3 Meeting with beneficiaries (survey) 3:30-4:00pm Debrief with Community Facilitator for Project #1 Debrief with Community Facilitator for Project #2 Debrief with Community Facilitator for Project #3 DAY 2 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:00am Community Facilitator for Project #4 Community Facilitator for Project #5 Line ministry individual interview 10:00-11:00am Focus group with CAG for Project #4 (survey) Focus group with CAG for Project #5 (survey) Line ministry individual interview 11:30am-12:30pm Focus group with local government for Project #4 (survey) Focus group with local government for Project #5 (survey) Provincial Council member individual Interview 12:30-1:30pm Lunch 1:30-3:30pm Site Visit to Project #4 Meeting with beneficiaries (survey) Site Visit to Project #5 Meeting with beneficiaries (survey) Meeting and survey with Marla Fund recipients (approx. 20) at central location 3:30-4:00pm Debrief with Community Facilitator for Project #4 Debrief with Community Facilitator for Project #5 Additional Field Visit Activities: 1. Distribute Community Facilitator Survey (web link or paper survey) 2. Identify relevant Council of Representative member (Baghdad interview) 3. Collect any relevant documentation Community outreach materials Training agendas Program resources USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 81 ANNEX C-3. SUMMARY OF ESTIMATED NUMBER OF RESPONSES BY DATA COLLECTION INSTRUMENT AND RESPONDENT TYPE USAID/Iraq CAP III Performance Evaluation Data Collection Instrument Respondent Type # of Persons Unit Total Respondents INTERVIEW Chief of Party 1 4 IPs 4 CAP III Program Manager 1 4 IPs 4 Marla Fund Program Manager 1 4 IPs 4 M&E Specialist 1 4 IPs 4 Community Mobilizers 3 5 5 gov. 10 gov. 65 Trainers 2 4 IPs 8 Local NGO 2 4 IPs 8 Engineers/Technical Experts 2 4 IPs 8 Line Ministries 1 15 gov. 15 Provincial Council Members 1 15 gov. 15 COR Members 1 15 gov. 15 Total 150 FOCUS GROUP RESPONDENTS CAG Members 5 Mbrs x 3 projects 5 Mbrs x 5 projects 5 gov. 10 gov. 325 Local Government (LG) Council Members 5 Mbrs x 3 projects 5 Mbrs x 5 projects 5 gov. 10 gov. 325 CAG Project Beneficiaries 10 Ben. x 3 projects 10 Ben. x 5 projects 5 gov. 10 gov. 650 Marla Fund Beneficiaries 10 15 gov. 150 Total 1,450 MINI-SURVEY RESPONDENTS Community Facilitators 5 4 IPs 20 CAG Members 5 Mbrs x 3 projects 5 Mbrs x 5 projects 5 gov. 10 gov. 325 LG Members 5 Mbrs x 3 projects 5 Mbrs x 5 projects 5 gov. 10 gov. 325 CAG Project Beneficiaries 10 Ben. x 3 projects 10 Ben. x 5 projects 5 gov. 10 gov. 650 Marla Fund Recipients 10 15 gov. 150 Total 1,470 PROJECT OBSERVATION CHECKLIST CAG Projects 10 Ben. x 3 projects 10 Ben. x 5 projects 5 gov. 10 gov. 65 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 82 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 83 ANNEX D. MATRIX OF RESEARCH QUESTIONS AND SUB-QUESTIONS In order to focus the data collection activities on areas of particular relevancy to the USAID/Iraq’s Democracy and Governance Office, the team has developed a set of key research questions consistent with the five evaluation questions required in the statement of work. These questions will guide all data collection to ensure priority themes are covered. Although each respondent may not be able to address each theme, the total of all interviews will be combined to develop conclusions and recommendations responding to each of the research questions. Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators I. Did the CAP Program contribute to increased participation in collective community actions? I-1. How has CAP III contributed to changes in the formal processes and institutions that Iraqis use to advocate on behalf of community needs? Descriptive Local Council focus group CAG focus group CAP beneficiary focus group Provincial Council and ministry interviews Examples of new/refined mechanisms used in specific cases I-2. How has CAP III contributed to Iraqi communities feeling a greater sense of empowerment in addressing the needs of their community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey CAG focus group CAP beneficiary focus group Local Council focus group Provincial Council and ministry interviews Self-assessment of change in agency of non-government Iraqis Government officials' observed changes in citizen advocacy I-3. Who do Iraqis rely upon to address community needs? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey Local Council focus group Provincial Council and ministry interviews Response to multiple-choice question: Whom would you go to first to address an urgent community need? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 84 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators I-4. Is collective community action measurable by tangible infrastructure (i.e., improved school, increased water access, repaired road) or is collective community action an attitude or behavior? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Identification of greatest achievement through CAP involvement: development project or participatory process? II. Are CAP communities better able to articulate their needs and mobilize resources within/outside the community to solve common problems? If not, why not? II-1. To what extent are CAGs representative of and respected by the community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interview CAP beneficiary survey or focus group Methods by which CAGs were selected Description of CAGs’ contributions to the community by non-CAG respondents II-2. Has CAP III contributed to CAGs capacity to identify community needs through participatory processes? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interview CAP beneficiary survey or focus group Project Site Visit Observation Guide Projects are valued by beneficiaries and seen as addressing important community needs CAG self-assessment of ability to identify community needs Government officials valuing CAG/citizen input II-3. Has CAP III contributed to CAGs capacity to mobilize resources on behalf of community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group CAGs understanding of government budget processes for accessing GOI cost share for projects Perception of government bodies regarding response to CAG requests II-4. How do communities engage with the national and provincial budgeting process? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Explanation of processes by respondents USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 85 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators II-5. Did CAP III lead to an increased role of vulnerable groups in community mobilization efforts? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Testimony by CAG members, especially representatives of vulnerable groups Local Council understanding of vulnerable communities' needs II-6 How will support for CAGs' advocacy on behalf of communities continue after CAP III ends? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus groups Evidence of institutionalized/formal support Evidence of individual commitment by CAG members III. Does local executive and representative government (defined as governorate-level ministry representatives, governorate councils, Local Councils, and the Council of Representatives) in CAP communities better meet the articulated needs of the community? If not, why not? III-1. How has CAP III contributed to changes in the how local government councils see their role in addressing community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Self-assessment of Local Council respondents Descriptions of changes from non-Local Council respondents III-2. To what extent are local government councils representative of and respected by the community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Methods by which Local Councils were selected Description of Local Councils’ contribution to the community by community members Expressions of appreciation/respect by non-Local Council respondents III-3. Has CAP III contributed to local government councils' ability to identify community needs through participatory processes? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Local Council self-assessment of ability to identify community needs Government officials valuing Local Council input to budgeting/planning processes USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 86 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators III-4. Has CAP III contributed to local government councils' capacity to mobilize resources on behalf of community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Local Council understand national and provincial budget and planning processes Local Council supports CAG access to GOI cost share for projects Government interviews indicate responding to Local Council requests III-5. Did CAP III lead to local governments' increased understanding and involvement of vulnerable groups and their needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group CAP beneficiary survey or focus group CAG and beneficiaries testimony of Local Council supportive actions, especially representatives of vulnerable groups Local Council expression of understanding of vulnerable communities' needs III-6 How will support for Local Government advocacy on behalf of communities continue after CAP III ends? Descriptive IP staff (COP, M&E, , trainer, mobilizer, engineer) CAG survey or focus group Local Council focus groups Evidence of institutionalized/formal support Evidence of individual commitment by Local Councils IV. Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the fund’s defined purpose? If not, why not? IV-1. Has the Marla Fund succeeded in reaching eligible war victims? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Testimony that most eligible victims have been served IV-2. What are the reasons war victims do not access Marla Fund? Descriptive IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Identification of reasons (e.g., uninformed, ineligible, opposed to U.S. support, needs different kind of support) IV-3. Has the Marla Fund developed a mechanism within Iraqi society for the ongoing support of war victims? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Identification of institution or mechanisms within civil society or the government for providing ongoing support IV-5 Should Marla Fund eligibility criteria be expanded to all victims of war? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Examples of potentially eligible victims unable to provide adequate verification Examples of transformative potential in U.S.-Iraqi relations if broader service base USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 87 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators V. What are lessons learned that could be applied to future USAID work with Iraqi communities and local governments specifically on approaches to: (1) community engagement/participation, (2) training methodology, (3) securing government buy-in/participation? V-1. What training formats are most effective in the Iraqi context? Descriptive IP staff (COP, M&E, trainer, mobilizer) CAG focus group Local Council focus group Self-assessment of capacity in key skill training areas Testimony of IP trainers V-2. Does effectiveness of training formats vary for vulnerable groups as compared to the general population? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Testimony of preferences by CAG and local government training participants that represent vulnerable populations V-3. What strategies for outreach in the community were most effective in the CAG formation process? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey or focus group Testimony of Community Mobilizers, CAG members, CAP beneficiaries Strategies that led to formation of CAGS that are representative, active, community-supported V-4. What types of CAP projects generated broad and enthusiastic community support and/or involvement? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Community cost-share in CAP projects GOI cost share in CAP projects Enthusiasm expressed by beneficiaries V-5. What strategies for outreach were most effective in building relationships between the community and government officials? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group CAP beneficiary survey or focus group Provincial Council and ministry interviews Testimony of how new citizen￾government relationships were initiated Examples of government officials' appreciation for value added of citizen participation Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators I. Did the CAP Program contribute to increased participation in collective community actions? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 88 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators I-1. How has CAP III contributed to changes in the formal processes and institutions that Iraqis use to advocate on behalf of community needs? Descriptive Local Council focus group CAG focus group CAP beneficiary focus group Provincial Council and ministry interviews Examples of new/refined mechanisms used in specific cases I-2. How has CAP III contributed to Iraqi communities feeling a greater sense of empowerment in addressing the needs of their community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey CAG focus group CAP beneficiary focus group Local Council focus group Provincial Council and ministry interviews Self-assessment of change in agency of non-government Iraqis Government officials' observed changes in citizen advocacy I-3. Who do Iraqis rely upon to address community needs? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey Local Council focus group Provincial Council and ministry interviews Response to multiple-choice question: Whom would you go to first to address an urgent community need? I-4. Is collective community action measurable by tangible infrastructure (i.e., improved school, increased water access, repaired road) or is collective community action an attitude or behavior? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Identification of greatest achievement through CAP involvement: development project or participatory process? II. Are CAP communities better able to articulate their needs and mobilize resources within/outside the community to solve common problems? If not, why not? II-1. To what extent are CAGs representative of and respected by the community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interview CAP beneficiary survey or focus group Methods by which CAGs were selected Description of CAGs’ contributions to the community by non-CAG respondents USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 89 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators II-2. Has CAP III contributed to CAGs capacity to identify community needs through participatory processes? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interview CAP beneficiary survey or focus group Project Site Visit Observation Guide Projects are valued by beneficiaries and seen as addressing important community needs CAG self-assessment of ability to identify community needs Government officials valuing CAG/citizen input II-3. Has CAP III contributed to CAGs capacity to mobilize resources on behalf of community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group CAGs understanding of government budget processes for accessing GoI cost share for projects Perception of government bodies regarding response to CAG requests II-4. How do communities engage with the national and provincial budgeting process? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Explanation of processes by respondents II-5. Did CAP III lead to an increased role of vulnerable groups in community mobilization efforts? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Testimony by CAG members, especially representatives of vulnerable groups Local Council understanding of vulnerable communities' needs II-6 How will support for CAGs' advocacy on behalf of communities continue after CAP III ends? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus groups Evidence of institutionalized/formal support Evidence of individual commitment by CAG members III. Does local executive and representative government (defined as governorate-level ministry representatives, governorate councils, Local Councils, and the Council of Representatives) in CAP communities better meet the articulated needs of the community? If not, why not? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 90 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators III-1. How has CAP III contributed to changes in the how local government councils see their role in addressing community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Self-assessment of Local Council respondents Descriptions of changes from non-Local Council respondents III-2. To what extent are local government councils representative of and respected by the community? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews Methods by which Local Councils were selected Description of Local Councils’ contribution to the community by community members Expressions of appreciation/respect by non-Local Council respondents III-3. Has CAP III contributed to local government councils' ability to identify community needs through participatory processes? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Local Council self-assessment of ability to identify community needs Government officials valuing Local Council input to budgeting/planning processes III-4. Has CAP III contributed to local government councils' capacity to mobilize resources on behalf of community needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Provincial Council and ministry interviews CAP beneficiary survey or focus group Local Council understand national and provincial budget and planning processes Local Council supports CAG access to GoI cost share for projects Government interviews indicate responding to Local Council requests III-5. Did CAP III lead to local governments' increased understanding and involvement of vulnerable groups and their needs? Normative IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group CAP beneficiary survey or focus group CAG and beneficiaries testimony of Local Council supportive actions, especially representatives of vulnerable groups Local Council expression of understanding of vulnerable communities' needs USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 91 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators III-6 How will support for Local Government advocacy on behalf of communities continue after CAP III ends? Descriptive IP staff (COP, M&E, , trainer, mobilizer, engineer) CAG survey or focus group Local Council focus groups Evidence of institutionalized/formal support Evidence of individual commitment by Local Councils IV. Did CAP partners assist civilian victims of conflict through the Marla Fund, as per the fund’s defined purpose? If not, why not? IV-1. Has the Marla Fund succeeded in reaching eligible war victims? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Testimony that most eligible victims have been served IV-2. What are the reasons war victims do not access Marla Fund? Descriptive IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Identification of reasons (e.g., uninformed, ineligible, opposed to U.S. support, needs different kind of support) IV-3. Has the Marla Fund developed a mechanism within Iraqi society for the ongoing support of war victims? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Identification of institution or mechanisms within civil society or the government for providing ongoing support IV-5 Should Marla Fund eligibility criteria be expanded to all victims of war? Normative IP staff (COP, mobilizer, Marla Fund manager) Local NGOs Marla Fund beneficiaries survey and focus group Examples of potentially eligible victims unable to provide adequate verification Examples of transformative potential in U.S.-Iraqi relations if broader service base V. What are lessons learned that could be applied to future USAID work with Iraqi communities and local governments specifically on approaches to: (1) community engagement/participation, (2) training methodology, (3) securing government buy-in/participation? V-1. What training formats are most effective in the Iraqi context? Descriptive IP staff (COP, M&E, trainer, mobilizer) CAG focus group Local Council focus group Self-assessment of capacity in key skill training areas Testimony of IP trainers V-2. Does effectiveness of training formats vary for vulnerable groups as compared to the general population? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Testimony of preferences by CAG and local government training participants that represent vulnerable populations USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 92 Main Evaluation Question and Research Sub-Questions Type of Question Source of Data Measures/Indicators V-3. What strategies for outreach in the community were most effective in the CAG formation process? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group CAP beneficiary survey or focus group Testimony of community mobilizers, CAG members, CAP beneficiaries Strategies that led to formation of CAGS that are representative, active, community-supported V-4. What types of CAP projects generated broad and enthusiastic community support and/or involvement? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group Community cost-share in CAP projects GoI cost share in CAP projects Enthusiasm expressed by beneficiaries V-5. What strategies for outreach were most effective in building relationships between the community and government officials? Descriptive IP staff (COP, M&E, trainer, mobilizer, engineer) CAG focus group Local Council focus group CAP beneficiary survey or focus group Provincial Council and ministry interviews Testimony of how new citizen￾government relationships were initiated Examples of government officials' appreciation for value added of citizen participation USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 93 ANNEX E. EVALUATION ACTIVITIES AND SITE VISIT SCHEDULE April 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Desk Review Desk Review Desk Review Desk Review Desk Review 29 30 Work Plan Development and DC Meetings Planning/ Schedule Setup USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 94 May 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 3 4 5 Work Plan/ DC Meetings Planning/ Schedule Work Plan/ DC Meetings Planning/ Schedule Work Plan/ DC Meetings Planning/ Schedule Work Plan/ DC Meetings Contact IP COPs Draft Work plan submitted via QED Planning/ Schedule Local team meeting 6 7 8 9 10 11 12 Planning/Schedule Follow-up with IP COPs Data Collection Tools Planning/ Schedule USAID Conference Call w/ QED Data Collection Tools Planning/ Schedule Expats Depart DC Travel (Expat) Planning/ Schedule Expats Arrive in Iraq Data Collection Tools Rest Day Training Prep Translation of Tools 13 14 15 16 17 18 19 Team Training (all) Fieldwork Begins Mercy Corps (MC): Travel to Basrah ACDI/VOCA (A/V):Travel to Kirkuk USAID In-Briefing MC: Basrah CHF: Anbar (Ramadi) A/V: Kirkuk Final Work Plan Submitted MC: Basrah A/V: Kirkuk MC: Maysan A/V: Kirkuk Rest Day A/V: Travel to Baghdad Team Meeting / Debrief: Basrah, Kirkuk, Anbar (Day 1), Maysan (Day 1) 20 21 22 23 24 25 26 MC: Maysan CHF: Anbar (Fallujah) MC: Dhi Qar IRD: Baghdad IP staff A/V: Travel to Mosul A/V: Kirkuk NOTES DUE MC: Dhi Qar A/V Team: Ninawa CHF: Anbar NOTES DUE MC: Muthanna A/V: Ninawa CHF: Karbala IRD: Provincial Council Mobilizer Survey Drafted IRD: IP Mtg Notes DUE MC: Muthanna A/V: Ninawa CHF: Karbala IRD: Proj #1 Nissan Mobilizer survey translated Rest Day MC: Travel to Baghdad A/V: Travel to Baghdad Team Meeting/Debrief: Anbar (Day 2), Maysan (Day 2), Baghdad, Ninawa, Karbala, Dhi Qar, Muthanna USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 95 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 27 CHF: Babil (Hilla) IRD: Baghdad Proj #2 IRD: Proj #2 Rusafa 28 CHF – Babil (Hilla) IRD- Marla Beneficiaries A/V: Ninawa NOTES DUE (Z) Mid-point USAID Briefing Mobilizer Survey Translated 29 A/V: Salah ad Din IRD: Project #3 Issue CAG CHF: Karbala NOTES DUE 30 A/V Team: Salah ad Din CHF: Najaf IRD: Proj #4 Adhamiya 31 A/V: Salah ad Din CHF: Najaf IRD: Proj #5 Karada USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 96 June 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 Rest day Team Meeting/Debrief: Babil, Salah ad Din, Najaf MC: All Notes DUE A/V: Kirkuk Notes Due (A) 3 4 5 6 7 8 9 CHF: Qadissiyahh CHF – Qadisiyyah A/V: Salah ad Din NOTES DUE (Z) Mobilizer survey disseminated CHF: Babil NOTES DUE A/V: Ninawa Notes DUE (A) CHF: Wasit CHF: Wasit Rest day Team Meeting/Debrief: Qadisiyyah, Diyala, Wasit A/V: Salah ad Din Notes DUE 10 11 12 13 14 15 16 Fieldwork 2/Data Analysis Mobilizer survey DUE Fieldwork 2 / Data Analysis CHF: Najaf NOTES DUE Data Analysis (Expats, 5 days, 4 members of local team, 5 day) CHF: Qadisiyyah NOTES DUE Data Analysis TEAM ANALYSIS Data Analysis TEAM ANALYSIS Draft outline to USAID Rest Day Data Analysis CHF: Wasit NOTES DUE 17 18 19 20 21 22 23 Data Analysis Exit Briefing to USAID Report Drafting Travel (Expats) Report Drafting Travel (Expats) 24 25 26 27 28 29 30 Report Drafting Report Drafting Report Drafting Report Drafting Draft Report to USAID USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 97 July 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 3 4 5 6 7 USAID Review of Draft Report USAID Review of Draft Report USAID Review of Draft Report USAID Review of Draft Report USAID Review of Draft Report 8 9 10 11 12 13 14 USAID Review of Draft Report USAID Review of Draft Report USAID Review of Draft Report USAID submit comments to QED Final report preparation Final report preparation Final report preparation 15 16 17 18 19 20 21 Final report preparation Final report preparation Final report preparation Editing and Formatting (QED HQ) Editing and Formatting (QED HQ) 22 23 24 25 26 27 28 Editing and Formatting (QED HQ) QED Submit final report to USAID Report 508 compliant and submitted to DEC USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 98 ANNEX F. SAMPLE AGENDAS FOR FIELD VISITS (CENTRAL OFFICE, OTHER PROVINCES) Sample Field Visit Agenda Governorate with Implementing Partner Main Office (CHF: Anbar and Hilla; Mercy Corps: Basrah, IRD: Baghdad, ACDI/VOCA: Kirkuk) DAY 1 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:30am CAP III Program Manager Marla Fund Program Manager 11:00am-12:30pm Provincial Council member(s) Line ministry/ministries Marla Fund Local NGO Partner(s) 12:30-1:00pm Lunch 1:00-2:00pm CAG and Local Council trainers Meeting and survey with Marla Fund recipients (approx. 10) at IP office or central/location 2:00-3:00pm CAP III engineers or technical experts 3:00-4:00pm Community Mobilizer for Project #1 Community Mobilizer for Project #2 Community Mobilizer for Project #3 DAY 2 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:30am Focus group with CAG for Project #1 Focus group with CAG for Project #2 (survey) Focus group with CAG for Project #3 (survey) 11:00am-12:00pm Focus group with Local Council for Project #1 Focus group with Local Council for Project #2 Focus group with Local Council for Project #3 12:00-1:00pm Lunch 1:00-3:00pm Site Visit to Project #1 Meeting or survey of beneficiaries Site Visit to Project #2 Meeting or survey of beneficiaries Site Visit to Project #3 Meeting or survey of beneficiaries 3:30-4:00pm Debrief with Community Mobilizer for Project #1 Debrief with Community Mobilizer for Project #2 Debrief with Community Mobilizer for Project #3 Additional Field Visit Activities: 1. Complete Project Observation Form while at project site. 2. Collect any relevant documentation Community outreach materials Training agendas Program resources Interviews In-Person or by Phone/Skype (conducted by expat team) 1. Implementing partner Chief of Party: implementing partner M&E Specialist USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 99 USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation Sample Field Visit Agenda Governorate without Implementing Partner Main Office (CHF: Karbala, Qadissiyah, Najaf, Wasit; Mercy Corps: Maysan, Dhi Qar, Muthanna; ACDI/VOCA: Ninawa, Salah ad Din, Diyala44) 44 Due to security concerns, the field visit to Diyala was limited to a two-day visit conducted by one evaluation team member. The evaluator conducted focus groups with CAG and LC members, surveys of CAP III beneficiaries, line ministries, and site visits for two projects instead of five projects. The evaluator also interviewed the Marla Fund local NGO. Due to the reduced schedule, there were no meetings with Marla Fund beneficiaries or Provincial Council members. DAY 1 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3* 9:00-10:00am Community Mobilizer for Project #1 Community Mobilizer for Project #2 Community Mobilizer for Project #3 10:00-11:00am Focus group with CAG for Project #1 Focus group with CAG for Project #2 Focus group with CAG for Project #3 11:30am-12:30pm Focus group with Local Council for Project #1 Focus group with Local Council for Project #2 Focus group with Local Council for Project #3 12:30-1:30pm Lunch 1:30-3:30pm Site Visit to Project #1 Meeting or survey of beneficiaries Site Visit to Project #2 Meeting or survey of beneficiaries Site Visit to Project #3 Meeting or survey of beneficiaries 3:30-4:00pm Debrief with Mobilizer Facilitator for Project #1 Debrief with Mobilizer Facilitator for Project #2 Debrief with Mobilizer Facilitator for Project #3 * If there are other interviews of interest in this governorate, this project site visit can be dropped in order to accommodate the interviews. DAY 2 DATE/TIME Evaluator 1 Evaluator 2 Evaluator 3 9:00-10:00am Community Mobilizer for Project #4 Community Mobilizer for Project #5 Line ministry interview 10:00-11:00am Focus group with CAG for Project #4 Focus group with CAG for Project #5 Line ministry interview 11:30am-12:30pm Focus group with Local Council for Project #4 Focus group with Local Council for Project #5 Provincial Council Member interview 12:30-1:30pm Lunch 1:30-3:30pm Site Visit to Project #4 Meeting or survey of beneficiaries Site Visit to Project #5 Meeting or survey of beneficiaries Meeting and survey with Marla Fund recipients (approx. 10) at central location 3:30-4:00pm Debrief with Community Mobilizer for Project #4 Debrief with Community Mobilizer for Project #5 Additional Field Visit Activities: Complete Project Observation Form while at project site. Collect any relevant documentation: Community outreach materials, training agendas, program resources USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 100 ANNEX G. DATA COLLECTION TOOLS 1. CAP III Evaluation Implementing Partner Interview Form – Headquarters (Washington, DC) NAME: TITLE/POSITION: ORGANIZATION: YEARS WORKING ON CAP/IRAQ: BEST CONTACT INFO FOR FOLLOW-UP: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. Our team of 2 expats and 8 Iraqi researchers has been contracted by USAID through QED’s PERFORM contract. USAID plans to use the results of this evaluation to determine to what extent CAP III has achieved its objectives and to identify lessons learned going forward with future programs that seek to engage Iraqi civilians in community based development and build local governance capacity. We anticipate that the findings will be useful for the new program—Broadening Citizen Participation through Civil Society—and the Governance Support Program. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that I do not attribute in my report anything you tell me by name or organizational affiliation. All interviews are aggregated and any quotations would simply refer to you as an Implementing Partner. A version of this report will be made public by USAID, but will not include names of those we have interviewed. Do you have any questions before we begin? * * * * * BACKGROUND 1. What are your responsibilities as they related to the CAP III program? [Probe: title, duties, length of time on project] 2. What, if any, responsibilities did you have in terms of the CAP I and CAP II programs? PROGRAM STRATEGY 3. What were the major differences in program design between CAP II and CAP III? How would you characterize these changes? [Probe: challenges, improvements] What was learned from earlier phases of CAP that was incorporated into the current phase? [Probe: possible areas of change are listed below.] CAG selection/formation Training of CAGs Training of local government Project Development/Implementation Mobilizing of resources (e.g., community, Provincial Council, ministries) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 101 Targeting IDPs, ethnic/religious minorities, women, youth [Note: If the respondent is not familiar with CAP II, you can skip this question.] 4. How would you define the theory of change for CAP III? Was this theory of change validated through program implementation and results? How could it have been improved? 5. How did the program adjust to the failure to pass the provincial powers law? What were the adjustments regarding work with Local Councils? [Probe: training, consultations, mobilizing resources, institutionalizing citizen participation, cooperation with CAGS, monitoring project implementation] 6. How did the program adjust to the focus on vulnerable groups (e.g., IDPs, ethnic/religious minorities, female headed households, youth)? Was this shift realistic? [Probe: slow down in implementation, CAG member dynamics, project priorities] PROGRAM IMPLEMENTATION 7. What were the implementation components for a typical CAP III program (specific to respondent’s IP)? Probe: Community outreach CAG: selection, training, mentoring Local Government: outreach, training, follow-up Projects: needs assessment/prioritization, identification, design, implementation, monitoring, maintenance 8. What were the challenges in conducting community outreach for the development of CAGs? How where these challenges addressed (i.e., lessons learned)? [Note: Important question for lessons learned for Broadening Participation through Civil Society program.] 9. What were the challenges in selecting CAG members? How did you ensure women’s participation? Participation of IDPs, vulnerable minorities, youth? What are lessons learned from this process? 10. How were partnerships with local government councils initiated? What made some partnerships more successful than others? 11. What steps were taken to engage vulnerable groups in CAG membership, community needs assessment and project design and implementation? How could their involvement been improved? 12. A significant shift between CAP III and CAP II was the increased involvement of local government. To what degree was this achieved? How was progress on this objective determined, and how could it be measured more effectively? 13. What are the salient features of projects implemented by your organization? How has this evolved over the course of CAP III? 14. Did community projects provide an opportunity for local leaders (districts and sub districts) and community representatives (CAGs) to work together? If so, in what ways was cooperation demonstrated? Probe: Assess/prioritize community needs Design appropriate interventions Mobilize resources from within and outside community to meet those needs Solicit regular community input USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 102 Give feedback on progress to the larger community. 15. What constitutes an especially successful project? Was this understanding of success shared across partners, or among the donors (USAID, community, Provincial Council, ministries)? 16. What are the variables that explain differences in CAP program implementation among the various communities? [Probe: type of project, location, urban/rural, availability resources, IP implementation design] 17. What were the implementation challenges and/or obstacles? MARLA FUND 18. How did your team identify and determine eligibility for Marla Fund recipients? [Probe: local NGO, IP staff, CAGs, police, PRTs] 19. Have most eligible victims been served? If so/not, how would you be able to make this determination? 20. Would you suggest changing in any way the eligibility criteria for Marla beneficiaries? 21. What role did the CAGs fulfill in the Marla program? 22. What are appropriate outcome indicators for the success of Marla activities? 23. What were the types of projects you initiated for Marla beneficiaries? Range of grant amounts? [Probe: training, technical assistance, business start-up, market research, infrastructure projects, occupational therapy, rehabilitation, home reconstruction] 24. What are the outcomes of assistance to Marla beneficiaries at the individual and family level? At the community level? [Probe: livelihood/self-sufficiency through employment, good will toward U.S., stable housing, mental health] PROGRAM RESULTS 25. What unit of analysis is most appropriate to understanding the impact of CAP III? Where is the change most visible? Among CAG members? Their community? At the provincial level? Nationwide? 26. In what ways have communities changed as a result of participation in/partnership with CAP III? 27. To what extent has CAP III succeeded in building productive relationships between CAGs and local government leaders? Governorate leaders (Governor, directorates, governorate councils)? National leaders (ministries, COR)? [Probe: ability to identify/respond to community needs, evidence of collaboration/mutual respect] 28. To what extent were CAGs able to advocate on behalf of their communities? [Probe: identify/prioritize needs, institutionalize broad community input, mobilize community/GoI resources, design/implement/monitor projects] USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 103 29. How did CAP III activities contribute to greater government responsiveness to community needs/service delivery? How could this contribution be further strengthened? 30. Which CAP activities were considered the most successful? [Probe: training, CAG formation, needs assessment, project implementation, training Local Government, building citizen participation mechanisms] Why? [Probe: How could this have been replicable/scalable?] 31. What are the lessons learned in terms of conducting training to support citizen participation? [Probe: format, approaches, classroom v. lecture v. experiential v. coaching, number of participants, participant types, trainer qualifications, topics to be covered, cultural considerations, gender dynamics] 32. How do you see CAP III ’s efforts being sustained or continuing into the future? [Probe: CAGs, citizen participation mechanisms, projects] 33. To what extent are the objectives of the program still valid? [Probe: encouraging citizen participation, training local government, funding community based development, training] OTHER ISSUES 34. If there was one change you would make to CAP III design or implementation, what would it be? 35. What was CAP III ’s biggest achievement? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 104 2. CAP III Evaluation Field Interview Form – CAP III Chief of Party (Interviewer’s Last Name; Province, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1-1/2 HOURS DATE OF INTERVIEW: NAME OF RESPONDENT: TITLE/POSITION: IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to understand the extent to which CAP III achieved its objectives and to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? * * * * * PROGRAM STRATEGY AND IMPLEMENTATION 1. In order to have a clear understanding of your organization’s approach to CAP III as compared to other implementing partners, please briefly outline the major programmatic components for your program? Probe: Community outreach CAG: selection, training, mentoring Local government: outreach, training, follow-up Projects: needs assessment/prioritization, identification, design, implementation, resource mobilization, monitoring, operations, maintenance 2. What have been successful strategies for conducting community outreach for initiating the CAP III program? How where these challenges addressed (i.e., lessons learned)? Did you need to have a letter of support from the Provincial Council to operate your program? 3. What were the challenges in selecting CAG members? How did you ensure women’s participation? Participation of IDPs, vulnerable minorities, youth? In your opinion, how representative is the CAG of the population as a whole? Are there new voices or do CAG members tend to be the same traditional community leaders? Y o u r L a s t N a m e – P r o v i n c e – C A P I I I F u n d M g r - D a Y o u r L a s t N a m e – P r o v i n c e – C A P I I I F u n d M g r - USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 105 4. What steps were taken to engage vulnerable groups in other CAP activities such as training or community needs assessment (women, female heads of households, youth, IDPs, ethnic/religious minorities)? What are the challenges of including these populations? 5. Do all of your CAGs develop at least one project or are there some CAGs that are formed that do not get to that stage? If so, what proportion of your CAP III CAGs have not developed at least one project? What are the reasons? 6. What has been your program’s experience with cluster CAGs (i.e., more than one CAG working together on a project)? What were the challenges and successes of this approach? 7. What has been your program’s experience with thematic CAGs (e.g., women’s issues, youth programs, people with disabilities)? What were the challenges and successes of this approach? 8. How is your CAP program different from the other implementing partners? ENGAGEMENT OF GOVERNMENT OFFICIALS 9. How does the CAP program work to engage Local Council officials (qadaa, nahiya, neighborhood councils) in the CAP program? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Training? b. Community needs assessment? c. Mobilization of GoI resources? 10. What types of support provided by CAP is especially important for Local Council members? After the CAP program ends, who do you think can provide this support? 11. Did community projects provide an opportunity for local leaders (districts and sub districts) and community representatives (CAGs) to work together? If so, in what ways was cooperation demonstrated? LESSONS LEARNED FROM TRAINING 12. USAID is especially interested in learning about effective training strategies. What training formats did you use and which do you think works best in promoting learning in the Iraqi context? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Classroom/Lecture b. Experiential/leaning by doing c. On-the-job d. Coaching/Mentoring e. Joint trainings (e.g., CAG with Government officials, men with women, youth with adults) MARLA FUND 13. How does your organization locate potential Marla Fund participants? 14. If you know victims of war that are not helped by the Marla Fund, what were the reasons they are not helped? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. They were not harmed by U.S. Coalition Forces (e.g., by terrorist attack, militia group)? b. They could not provide verification/documentation to prove eligibility? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 106 c. They did not want to accept support from the United States? d. They had all the support they needed? e. Marla Fund did not provide the support they needed? (please explain what was needed) 15. After the end of the CAP program, how will training of CAGs and Local Councils continue? 16. Do you think all the eligible war victims have been identified? If not, what is needed in order to identify all who are eligible for the Marla Program? How many more eligible victims do you think there might be? 17. Do you think the Marla Fund eligibility criteria should be changed in any way? a. If so, how would you recommend changing? b. What are the considerations of expanding it to include all victims of war? 18. After CAP ends, how do you expect the local NGO partner to continue to provide Marla Fund services? 19. What do you think are the most important lessons learned in supporting Marla Fund participants? 20. What are the outcomes of assistance to Marla beneficiaries at the individual and family level? At the community level? [Probe: livelihood/self-sufficiency through employment, good will toward US, stable housing, mental health] PROGRAM RESULTS 21. How well are CAGs now able to advocate on behalf of their communities to government officials? Can they be effective without the help of the CAP program? 22. Since the beginning of CAP III in 2008, how has CAP III built productive relationships between CAGs and local government leaders? 23. Since the beginning of CAP III in 2008, how has CAP contributed to improving local government’s responsiveness to community needs (e.g., understanding/meeting needs, mobilizing resources)? 24. Since the beginning of CAP III in 2008, how has CAP contributed to improvements in grassroots citizen engagement? 25. How do you see CAP III ’s efforts being sustained or continuing into the future? 26. If there were one change you would make to CAP III , what would it be? 27. What was CAP III ’s biggest achievement? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 107 3. CAP III Evaluation Field Interview Form – IP Community Mobilizer/Facilitator (Interviewer’s Last Name; Province, Mobilizer/Facilitator’s Name, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1 HOUR DATE OF INTERVIEW: NAME OF RESPONDENT: TITLE/POSITION: IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** BACKGROUND 1. What is the structure of your M&E office? a. How many staff members do you have? b. What is each of their responsibilities? 2. What lessons learned from CAP II were applied in CAP III in terms of performance monitoring and data collection (probe: PRS)? 3. What interaction did you have with the M&E staff of other IPs? SYSTEM 4. How are M&E findings used to improve program performance? 5. Is the PMP reflective of program activities? What revisions or changes would you suggest? 6. Was the theory of change for the program plausible? How would you adjust it? 7. What innovations or improvements were initiated over the course of the M&E system? INDICATORS 8. Of the PMP indicators, which were the best measures of program performance? Which were the worst? 9. What do you use as a baseline for your indicators from program start (2008), including: a. Community and/or CAG ability to advocate on behalf of community needs b. Local Government’s openness to community input in planning/budgeting process USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 108 c. Meeting the needs of war victims eligible for the Marla Fund 10. In addition to the PMP, what custom indictors did your organization develop to monitor and evaluate your CAP III program? How did these indicators change over the 4 years of program implementation? 11. What improvements to indicators would you recommend? DATA QUALITY AND COLLECTION 12. How do you collect information for indicators? What were the challenges? 13. What is the verification or quality assurance process for data collected from the field? 14. What are the quality concerns in terms of using your data in order to measure performance and also to compare performance over the four year period? 15. What are the barriers to aggregating data across the four implementing partners or comparing performance on the PMP indicators? OUTPUTS 16. What output indicators did you use in addition to the PMP? 17. At what point in the process of forming a CAG was it considered to be a CAG? 18. What proportion of your CAGs did not develop at least one project? At what point in the training process is a CAG considered to be trained? A CAG member? A Local Council? OUTCOMES 19. How did the program measure changes in government responsiveness to community needs? How could this have been improved? 20. How did the program measure changes in community’s ability to identify and advocate for its own needs? 21. To what extent were outcomes quantifiable? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 109 4. CAP III Evaluation Field Interview Form – CAP III PROGRAM MANAGERS (Interviewer’s Last Name; Province, Program Manager’s Name, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1 HOUR DATE OF INTERVIEW: NAME(S) OF RESPONDENT: TITLE/POSITION: IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. How long have you been working on CAP? 2. What are your main duties? CAG FORMATION AND SUPPORT 3. How are CAG members selected? a. Who is involved in the selection decision? b. How are candidates identified? c. What are the mechanisms for selecting – election, appointment, consensus? d. How are vulnerable groups included on the CAG (e.g., women, female heads of households, youth, ethnic/religious minorities, IDPs)? e. How long do they serve on the CAG? f. How are new members included in the CAG? 4. What are all the ways that CAP supports the CAGs? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Training? b. Coaching/Mentoring? c. Making connections with other people in the Iraq government? d. Making connections with other people in the community? Y o u r L a s t N a m e – P r o v i n c e – C A P I I I F u n d M g r - D a t e USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 110 e. Providing technical expertise on project design? f. Providing funding? 5. What were the challenges and/or obstacles to working with CAGs (e.g., recruiting, selecting, managing, developing advocacy capacity, training, decision making)? 6. How well are CAGs now able to advocate on behalf of their communities to government officials? Can they be effective without the help of the CAP program? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 111 5. CAP III Evaluation Field Interview Form – IP Community Mobilizer/Facilitator (Interviewer’s Last Name; Province, Mobilizer, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1 to 1-1/2 HOURS DATE OF INTERVIEW: NAME OF RESPONDENT: TITLE/POSITION: IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. How long have you been working on CAP? 2. What are your main duties? CAG FORMATION AND SUPPORT 3. How are CAG members selected? a. Who is involved in the selection decision? b. How are candidates identified? c. What are the mechanisms for selecting – election, appointment, consensus? d. How are vulnerable groups included on the CAG (e.g., women, female heads of households, youth, ethnic/religious minorities, IDPs) e. How long do they serve on the CAG? f. How are new members included in the CAG? 4. What are all the ways that CAP supports the CAGs? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Training? b. Coaching/mentoring? c. Making connections with other people in the Iraq government? d. Making connections with other people in the community? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 112 e. Providing technical expertise on project design? f. Providing funding? 5. What were the challenges and/or obstacles to working with CAGs (e.g., recruiting, selecting, managing, developing advocacy capacity, training, decision making)? 6. How well are CAGs now able to advocate on behalf of their communities to government officials? Can they be effective without the help of the CAP program? ENGAGEMENT OF GOVERNMENT OFFICIALS 7. How does the CAP program work to engage Local Council officials (qadaa, nahiya, neighborhood councils) in the CAP program? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Training? b. Community needs assessment? c. Mobilization of GoI resources? 8. What types of support provided by CAP is especially important for Local Council members? After the CAP program ends, who do you think can provide this support? 9. How does your program work to engage Provincial Council members in the CAP program? 10. How does your program work to engage line ministries and directors general in the CAP program? 11. How does your program work to engage parliamentarians in the CAP program? 12. How open are these government officials to receiving CAG input? PROMOTING COOPERATION BETWEEN COMMUNITY AND GOVERNMENT 13. Since the beginning of CAP III in 2008, how has CAP III built productive relationships between CAGs and local government leaders? 14. Since the beginning of CAP III in 2008, how has CAP contributed to improving local government’s responsiveness to community needs (e.g., understanding/meeting needs, mobilizing resources)? 15. What steps were taken to engage vulnerable groups in other CAP activities such as training or community needs assessment (women, female heads of households, youth, IDPs, ethnic/religious minorities)? What are the challenges of including these populations? CAP PROGRAM RESULTS 16. If there were one change you would make to CAP III , what would it be? 17. What was CAP III ’s biggest achievement? PROJECT SITE VISIT 18. Please provide a brief description of the project we will see today/tomorrow. 19. How was this project selected by the CAG? What community needs will it address and how urgent are those needs? Who are the main beneficiaries? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 113 20. What were the challenges in deciding to implement this project? 21. Who provides resource support (financial and in-kind) for this project? 22. How has the government (e.g., Local Council, Provincial Council, line ministries, mayor) been involved in this project (e.g., selection, implementation, providing or mobilizing resources, procurement process, advocacy, resistance)? 23. Who currently operates and makes repairs on the project? Who will continue to operate and make repairs on this project in the future? 24. What is the current responsibility of the CAG in relation to this project? 25. What was the greatest success or accomplishment of this project? 26. Anything else I should know about this project? MARLA FUND (if involved with Marla cases) 27. If you work with Marla Fund participants, what are your responsibilities? 28. How do you or your organization locate potential Marla Fund participants? 29. If you know victims of war that are not helped by the Marla Fund, what were the reasons they are not helped? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. They were not harmed by U.S. Coalition Forces (e.g., by terrorist attack, militia group). b. They could not provide verification/documentation to prove eligibility. c. They did not want to accept support from the United States. d. They had all the support they needed. 30. Marla Fund did not provide the support they needed? (please explain what was needed) Do you think all the eligible war victims have been identified? If not, what is needed in order to identify all who are eligible for the Marla Program? How many more eligible victims do you think there might be? 31. Do you think the Marla Fund eligibility criteria should be changed in any way? a. If so, how would you recommend changing? b. What are the considerations of expanding it to include all victims of war? 32. After CAP ends, how do you expect the local NGO partner to continue to provide Marla Fund services? 33. What do you think are the most important lessons learned in supporting Marla Fund participants? 34. What do you think are the most important accomplishments of the Marla Fund? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 114 6. CAP III Evaluation Field Interview Form – Engineers and Technical Support (Interviewer’s Last Name; Province, Engineer’s Name, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1 HOUR DATE OF INTERVIEW: NAME(S) OF RESPONDENT(S): TITLE/POSITION(S): IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. What are your responsibilities as they relate to the CAP III program? 2. What is the most important support that you provide to the CAGs? 3. If you work with government officials through the CAP program, what do you do? 4. What technical training needs do you think are especially important for CAG members so that they are able to develop and design appropriate projects? 5. What were the challenges and/or obstacles to providing technical support to CAGs through the CAP program? 6. After the CAP program ends in September, who do you think can provide this technical support in the future? CAP PROGRAM RESULTS 7. Do CAGs know how to get the expertise needed to help design projects? Can they do this without the CAP program’s assistance? 8. Do you think CAGs are able to advocate effectively on behalf of their communities? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 115 9. Since the beginning of CAP III in 2008, how has CAP III built productive relationships between CAGs and local government leaders? 10. Since the beginning of CAP III in 2008, how has CAP contributed to improving local government’s responsiveness with communities (e.g., understanding/meeting needs, mobilizing resources)? 11. If there were one change you would make to CAP III program, what would it be? This change can be regarding providing your technical expertise or any other part of the program. 12. What was CAP III’s biggest achievement? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 116 7. CAP III Evaluation Field Interview Form – CAG and Local Council Trainers (Interviewer’s Name; Province, CAG or LC Trainer Name, Date of Interview) DATE OF INTERVIEW: NAME(S) OF RESPONDENT(S): TITLE/POSITION(S): IMPLEMENTING PARTNER: Male: BEST CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program. The results of this evaluation will assist USAID to improve programs like CAP in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. How long have you been working as a CAP trainer? 2. What are your main duties and who do you train (e.g., CAG, Local Council members, others)? 3. How are CAG trainings organized? a. How are participants recruited? b. What are the training topics [Note: Ask for sample agendas for CAG trainings. They can email it to you if no hard copy is easily available.] c. How long do they last (e.g., hours, days)? d. What are the training goals? e. When is a CAG member or CAG considered to be “fully trained”? 4. How are local government trainings organized? a. How are participants invited to be in the training? b. What are the training topics [Note: Ask for sample agendas for local government (LG) trainings. They can email it to you if no hard copy is easily available.] c. How long do they last (e.g., hours, days)? d. What are the training goals? e. When is a local government official or council considered to be “fully trained”? TRAINING STRATEGIES AND LESSONS LEARNED 5. How does training promote partnerships between CAGs and Local Councils? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 117 6. What steps were taken to engage vulnerable groups in trainings (women, female heads of households, youth, IDPs, ethnic/religious minorities)? Did you need to adjust any part of the training to include vulnerable groups? 7. USAID is especially interested in learning about effective training strategies. What training formats did you use and which do you think work best in promoting learning in the Iraqi context? (Examples are below but you do not need to read unless the trainer does not have any ideas.) a. Classroom/lecture b. Experiential/leaning by doing c. On-the-job d. Coaching/mentoring e. Joint trainings (e.g., CAG with LG, men with women, youth with adults) 8. What were the challenges and/or obstacles to providing CAP training? 9. What training needs do you think are especially important for Local Council officials? After the CAP program ends in September, who do you think can provide this training? 10. What training is needed to help Iraqi citizens advocate on behalf of their community’s needs? After the CAP program ends, who do you think can provide this training? CAP PROGRAM RESULTS 11. After the training, do you think CAGs are able to advocate on behalf of their communities? 12. Since the beginning of CAP III in 2008, how has CAP III built relationships between CAGs and local government leaders? 13. Since the beginning of CAP III in 2008, how has CAP improved local government’s responsiveness with communities (e.g., understanding/meeting needs, mobilizing resources)? 14. What was CAP III ’s biggest achievement? MARLA FUND 15. If you trained Marla Fund participants, what were the training topics (e.g., job skill)? [Note: Please ask for sample agendas.] 16. What do you think are the most important lessons learned in providing useful training to Marla Fund participants? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 118 8. CAP III Evaluation Field Interview Form – MARLA FUND MANAGER ESTIMATED LENGTH OF INTERVIEW = 1 HOUR (Interviewer’s Name; Province, Marla Fund Manager’s Name, Today’s Date) DATE OF INTERVIEW: NAME OF RESPONDENT: TITLE/POSITION Male: IMPLEMENTING PARTNER: CONTACT INFO FOR FOLLOW-UP: Female: INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program, including administration of the Marla Fund. The results of this evaluation will assist USAID to improve this program. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. How long have you been the Marla Fund Manager? 2. What are your main job responsibilities? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Do you identify potential participants? b. Are you involved with verification of eligibility? c. Do you design Marla project to meet war victims’ needs? 3. How many Marla Fund participants is your organization currently serving? 4. What are the different kinds of Marla Fund projects that your organization provides? IDENTIFICATION OF MARLA FUND PARTCIPANTS 5. What are the responsibilities of the local NGO partner for the Marla Fund? 6. How do you / your organization/the local NGO partner locate potential Marla Fund participants? a. What are the challenges? b. What are the most effective strategies? 7. How do you/the local NGO determine eligibility for the Marla Fund? a. What percentage do you find are ineligible for the Marla Fund? b. What are the typical reasons for being ineligible? c. What are the challenges of the eligibility verification process? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 119 8. If you know victims of war that are not helped by the Marla Fund, what were the reasons they are not helped? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. They were not harmed by U.S. Coalition Forces (e.g., by terrorist attack, militia group). b. They could not provide verification/documentation to prove eligibility. c. They did not want to accept support from the United States. d. They had all the support they needed. e. Marla Fund did not provide the support they needed. (please explain what was needed) 9. Do you think all the eligible war victims have been identified? If not, what is needed in order to identify all who are eligible for the Marla Program? How many more eligible victims do you think there might be? ELIGIBILITY FOR MARLA FUND 10. Do you think the Marla Fund eligibility criteria should be changed in any way? a. If so, how would you recommend changing? b. What are the considerations of expanding it to include all victims of war? SERVICES PROVIDED BY MARLA FUND 11. Which of the projects you provide to Marla Fund participants are most important to the recipients? 12. What services are you unable to provide that you wish you could? MARLA FUND IMPACT ON VICTIMS 13. What do you think is the impact on victims of Marla services? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Physical healing? b. Psychological /trauma healing and support? c. Employment/livelihood stabilization? d. Property repair /restoration of home? e. Increased community-based capacity to respond to victims? 14. To what extent are victims aware that the support is paid for by the U.S. government? a. Does this awareness change the victim’s attitude toward the U.S.,, either improving or worsening it? 15. On average, what is the length of time that you provide support for a Marla Fund participant? What is the range from shortest and longest (approximately)? 16. After CAP ends, how do you expect the local NGO partner to continue to provide Marla Fund services? 17. What do you think are the most important lessons learned in supporting Marla Fund participants? 18. What do you think are the most important accomplishments of the Marla Fund? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 120 9. CAP III Evaluation Field Interview Form – MARLA Fund Local NGO Partners (Interviewer’s Last Name; Province, Marla Fund NGO, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 1 HOUR DATE OF INTERVIEW: NAME(S) OF RESPONDENT(S): TITLE/POSITION(S) ORGANIZATION: Male BEST CONTACT INFO FOR FOLLOW-UP: Female INTRO: Thank you for meeting with me as part of the USAID evaluation of the CAP III program, including administration of the Marla Fund. The results of this evaluation will assist USAID to improve this program. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your name or anything that can be used to identify you with anyone outside of the research team. A version of the report will be made public by USAID, but will not include names of those interviewed. Do you have any questions before we begin? ***** JOB RESPONSIBILITIES 1. When did your organization begin to work on the Marla Fund? 2. What does your organization do as it relates to the Marla Fund? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Do you identify potential participants? b. Are you involved with verification of eligibility? c. Do you design Marla project to meet war victims’ needs? d. Do you provide services to war victims (other than Marla Fund)? 3. What is your official job title? 4. How long have you been working on Marla Fund activities? 5. What are your main job responsibilities? ((Examples are listed below but you do not need to say them unless he/she is not sure how to answer the question.) a. Do you identify potential participants? b. Are you involved with verification of eligibility? c. Do you design Marla project to meet war victims’ needs? 6. How many Marla Fund participants is our organization currently serving? 7. What are the different kinds of Marla Fund projects that your organization provides? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 121 IDENTIFICATION OF MARLA FUND PARTCIPANTS 8. How do you / your organization / the local NGO partner locate potential Marla Fund participants? a. What are the challenges? b. What are the most effective strategies? 9. How do you / the local NGO determine eligibility for the Marla Fund? a. What percentage do you find are ineligible for the Marla Fund? b. What are the typical reasons for being ineligible? c. What are the challenges of the eligibility verification process? 10. If you know victims of war that are not helped by the Marla Fund, what were the reasons they are not helped? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. They were not harmed by U.S. Coalition Forces (e.g., by terrorist attack, militia group)? b. They could not provide verification/documentation to prove eligibility? c. They did not want to accept support from the United States? d. They had all the support they needed? e. Marla Fund did not provide the support they needed? (please explain what was needed) 11. Do you think all the eligible war victims have been identified? If not, what is needed in order to identify all who are eligible for the Marla Program? How many more eligible victims do you think there might be? ELIGIBILITY FOR MARLA FUND 12. Do you think the Marla Fund eligibility criteria should be changed in any way? a. If so, how would you recommend changing? b. What are the considerations of expanding it to include all victims of war? SERVICES PROVIDED BY MARLA FUND 13. Which of the projects you provide to Marla Fund participants are most important to the recipients? 14. What services are you unable to provide that you wish you could? MARLA FUND IMPACT ON VICTIMS 15. What do you think is the impact on victims of Marla services? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Physical healing? b. Psychological / trauma healing and support? c. Employment / livelihood stabilization? d. Property repair / restoration of home? e. Increased community-based capacity to respond to victims? 16. To what extent are victims aware that the support is paid for by the U.S. government? a. Does this awareness change the victim’s attitude toward the U.S., either improving or worsening it? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 122 17. On average, what is the length of time that you provide support for a Marla Fund participant? What is the range from shortest and longest (approximately)? 18. After CAP ends, how do you expect your organization to continue to provide Marla Fund services? 19. What support or training does your organization need in order to continue to provide these services? 20. What do you think are the most important lessons learned in supporting Marla Fund participants? 21. What do you think are the most important accomplishments of the Marla Fund? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 123 10. CAP III Evaluation Field Interview Form – Provincial Council and Line Ministries (Interviewer’s Last Name; Province, Provincial Council or Line Ministry, Date of Interview) ESTIMATED LENGTH OF INTERVIEW = 15 MINUTES DATE OF INTERVIEW: NAME(S) OF RESPONDENT: TITLE/POSITION(S) COUNCIL OR MINISTRY: Male BEST CONTACT INFO FOR FOLLOW-UP: Female INTRODUCTION: Thank you for meeting with me as part of the evaluation of the CAP III program. The results of the evaluation will help to improve these kinds of programs in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your names or anything that can be used to identify you. A version of this report will be made public but will not include names of those we have interviewed. Do you have any questions before we begin? ***** CAP PROGRAM 1. How did you first hear of the CAP program? For example, from the implementing partner, a co-worker, the media, CAG Member, or friends and family. 2. In what ways have you worked with the CAP program or a Community Action Group (CAG)? (Examples are below but you do not need to read these unless he/she does not know how to respond.) a. Authorized CAP project? b. Arranged in-kind contribution? c. Arranged financial contribution? d. Participated in training? e. Met with CAP or CAG? f. Introduced CAP or CAG to other government officials? 3. How do you usually understand the community needs of qadaas and nahiyaas where you do not live? How has the CAP program helped you to understand better those community needs? 4. How has the CAP program changed the way you work with communities? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 124 5. How has the CAP program changed the way you work with Local Council officials (qadaa, nahiyaa, neighborhood)? COMMUNITY ENGAGEMENT 6. How do community groups usually communicate with your office in order to propose a project or explain a need, such as a school, road, bridge, electricity, or drinking water? (e.g., public meetings, private meetings, letter, email, phone call, newspaper, complaint boxes) 7. Can these communities typically access your office without the help of CAP or an international partner like CAP? 8. What is the most effective way for communities to propose community projects as part of the budgeting and planning processes? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 125 11. CAP III Evaluation Focus Group Protocol – CAG Members (Interviewer’s Last Name; Province, Project Code/CAG Focus Group, Date of Focus Group) Note to the Researcher: Please use the specific terminology appropriate for your IP and province. For example instead of asking about the “implementing partner,” ask about “Mercy Corp”; instead of “CAG,” ask about the specific name used for this term in that area; instead of “CAP III, ” ask about the name of the program used, such as the name of the IP. INTRODUCTION: Thank you for meeting with me as part of the evaluation of the CAP III program. The results of the evaluation will help to improve these kinds of programs in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your names or anything that can be used to identify you. A version of this report will be made public but will not include names of those we have interviewed. STRUCTURE: As part of this focus group, I will ask you several questions about the program. It is important to give everyone an opportunity to speak about his or her experience. Do you have any questions before we begin? FOCUS GROUP QUESTIONS: 1. Please introduce yourself one-by-one by saying your name and how many years you have been on the CAG and your work outside of the CAP program. Name________________ (M or F) Number of years______ Job__________________ Name________________ (M or F) Number of years______ Job__________________ Name________________ (M or F) Number of years______ Job__________________ Name________________(M or F) Number of years______ Job__________________ Name________________(M or F) Number of years______ Job__________________ 2. How were you chosen to be on the CAG? 3. How did the process to choose CAG members encourage people like women, young people (youth) and vulnerable groups like IDPs, female heads of households, and ethnic or religious minorities, to serve on the CAG? 4. How did the CAG decide what were the community needs? 5. How did the CAG prioritize which projects you would implement? 6. How did you get the resources you needed (i.e., funding, labor, land) for this project from the community? 7. How did you get the resources you needed (i.e., funding, labor, land) for this project from the government (e.g., Local Council, Provincial Council, DG)? 8. Who was responsible for supervising the implementation of this project? 9. Who is responsible for operating and repairing this project? 10. How do people in the community usually contact their government leaders to let them know what they need and how successful are they in getting their needs met from the government? And has this process gotten easier or harder in the past four years? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 126 11. How has your involvement with this program changed your relationship with government leadership (e.g., Local Council, Provincial Council, DG)? For example, are you more or less likely to go to the Local Council for help? Are you more or less confident that the government will respond to your community’s needs? 12. What part of the training provided by the implementing partner was most helpful to you? 13. What additional training would you like to receive? 14. After the CAP program ends in September, what will happen to the CAG and its work representing the needs of the community? Do you think you can continue to do your work without the involvement of the implementing partner? 15. What was the hardest part of your work on the CAG? 16. What do you consider your biggest success? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 127 12. Survey and Focus Group Protocol – Local Government Officials (Interviewer’s Last Name; Province, Project Code, Local Council Focus Group, Date of Focus Group) Note to the Researcher: Please use the specific terminology appropriate for your IP and province. For example instead of asking about the “implementing partner,” ask about “Mercy Corp”; instead of “CAG,” ask about the specific name used for this term in that area; instead of “CAP III, ” ask about the name of the program used, such as the name of the IP. INTRODUCTION: Thank you for meeting with me as part of the evaluation of the CAP III program. The results of the evaluation will help to improve these kinds of programs in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your names or anything that can be used to identify you. A version of this report will be made public but will not include names of those we have interviewed. STRUCTURE: As part of this focus group, I will ask you several questions about the program. It is important to give everyone an opportunity to speak about his or her experience. Do you have any questions before we begin? CAP PROGRAM OVERALL 1. Please introduce yourself one-by-one by saying your name and how many years you have been on the Local Council. Name______________________________________(M or F) Number of years______ Name______________________________________(M or F) Number of years______ Name______________________________________(M or F) Number of years______ Name______________________________________(M or F) Number of years______ Name______________________________________(M or F) Number of years______ 2. How were you involved in the selection and design of this project for the community? 3. From your perspective, did the CAG involve all parts of the community in developing this project, including women, youth, religious/ethnic minorities, IDPs? 4. How did the project get the resources needed to create the project (i.e., funding, labor, land)? 5. Does this project meet an important need of the community? 6. Who was responsible for supervising the implementation of this project? 7. Who is responsible for operating and repairing this project? 8. How do you understand the needs of the people in the community that you serve? And how do you decide which are the most important? Does the CAG help you to understand these needs? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 128 9. How do people in the community usually contact you and other government leaders to let them know what projects they need? How successful are they in getting their needs met from the government? And has this process gotten easier or harder in the past four years? 10. How did your involvement in the CAP program change the way you advocate on behalf of your community’s needs to the ministries, Provincial Councils and Parliament? 11. What part of the training provided by the Implementing Partner was most helpful to you? And what additional training would you like to receive? 12. After the CAP program ends in September, who will continue to support Local Council members for training, understanding the needs of the community or implementing community projects? 13. What was the hardest part of your work in developing this project? 14. What do you consider your biggest success through your work on this project or the CAP program? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 129 13. Surveys or Focus Group Protocols – CAP III Beneficiaries (Interviewer’s Last name, Province, Project Code, CAP Beneficiary Focus Group, Date of Focus Group) INTRODUCTION FOR SURVEY: Hello. I am part of a research team that is talking to people about this [name of project]. Would you be willing to take a survey about your experience with [this project}? It will take about 15 minutes. Everything you tell me is confidential so I do not put your name on the form. OR INTRODUCTION FOR FOCUS GROUP: Thank you for meeting with me to talk about your experience with [name of project]. The results of our research will help to improve these kinds of programs in the future. CONFIDENTIALITY: Everything we discuss is considered to be confidential. What this means is that we do not share your names or anything that can be used to identify you. A version of this report will be made public but will not include names of those we have interviewed. STRUCTURE: As part of this focus group, I will ask you about 10 questions about the program. It is important to give everyone an opportunity to speak about his or her experience. Do you have any questions before we begin? NOTE TO RESEARCHER: How many men?_____ How many women?____ FOCUS GROUP QUESTIONS 1. How many of you were part of the decision-making process to implement [name of project]? What were the number of people involved? 2. Of those of you who were involved in the decision making, how did the process work and who else was involved? 3. Do you think the way the decision was made should be changed in any way? If so, in what ways? 4. How important is this project to you and to your community? 5. Did any of you help to implement this project? If so, what did you do to help? What was the number of people who helped implement? 6. If your community needs important projects—such as water, sanitation, roads, bridges, schools, electricity—where do you go for help? (e.g., Local Council, Provincial Council, COR member, DG, tribal leader, religious leader, muhktar, etc.) 7. How do you let these people know what your community needs (e.g., call them, go to a public meeting, make a private meeting, sign a petition, submit a report with data about what is needed, etc.)? 8. Has this process of letting government leaders know what your community needs become easier or harder in the past four years? 9. Who do you think should be responsible for taking care of the operations and repairs for this project? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 130 10. How were your Local Council, line ministry, or Provincial Council members involved in creating this project? 11. How satisfied are you with this project? Do you think anything about the project needs to be changed? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 131 14. Surveys and Focus Group Protocols – Marla Beneficiaries (Interviewer’s Last Name, Province, Marla Beneficiary’s Name, Date of Focus Group) Note to the Researcher: Please use the specific terminology appropriate for your IP and province. For example instead of asking about the “implementing partner,” ask about “Mercy Corp”; instead of “CAG,” ask about the specific name used for this term in that area; instead of “CAP III, ” ask about the name of the program used, such as the name of the IP. INTRODUCTION: Thank you for meeting with me as part of the evaluation of the [CAP III] program. The results of the evaluation will help to improve the program. CONFIDENTIALITY: Everything you write and we discuss is considered to be confidential. What this means is that we do not share your names or anything that can be used to identify you. A version of this report will be made public, but will not include names of those we have interviewed. SURVEY: I am going to give you a questionnaire first. Once everyone has completed that form we will discuss the program as a group. The survey should take about 15 minutes. If you have any questions about the survey, please let me know and I will be happy to help you. [Pass out the survey to the respondents.] FOCUS GROUP QUESTIONS Thank you for completing the survey. Now, we will talk about the Marla Fund program. 1. Is the support you receive through the Marla program continuing or has it ended? (Please write the number of people for each option on the line below) ______support continuing ______support ended ______don’t know 2. What kind of help do you need from the Marla program now? 3. Have you received your help from the implementing partner or from the local Iraqi NGO? ______Implementing partner ______Local NGO ______other (please explain) 4. Do you think all the victims of the U.S. coalition forces who need help are receiving support through the Marla Fund? ______yes ______no ______don’t know 5. How many more victims that aren’t receiving help do you think there are in this province? 6. What is the best way to find or identify other people who were hurt by the U.S. coalition forces so that they can also receive help from the Marla program? 7. How helpful was the staff in helping you to receive support through the Marla Program? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 132 [Note to researcher: Be sure you understand if they are referring to IP staff or the local NGO staff.] 8. Did you receive the support you expected to receive from the Marla Program? 9. How can the support you have received through the Marla program be improved? 10. If there were no Marla Program, where would you go for support? 11. In the future, what support do you think you will need to help you recover from the harm caused by the U.S. Coalition? USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 133 15. IP Mobilizer/Facilitator Questionnaire BACKGROUND 1. What is your name? 2. Are you a man or woman? Multiple choice (choose one): Man, Woman 3. Which organization do you work for? Multiple choice (pick one): ACDI/VOCA, CHF, Mercy Corps, IRD 4. Which provinces do you cover? Multiple choice (pick all that apply): Anbar, Babil, Baghdad, Basrah, Dhi-Qar, Diyala, Karbala, Kirkuk, Maysan, Muthanna, Najaf, Ninawa, Qadissiyah, Salah ad Din, Wasit 5. When did you begin your job as a CAP III community mobilizer/facilitator/developer? CAG ACTIVITIES 6. When beginning the process to form a CAG, who in the community and/or province should your organization contact first in order to be most successful? Multiple choice (pick one): Local Council Provincial Council Ministry/DG Governor Mayor Tribal leader/sheikh/muhktar Your own personal social network Religious leader NGO/civil society organization Participants from other CAGs IP staff Other 7. What are obstacles to forming a CAG? Multiple choice (choose all that apply): Insufficient community support for forming a CAG No ability to include vulnerable groups on the CAG (such as women, youth, ethnic/religious minorities) Lack of political support from Local Council Lack of political support from Provincial Council Lack of political support from Parliament Lack of adequate funding from USAID for projects USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 134 Lack of adequate funding from ministries or Provincial Council for projects Lack of community participation in the CAG selection process Inability for community to understand the CAP program Other (please explain) 8. Which strategies were most important in building relationships between the CAG and government officials? Multiple choice (choose all that apply): Include well-known and respected members of the community on the CAG Include government officials on the CAG Implementing partner assistance in making initial introductions Joint efforts on a project that benefits the community Participating in a joint skill-building training Building on established personal relationships Attending conferences together Other (please explain) 9. Which types of CAP III projects generated the most community support and/or involvement? Multiple choice (choose one): Water Electricity Athletic facilities Public parks Job training Sanitation Schools/educational materials Health facilities Roads Other (please explain) 10. What reasons best explain why a CAG successfully implements a project? Multiple choice (choose all that apply): Funding from CAP program (USAID funding) Good relations between the CAG and Local Council Good relations between CAG and Provincial Council Good relations between CAG and line ministries, Good relation between the implementing partner (your organization) and government officials USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 135 Training of CAG members in important skills Strong level of community support for project Urgent need for project in the community Strong motivation by individual community or Local Council leaders Other (please explain) TRAINING 11. What training do you think is most important for community leaders to receive in order to advocate on behalf of their communities? Multiple choice (choose one): Conducting needs assessments Prioritizing community needs Designing projects Participatory decision-making processes Advocacy with government officials National and/or provincial budgeting process Working with vulnerable groups Project management NGO management Iraqi government structure and function Accounting Other (please explain) 12. What skill-building training approach/format is most effective for Iraqi participants? Multiple choice (choose one): Classroom lecture On-the-job training in which people are trained in their own offices Coaching/advising Experiential training (learning by doing in which people practice a skill by conducting an actual activity) Other (please explain) 13. What combination of participants work well for training: Multiple choice (choose all that apply): Joint training with both government and non-government participants – 5 Separate trainings with participants who are all government participants or who are all non-government participants training – LC on how to advocate projects for PCs, specialized skills Men and women trained together – 3 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 136 Men and women trained separately – 3 tribal community, village, only chance for women, vote for project Intergenerational training (i.e., both youth and older adults) – 3, old ones believe in young people’s ability All-youth training – 2 young people are more motivated, computer course, may be different skill levels Multiple levels of government officials together (e.g., Provincial Council, Local Council, ministries, Parliament, governor) in order to understand each other’s roles, want more training to understand each relevant, relationship building IDP-only IDP with non-IDP participants – if trained separately, feel isolated Ethnic/sectarian minorities only – never separate, no reason, reemphasize separating Joint ethnic/sectarian groups with majority groups Other (please explain) 14. How should training formats be changed for vulnerable groups (e.g., female heads of household, youth, IDPs, ethnic/sectarian minority groups)? [Open-ended] LOCAL COUNCILS 15. What is the best strategy from CAP III for Local Councils to gather public input on community needs? Multiple choice (choose one): Open public meeting for whole community organized by community leaders Local Council meeting open to public Provincial Council meeting open to public Public opinion poll or survey Individual meetings organized between Local Council members and community advocates Other (please explain) 16. How has CAP III contributed most to changes in how Local Councils see their role in addressing community needs? Multiple choice (choose one): Provided training useful to their job Built relationships between Local Councils and the community (CAGs) Built relationships between Local Councils and other government officials (Provincial Council, Parliament, line ministries, governors), USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 137 Helped Local Council members understand their role and accountability to their community Helped Local Councils better understand the needs of their communities Helped Local Councils understand how to implement projects that serve their communities Other (please explain) 17. After the CAP III program finishes, how likely do you think it will be for Local Councils to advocate on behalf of community needs? Multiple choice (pick one): Very likely Somewhat likely Somewhat unlikely Very unlikely Don’t know 18. After the CAP program finishes, how likely do you think it will be for Local Councils to mobilize resources on behalf of community needs? Multiple choice (pick one): Very likely Somewhat likely, Somewhat unlikely Very unlikely Don’t know 19. In addition to the Marla Ruzicka Iraqi War Victims Fund, who within Iraqi society is the most important provider of support to victims of war and their families? [Multiple choice (choose one)]: Tribes/family networks Religious institutions Iraqi NGOs/CSOs International NGOs International governments Iraqi Government Other (please explain) 20. What are the reasons war victims do not access the Marla Fund? Multiple choice (choose all that apply): USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 138 Victims were not harmed by U.S. Coalition Forces (e.g., they were harmed instead by terrorist attack, militia group) Victims could not provide verification/documentation to prove eligibility Victims did not want to accept support from the United States Victims had all the support they needed Victims had not heard of the Marla Fund Marla Fund did not provide the support they needed Other (please explain) SUSTAINABILITY 21. How often do Iraqi communities go to the following people for help in meeting their communities’ needs: Many Times A Few Times Never Local Council: qadaa, nahiya, etc. Provincial Council Governor Line ministry Member of Parliament Tribal leader, sheikh, mukhtar Family or friend Iraqi CSO or NGO International NGO Other (please explain) 22. After the CAP program finishes, how likely do you think it will be for most of your CAGs to continue to advocate on behalf of community needs? Multiple choice (pick one): Very likely Somewhat likely Somewhat unlikely Very unlikely Don’t know 23. What percent of your CAGs do you think could successfully develop a project without the CAP program support? Multiple choice (pick one): None less than 25% 25-50% 50-75% USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 139 More than 75% All 24. What is the most important contribution to communities of the CAP program? Multiple choice (pick one): Development projects provided through CAP III (e.g., water, electricity, schools, parks) Developing community-based leaders to advocate on behalf of community needs Developing Local Council’s capacity to advocate for community needs Support to Iraqi war victims through the Marla Fund Other (please explain) 25. How has CAP III contributed to building formal processes and institutions that Iraqis use to advocate on behalf of community needs? [open ended] 26. As a result of your involvement in CAP III , do you feel more able to advocate on behalf of your own community? Multiple choice (choose one): Yes No No change USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 140 16. USAID/Iraq’s Community Action Program (CAP III) End of Project Performance Evaluation CAP Beneficiaries Questionnaire Background Questions – to be completed by CAP III Evaluator 1. Governorate ________________________________ 2. Name of IP service provider (circle one) ACDI/VOCA…………….………………..1 CHF………………………………………….2 IRD…………………………………………..3 Mercy Corps…………………………….4 3. Qadaa Council Name_________________________ 4. Nahiyaa Council Name ________________________________ 5. Neighborhood Council Name ________________________________ 6. Visited project’s Name or Project Code ________________________________ 7. Sex of the respondent (please check one): Male Female Questions for Beneficiary 8. How did you first hear about the CAG that organized this project? Please circle one: Relative or friend participates in CAG Saw public announcement about CAG Attended event sponsored by CAG Heard about CAG through other means Have not heard of CAG 9. Did you help in selecting this project? Please circle one: YES NO 10. Did you help in designing the project? Please circle one: YES NO USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 141 11. What contribution did you make to help implement this project? Please circle ALL that apply: I gave my labor I gave money I gave equipment I gave my land I made another contribution (please describe):_______________ I did not make a contribution 12. Do you help to operate or repair the project? Please circle one: YES NO 13. What is your opinion of the quality of the project? Please circle one: Excellent Good Satisfactory Mediocre Poor 14. What is your opinion of the importance to the community of the project? Please circle one: Very important Somewhat important Not very important Not important 15. Does this project help any vulnerable members of your community? Please circle ALL that apply: Female heads of households / widows Youth Internally displaced persons (IDPs) Ethnic or religious minorities Persons with disabilities None of the above 16. How often have I gone to these people for help in meeting a community need? Many times A few times Never CAG Member Local Council: qadaa, nahiya, neighborhood Provincial Council Governor Line Ministry/Director General Parliamentarian USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 142 Tribal leader/Sheikh A member of my family or a friend Iraqi CSO or NGO International donor or NGO Other (please explain): 17. In the past four years, has it become harder or easier to let your Local Council know what your community needs? Please circle one: Harder Easier Stayed the same Don’t know 18. How satisfied are you with the level of support you receive from your Local Council? Please circle one: Very satisfied Satisfied Not Satisfied USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 143 17. Marla Beneficiaries Questionnaire Ba Background Questions 1. City/Village and Governorate of Residence ____________________________ 2. Name of IP service provider: ACDI/VOCA……………………………..1 CHF………………………………………2 IRD……………………………………….3 Mercy Corps……………………………..4 3. Sex of the respondent (please check one): Male Female Case Specifics 4. When did you first go to the Marla Fund program to get help? Please circle one: 2003 2004 2005 2006 2007 2008 2009 2010 5. How did you first hear about support available from the Marla program? Please circle one: US government representative/PRT/US military Iraqi government official Member of the CAG Friend or Family Other (please explain): USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 144 6. What support did you receive through the Marla Fund program? Please circle all that apply: A medical device, prosthesis or treatment for myself. A medical device, prosthesis or treatment for my family member. Participated in a work training program Received tools / supplies for my work Repair of property damage to my home Repair of property damage to my business Other (please explain): 7. Are you continuing to receive support or has the support from the Marla Program ended? Please circle one: Continuing to receive support Support has ended Don’t know Sustainability 8. Where would you go to replace the support you are receiving if the program stopped providing services? Please circle one: Local Council (qadaa, nahiyaa) Ministry/Director General Religious organization Iraqi NGO or CSO Family International organization I am no longer receiving support Don’t know USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 145 9. What is your opinion of support provided by the Marla Fund program? Please circle one: Extremely useful Useful Not at all useful 10. If you know victims of war that were not helped by the Marla Fund, what were the reasons they were not helped? Please circle all that apply: I do not know any war victims that were not helped They were not harmed by the U.S. Coalition Forces so were not eligible They could not provide verification documentation They did not want to accept the Marla Fund They did not know about the Marla Fund Other reason (please explain): 11. To what extent do the services provided by the Marla program meet your needs? Please circle one: It meets all my needs It meets most of my needs It meets some of my needs It meets very little of my needs It meets none of my needs 12. Did support from the program change your opinion of the United States? Please circle one: Yes, significantly improved Yes, improved a little Yes, worsened a little Yes, significantly worsened No, unchanged 13. In five years, which option best describes your expectations of the Marla Fund program related to your case? Please circle one: I will receive better services and improved care I will receive slightly better services and care than I receive now I will receive the same level of services and care will be the same as it is now I will receive fewer services and care than I receive now I will not receive any services and care USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 146 18. CAP III Evaluation / Field Visit Report Form Evaluator Name: Date of visit: Province: District: Sub-District /Village: Activity Name: Project Identification : Implementer ACDI/VOCA CHF IRD Mercy Corps Dates Project start date (m/yr) Project end date (m/yr) if completed Project Type Education Water/Sanitation Electricity Transportation Economic growth/Livelihoods Health Governance Other Location Urban Describe any noteworthy features of the location (ex: takes several hours to reach from nearest city, located adjacent to the Local Council, etc.): Peri Urban Rural Condition: Describe any important aspects of its condition (ex: the lights have burned out, the equipment is in boxes, everything is covered in dust ) Excellent Poor Good Awful Mediocre Relevance Describe in what ways this responds to the community’s need (ex: The community selected a school, but this only serves the members of the CAG who had school age children. The farmers nearby the school have a desperate need for improved irrigation, which was evident by their barren fields. They said that they had never heard of the CAG, and wish that their views about the community’s need had been heard.): Use Breadth Frequency Describe important aspects of its use (ex: it is used daily for trainings of women from inside and outside the community; the rooms repaired by the program are used for a fourth grade class of 20 students) Everyone in the community Always/Very Frequently Most in the community Often Some/Many in the community Sometimes Few in the community Rarely Almost none/None in the community Almost never/Never Budget ICAP Contribution Government Contribution Community Contribution Other Comments Discuss anything else that deserves mention (ex: The project is directly adjacent to another program funded by LGP; The activity involves two parts, the second part is not visible; It is unlikely that the project will be maintained as well in two months as the guard is leaving and a new one has not been hired): Maintenance Describe any issues its maintenance (e.g., The area is maintained by a local guard who visits daily; the ministry visits once a month, which is not enough) USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 147 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 148 ANNEX H. LIST OF VISITED PROJECTS Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution ACDI/VOCA Construct a soccer field and service building Diyala Baquba Baquba 1 Youth Activities Completed 13-Oct-11 $69,080 $0 $2,541,960 Support the 2011-2012 Apprenticeship Program for Youth (Phase VI), Diyala South Diyala Diyala Multi District Economic Growth/ Livelihoods Completed 25-Apr-12 $95,660 $1,080 $7,560 Construct 6 classrooms at mixed primary school Kirkuk Dibis Mesherfa Village Education Active 30-Apr-12 $98,796 $0 $52,000 Construct WC building in Brayati Park and Nawzing Park Kirkuk Dibis Public Space Completed 14-Nov-11 $32,034 $0 $15,000 Establish soccer stadium Kirkuk Kirkuk K4 Youth Activities Completed 28-Jan-10 $99,400 $0 $1,000,000 Construct Laboratory Unit in Ba'asheqa Veterinary Dispensary Ninawa Mosul Ba'asheqa Subdistrict Economic Growth/ Livelihoods Active n/a $91,258 $0 $52,500 Promote public hygiene in 3 Quarters Ninawa Mosul Ba`asheqa Subdistrict Health Completed 27-Oct-11 $9,005 $0 $12,900 Shikhan Hospital Blood Bank Construction Ninawa Shikhan Health Completed 10-Mar-10 $85,495 $0 $40,220 Install water pump in Al￾Gubba Water Station Ninawa Talkef Water Completed 17-Oct-10 $67,080 $0 $80,000 Establish soccer field Ninawa Telafar Zummar Subdistrict, Bardiya Collective Town Youth Activities Active n/a $99,654 $0 $150,000 Construct a Nursing High School Salah ad Din Balad Education Completed 12-Dec-10 $98,658 $0 $246,400 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 149 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Improve and supply neighborhood parks with children's playground equipment Salah ad Din Tikrit Ma’at Dar and Sitteen Dar Neighborhoods Public Space Completed 14-Nov-11 $28,683 $0 $86,440 Upgrade electricity grid in 4 neighborhoods Salah ad Din Tikrit Tikrit City Electricity Active n/a $78,412 $0 $31,800 Improve electricity network Salah ad Din Tuz Kurmatu Al Sroor Neighbourhood Electricity Completed 30-Jun-10 $88,966 $0 $39,250 Construct Physiotherapy Unit Salah ad Din Tuz Kurmatu Health Completed 11-Feb-10 $76,715 $0 $24,000 CHF Improve electricity network by providing 2 transformers (250 KVA) , poles and wires Anbar Falluja Al-Takiya Village Electricity Completed 26-Jan-12 $66,681 $0 $0 Provide materials and equipment for vocational training on sewing and embroidery arts for women Anbar Falluja Al-Armeel Hay Economic Growth/ Livelihoods Completed 20-Mar-12 $25,115 $0 $0 Construct annex of 3 classes and rehabilitation of bathrooms for Al-Missabeih primary school Babil Al Mahawil Al-Nile subdistrict Education Completed 17-Aug-11 $61,400 $2,682 * Enhance electricity network through provision of transformers and electric material Babil Hashimiya Al-Mazedia Village Electricity Completed 8-Sep-10 $50,301 $417 $10,060 Cover Al-Hadadeen Market with wavy plates and lighting Babil Hilla Economic Growth/ Livelihoods Completed 14-Sep-11 $61,200 $4,196 * Provide 1 transformer with electrical materials Karbala Al Hindiya Al-Reyhth Hay Electricity Completed 1-Nov-10 $47,890 $2,483 $12,660 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 150 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Provide Al-Rawdthateen Sport Club with sport equipment Karbala Al Hindiya Youth Activities Completed 23-Aug-09 $60,650 $2,332 $1,474,915 Construct garden with playground Karbala K-III Al Husseiniyya, Al-Rasool Hay Public Space Completed 11-Mar-12 $48,500 $1,502 * Provide poles & wires to construct electricity network Karbala Karbala Al Hur, Sewadat Al-Bazel Al￾Kabeer Village Electricity Approved n/a $0 $0 $0 Provide 9 schools with furniture and some needs Karbala Karbala Al-Mulhaq al￾Askan Cluster Education Completed 25-Oct-10 $44,680 $95 $105 Provide 3 transformers with all equipment to Al-Joban Al￾Janubi electricity network Najaf Kufa Abbasiyya Electricity Completed 24-Oct-10 $69,000 $410 $840 Provide electrial equipment to upgrade the electrical network Najaf Kufa Al-Ramahi Village Electricity Completed 28-Nov-11 $43,356 $1,176 * Provide Hay Al-Askary clinic with medical equipment Najaf Menathera Al-Askary Hay Health Completed 13-Oct-10 $37,472 $1,042 $1,052,114 Construct annex for 3 classrooms in Al-Shomos primary school Najaf Najaf Al-Haidariya Education Active n/a $64,725,00 $216 * Rehabilitate and supply 4 schools (Abi Thar - Awrras - Al￾Shareef Al-Radi - Al-Risala) Najaf Najaf Hay Al-Zahra Education Completed 14-Oct-10 $76,925 $282 $10,715 Construct 1 park Qadissiyah Diwaniya AL-Sedr AL￾Thaniyah Hay Public Space Completed 18-Feb-10 $96,845 $674,160 * Provide water pipes to Alta`an Cluster Qadissiyah Diwaniya Daghara Water Completed 22-Mar-12 $75,925 $215,25 * USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 151 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Rehabilitate bathrooms of Al￾Ajiyal Al-Ola Nursery and provision of materials for the nursery and elderly care centers Qadissiyah Diwaniya Health Active n/a $53,325 $436 $30,00 Provide computers, furniture, water coolers for 10 youth centers Qadissiyah Diwaniya Youth Activities Completed 11-Mar-10 $95,650 * * Provide electrical equipment to improve and expand the electricity network Qadissiyah Hamza Mossa Nabhar Village Electricity Completed 25-Mar-12 $46,970 $104 * Provision of toys for Al￾Zanbaq & Al-Baraa kindergartens Wasit Al Aziziya Hafriya, AL Mazraa Education Completed 23-Sep-10 $22,500 $609 $13,560 Provision of toys, benches and iron fence for 2 parks Wasit Numaniya Al Askry Hay Public Space Completed 14-Oct-10 $81,150 $743,500 $2,773,440 Provision of twisted cable & aluminum wire to install electricity network Wasit Kut Al- Wafedeen Hay Electricity Completed 22-May-11 $96,000 $1,834 $599,084 Provision of 2 forums with furniture and athletic equipment for persons with disabilities Wasit Kut Other: Disabled Completed 15-Sep-09 $88,360 $415 $53,700 Provision of water pump and a transformer Wasit Sewera Shuhaymiya Water Completed 15-Sep-09 $68,740 $143,145 $3,550 IRD Supply and laying of 1,860 M UPVC water pipe Baghdad Adhamiya Al-Huzyran Village Water Active 22-Mar-12 $47,598 $30,000 $160,000 Provide water filter for Health Awareness Campaign for 90 IDP Women Baghdad Karada Health Completed 21-Aug-11 $2,097 $2,700 $0 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 152 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Construct multi-purpose hall, add new bathrooms, rehabilitate old bathrooms and pave yard of al-Qairawan Primary School Baghdad Karada Education Active 30-May-12 $87,642 $510 $115,700 Construct 4 addditional rooms in Al-Ameen Health Center Baghdad Nissan 9 Health Completed 6-Feb-11 $40,804 $200 $68,640 Provide sport supplies for football league Baghdad Rusafa Al-Mustansirya Neighborhood Youth Activities Completed 26-Dec-11 $14,640 $3,300 $1,200 Mercy Corps Construct Al-Hindyia Family Park Basrah Basrah City Center Al Maqa’al Public Space Active n/a $71,610 $500 $312,600 Women’s Inclusion and Awareness – (4 centers) (literacy) Basrah Basrah City Center Basra Women Inclusion and Awareness CAG Education Active n/a $32,805 $0 $9,600 Lay and connect water piping network with main pipe line network Basrah Abu Al Khaseeb Siba subdistrict, Al-Rabdha Village Water Completed 24-Apr-12 $26,900 $3,000 $3,750 Pave 2 km of rural road Dhi Qar Al Shatra AL-Garaf subdistrict, AL￾Dawood Village Roads Completed 6-Sep-09 $80,000 $6,600 $600 Construct and supply student dormitory for women in Administration & Economics basic education colleges Dhi Qar AL-Refayee Education Active n/a $98,205 $10,480 $45,210 Construct and supply 4 classrooms annex, Barada primary school for boys Dhi Qar Nassiryah AL-Bat-haa district, Al￾Mostashfaa neighborhood Education Active n/a $94,950 $2,780 $7,750 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 153 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Construct water plant (Reverse Osmosis) with capacity 2m3/hr Dhi Qar Nassiryah Al-Sedenawiya subdistrict, Al￾Shamamra village Water Completed 6-Sep-09 $70,500 $1,040 $3,180 Extend electricity network Dhi Qar Souk Al Shyuok Al Fadliyah subdistrict, Al Nasir Village Electricity Completed 22-Nov-11 $89,036 $2,460 $38,331 Establish IT Lab at Al-Mejar Al￾Kaber Secondary School for Boys Maysan Al-Majar Education Completed 21-Feb-10 $37,969 $0 $1,000 Outfit Theater Hall of Al￾Jumhoriya Primary School for Boys Maysan Al-Majar Education Completed 23-Mar-11 $48,685 $0 $500 Upgrade Ali Al-Sharqi Sports Facility Maysan Ali Al￾Sharqi Hay Al-Askary Youth Activities Completed 18-Mar-10 $97,559 $0 $4,938 Construct Al Ufia Kindergarten Maysan Amarah Education Completed 7-Jun-10 $332,205 $0 $197,000 Upgrade Qalat Salih Sports Club Facility Maysan Qalat Saleh Youth Activities Completed 1-Apr-10 $97,220 $13,000 $0 Construct and supply 4 classrooms, Shaheed Al￾Mihrab coed primary school Muthanna Al-Khidir Education Completed 11-Oct-10 $94,108 $780 $4,540 Pave4.1 km of rural road (Al￾Azreq road) Muthanna Rumaitha Al-Warka'a: Al￾Hajarah, Al-Siad Gatah, Al-Siad Ghazi villages Roads Completed 22-Jan-10 $93,685 $1,320 $2,690 Construct football field Muthanna Rumaitha Al-Hilal subdistrict Youth Activities Completed 9-Aug-09 $78,928 $540 $47,020 Construct fence, Jewad Al￾Emma primary school Muthanna Samawah Al-Sewer sub￾district, Al￾Heleyil Village Education Completed 28-Nov-10 $58,800 $540 $300 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 154 Activity Name Province District/ City Sub District/ Nahiya Project Type Status End Date Estimated USAID Contribution Estimated Value of Community Contribution Estimated Value of GoI Contribution Construct 3 classrooms in Abin Zaidoon primary school for boys Muthanna Samawah Education Completed 21-Aug-09 $75,882 $720 $4,140 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 155 ANNEX I. TEAM BIOGRAPHIES JENNIFER KUIPER, TEAM LEADER Jennifer Kuiper has extensive experience managing and designing programs, evaluations, projects, and budgets in diverse policy arenas, often in international, cross-cultural, politically charged environments. She has carried out multiple assignments involving the design and implementation of needs assessments and program evaluations with a focus on qualitative research and the development of state/national-level policy recommendations. Her fields of expertise include the Middle East, third-party civilian peacekeeping, conflict analysis, nonviolence, congressional relations, afterschool education and youth, welfare reform, and child welfare. Most recently, Ms. Kuiper served as Team Leader for the USAID/Iraq and USAID/CMM Conflict Vulnerability Assessment Under the leadership of USAID/CMM, she managed a four-person team of expatriates and Iraqis in fieldwork in multiple provinces in Iraq and managed the team’s planning and preparation, data collection and analysis, and communications of findings and recommendations. In an earlier assignment, she served as Program Director, Palestinian Leadership Project, for the Council of International Programs USA (CIPUSA), where she oversaw all U.S. and overseas-based project implementation, including establishment of an international partnership between CIPUSA and Birzeit University, with support from the Palestinian NGO Network, German Technical Cooperation (GTZ), and the Palestinian Authority’s Council of Ministers. Ms. Kuiper holds an M.P.P. from the University of California, Berkeley. ETHAN ARNHEIM Ethan Arnheim is an evaluation specialist with expertise in the collection and analysis of data in conflict zones and in transition countries. Previously, Mr. Arnheim served as a Transition Specialist, advising country programs for USAID's Office of Transition Initiatives (OTI). Prior to that post, he was Field Program Manager for the Pakistan Transition Initiative, a USAID-funded effort designed to improve governance and increase stability in the restive border area between Afghanistan and Pakistan. During his previous work on the USAID Iraq Community Action Project in 2005, Mr. Arnheim managed reporting and performance management requirements, including program work plans, project management plans, and semiannual and monthly reports. As Program Specialist for the Iraq Civil Society Program (ICSP), he developed project-wide implementation plans and advised component managers on improving project synergy and impact. Mr. Arnheim holds a Master’s of International Development from Johns Hopkins University. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 156 USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 157 ANNEX J. DISCLOSURE OF CONFLICT OF INTEREST FORM Disclosure of Real or Potential Conflict of Interest for USAID Evaluations Instructions: Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest. 45 For external evaluations, all evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated.46 Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s). In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 47 Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 45 USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part 9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct. 46 USAID Evaluation Policy (p. 11). 47 FAR 9.505-4(b). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 158 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Title Team Member Organization The QED Group LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Community Action Program (CAP) End of Project Performance Evaluation The QED Group LLC I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date x x USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 159 ANNEX K. BIBLIOGRAPHY ACDI/VOCA (2012). CAP III Quarterly Report 14. Baghdad, Iraq: ACDI/VOCA. ACDI/VOCA (2011). CAP III Yearly Work Plan. Baghdad, Iraq: ACDI/VOCA. ACDI/VOCA (2011). CAP III Quarterly Report 13. Baghdad, Iraq: ACDI/VOCA. ACDI/VOCA (2009). ACDI/VOCA Community Action Project (CAP) III Performance Management Plan (PMP). Baghdad, Iraq: USAID/Iraq. Barbour, Josephine, Melissa Brill, and Jeffrey Swedberg. (2012) Iraq Civil Society Assessment. The QED Group, LLC. Britan, Gerald. Performance Monitoring & Evaluation: Constructing an Evaluation Report. Tips, 17(1). CHF (2012). CAP III Quarterly Report 14. Baghdad, Iraq: CHF. CHF (2011). CAP III Quarterly Report 13. Baghdad, Iraq: CHF. CHF (2011). CAP IIII Year IV Implementation Plan. Baghdad, Iraq: CHF. CHF (2009). CHF Community Action Project (CAP) III Performance Management Plan (PMP). Baghdad, Iraq: USAID/Iraq. CHF (2008). Iraq Community Action Program National Conference Report. Irbil, Iraq: USAID/Iraq. Grant Thorton (2011) Financial audit of USAID resources managed and expenditures locally incurred by Mercy Corps under "Community Action Program (CAP) III. Cairo, Egypt. Haydan, Mary et al. (2009) Community Action Program III: Data Quality Assessment of Mobilization of Resources by Qadas and Nahiyas and the Marla Ruzicka Iraq War Victims Fund. Baghdad, Iraq: USAID/Iraq. Herr, Harvey et al. (2006). Looking Back and Looking Forward: Iraq Community Action Program (ICAP) Evaluation. Washington, DC: USAID. Herr, Harvey et al. (2006). Community Action Program Phase II (CAPII) Mini-Evaluation and Design Study Final Report. Baghdad, Iraq: USAID/Iraq. IRD (2012). CAP III Quarterly Report 14. Baghdad, Iraq: IRD. IRD (2011). CAP IIII Year IV Implementation Plan. Baghdad, Iraq: IRD. IRD (2011). CAP III Quarterly Report 13. Baghdad, Iraq: IRD. IRD (2009). International Relief and Development (IRD) Community Action Project (CAP) III Peformance Management Plan. Baghdad, Iraq: USAID/Iraq. Karagiozakis, Maria (2009) Counting excess civilian casualties of the Iraq War: Science of Politics? The Journal of Humanitarian Assistance (Feinstein International Center). USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 160 Kumar, Krishna (2006) Conducting Mini Surveys in Developing Countries. Washington, DC: USAID Bureau for Policy and Program Coordination. Mercy Corps (2012). CAP III Quarterly Report 14. Baghdad, Iraq: Mercy Corps. Mercy Corps (2011). CAP III Year IV Annual Implementation Plan. Baghdad, Iraq: Mercy Corps. Mercy Corps (2011). CAP III Quarterly Report 13. Baghdad, Iraq: Mercy Corps. Mercy Corps (2009). Mercy Corps Community Action Project (CAP) III Performance Management Plan (PMP). Baghdad, Iraq: USAID/Iraq. Special Inspector for Iraq Reconstruction (2011). The Iraq Community Action Program: USAID's Agreement with CHF Met Goals, but Greater Oversight Is Needed. Baghdad, Iraq: SIGIR. USAID (2011). Learning from Experience: USAID Evaluation Policy. Washington, DC: USAID. USAID (2011). Audit of USAID/Iraq’s Community Action Program Activities Implemented by International Relief and Development. Baghdad, Iraq: USAID/Office of Inspector General. USAID (2008). Audit of USAID/Iraq’s Management of the Marla Ruzicka Iraqi War Victims Fund. Baghdad, Iraq: USAID/Office of Inspector General. USAID/IRAQ COMMUNITY ACTION PROGRAM III END OF PROJECT PERFORMANCE EVALUATION 161 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov