Iraq Rapid Assistance Program Evaluation July – September 2010 xii IRAQ RAPID ASSISTANCE PROGRAM (IRAP) EVALUATION Final Report November 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by The QED Group, LLC, under contract number 267-M-00-09-00513-00. Iraq Rapid Assistance Program (IRAP) Evaluation Report ii IRAQ RAPID ASSISTANCE PROGRAM (IRAP) EVALUATION Final Report The QED GROUP, LLC 1250 Eye Street, NW Suite 1100 Washington, DC 20005 Phone: 202-521-1900 Fax: 202-521-1901 www.qedgroupllc.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii Iraq Rapid Assistance Program (IRAP) Evaluation Report CONTENTS ACRONYMS...........................................................................................................................IV TABLE OF FIGURES.............................................................................................................. V EXECUTIVE SUMMARY.......................................................................................................VI INTRODUCTION.................................................................................................................... 1 THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE.......................................... 2 PURPOSE OF THE EVALUATION....................................................................................... 5 RESEARCH DESIGN AND EVALUATION METHODOLOGY ......................................... 6 FINDINGS...............................................................................................................................12 CONCLUSIONS ....................................................................................................................25 LESSONS LEARNED.............................................................................................................29 ANNEX A: Ten Steps From Conflict to Development.....................................................31 ANNEX B: Scope of Work...................................................................................................32 ANNEX C: Key Documents and Bibliography ...................................................................36 ANNEX D: Summary of USAID Transition Strategy & The New Way Forward (2007) ..................................................................................................................................................38 ANNEX E: Administrative Delays.......................................................................................39 ANNEX F: End Notes...........................................................................................................41 ANNEX G: Charts.................................................................................................................41 Iraq Rapid Assistance Program (IRAP) Evaluation Report iv ACRONYMS CAP III Community Action Program, Phase III CERP Commander’s Emergency Reconstruction Program COTR CSCM Contracting Officer’s Technical Representative Civil Society and Conflict Mitigation CSO Civil Society Organization DAI Development Alternatives, Inc. DOD Department of Defense DOS Department of State ePRT Embedded Provincial Reconstruction Team ETEC Embassy Technical Evaluation Committee GiK Grant in Kind GUC Grant Under Contract IFES International Foundation For Election Support IRAP Iraq Rapid Assistance Program KRG Kurdistan Regional Government LGP Local Governance Program, Phase III M&E Monitoring and Evaluation NCT Non-Combatant Training NGO Non-Governmental Organization OGPRT Office of Governance and Provincial Reconstruction Team OIG Office of Inspector General PCM Provincial Council Member PDP Provincial Development Plans RRT Regional Reconstruction Team PERFORM Performance Evaluation and Reporting for Results Management PRT Provincial Reconstruction Team SIGIR Special Inspector General for Iraq Reconstruction SOW Scope of Work USAID United States Agency for International Development USAID/OTI USAID Office of Transition Initiatives USG United States Government WTEC Washington Technical Evaluation Committee v Iraq Rapid Assistance Program (IRAP) Evaluation Report TABLE OF FIGURES FIGURES Figure 1: IRAP Approval and Implementation Structure.................................................... 4 Figure 2: Focus Area Sample Distribution ..........................................................................18 Figure 3: Project Identification and Planning ......................................................................18 Figure 4: Project Objectives..................................................................................................19 Figure 5: Intervention Spectrum..........................................................................................25 Figure 6: Speed of Program Progress with Time................................................................26 Figure 7: Approval to USAID Approval: Delay in Days for Each Project........................39 Figure 8: Grant Application to ETEC Approval: Delay in Days for Each Project............39 Figure 9: USAID Approval to Grant Signature: Delay in Days for Each Project............40 Figure 10: Grant Signature to First Disbursement: Delay in Days for Each Project......40 Figure 11: Number of CSCM Grants by Year .....................................................................43 Figure 12: Value of CSCM Grants by Year ..........................................................................40 Figure 13: Grants Amount by CSCM Status (USD) ...........................................................40 Figure 14: Number of CSCM Grants....................................................................................40 Figure 15: Distribution of QRF Grants by Province............................................................40 Figure 16: Distribution of Grant Fund by Province ............................................................40 Figure 17: Distribution of QRF Grants by Province............................................................40 Figure 18: Distribution of Grant Fund by Province ............................................................40 Figure 19: QRF Grants by Type............................................................................................40 Figure 20: Number of Projects by Year ...............................................................................40 Figure 21: Number of QRF Grants by Type ........................................................................40 Figure 22: Total Amount of Grants by Year........................................................................40 TABLES Table 1: Evaluation Matrix ...................................................................................................... 8 Table 2: PRT Interviews ........................................................................................................14 Table 3: Budget, Procurement and Implementation .........................................................21 Table 4: Grants by Year and Status......................................................................................43 Iraq Rapid Assistance Program (IRAP) Evaluation Report vi EXECUTIVE SUMMARY Background and Purpose The Iraq Rapid Assistance Program (IRAP) was initiated by the United States Agency for International Development (USAID) in 2007 in the midst of sectarian chaos and evolved as the security situation ebbed and flowed over the following three years. Because USAID’s Office of Transition Initiatives (USAID/OTI) left Iraq in May 2006, there was a need for a responsive transitional program capable of reaching out to bridge possible gaps between United States Government (USG) programs and reducing Iraqi disillusionment caused by escalating violence. IRAP came online in tandem with 2007’s troop surge and was managed by the Provincial Reconstruction Teams (PRTs), staffed by specialists from the United States Department of Defense (DOD), Department of State (DOS) and USAID. USAID funds IRAP through two streams: Quick Response Funds (QRF: $143M) that originate with the DOS and Civil Society and Conflict Mitigation (CSCM: $30M) from USAID. Each funding source has its own approval procedure. QRF grants are approved through the Embassy Technical Evaluation Committee (ETEC), the Washington Technical Evaluation Committee (WTEC) and USAID’s Contracting Officer’s Technical Representative (COTR) while the CSCM projects are approved directly by the COTR. Program successes reside in the ability of the implementing contractor, Development Alternatives, Inc., (DAI) and DAI’s subcontractor-resource organizations, to support the PRT grant project design and to procure and deliver in-kind goods and services in an effective and timely manner. USAID tasked PERFORM to conduct an evaluation to “determine the progress and success of the project, as well as document accomplishments and lessons learned, over the entire IRAP subcontract period of performance, from September 2007-September 2010,” and to answer three broad questions: • Did the Program contribute to USG objectives expressed in the mandate of the PRTs, the USAID Transitional Strategy and, ultimately, The New Way Forward? Was IRAP an effective counterinsurgency and stabilization tool? • Has the management and implementation of the grants and projects been effective and efficient? To what extent did IRAP institute corrective measures in response to the OIG audit? • What lessons can be learned from the program experiences? Findings Contribution of IRAP to USG Objectives and Effectiveness as a Counterinsurgency/Stabilization Tool IRAP was designed to provide the PRTs with a financial tool to reach out into communities with goodwill and humanitarian intent, comperable to the DOD’s Commander’s Emergency Response Program (CERP) funds. The counterinsurgency/stabilization element intended to provide temporary employment for unemployed groups susceptible to militancy would serve as an effective counterinsurgency tool and improving the quality of life of people would help stabilize the country. IRAP was also intended to further USG aims of promoting democracy as an essential element of stabilization. Breakdown of grant types is discussed in the report under Findings. vii Iraq Rapid Assistance Program (IRAP) Evaluation Report Counterinsurgency and stabilization are complex processes that require military and civilian intervention as well as a host government committed to establishing a viable political system and earning the trust of constituents. Therefore, IRAP grants could only comprise a small part of an overall strategy. A total of 58 grantees were interviewed. Eleven of those were from government and 47 were from non-governmental organizations (NGOs). Fifty-three focus groups were held with beneficiairies of both governmental and NGO grants. Results from interviews and focus groups show IRAP projects had an overall positive impact. More than 90 percent of the surveyed respondents said they would choose to do a repeat project with the PRT. When asked if they thought their project had helped stabilize the country, 87 percent (grantees) and 93 percent (beneficiaries) said it had. Also, 91 percent of both grantees and beneficiaries said IRAP projects improved their lives, with 73 percent of government grantees affirming their lives were better. Of the reasons given for the perceived contributions to Iraq’s stability, nearly two-thirds of the non￾governmental grantees (60 percent) said their projects had improved employment and economic possibilities for beneficiaries. Other stabilization effects cited included contributions to community harmony (15 percent), contributions to women’s issues (10 percent) and improved health and election results (8 percent). Dissenting voices from the grantees (13 percent) said the projects were too small to actually have any impact on the stabilization of Iraq. Governmental grantees gave similar reasons for their projects’ contributions to stability. Four said their projects had improved health; three said their projects had improved community and cultural harmony; two cited economic improvements; and one cited improved electoral results. The one dissenting voice from the government grantees said the project was "far from (the) goal" of achieving stability because it was too small to have a real impact. Of the 53 focus groups held, 41 said IRAP projects had a positive impact on Iraq’s stability. They categorized this impact as increased employment and an improved economy (43 percent), contributions to community harmony (20 percent) and the delivery of services to the community (29 percent). The final 8 percent of the positive responses said the contributions to the electoral process helped stabilize the country. Twelve groups of beneficiaries did not think the projects they participated in helped stabilize Iraq because the projects were too small. These groups said stability required the end of armed conflict and the establishment of a government capable of protecting its citizens. Respondents directly interviewed by the evaluation team almost unanimously said the template voter education projects contributed to the high voter turnout in parliamentary elections, which indicates Iraqis have a growing sense of confidence in a democratic and stable system. Two respondents said IRAP projects, especially those specifically targeted at minority groups, created jealousy within a community and may have exacerbated ethnic tensions. Programs such as IRAP are an important element, but not the only element, in an overall counterinsurgency/stabilization strategy. It is premature to evaluate IRAP’s longer-term impacts on economic and social well-being. Interviews with grantees and beneficiaries of agricultural and women’s projects indicated that these projects have initiated sustainable improvements in their economic and social status. However, the evaluation could not quantify the overall impact of IRAP because the monitoring system collected only input and output data, not outcome and results data. IRAP projects were planned to deliver goods and services (inputs) for activities (rapid actions) in order to contribute to objectives of counterinsurgency and stabilization. Outputs were monitored (e.g., people trained, equipment delivered) but DAI was not required to measure the impact of these outputs on counterinsurgency and stabilization (e.g., do people trained in business practices implement them and how does that contribute to stability?). PRT Work Plans did not pay attention to the objectives of their actions until the last two quarters of Fiscal Year (FY) 2010, so the evaluation team could not evaluate whether IRAP helped the PRTs Iraq Rapid Assistance Program (IRAP) Evaluation Report viii accomplish their Work Plans. Work Plans prior to the last two quarters of 2010 were specific action plans, and in this case, IRAP projects were listed as actions. The majority of PRT representatives reported that IRAP was a useful tool to assist them in carrying out their mission, with the caveat, discussed below, that the IRAP approval process was seen to be too slow for the necessary rapid response. Some PRT Representatives saw IRAP as a useful complement to CERP funds since IRAP could be used for complementary activities such as training, whereas CERP funds provided equipment and commodities. The impact of PRT and other USG investments at the provincial level was to be measured by the Maturity Model introduced to assess progress through stages of maturity. However, this model, deemed to be too subjective to be useful, has been abandoned. The DOS’s Office of Provincial Affairs (OPA) is now studying the possibility of using the Gallup Stability Index based on 47 indicators. Effectiveness of Implementation The evaluation team determined that overall the sampled IRAP projects were implemented effectively. The two problems identified regarding implementation were timeliness of the project approval process and weak project monitoring. Project monitoring was a joint responsibility of DAI and the PRTs. Project Approval Process For 85 percent of projects, the time from the receipt of the grant proposal to the signing of the grant agreement was four months. In some cases, DAI had to rewrite grant proposals, resulting in a longer delay. The lag frustrated some PRTs, which were accustomed to fast CERP decisions. On the other hand, it was also reported that the imperative of disbursing program funds according to an ideal "burn rate" led to pressures on the PRTs and meant that no credit was given to PRTs for being prudent in their project ideas; nor was credit given for the time spent ensuring grantee verification and possible project rejection. CSCM-funded projects were approved more quickly than QRF-funded ones because only the USAID Mission had to approve those projects. Additionally, the introduction of a template design for 518 projects ($10.5 million QRF disbursed, $5.7million CSCM disbursed) in voter education saw minimal approval delays. This template project took less time to approve because it was jointly designed by the USAID Mission and the DOS. Project Monitoring While the PRT was assumed to be an important part of the monitoring function for the program, USAID representatives admitted it was difficult to visit the projects as they were being implemented. The survey showed that 49 percent of the grantees said the PRT visited while 28 percent of the beneficiary focus groups said the PRT visited. Upon closer data examination, it was noted that only 16 (34 percent) of the 47 grantees said a DAI representative visited while only five (9 percent) of the 53 beneficiary focus groups said a DAI representative visited their program. The DAI monitoring site visit reports do not have a standard format and descriptions of the visits are not detailed nor do they monitor the project implementation or results. Following the USAID Office of Inspector General (OIG) audit in August 2009, DAI introduced tools and training in November 2009 designed to improve resource organizations’ management of, and relations with, grantees and locally hired labor. Following the audit, DAI strengthened its grants monitoring system. The restructured system is a stronger and more effective mechanism to correct implementation issues the audit identified. A monitoring and evaluation system was established in February 2010 that tracks not only the four agreed-upon performance indicators for IRAP but also parameters for 24 "F" economic and other indicators. While the IRAP quarterly reports discuss the progress of grants on a quarterly basis, there has been no exercise to look at the grant outcomes or impacts over time. An analysis of a sample of project close-out reports indicated the “impacts” ix Iraq Rapid Assistance Program (IRAP) Evaluation Report reported were outputs rather than outcomes and impacts achieved. The decision to focus on quick outputs instead of impacts was made by the PRTs. Other Findings in Regard to Implementation Overall, key informants reported DAI’s subcontractor-resource organizations were effective resource companies, but they noted a few difficulties. Some complaints stemmed from the misperception that these companies are Kurdish companies that only employ Kurds. In fact, the companies employed local labor and purchased local materials for their projects. Three PRT key informants reported that DAI’s subcontractor-resource organizations sometimes did not deliver equipment with the correct specification to grantees and the PRTs had to intervene to make sure the companies followed through with their commitments. The evaluation team could not find any evidence that the involvement of so many players in the QRF-funded project approval process actually enhanced the success of the projects. Although it is clear USG regulations require approval from Washington for certain levels of procurement and certain types of construction, a review of the Washington-based approval process could be useful for ensuring maximum efficiency. Some key informants said most of the input amounted to edits on grant proposals rather than comments of substance. The evaluation survey confirmed the performance data reported by DAI, and the resource organizations are valid. The majority of grantees reported they have received the equipment included in the grant agreement; the majority of beneficiaries said their projects achieved the results they expected and have not reported any instances of fraud. In the case of an immunization project in Fallujah, beneficiary numbers were reported that were higher than project targets because the clinic reported the total number of beneficiaries who visited the clinic (which exceeded the number immunized), since they did not maintain a separate list of those immunized. This was the only reporting discrepancy the evaluation identified. Conclusions and Recommendations Key informants repeatedly said grants were given on a “first come, first serve” basis and depended primarily upon the personalities of the PRT representatives and whether they thought that projects were an important component of their work. It is not clear that responding to often random targets of opportunity had the same magnitude of impact as a more strategic approach to project selection according to a theory and framework of transition. Regardless of the project selection process, project planning must include more attention to project logic than IRAP projects did, and include outcomes and goals that can be monitored by appropriate indicators. Given the finding that some of the Washington-based approval processes were not always required by USG regulations – and in some case, created delay – USAID and DOS need to reconsider when such approval should be included in a rapid response program. Moreover, in the early stages of transition when rapid response is necessary to quell violence or show support to citizens, appropriate projects should be selected that can be approved more expeditiously. If USAID Missions will be implementing rapid response projects in future post-conflict situations, then the agency might consider adopting USAID/OTI’s approaches to expedited approval, including USAID’s notwithstanding authorities. Given the success of the voter education program and others, the evaluation team also recommends the development of a library of projects based on successful IRAP projects. This library would include both template projects and individual projects reported to have a large impact on improving stability. PRTs or other program agents (e.g. contracted support agencies) must be able modify the library’s design in response to field realities and specificities. Template projects are appropriate for certain types of mass-appeal projects in certain situations. Voter education is a good example, as are public health education and other information campaigns, Iraq Rapid Assistance Program (IRAP) Evaluation Report x child immunization campaigns, education backpack projects and so on. However, such rapid response programs also need the flexibility to respond to local concerns. The evaluation team recommends USAID compile such a library of projects based on a more in-depth evaluation of IRAP projects. Such a library would be extremely useful to the greater donor community that increasingly finds itself in conflict situations. Many key informants recommended that a program such as IRAP should focus on "traditional" transitional sectors rather than blending development and transition objectives. Most USAID PRT representatives spoke of the tensions existing within the PRTs stemming from the debate between those who supported “transition” projects versus those who supported “development” projects. These tensions have been blamed on personality clashes; however, the clash may also be between agency cultures and paradigms rather than individual personalities. While USAID/OTI proposes a flexible approach in chaotic post-conflict or post-disaster situations, evaluating context-appropriate project approaches and sectors for the various stages of transition, as suggested above, may help reduce expectations and eliminate struggles to find the correct project formulation. There were positive outcomes in several provinces when PRTs promoted and nurtured linkages of two kinds – namely, links to local government and links to other USAID programs. The links to local government clearly fall under the overarching principle of The New Way Forward, which recognizes the necessity for Iraqis to be in the lead. Representatives reported local governments were excited to be included in the planning and delivery processes of IRAP. Local government interest is an additional monitoring locus that can ensure project quality. Links to other USAID projects permit the exploitation of complementarities. IRAP was able to fund projects that supported the goals of a number of USAID projects, including IFES, Tijara and Inma and to fill gaps not covered by other USG programs. Such linkages should be promoted when future rapid￾response projects are implemented. Monitoring and evaluation of IRAP-like programs should be conducted as appropriate in high-threat environments, as suggested by USAID/OTI. Their model includes monitoring by implementing partners and USAID country offices, but also by external independent organizations (such as NGOs, universities, or consulting companies) and, where possible, by community oversight committees. Implementors of IRAP programs should increase the number of visits to grantees, especially in areas where there has been a history of grant implementation issues. Site visit reports should be structured and uniform, and seek information on the purpose and outcome of the visit and recommended follow-up steps. Lessons Learned • The Program’s contribution to stabilization cannot be assessed without consideration of the Government of Iraq’s (GoI’s) commitment to creating an enabling environment for local institutions. As USAID/OTI underlined in their document on Lessons Learned[1 • Since IRAP was a transition project, it is inappropriate to introduce sustainability as a goal, especially when timeframes permitted are limited to four to 12 months. Sustainability in project outcomes requires midterm commitment of at least five years to building up local institutions and the appropriate facilitating political environment. ] “… Unless trust in credible local institutions is restored, no stabilization effort will be sustainable.” 1 USAID Transition Initiatives Lessons Learned in Counterinsurgency Planning December 2009 accessed 1st September 2010 http://www.usaid.gov/our_work/cross-cutting_programs/transition_initiatives/lessons_coin.html xi Iraq Rapid Assistance Program (IRAP) Evaluation Report • More thought should have been put into conceptualizing the stages of conflict and transition and the types of projects that are appropriate for each stage. The evaluation team has proposed a 10-step framework for transition activities and projects that could bridge the counterinsurgency/stabilization gap between conflict and development. The framework considers projects from immediate, post-conflict repair and restitution through approaches that maximize the outreach and benefits of the proposed projects. This framework is only illustrative. Other frameworks and typologies are available, including those developed by USAID/OTI and the USAID Office of Conflict Management and Mitigation. • IRAP-style programs should use organizations that can assure close local contact with grantees. These may be NGOs or local contractors. Ideally, these local entities would have sufficient capacity to partner with the program in order to rapidly implement activities; however, many local NGOs and contractors lack the capacity to write grant proposals, properly track implementation, conduct monitoring and evaluation and develop budgets for GiKs. Partnering with other, more traditional development programs that could provide this type of training and technical assistance could serve as a useful complement to future IRAP-style programs. • For a project such as IRAP to be a more effective counterinsurgency, stabilization and transition tool, more research needs to be completed on the impact of the various types of projects and how USAID should select appropriate projects for various stages of conflict and transition. Such a study could be conducted on IRAP but would require a larger sample size and a longer time frame than this evaluation to obtain valid, accurate and useful data. A number of authors have developed schemas that propose project typologies in accordance with a country’s stage of transition, and/or typologies designed to promote the peace￾building process. USAID and the DOS should employ such schemas when implementing IRAP-like projects in the future to more strategically select appropriate projects. John Paul Lederach is a well-known author who has proposed such a schema. His schema and those of others are discussed in the Lessons Learned Section of this evaluation to serve as a reference guide for future programs. • IRAP would have benefitted from training on grant preparation to enhance the capability of NGOs and other grantees to write grant proposals. DAI reported they spent considerable time re-drafting proposals, most of which were written by Grants Officers Responsible (GORs) at PRTs. One grantee reported that in competitive grant situations, well-written proposals were obviously given priority, regardless of the project merits. A computer-based support tool was proposed to augment the online project proposal format available to USAID representatives. In this manner, PRT officers with little international or development experience could access checklists and critical issues with a click, thus complementing pre￾deployment training. Such an online proposal format could also be shared with local NGOs through training. 1 Iraq Rapid Assistance Program (IRAP) Evaluation Report INTRODUCTION As stated above, the 2006 exit of USAID/OTI created a need for a responsive transitional program that could bridge gaps between USG programs and fight Iraqi disillusionment. IRAP came online as part of the 2007 troop surge and was managed by USAID to enable the PRTs to implement projects, staffed by specialists from the Department of Defense, DOS and USAID, thus co-locating personnel from all three main US actors operating in Iraq in provincial locations. On January 10, 2007, President George W. Bush, faced with a country on the brink of civil war, presented The New Way Forward. This new strategy translated into the surge with 20,000 additional troops deployed. The President also promised: “We will give our commanders and civilians greater flexibility to spend funds for economic assistance. We will double the number of provincial reconstruction teams. These teams bring together military and civilian experts to help local Iraqi communities pursue reconciliation, strengthen moderates, and speed the transition to Iraqi self-reliance.” [2 PRTs were introduced in Afghanistan in 2004, and the model was exported to Iraq to include 70 military civil affairs and civilian specialists with expertise in government, engineering and other non￾military fields. PRTs also include Iraqi experts in education, government, language and other areas. ] On January 11, 2007, Secretary of State Condoleezza Rice reiterated the importance of the PRTs. As she stated, “The logic behind the PRTs is simple. Success in Iraq relies on more than military efforts; it requires robust political and economic progress. Our military operations must be fully supported and integrated with our civilian and diplomatic efforts across the entire USG to help Iraqis clear, hold and build throughout all of Iraq.” In August 2007, DOS created the QRF to provide a flexible mechanism to enable PRTs, embedded Provincial Reconstruction Teams (ePRTs) and Regional Reconstruction Teams (RRTs) to facilitate economic and social development through “grants in kind.” (GiKs) [3 IRAP provided grants through two USAID funding streams: QRFs ($143 million, originating from the DOS), and $30 million in CSCM funds. Each funding source has its own approval procedure. QRF grants are approved through the ETEC, the WTEC and USAID’s COTR while CSCM projects are approved directly by the COTR and Contracting Officer. The program successes reside in the ability of the implementing contractor, DAI, and DAI’s subcontractor-resource organizations to support the PRT grant project design and to procure and deliver GiK goods and services in an effective and timely manner. ] The US Embassy’s Office of Provincial Affairs (OPA) in Baghdad is responsible for the overall management of the QRF program. PRTs initiate all QRF and IRAP project proposals. One of OPA’s major roles is the review and approval of proposals and grants over $25,000 to ensure that projects support program goals. An ETEC made up of OPA, other Embassy offices and USAID technical advisors initially review and approve these proposals and grants, with a similar committee in Washington (the WTEC) authorizing the final approval. 2 CBS News’ full transcript of President Bush’s Iraq Speech http://www.cbsnews.com/stories/2007/01/10/iraq/main2349882.shtml, Accessed 12th August 2010. 3 Special Inspector General for Iraq Reconstruction (SIGIR) Opportunities to Improve Management of the Quick Response Fund 29th January 2009 http://pdf.usaid.gov/pdf_docs/PCAAC060.pdf, Accessed 13th August 2010. Iraq Rapid Assistance Program (IRAP) Evaluation Report 2 THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE The Conflict-Stabilization-Development Spectrum IRAP is not a traditional development project. It is a post-conflict response mechanism where the primary objective is to win hearts and minds, counter insurgency, stabilize Iraq, move the country toward a democracy and reduce the military footprint of the coalition forces. USAID/OTI was created in 1994 and is responsible for immediate post-conflict interventions that "…bridge the gap between emergency disaster relief programs and long-term development assistance.”[4] USAID/OTI operated in Iraq from April 2003 to March 2006. During this time "most activities initially fit into one of three categories: democracy building, civil society organization and human rights.”[5 As part of the civilian arm of the 2007 surge, the PRTs (which were introduced to Iraq by Ambassador Zalmay Khalilzad from his experience in Afghanistan) were tasked with reaching out to support civil society in its state of infancy and confusion, and contributing to the stabilization of the post-conflict environment. As described in Hard Lessons: The Iraq Reconstruction Experience, the report of the Special Inspector General for Iraq Reconstruction (SIGIR) released in February 2009, the program was called upon to respond to an ever-changing policy environment.[ ] 6]{i Ultimately, as cited by SIGIR in its February 2010 document on applying the hard lessons, stabilization and reconstruction operations “straddle an uncomfortable perch between conventional war-fighting and traditional development assistance.”[ } As the role of the PRTs evolved, emphasis shifted from delivering projects with a counterinsurgency rationale to those with a stabilization rationale. 7 ]{ii} IRAP was an attempt to fund activities in this ever-shifting context. IRAP’s Performance Management Plan (PMP) Framework: According to the IRAP PMP, DAI was responsible for reporting on four indicators: 1. Average number of days from concept summary and draft proposal submission from PRT/ePRT and CSO to DAI until grant is submitted to ETEC/COP/COTR; 2. Percentage of approved grantees that have signed grant agreements within four weeks; 3. Percentage of grants that show a disbursement within four weeks of grant signature; 4. Percentage of PRT USAID representatives reporting they are satisfied or very satisfied with DAI’s implementation of IRAP. 4 Lawson, M. L. USAID’s Office of Transition Initiatives After 15 years: Issues for Congress Summary 27th May 2009 Congressional Research Service, Accessed 1st September 2010. http://www.fas.org/sgp/crs/row/R40600.pdf 5 Ibid., p17. 6 Hard Lessons: The Iraq Reconstruction Experience February 2009 SIGIR, Accessed 5th September 2010. http://www.sigir.mil/files/HardLessons/Hard_Lessons_Report.pdf 7 Applying Iraq’s Hard Lessons to the Reform of Stabilization and Reconstruction Operations SIGIR February 2010, page 3, Accessed 5th September 2010. http://www.wartimecontracting.gov/docs/SIGIR_ApplyingHardLessons.pdf 3 Iraq Rapid Assistance Program (IRAP) Evaluation Report The Program Implementation: Speed and Accountability IRAP was organized according to a relatively simple structure, as illustrated in Figure 1 on the following page. Grant management rested on two factors: 1. The speed of the process from grant application to disbursement of the grant funds. Any delay ultimately results in frustrations for the grantee and the beneficiaries – and a loss of credibility for the different actors in the delivery process. 2. The monitoring of the actions undertaken by the grantee. This monitoring is crucial to the success of a grant program. If it is not carried out rigorously and correctly the result can be abuses in the use of funds and ridicule of the program by recipients. Having a GiK mechanism was intended to remove the danger of fraud and abuses. However, this mechanism also increased the risks of perceived fraud and favoritism. Iraq Rapid Assistance Program (IRAP) Evaluation Report 4 Figure 1: IRAP Approval and Implementation Structure Department of State USAID Quick Response Fund (QRF) Civil Society & Conflict Mitigation (CSCM) Iraq Rapid Assistance Program (IRAP) Provincial Reconstruction Team (ePRT/PRT) Development Alternatives, Inc. (DAI) Grantees Embassy Technical Evaluation Committee Washington Technical Evaluation Committee CO Technical Representative Grant Application ? ? Beneficiaries Choose how to manage if over $25K Choose which fund to use PRT￾implemented grants PRT application up to $100K Micro-purchases and projects under $25K Resource Organizations (DAI’S SUBCONTRACTOR￾RESOURCE ORGANIZATIONS) 5 Iraq Rapid Assistance Program (IRAP) Evaluation Report PURPOSE OF THE EVALUATION USAID tasked the Peformance Evaluation and Reporting for Results Management (PERFORM) project with the final evalaution of IRAP “to determine the progress and success of the project, as well as document accomplishments and lessons learned, over the entire IRAP subcontract period of performance, from September 2007-September 2010.” The Scope of Work (SOW) required that PERFORM answer three major questions: 1. Has the program contributed to the various objectives? • Evaluate IRAP contributions to USAID Objectives as outlined in the USAID Transition Strategy for Iraq. • Evaluate IRAP contributions to the USG’s New Way Forward Strategy and, in particular, its success as a counterinsurgency/stabilization tool. • Analyze extent to which IRAP QRF grants further PRT objectives as defined in the PRT Work Plan or Joint Common Plan. 2. Has the management been effective and efficient? • Evaluate IRAP grant process efficiency and identify lessons learned. • Assess effectiveness of IRAP’s corrective measures in response to recommendations from the OIG Audit of IRAP conducted in August 2009. • Verify that performance data reported by IRAP is accurate and that reported impacts are occurring or have occurred as stated. 3. What lessons can be learned from this experience? • Evaluate IRAP grant process efficiency and identify lessons learned. • Identify programmatic lessons learned and best practices, and determine how IRAP experience can help USAID in designing Grants Under Contract (GUC)-type programs in the future. Iraq Rapid Assistance Program (IRAP) Evaluation Report 6 RESEARCH DESIGN AND EVALUATION METHODOLOGY Sample Project Selection The evaluation team selected a sample of 58 projects out of the total number of 809 projects listed on DAI’s most recent project data sheet. Projects were selected in the provinces which USAID stated were of most interest to them, namely Anbar, Babil, Baghdad, Basra, Dhi Qar, Dohuk, Erbil, Kirkuk, Mosul, and Sulaimaniya. Projects were selected out of the following sectors: business development, civil society, economic development, education, governance issues, health, rule of law, women’s programs, youth programs and minorities earmarks. Charts illustrating the composition of the sample are available in Annex G of this report. Evaluation Matrix The evaluation matrix (See Chart 1 on the following pages) consists of the questions that were to be asked of the interviewees, informants and data sources. Document Analysis Analysis of key program documents has included a thorough review of evaluations and other studies on PRTs and their mission, lessons learned from USAID/OTI, lessons learned and other studies from SIGIR, evaluations of Community and Program, as well as all the documents related to IRAP. Interviews with Key Informants Interviews were held with key actors in IRAP throughout Iraq. As well as holding interviews, the evaluation team invited case studies or commentaries from USAID representatives in the PRTs and DAI representatives. Interviewees included: • USAID IRAP Management Staff; • The US Embassy’s Office of Provincial Affairs (OPA); • USAID representatives in 13 of 17 PRTs; • DAI offices in Baghdad, Basra and Erbil; • DAI’s subcontractor-resource organizations in Erbil; and • Staff from USAID/Iraq’s Local Governance Program, Phase III (LGP III), Community Action Program, Phase III (CAP III) and the Tijara Provincial Economic Growth Program (Tijara). The interviews with the USAID representatives have been summarized into bullet form and placed in a matrix of three columns identifying the valuable suggestions for project improvement and the common themes running through the interviews (such as the tensions within the PRT teams, and the issues and specificities related to each geographic PRT experience). Survey of Grantees and Beneficiaries To complete the collection of data on IRAP, a survey was conducted with project grantees (governmental and non-governmental) and beneficiaries. A total of 58 grantees were interviewed. Eleven of those were from government and 47 were from NGOs. Fifty-three focus groups were held with beneficiaires of both governmental and NGO projects. 7 Iraq Rapid Assistance Program (IRAP) Evaluation Report Questionnaire and Data Collection Three survey questionnaires were developed to structure interviews with grantees and beneficiaries. PERFORM employs a survey development process that presents and explains the questionnaire structure and the rationale behind the questions, and asks data collectors to offer their opinions on how best to ask questions within the Iraq context and what the most appropriate translation of the questions might be. This method allows the data collectors to assume ownership of the survey process and to be able to address any respondent’s need for clarification that invariably occurs when surveys are translated from foreign languages (in this case, English). The surveys were then translated into Arabic. Survey results were anlayzed and re-translated back into English. Focus Groups of Beneficiaries To enrich the survey, PERFORM’s Iraqi data collectors were asked to conduct focus groups with beneficiaries from the program. Again, the ownership-sharing method used by PERFORM allows for particularly rich responses from the focus groups with answers that are structured but nuanced. In particular, data collectors proposed that rather than simply asking for yes/no answers to certain questions, the numbers of group members saying “yes” and “no” be taken and that the different responses be recorded in the open-ended section of the question. The analysis of the open-ended questions was carried out by coding common themes from the responses while underlining positive and negative outliers. Iraq Rapid Assistance Program (IRAP) Evaluation Report 8 Table 1: Evaluation Matrix Evaluation objectives (from SOW) Interviewees Evaluate IRAP contributions to USAID Objectives as outlined in the USAID Transition Strategy for Iraq Analyze extent to which IRAP QRF grants further PRT objectives as defined in the PRT Work Plan or Joint Common Plan Evaluate IRAP grant process efficiency and identify lessons learned Assess effectiveness of IRAP’s corrective measures in response to recommendations from the OIG Audit of IRAP conducted in August 2009 Identify programmatic lessons learned and best practices, and determine how IRAP experience can help USAID in designing GUC-type programs in the future Verify that performance data reported by IRAP is accurate and that reported impacts are occurring or have occurred as stated USAID representatives • With respect to your mission objectives, what progress have you seen in IRAP? • Successes? • Problems? • Has the program helped you to change the Transition strategy? • Do you feel that IRAP fills in important gaps in the USAID program left by other programs? • Where/what are they? • What kind of coordination or monitoring do you have with the activities of the PRT/ePRTs? • Do you experience any tensions between development and security strategic priorities? • What indicators would you need to decide that PRTs were no longer necessary? • Would you say that the IRAP process is effective in delivering beneficiary￾identified aid? • In comparison with other programs in the Iraq portfolio, would you say that the IRAP process is efficient? • Are there any lessons learned or things to be repeated in other conflict and post￾conflict countries? • Have you noticed any difference in IRAP’s modus operandi since the OIG audit? • Has there been a noticeable improvement in IRAP’s response? • How do you measure that? • Do you feel that it was the correct time to introduce a program like IRAP? • Can you suggest any improvements to the planning/implementation of the program? • Would you agree that placing projects in insecure areas effectively rewards violence and removes incentives from peaceful areas? • How is Monitoring and Evaluation carried out in other programs? • Do you feel that IRAP reports reflect actual progress on the ground? How is that measured? PRT/ePRT members • How do you feel IRAP contributes to your objectives and the objectives outlined in the Transition Strategy? • What principle PRT objectives are described in the Joint Common Plan? • How have the IRAP QRF grants helped you reach your objectives? • When do you expect to be able • If you were asked to improve the IRAP grant process, what recommendations would you make? • What methods in the grant process have you found • Did you find any major difficulties in applying the “Maturity Model” to your field activities? • Have you carried out any evaluation of your existing beneficiaries on the basis of • Can you suggest any improvements to the planning/implementation of the program? • Would you agree that placing projects in insecure areas effectively rewards violence and • How do you carry out Monitoring and Evaluation? • Is there any process of verification or ground truthing of results? 9 Iraq Rapid Assistance Program (IRAP) Evaluation Report to achieve the indicators as presented in your PRT Work Plans? • What period do your Work Plans cover? • How are projects approved and either placed on or removed from the Work Plan? particularly useful in achieving your objectives? • What have you found particularly problematic in the grant process? their position within the ”Maturity Model”? • How do you intend to use the ”Maturity Model” in your management of projects? removes incentives from peaceful areas? QRF/OPA representatives • How does IRAP contribute to the objectives outlined in the USAID Transition Strategy 2008-2012? • Do you find it easy to respond to the needs of the PRT/ePRTs and their Work Plans? • Do you have regular monitoring/ coordination meetings with the PRTs? Under whose aegis are they conducted? • Do you feel your grants disbursement work is appreciated by the PRTs and the USAID Mission? • Given the present modalities in place for QRF processes, can you identify any issues or hurdles that you might change? • Are there any methods that might be used to improve your delivery of grants management through IRAP? • What were the principal recommendations of the OIG Audit report and how did you respond to them? • Have you experienced any difficulties or delays in implementing the corrective measures proposed by the SIG? • What is your progress in implementing the Audit recommendations? • What successes would you point to as program lessons learned that could be used in other countries in similar post-conflict situations? • Are there any issues that need to be avoided in any similar program? • How do you carry out Monitoring and Evaluation? • Is there any process of verification or ground truthing of results? Iraqi beneficiaries • How did you identify your project and what kind of support did you get from the IRAP/PRT Team? • Do you know what the USAID objectives are for the IRAP and how does your project fit into those objectives? Why do you think USAID is funding you? • Have you been able to tell neighbors about your project or do you feel that this would create tensions and/or jealousies? • Do you find that security issues hamper your ability to complete your project or achieve project results? • Did you experience any delays in receiving your funding and was it the amount you felt was necessary to achieve results? • Were you expected to prepare reports for IRAP? How often did you report and did you find the reporting format helpful or difficult? • Did you feel that the IRAP/PRT team had expectations beyond the grant contract? Did you ever feel pushed to hurry and get the project finished? • Did you have to file an end-of-project report to get all of your grant? • Can you suggest any improvements to the planning/implementation of the program? • If you could change any aspect of your grant project, what would it be? • How could we try to make the program better? • Do you receive any monitoring visits from IRAP or the PRT? • How often do they visit? • Do you find their visits helpful or are they a hindrance to your work? Iraq Rapid Assistance Program (IRAP) Evaluation Report 10 • How do issues with security play out if they have an impact on your project? • Do you feel that letting people know about your project and funding with IRAP would make you vulnerable to threats or attacks by insurgents? Subcontracted companies • Do you find that security issues hamper your ability to complete your project or achieve project results? • How do issues with security play out if they have an impact on your project? • Do you feel that letting people know about your project and funding with IRAP would make you vulnerable to threats or attacks by insurgents? • What special precautions do you take to ensure that you are safe from attack? • Were you expected to prepare reports for IRAP? How often did you report and did you find the reporting format helpful or difficult? • Did you feel that the IRAP/PRT team had expectations beyond the grant contract? Did you ever feel pushed to hurry and get the project finished? • Do you receive any monitoring visits from IRAP or the PRT? • How often do they visit? • Do you find their visits helpful or are they a hindrance to your work? DAI IRAP Team • How do you believe IRAP contributes to the objectives outlined in the USAID Transition Strategy 2008- 2012? • Do you feel that you were hampered or especially supported at any time by other actors/stakeholders in the program? • Do you find it easy to respond to the needs of the PRT/ePRTs and their Work Plans? • Do you have regular monitoring/ coordination meetings with the PRTs? Under whose aegis are they conducted? • Do you feel your grants disbursement work is appreciated by the PRTs and the USAID Mission? • If you were asked to improve the IRAP grant process, what recommendations would you make? • What methods in the grants process have you found particularly useful in achieving your objectives? • What have you found particularly problematic in the grant process? • Have you encountered any problems in receiving funds in the Grant Account? Have • What were the principal recommendations of the SIG Audit report and how did you respond to them? • Have you experienced any difficulties or delays in implementing the corrective measures proposed by the SIG? • What is your progress in implementing the Audit recommendations? • Can you suggest any improvements to the planning/implementation of the program? • Would you agree that placing projects in insecure areas effectively rewards violence and removes incentives from peaceful areas? • What special precautions have you taken to avoid security incidents, and how do these considerations delay your work? • How do you carry out Monitoring and Evaluation? • Is there any process of verification or ground truthing of results? 11 Iraq Rapid Assistance Program (IRAP) Evaluation Report you had problems with grant disbursement? USAID Partners: CAP III; LGP III; INMA; Tijara • How did your project fit into USAID’s Transition Strategy? • Did you have any interactions with the PRTs? • Did you find that the modus operandi of the PRTs hampered your own development messages? • How could this have been done better? • Have you had any opportunity to work with IRAP or to seek complementarities with IRAP funded projects? • If you had no contact, do you think that inter-project collaboration is valid or even feasible? • How did you carry out your M&E functions? Iraq Rapid Assistance Program (IRAP) Evaluation Report 12 FINDINGS Distribution of IRAP Projects by Provinces and Types The evaluation team determined there was no overall plan for deciding in which province IRAP grants should be implemented. Instead, a first-come, first-serve approach was employed and the number of grants implemented by province was determined by PRT staff and their determination to achieve their work plans. As illustrated in the charts in Annex H, the vast majority of projects were implemented in Badghad Province (229 out of 809). Other priority provinces included: Anbar (80); Erbil (63); Babil (48); Al Ta’mim (46); Dahuk (46); and Diyala (39). In terms of types of grant, civil society and governance were the largest categories, followd by economic growth, minority ermark, education, health, women’s programs, rule of law, agriculture, and youth. Other charts in Annex F illustrate distribution of the projects by other categories. Management and Program Implementation Organizational Structure of PRT There are several studies on the merits and difficulties of PRTs in the literature, media and blogs available online. PERFORM’s interviews with key informants confirmed that little has changed over the past two years: • Potential tensions between brigade commanders and PRTs – highlighted in the Perito study[ 8 ]{iii • PRTs are often personality-driven, creating tensions in the PRT structures – also reported by the House Armed Services Subcommittee on Oversight and Investigations[ } quotation in end notes; 9]){iv • Insufficient training of civilian personnel (Non-Combatant Training, or NCT) – proposed by Major Tom Kinton[ } 10]{v • It is difficult to pin down the mandate and objectives of the PRTs (e.g., The Joint Campaign Plan is deemed secret) – As Luehrs says, “The basic understanding of what a PRT should be trying to achieve and what it realistically can achieve has been in flux.”[ } 11]{vi} PRT Interviews Thirteen out of 17 PRTs were interviewed and the results are presented in the table on the following page. The following paragraphs present the information gleaned from the many points made by USAID representatives. All representatives were eager to talk to the evaluation team and gave their opinions on improvements or issues that needed attention. These opinions are called “Recommendations.” There were also common themes that could be traced over several 8 Perito, R. et al., Provincial Reconstruction Teams: Lessons and Recommendations, Princeton University Woodrow Wilson School, January 2008. 9 The Washington Post, U.S. Effort to Rebuild from War Criticized. 18th April 2008. Accessed 15th August 2010 at http://www.washingtonpost.com/wp-dyn/content/article/2008/04/17/AR2008041703701.html 10 Kinton, Maj. Tom. Interagency Cultural Similarities in Iraq and Afghanistan Peacekeeping & Stability Operations Institute accessed 16th August 2010 at http://pksoi.army.mil/PKM/publications/bulletin/volume2issue3/interagency.cfm 11 Luehrs, Christoff. Provincial Reconstruction Teams: A Literature Review. Prism vol. 1, National Defense Univ Press, December 2009. 13 Iraq Rapid Assistance Program (IRAP) Evaluation Report interviews. These themes are called “General Commentary.” Finally, specific issues or problems that seem to relate to a single PRT are highlighted as “PRT Specifics.” Several of the interviewed USAID representatives said the comprehension and approach of the PRTs over the life cycle of IRAP had changed from being the rapid response facility to focusing more on development and sustainability. Some said they felt the PRTs had finally "got it," reflecting not only progress within the PRTs but also tensions in former years. Iraq Rapid Assistance Program (IRAP) Evaluation Report 14 Table 2: PRT Interviews Recommendations General Commentary PRT Specifics • Need to be conscious of outcomes when designing IRAP projects. Monitoring systems are effective if front-end design is good and a good evaluation matrix is developed that gives people something to nurture. • IRAP should be focused on transitional sectors: voter education (this worked well), health, media, education like activities implemented under USAID/OTI. • Establish a project review committee within the PRT to review all projects. • Maximize IRAP impact through linkages to other programs in the field (although care must be taken not to spoil the development logic of other programs i.e., CAP). • Publishing the program and asking for proposals can work in more secure areas such as KRG. • Provincial Councils seemed excited about sharing the projects with IRAP. Collaboration with Provincial Councils could have produced more bang for the buck; they object to not knowing what activities have been done. • The online application forms could be used as electronic performance support systems using drop-down menus to provide previous lessons learned. • Re-emphasize the need for linkages with other USAID programs. • There were tensions within many of the PRTs because of inter- agency jurisdictions, but these were improved when PRT Leaders and USAID representatives understood each agency better and could work together. • Military and PRTs focus on weekly “story board” and photo ops and less on actual development results, even accepting that the program was not a development program. • Overall, IRAP was beneficial: it was akin to the Ambassador’s discretionary fund; “PRTs were given money to do things with.” • While being faster than traditional USAID development projects to plan, design and fund, the delays in design and approval were longer than a Rapid Assistance program might expect. • DAI was very helpful and DAI’s subcontractor-resource organizations helped to get things moving on the ground (three out of 14 USAID representatives reported difficulties with DAI). • Monitoring & Evaluaiton has been assured by DAI and has been particularly evident over the last 6-8 months since the IG audit. • New Subcontractor-Resource Organization staff have started mentoring grantees • Five PRT staff interviewed felt they lost control of their projects: once DAI started implementing them. • Fear that a lot of relationships with local government, NGOs will initially be “lost through the cracks” when PRT disbanded, but USAID will build them up again and use resources like DAI and DAI’s subcontractor-resource organizations in a facilitation role. Local nationals in the mentioned organizations really became involved/committed to IRAP projects. • Ninewa: Major problems arose as local populations perceived the IRAP resource organizations as being fundamentally Kurdish. They were then convinced that this was the reason for delays and approval decisions; • Ninewa also felt that DAI would do their own thing. • Karbala PRT meets on a weekly basis with USAID and its implementing partners and worked closely with the Chamber of Commerce to register the Small Business Development Center. • Diwaniya PRT was very successful in coordination with the provincial govt., i.e., USAID/IRAP agriculture project. Province sent agriculture officer to assist in training and PRT field work. PRT also involved provincial and local councils on many projects. PRT also accessed LGP, CAP, etc. • Anbar reported that the Iraq Db MSI was extremely difficult to use, e.g., QRF and IRAP were recorded but CSCM was not. Impossible to track payment dates; cannot call up key items like close-out reports that have an assessment of impact. • In Anbar, it was difficult to find out what lessons were learned in other provinces. Iraq Rapid Assistance Program (IRAP) Evaluation Report 15 USAID Representative General Comments Findings in this section are presented both by generalized statements that were repeated by many interviewees and by illustrative quotations that specify individual perspectives. The interviews revealed that, on the whole, PRTs, or rather USAID representatives on PRTs, were very positive about IRAP, as well as by the contribution made by DAI and its resource organizations. One USAID PRT representative summed up the general attitude: “The program was very positive, putting money into the hands of the PRTs and getting money into the field.” The interviewees reported DAI contributed important experience in project planning and design. As one PRT representative stated, “We would have some ideas and they would say that they had tried something elsewhere that had worked, and we would develop that.” As another one said, ”The real strength of IRAP was DAI. DAI managed to get things going on the ground, while their resource organizations are out there with the local people getting things done.” USAID representatives were asked their opinions about DAI’s response to the OIG Audit Report. Since most were not monitoring the projects, knowledge about the issue was slight. The representative from Anbar said she had noticed a marked improvement over the past six to eight months and that the Subcontractor-Resource Organization staff had started to mentor program grantees. One informant voiced the fear that many relationships with local government and NGOs will initially be “lost through the cracks” when the PRTs are disbanded, but also expressed the belief that USAID will build them up again if they use resources like DAI and DAI’s subcontractor-resource organizations in a facilitation role. Local nationals in these organizations became very involved and committed to IRAP projects. USAID Representative Criticisms The greatest criticism of IRAP was the lag between discussing project ideas with the grantees and eventual project approval and fund disbursement. These delays were particularly frustrating to PRTs who had access to CERP funds with considerably less oversight mechanisms and greater ease of disbursements. There were a great deal of comments about the profligate use of CERP funds. One USAID representative commented: “The CERP funds were a disaster.” Another commented, “Hey, $600,000 CERP funds just disappeared in one province.” Yet another one said, “CERP funds were used to simply buy time.” Representatives were also embarrassed by the time it took to see action on the ground when using IRAP funds. The representatives highlighted the problems arising from the delivery of the IRAP funds and GiKs: CERP funds had been delivered as cash, and grantees were perhaps predictably disappointed that this was not the case within IRAP. These differences may have led to grantees accusing the resource organizations of favoritism. Ninewa indeed seems to have been a province with particular problems between grantees and DAI’s resource organizations. While DAI’s subcontractor-resource organizations have offices and staff throughout Iraq, some Ninewa grantees chose to see them as Kurdish and foreign. This may have had its source in turf struggles for the contracts to build three schools in differing minority areas. DAI’s subcontractor-resource organizations’ responses are recorded below. A couple of PRT representatives also criticized DAI for its poor communications, while another said DAI would “go off and do their own thing” and a third, while being positive in general, still felt he lost control of his projects when DAI began implementation because DAI did not communicate progress regularly enough. USAID Representative Recommentations Recommendations ranged from the global, strategic level, to day-to-day implementation issues with three major common themes emerging from the analysis of the interviews. Iraq Rapid Assistance Program (IRAP) Evaluation Report 16 Firstly, USAID representatives recommended IRAP-like projects should focus on “traditional” transitional sectors rather than mixing development and transition objectives. Most of the PRT USAID representatives spoke of the tensions existing within the PRTs. This was also underscored elsewhere in PRT evaluation studies and Congressional reports. These tensions have been put down to personality clashes; however, the clash may be between agency cultures and paradigms rather than individual personalities. While USAID/OTI proposes a flexible approach in chaotic post-conflict or post-disaster situations, identifying the most appropriate project approaches and sectors for a transitional period may help reduce expectations and eliminate struggles to find the correct project formulation. Secondly, USAID representatives said monitoring systems must be included in the planning process. When this is done, data collection is easy. Outcomes and impacts must also be considered in the planning exercise. A major area for improvement arises from positive experiences in several provinces when PRTs promoted and nurtured connections of two kinds: links to local government and links to other USAID programs. The links to local government clearly fall under the overarching principle of The New Way Forward, which recognizes the necessity for Iraqis to be in the lead. USAID representatives reported that local governments were positive and excited to be included in the planning and delivery processes of IRAP. Local government interest is also an additional monitoring locus that can ensure project quality. The links to other USAID projects permit the exploitation of complementarities. This worked through the program implemented by the International Foundation for Elections Systems (IFES) and the Tijara project, while the search for local government links could be assisted by LGP III. Communications with CAP III indicated links were difficult because of the military aspect of PRT engagement (military security, body armor and weapons) and the assumption by some PRTs that the slow-and-steady approach of CAP was not appropriate. DAI and DAI’s subcontractor-resource organizations: Management, Procurement and Monitoring & Evaluation DAI’s impression of IRAP has been positive. Having worked in Iraq on the USAID/OTI program, it was able to mobilize quickly and take advantage of existing relationships with the resource organizations. DAI staff was very conscious of the inherent tensions in the PRTs and recognized that many PRT staff had little or no international experience and certainly did not have the development background of USAID. The company sees one of its contributions in terms of offering experience in foreign service, particularly in Iraq. DAI also recognized difficulties for PRTs with high staff turnover, and responded by providing a full briefing each time an officer was replaced. These briefings seemed incessant, since many PRT technical staffers were only hired for one-year tours. DAI reported PRTs did not always share work plans for security reasons (This was also reported in the IRAP Monitoring and Evaluation Plan.[12 12 Key Development Services Iraq Rapid Assistance Program Monitoring and Evaluation February 2009 ]). According to DAI, this meant monitoring visits were difficult to plan. Following the OIG audit in August 2009, DAI prepared and delivered training in October and November 2009 to improve the record-keeping procedures and internal administration of resource organizations. The PRT in Ninewa rasied questions about DAI’s reliance on its subcontractor-resource organizations, which they perceived as being Kurdish firms, and thus politically unwelcome by minority groups in Ninewa Province. DAI senior staff suggested this was primarily a problem of chosen perceptions within minority groups, which felt they had an ear for their constituency in Washington. Both of DAI’s subcontractor-resource organizations declared that importing materials and labor is not economical and certainly not profitable; local sources are cheaper for both and both of them relied on local labor to implement projects. 17 Iraq Rapid Assistance Program (IRAP) Evaluation Report DAI’S Subcontractor-Resource Organizations said they had chosen to work with DAI to learn about USG regulations and how to operate within them. These regulations required administrative procedures that differ from traditional Iraqi management methods; this was highlighted by the OIG Audit. DAI’s subcontractor-resource organizations acknowledged the introduction of different procedures in recent months (essentially since the OIG recommendations) and recognized the advantages of transparent administration, which ultimately removes the possibility of being accused of malfeasance. All three companies highlighted the need for constant reiteration of the principles of GiK funding to sub-grantees and the possibilities of variation between budget provisions and actual expenditures. Grantees were often disappointed to receive a GiK and not cash. Changing equipment specifications to meet USG criteria often upset grantees. These experiences led DAI’s subcontractor-resource organizations to pay particular attention to the issues as they communicated with new grantees. However, they feel that this message should be delivered up front when initial planning is started by whoever initiates the project design process. Program’s Achievement and Consequences The program’s achievements have been drawn from an analysis of information provided by DAI. The program had, by June 31, 2010, disbursed $73.3 million out of a committed $87.4 million of Quick Response Funds to a total of 809 projects in 11 focus areas, and $16.8 million of CSCM funds had been disbursed out of a committed $22.6 million to 229 projects for totals of $90.1million disbursed and $110 million committed funds. The introduction of a template design for 518 projects in voter education ($10.5 million QRF, $5.7 million CSCM disbursed) saw greatly reduced approval times for a series of projects throughout Iraq that were deemed successes: voters turned out in exceptional numbers. These projects were, however, part of a wider IFES/USAID/United Nations effort and it is therefore difficult to accurately assess the contribution to voter turnout the IRAP projects may have made. The checks and balances imposed through the program structure have delivered important support to PRT project design processes and introduced oversight on projects with budgets above $25,000. The speed of delivery has fallen somewhere between that of traditional USAID development projects and the CERP. Those delays have frustrated PRTs and some grantees. Contracting Iraq-wide national resource organizations with appropriate management and oversight systems in place has allowed local purchase of materials and labor with consequent savings and stimuli to local economies. The resource organizations reported no direct threats to their workers throughout Iraq. For them, security deterioration merely increased the possibility to be in the wrong place at the wrong time, or created delays due to conflicts or bomb disposal. Following the OIG audit in August 2009, DAI introduced tools and training in November 2009 that were designed to improve the resource organizations’ management of, and relations with, grantees and locally hired labor. The evaluation team found that DAI strengthened its grant-monitoring system after this audit. It appears that the restructured system is a more adequate and effective mechanism that corrected the implementation issues identified by the OIG audit. Survey and Data Analysis Project Sampling and Survey Distribution The objective of the survey was to give a voice to the Iraqi populations effected by IRAP. Grantees were interviewed and project beneficiaries were asked to participate in focus groups. The responses to the survey questions represent the views of these two groups. The survey of Iraqi grantees and beneficiaries was given to a random sample of 58 grantees (11 government Iraq Rapid Assistance Program (IRAP) Evaluation Report 18 0 1 2 3 4 5 6 7 8 9 Sample distribution in each focus area Projects surveyed Figure 2: Focus Area Sample Distribution departments: 47 associations or NGOs) and from 53 focus groups of beneficiaries. The focus areas of the projects sampled is given in Figure 2. Of the 58 projects surveyed, 10 were specifically designed for women, 45 could apply to women and men together, while 3 were intended to reach out to young men in particular. Nevertheless, the senior management of the grantees responding to the survey questions were predominantly men. The focus groups had three profiles: 14 groups out of 53 (26 percent) were composed of female respondents, 23 groups (43 percent) were mixed and 16 groups (30 percent) were composed solely of men. In all (347 persons), 151 women (44 percent) and 196 men (56 percent) attended the focus groups. The level of women’s participation may be due to the fact the evaluation team placed particular emphasis on women’s projects at USAID’s request. Additionally, women were beneficiaries in projects in Civil Society, Health, Agriculture, Rule of Law and those earmarked for minorities. Six projects in Agriculture, another area of interest for USAID, were also surveyed. Project Identification and Planning Respondents were asked who had identified and planned their project. The majority of the grantees were clear that they had both identified (67 percent said the NGO/Association) and planned (63 percent) the project. The beneficiaries were not so sure: 23 percent said they did not know who identified the project and 27 percent did not know who had planned it. Few respondents said the PRT or outside consultants actually helped plan the project. Of the 27 percent of grantees who responded "Other,” 4 percent clarified they had worked on the plan with the PRT while 23 percent said they, the grantee, had planned the project, demonstrating considerable ownership. Figure 3: Project Identification and Planning 63% 25% 39% 4% 0% 4% 6% 17% 4% 0% 8% 27% 27% 50% 27% 0% 10% 20% 30% 40% 50% 60% 70% Grantees Government Officials Beneficiaries Who Developed the Plan? NGO Community PRT Don't Know Other 19 Iraq Rapid Assistance Program (IRAP) Evaluation Report The survey also asked if the respondents felt they had been consulted enough about the projects and if the purpose of the support provided for projects was explained. Forty-three out of 47 (91 percent) of the grantees felt they had been sufficiently consulted by the PRT during planning, while 34 focus groups out of 53 (64 percent) of the beneficiaries said they had not been consulted. Thirty￾six out of 47 (77 percent) of grantees and 33 focus groups out of 53 (62 percent) of beneficiaries said the purpose of PRT support had been made clear. Project Objectives While project grantees and beneficiaries might not be able to describe the objectives of their projects, they were able to express what they hoped to achieve through participation in the project. Three categories were given to close the question: Did you seek financing? And did you seek to improve your capacities? And the last category was “other” if any. Thirty-four out of 68 (50 percent) of the civil grantees’ responses and 40 out of 76, (53 percent) of beneficiaries’ responses stated they sought capacity while 6 out of 15 (40 percent) of the government grantees’ responses said they, too, wanted to build their capacity. Sixteen out of 68 (24 percent) of grantees’ responses and 22 out of 76 (29 percent) of beneficiaries’ responses said they wanted financing while 5 out of 15, (33 percent) of government grantees’ responses said they wanted financing. Eighteen out of 68 (26 percent) of grantees’ responses and 14 out of 76 (18 percent) of beneficiaries’ responses and 4 out of 15 (27 percent) of government grantees’ responses have different “other” answers. The open-ended part of the question revealed that one-third of respondents were very clear on the objectives of their projects and organization. Grantees (38 percent), government grantees (36 percent) and beneficiaries (32 percent) were able to provide a clear definition of the project objectives. 0% 10% 20% 30% 40% 50% 60% Grantees Benefeciaries Government Officials Finance Capacity Other Figure 4: Project Objectives Iraq Rapid Assistance Program (IRAP) Evaluation Report 20 Were there any problems for women’s participation? YES: “The nature of tribal customs and norms prevailing in the region did not allow for women to participate as desired manner, in addition, the project was held in 2008 and the security situation was not stable fully, as well the people were ignorant of what are the activities and background of the Centre for Small Enterprise Development because it was a newly established.” NO: “Because women have an active role in the community and there is acceptance of the equality of men and women, communities accept the right of women to work, especially in order to help the husband and the family make ends meet.” Women’s Participation in Project Identification and Planning To measure the participation of women as decision makers in their projects, the survey asked respondents for their estimate of the number of women having participated in project identification and planning. The survey also asked whether women felt their project had been important for the community. Of the non-government grantee respondents who had been responsible for planning the project, 87 percent said women had participated in both identification and planning. Only 9 percent of the government grantee officials surveyed said women had participated in identification and planning while 28 percent of the beneficiaries felt women had been involved in project planning. When asked about the importance of the project for women, grantees were again almost unanimous in their assertion that the project was important for women. The survey asked all focus group participants if they felt the project was important to women. Two hundred seventy-five participants (81 percent) said they believed the project was important to women while 12 participants (4 percent) said it was not important and 51 participants (15 percent) said they did not know. The respondents were also asked if they knew of any problems for women’s participation in their projects. The responses were varied. Of the 47 grantees, 41 said there were no problems while the other six cited tribal customs as hindering women’s participation. One hundred six beneficiairy respondents (31 percent) identified problems for women respondents, while 235 beneficiary respondents (69 percent) said there were no problems. Grantees (NGO/association and government officials) were asked to express their agreement with the statement: “There is an important role for women in the project.” Ninety-six percent of the non-governmental grantees agreed or strongly agreed with the statement (2 percent had no opinion); 82 percent of the government officials agreed or strongly agreed while 18 percent disagreed. Budget, Procurement and Implementation Since the grantees have a different relationship to budgeting and implementation than the beneficiaries and the nature of responses is different between individual surveys and focus groups, different questions were posed. The grantees were asked to rate their agreement or disagreement with a series of questions (See Table 3). 21 Iraq Rapid Assistance Program (IRAP) Evaluation Report Did the project make your lives better? YES: “Our income has increased due to our increased animal production and we can now afford to have breakfast.” NO: “We gained no benefit from the project financially or culturally but it was an additional burden on us and the station.” Table 3: Budget, Procurement and Implementation “We were given enough money for the project” Strongly Disagree Disagree No Opinion Agree Strongly Agree Grantees 6 % 34 % 0 % 45 % 15 % Government official 9 % 36 % 9 % 27 % 18 % “The equipment was delivered on time” Grantees 7 % 4 % 7 % 36 % 47 % Government official 9 % 36 % 9 % 45 % 0 % “The project was finished on time" Grantees 2 % 15 % 0 % 36 % 47 % Government official 18 % 18 % 9 % 55 % 0 % It is believed that the responses reflect grantee satisfaction with the budget offered by IRAP (60 percent agree or strongly agree) although three grantees (6 percent) strongly disagreed with the statement and those who disagreed felt they were not able to reach the target populations. The government officials were split in their attitude to the budgets (45 percent both positive and negative). With respect to equipment delivery, 82 percent of the grantees were positive with 47 percent strongly agreeing the equipment delivery was timely. Three grantees strongly disagreed that equipment was delivered on time. One of these grantees attributed the delays to DAI. A review of the project documentation of this grantee revealed that it took 78 days between the application date and ETEC approval, and then another 233 days for the project to go through WTEC and USAID for approval, suggesting considerable back and forth to improve the project design. When asked if the project was completed on time, 39 (83 percent) of the non-government grantees and six (55 percent) of the government officials were positive (agree or strongly agree). The examples of the four government department projects that did not finish on time illustrate how frustrating project delays can be. As one government official stated, “We designed a project to implement an English language training course using a computer and a language CD, but we have never been able to initiate the project because we have never received either the computer or CD.” Project Results and Effects Respondents were asked if their projects had fuly achieved the results that had been planned. The non-governmental grantees were the least positive, with 28 percent saying results were less than they had hoped for. Seventy-eight percent of beneficiaries and 82 percent of government officials were more positive, saying the results were as expected. When beneficiaries were asked if there were stories of fraud, three focus groups (5 percent) said yes. Two cited the delivery of material that was not up to standard and the other was the group trying to implement the English language training course cited above. Ninety-five percent of the focus groups said they had heard no stories of fraud. Grantees were asked if they Iraq Rapid Assistance Program (IRAP) Evaluation Report 22 had received the equipment specified in their grant agreement. Eighty-eight percent of the non￾governmental grantees affirmed they had received the specified equipment. Twenty-seven percent of the government officials said they had not received the equipment specificed in their grant agreements. When asked, ”Did the project make your lives better?” 308 beneficiaries (89 percent) responded positively while 34 beneficiaries (10 percent) responded "no." Ninety-six percent of the non￾government grantees responded positively while 73 percent of the government officials stated they agreed or strongly agreed with the above question. Monitoring and Evaluation To assess the degree to which projects received support through monitoring visits, the grantees and beneficiaries were asked whether they received visits, from whom and how often these occurred. Almost all non-government grantees (94 percent) reported receiving visits, although only half of those (49 percent) said the visits were from PRTs. The government departments reported visits in seven projects (66 percent) while 36 percent of the respondents said these visits were from the PRTs. The beneficiary focus groups had varying responses within the groups, so individual responses of the participants in these focus groups are provided. Out of a total of 362 focus group participants, 228 (63 percent) said they received visits while 28 percent said they did not. Fifteen groups out of the 53 surveyed said the visits were from the PRT. It is interesting to note the diversity of organizations and institutions that were reported to have visited the projects: Beneficiaries cited: • The Manager of the Civil Defense Office and some of Municipal Council members; the Tijara project; the City or Provincial Council (four grantees); foreign organizations that they did not know; media; the Department of Veterinary Science; members and management of the grantee association (11); DAI (four); DAI’S Subcontractor-Resource Organizations (two). Governmental grantees cited: • Municipal Council members (2 cases) and DAI/IRAP (1 case). Civil grantees cited: • US Embassy (one case); DAI/DAI’S Subcontractor-Resource Organizations (19 grantees); City Council; Tijara; Provincial Economic Growth Council; Manager of Civil Defense; media and government agencies; local organizations and university. From these responses it is evident that some projects enjoyed a relatively high profile. Maintenance and Operations, Sustainability In general, in all projects, provisions made for the maintenance, repair and ultimately the replacement of equipment is a very clear indicator of the sustainability of a project. Without such plans the project risks failure once equipment breaks down or becomes obsolete. To address whether the IRAP projects had any hope of sustainability, the survey asked if the grantees had any plans to maintain, repair and replace the equipment provided by the project (beneficiaries were not asked as these questions would not normally be in their purview). 23 Iraq Rapid Assistance Program (IRAP) Evaluation Report Did the project help stabilize the country? YES: “Because when young men get a full-time job with a decent fixed income, then young men won’t get carried away in violence. This would reflect positively on stability.” NO: “Stability is connected to security situation and to bringing big investments for the country; but as for this project, it is far too small to have such an impact.” YES: “Some of the participating women could develop their skills at least in limited fields.” NO: “This project was short-term and the country’s stabilization is a long-term process.” According to the COTR, all equipment deliveries under grants had six to 12-month warranties and claims on warranties were filled multiple times over the life of the project. Nonetheless, the COTR stated grantees were unhappy when the warranties expired. The evaluation survey results confirmed grantee discontent regarding warranties and plans for sustainability. About two-thirds of the surveyed grantees and government officials cited the absence of any plans for maintenance (60 percent), repair (63 percent) or replacement (66 percent). Project Improvement and Impact To gain an appreciation of the Iraqi experience with the program, the survey asked if the grantees and beneficiaries would choose to do a project with the PRT again. It was assumed that a “yes” vote would imply that on the whole the experience within the project had been positive, notwithstanding any delays, complaints or disappointments. The overall response to the program was very positive: 91 percent of the government grantees (one dissenter), 94 percent of the civil grantees (six dissenters) and 98 percent of the total number of participants in the beneficiairy focus groups (only 36 people spoke out against repeating the experience with the PRTs). Only one focus group provided comments as to why they would not work with the PRTs again. This group stated the government projects should have been implemented under the supervision of the relevant governmental department to assure that the department had ownership, and hence responsibility for the project. One government grantee suggested the government department should have been included in planning and implementation. Three non-governmental grantees with negative attitudes felt PRT planning was not appropriate. One grantee stated simply, “The PRT poorly planned our project.” Another stated he had “doubts about the PRT policies in regard to grantees and projects selected” and wondered “if projects were being given to the groups who needed them the most.” The third grantee, from a minority group, stated, “We had problems with majority groups in our community because our projects and other projects were mandated only for minority groups.” When asked how the project could be improved, all respondents had suggestions. The beneficiaries and non-governmental grantees expressed the need for more money to continue their activities or to build on their project as well as to extend the number of people benefiting from the projects (beneficiaries 54 percent, grantees 72 percent). Another 23 percent of the beneficiaries wished to extend the project horizon into the future. The governmental grantees wished to build on their project by extending the activities into the future (27 percent), while another 27 percent felt that the project had worked well and did not need improvement. The governmental grantees were also more insistent on their inclusion in the consultation and planning process; namely, 36 percent as opposed to 15 percent of non￾government grantees and 8 percent of beneficiaries. Again, this reflects the overall positive attitude of the respondents to their experience with the project and to their desire to work closely with the program implementation. One grantee wished there were fewer reporting Iraq Rapid Assistance Program (IRAP) Evaluation Report 24 requirements while another suggested the need to include rural clergy in the planning process. Finally, survey respondents were asked if they felt the projects had helped stabilize the country. The response was overwhelmingly positive with 87 percent of non-government grantees, 90 percent of focus group members and 91 percent of the governmental grantees affirming the impact on the stability of the country. Of the reasons given for the perceived contributions to Iraq’s stability, almost two-thirds of the non-governmental grantees (60 percent) felt their projects had improved employment and economic possibilities for the people effected by the program. Other stabilization effects were contributions to community harmony (15 percent) and contributions to women’s issues (10 percent) while improved health and election results were cited by 8 percent of the grantees. Dissenting voices from the grantees (13 percent) stated the projects were too small to actually have any impact on the stabilization of Iraq. The governmental grantees gave similar reasons for their projects’ contributions to stability: Four said they had improved health, three felt their projects had improved community and cultural harmony, two cited economic improvements and one noted improved electoral results. The one dissenting voice from the government grantees felt the project was ”far from [the] goal” of achieving stability. The beneficiaries were also convinced about the contribution to Iraq’s stability through employment and an improved economy (43 percent), contributions to community harmony (20 percent) and the delivery of services to the populations (29 percent). The final 9 percent of the positive responses said contributions to the electoral process helped stabilize the country. Not all focus groups felt the program had contributed to the stability of the country. Of the 12 groups that answered negatively, five said the project was far too small to have any stabilization impact while another four said the project failed to achieve stability. It is important to point out that the cumulative impact of over 1,000 IRAP projects having a small impact on stability for the reasons listed above may have had an overall stabilizing effect on the country. Mechanisms for Complaints DAI instituted a hotline for staff to report any complaints or allegations of fraud. Although this hotline was designed only for DAI staff, the evaluation asked grantees and beneficiaires whether they knew of such a hotline or any other mechanism through which they could report their complaints. Eighty-five percent of the beneficiaries did not know the hotline existed nor did they know of any other avenue for reporting complaints; 55 percent of the governmental grantees did not know of the hotline and 45 percent of the non-governmental grantees were not aware of its existence. However, these percentages indicate some beneficiairies and grantees were aware of the hotline despite DAI’s assertion that it was meant only for DAI employees. Three beneficiaries identified the DAI hotline while five identified the grantee as the receiver of complaints. Of the five governmental grantees who knew of the hotline’s existence, three knew the number and one knew someone who had it. The non-governmental grantees were much more aware of the hotline function, or at least the possibility of contacting the PRT or DAI if there were complaints. One grantee responded that he did not recall the number ”but it was written in the contract.” Iraq Rapid Assistance Program (IRAP) Evaluation Report 25 CONCLUSIONS Contributions to USAID Transition Strategy 2006-2008 and The New Way Forward, January 2007 The immediate post-conflict response system operates according to the assumption that it is appropriate to disburse large amounts of money (CERP) or GiKs (IRAP) for “stabilization” projects in order to calm down fighting and buy time. As one USAID representatives said: "They don’t realize that they are not buying time, they are only renting it. Not only are they renting it, but the price goes up each time the rent is paid.” IRAP was implemented during the surge; it is impossible to conclude what impact the increased military presence had on stabilization versus the impact of projects such as CERP and IRAP. It is clear that reports of violence decreased during the period during which IRAP projects were implemented, but the country was still far from stable. And, as stated in the previous section on “Findings,” the majority of IRAP grantees and beneficiaires believe their IRAP projects did contribute to stablizing the country. This evaluation coincided with the USG withdrawal of combat troops from Iraq on Aug. 31, 2010. At the time of writing this report, it is not clear whether there will be resurgence in the sectarian fighting that prevailed when IRAP was first planned and designed. Many Iraqis surveyed said the program contributed to stability, while others say this objective is far beyond the scope of the simple projects implemented in the program. Both IRAP projects and local political forces act upon the population, and only time will tell if the internal and external divisive forces brought to bear on Iraqi society will have the ability to disrupt the situation in the future. Contribution to PRT Objectives IRAP did contribute to PRT objectives as reported by USAID PRT representatives. IRAP complemented CERP funds and USAID representatives were able to draw on both funding sources to help them accomplish their objectives. Figure 5: Intervention Spectrum IRAP is a valid part of the intervention spectrum from conflict to development. Hard on the heels of military intervention, there is a need for both flexibility and speed in response to humanitarian need. As governance bodies are set up or governmental structures are supported, and as economic projects are attempted, there is a necessary process of bureaucratization with careful project planning, due diligence in accountability and monitoring of project results. IRAP provided this. IRAP Grant Process Efficiency The success of a grant management facility is evaluated in the quality of the projects planned, the speed of their approval and fund disbursement, the quality of the project implementation and the impact on overall objectives. If projects have been planned correctly, then proper implementation should have measurable impacts on the beneficiary populations. Commander’s Emergency Response Programs Counterinsurgency Programs - COIN Iraq Rapid Assistance Program - IRAP Post-Conflict Capacity Building Traditional Development Iraq Rapid Assistance Program (IRAP) Evaluation Report 26 Figure 6: Speed of Program Progress with Time 0 100 200 300 400 500 600 700 800 900 Nov, 2007 Feb, 2008 May, 2008 Aug, 2008 Nov, 2008 Feb, 2009 May, 2009 Aug, 2009 Nov, 2009 Feb, 2010 Speed of program progress with time # Received # ETEC approved # GA signed 1st Disbursement CSCM applications CSCM grants cleared CSCM GA signed In the transitional context of IRAP, projects were planned and monitored on the basis of inputs and outputs rather than outcomes and impacts. With respect to program speed, the mandatory oversight and approval stages meant there would be delays. On average, 85 percent of the projects traversed each stage of the approval process within 30 days; however, these delays add up to an average of about four months for project approval to go from grant application to fund disbursement. If the imperative is to deliver funding rapidly, it would be appropriate for IRAP management to study ways in which the approval process could be accelerated. Reference to Figure 2 on this page shows that using an accelerated decision process for the CSCM funds meant that delays were kept to a minimum. According to Figure 2, money was obligated at a high pace from November 2007 through January 2009. The high pace was catalyzed by the violence in the country and the need to get projects implemented in an effort to stabilize the country. In January 2009, funding obligations for QRF slowed down due to several factors: Violence diminished and USAID shifted its focus to more development-oriented projects; and USAID focused on spending CSCM funds at a high rate from January to March 2009, relying less on QRFs. This was largely due to the lull in violence and USAID’s decision to build civil society, as well as the desire to take advantage of a funding source that benefitted from a faster approval process. Obligation of QRFs then accelerated sharply in December 2009, after USAID and DOS decided to fund 518 voter education template projects. Corrective Measures Following OIG Audit 2009 IRAP’s corrective measures appear to be adequate and effective for most of the grants included in the evaluation sample. Data from a sample of 46 grantees and 51 beneficiaries, plus numerous interviews with IRAP’s stakeholder and supporting document reviews, suggests DAI strengthened its procedures system to monitor active IRAP grants. As a result, problems identified in the OIG audit report of August 2009 were addressed and rectified systematically in a timely and coherent manner. DAI staff conducted spot checks and visited grantees, and senior DAI staff also visited some grant recipients. Site visits are all documented and the reports are available in the DAI administrative database. The view of a USAID PRT representative gained upon visiting several IRAP grant projects, and reading DAI monitoring and evaluation reports, summarizes what other PRT representatives conveyed to us: “DAI’s procedures for monitoring and evaluating grants were adequate and effective.” DAI took numerous measures to address fraud and other misconduct. Those efforts included but were not limited to: retraining staff, adjusting grant reporting documentation and taking a more vigorous approach to grant implementation oversight. As a result, feedback from grantees and grant beneficiaries indicated that issues cited in the OIG audit reportvii appeared to be adequately corrected among the sample of grants selected for this evaluation. 27 Iraq Rapid Assistance Program (IRAP) Evaluation Report Data analysis suggests that DAI: • Improved the work performed, and goods/services provided by DAI subcontractors • Provided informative updates to stakeholders of issues concerning progress of grant implementation • Contained improper practices with regard to payroll procedures, expense reimbursements, transference of monies to grantee bank accounts and improper salary payments among grantee staff. Project Design and Selection Process Key informants repeatedly said grants were given on a “first come, first serve” basis and depended primarily upon the personalities of the PRT representatives and whether they thought that projects were an important component of their work. It is not clear that responding to often random targets of opportunity had the same magnitude of impact as a more strategic approach to project selection according to a theory and framework of transition. Regardless of the project selection process, project planning must include more attention to project logic, and include outcomes and goals that can be monitored by appropriate indicators. USAID and DOS need to reconsider whether and when Washington-based approvals should be included in a rapid response program. Moreover, in the early stages of transition when rapid response is necessary to quell violence or show support to citizens, projects that can be approved quickly should be selected. If USAID Missions, rather than USAID/OTI, will be implementing rapid response projects in future post￾conflict situations, the agency might consider adopting some USAID/OTI approaches to expedite approval or employ the USAID notwithstanding approval authorities. Given the success of the voter education program and others, the evaluation team also recommends the development of a library of different types of projects based on projects found to be successful in all sectors in which IRAP worked. The use of this library must, however, be flexible to allow PRTs or other program agents (e.g., contracted support agencies) to modify the design to adapt to field realities and specificities. Template projects are appropriate for certain types of mass-appeal projects in certain situations. Voter education is a good example, as are public health education and other information campaigns, child immunization campaigns, education backpack projects, and so on. However, such rapid response programs also need to have the flexibility to respond to local concerns. Such a library would be extremely useful to the greater donor community that finds itself increasingly in conflict situations. Projects must also be selected according to an overarching schema of stage of transtition and/or peace-building. Such possible schemas are discussed in the Lessons Learned section of this evaluation report. Many key informants recommended that a program such as IRAP should focus on ”traditional” transitional sectors rather than mixing up development and transition objectives. Most of the USAID PRT representatives spoke of the tensions existing within the PRTs stemming from the debate between those who supported “transition” projects versus “development” projects. Such tensions were also underlined elsewhere in PRT evaluation studies and Congressional reports. These tensions have been attributed to personality clashes; however, the clash may be between agencies rather than individual personalities. While USAID/OTI proposes a flexible approach in chaotic post-conflict or post-disaster situations, evaluating appropriate project approaches and sectors for the various stages of transition, as suggested above, may help manage expectations and eliminate struggles to find the correct project formulation. There were positive experiences in several provinces when PRTs promoted and nurtured linkages of two kinds – namely, links to local government and links to other USAID programs. The links to local government clearly fall under the overarching principle of The New Way Forward, which recognizes Iraq Rapid Assistance Program (IRAP) Evaluation Report 28 the necessity that Iraqis be in the lead. PRT representatives reported that local governments were positive and excited to be included in the planning and delivery processes of IRAP. Local government interest and participation is also an additional monitoring locus that can ensure project quality. The links to other USAID projects permit the exploitation of complementarities. IRAP was able to fund projects that supported the goals of a number of USAID projects, including IFES, Tijara and Inma. Such linkages should be promoted when future rapid response projects are implemented. DAI reported it spent considerable time redrafting proposals. One grantee reported that in competitive grant situations, grantees who could write a good proposal were obviously given priority, regardless of the project merits. A computer-based support tool was proposed to augment the online project proposal format available to USAID representatives. In this manner, PRT officers with little international or development experience could access checklists and critical issues with a click. Such an online proposal format could also be shared with local NGOs through training. Monitoring and Evaluation Monitoring and evaluation of IRAP-like programs should be conducted as appropriate in high-threat environments, as suggested by USAID/OTI. Its model includes monitoring by implementing partners and USAID country offices, but also by external independent organizations (such as NGOs, universities, or consulting/contracting companies) and, where possible, by community oversight committees. Implementors of IRAP programs should increase the number of visits to grantees, especially in areas where there has been a history of grant implementation issues. Site visit reports should be structured and uniform, and seek information on the purpose and outcome of the visit and recommended follow-up steps with clear roles, responsibilities and deadlines. Veracity of Performance Data The evaluation survey confirmed the performance data that was being reported by DAI and the resource organizations was valid. As one USAID representative said: ”You can take all the photographs you want, but you cannot know what happens when you turn your back.” The grantees say they received the equipment that was agreed upon in the grant agreement, beneficiaries say they had the project and achieved the results that they expected and, more tellingly, there have been no stories of fraud told to interviewers. 29 Iraq Rapid Assistance Program (IRAP) Evaluation Report LESSONS LEARNED • The Program’s contribution to stabilization cannot be assessed without consideration of the GoI’s commitment to creating an enabling environment for local institutions. As USAID/OTI underlined in its document on Lessons Learned[13 • Since IRAP was a transition project, it is inappropriate to introduce sustainability as a goal, especially when the time frame permitted is limited to four to twelve months. Sustainability in project outcomes requires mid-term commitment (five years) to building up local institutions and the appropriate facilitating political environment. ] “Unless trust in credible local institutions is restored, no stabilization effort will be sustainable.” • As more bureaucratic procedures are implemented, it becomes appropriate to seek ways to streamline the approval and disbursement processes. • A 10-step framework (see Annex A) has been proposed for transitional activities and projects that could bridge the counterinsurgency/stabilization gap between conflict and development. The framework considers projects from immediate, post-conflict repair and restitution through approaches that maximize the outreach and benefits of the proposed projects. IRAP-style programs should partner with organizations that can assure close local contact with grantees; these may be NGOs or local contractors (as with IRAP). However, it is imperative to include appropriate training and management tools in outreach to communities since these organizations are the operational arm of transitional and development projects. The organizations must also be constantly monitored to ensure good governance in their deliveries. Training must include tools for explaining and promoting no cash, GiK projects and budgeting. • For a project such as IRAP to be a more effective counterinsurgency, stabilization and transition tool, more research needs to be completed on what the impact of the various types of projects was and how USAID should select appropriate projects for various stages of conflict and transition. Such a study could be conducted on IRAP, but would require a larger sample size and a longer time frame than this evaluation to obtain valid, accurate and useful data. A number of project typologies based on the stage of transition and/or stability a country is in have been proposed and should be referred to when designing and implementing an IRAP-like project in the future. The recommendations of peace-making theorists and practitioners such as John Paul Lederach, Peter Uvin and many others offer project typologies that are useful. Such schemas could serve as the framework for developing appropriate types of projects for different stages of transition. • USAID might also apply more of the lessons learned from USAID/OTI. USAID/OTI has many documents that lay out different approaches to transition situations and peacebuilding that would be useful to help more appropriately design and implement IRAP-like projects. Evaluator Richard Blue has conducted a number of evaluations of USAID/OTI projects that offer useful lessons learned to apply also. The Journal of Peacebuilding and Development is a valuable reference source for such projects. In particular, Peter Uvin’s article in this journal titled The Development/Peacebulding Nexus: A Typology and History of Changing Paradigms (Vol 1, No. 1, 2002). 13 USAID/OTI Lessons Learned in Counterinsurgency Planning December 2009, Accessed 1st September 2010 http://www.usaid.gov/our_work/cross-cutting_programs/transition_initiatives/lessons_coin.html Iraq Rapid Assistance Program (IRAP) Evaluation Report 30 • The 1997 OECD publication Guidelines on Peace, Conflict, and Development Cooperation and the UN’s 1998 Priorities for Post-Conflict Peace Building offer valuable guidelines for a framework on similar post-conflict project typologies. Iraq Rapid Assistance Program (IRAP) Evaluation Report 31 ANNEX A: TEN STEPS FROM CONFLICT TO DEVELOPMENT Iraq Rapid Assistance Program (IRAP) Evaluation Report 32 ANNEX B: SCOPE OF WORK PROGRAM EVALUATION OF THE IRAQ RAPID ASSISTANCE PROGRAM (IRAP) Scope of Work July-September 2010 USAID Iraq Program Description USAID’s IRAP Project was created in 2007 to manage large grants given under the Department of State’s QRF IRAP was designed with a GUC structure, enabling quick and flexible disbursement given a rapidly changing operating environment. The grants awarded under IRAP are designed to foster development, security, and sustainable prosperity in Iraq. IRAP grantees can be local Iraqi government entities, NGOs, or CSOs. Large grants under QRF are defined as any grant amounting to more than US$25,000. Activities are normally identified and developed by the PRTs/ePRTs (embedded PRT). Grant proposals are developed and submitted to DAI for review and approval before being forwarded to USAID’s Office of Acquisition and Assistance for final approval. Grants typically range between $25,000 and $500,000, though a few have totaled more than US$2,000,000. Total authorized funding for IRAP is $200 million for the period of performance from Sep 2007 – Sep 2010. A total of US$131 million has been expended as of March 31, 2010. The current pipeline analysis indicates that pending successful implementation of the 2010 annual work plan, the total obligated amount of US$173 million will be fully expended by the end of the contract period, September 30, 2010. Background In the course of implementation, IRAP has faced many challenges. As any Iraq-based project, IRAP had to manage security risks for project staff and partners resulting from country-wide instability. Many IRAP Iraqi partners were inexperienced and operationally immature. Additionally, many partners lacked the structures, controls, and experience required for cash grants in compliance with USAID regulations, therefore, IRAP used novel grant mechanisms to allow for smooth implementation. The two main grant mechanisms used were GiKs and ‘fixed obligation grants’. From July-August 2009, the Office of the Inspector General (OIG) conducted an audit and provided audit report on the Iraq Rapid Assistance Program (IRAP). The audit report highlighted several issues that the IRAP project was to address. Purpose The United States Agency for International Development (USAID) is seeking a final evaluation for the IRAP project. The purpose of the IRAP final evaluation is to determine the progress and success of the project, as well as document accomplishments and lessons learned, over the entire IRAP subcontract period of performance, from September 2007 – September 2010. The IRAP Evaluation shall be conducted in seven (7) weeks, during which time the evaluation team will assess performance based on key factors including but not limited to: 1. Evaluate IRAP contributions to USAID Objectives as outlined in the USAID Transition Strategy for Iraq ; 2. Evaluate IRAP contributions to the USG’s Way Forward Strategy and in particular, its success as a counter-insurgency/stabilization tool. 3. Analyze extent to which IRAP QRF grants further PRT objectives as defined in the PRT Work Plan or Joint Common Plan. 33 Iraq Rapid Assistance Program (IRAP) Evaluation Report 4. Evaluate IRAP grant process efficiency and identify lessons learned; 5. Assess effectiveness of IRAP’s corrective measures in response to recommendations from the USAID OIG Audit of IRAP conducted in August 2009; 6. Identify programmatic lessons learned and best practices, and determine how IRAP experience can help USAID in designing GUC-type programs in the future; 7. Verify that performance data reported by IRAP is accurate and that reported impacts are occurring or have occurred as stated. Key Tasks & Questions to Be Answered Key research questions: Evaluate IRAP contributions to USAID and USG Objectives in Iraq: • To what extent did IRAP grant activities contribute to achieving the objectives of the USAID Transition Strategy for Iraq 2006-2008, which includes support for transition to a more prosperous, democratic and secure Iraq? USAID Transition Strategy document will be provided by the Mission. • Did IRAP help to fill important gaps not funded through existing economic and/or governance programs? • Did IRAP help to mitigate conflict in Iraq by strengthening civil society through grants to CSOs? Was it an effective counter-insurgency tool? Evaluate IRAP contributions to PRT Objectives: • Did IRAP stabilization activities contribute to and support PRT/ePRT objectives as outlined in the PRT Work Plans, Joint Campaign Plan, the Provincial Development Plans (PDPs) or the Maturity Models? To what extent did IRAP fulfill the objectives of the USG New Way Forward strategy and the PRT workplans designed to implement this strategy? These objectives are defined by the QRF Country Objectives as Political Development, Economic Development, Improving Governance, Supporting Rule of Law, and Community Building. The relevant documents will be provided by the PRTs, the USAID Mission and the Office of Provincial Affairs (OPA). Evaluate IRAP grant process efficiency and identify Lessons Learned: • What major problems did IRAP encounter during implementation and what was successful as a response to overcoming those problems? • How can IRAP experience help USAID in developing and implementing grant programs in the future? Assess effectiveness of IRAP’s corrective measures: • In response to the recommendations of the USAID OIG Audit of IRAP conducted in July￾August 2009, were IRAP’s corrective measures adequate and effective? The Audit Report will be provided by the Mission. Verify that performance data reported by IRAP is accurate: • Are reported impacts occurring or have these occurred as stated? If NOT, can the reasons for inaccuracies be identified and recommendations made on how these issues might be remedied in future programs? Iraq Rapid Assistance Program (IRAP) Evaluation Report 34 Sector Assessment: • The IRAP evaluation team should assess the impact of IRAP grants on the development of the agriculture sector and in improving the status of Iraqi women by means of the grants in support of women’s organizations and issues. Methodology of the Evaluation The Evaluation Team shall be in Iraq for a period of six weeks. Through interviews with key informants and site visits to IRAP/DAI, USAID, the US Embassy Quick Response Fund /Office of Provincial Affairs (QRF/OPA), with IRAP stakeholders and partners on PRTs/ePRTs. Most interviews will be conducted in English. At the PRTs/ePRTs, the evaluators will interview key observers available there (such as BBAs, USDA reps, USAID reps, military Civil Affairs Teams, etc.). Field trip logistical support will be provided by the PRTs/ePRTs. Interviews with Iraqi key informants and IRAP beneficiaries shall be conducted in Arabic. A list of potential Data sources and Interviewees from among the following groups will be provided: • PRT teams members, including USAID representatives, in the selected provinces (identified by PRT Office and COTR); • Benefiting Iraqi local Government institutions, CSOs and community leaders or representatives (identified by DAI/IRAP and COTR); • Benefiting Iraqi citizens (identified by DAI/IRAP); • Sub-contracted resource organizations (identified by DAI/IRAP and COTR); • QRF/OPA representatives in Baghdad (identified by PRT Office and COTR).; • USAID representatives in Baghdad (identified by PRT Office and COTR). Team Qualifications and Selection Criteria The evaluation team will consist of 3-4 expatriate experts, one of whom will be designated Team Leader; and 3-4 local experts in civil society and conflict mitigation. The expatriate team leader should have the following minimum qualifications: 1. A master’s degree or higher, with extensive experience in the management of large development projects. 2. A minimum of 10 years of experience working on stabilization, conflict mitigation and related development programs, preferably in transitional, post-conflict, or fragile countries. 3. Senior-level project management experience and associated technical program management skills. 4. Strong experience designing and/or implementing project evaluations on similar programs, preferably in post conflict countries. Other expatriate team members should have the following minimum qualifications: 1. Advanced degree in social sciences, international development, economics or related field preferred, and relevant experience with either government programs or development projects in Iraq or similar country contexts; experience in the region is preferred. 2. Familiarity with the Iraqi civil society and governance sector. 3. Previous experience as members of a team conducting research or evaluation of similar large-scale projects. 35 Iraq Rapid Assistance Program (IRAP) Evaluation Report Period of Performance USAID would like this evaluation to commence as soon as possible and preferably by July 20, 2010. The estimated duration of performance is eight weeks total, broken out as follows: one week, prior to arrival of the evaluation team in Iraq, for literature review and collection of background information; six weeks of field work in Iraq; and subsequent to receipt of USAID’s comments on the draft report, one-week to produce and submit the final report. The team will arrive in Iraq on or about August 1, 2010. Upon arrival in Iraq, the team is required to meet with the IRAP and PERFORM COTRs at USAID to receive an in-brief and discuss the SOW to ensure clarity of goals and objectives of the study. Deliverables 1. Within 2 working days in country, submit to the IRAP and PERFORM COTRs a draft evaluation work plan with benchmarks listed. 2. Within 4 working days, submit a draft outline of the Evaluation Report, including planned questions and detailed field-study methodology and timeline that will be used for the evaluation process. 3. At the end of the third week of field work, the evaluation Team Leader must submit to the IRAP and PERFORM COTRs a brief (2-3 pages) progress report on the evaluation. 4. At the beginning of week seven of the evaluation, that is, the beginning of the sixth week of field work, the Team Leader must submit to the IRAP and PERFORM COTRs a 2-5 page summary of key findings and recommendations for Mission review. The estimated date for this summary is September 6, 2010. 5. Also at the beginning of week seven of the evaluation, that is, the beginning of the sixth week of field work, the evaluation team must make a verbal presentation on the key findings and recommendations to the Mission. The team would be at liberty to use any graphic aids it chooses. The estimated date for this presentation is September 6, 2010. 6. At the end of week seven of the evaluation, that is at the end of the sixth week of field work, the Team Leader must submit a complete draft report, of not more than 40 pages (excluding annexes), to USAID for review, comments and suggested changes. The estimated date of this submission is September 12, 2010. 7. One week after receiving USAID’s comments, QED must submit the final report. (USAID expects to provide its comments within a week of receiving the draft report.) The date of the final submission is estimated to be September 25, 2010. Iraq Rapid Assistance Program (IRAP) Evaluation Report 36 ANNEX C: KEY DOCUMENTS AND BIBLIOGRAPHY Audit of USAID/Iraq’s Iraq Rapid Assistance Program (IRAP), Audit Report No. E-267-09-005-P, August 16, 2009 Constructing an Evaluation Report, April 14, 2006 DAI Action Tracker Table, June 2010 IRAP Annual Work Plan (Year One), October 2007 to October 2008 IRAP Annual Work Plan (Year Two), October 1, 2008 to October 31, 2009 IRAP Annual Work Plan (Year Three), October 1, 2009 to September 30, 2010 IRAP Scope of Work (SOW), Contract No. 267-C-00-07-00505-00 IRAP Grant Tracker IRAP Monitoring and Evaluation, Quarterly Report, (April 01- June 30, 2010) IRAP Monitoring & Evaluation, Quarterly Report, (January 01- March 31, 2010) IRAP Monitoring & Evaluation , February 06, 2009 IRAP First Quarterly Report (October 1 – December 31, 2007) IRAP Third Quarterly Report (April 1 – June 30, 2008) IRAP Fourth Quarterly Report (July 01 - September 30, 2008) IRAP Fifth Quarterly Report (October 01 - December 31, 2008) IRAP Sixth Quarterly Report (January 01 - March 31, 2009) IRAP Seventh Quarterly Report (April 01 - June 30, 2009) IRAP Eighth Quarterly Report (July 01 - September 30, 2009) IRAP Ninth Quarterly Report (October 01 – December 31, 2009) IRAP Tenth Quarterly Report (October 01 – December 31, 2009) IRAP Eleventh Quarterly Report (April 1 – June 30, 2010) IRAP Performance Management Plan (2007-2009) IRAP Grants Administration Handbook PRT Assessment Hand Book PRT work Plan 2010 by provinces Quick Guide to ADS Gender Integration and Analysis Requirements 37 Iraq Rapid Assistance Program (IRAP) Evaluation Report Quick Response Funds (QRF) Guidelines Republic of Iraq provincial Development strategic Executive Summaries United States Government Accountability Office Report: Provincial Reconstruction Teams in Afghanistan and Iraq USAID/IRAQ Transition Strategy Plan (2006-2008) Iraq Rapid Assistance Program (IRAP) Evaluation Report 38 ANNEX D: SUMMARY OF USAID TRANSITION STRATEGY AND THE NEW WAY FORWARD 2007 USAID Transition Strategy 2006–2008 USAID Transition Strategy Plan 2006–2008 was conceived to bridge the transition from the short￾term provision of essential services to long-term, integrated and Iraqi-led development. The Plan consisted of three strategic tracks: Security (clear, hold, build): stabilizing strategic cities, improving local service and government capacity, supporting Iraqi communities Economy (restore, reform, build): increasing access to financial services; strengthening agricultural capacity and productivity; promotion of privatization and business development; advancing policy, subsidy, regulatory and transparency reforms. Political (isolate, engage, build): developing capacity and core functions of national institutions, supporting budget development.14 Additional cross-cutting themes included: Sustainability and Capacity Building, Responsiveness, Transparency and Accountability, Global Development Alliance. The five core principles of The New Way Forward were: 1. The government of Iraq is in the lead and is committed and performing. 2. Help Iraqis build their own capacity to govern. 3. Decentralize and diversify our civilian presence and assistance to the Iraqi people. 4. Target assistance to isolate extremists and empower moderates who support peace and democracy. 5. Reinvigorate regional diplomacy and strengthen support for the Iraq government.[15] 14 USAID Iraq Transition Strategy Plan (2006–2008) Summary March 2006, Accessed 28th August 2010 http://www.usaid.gov/iraq/pdf/USAID_Strategy.pdf 39 Iraq Rapid Assistance Program (IRAP) Evaluation Report ANNEX E: ADMINISTRATIVE DELAYS Figure 7: Approval to USAID Approval: Delay in Days for Each Project -150 -120 -90 -60 -30 0 30 60 90 120 150 180 210 240 270 2007/11/14 2007/12/14 2008/01/13 2008/02/12 2008/03/13 2008/04/12 2008/05/12 2008/06/11 2008/07/11 2008/08/10 2008/09/09 2008/10/09 2008/11/08 2008/12/08 2009/01/07 2009/02/06 2009/03/08 2009/04/07 2009/05/07 2009/06/06 2009/07/06 2009/08/05 2009/09/04 2009/10/04 2009/11/03 2009/12/03 2010/01/02 2010/02/01 2010/03/03 2010/04/02 2010/05/02 Grant Application to ETEC Approval Delay in days for each project days delay Application date Figure 8: Grant Application to ETEC Approval: Delay in Days for Each Project 0 30 60 90 120 150 180 210 240 270 300 2007/08/06 2008/02/22 2008/09/09 2009/03/28 2009/10/14 2010/05/02 ETEC Approval to USAID Approval Delay in days for each project days delay ETEC approval dates Iraq Rapid Assistance Program (IRAP) Evaluation Report 40 Figure 9: USAID Approval to Grant Signature: Delay in Days for Each Project -240 -210 -180 -150 -120 -90 -60 -30 0 30 60 90 120 150 180 210 240 2008/06/01 2008/09/09 2008/12/18 2009/03/28 2009/07/06 2009/10/14 2010/01/22 2010/05/02 2010/08/10 USAID Approval to Grant Signature Delay in days for each project days delay USAID approval dates Negative values indicate projects approved prior to obligatory COTR sign-off Figure 10: Grant Signature to First Disbursement: Delay in Days for Each Project -30 0 30 60 90 120 150 180 210 240 270 300 2007/08/06 2007/11/04 2008/02/02 2008/05/02 2008/07/31 2008/10/29 2009/01/27 2009/04/27 2009/07/26 2009/10/24 2010/01/22 2010/04/22 2010/07/21 Grant Signature to 1st Disbursement Delay in days for each project days delay Gant agreement dates 41 Iraq Rapid Assistance Program (IRAP) Evaluation Report ANNEX F: END NOTES “The logic behind PRTs is simple: Success in Iraq relies on more than military efforts; it requires robust political and economic progress. Our military operations must be fully supported and integrated with our civilian and diplomatic efforts across the entire U.S. government to help Iraqis clear, hold, and build throughout all of Iraq.” Secretary of State Condoleezza Rice, January 11, 2007. http://wws.princeton.edu/news/PRTgradpolicyworkshop/ i “Over nearly six years, the U.S. program had undergone an extraordinary evolution. What was originally conceived as a modest program to repair war damage and treat refugees had ballooned into an expansive and expensive nation-building effort. This in turn was supplanted by a counterinsurgency campaign and then a countrywide initiative to build Iraqi capacity. Constant re-evaluations of how U.S. resources could be employed to achieve the desired result of a stable Iraq led to a shift from large infrastructure reconstruction to a program that combined ’soft’ and ’hard’ projects aimed at mitigating security problems and building capacity.” Hard Lessons: The Iraq Reconstruction Experience, February 2009 SIGIR, Accessed 5th September 2010. http://www.sigir.mil/files/HardLessons/Hard_Lessons_Report.pdf ii “Stabilization and reconstruction operations: straddle an uncomfortable perch between conventional war￾fighting and traditional development assistance, both of which – and particularly the former – the United States can do well. These operations require a mix of skills and training addressing a range of issues, including establishing public security and the rule of law, facilitating political transitions, rebuilding infrastructure, and jumpstarting economic recovery. To complicate matters, stabilization and reconstruction missions must operate in far more demanding and often hostile environments than do traditional economic development programs. And they face narrow windows of opportunity to produce results. Stabilization and reconstruction encompasses military and civilian activities across the full spectrum of conflict.” Applying Iraq’s Hard Lessons to the Reform of Stabilization and Reconstruction Operations, February 2010, SIGIR, Page 3, Accessed 5th September 2010. http://www.wartimecontracting.gov/docs/SIGIR_ApplyingHardLessons.pdf iii “Poor interagency relations in Rome created misunderstandings about the purpose and focus of the PRT, resulting in de facto subordination of the civilian personnel to the military. This abdication of responsibilities to the military then caused a delay in the civilian participation in PRT operations. The US and Canadian experiences have also shown that institutional cultural friction sometimes causes turf battles between military and civilian personnel in the PRT.” Perito, R. et al. Provincial Reconstruction Teams: Lessons and Recommendations, Princeton University Woodrow Wilson School of International and Public Affairs, January 2008, Accessed 16th August 2010. http://wws.princeton.edu/research/pwreports_f07/wws591b.pdf iv ”The U.S. effort to rebuild Iraq and Afghanistan through local reconstruction teams lacks clear goals, organizational structure and lines of command, according to a new congressional report … They also recognized the dedication of individuals working on the teams, often under dangerous ‘personalities’ of staff individuals. It says that training is insufficient and that many staffers are unsuited for the jobs they are expected to perform.” Iraq Rapid Assistance Program (IRAP) Evaluation Report 42 The Washington Post, U.S. Effort to Rebuild from War Criticized, 18th April 2008, Accessed 15th August 2010 http://www.washingtonpost.com/wp-dyn/content/article/2008/04/17/AR2008041703701.html v “Prior to deployment (actually getting on the airplane) selected civilian and military personnel receive training from their own agencies and then come together for a two- to three-week cross-pollination exercise at a military base in the United States. Although not all team players are able to participate in the Forming phase of the pre-deployment, the attempt is well intentioned.” Kinton, Major Tom. Interagency Cultural Similarities in Iraq and Afghanistan Peacekeeping and Stability Operations Institute, Accessed 16th August 2010. http://pksoi.army.mil/PKM/publications/bulletin/volume2issue3/interagency.cfm vi “The most prominent of these trends is the failure to learn the lessons throughout this period such that the challenges and gaps identified in 2004 persist into 2009. Issues identified include the need for: • better defined mission objectives and transition strategies • integrated interagency training with greater input from subject matter experts • resolution of command and control issues and “culture clash” between civilians and military, and among civilian interagency partners • increased planning to integrate civil-military and interagency members • streamlined and integrated funding mechanisms • augmented host-nation involvement throughout the reconstruction and stabilization process • continuity of human resources and enhancement of institutional knowledge retention • coordination of and integration across the sectors and programs – breaking down stovepipes.” Luehrs, Christoff. Provincial Reconstruction Teams: A Literature Review, Prism vol. 1, December 2009 National Defense University Press, Accessed 21st August 2010. http://www.ndu.edu/press/lib/images/prism1-1/10_Prism_95-102_Luehrs.pdf vii USAID OIG Audit Report, 8,9,2009 43 Iraq Rapid Assistance Program (IRAP) Evaluation Report ANNEX G: CHARTS Table 4: Grants by Year and Status Year Total Amount No. of Grants 2008 15,024,133 63 2009 2,764,691 148 2010 4,761,402 18 22,550,226 229 Status Total Amount No. of Grants Cleared 2,537,577 12 Closed 18,675,219 215 Completed 1,337,430 2 22,550,226 229 63 148 18 0 20 40 60 80 100 120 140 160 2008 2009 2010 Figure 11: No. of CSCM Grants by Year Iraq Rapid Assistance Program (IRAP) Evaluation Report 44 15,024,133 2,764,691 4,761,402 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 14,000,000 16,000,000 2008 2009 2010 Figure 12: Value of CSCM Grants by Year $2,537,577 $18,675,219 $1,337,430 Figure 13: Grants Amount by CSCM Status (USD) Cleared Closed 12 215 2 Figure 14: Number of CSCM Grants Cleared Closed Completed 45 Iraq Rapid Assistance Program (IRAP) Evaluation Report _ _ $0 $5,000,000 $10,000,000 $15,000,000 $20,000,000 $25,000,000 $30,000,000 Figure 15: Distribution of QRF Grants by Province 0 200,000 400,000 600,000 800,000 1,000,000 1,200,000 1,400,000 1,600,000 1,800,000 2,000,000 Projects Amount (USD) Figure 16: Distribution of Grant Fund by Province (58 Projects) Iraq Rapid Assistance Program (IRAP) Evaluation Report 46 _ _ 80 28 17 26 37 43 48 229 46 14 16 39 63 15 22 31 31 24 0 50 100 150 200 250 Figure 17: Distribution of QRF Grants by Province 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 Projects Amount (USD) Figure 18: Distribution of Grant Fund by Province (58 Projects) 47 Iraq Rapid Assistance Program (IRAP) Evaluation Report _ _ 0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000 12,000,000 14,000,000 16,000,000 Total Amount (USD) Figure 19: QRF Qrants by Type 24 295 143 347 0 50 100 150 200 250 300 350 400 2007 2008 2009 No. of Projects Figure 20: Number of Projects by Year 2010 Iraq Rapid Assistance Program (IRAP) Evaluation Report 48 _ _ 19 362 58 105 33 37 44 48 1 29 47 26 0 50 100 150 200 250 300 350 400 No. of Projects Figure 21: Number of QRF Grants by Type 2,876,510 46,943,928 22,844,753 14,754,236 $0.00 $5,000,000.00 $10,000,000.00 $15,000,000.00 $20,000,000.00 $25,000,000.00 $30,000,000.00 $35,000,000.00 $40,000,000.00 $45,000,000.00 $50,000,000.00 2007… 2008 … 2009 … 2010… Total of Grants Amount Figure 22: Total Amount of Grants by Year