May 19, 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Rich Mason and Harvey Herr. USAID COMMUNITY STABILIZATION PROGRAM (CSP) COUNTERINSURGENCY (COIN): CIES AND BDP PROGRAM REVIEW IN KIRKUK USAID COMMUNITY STABILIZATION PROGRAM (CSP) – COUNTERINSURGENCY (COIN): CIES AND BDP PROGRAM REVIEW IN KIRKUK International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Program, Phase II (MEPP II) 8614 Westwood Center Drive Suite 400 Vienna, VA 22182 Contracted under 267-C-00-05-00508-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) i MEPP II CSP Kirkuk BDP and CIES Study Table of Contents LIST OF ACRONYMS...................................................................................................................II EXECUTIVE SUMMARY .............................................................................................................III I. INTRODUCTION ....................................................................................................................................1 II. PURPOSE OF THE KIRKUK CIES AND BDP ASSESSMENT ..............................................................1 III. ASSESSMENT METHODS ....................................................................................................................2 IV. FINDINGS...............................................................................................................................................1 V. RECOMMENDATIONS ........................................................................................................................17 VI. CONCLUSIONS ...................................................................................................................................17 ANNEX I: BDP MONITORING INSTRUMENT ...........................................................................18 ANNEX II: CIES MONITORING INSTRUMENT .........................................................................24 ANNEX III: FSO AND CSP RESPONSES TO IBTCI RECOMMENDATIONS. ..........................28 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) ii MEPP II CSP Kirkuk BDP and CIES Study List of Acronyms AoR Area of Responsibility (of Implementing Partners) AOTR Agreement Officers Technical Representative BDP Business Development Program BOQ Bill of Quantity CIES Community Infrastructure and Essential Services CoAg Cooperative Agreement COTR Contracting Officer’s Technical Representative CSP Community Stabilization Program DOE Directorate of Education DOH Directorate of Health DWR Directorate of Water Resources EGY Employment Generation and Youth FOM Field Operations Manual FSO Focused Stabilization Office GOI Government of Iraq HQ Headquarters HR Human Resources IBTCI International Business & Technical Consultants, Inc. M&E Monitoring and Evaluation MEPP II Monitoring and Evaluation Performance Program, Phase II MSME Micro, Small or Medium Enterprise PMP Performance Management Plan POD Program Office Director POO Program Operations Officer QAQC Quality Assurance and Quality Control SOW Scope of Work SPSS SPSS predictive analytics software www.spss.com USAID United States Agency for International Development USG United States Government International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) iii MEPP II CSP Kirkuk BDP and CIES Study EXECUTIVE SUMMARY In October 2008, International Business and Technical Consultants Inc.’s (IBTCI) Monitoring and Evaluation Performance Program, Phase II (MEPP II) team received a Scope of Work (SOW) from USAID/Iraq’s Focused Stabilization Office (FSO) requesting an assessment of the Community Stabilization Project (CSP) Kirkuk office Business Development Program (BDP) and Community Infrastructure and Essential Services (CIES) projects. The purpose of this assessment was “to determine if these projects and grants were properly managed and completed within the CSP Cooperative Agreement and International Relief and Development’s (IRD) own internal policy and guidance.” The SOW required the study to answer a variety of questions related to CSP Kirkuk Quality Assurance and Quality Control (QAQC), Monitoring and Evaluation (M&E), and BDP and CIES implementation procedures. The specific areas of focus under each question were negotiated in a series of meetings with FSO staff. The findings are noted below organized by question. 1. Were these programs and grants completed per the program guidance in the CoAg and internal IRD policies and directives (was program fidelity demonstrated)? Based on the results of site visits conducted for a random stratified sample of 82 completed BDP grants and for 23 completed non-irrigation CIES projects, Kirkuk programming in these two technical areas has been correctly and effectively implemented according to the program’s goals. One hundred percent of the monitored BDP grants were still in operation, with 95% of projects possessing all of the delivered grant materials months after the grant’s completion. Ninety percent (79) of the monitored business owners reported having attended IRD sponsored business training. All of the 23 monitored CIES projects were being utilized for the intended purpose and exhibited no observed quality deficiencies. 2. Were proper procedures in place to monitor project award or implementation? CSP Kirkuk CIES procedures assessed under this question included: the requirement for signed time sheets for CIES short-term workers; the verification of community and local government cost share; the reported cost share percentages in the tracking sheet; and the conduct of M&E site visits. For BDP projects, the assessment focused on the reported cost share percentages in the tracking sheets and M&E department site visits. Short-term worker documentation. The assessment found that all of the relevant Kirkuk CIES project files contained the appropriate documents for establishing the presence of reported numbers of short-term workers. These included worker-signed or thumb-printed daily and weekly sign-in and/or time sheets and monitor-signed daily worksite sheets verifying worker numbers and equipment. There was a shift in the format of time sheets in September 2007 from daily sign-in sheets with individual worker signatures accompanied by weekly time sheets not signed by the individual worker to weekly time sheets signed each day by the worker International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) iv MEPP II CSP Kirkuk BDP and CIES Study and a site monitor’s daily count of workers. Both methods appear to satisfy the requirement that there should be a daily signature from each worker in the project documentation. In addition, CIES project files include substantial numbers of photographs documenting the volume of short-term workers, as well as individual project management and monitoring staff, and clearly demonstrating project progress. CIES community and government contribution verification. In sum, all CIES project included the CIES Project Share Cost Form. QAQC site visits to confirm granttee contribution are conducted for only the BDP program and for contributions that are over $5,000. Therefore, the Kirkuk program office was unable to independently confirm the value of substantial amounts of proposed and reported community and local government cost share for the CIES program. The CSP Community Contributions Financial Review Summary Sheet was not employed until August of 2008 or later. As a result, the form was not included in the majority of CIES project files. This is most evident in the Phase I irrigation canal cleaning projects. The only cost share documentation provided to IBTCI consisted of letters in Arabic detailing proposed contributions including community cleaning of local sub-canals and the provision of Directorate of Water Resources (DWR) staff and equipment. Phase II irrigation projects contained some additional documentation, including time sheets for government workers, equipment provided and a certification letter from the DWR stating that the community had cleaned the local canals in the amounts and values as proposed. There was no additional documentation to show that IRD had verified this cleaning. Cost share documentation was more advanced for the local government building, school rehabilitation and water supply projects. These files included time sheets, DWR warehouse stock-out forms and, in the case of the water supply projects, photographs of the wells and pumps provided as contribution. IBTCI received conflicting guidance from CSP program office staff on the Kirkuk Office’s procedures relating to documenting and verifying cost share. IRD’s FOM and amended cost-share verification procedures post date the CIES projects reviewed. Hence, they did not provide a yardstick for measuring cost-share documentation. As a result, it is not included in the majority of CIES project files. Monitoring and evaluation site visits for CIES and BDP projects. IBTCI found that the Kirkuk M&E unit exceeded the minimum required number of M&E site visits for both project types and were collecting, documenting and reporting information that could be utilized for maximizing the quality and effectiveness of Kirkuk projects. According to the provided records, three-quarters (77.9%) of the sampled BDP projects and nearly 70% of the sampled CIES projects had been visited one or more times by M&E staff, well beyond the minimum levels established by even the most rigorous standards in CSP’s various guidelines. A review of the content of the M&E reports show that M&E officers conduct visits according to the standards established and described in CSP’s M&E Plan and Field Operations Manual. Visit reports included tallies of observed workers and equipment, descriptions of project progress, quality and other implementation shortfalls, beneficiary feedback and photographs of on-going or completed projects. Any follow-up from the M&E visit was not documented in project files. IBTCI was unable to assess how the M&E International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) v MEPP II CSP Kirkuk BDP and CIES Study findings were addressed by the project office and technical area management. This potentially valuable management information was missing. Reported cost-share percentages. IBTCI observed that the Kirkuk office reports the percentage of CIES and BDP project budgets covered by community cost share differently than what is described in the relevant project documents. In an April 29, 2008 document, IRD defined the “Total Project Cost” as the sum of CSP cost and community contribution as indicated in the project proposal. The CIES tracking sheet guidelines state that this percentage will be “the percentage of the completed amount against the cost of the total project” and the BDP tracking sheet guidelines state that the contribution percentage is “the percentage of the completed amount against the cost of the total grant project”. The CIES and BDP tracking sheets provided to IBTCI for Kirkuk calculated this percentage by dividing the local contribution by the CSP budget, resulting in a higher percentage of local cost shares. For example, for CIES 17900, the reported cost share percentage is 57% or $64,855 (verified contribution) divided by $114,022 (verified CSP cost) multiplied by 100. Calculated according to the revised guidance where total project cost is $178,877 (verified contribution + verified CSP cost) the cost share percentage would be 36% - $64,855 (verified contribution) divided by $178,877 (CSP cost of $114,022 plus the contribution of $64,855) multiplied by 100. 3. Does IRD have the procedures in place to implement an effective and independent QAQC program to ensure discrepancies are reported to program management? The formal Kirkuk QAQC program is limited in scope to three procedures, all of them focused on BDP grants. IBTCI’s assessment confirmed that these QAQC procedures are being effectively implemented and that program management receives information on identified issues. The QAQC unit in Kirkuk is a part of the M&E unit, utilizing the same staff and Director, and is not an independent unit as established in other offices. In the document review for sampled projects, IBTCI was able to see that QAQC procedures for the verification of CSP grant items were followed for 88% of the eight projects that were implemented after the establishment of this process and that qualified based on the amount of the grant. QAQC verification of grantee contribution was conducted for 86% of the seven eligible grants. QAQC Joint Final Closeouts were performed for 57 of the grants, all of which were completed after September 2007, providing useful additional data on the durability of BDP created jobs. The number of long-term jobs for the sampled grants observed during these Final Closeout site visits conducted by CSP QAQC and BDP staff, were almost identical to the numbers reported in the BDP tracking sheet. According to the numbers entered on the Final Closeout forms, seven BDP grants (11.9%) saw an increase in jobs compared with the proposed number, three (5.1%) experienced a decrease in jobs and 49 (83.1%) reported no change in the overall number of jobs. The total number of jobs across the 59 projects increased from 204 to 208. The number of female jobs observed at Final Closeout for these projects was nine less than the number projected in the proposals. At International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) vi MEPP II CSP Kirkuk BDP and CIES Study the same time, the number of male jobs at closeout increased by 13. The tracking sheets are not updated to reflect such changes. IBTCI was able to document the reporting procedures and communication chain for QAQC issues. In the case of the Joint QAQC Final Closeout Certifications, which include comments on issues with each grant, the POD’s signature is required along with those of the BDP and M&E/QAQC managers. These signatures demonstrate that relevant QAQC information is circulated through all levels of PO management and that issues should have been resolved to the satisfaction of the POD before that time. The Kirkuk M&E Unit performs tasks that can be considered as contributing to quality assurance for BDP and, more importantly in the absence of formal QAQC procedures, for CIES projects. IBTCI was able to document that the M&E Unit conducts site visits of BDP and CIES projects and prepares reports of their findings, including observed deficiencies in project implementation and employment figures that are sent to the POD and shared with the program managers. While it was reported that the POD follows up with relevant technical staff to address problems, documentation of such communication is not kept in the individual project or M&E files and IBTCI was unable to confirm actions taken in response to M&E findings. 4. Does the staff in the Kirkuk offices have the proper manpower, guidance, authority and procedures established to execute established M&E and QAQC roles and responsibilities? The Kirkuk M&E/QAQC unit director reported that he did not have adequate staffing to implement the full complement of QAQC procedures and forms that are utilized in larger CSP offices such as Baghdad/Karrada. As a result, a more limited QAQC role was established for the Kirkuk office and was implemented as part of a combined M&E and QAQC unit. As established under questions two and three of the SOW, these more circumscribed responsibilities and roles were effectively implemented by the six-person M&E/QAQC unit and the Kirkuk office’s minimum standards for M&E site visits and QAQC verification procedures were exceeded. However, the establishement of a combined M&E and QAQC unit is inconsistent with the formal requirement per Cooperative Agreement Modification 16 that was signed on March 19, 2009. Recommendations Establish the minimum required documentation for all relevant types of cost share for the project. Ideally, this should be done at the outset of such projects and the documentation guidelines should be revisited periodically to reflect changes or additions to the program’s project scope. While existing regulations such as 22 CFR 226.23 [Administration of Assistance Awards to U.S. Non-Government Organizations – Cost Sharing or Matching], are a useful starting point, they are minimum guidelines that do not provide detailed guidance for the universe of potential cost share types and corresponding documentation. Establish a system that allows key project results, such as reported long-term jobs, to be updated based on M&E and QAQC processes. The processes established for this should be uniform across the program offices, such as using data International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) vii MEPP II CSP Kirkuk BDP and CIES Study from initial M&E visits and/or final closeout jobs numbers for BDP projects instead of figures from the proposal. This may result in a delay in reported results to USAID and a need for additional personnel resources to conduct necessary follow-up and verification. Require that documentation be included in project files to demonstrate follow-up on identified M&E problems. For example, direct M&E to update their relevant M&E reports to reflect decisions that were taken to address concerns raised in previous reports, to store relevant emails in the project files, to make copies of minutes taken during meetings between the POD or POO, M&E staff and technical directors where such issues are addressed and action items identified. Conclusions CSP Kirkuk has implemented a large number of project processes designed to ensure accurate, effective, high-quality implementation of its CIES and BDP programming. Procedures evolved over time responding to identified program issues and evolving guidance from the CSP HQ offices. These include several quality assurance processes, documentation and implementation protocols and project monitoring systems. All of these elements produce useful, timely and valid data that can be utilized by program management to maximize the impact of CSP’s efforts to promote stability in the Kirkuk AoR. The data collected during this assessment confirms that Kirkuk’s efforts in these areas resulted in the successful completion of projects that are, for the most part, well monitored and documented. If the exceptions to this record of achievement are addressed successfully in other program offices, these offices and the resulting programming will be further strengthened. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 1 MEPP II CSP Kirkuk BDP and CIES Study I. INTRODUCTION On May 29, 2006, USAID/Iraq awarded cooperative agreement number 267-A-00-06-00503-00 to International Relief and Development (IRD) to implement. The primary objective of the Community Stabilization Program (CSP) falls within USAID/Iraq’s Transition Strategic Objective 7 “Focused Stabilization: Reduce the incentives for participation in violent conflict.” Initial funding under the CSP award limited activities to areas of significant insurgent activity in Baghdad. CSP was later expanded to cover similar areas in Tameem, Ninewa/Tel Afar, Ramadi, Falluja, North Babil and Diyala. Initially designed for two years, the project has been extended to February 28, 2010. The CSP is seen as a key element to transition Iraq to a stable, democratic and prosperous state. As defined in the Request for Application (RFA) the purpose of CSP is to complement military security efforts, and civilian local government development, with economic and social stabilization efforts. The design of the CSP project includes: 1) creation of jobs and development of employable skills with a focus on unemployed youth; 2) revitalization of community infrastructure and essential services; 3) support for established businesses and development of new sustainable businesses; and 4) help to mitigate conflict in selected communities. By carrying out these activities the CSP is expected to achieve measurable progress towards achieving the strategic objective. The CSP Performance Management Plan (PMP) identifies the measurable indicators that will evidence the achievement of the strategic objective. This assessment focuses on two technical areas, the Business Development Program (BDP), which supports established businesses and develops new ones, and the Community Infrastructure and Essential Services (CIES) program, which implements cleaning, construction and rehabilitation projects to improve the delivery of services and provide short-term employment opportunities. The assessment is specifically concerned with documenting internal program procedures and protocols in the Kirkuk Program Office, specifically those related to monitoring and evaluation (M&E) and quality assurance and quality control (QAQC), and evaluating concomitant compliance. II. PURPOSE OF THE KIRKUK CIES AND BDP ASSESSMENT In October 2008, International Business and Technical Consultants Inc’s (IBTCI) Monitoring and Evaluation Performance Program, Phase II (MEPP II) team received a Scope of Work (SOW) from USAID/Iraq’s Focused Stabilization Office (FSO) requesting an assessment of the CSP Kirkuk office BDP and CIES projects (Annex I). The purpose of this assessment was “to determine if these projects and grants were properly managed and completed within the CSP Cooperative Agreement, and IRD’s own internal policy and guidance.” The SOW required the study to answer a variety of questions related to CSP Kirkuk QAQC, M&E, and BDP and CIES implementation procedures. The specific areas of focus under each question were negotiated in a series of meetings with FSO staff. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 2 MEPP II CSP Kirkuk BDP and CIES Study Study Questions from the SOW 1. Were these programs and grants completed per the program guidance in the Cooperative Agreement (CoAg) and internal IRD policies and directives? 2. Were proper procedures in place to monitor project implementation and completion? 3. Does IRD have the procedures in place to implement an effective and independent QAQC program to ensure discrepancies are reported to program management? 4. Does the staff in the Kirkuk offices have the proper manpower, guidance, authority and procedures established to execute established M&E and QAQC roles and responsibilities? III. ASSESSMENT METHODS This assessment combined multiple methods including in-depth unstructured interviews with expatriate and local office staff; a review of written CSP Headquarters (HQ) and Kirkuk operating procedures; a structured review of individual project files and documentation; and site visits with a sample of CIES and BDP projects. A detailed list of methods by study questions is included in Annex II. In-depth Interviews in the Kirkuk Offices An IBTCI expatriate staff member travelled to the CSP Kirkuk offices a total of three times in March of 2008 from a base in Erbil. During these visits and an additional meeting in Erbil, he conducted interviews with the current Program Office Director (POD), the previous POD, the M&E Director, the CIES Infrastructure unit Director, the CIES Essential Services unit Director, and the BDP Director. The interviews focused on several tasks essential to the SOW: • Documenting the QAQC and M&E procedures being utilized in the Kirkuk office, including the staffing structure, methodologies, instruments, data entry and analysis and results reporting to office and unit management. • Documenting CIES and BDP implementation, management and monitoring practices. • Establishing CIES and BDP documentation procedures, including the locations and formats of project files and the specific content of individual project records. Each of these tasks included a thorough, hands-on review with CSP local staff of the program office records in question to ensure that the IBTCI staff member possessed as complete an understanding as possible of the relevant procedures, content and results. CSP Procedures Review IBTCI requested, received and reviewed multiple CSP procedural documents. The purpose of this review was to supplement the information provided through the program staff interviews on project guidelines related to the procedures of interest in the assessment. These documents included the CSP Field Operations Manual (FOM) and International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 3 MEPP II CSP Kirkuk BDP and CIES Study the included attachments, the CSP M&E plan, and the CSP Cooperative Agreement and Modifications. A complete list of reviewed documents can be found in Annex III. The review of procedures was complicated by differences between the written protocols and the verbal descriptions from CSP staff as well as between various versions of the written guidelines. The evolution of protocols and procedures over time created additional challenges in determining the correct standard against which to judge actual practices. Project File Review IBTCI reviewed the complete scanned project files for all completed CIES projects in Hawija and a sample of 82 closed out BDP projects. The purpose of the file review was to confirm that CSP QAQC, M&E and project implementation and documentation procedures were implemented as described by CSP personnel and in CSP guidelines. The file review was not intended to be a financial audit focused on comparing specific reported expenditures against documented receipts. Instead, the file review was focused primarily on the documentation of short-term workers; the verification of CIES cost share; the implementation of Kirkuk QAQC and M&E procedures; and the use of resulting QAQC and M&E data for program management. The total volume of pages reviewed by IBTCI for this assessment includes thousands of scanned items such as timesheets, receipts, site-visit forms, photographs, proposals, bills of quantity (BOQ) and other documents. The original SOW was predicated on several assumptions that proved to be inaccurate and that complicated the task of reviewing project documentation. The first incorrect assumption was that all relevant records would be present at the Kirkuk offices. In January 2008, a substantial number of files for sampled projects were sent to IRD’s storage facilities in Amman, Jordan for scanning and archiving. This had two consequences that hampered the assessment. First, it meant that the consultant was not able to review representative records from early in the life of the project and, as a result, was not exposed to the record-keeping procedures that were in effect at the time. Second, the lack of project files and the need to have them sent to IBTCI’s office in the IZ necessitated a change of strategy. A decision was made to have IBTCI expatriate staff conduct the document review, in lieu of the original project design of utilizing Iraqi data collectors to review a random sample of project files with a standardized checklist. The second assumption was that the assessment could be completed before the Kirkuk office had closed. Due to delays in the receipt of project files and the resulting difficulties described above in establishing the content of some records, the Kirkuk office began closeout before the assessment was complete and while important questions relating to implementation and documentation procedures were still unanswered. The resulting multi-week suspension of communication from the Kirkuk POD and local staff, some of whom left the project, contributed to considerable delay in completing the assessment. Survey Instrument Design Two field surveys were conducted – one of BDP grants (Annex IV) and one for CIES projects (Annex V) – with the intent to verify the successful implementation of CSP Kirkuk programs as required under the first question of the SOW. The CIES site instrument was designed to answer three key questions: had the projects been International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 4 MEPP II CSP Kirkuk BDP and CIES Study implemented as described in the project proposals, did the sites appear to have been implemented according to an acceptable level of quality and were the sites being utilized as intended. While time constraints did not allow for a rigorous assessment of project quality against the detailed BOQs, IBTCI has successfully monitored CIES projects in the past using the more general approach with non-specialist monitors. Because of USAID’s request that IBTCI monitor only completed projects in a particular geographic area, the CIES survey was not able to verify or assess the reported short-term employment figures. The BDP questionnaire was adapted from previous surveys that IBTCI has utilized in its extensive previous fieldwork with CSP BDP grants monitoring. The survey instrument focused on two main elements: the presence of grant materials delivered by IRD to grantees and the creation and durability of jobs supported by the grants. Both of these surveys were intended to establish whether CSP had successfully delivered on its program goals of developing community infrastructure, improving essential services and providing short and long-term employment. Survey Sample A total of 24 CIES projects were selected for on-site monitoring by IBTCI monitors. The list of CIES projects that were included in the Kirkuk assessment was provided by USAID and included all completed projects in the Hawija sub-district of Tameem province. After interviews with CSP program staff, a thorough review of program files and informal interviews with various PRT members, a decision was made to exclude irrigation canal cleaning and reconstruction projects from the site visits. This decision was based on concerns that, due to the temporal nature of expected improvements in the removal of vegetation and silt from what were mostly unlined earthen ditches, any improvements produced by the projects would not be visible on-site six months or more later. The one street-cleaning project on the Hawija project list was also excluded based on similar reasoning. The sample of BDP projects was taken from the February 20, 2009 tracking sheet, which contained a total of 829 completed projects. A stratified random sample of 86 projects consisting of 40 micro grants, 40 small grants and all six medium grants was selected to ensure that the study looked at a representative mix of completed grants. Implementation of the Surveys and Survey Non-Responses Monitors made every effort to determine the disposition of the projects and grantees selected by the sample in order to minimize non-response. For BDP grant recipients, field monitors made at least four attempts to contact the grantee or neighbors who might know the whereabouts of the grantees. MEPP II monitors were careful not to confuse non-responses with enterprises that were no longer in business. A non-response for purposes of this survey means that IBTCI was unable to locate the grantee or access a CIES project. Non-respondents are not included in the data analysis. It is accepted that approximately one in ten persons has been displaced (International Organization for Migration estimations of Iraq internally displaced persons (IDP) during the war and insurgency so that movement of grantees from previously known address is not considered unusual. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 5 MEPP II CSP Kirkuk BDP and CIES Study Out of the 24 selected CIES projects, IBTCI monitors located and visited all but one. For security reasons monitors did not access project CIES 370/7323 renovation of the Municipal building in Al Rashad. The disposition of the remaining projects by project type is included in Table 1 below. Table 1. Monitored CIES Projects by CSP Sector Sector Number Percent Local Government 8 35 Education 7 30 Water/Sewer 6 26 Healthcare 1 4.5 Other 1 4.5 Total 23 100 For the BDP grants, actual survey response was 95% (82 out of 86). Of the four non￾respondents, one refused the interview, one was not home at the time of the survey visit, one had been arrested by security forces and one was reported as being out of the country. The resulting completed surveys included the following distribution of project type of grant size shown in Table 2. Table 2. Survey Responses by Grant Size and Sector Grant Size/ Business Sector Medium Small Micro Total Agriculture 1 14 0 15 Industrial/Manufacturing 5 9 4 18 Trade 0 3 18 21 Service 0 11 17 28 Total 6 37 39 82 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 1 MEPP II CSP Kirkuk BDP and CIES Study IV. FINDINGS 1. Were these programs and grants completed per the program guidance in the CoAg and internal IRD policies and directives? According to site visits conducted for a random sample of completed BDP grants and for all completed non-irrigation CIES projects, Kirkuk programming in these two technical areas has been correctly and effectively implemented according to the programs goals. One hundred percent of the monitored BDP grants were still in operation, with 95% of projects possessing all of the delivered grant materials months after the grant’s completion. All of the 23 monitored CIES projects were being utilized for the intended purpose and exhibited no observed quality deficiencies. CIES Monitoring Findings The CIES site monitoring confirmed that Hawija projects were implemented successfully achieving the intended results. Based on the monitors’ evaluation of the level of completeness and construction quality for the six or seven main components assessed for each project, 100% of the 23 visited CIES projects were reported as having been completed to an acceptable standard of quality in all assessed areas. Furthermore, there were no quality concerns or construction deficits identified for follow-up by engineers. All of the projects were being utilized for their intended purpose, such as schools or municipal buildings, and were delivering improved water services in the case of water supply projects. Monitors comments indicate that the projects were well received by beneficiaries and no defects or deficits were identified that require follow-up by IRD. BDP Monitoring Findings Of the 82 grantees surveyed, all of them still operated their original type of business. IBTCI monitors found that two of the 82 businesses were missing equipment that was delivered by IRD. One of these businesses reported selling the equipment because of low demand for the service and one reported that it was at their house, which was not confirmed. Three of grantees reported not using some of their equipment, one because the equipment was broken, one because of a lack of demand for the service and one because they did not need to use the equipment (a heater) because cool weather had not yet arrived. Long-Term Jobs (LTJ) The BDP monitoring survey included detailed questions about the total number of workers employed in the businesses at the time of the site visit. Comparisons with the reported numbers of long-term jobs from the tracking sheets and the number observed by QAQC at final closeout show that, including part-time jobs, the average employment increased in the surveyed businesses. A net total of 26 additional workers (319), full and part-time, were observed by the IBTCI monitors compared with the number reported in the tracking sheets (293). If part-time International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 2 MEPP II CSP Kirkuk BDP and CIES Study workers are excluded from this analysis, there was a net loss of 13 jobs (280) compared with the reported figures. Table 3. Difference in Observed Jobs and Reported Jobs in Tracking Sheet Difference Frequency Percent -4 2 2.4 -3 2 2.4 -2 5 6.1 -1 7 8.5 0 33 40.2 1 19 23.2 2 8 9.8 3 3 3.7 4 2 2.4 5 1 1.2 Total 82 100.0 A net total of 13 additional workers (219), full and part-time, were observed by the IBTCI monitors compared with the number reported at final closeout (206). If part-time workers are excluded from this analysis, there was a net loss of 13 jobs (190) compared with the reported figures. Table 4. Difference in Observed Jobs and Reported Jobs at Final Closeout Difference Frequency Percent -4 2 3.4 -2 3 5.2 -1 8 13.8 0 25 43.1 1 13 22.4 2 3 5.2 3 2 3.4 5 2 3.4 Total 58 100.0 According to the survey responses, IRD provided follow-up grant visits for the majority (95%) of grantees. The reported number of visits ranged from three to 15 with an average number of six visits, which was also the number mentioned the most frequently (18 respondents). CSP Kirkuk staff reported to IBTCI that 100% of their BDP trainees were required to attend business training. Nearly 90% of the grantees reported that they had in fact attended IRD-sponsored business training. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 1 MEPP II CSP Kirkuk BDP and CIES Study 2. Were proper procedures in place to monitor project award or implementation? CSP Kirkuk CIES procedures assessed under this question included: the requirement for signed time sheets for CIES short-term workers; the verification of community and local government cost share; the reported cost share percentages in the tracking sheet; and the conduct of M&E site visits. For BDP projects, the assessment focused on the reported cost share percentages in the tracking sheets and M&E department site visits. These procedures were identified through meetings and consultation with USAID FSO staff that requested the assessment. Short-term worker documentation. In the Regional Inspector General (RIG) audit of the CSP conducted in mid to late 2007 and issued in May 2008, the inspectors expressed concern about the absence of documentation related to short-term employees on CIES projects. CIES projects in Kirkuk were implemented in two ways: direct implementation where CSP is responsible for overseeing the project and hiring and paying employees; and indirect implementation, which contracts with a local business. For indirect implementation the CSP FOM provides the following guidance: “Contractor shall be instructed to submit a daily report stating the progress of work and number of deployed manpower and machinery.” (CSP Field Operations Manual, pg. 45, April 29, 2008) According to CSP Kirkuk staff, the same standard of documentation is applied to projects that are directly implemented by IRD. On-site monitors provide a daily report with the requisite information and are responsible for ensuring that hires sign time sheets and/or attendance logs. The assessment found that all of the relevant Kirkuk CIES project files contained the appropriate documents for establishing the presence of reported numbers of short-term workers. These included worker-signed or thumb-printed daily and weekly sign-in and/or time sheets and monitor-signed daily worksite sheets verifying worker numbers and equipment. There was a shift in the format of time sheets in September 2007 from daily sign-in sheets with individual worker signatures accompanied by weekly time sheets that were not signed by the individual worker to weekly time sheets signed each day by the worker and a daily count of workers provided by a site monitor. Both methods appear to satisfy the requirement for a daily signature from each worker in the project documentation. In addition, CIES project files include substantial numbers of photographs documenting the volume of short-term workers, as well as individual project management and monitoring staff, and clearly demonstrating project progress. CIES community and government contribution verification. IBTCI, in consultation with the USAID FSO team, focused much of its assessment of CSP Kirkuk procedures on CIES community and local government cost share. The first part of the assessment required that IBTCI establish the procedures in use for documenting and/or verifying the reported cost share amounts. The second part of the assessment included a detailed review of each individual CIES project file to determine if procedures had been followed. This was a difficult task for several reasons. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 2 MEPP II CSP Kirkuk BDP and CIES Study CIES community contribution verification policies and procedures, like many others, have varied over the life of the CSP project in Kirkuk. According to interviews with project staff, including the Program Operations Officer (POO) and POD in charge during the development, approval and implementation of the CIES projects included in this assessment, the initial guidance received from the CSP senior management in the IZ was that letters from the appropriate GOI authority detailing the proposed contribution from their office and from the community, would be sufficient documentation. This guidance reportedly changed, most likely because of the RIG audit, during late 2007, to include more rigorous documentation, such as: signed time sheets for workers and equipment; photos of delivered items; warehouse receipts; and other documents. The most formal written guidance for community contribution provided to IBTCI as part of this assessment is a four-page document entitled “Community Contributions Directive” (Annex VI), circulated by CSP HQ management, which was sent to CSP offices in April/May of 2008 and inserted in the CSP FOM issued around the same period. Excerpts from this document are inserted in the text box below. It is important to note that, while the examples in this document of required documentatin for cleaning campaign projects are specific, there are no guidelines providd relating to documentation for irrigtion canal cleaning projects. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 3 MEPP II CSP Kirkuk BDP and CIES Study Requirements for Documenting the Cost Share: • Volunteer services shall be documented and, to the extent feasible, supported by the same methods used by CSP for its own employees. • The basis for determining the valuation for personal services, material, equipment, buildings and land shall be documented. • MOUs, letters of agreement, contracts, formal contribution letters, and time sheets, lease, and services receipts are to be used as formal documentation. • Grantee Application Form, Grant Proposal Form, Verification of Contribution, and Site Assessment Form” (CSP Community Contributions Directive, pg. 2, April 29, 2008) Guidance for Specific Types of Projects: • “Cleaning campaign projects. For cleanup campaigns, CSP provides all necessary tools and labor to clean the streets and remove the trash and debris to the dump station; the local municipality might cost share by providing machineries for cleaning the sewer system for the same neighborhood. The required documentation in this case will be the timesheets for the municipality’s machineries and labors engaged in municipality work. A letter of understanding with the local authorities regarding the cost share will be preferred. • Irrigation canals restoration projects. As CSP restores the canal, the community may rehabilitate the pump station, the ministry of water resources may provide technical supervision or the local council may provide workforce.” (CSP Community Contributions Directive, pg. 2, April 29, 2008) • “All contribution documents are to be translated. The translation can be done on the margins of the letter or document or translated on a separate document and stapled with the document.” (CSP Community Contributions Directive, pg. 4, April 29, 2008). Another version of this document, entitled “CIES Cost Share Guideline” was provided to IBTCI late in the assessment by the Kirkuk POD (Annex VII). The main difference between the two versions is that the four numbered points listed above were reduced to include only the first two items. The guidance for the specific types of projects was unchanged. Both versions include the guidance that all of these records should be kept in the CIES project files and should be translated into English. There is no guidance in either version that these requirements should be applied retroactively. At the same time, an additional form was developed, the CSP Community Contributions Financial Review Summary Sheet (Annex VIII). The use of this form and other relevant International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 4 MEPP II CSP Kirkuk BDP and CIES Study procedures are detailed below in an excerpt from the CSP Field Operations Manual of May 2008: “This form (CSP Community Contributions Financial Review Summary Sheet) is devised to clarify and summarize all financially related aspects of CSP projects including linking and reporting Community Contributions to the financial reporting requirement of CSP. Thus through this form, CSP finance staff will be able to post and include community contributions in the quarterly financial reports sent to USAID. The form will be an integral component of each project file and is to be regularly updated by Finance staff. The form is to be applied immediately for all open and ongoing projects from now on.” Once a document outlining the contribution by the project beneficiaries is received and filed, Program Field Officers in charge of the project are responsible to alert the designated finance staff who in turn will do the necessary posting of the specific contribution value as outlined in the contribution document after registering the exchange rate from Iraqi Dinars to USD. The form will also indicate all other financial related matters such as payments made and variation orders. Each document received verifying community contribution, must be properly and clearly translated and shared with Finance in each unit which in turn records the amount in the “Financial Review Summary Sheet” form in USD value as per the exchange rate to the Iraqi Dinar on that specific date and posting the amount in their QuickBooks. Additional instructions will follow to finance staff clarifying the posting of community contributions in their QuickBooks. Each POD or his designated person and the Finance Manager in each unit will sign the form at the completion of the project insuring that all related documents are enclosed and properly translated. Each time a contribution document is received and registered in the form Program Field Officers will alert the M&E staff in each unit to update the contribution figures in the tracking sheet.”(CSP Field Operations Manual, pgs. 97-98, May 29, 2009.) The CIES Project Cost Share Verification form (Annex IX) was utilized and was included in the project files about mid-2008. The form required the signature of the POO and other management staff and included a list of supporting documentation for cost share types and amounts. The form was present in seven of the 43 Kirkuk CIES projects. All but one of these projects was completed in August of 2008 or later. The one project that was completed earlier than this, CIES 559/10232, included a cost share form that was dated March of 2009. In most instances where these forms were utilized, the dollar amounts for cost share indicated on the forms matched the amounts provided in the accompanying documentation. The one exception to this was CIES 959/21076, which was missing two time sheets listed on the verification form. For CIES 559/10232, the dollar amounts matched the description of the provided cost share, which included furniture from the DOE for the newly renovated school. Unlike the other similar projects, however, the warehouse stock-out forms were not included in the record. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 5 MEPP II CSP Kirkuk BDP and CIES Study One of the challenges in reviewing the CIES contribution component of the monitoring exercise was determining exactly what the scope and value of the contributions should be from the documents provided. IRD procedures in Kirkuk did not, apparently, call for a detailed description of the local and government contributions as part of the project proposal. As observed in the file review, the proposal usually includes a one or two sentence description of the community contribution proposed and a table listing the breakdown of contribution by line items, such as labor, equipment, etc. Project proposal procedures appear to be different for different types of CIES projects. For the indirect implementation projects, there is some detail about both the community and local government cost shares including the number of workers and their daily wage and the type and daily value of equipment contributions. For the direct implementation projects, such as irrigation canal cleanings, there is no information about the contribution in the proposals themselves beyond the dollar amounts in the table. Additional information about the scope of the proposed contribution, where it exists, is contained in a separate document that may or may not have been attached to the overall proposal. The type and presence of cost share verification documents for CIES projects varied by the implementation date and project type. The first factor appears to be a result of evolving guidance from the CSP policies and procedures and USAID related to the RIG CSP Audit that was issued in late 2007. Variation by project type is to be expected as the type of contribution changed based on the nature of the project. Table 5. Cost Share Verification Documentation present in CIES Files Project Type Contribution Description in Proposal Proposed Contribution Letter from GoI Verified Contribution Letter from GoI Time Sheets for Community Personnel Time Sheets For GoI Personnel Time Sheets for Equipment Store-Outs On-site Delivery Receipts Photos of Contribution Verification of Applicant Contribution Form Irrigation Canals Phase 1 No Yes No No No No N/A N/A No No** Irrigation Canals Phase 2 No No Yes No Yes Yes No No No No** School Renovations Yes Yes No N/A Yes Yes Yes N/A No Yes Municipal Building Renovations Yes Yes No N/A Yes Yes Yes N/A No Yes Water Network Yes Yes Yes N/A N/A N/A N/A N/A Yes No** Hospital Sanitation Yes Yes Yes N/A N/A N/A N/A Yes No Yes Rubble Removal No Yes No N/A Yes Yes No No No Yes *N/A indicates that the documentation type is not applicable to the type of contribution ** Indicates that the form was reportedly not a requirement at the time of closeout International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 6 MEPP II CSP Kirkuk BDP and CIES Study For example, CIES 739/8323, which consisted of a sanitation and cleaning campaign in Hawija Hospital that was conducted from December 2007 to January 2008, cost share consisted of the monthly provision of the requisite cleaning supplies on a monthly basis. Verification documents contained in the file included: an itemized list of monthly needs and costs for the five area hospitals signed by the local Directorate of Health (DOH) Director of Administration and the Director of Monitoring and Inspection; four itemized receipts for delivery of cleaning supplies signed by the Hospital Director, IRD’s Supervising Engineer, the Contractor and the DOH’s Health Engineer; a signed and stamped letter from the Director of Monitoring and Inspection certifying the proposed total and monthly dollar amounts of the contribution for the five hospitals; and a letter of appreciation from the DOH for IRD’s assistance and expressing their willingness to continue the project and provide additional contribution as required. Based on the detailed documentation, it was possible to calculate the value of the contribution, 27,003,100 ID ($2310 at an 1170 dinar to $1 rate), which is close to the $2,859 value listed by IRD. Cost-share documentation for CIES 910/18762, 911/18775, 912/18780 and 559/10232, which were school renovation projects where the Directorate of Education (DOE) provided desks, blackboards and other furniture, consists of a different set of verification documents. This is despite the fact that the nature of the type of cost share is similar: products provided by a local Directorate office. For each of these projects, there is a letter in Arabic listing the items that will be provided by the DOE along with warehouse receipts for the issuance of similar materials (the proposed items and the issued items was not always the same). The cost share for these projects was well documented, including descriptions in the proposals and, where required, warehouse issuance receipts or “store-outs” for provided materials. There was one potentially questionable school renovation cost share amount for CIES 539/16861 in the village of Al Manzila. For this project, IRD agreed to complete a school that was partially constructed in 1998. Forty thousand dollars of the $46,000 cost share was apparently credit for the construction costs incurred ten years ago, which are detailed in an updated BOQ included in the file. CSP implemented a series of municipal building renovations for which the cost share consisted generally of labor, equipment and materials to furnish the completed structures. For these projects, the proposals contain a brief description of the contribution, which is further documented in a letter in Arabic, with an English translation, detailing the specifics of the cost share. For the water projects, such as CIES 860/15285 and 861/15284, the local contribution consisted of pre-drilled wells with pumps, generators and other complimentary machinery, which generated the water utilized in the networks installed by IRD. Verification consisted of: letters from the DWR describing the content and value of the well packages, letters verifying their completion in each location so IRD could begin their work, and pictures of the completed wells. Irrigation projects demonstrated some of the highest values for “verified” local contribution as well as the least comprehensive documentation practices for cost share. None of the irrigation project proposals provided to IBTCI included a description of the cost share within the proposal itself, with the exception of dollar values in the budget tables. These were often not broken down by the listed subcategories, such as unskilled International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 7 MEPP II CSP Kirkuk BDP and CIES Study labor and equipment and they grouped community contribution values with those from the local government. Phase I project files included an Arabic-only description of the cost share provided by the DWR in a separate document. Phase II projects had neither. Phase I irrigation canal projects have the smallest amount of cost share documentation, consisting of only a letter detailing proposed contribution prepared by the DWR. The main value of the contribution, $50,000 in many cases, comes from the cleaning of local sub-canals by farmers. This contribution was not verified during or after the project by either the DWR or IRD based on the provided documentation. Additional proposed contribution consisted of the provision of DWR engineers and DWR equipment; presumably, to assist with either the IRD-led or community-led canal cleanings. In the files provided to IBTCI, there were no documents verifying that this contribution was provided either. Phase II irrigation projects included some additional documentation and were also missing documentation present for the phase I projects. The Phase II files included a sub-folder labeled “DWR Contribution”, which included time sheets for the one or two staff and equipment provided by the DWR. Additionally, a letter was provided from the DWR signed March 19, 2009 that confirmed the amounts and values of the sub￾channels that were cleaned as part of the community contribution, valued at $50,000 for each project. Missing from the files was the letter of proposed contribution as well as any time sheets for community members. These are expected because the total contribution amount listed in each proposal was placed in the unskilled labor column of the Community Contribution table, despite the fact that it apparently included, based on the total contribution and the included DWR time sheets, thousands of dollars of DWR labor and equipment that should have been in the adjacent table. In sum, all CIES project included the CIES Project Share Cost Form. QAQC site visits to confirm granttee contribution are conducted for only the BDP program and for contributions that are over $5,000. Therefore, the Kirkuk program office was unable to independently confirm the value of substantial amounts of proposed and reported community and local government cost share for the CIES program. The CSP Community Contributions Financial Review Summary Sheet was not employed until August of 2008 or later. As a result, the form was not included in the majority of CIES project files. This was most evident in the Phase I irrigation canal cleaning projects where the only documentation provided to IBTCI consisted of letters in Arabic detailing proposed contributions including community cleaning of local sub-canals and the provision of DWR staff and equipment. While Phase II irrigation projects contained some additional documentation, including time sheets for government workers and equipment and a certification letter from the DWR stating that the community had cleaned the local canals in the amounts and values as proposed, there was no additional documentation to show that IRD had verified this cleaning. Cost share documentation was more advanced for the local government building and school rehabilitation and water supply projects. These files included time sheets, DWR warehouse stock-out forms and, in the case of the water supply projects, pictures of the wells and pumps provided as contribution. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 8 MEPP II CSP Kirkuk BDP and CIES Study Limitations of the Cost-Share Assessment As IBTCI’s expatriate staff are not trained auditors, they are not conversant in the exact requirements of USAID’s regulations in regards to cost share documentation. It is unclear from the CoAg whether USAID can and should expect and require that IRD staff provide independent verification and certification of all elements of reported cost share. This is complicated by CSP’s use of the terms ‘verified’ and ‘verification’ in relation to contributions in the tracking sheets and other guidelines. On its face, ‘verification’ implies that IRD is reasonably sure that these amounts are accurate and can demonstrate how the figures were calculated and confirmed by CSP staff. By this standard, much of the reported cost share for Kirkuk CIES can be questioned, especially the large dollar amounts provided by communities’ commitments to clear local sub￾channels in Phase I and Phase II of the irrigation projects. The main challenge encountered for addressing this component of the assessment centered on difficulty of establishing the exact procedures that were in effect at the time that these projects were designed and implemented. The only written guidance on cost share documentation procedures that was available to IBTCI staff was dated April 2008, more than one year after most of the CIES projects of interest were completed. Even this document, however, provided little guidance for irrigation projects. Additionally, the oldest CIES project files had been sent to Amman, Jordan in January 2008 for archiving and were not available for on-site review with CIES staff during IBTCI’s visit to the Kirkuk offices. This proved to be a substantial challenge as the documentation and file organization procedures in the files for sampled projects that were reviewed during this visit were not the same as in many of the files that were offsite, resulting in difficulty in understanding the expected contents of the files and establishing that IBTCI was provided with all relevant documentation. Another important complication in the process was the closeout of the Kirkuk office in March and April. This resulted in the suspension of communication with Kirkuk staff for several weeks and the loss of access to key CIES officers for clarification. Finally, the type of cost share documentation depended on the type of cost share provided. Documentation varied across the sectors and the project implementation mechanisms. As a result, documents were acquired from various locations and, as a result, extended the period of time needed for document review. Monitoring and Evaluation Site Visits for CIES and BDP projects. According to interviews with CSP M&E Staff in the International Zone and in the Kirkuk PO, CSP sets a minimum standard for M&E site visits for each technical program. CSP staff and written documents provided a variety of different guidance on the frequency of these visits. Written CSP guidelines contained within the CSP FOM dated May 29, 2008 and CSP M&E Plan dated May 15, 2008 provided different guidelines. For example, on page 102 of the FOM, it states that a minimum of 10% of all projects need to be visited, presumably once, to ensure a statistically valid sample. In the M&E plan, this figure increases to 20% (pg. 21). Both of these assumptions are predicated on the use of an actual random sampling technique. The document did not provide guidance on how projects should be randomly selected for monitoring visits. The Kirkuk M&E Manager confirmed that the sampling technique utilized in their office was not random but based on a variety of factors including requests from management, project size, project location, and other considerations. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 9 MEPP II CSP Kirkuk BDP and CIES Study The CSP M&E plan provides additional specifications for the content and timing of monitoring visits in a table on page 18. According to Table 6, CIES Public Works Projects should be visited at least three times and Essential Services Projects (ESP) should be monitored in the field only once. Infrastructure projects (INF) should receive two to three visits. For BDP projects, two site visits are listed. Further guidance is provided in another table on pages 23 and 24 where it states that M&E staff should visit 30% of CIES projects and 40% of BDP projects, presumably according to the guidelines listed on page 18. These minimum standards are rather low and, as a result, position IRD to easily exceed the standards. Of course, the most influential factor in determining the performance of M&E staff in terms of monitoring visits at the local level is how all of these guidelines are interpreted and implemented by the M&E Manager. According to the Kirkuk M&E Manager, 30% of CIES and 40% of BDP projects should be visited at least once or twice. Furthermore, the M&E Manager reported that his staff had conducted a total of 1,510 site visits as of February 22, 2009 with 85% of CIES and 66% of BDP projects visited at least once. In order to assess the actual frequency and contact of M&E site visits, MEPP II staff requested copies of all M&E site visit reports for the sampled BDP and CIES projects. The dates of each visit and total numbers of visits were recorded for each project and are presented in Table 6 below. Table 6: Number of Reported Site Visits by Program Type # of Visits BDP (N=86) ESP INF Total CIES (N=43) 0 19 (22.1) 7 (36.8) 1 (4.2) 8 (18.6) 1 61 (70.9) 10 (52.6) 8 (33.3) 18 (41.9) 2 6 (7.0) 0 (0) 12 (50.0) 12 (27.9) At least 2 67 (77.9) 10 (89.4) 20 (83.3) 30 (69.8) 3 or more 0 (0) 2 (10.6) 3 (16.7) 5 (11.6) Total 86 (100) 19 (100) 24 (100) 43 (100) According to the provided records, three-quarters (77.9%) of the sampled BDP projects and nearly 70% of the sampled CIES projects had been visited one or more times by M&E staff, well beyond the minimum levels established by even the most rigorous standards in CSP’s various guidelines. Content of Monitoring and Evaluation Site Visit Reports M&E officers utilized a standardized monitoring form to guide site visits for BDP and CIES projects. According to interviews with the Kirkuk M&E Director, templates for these forms and guidelines for preparing subsequent reports were provided by CSP HQ. The M&E Director and unit staff received training in the use of these forms conducted by CSP HQ staff at the onset of the CSP program in Kirkuk. The CSP FOM includes the following guidance concerning M&E reports and their use: International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 10 MEPP II CSP Kirkuk BDP and CIES Study M&E site visit reports: “For each visit, a brief monitoring report is prepared and reviewed by the M&E Manager for each location and then forwarded to the respective POD and POO. The POD and POO then shares those reports with the relevant technical staff to ensure that M&E lessons learned are fully shared with all stakeholders and that suggestions for improvement are widely shared.” (CSP FOM, April 29, 2008, pg. 102) The specific focus of site monitoring visits is summarized in the FOM as: “Typically, site monitoring of CSP projects would involve the following types of activities: • Interviewing local Iraqis involved in the project design; • Interviewing local Iraqis who are participating and/or benefited from the project; • Taking photographs of projects' achievements; • Observing and/or measuring infrastructures refurbished by the project; • Observing and/or measuring the quality of supplies provided; • Observing and/or measuring the quality of work done by the project or vendors The goal of the site monitoring visits is to take a “snapshot” of that project. The M&E staff member will collect basic information on the project. He/she will talk to people, make observations, take photos, take notes, etc. The goal is to be able to make a reasonable and accurate assessment about the quality of the project and its effectiveness to meet the stated objectives.” (pg. 102) The manual included a section on the preparation of the site-visit reports: “The reports are designed to capture basic project information for identification purposes. More importantly, however, the report asks the M&E staff member to summarize his/her observations of the project site visit. In addition, the report now contains information on the objectives of the project, whether or not these objectives seem to be being met, as well as any recommendations, suggestions, or required actions that are needed. In some cases, photos, records, project documents, etc. may be attached to the report or otherwise made available.” (pg. 103) As part of this assessment, IBTCI staff reviewed the content of all of the M&E reports conducted for the sampled projects. In each instance, a variety of useful information was provided in the reports including counts and photos of the numbers of workers present on-site, the presence or absence of BDP grant materials and contribution, the presence or absence of local cost share, photos of project progress and accomplishments, and, most importantly, identified strengths and shortcomings in terms of implementation, results and quality. Much of this information would be quite useful to program office and technical area management in order to know how well monitored projects were advancing and would provide advance warning of potential implementation issues. In Kirkuk, the M&E Director was responsible for taking the individual monitoring reports and synthesizing them into a single document that was sent to the POD/POO each week. Much of the content of the individual reports is kept in this report along with summary statistics of M&E visits conducted for the week. IBTCI was able to review International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 11 MEPP II CSP Kirkuk BDP and CIES Study these reports at the Kirkuk offices and view examples of emails sent each week to the POD and POO with the reports as an attachment. The Kirkuk CSP office had a strict policy that written reports from the M&E unit were not to be shared with the BDP and CIES technical teams. This contradicts the guidance contained in the FOM that “Once the POD and POO have reviewed the reports, they should be circulated to the relevant technical team leaders.” (pg. 104). Instead, a weekly meeting was held between the M&E Director, the POD/POO and the technical directors where the M&E Director presented the findings of his unit’s monitoring visits for the week verbally. This policy was implemented reportedly based on concerns about retribution against M&E officers for negative findings. The Kirkuk M&E Unit performs tasks that can be considered as contributing to quality assurance for BDP and, more importantly in the absence of formal QAQC procedures, for CIES projects. IBTCI was able to document that the M&E Unit conducts site visits of BDP and CIES projects and prepares reports of their findings, including observed deficiencies in project implementation and employment numbers that are sent to the POD and shared with the program managers. It is difficult to establish the impact of the M&E site visits on the quality of project implementation and reported results for CIES or BDP projects. As explained above, details of problems found during BDP and CIES site visits are sent directly to the POO and POD in the weekly M&E Site Visit Summary Reports. At that point it is up to the POO and POD to decide what, if any, action to take with the relevant technical program directors. While the M&E Director and the POD and POO stated that specific action had been taken in the form of meetings with the BDP or CIES Directors requesting that they follow-up with their staff to address specific problems, information on the content of such meetings, content of follow-up discussions and any actions taken are not kept in the project files and, as a result, cannot be verified or analyzed as part of this review. A good example of this gap is BDP 1157/18149, which consisted of a lathing shop. IRD spent $5,500 providing equipment and lathing supplies to the shop, which was intended to provide a total of four long-term jobs. The project was completed in May of 2008 and an M&E site visit was conducted on June 29 of the same year. At the time of the site visit, the shop was closed with “For Rent” spray painted on the front door. While there should have been a QAQC visit made to verify the delivered items due to their value being in excess of $5,000, this was not done. Despite the apparent closure of the business, the joint final closeout form from January 2009 shows that the shop was open and employing the requisite four employees. Nowhere in the file or in the M&E report is there any explanation of how the shop went from closed and out of business to a thriving enterprise in six months. The explanation could be as simple as the grantee having closed for lunch or some other reason the day that M&E staff made their visit. The important issue here is the inability to close the loop from monitoring data to program management information to action to correct the issue. In conclusion, IBTCI found that the Kirkuk M&E unit exceeded the minimum required number of M&E site visits for both project types. Based on a review of the content of the M&E reports, it is clear that M&E officers are conducting these visits according to the standards established and described in CSP’s M&E Plan and Field Operations Manual. Visit reports included tallies of observed workers and equipment, descriptions of project International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 12 MEPP II CSP Kirkuk BDP and CIES Study progress, quality and other implementation shortfalls, beneficiary feedback and pictures of on-going or completed projects. Reported cost-share percentages. IBTCI observed that the Kirkuk office is reporting the percentage of CIES and BDP project budgets covered by community cost share different than described in the relevant project documents. In an April 29, 2008 document entitled “Report on the Community Contribution in CSP” (Annex IX), IRD defined the “Total Project Cost” as the sum of CSP cost and community contribution as indicated in the project proposal”. • “The program implementers are encouraged from now on to consider “Total Project Cost” as the sum of “CSP project cost” and the “Community Contribution”. • “Proposed CSP Cost” is the CSP contribution towards project implementation as indicated in the project proposal form. • “Total Project Cost” is the sum of CSP Cost and Community Contribution as indicated in the project proposal. • “Revised and Final CSP Project Cost” is the total final CSP costs that include all contracts and variation orders upon the completion of a project. • “Revised and Verified CC/GC/Grantee Contribution” is the total amount of money contributed by the community/government/Grantee and properly documented in the project file. “ (Report on the Community Contribution in CSP, April 29, 2008, pg. 2) Utilizing such a definition would make sense in a situation where programs worked with local partners to develop a full project budget based on the total need, which was then divided up between CSP and the partner according to their relative resources for this project. Instead, it appears that, prior to the issuance of this guidance, CSP planned its projects as wholly independent initiatives and then went to the community and local government to see what these groups could add-on to meet the contribution requirement. The first approach makes the local cost share an essential component of project completion and success while the latter method removes dependence on the cost share for the fulfillment of its objectives. The CIES tracking sheet guidelines state that the community cost share percentage will be “the percentage of the completed amount against the cost of the total project” and the BDP tracking sheet guidelines state that the contribution percentage is “the percentage of the completed amount against the cost of the total grant project” (for completed grants) (Annexes IX and X). The CIES and BDP tracking sheets provided to IBTCI for Kirkuk calculated this percentage by dividing the local contribution by the CSP budget only, potentially resulting in a higher percentage of local cost share being reported. For example, for CIES 17900, the reported cost share percentage is 57% or $64,855 (verified contribution) divided by $114,022 (verified CSP cost) multiplied by 100. Calculated according to the revised guidance where total project cost is $178,877 (verified contribution + verified CSP cost) the cost share percentage would be 36% - $64,855 (verified contribution) divided by $178,877 (CSP cost of $114,022 plus the contribution of $64,855) multiplied by 100. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 13 MEPP II CSP Kirkuk BDP and CIES Study This observation may be irrelevant as CSP’s cooperative agreement specifies a single total amount of overall cost share for the life of the project instead of a targeted percentage for the program or for each individual project. 3. Does IRD have the procedures in place to implement an effective and independent QAQC program to ensure discrepancies are reported to program management? The formal Kirkuk QAQC program is limited in scope to three procedures, all of them focused on BDP grants. IBTCI confirmed that these QAQC procedures are being effectively implemented and that program management receives information on identified issues. In the document review for sampled projects, IBTCI was able to see that QAQC procedures for the verification of CSP grant items were implemented for 88% of the eight projects that were implemented after the establishment of this process and that qualified based on the amount of the grant. QAQC verification of grantee contribution was conducted for 86% of the seven eligible grants. QAQC Joint Final Closeouts were performed for 57 of the grants, all of which were completed after September 2007, providing useful additional data on the durability of BDP created jobs. IBTCI was able to document the reporting procedures and communication chain for QAQC issues. In the case of the Joint QAQC Final Closeout Certifications, which include comments on issues with each grant, the POD’s signature is required along with those of the BDP and M&E/QAQC managers. These signatures demonstrate that relevant QAQC information is circulated through all levels of PO management and that issues should have been resolved to the satisfaction of the POD. The QAQC unit in Kirkuk is not independent, however, as it is a part of the M&E unit, utilizing the same staff and Director. The Kirkuk M&E Unit performs tasks that can be considered as contributing to quality assurance for BDP and, more importantly in the absence of formal QAQC procedures, for CIES projects. IBTCI was able to document that the M&E Unit conducts site visits of BDP and CIES projects and prepares reports of their findings, including observed deficiencies in project implementation and employment figures that are sent to the POD and shared with the program managers. While it was reported that the POD follows up with relevant technical staff to address problems, documentation of these meetings is not kept and IBTCI was unable to confirm any actions taken as a result of M&E feedback. Based on a quantitative comparison of reported BDP job numbers that were included in the tracking sheet, the number of jobs observed by M&E officer during site visits to the grantees, and the final observed number of jobs included in the Joint QAQC Final Closeout Certifications, all but four of the 23 grants that had lower numbers of employees during M&E site visits supported the reported number by the time of final closeout. This result could provide some evidence that M&E findings are acted upon, resulting in adjustments that improved project implementation. QAQC Site Visits QAQC site visits in Kirkuk are only made for BDP projects and only for the following three reasons: 1) Confirmation of Grantee Contribution over $5,000.00; 2) Verification of Grant Materials Delivery over $5,000.00; and 3) Joint Final Closeout visits. QAQC forms that are unique, although not substantially different according to Kirkuk staff from the International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 14 MEPP II CSP Kirkuk BDP and CIES Study QAQC forms utilized in the Karrada office, guide Kirkuk QAQC site visits (Annex XI). According to interviews with Kirkuk M&E staff and the Program Office senior management, these various procedures were introduced at different times in response to the results of the RIG Audit and the development and dissemination of QAQC procedures from HQ. As a result, verification of implemented QAQC procedures varies according to the completion date of the project. Only projects that meet the first two criteria for on-site QAQC verifications and were closed-out after November 2008 would have been subject to all three procedures, for example, and would have corresponding records in the project files. In order to deal with this issue, the BDP data for sampled projects were recoded based on closeout dates, grantee contribution and grant size and the appropriate use of QAQC procedures was analyzed. IBTCI was able to document the reporting procedures and communication chain for QAQC issues. According to interviews with management staff, completed QAQC site visit forms are sent by the M&E Director to the office POD and POO and to BDP program officers for follow-up and inclusion in the project files. Deficits identified through the QAQC site visits appear to be acted upon. While there was not an instance of this in the sampled projects, a random file reviewed during IBTCI’s visit to the Kirkuk office contained a project where a deficiency in a delivered grant item was identified by the QAQC officer, resulting in a rejection of the previously prepared completion report. The issue in question was the horsepower rating of a particular piece of machinery. The delivered equipment was of an inferior specification to that provide for in the BOQ and contract. The BDP file included another QAQC verification form issued at a later date that confirmed that the correct item was now in place. QAQC verification of grantee contribution was correctly completed in each instance with types and amounts of contribution clearly indicated and the relevant documentation included in the file. In the case of the Joint QAQC Final Closeout Certifications, which include comments on issues with each grant, the POD’s signature is required along with those of the BDP and M&E/QAQC managers. These signatures demonstrate that relevant QAQC information is circulated through all levels of PO management and that issues should have been resolved to the satisfaction of the POD before that time. BDP Jobs at Final Closeout A total of 59 BDP projects from the sample of 86 went through Final Closeout through the joint QAQC/BDP closeout process. As previously noted, CSP does not enter the observed number of jobs at Final Closeout into its M&E database or BDP tracking sheet. IBTCI entered these data in order to calculate the differences, if any, in job numbers observed months later through CSP’s own QAQC/M&E procedures. These numbers were then compared with the data from the proposals reported in the BDO tracking sheets. According to the figures provided on the Final Closeout forms, seven BDP grants (11.9%) saw an increase in jobs compared with the proposed number, three (5.1%) experienced a decrease in jobs and 49 (83.1%) reported no change in the overall International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 15 MEPP II CSP Kirkuk BDP and CIES Study number of jobs. The total number of jobs across the 59 projects increased from 204 to 208. While little change may have been observed in the overall number of jobs, there were more pronounced differences in the numbers when broken down by gender. The number of female jobs observed at Final Closeout for these projects was nine less than the number projected in the proposals. At the same time, the number of male jobs at closeout increased by 13. BDP Jobs Observed during Monitoring and Evaluation Site Visits As described in the section on M&E site visits and reporting above, a total of 67 of the 86 (78%) sampled projects received at least one M&E visit after the delivery of the grant materials. Unlike Final Closeout visits, M&E fieldwork is not announced to either the grantee or the BDP department. This has the advantage of allowing for a more candid assessment of the numbers of employees and day-to-day condition of the business. It also has the potential disadvantage of situations where monitored businesses are closed or have reduced numbers of employees for reasons unrelated to grant compliance. Among other information, M&E officers entered the number of observed jobs during their site visits, allowing for a comparison with the reported data in the tracking sheets. The results of this comparison show that nearly half (47.8%) of the monitored projects had a smaller number of employees on site at the time of the monitoring visit than the proposed amount. The same percentage of businesses had the correct number of staff present and three projects, slightly less than 5%, had more employees. A second M&E visit was made for four of the projects that had fewer jobs during the M&E visit, including for the two projects with the largest differences – eight and seven jobs fewer. When the jobs numbers from the second visit are compared to those in the same project’s first visit and the proposed amount, for three of these four projects, the numbers of employees were even fewer. Two of these projects had Joint QAQC Final Closeout forms, which allows for comparison of job numbers some months after the M&E visits. Presumably, the M&E reports, once communicated to the BDP and Program Office head management, would result in follow-up with the affected BDP projects and either improvements in the business’ performance or an adjustment to the reported jobs figure. Unfortunately, there are no records of any action taken by Kirkuk BDP staff in either of these projects. Based on the reported final closeout figures, it appears that some action was taken as the difference in jobs at that time was less than during the monitoring visits. However, Table 7 indicates that LT job figures were lower during the first M&E visit and the QAQC’s figures matched the initial proposed numbers. On one hand, findings from Table 7 suggest tht close-out numbers were artifically inflated. On the other hand, additional interviews with CSP Kirkuk and business owners would be needed in order to explain the variation in the figures and the resulting achievement of higher numbers. Table 7 below combines data from the BDP tracking sheets, M&E site visits and Joint Final Closeout reports to illustrate the fluctuation in jobs numbers observed by various members of the Kirkuk CSP team. The first column is taken directly from the final International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 16 MEPP II CSP Kirkuk BDP and CIES Study tracking sheet. The second and third columns are the differences in observed jobs numbers from M&E site visits and the tracking sheet and the fourth column was entered from the Joint QAQC Final Closeout Reports. The final column is the difference between the reported figures from the tracking sheet and the observed numbers at Joint Closeout. An additional finding from this analysis is that the use of jobs numbers from the proposals as data for the tracking sheets is a fairly reliable practice based on the total number of jobs observed at final closeout by IRD staff in Kirkuk. While there was change in the numbers of jobs for individual projects, more substantially, in the mix of male and female jobs numbers, the overall total number of jobs did not vary significantly from the proposal figures to the closeout numbers. Table 7. CSP Kirkuk Reported and Observed Employee Figures CSP Project Code Total Proposed Long Term Jobs* 1st M&E Visit Difference 2nd M&E Visit Difference Total QAQC Joint Closeout Jobs Total Closeout Difference CSP-KK-BDP-8502 14 -7 -10 12 -2 CSP-KK-BDP-7956 8 -2 -3 7 -1 CSP-KK-BDP-15990 8 -8 . 8 0 CSP-KK-BDP-8060 8 -7 . 8 0 CSP-KK-BDP-8084 13 -7 . 13 0 CSP-KK-BDP-8112 14 -7 . 14 0 CSP-KK-BDP-18149 4 -4 . 3 -1 CSP-KK-BDP-7360 6 -4 . 6 0 CSP-KK-BDP-11004 5 -3 . 5 0 CSP-KK-BDP-8515 7 -3 . 7 0 CSP-KK-BDP-10734 3 -2 . 3 0 CSP-KK-BDP-20302 3 -2 . 3 0 CSP-KK-BDP-7347 4 -2 . 4 0 CSP-KK-BDP-11511 3 -1 . 3 0 CSP-KK-BDP-15140 2 -1 . 2 0 CSP-KK-BDP-16639 2 -1 . 2 0 CSP-KK-BDP-16643 2 -1 . 2 0 CSP-KK-BDP-16691 3 -1 . 3 0 CSP-KK-BDP-16707 2 -1 . 2 0 CSP-KK-BDP-16820 2 -1 . 2 0 CSP-KK-BDP-17765 2 -1 . 2 0 CSP-KK-BDP-7315 14 -1 . 14 0 CSP-KK-BDP-7704 3 -1 . 3 0 Total 132 -68 -13 128 -4 International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 17 MEPP II CSP Kirkuk BDP and CIES Study V. RECOMMENDATIONS Establish the minimum required documentation for all relevant types of cost share for the project. Ideally, this should be done at the outset of such projects and the documentation guidelines should be revisited periodically to reflect changes or additions to the program’s project scope. While existing regulations such as 22 CFR 226.23 [Administration of Assistance Awards to U.S. Non-Government Organizations – Cost Sharing or Matching], are a useful starting point, they are not specific enough to provide detailed guidance for the universe of potential cost share types and corresponding documentation. Establish a system that allows key project results, such as reported long-term jobs, to be updated based on M&E and QAQC processes. The processes established for this should be uniform across the program offices, such as using data from initial M&E visits and/or final closeout jobs numbers for BDP projects instead of figures from the proposal. This may result in a delay in reported results to USAID and a need for additional personnel resources to conduct necessary follow-up and verification. Require that documentation be included in project files to demonstrate follow-up on identified M&E problems. For example, direct M&E to update their relevant M&E reports to reflect decisions that were taken to address concerns raised in previous reports, to store relevant emails in the project files, to make copies of minutes taken during meetings between the POD or POO, M&E staff and technical directors where such issues are addressed and action items identified. VI. CONCLUSIONS CSP Kirkuk has implemented a large number of project processes designed to ensure accurate, effective, high-quality implementation of its CIES and BDP programming. Procedures evolved over time responding to identified program issues and evolving guidance from the CSP HQ offices. These include several quality assurance processes, documentation and implementation protocols and project monitoring systems. All of these elements produce useful, timely and valid data that can be utilized by program management to maximize the impact of CSP’s efforts to promote stability in the Kirkuk AoR. The data collected during this assessment confirms that Kirkuk’s efforts in these areas resulted in the successful completion of projects that are, for the most part, well monitored and documented. If the exceptions to this record of achievement are addressed successfully in other program offices, these offices and the resulting programming will be further strengthened. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 18 MEPP II CSP Kirkuk BDP and CIES Study Annex I: BDP Monitoring Instrument CSP KIRKUK/HAWIJA BUSINESS DEVELOPMENT PROJECTS -SPECIAL STUDY INSTRUMENT￾INSTRUCTION TO THE MONITOR: COMPLETE ITEM 1 TO 10 OF THIS FIRST PAGE OF THE MONITORING FORM BEFORE PROCEEDING TO THE PROJECT SITE. GRANTS INFORMATION PANEL (1-10 ARE TO BE COMPLETED FROM IRD DATABASE) 1 GRANT PROJECT CODE: 2 GRANT TITLE 3 GRANTEE NAME 4 SITE VISIT DATE (DAY 0-31/MONTH 01-12/YEAR 2009) 5 GOVERNORATE: 6 DISTRICT (QADA1): 7 SUB-DISTRICT (NAHIYA): 8 MAHALLA/ZUQAQ: 9 MONITOR’S NAME 10 11 RESULT OF INTERVIEW: COMPLETED ............................................1 REFUSED ................................................2 NOT AT HOME/BUSINESS..........................3 INVALID ADDRESS.....................................4 SECURITY PREVENTED ACCESS ................5 BUSINESS HAS CLOSED............................6 BUSINESS HAS RELOCATED......................7 OTHER (SPECIFY) ....................................9 12 DATA ENTRY CLERK: NAME: ________________________________ DATE OF ENTRY _________________________________ MONITOR NOTES: IN THIS SPACE THE MONITOR RECORDS NOTES ABOUT WHY THE SITE VISIT WAS NOT COMPLETED. MONITOR NOTES: IN THIS SPACE THE MONITOR RECORDS HIS OBSERVATIONS ABOUT THE MONITORING VISIT. RECORD HERE IMPRESSIONS ABOUT THE VISIT. WHO WAS PRESENT. WERE THEY WELCOMING OR DISTANT... International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 19 MEPP II CSP Kirkuk BDP and CIES Study PART I: GRANT UTILIZATION AND COMPLIANCE # QUESTION RESPONSE SKIP HERE PREPARE THE RESPONDENT FOR WHAT IT IS YOU WANT TO TALK TO HIM ABOUT. TELL HIM HOW LONG IT WILL TAKE AND WHAT THE INFORMATION WILL BE USED FOR. ASK PERMISSION TO PROCEED. FIRST I WOULD LIKE TO ASK ABOUT YOU AND THE PEOPLE IN YOUR HOUSEHOLD 1 NAME AND TITLE OF RESPONDENT 2 SEX OF RESPONDENT MALE .........................................1 FEMALE......................................2 3 HOW MANY PERSONS LIVE IN YOUR HOUSEHOLD INCLUDING YOURSELF? 4 AGE OF RESPONDENT 5 HOW MANY PERSONS IN YOUR HOUSEHOLD DEPEND ON THE INCOME YOU EARN FROM THIS BUSINESS? 6 WHAT IS YOUR MARITAL STATUS? SINGLE, NEVER MARRIED ............1 MARRIED....................................2 DIVORCED..................................3 WIDOWED ..................................4 7 WHAT IS THE HIGHEST LEVEL OF EDUCATION YOU HAVE OBTAINED? (CATEGORIES FROM COSIT) ILLITERATE.................................1 READ AND WRITE........................2 ELEMENTARY .............................3 INTERMEDIATE ...........................4 SECONDARY...............................5 VOCATIONAL SCHOOLS ...............6 VOCATIONAL CENTERS ...............7 DIPLOMA ....................................8 BACHELOR .................................9 HIGH DIPLOMA..........................10 MASTER ...................................11 DOCTORATE.............................12 OTHER .....................................99 8 HAVE YOU ATTENDED ANY TECHNICAL OR VOCATIONAL EDUCATION RELATED TO YOUR BUSINESS? YES ...........................................1 NO.............................................2 NOW I WOULD LIKE TO ASK YOU ABOUT YOUR BUSINESS AND ANY GRANTS THAT YOU HAVE RECEIVED THAT SUPPORT YOUR BUSINESS 9 WHEN DID YOU FIRST OPEN YOUR BUSINESS? (MONTH 01-12/YEAR) ENTER MONTH AND YEAR ONLY; USE 4 DIGIT REFERENCE FOR THE YEAR / International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 20 MEPP II CSP Kirkuk BDP and CIES Study 10 PLEASE CHARACTERIZE HOW YOUR BUSINESS IS OPERATED (CONFIRM WITH OTHER RESEARCH) SOLE PROPRIETOR .....................1 FAMILY OWNED BUSINESS EMPLOYING ONLY FAMILY MEMBERS ..................................2 FAMILY OWNED BUSINESS WITH NON-FAMILY EMPLOYEES............3 PARTNERSHIP/COOPERATIVE .....4 OTHER........................................5 11 PLEASE IDENTIFY THE KIND OF BUSINESS YOU CURRENTLY OPERATE AGRICULTURE ............................1 LIVESTOCK.................................2 CONSTRUCTION..........................3 MANUFACTURING .......................4 HANDICRAFT ..............................5 WHOLESALE/RETAIL SHOP...........6 REPAIR SERVICES.......................7 PERSONAL SERVICES..................8 OTHER .......................................9 12 DESCRIBE THE NATURE OF YOUR BUSINESS AND HOW IT OPERATES WRITE ALL RESPONSES 13 DID YOU RECEIVE A GRANT FROM THIS PROGRAM TO HELP SUPPORT OR OPEN YOUR BUSINESS? YES ...........................................1 NO.............................................2 END 14 DID YOUR BUSINESS RESTART OR REOPEN BECAUSE OF THIS GRANT AFTER HAVING BEEN CLOSED? YES ...........................................1 NO.............................................2 15 WHEN DID YOU FIRST START USING THE GRANT TO MAKE MONEY? (MONTH 01-12/YEAR) ENTER MONTH AND YEAR ONLY; USE 4 DIGIT REFERENCE FOR THE YEAR 16 WHAT MATERIALS AND EQUIPMENT DID YOU RECEIVE (MONITOR TO CONFIRM BASED ON BOQ)? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 21 MEPP II CSP Kirkuk BDP and CIES Study 17 DID YOU MAKE USE OF ALL THE MATERIALS AND EQUIPMENT THAT WERE PROVIDED? YES ...........................................1 NO.............................................2 18 IF EQUIPMENT WAS MISSING (Q16) OR WAS NOT BEING USED, PLEASE SPECIFY WHAT ITEMS AND EXPLAIN WHY? 19 ARE YOU STILL OPERATING THE SAME BUSINESS THAT WAS ASSISTED WITH MATERIALS AND EQUIPMENT PROVIDED BY THE GRANT? YES ...........................................1 NO.............................................2 20 HOW MANY TIMES DID IRD VISIT YOUR BUSINESS AFTER THE GRANT WAS PROVIDED? WRITE NUMBER 21 DID YOU ATTEND BUSINESS TRAINING SPONSORED BY IRD? YES ...........................................1 NO.............................................2 22 DID IRD PROVIDE ANY TECHNICAL AND BUSINESS ADVICE AFTER YOU RECEIVED THE GRANT? YES ...........................................1 NO.............................................2 GOTO PART II 23 PLEASE EXPLAIN WHAT ADVICE WAS PROVIDED? 24 DID YOU MAKE USE OF THE ADVICE THAT WAS PROVIDED? YES ...........................................1 NO.............................................2 25 PLEASE EXPLAIN WHY YOU DID OR DID NOT USE THIS ADVICE? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 22 MEPP II CSP Kirkuk BDP and CIES Study PART II: EMPLOYMENT CREATED AS A RESULT OF THE GRANT # QUESTION RESPONSE SKIP NOW I WOULD LIKE TO ASK YOU SOME QUESTIONS ABOUT PAID EMPLOYEES IN YOUR BUSINESS MALE FEMALE PART￾TIME FULL￾TIME 1 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DID YOU HAVE IN THE FOLLOWING CATEGORIES BEFORE YOU RECEIVED THE GRANT? ENTER 00 IF NONE TOTAL MALE FEMALE PART￾TIME FULL￾TIME 2 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DO YOU HAVE IN THE FOLLOWING CATEGORIES AT THE PRESENT TIME? ENTER 00 IF NONE TOTAL MALE FEMALE PART￾TIME FULL￾TIME 3 HOW MANY PAID EMPLOYEES (INCLUDING THE GRANTEE AND ANY FAMILY MEMBERS) DID YOU HAVE IN THE FOLLOWING CATEGORIES WHEN THE GRANT ENDED? ENTER 00 IF NONE TOTAL IF 2 AND 3 ARE THE SAME GOTO 6 4 WHY DID THE NUMBER OF JOBS IN YOUR BUSINESS CHANGE BETWEEN NOW AND THE TIME YOUR GRANT ENDED? WRITE ALL RESPONSES 5 EXPLAIN THE MOST CRITICAL FACTOR THAT HAS PERMITTED THE BUSINESS TO ADD NEW EMPLOYEES? WRITE ALL RESPONSES MALE FEMALE PART￾TIME FULL￾TIME 6 HOW MANY UNPAID FAMILY WORKERS (INCLUDING THE GRANTEE) DO YOU HAVE IN THE FOLLOWING CATEGORIES AT THE PRESENT TIME? TOTAL International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 23 MEPP II CSP Kirkuk BDP and CIES Study # QUESTION RESPONSE SKIP ENTER 00 IF NONE NOW I WANT TO ASK ABOUT YOUR BUSINESS PROSPECTS 7 WHAT DIFFICULTIES DO YOU ENCOUNTER TRYING TO OPERATE YOUR BUSINESS? WRITE ALL RESPONSES 8 IN YOUR OPINION, WHAT ANSWER BEST DESCRIBES THE FUTURE OF YOUR BUSINESS INCOME? IT WILL INCREASE....................... 1 IT WILL STAY THE SAME .............. 2 IT WILL DECREASE...................... 3 DON’T KNOW/UNSURE ................ 8 9 PLEASE EXPLAIN YOUR ANSWER ABOUT YOUR FUTURE BUSINESS PROSPECTS: 10 IN YOUR OPINION, WHAT ANSWER BEST DESCRIBES YOUR FUTURE NUMBER OF EMPLOYEES? IT WILL INCREASE....................... 1 IT WILL STAY THE SAME .............. 2 IT WILL DECREASE...................... 3 DON’T KNOW/UNSURE ................ 8 11 PLEASE EXPLAIN YOUR ANSWER ABOUT YOUR FUTURE NUMBER OF EMPLOYEES? International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 24 MEPP II CSP Kirkuk BDP and CIES Study Annex II: CIES Monitoring Instrument CIES COMPLETED PROJECT MONITORING FORM INSTRUCTION TO THE MONITOR: COMPLETE THIS FIRST PAGE OF THE MONITORING FORM BEFORE PROCEEDING TO THE PROJECT SITE. PROJECT INFORMATION PANEL (1-8 ARE TO BE COMPLETED AT THE PARTNERS OFFICE) 1 PROJECT ID 2 PROJECT TITLE 3 TYPE OF PROJECT SCHOOL RENOVATION............................. 1 GOVERNMENT OFFICE RENOVATION ........ 2 WATER SUPPLY ...................................... 3 4 5 GOVERNORATE: 6 DISTRICT (QADA1): 7 SUB-DISTRICT (NAHIYA): 8 MAHALLA/ZUQAQ: 9 MONITOR NAME: 10 SITE VISIT DATE: DD/MM/YEAR ___/__/____ 11 RESULT OF INTERVIEW: COMPLETED ........................................... 1 REFUSED ............................................... 2 COULD NOT FIND PROJECT...................... 3 SECURITY PREVENTED ACCESS............... 4 OTHER (SPECIFY) ................................... 9 12 MONITOR NOTES: IN THIS SPACE THE MONITOR RECORDS NOTES ABOUT WHY THE SITE VISIT WAS NOT COMPLETED. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 25 MEPP II CSP Kirkuk BDP and CIES Study PART I – INTERVIEW WITH PROJECT REPRESENTATIVE # question response skip 12 PLEASE INDICATE THE NAME, TITLE AND PROJECT ROLE OF THE RESPONDENT. 13 WHEN WAS THIS PROJECT COMPLETED? DD/MM/YEAR ENTER 99/99/9999 IF UNKNOWN ___/__/____ 14 TYPE OF PROJECT IF MULTIPLE TYPES SELECT ONLY THE DOMINANT TYPE NEW BUILDING CONSTRUCTION................... 1 BUILDING REHABILITATION.......................... 2 WATER SUPPLY.......................................... 3 SANITATION............................................... 4 OTHER (PLEASE SPECIFY) .......................... 8 15 IS THE SITE OR SERVICE BEING USED? YES........................................................... 1 NO ............................................................ 2 GOTO 8 16 IS THE SITE OR SERVICE BEING USED FOR ITS INTENDED PURPOSE? YES........................................................... 1 NO ............................................................ 2 PART II 17 IF NO, WHY IS THE SITE NOT BEING USED OR NOT BEING USED FOR ITS INTENDED PURPOSES? WRITE ALL RESPONSES NEXT TABLE International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 26 MEPP II CSP Kirkuk BDP and CIES Study PART II – INTERVIEW WITH PROJECT REPRESENTATIVE # ITEM ENGLISH ITEM ARABIC IS THE WORK COMPLETED? Y/N IS THE QUALITY OF THE WORK TO STANDARD? Y/N IF NO, DESCRIBE FULLY WHAT WAS FOUND TO BE DEFICIENT IF WORK WAS BELOW STANDARD HAS THE USEFULNESS OF THE PROJECT BEEN AFFECTED? Y/N IF THE USEFULNESS OF THE PROJECT HAS BEEN AFFECTED, DESCRIBE FULLY HOW IT IS AFFECTED. 1. General repair of the damaged building: Replastering with cement and gypsum 2. Reinstall doors 3. Reinstall windows 4. Reinstall main gate 5. Repair electrical installation 6. Rehabilitate the big hall 7. Repair sanitary installation International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 27 MEPP II CSP Kirkuk BDP and CIES Study PART III – MONITOR’S COMMENTS # QUESTION RESPONSE 1 WHAT ARE THE MAJOR STRENGTHS AND ACCOMPLISHMENTS OF THIS PROJECT? 2 WHAT ARE THE MAIN OBSTACLES FACING THE PROJECT BASED ON YOUR VISIT? 3 MONITOR’S COMMENTS AND OR RECOMMENDATIONS REGARDING THIS PROJECT 4 WAS PROJECT STAFF PRESENT DURING THE INTERVIEW? YES ...........................................................1 NO.............................................................2 5 MONITOR’S FIELD NOTES REGARDING THE MONITORING PROCESS International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 28 MEPP II CSP Kirkuk BDP and CIES Study ANNEX III: FSO and CSP Responses to IBTCI Recommendations. IBTCI Recommendation FSO Response IRD/CSP response Action Taken and Date Establish the minimum required documentation for all relevant types of cost share for the project. Ideally, this should be done at the outset of such projects and the documentation guidelines should be revisited periodically to reflect changes or additions to the program’s project scope. While existing regulations such as 22 CFR 226.23 [Administration of Assistance Awards to U.S. Non￾Government Organizations – Cost Sharing or Matching], are a useful starting point, they are not specific enough to provide detailed guidance for the universe of potential cost share types and corresponding documentation. USAID agrees that 22 CFR 226.23 outlines basic assistance parameters for cost-sharing but that additional guidelines should be developed, such as through the CSP Operations Manual. Various steps have been taken. IRD refined the CSP Operations Manual in May 2008 to develop more detailed community contribution guidelines for CSP projects than existed at the project outset. CSP Modification 16 (dated 19 March 2009) also required that IRD have in place and in its files consistent and detailed valuing, tracking, and verification systems for community and GoI contributions. While most CSP offices are now closed including Kirkuk, FSO recognizes the importance of cost-share verification and had planned a Community contributions involve two aspects: first, accounting for the value of contributions; and second, verification of the contributions was made at the promised value. In the case of the CSP Kirkuk Phase I Irrigation Cleaning projects, CSP secured the supporting letters documenting the GOI contribution and value. This completed the accounting and valuation requirement. CSP Kirkuk QA/QC, M&E, CIES program staff include reports during site visits, or photographs of the contribution to verify the contribution was made. This completed the verification requirement. Accordingly, CSP Kirkuk accepted and reported the contribution valuation figures from the GOI. Although much effort has been made to provide all project documentation supporting cost shares, all the supportive files that were stored in Amman were not sent to IBTCI during the study period. CSP adopted the following community contribution procedure in 2008: 1) Before a project is approved, documentation of proposed cost share structure is reviewed by the appropriate CSP program director and is a factor in the project approval process. 2) GOI officials submit formal letters outlining their proposed contribution, including financial value, to CSP during project development. 3) CSP Program Officers monitor for actual contributions as they are made (generally implemented in phases) during project implementation through project site visits and reports. 4) When the project is completed, GOI officials are required to submit formal letters confirming their contribution has been made, including financial value. 5) CSP Monitoring and Evaluation staffs verify CSP Program Officer reports on contributions through actual site visits and review of documentation. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 29 MEPP II CSP Kirkuk BDP and CIES Study review of this for the two remaining CSP offices in Mosul and Baqubah for the next IBTCI QAQC study. Verification includes costing of all assets and labor by CSP staffers. 6) All documentation of GOI contributions is audited by an internal (non-program) Contribution Verification unit. 7) Documentation of all verified GOI contributions, which may differ from proposed contributions, is added to the CSP project file once approved by the internal Contribution Verification unit. 8) GOI contributions are considered final and “verified” only after the CSP project has been officially closed and the verification process completed. 9) Now all Kirkuk files are in Baghdad for consolidation. If necessary, IBTCI could recheck the desired project documentation for more evidence. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 30 MEPP II CSP Kirkuk BDP and CIES Study IBTCI Recommendation FSO Response IRD/CSP response Action Taken and Date Establish a system that allows key project results, such as reported long-term jobs, to be updated based on M&E and QAQC processes. The processes established for this should be uniform across the program offices, such as using data from initial M&E visits and/or final closeout jobs numbers for BDP projects instead of figures from the proposal. This may result in a delay in reported results to USAID and a need for additional personnel resources to conduct necessary follow-up and verification. FSO will look for reporting of the DCoP/Compliance Officer’s findings in the Monthly and/or the Weekly report. The system to validate CSP performance reporting data is in place. However, the mechanisms set forth to update the information in the tracking sheets have not always been uniformly operational. A good feedback system between CSP city management, M&E and QAQC through follow-up site visits is necessary to make efficient and sound corrective management decisions. In March 2009, and in response to USAID Modification 16, CSP ensured all M&E units will provide accurate, reliable, consistent and verifiable data with respect to progress towards achieving results. The M&E units are tasked with conducting random site visits of CSP project sites in order to provide independent monitoring, review, feedback, analysis, and recommended actions for improvement if appropriate. The M&E unit in each city works for their respective POD/POO, and receives guidance, support, and technical direction from the M&E Manager and Director for Iraq. The QA/QC effort was modified at this time, including the designation of a HQ QA/QC specialist, to accomplish two goals: First, the Quality Assurance (QA) side of the system is designed to ensure that processes established by CSP are adequate and effective in meeting program objectives. The QA side identifies the strengths and weaknesses in the procedures set in place for program implementation and propose corrective measures if International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 31 MEPP II CSP Kirkuk BDP and CIES Study needed. The second goal is to maximize the synergies between QA/QC and M&E. This is because the Quality Control (QC) side of the effort is designed to further monitor and evaluate project implementation activities in relation to the CSP M&E system. Its objective is to test the validity and the reliability of the knowledge base generated towards the intended results. QA/QC staffs also verify the outputs during their site visits and any discrepancies are adjusted in the data management for CSP at both the city and country level. With guidance from the M&E Manager, M&E city specialists work closely with program component staff, and PODs/POOs work closely with city QA/QC staffs to verify these numbers, and resolve discrepancies that may be noted. This information is acted upon by the PODs/POOs. The HQ QA/QC specialist regularly reviews procedures, including the uniformity of implementation, with CSP cities. In addition, spot checks of M&E and QA/QC compliance by the HQ QA/QC specialist, and DCOPs have been completed, and continue. International Business & Technical Consultants, Inc. Monitoring and Evaluation Performance Project, Phase II (MEPP II) 32 MEPP II CSP Kirkuk BDP and CIES Study IBTCI Recommendation FSO Response IRD/CSP response Action Taken and Date Require that documentation be included in project files to demonstrate follow-up on identified M&E problems. For example, direct M&E to update their relevant M&E reports to reflect decisions that were taken to address concerns raised in previous reports, to store relevant emails in the project files, to make copies of minutes taken during meetings between the POD or POO, M&E staff and technical directors where such issues are addressed and action items identified. FSO will work with on the IRD’s recommended action to systematically document and archive information based on M&E and QAQC visits; that it is necessary for IRD to take steps to “marry up” the existing archived data; and that if security is an issue, that procedures should be in place for qualified users (such as the COP/DCOP/POD) to fully access the data. M&E and QA/QC documentation identifying project deficiencies exists. The documentation is often not placed in project files for security reasons, but rather archived separately or in computer files. From now on, CSP M&E will systematically document project files regarding management information that has been brought about by site visit recommendations. On May 21, 2009 CSP M&E Iraq￾Wide HQ team sent to M&E city managers in all Phase III and Phase IV cities direction and guidance on methods to systematically document and archive information that has been brought about by M&E and QA/QC site visit recommendations, and actions taken based on the recommendations.