Evaluation of the Small Grants Program Performed by IHSS Project: A Consolidated Report Prepared for: IHSS Project funded by USAID Prepare by: Ghaith J. Al-Eyd, MD, M.Sc. Namir G. Al-Tawil, MD, PhD IHSS Project Baghdad, Iraq May 2004 2 Table of Contents 1. Introduction………………………………………………………... 3 2. Objectives and Approaches……………………………………….. 5 3. Results…………………………………………………………….. 7 4. Tables and Figures………………………………………………... 8 5. Conclusions………………………………………………………. 14 6. Appendices ……………………………………………………….. 15 3 1. Introduction The Iraq Health System Strengthening Project (IHSSP) provided rapid response grants for both local and international NGOs to address the specific health needs of the Iraqi people. In order to assure that the grant is being best used for the expected results, the consolidated work plan of IHSSP as approved by USAID specified that IHSSP should “design and implement a grants monitoring and evaluation (M&E) program that includes internal M & E, performance-related payment, and collaborative review of grant achievements before final payment is made”. M&E, both internal and external, can serve as an important tool to increase grant performance, through its inspection mechanism as well as provision of necessary technical assistance. This is particularly true for those grantees whose experiences are limited and who lack financial management systems. The framework of M&E of the small grants is presented in Figure 1. It provides general guidance for the project specific M&E as well as the overall M&E of the small grant program. As shown in the Figure, the overall objectives, including a general objective to provide rapid response grant to address the specific health needs of the Iraqi people and the associated numerous sub-objectives as specified in Box 1. These objectives are to be achieved through supporting various projects. Each of the project funded by the small grant program has it’s own proposed activities, the implementation of which are to achieve the project specific objectives as they are proposed in the grant application. The achievements of the project specific objectives will mean the attainment of the program objectives. The performance of the project (for both midterm and end-of-project) can be monitored and evaluated according to whether or not and how well the grantees implemented their proposed activities, and the direction and level of attainment of the proposed objectives. The performance of the small grant program can be evaluated according to the results of M&E of the specific projects as well as an overall and cross-project analysis the grants. A total of 35 grants’ projects have been proposed and then approved, but unfortunately only 28 grants have been implemented and evaluated completely. There was an incorrigible cut in the total small grants budget that interrupted some of the approved grants. 4 Figure 1. The framework for M&E of the small grants Project n Project objective 1 Project objective 2 Project objective 3 Project objective 1 Project objective 2 Project objective 3 • • • • • • • Small Grant Program Objectives Project 1 Project 2 Project 3 Project 1 Project 4 Project 5 Project 6 Project 7 Project 8 Project 9 Project 10 Project 11 Project n SMALL GRANT PROGRAM SMALL PROJECTS PROJECT OBJECTIVES • • • • • • • Project activity 1 Project activity 2 Project activities Project activity 4 Project activities Project activity 5 Project activity Project activity n Project 1 Project activity 1 Project activity 2 Project activities Project activity 4 Project activities Project activity 5 Project activity Project activity n PROJECT ACTIVITIES Box1. Objectives of the Small Grants A small grant is designed and implemented to quickly and visibly contribute to one or more of the following objectives: 1. Speed up the reconstruction and re-equipment of medical facilities and health sector infrastructure – priority will be given to those facilities that support primary health care, particularly, care focused on women and children; 2. Support service providers and community initiatives to restore and maintain health services in areas and facilities disrupted by the conflict; 3. Enhance IHSSP efforts and roll out project results to more facilities, regions, and communities of Iraq to contribute to the following important goals: 3.1 restore and modernize primary health services, particularly, women’s and children’s care; 3.2 reconstitute health financing and administration; 3.3 modernize disease surveillance and health reporting; 3.4 reconnect Iraqi health providers with international standards and best practices through modernized clinical training and care management; 3.5 develop health and management data systems to provide policy- and decision-makers with the evidence of health needs and health sector performance; 3.6 increase participation of the Iraqi communities and families in the matters of health, particularly disease prevention and health promotion; 3.7 help health system leaders of Iraq develop a broader look on the present and future of the Iraqi health care sector; 3.8 foster health care and policy research; 3.9 support for professional associations such as doctors, nurses, pharmacists, lab technicians, public health and allied health sciences. 4. Other objectives that target important and effective solutions in the health care sector of Iraq. 5 2. Objectives The general objective of this report is to provide an overall evaluation of the effectiveness of small grants program, and to provide necessary information that can be used in preparation of the final report of the IHSS project. Specifically, the report is to: a. Check if the activities have been performed as they were designed (timing, quantity, and quality). b. Assessing whether the grants have been implemented towards the achievement of designed objectives. c. Assessing whether the objectives of the grant have been achieved. d. Estimating the social benefits of the small grant program (e.g. the number of people hired, improvement in capacity, the number of direct beneficiaries, the quantity of services, and effects on health). 3. Approaches Project-specific and small grant program M&E have been conducted with close collaboration between M&E team and the small grant program (I-HELP). While in principle, the M&E will be external (external of internal), the two teams are on the common ground in their objectives to improve the performance of the grants and the program as a whole. During the process of designing and implementation of this M&E work, attitude of “inspections” and “criticisms” has been avoided, but rather the two teams have maintained frequent dialogues and worked together to seek strategies and take actions for the benefits of project and program performance. Achievements have been documented as much as possible, problems have been identified, and recommendations for performance improvement have been provided. Grantees have been informed that they have to accept external M&E on their performance, including process and results before the contract is signed. M&E Tools & Strategy: 1- Midterm M&E have been conducted at end of first half of the project implementation period. Ideally, I-HELP have received the progress report from the grantee (See appendix 1). 2- The end-of-project M&E have been done after the submission of the final report of a project. However, if the grantees fail to submit final report in time, the end￾of-project M&E should begin as soon as possible after the specified deadline (see appendix 2). 3- I-HELP shared the grant files with M&E team through a mechanism of easy access to “mother” files or obtaining a copy of each of the grant files. 4- M&E team communicated with I-HELP and filled parts of the form and questionnaire before visiting the grantees. 6 5- Midterm & end-of-project M&E of the grants included field visits, interviews of grantees, interviews of the clients (or beneficiaries) of the grants, observations, and checking grant documents. 6- Upon finalization of the field visit, M&E team submitted the Small Grant Midterm and End – of project M&E report to I-HELP team as soon as possible for the purpose of development and implementation of performance improvement strategies and actions. 7- All reports and forms have been well managed by M&E team and have been accessible to I-HELP staff. 8- Midterm M&E of the grants will include field visits, interviews of grantees, interviews of the clients (or beneficiaries) of the grants, observations, and checking grant documents. 9- Upon finalization of the End-of-project M&E for a grant, M&E team should submit the Small Grant End-of-Project report to I-HELP as soon as possible. 10- Data analysis have been done using SPSS version 9 program, based on all data collected (including midterm and end-of-project M&E reports and forms, accumulated narratives, and other project files available). In addition, M&E team have provided a Workshop Evaluation Questionnaire (See appendix 3) to evaluate the workshop directly conducted by I-HELP. This questionnaire forms have been analyzed and documented in the consolidated trainings’ report with primary health care delivery training workshops. 7 4. Results A total of 28 projects had been granted with a total budget of $1,162,566.73 . Half (50%) of the grants were implemented in Baghdad, where as the others were implemented in Mosul, Wassit, Basra, Erbil, Sulaimania, Kirkuk, Thiqar, and Tikrit (See table 1, Fig. 2 for details). Table 2 shows that the duration of one third (32.14%) of the grants ranges from 31 – 60 days and the duration of 14.29% of the grants was ≥ 121 days. More than two thirds (67.86%) of the grants were implemented in urban areas, 3.57% were implemented in rural areas, 7.14% were implemented in suburban areas, and the others where implemented in a combination of locations (See table 3 for details). Seventy five percent of the grants were sponsored by NGOs and the rest were sponsored by a variety of community groups (See table 4, Fig. 3). The objectives of more than half (53.57%) of the grants were consistent with one or two of the I-HELP objectives (See table 5 for details). More than two thirds (67.86%) of the beneficiaries were non medical personnel, and 70.86% were medical personnel as shown in table 5. The majority (92.86%) of the grants succeeded to accomplish their activities and only two (7.14%) grants failed to do so (See table 6). Regarding security status (table 7), nearly all (96.4%) of the grants had no security problems, and one grant had faced a minor security problem that was fixed. Total number of staff hired by the projects was1051 persons with a mean of 37 persons and a median of 10, ranging from 2-209 persons. Half of the projects hired ≤10 persons (See table 8 for details). More than half (53.57%) of the grantees were newly created NGOs or community groups that will be sustained after the end of I-HELP. While the rest were established organizations (See Fig. 4). In general, the status of the already established organizations was improved compared with their status before, specifically; skills & knowledge, ability to provide technical assistance, and general performance of the organization as stated by the organizations managers (See table 9 for details). All of these grants’ managers were aiming to expand their projects to include more areas in the country in the future. The majority of the grantees succeeded to implement their projects toward the expected results as shown in table 10, where the M&E staff has rated the projects performance according to certain selected parameters. 8 Table 1. Grants by governorate Governorate Frequency Percentage Baghdad 13 46.43 Baghdad & Wassit 1 3.57 Basrah 2 7.14 Erbil 2 7.14 Ninewa 3 10.71 Salah al-Din 1 3.57 Sulaymaniyah & Tameem 1 3.57 Thi-Qar 1 3.57 Thi-Qar & Basrah 1 3.57 Wassit 3 10.71 Total 28 100.00 Table 2. Grants by duration Table 3. Grants by location Project duration (days) Frequency Percentage ≤ 30 4 14.29 31-60 9 32.14 61-90 6 21.43 91-120 5 17.86 ≥121 4 14.29 Total 28 100.00 Location of the project Frequency Percentage Rural 1 3.57 Sub-urban 2 7.14 Urban 19 67.86 Urban & suburban 4 14.29 Rural & suburban 1 3.57 Urban & rural 1 3.57 Total 28 100.00 9 Table 4. Grants by ownership Table 5. Grants by reference to I-HELP objectives Table 5. Grants by type of beneficiaries Ownership status Frequency Percentage NGO 21 75.00 Community group 7 25.00 Total 28 100.00 No. of objectives consistent with I￾HELP objectives No. of grants Percentage 1 6 21.43 2 9 32.14 3 6 21.43 4 4 14.29 5 1 3.57 7 1 3.57 8 1 3.57 Total 28 100.00 Beneficiaries Frequency Percentage Medical personnel 5 17.86 Non-medical personnel 19 67.86 Medical & non-medical 4 14.29 Total 28 100.00 10 Table 6. Grants by accomplishment of activities % of accomplished activities Frequency Percentage 75 2 7.14 100 26 92.86 Total 28 100.00 Table 7. Grants by security status Table 8. Staff hired by the projects No. of staff hired Frequency Percentage 2 – 5 5 17.86 6 – 10 9 32.14 11 – 30 5 17.86 31 – 100 5 17.86 101 – 209 4 14.29 Total 28 100 Security status Frequency Percentage Secured 27 96.43 Not secured 1 3.57 Total 28 100.00 11 Table 9. Opinion of the previously established organizations’ managers regarding their organization status Table 10. Rating of the projects’ performance by M & E staff skill and knowledge improvement ability to provide technical assistance better equipped organization than before better performance of the organization more staff than before for the organization Opinion of organization manager No. % No. % No. % No. % No. % Strongly agree 11 84.62 13 100.00 5 38.46 13 100.00 5 38.46 Agree 2 15.38 0 0 4 30.77 0 0 5 38.46 Disagree 0 0 0 0 3 23.08 0 0 2 15.38 Strongly disagree 0 0 0 0 1 7.69 0 0 1 7.69 Total 13 100.00 13 100.00 13 100.00 13 100.00 13 100.00 Indicators of Very good Good acceptable Poor Very poor Total project performance No. % No. % No. % No. % No. % No. % activity performance 19 67.86 4 14.29 3 10.71 1 3.57 1 3.57 28 100 objective achievement 13 46.43 8 28.57 5 17.86 1 3.57 1 3.57 28 100 social benefit 14 50 10 35.71 2 7.14 0 0 2 7.14 28 100 health benefit 16 57.14 7 25 4 14.29 0 0 1 3.57 28 100 capacity improvement 8 28.57 14 50 4 14.29 1 3.57 1 3.57 28 100 project sustainability 14 50 5 17.86 6 21.43 2 7.14 1 3.57 28 100 12 Figure 2 Grants by governorates Baghdad 45% Baghdad & Wassit 4% Basrah 7% Erbil 7% Ninewa 10% Salah al-Din 4% Sulaymaniyah & Tameem 4% Thi-Qar 4% Thi-Qar & Basrah 4% Wassit 11% Figure 3 Grants by ownership NGO 75% Community group 25% 13 Figure 4 Grants by time of organization establishment Newly created because of the project 54% Established before the project 46% 14 5. Conclusions - I-HELP succeeded to cover 9 governorates with rapid impact small grants to strengthen the health system in urban and rural areas. - The grantees were either NGOs or community groups, more than half of them were newly created because of I-HELP. - Around half of the grants comply with 3 or more of I-HELP objectives. - Total number of staff hired by all projects was 1051 Iraqi persons. - The majority of the grantees had accomplished their proposed activities. - There was an improvement in the skills & knowledge, ability to provide technical assistance, and better performance of the previously established organizations as stated by their managers. - The majority of the grants were evaluated as very good or good by the M&E staff. - In general the I-HELP succeeded to achieve it’s objectives in spite of the bad challenging conditions. 15 Small Grant Midterm M&E Report Form GENERAL INFORMATION 1. Project reference No._____________ 2. Starting date of the grant: MM_____DD_____YY_____ 3. Proposed end date of the grant: MM_____DD_____YY_____ 4. Date of midterm M&E: MM_____DD_____YY_____ 5. Project title: 6. Awarded amount ($): 7. Name of project manager: 8. Name of organization: 9. Address: 10. Phone number of the manager: Office________________; Mobile:________________ Email address:_______________________ 11. Project implementation location: (1) rural; (2) suburban; (3) urban 12. Ownership status: (1) Government; (2) NGO with government share; (3) NGO; (4) IO; (5) Community Group TECHNICAL INFORMATION 13. Proposed objectives: Appendix 1 16 14. Reference to I-HELP objectives: (1) (2) (3.1) (3.2) (3.3) (3.4) (3.5) (3.6) (3.7) (3.8) (3.9) (4) 15. Proposed beneficiaries: 16. Proposed quantity of beneficiaries: 17. Proposed activities and timeline: 17 18. Number of activities proposed to be finished by midterm: 19. The number of activities finished by midterm: 20. Number of activities poorly performed: 21. Security problem: (1) yes; (2) no. If yes, specify:____________________________________ _________________________________________________________________________________ 22. Reasons and recommendations for non-performance and delayed performance of activities: Name of activity: Proposed deadline of performance: The _____ week of MM_________YY_________ Status of performance: (1) non-performance; (2) delayed performance Reasons: Grantee’s plan to fix the problem: Recommendations (communicate with grantee): Name of activity: Proposed deadline of performance: The _____ week of MM_________YY_________ Status of performance: (1) non-performance; (2) delayed performance Reasons: Grantee’s plan to fix the problem: 18 Recommendations (communicate with grantee): Name of activity: Proposed deadline of performance: The _____ week of MM_________YY_________ Status of performance: (1) non-performance; (2) delayed performance Reasons: Grantee’s plan to fix the problem: Recommendations (communicate with grantee): Name of activity: Proposed deadline of performance: The _____ week of MM_________YY_________ Status of performance: (1) non-performance; (2) delayed performance Reasons: Grantee’s plan to fix the problem: 19 Recommendations (communicate with grantee): Name of activity: Proposed deadline of performance: The _____ week of MM_________YY_________ Status of performance: (1) non-performance; (2) delayed performance Reasons: Grantee’s plan to fix the problem: Recommendations (communicate with grantee): 23. Reasons and recommendations for poorly performed activities: Name of activity: Description of the problem: Reasons and excuses: Recommendations (communicate with grantee): Name of activity: Description of the problem: 20 Reasons and excuses: Recommendations (communicate with grantee): Name of activity: Description of the problem: Reasons and excuses: Recommendations (communicate with grantee): Name of activity: Description of the problem: Reasons and excuses: Recommendations (communicate with grantee): 24. Grantee’s opinion of the likelihood of achieving the proposed objectives, plan for improvement, and requests for technical assistance Objectives For sure Possible Not possible 21 Plan for achieving the objectives: Request for modifications: Requests for technical assistance: 25. M&E staff’s assessment of the likelihood of achieving the proposed objectives, and recommendations: Objectives For sure Possible Not possible Recommendations and rationale (technical assistance, modification of objectives, re-organization of the grant implementation, fine-tuning the grant activities, and termination of the grant): 26. Technical assistance provided by M&E: 22 27. Conclusions: Name of M&E staff:__________________ Signature:______________________ Date of report: MM______DD______YY_______ 23 Small Grant End-of-Project M&E Report Form GENERAL INFORMATION 1. Project reference No._____________ 2. Starting date of the grant: MM_____DD_____YY_____ 3. Proposed end date of the grant: MM_____DD_____YY_____ 4. Date of end-of-project M&E: MM_____DD_____YY_____ 5. Project title: 6. Awarded amount ($): 7. Name of project manager: 8. Name of organization: 9. Address: 10. Phone number of the manager: Office________________; Mobile:________________ Email address:_______________________ 11. Project implementation location: (1) rural; (2) suburban; (3) urban 12. Ownership status: (1) Government; (2) NGO with government share; (3) NGO; (4) IO; (5) Community Group TECHNICAL INFORMATION 13. Proposed objectives: Appendix 2 24 14. Reference to I-HELP objectives: (1) (2) (3.1) (3.2) (3.3) (3.4) (3.5) (3.6) (3.7) (3.8) (3.9) (4) 15. Proposed beneficiaries: 16. Proposed quantity of beneficiaries: 17. Proposed/revised activities and timeline: 25 18. Number of activities proposed to be finished by the end of project: 19. The number of activities finished by the end of the project: 20. Number of activities poorly performed: 21. The total number of objectives proposed/revised: 22. The number of objectives achieved: 23. Security problem: (1) yes; (2) no. If yes, specify:____________________________________ _________________________________________________________________________________ 24. Reasons for non-performed activities: Name of activity: Reasons: Name of activity: Reasons: 26 Name of activity: Reasons: 25. Reasons for poorly performed activities: Name of activity: Description of the problem: Reasons and excuses: Name of activity: Description of the problem: Reasons and excuses: Name of activity: Description of the problem: Reasons and excuses: 26. Grantee’s opinion of the achievement of the proposed/revised objectives: Objectives Fully achieved Partially achieved Not achieved 27 27. M&E staff’ assessment of the achievement of the proposed/revised objectives: Objectives Fully achieved Partially achieved Not achieved 28. The proposed number of staff hired: 29. The number of staff hired by the project: 30. The total number of the person-months used in the project: (Person-months) 31. Is your organization newly created because of this project? (1) Yes; (2) No If it is newly created, will it be sustained after the completion of this project? (1) Yes; (2) No. If it is not newly created, go to item 32. 32. Capacity improvement for existing organization: Statement Strongly agree Agree Disagree Strongly disagreed The skill and knowledge of your staff are improved Your ability to provide technical assistance to another project is improved Your organization is better equipped than before Your organization will perform better in the future because of this project You have had more staff than before because of implementation of this project 33. Services and beneficiaries: The list of services provided Number of beneficiaries/Quantity of services 1. 2. 3. 4. 28 34. Did you receive any technical assistance during the implementation of the project? (1) Yes; (2) No. If yes (received), is the technical assistance beneficial for the improvement in your project performance? (1) Yes; (2) No. Explain: 35. Lessons and experience learned (by staff and program manager): 36. Lessons and experience learned (by M&E staff): 37. Explanation by the project manager of the secondary, long-term, expanded effects (eg from project region to other regions and to the whole country) of the project: 29 38. Rating of the project performance by M&E staff: Dimensions of performance Evidence-based rating by M&E staff from 1 (very poor) to 5 (very good) Activity performance 1 2 3 4 5 Objective achievement 1 2 3 4 5 Social benefit 1 2 3 4 5 Health benefit 1 2 3 4 5 Capacity improvement 1 2 3 4 5 Project sustainability 1 2 3 4 5 39. Concluding Remarks by M&E staff (success and failure of the project, and the lesson learned): Name of M&E staff:__________________ Signature:___________________ Date of report: MM______DD______YY_______ 30 Workshop Evaluation Questionnaire 1. Name of workshop:________________________________ 2. How old are you? _______years 3. Your gender? 1) Male 2)Female 4. Which governorate are you from?_________________ 5. City where workshop took place?________________ 6. What is your main profession (check only one)? 1) Medical doctor 2) Nurses 3) Technician 4) Statistician 5) Community medicine specialist 6) Engineer 7) Business administration 8) Social scientists 9) Other (specify): _______________ 7. What is your administrative position? 1) Ministry of health officer 2) Director general of the governorate 3) Department director of directorate of health 4) Health center manager 5) Health clinic manager 6) Hospital manager 7) Other 8) No administrative position 8. What is the length of this workshop? _____hours/days 9. What do you think the length of the workshop? 1) Too short 2) Short 3) Just about right 4) Long 5) Too long 10. Is this workshop useful? 1) Very useful 2) Useful 3) Unuseful 11. How is your improvement in your knowledge and skills? 1) Major improvement 2) Minor improvement 3) No improvement 12. How will be the impact of the workshop on your work? 1) Very positive impact 2) Positive impact 3) No impact 13. Would you be willing to participate in additional workshops like this? 1) Very willing to 2) Willing to 3) Unwilling to 4) Very unwilling to 14. What is your satisfaction with the trainers? 1) Very satisfied 2) Satisfied 3) Unsatisfied 4) Very unsatisfied 15. What is your overall satisfaction with the workshop? 1) Very satisfied 2) Satisfied 3) Unsatisfied 4) Very unsatisfied 16 . Please provide your recommendations on how we can improve the quality of the workshop This anonymous questionnaire is used for evaluating the effectiveness of the workshop and asking you for recommendations for further improvement of future workshops. It is greatly appreciated if you would be willing to take a few minutes to answer the following questions. Thank you very much for cooperation. Appendix 3