Submitted to USAID on March 7, 2012 This publication was produced for review by the United States Agency for International Development. EVALUATION Final Evaluation of the Armenia Small Scale Infrastructure Project (SSIP) ARMENIA Serouj, via Wikimedia Commons SSIP EVALUATION REPORT Submitted to USAID on March 7, 2012 This publication was produced for review by the United States Agency for International Development. SSIP EVALUATION REPORT MARCH 2012 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive Suite 220 Vienna, VA 22182 USA Contracted under RAN-I-00-09-00016-00, Task Order Number AID-111-TO-11-00002 Evaluation of ABA-ROLI, CEPPS, SATR AND SSIP Projects DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Authors: Vahé Heboyan and Mayis Vanoyan SSIP Evaluation Report International Business & Technical Consultants, Inc. ACKNOWLEDGEMENTS We would like to express our deep appreciation to the leadership of CHF for providing data and background information on the SSIP activities and patiently responding to our inquiries. We would like to express our appreciation to the Civic Development Partnership Foundation (CDPF) for assisting us with carrying our intensive surveys throughout Armenia under tight time constraints and severe weather conditions. Last, but not least, we would like to thank the village mayors for their time and efforts to assist us in completing field surveys and providing vital information and opinions necessary to properly evaluate SSIP activities. SSIP Evaluation Report International Business & Technical Consultants, Inc. TABLE OF CONTENTS UExecutive SummaryU ......................................................................................................................... i UI.U UIntroductionU ......................................................................................................................... 1 UII.U UScope and MethodologyU ..................................................................................................... 2 ULimitations of Evaluation MethodologyU ....................................................................... 6 UGender ImplicationsU ...................................................................................................... 6 UIII.U UFindings and AnalysisU ......................................................................................................... 7 UA.U UUSAID General Evaluation QuestionsU ......................................................................... 7 UB.U UUSAID Evaluation Questions Specific to SSIPU .......................................................... 12 UIV.U UBest Practices and Lessons LearnedU ................................................................................. 17 UA.U UBest PracticesU .............................................................................................................. 17 UB.U ULessons LearnedU .......................................................................................................... 19 UIV.U UConclusionsU ....................................................................................................................... 21 UV.U URecommendationsU ............................................................................................................. 22 UAnnex I: Evaluation Scope of WorkU ............................................................................................. 24 UAnnex II: Evaluation Work PlanU ................................................................................................... 32 UAnnex III: Informants for Field InterviewsU .................................................................................. 38 UAnnex IV: Illustrative Interview GuidesU ....................................................................................... 39 UAnnex V: Long-Term Employment Survey QuestionnaireU .......................................................... 44 UAnnex VI: Short-Term Employee Survey QuestionnaireU ............................................................. 47 UAnnex VII: Draft Work Plan ChartU ............................................................................................... 51 Annex VIII: Conflict of Interest Statements…………………………………………………….. 52 SSIP Evaluation Report International Business & Technical Consultants, Inc. LIST OF ACRONYMS CDPF Civic Development Partnership Foundation CHF Cooperative Housing Foundation International PRA Participatory Rapid Appraisal SSIP Small Scale Infrastructure Project WFP World Food Program SSIP Evaluation Report International Business & Technical Consultants, Inc. Page i EXECUTIVE SUMMARY Purpose International Business & Technical Consultants, Inc. (IBTCI), under the Evaluation Services IQC, conducted this final evaluation of the Small Scale Infrastructure Project (SSIP) for USAID/Armenia. The purpose of the evaluation is to inform the design and development of future projects for USAID; identify “lessons learned;” assess strengths and weakness of strategies and activities performed by Cooperative Housing Foundation International (CHF); and provide recommendations to USAID for project planning for the next three to five years. Background Since its independence from the Soviet Union in 1991, Armenia’s transition process has been constrained by: crumbled infrastructure, war with neighboring Azerbaijan, economic blockade imposed by Turkey, civil conflicts in neighboring Georgia, and lost trading links and markets. Furthermore, budget constraints have forced the Armenian government to prioritize government expenditures at the expense of infrastructure maintenance and new developments. The result has been the further deterioration of Soviet-built structures. This situation has been especially severe in rural and bordering communities. To mitigate the consequences of the crisis, the USAID initiated the Small Scale Infrastructure Project (SSIP) in 2009 with the objective of creating temporary employment opportunities in vulnerable rural communities through the implementation of projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls, gas and drinking water supply systems. Under the original cooperative agreement, signed in September 2009, the program was slated to last 15 months. Under two subsequent modifications, the program was extended by seven months and three and one-half months, respectively, bringing the new end date for the SSIP to November 15, 2011 Main Evaluation Questions The primary evaluation questions were designed to assess whether or not the project has met its job creation criteria and draw lessons learned from project implementation and end-results for future planning purposes. Methodology The methodology for this evaluation was designed to answer the main questions cited above by taking into consideration the goal, objectives of the SSIP, and the indicators as presented in the project documentation. The goal of the program was to help alleviate the consequences of the Global Economic Crisis on rural, vulnerable Armenian villages, by means of small-scale infrastructure projects that would generate short-term employment and, once the projects were completed, provide longer-term employment. In addition, the infrastructure projects were to be the basis for improved quality of life through expanded utilities and public infrastructure for education. The key objectives for the goal were: SSIP Evaluation Report International Business & Technical Consultants, Inc. Page ii 1. Improved quality of lives of Armenians. 2. Increased employment for vulnerable Armenians in targeted villages. The targets of the program were: • 58 Infrastructure projects completed, benefiting over 40,000 Armenians • Over 47,000 person-days of employment generated • 823 vulnerable Armenians obtaining short-term employment • 142 long-term jobs created. The evaluation began with a desk review of materials provided by CHF and the USAID Mission. Initial interviews were conducted with USAID/Am staff who managed the SSIP. This work was followed up with interviews with the staff of CHF. A sample of 21 projects out of overall 58 projects was selected to be representative of both the geographic coverage of the SSIP and the four types of projects. The Civic Development Partnership Foundation (CDPF), an Armenian NGO, was employed to conduct the interviews of the temporary and long-term employees, and the beneficiaries in all 21 projects sampled. The IBTCI team leader provided the supervision and oversight of this field work to assure data quality and the integrity of the sample. Findings - Overall Through interviews with the village mayors, it was revealed that a majority considered the rehabilitation/construction of infrastructure the principle important outcome for the community over job creation. All mayors interviewed were happy with the project implemented in their community and considered the project either successful or very successful. The evaluations observed some problems with contractors: in several cases they failed to make salary payments in a proper and timely manner and, in some of such cases, interventions by village mayors were needed to ensure that payments were made. In terms of creation of short-term jobs and salary having social impact on citizens it was stated that people who were hired for the job were paid by two types of payments: cash and foods. These payments are different by nature and have different social impact. The complex partnership structure (CHF with WFP, World Vision) made it difficult to properly manage and deliver materials on time. This also made it difficult for mayors to participate or monitor the process of implementation because the lines of responsibility were not clear to them. Findings – Beneficiaries by Type of Project The evaluation focused on the impact of the projects on end-users and beneficiaries of the newly developed or renovated infrastructures. The beneficiaries were asked various questions to help better understand the overall impact on the local communities. Gas system beneficiaries: At the time of the interviews, only 13% of households were subscribed to natural gas through the new systems. Further exploration revealed that the major reason for such low numbers is the high cost of in-house installations and that at current prices natural gas is beneficial only for cooking and bathing purposes. For example, only 22% of SSIP Evaluation Report International Business & Technical Consultants, Inc. Page iii respondents thought that use of natural gas would result in financial savings; 11% indicated that it would not save money; and 15% indicated that it would result in higher energy expenditures. Water system: Over 95% of the interviewees indicted that they currently receive water through the new system, and 95% state that their quality of life has been positively impacted. Similarly, 93% and 98% indicated high satisfaction with the current quality of water for cooking and bathing purposes, respectively. However, 87% and 74% indicated high satisfaction with the previous quality of water for cooking and bathing, respectively. Kindergarten beneficiaries: The surveys indicated that 90.3% of respondents stated that the renovated kindergarten was highly important for their household and that they were highly satisfied by the services provided by the kindergarten. Other infrastructure: One of the important questions in the Scope of Work (SOW) concerned the sustainability of the projects. The beneficiary interviews showed that over 95% of villagers benefited directly from the project. The change in satisfaction is shown by 92% of respondents whom reported dissatisfaction with the previous quality and 92% reported satisfaction of the current quality. 60% indicated that the new system/project has positively impacted their personal quality of life. These high levels of satisfaction bode well for the continued commitment of the community toward maintenance. Employment Generation: The major target of the program was to generate short-term employment for at least 823 vulnerable Armenians.Although the project failed to reach the target directly from the infrastructure employment, it did reach the target when considering the jobs obtained through the partnership with the World Food Program (WFP), where food was provided for labor instead of monetary compensation. Short-term employees: The majority of the interviewees, 88%, were satisfied with the amount of salary that they received and 97% indicated that the salary was very important for their household. However, the majority indicated that lack of alternative employment options limited their bargaining power and that “any money was better than no money” in high unemployment rural communities. A total of 920 short-term jobs were created. However, only 505 of these jobs were directly attributable to the CHF initiative and the remaining 415 were made possible by the World Food Organization (WFO), which was not involved from the inception of the program and provided food as a compensation for the work performed within the auspices of the CHF/WFO agreement. Therefore, it is difficult to attribute the final total of short-term jobs directly to CHF. Furthermore, the agreement did not provide clear results against which progress and impact could be measured, as it did not distinguish whether short term employees should be paid by cash or food. Long-term employees: The surveys indicate that the average salary reported was 31,656 drams per month and 63% of employees were satisfied with the amount of the salary. However, similar to the short-term employment, high unemployment and lack of alternative employment opportunities considerably decreased their bargaining power. Of all respondents, 83% indicated that the amount of the current salary was very important to their household’s ability to function. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page iv Conclusions: The targeted number of short-term employment (823) has been reached and actually exceeded (920) by means of 45% of the total job created by WFP program and 55% by CHF. The jobs created by CHF were paid by cash, while WFP paid with food. Thus if the objective of creation of short term jobs implies payment of short term employees by cash, the project did not reach its objective. Conversely, if the preliminary objective included any type of employment including those paid by food, it did reach its objective. Evaluation findings showed that SSIP was very helpful and did solve critical problems in vulnerable communities, especially for those villages situated in remote areas at a distance from regional centers and capital. One of the important objectives of the program - improved quality of lives of Armenians through the rehabilitaion of small scale community infrastructure has been achieved by the implementation of the program. The program was also an important step toward reducing the obvious disparities of development between capital and regional centers and remote areas. Community involvement, in terms both contributing by community (project co-financing) and participation in all decision making process creates good basis for the sustainability of projects, too. Therefore its seems that the sustainability of projects may suffer in those project where the community was attributed the least role. Or the community contribution is represented by other donor projects. In spite of the coordination issues one of the significant accomplishments of the SSIP was to build vital partnerships with other organizations that extended the reach of the assistance to Armenia’s vulnerable communities, i.e. more communities were assisted. Gas provision is an important issue in all villages, however in many places the gas connection is under question and by the completion of the project most (87%) of the villagers have not connected to the systems because of the high cost of connecting and using gas. The design of future programs should try to anticipate these difficulties beforehand. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 1 I. INTRODUCTION Since its independence from the Soviet Union in 1991, Armenia’s transition process has been constrained by: crumbled infrastructure, war with neighboring Azerbaijan, economic blockade imposed by Turkey, civil conflicts in neighboring Georgia, and lost trading links and markets. Furthermore, budget constraints have forced the Armenian government to prioritize government expenditures at the expense of infrastructure maintenance and new developments. The result has been the further deterioration of Soviet-built structures. This situation has been especially severe in rural and bordering communities. During the 2000s, Armenia has statistically experienced economic growth; however most of it has come from the construction boom in the capital Yerevan and its economy has remained undiversified. The recent Global Economic Crisis has underlined Armenia’s dependence on construction growth and vulnerability to the external economic and political events. To mitigate the consequences of the Crisis, United States Agency for International Development (USAID) initiated the Small Scale Infrastructure Project with the objective of creating temporary employment opportunities in vulnerable rural communities through the implementation of projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre￾schools, community centers, sport halls, gas and drinking water supply systems. The USAID Mission in Armenia partnered with CHF, in association with a leading Armenian community development organization, Shen NGO, to implement the $4.5 million Small Scale Infrastructure Program (SSIP), a program which supports job creation, particularly targeting increasingly poor and vulnerable rural communities. To increase the reach and scope of the projects vital partnerships with various local public and private organizations, such as Shen, World Vision, World Food Program, etc. were created. SSIP projects were selected to address critical issues in local communities and input from various stakeholders were taken into consideration to assess the projects presented by the communities as priorities. SSIP used Shen NGO’s annual Participatory Rapid Appraisal (PRA) process for the competitive selection of projects. The PRA process targets the poorest communities identified through Shen’s vulnerability index. As part of the PRA process, community members, local NGOs, and local governments submit priority project proposals for consideration. Shen NGO worked with communities to identify the labor force within the communities. Then contractors were required to hire laborers who are unemployed and from vulnerable families. Under the original cooperative agreement, signed in September 2009, the program was slated to last 15 months. Under two subsequent modifications, the program was extended by seven months and three and a half months, respectively, bringing the new end date of SSIP to November 15, 2011. SSIP is building upon the decades of CHF International’s construction management experience including the previous five years in Armenia and Shen NGO’s 20 years of community mobilization efforts to work with communities to: target the most-needed infrastructure improvements; identify community labor resources; rehabilitate potable water systems; install village gas distribution systems; and rehabilitate kindergartens, community centers and other essential infrastructure. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 2 The projects have been carried out under the overall supervision of CHF International. Through a competitive process overseen by various stakeholders, building contracts were awarded to a total of 26 firms for the implementation of 58 SSIP projects. Stakeholders participated with monetary as well as non-monetary inputs; Communities were required to contribute five to 10% of the USAID contribution. The purpose of this assignment is to provide a final evaluation of the SSIP which will determine the extent to which the project reached its goals and objectives, as well as to draw lessons learned for future use. II. SCOPE AND METHODOLOGY This is a final evaluation of the SSIP project as implemented by CHF, coinciding with the last months of project implementation. The purpose of the evaluation is to inform design and development of future projects for USAID. The evaluation will identify “lessons learned,” assess strengths and weakness of strategies and activities performed under these projects and provide recommendations to USAID for project planning purposes for the next three to five years. The Scope of Work and methodology (see Annex 1 for the Evaluation SOW and Annex 2 for the Work Plan) for SSIP evaluation activity has taken into consideration the goal and objectives of SSIP. The main goal of the USAID/CHF program was to help alleviate consequences of the Global Economic Crisis on rural, vulnerable Armenian villages by means of small scale infrastructure projects prioritized by the communities. The following key indicators were established to measure the achievement of the goal: 1. Improved quality of lives of Armenians through the rehabilitation of small scale community Infrastructure. 2. Increased employment for vulnerable Armenians in targeted villages. The following were the targets of the program: • 58 Infrastructure projects completed, benefiting over 40,000 Armenians • Over 47,000 person-days of employment generated • 823 vulnerable Armenians obtaining short-term employment • 142 long-term jobs created. The purpose of the evaluation is to reveal aspects mentioned above and particularly to answer the following questions posited in the SOW: how the targets have been met vis-à-vis original targets and benchmarks, particularly what is the ratio of “planned” and “actual” jobs generated throughout the project? In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? How did the partnership between several partners go? How did they (the partnerships) impact the project (timeliness, quality, and cost-benefit)? Did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally? What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies? What should be changed in the design and technical approach in order to get better results? How the completed projects will be maintained by the communities? What SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 3 are the grounds for their sustainability? What are social, economic and other impacts of the project on target communities and beneficiaries, etc? In addition to the SSIP-specific questions, this evaluation report aims at answering several general questions of importance to the USAID and other stakeholders. Specifically: 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? 3. What were the main achievements of the program? 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? 5. How did the implementer perform in terms of project management and how effective was the project leadership? 6. What lessons learned can be provided for future USAID programming in this area? 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? 9. Analyze attribution of project successes to USAID involvement. 10. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. For sampling, the small scale infrastructure projects were grouped into four types of projects: 1) water supply, 2) gas supply projects, 3) kindergarten projects, and 4) other construction projects (such as rehabilitation of community centers, bridge construction, street lighting, sport hall renovation, etc.) The evaluation was designed to use a desk review of materials provided by CHF as background and data, and to provide interviews with four groups of interviewees in the field, i.e. project sites: 1) interviews with 21 mayors, 2) interviews with short-term employees created during implementation of 21 project sites – villages where SSIP have been implemented, 3) interviews with long-term employees created in the villages where SSIP have been implemented, 4) interviews with CHF acting Chief of Party (CoP) and contractors/construction companies (by telephone). Questionnaires were prepared beforehand and had different specifications to fit with the type and character of projects. For example, specific questionnaires were developed for water projects, for gas construction, and for the nine Marzes in which CHF completed 58 projects. The sample size for the evaluation was planned to be approximately 33% of total number of projects. In Kotayk, Armavir and Vayots Dzor Marzes CHF implemented one project in each Marz. In order to cover all Marzes, these three projects were added to the sample. As a result, the total sample size was 36% of total number of implemented projects. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 4 Additionally, to cover all types of projects, those having specific and unique types of projects have been included in the sample “by default” as well, for example the ventilation project in Akhurian, bridge construction in Gegharkunik Marz, etc. To be consistent, the sample size for each breakdown (by Marzes and types of project) has been defined on the basis of proportionality. It means that the sample size based on location (Marzes) or types of project varies from 30-43% from the total number of projects with an average of 38%. For example, sample size of water projects to be evaluated is 33%, while the size of preschool renovation is 43% from the total number of projects of this type. Table 1 shows the breakdown of projects by type and by Marz and Table 2 provides sampling by types of projects. Table 1: Sampling by Marz Marz Number of implemented projects Number of projects to be evaluated (sampling by Marz) Breakdown by types of projects Water Gas Preschool Other projects/note Gegharkunik 12 4 1 1 1 (Community Centre) 1(Bridge) Shirak 11 3 1 1 1 (Ventilation) Lori 6 1 1 (Community center) Tavush 11 4 2 1 1 (Lighting) Syunik 5 2 1 1 (Sport hall) Aragatsotn 11 4 1 1 1 1(Computer classes) Armavir 1 1 1 (Preschool) Kotayk 1 1 1 Vayotsdzor 1 1 1 (Sport hall) Total 58 1 7 2 3 9 SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 5 Table 2: Sampling by type of projects Types of implemented projects Total number of implemente d projects by types Number of implemented projects in Marzes Sample size by types of project Samples pro rata to number of projects in Marzes Water projects (40% from total number) 23 Gegharkunik3, Shirak4, Lori 3,Tavush 7, Kotayk1, Syunik3, Aragatsotn2 7 Gegharkunik1, Shirak1, Tavush2, Kotayk1, Syunik1, Aragatsotn1 Gas projects – (10% from total number) 6 Lori 1, Shirak3, Aragatsotn2 2 Shirak1, Aragatsotn1 Preschool renovation – (12% from total number) 7 Lori 1, Aragatsotn4, Tavush1, Gegharkunik1 3 Aragatsotn1, Gegharkunik1, Tavush 1 Other projects (38% from total number) including: Ventilation, Lighting, Computer classes, Community Centers, Sport halls, Bridge construction 22 Ventilation 1U (Shirak), Lighting 4U (Tavush3 , Lori1), Computer classes 3 (Aragatsotn2, Gegharkunik1), Community centers 7(Gegharkunik3, Tavush1, Shirak1, Aragatsotn1, Lori 1), Sport halls6 (Syunik2, Tavush1, Shirak1, VayotsDzor1, Armavir1), Bridge 1 (Gegharkunik), 9 Ventilation1 (Akhurian/Shirak) Lighting1 (Teghut/Tavush) Computerclasses1 (Aragatsotn) CommunityCenters2 (Avazan/Gegharkunik1, Shahumian/Lori 1) Sporthall2, (Myasnikyan/Armavir1, Malishka/VayotsDzor1, Shinuhair 1/Syunik) Bridge 1 (Ttujur/Gegharkunik1) Total 58 21 (36%) With the assistance of CDPF, a total of 599 surveys were carried out. Table 3 displays the breakdown of each survey category. Surveys were administered within the community to solicit community input and opinion on various aspects of the projects. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 6 Table 3: Distribution of field surveys by project Region Community ST LT BO BK BW BG Total 1. Tavu sh Teghut 6 1 20 27 2. Tavush Sevqar 6 13 20 39 3. Tavu sh Kirants 5 1 20 26 4. Tav ush Voskevan 11 1 20 32 5. Sh irak Sarnaghbyur 1 0 24 25 6. Shi rak Pemzashen 7 0 20 27 7. A ragatsotn Zarindja 4 1 26 31 8. Gegharqunik Vahan 4 11 25 40 9. G egharqunik Kalavan 2 0 21 23 10. G egharqunik Ttujur 2 0 35 37 11. G egharqunik Dprabak 3 0 30 33 12. A ragatsotn Shenavan 5 0 30 35 13. A ragatsotn Ashnak 1 1 28 30 14. A ragatsotn Parpi 0 0 27 27 15. Vay ots Dzor Malishka 3 1 29 33 16. Syu niq Shinuhair 2 0 32 34 17. Sy uniq Kornidzor 4 1 20 25 18. Lori Shahumyan 4 4 21 29 19. K otaik Kaputan 5 0 20 25 20. Arm avir Miasnikyan 1 0 20 21 21. S hirak AkhuryanF 1 0 0 0 0 0 0 0 TOTAL 76 35 215 72 147 54 599 Note: ST=short-term employee, LT=long-term employee, BO=beneficiary other projects, BK=beneficiary kindergartens, BW=beneficiary water projects, and BG=beneficiary gas projects. Limitations of Evaluation Methodology The generalizations of the evaluation results are constrained to the sample studied and do not necessarily represent the whole SSIP population. Another limitation of the results is that many of the short-term employees were not available to be interviewed at the time the field work was conducted because they were either abroad on migrant work or serving in the army. Gender Implications 1 In Akhuryan, Shirak, a sp ecial project was included at th e request of USAID. SSIP h elped to install a v entilation system for the local hospital. Due to the nature of specialized installation, we learned that no locals were hired and all work was completed by professional installers represen ting the firm that provided the equipment. Community involvement was not so licited an d also its d irect im pact is not a pparent t o t he h ospital bene ficiaries. F or t hese reasons, beneficiary or employee surveys were not conducted. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 7 Due to the character of the construction works, there is no equal involvement of women and men in the process of implementation. We revealed only 3 cases of employment of women who were hired by WFP project. III. FINDINGS AND ANALYSIS The purpose of the evaluation is to answer questions posed in the evaluation SOW, which will reveal important information concerning short-term and long-term jobs, program partnership, cost share, sustainability, social and economic impacts of the projects and other issues. Utilizing the methodology described above, the Evaluation Team’s findings, described in the following section, are presented as they pertain to the specific evaluation SOW questions. Some questions are grouped because the findings are the same. A. USAID General Evaluation Questions • Did the agreement provide clear and achievable results against which progress and impact could be measured?2 The SSIP agreement presented clearly the following results against which progress and impact could be measured. • 58 Infrastructure projects completed, benefiting over 40,000 Armenians • Over 47,000 person-days of employment generated • 823 vulnerable Armenians obtaining short-term employment • 142 long-term jobs created. However, based on the second objective of the program, increased employment for vulnerable Armenians in targeted villages, there is no specification of how this employment is defined. For example, the evaluation team can measure the number of short-term and long-term jobs that were provided as a result of the program compared to the expected result. Yet, when looking at the short-term jobs, while a total of 920 were created, CHF is directly responsible for only 505. An outside partner who was not involved from the inception of the program brought an additional 415. Therefore, it is difficult to attribute the final total of short-term jobs to CHF. In other words the agreement did not provide clear results against which progress and impact could be measured for the objective of creating short term employment, as it did not distinguish whether short term employees should be paid by cash or food. • To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? Table 4 demonstrates that SSIP was able to meet or exceed all of the targeted objectives except for the number of long-term jobs. Table 4: SSIP targeted goals and actual achievements 2 The l atest sources CHF provided for desk review is SSIP Quarterly Report for submission to USAID, Reporting Period A pril 1-June 3 0, 2 011 (end of 7 th qua rter) and . The CHF fi nal report was una vailable be fore USAID approval. Therefore, some data such as long-term jobs is estimated to be 121 as of June 30, 2011. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 8 Objectives Targete d Actual Number of Infrastructure projects 58 58 Number of Long-term jobs 142 121 (estimation) Number of short-term jobs 823 505 CHF / 415 WFP (total 920) Labor days 47,000 person-days 71,077 Beneficiaries: o ver 40,000 72,876 One of the main objectives of SSIP was to create temporary jobs for vulnerable Armenians. Each project generated short-term jobs in the communities where SSIP activities were implemented. . The target number of Armenians who would obtain short-term employment is 823 and long-term employment is 142. The actual number of short-term jobs created, 920, exceeded this target and so the program did achieve this result. Of these short-term jobs, CHF directly contributed to 55% of them and paid the employees by cash. The World Food Program helped create 45% of the jobs and paid the employees by food. Long-term jobs were created after the completion of the infrastructure projects, and not all infrastructure projects demanded the creation of long-term jobs after completion. At the time of the evaluation, the evaluation team was provided an estimation that 121 long-term jobs were created, but it can be assumed that there is a high probability that additional long-term jobs will be created as a result of the projects. It is also important to note, that of the two objectives, rehabilitating small-scale infrastructure was more important to the mayors than creating jobs. Based on the interviews the evaluators did with the mayors, it appeared the mayors were not even aware that one of the program’s objectives was to create jobs. • Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? As in any other development project, sustainability of SSIP is crucial to ensure continued benefit and development for the affected communities. This is especially crucial for vulnerable communities with very limited financial resources. Our evaluation results indicated mixed results in terms of sustainability of the SSIP projects. For example, on the positive side, the gas delivery systems, upon completion, were transferred to the national gas delivery company jointly owned by Armenia and Russia, and they will assume all maintenance related activities. Similarly, because the management of the water delivery systems was handed over to the Armenian Water Sewage Company, sustainability has been achieved. On the other hand, projects such as those that helped renovate various vital infrastructure buildings (schools, kindergartens, community halls, etc.) remain under the jurisdiction of the local communities, they must assume all maintenance and upkeep of these constructions. Because their tax base is very limited, they may not be able to maintain these structures and therefore sustainability may be at issue. Therefore, they should request active participation from local government and seek community contributions to leverage a sense of ownership for community/local government and then it will be more likely that the community will take care of the new system. • What were the main achievements of the program? Analyze attribution of project successes to USAID involvement. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 9 The CHF small scale infrastructure projects in Armenian communities were very helpful and solved very important community problems, especially for those villages situated in remote areas and distant from regional centers and capital Marzes. Several villages improved their infrastructure which were considered to be one of the most important accomplishments of the SSIP. Therefore one of the important objectives of the program - improved quality of lives of Armenians through the rehabilitaion of small scale community infrastructure - has been achieved by the implementation of the program. The program was also an important step to decrease the extent of disparities in development between capital and regional centers and remote areas. SSIP is a valuable contribution by USAID necessary for the further development of Armenia. It is a successful practice of efficient and reasonable use of USAID resources in Armenia, in terms of building vital partnerships with other organizations that extended the reach of the assistance to Armenia’s vulnerable communities (so that more communities will be assisted). • How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? • How did the implementer perform in terms of project management and how effective was the project leadership? • Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Overall, CHF/SSIP achieved high degree of collaboration with other donors and the communities. This high degree of collaboration demonstrates that the interventions were highly relevant. Because the most of the resulting products of the interventions were also of high￾quality, the overall program itself was well-defined and responded to the theory of change/the development hypothesis of the program which was that brining communities together to improve infrastructure and increase jobs would improve the lives of Armenians. Additionally, overall project management and leadership performed well as evidenced by the high degree of success with bringing in donor and community involvement for the projects. This collaboration created a chance to expand the budget of projects and enabled for more activity to be implemented. Also the coordination with other donors contributed to the high level of quality, as with the case of Sevkar (described below). However, when the community was less involved, as was the case in some instances, the project management and oversight was weak because the community did not feel empowered to fully commit to the projects. This caused delays in implementation and a negative impact on the quality of the project, with Pemzashen and Malishka as examples. In sum, the success of the projects was reliant on the involvement of the community and local government in its implementation. When there was high involvement, the project was successful. Low community involvement created less desirable outcomes. Future small-scale infrastructure projects should ensure that no matter where the actual matching project funds comes from, community involvement and ownership of the project is essential for success. For example, in the case of Sevkar, the village provided $28,766 in matching funds to renovate the kindergarten. This input was approximately one-third of the total budget. The community itself led the project and organized the works properly, succeeded in monitoring the quality and kept control over the construction contractor, including payment for short-term employees, etc. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 10 Therefore in case of Sevkar Kindergarten Construction project not only succeeded in terms of completion of project on time, project fund efficiency, further sustainability, but also was very efficient and the collaboration of several partners was on highest level. In this case we can state that CHF/USAID fund triggered serendipity effect. Another successful case of the community participation and co-financing is Teghut Village Street Lighting construction project. Due to preliminary works with the community, starting with needs assessment and proper distribution of roles in the project, the community contributed the principle materials for implementation and provided all labor by itself, without provision of tender. This led to a very quick implementation and the project was completed on time, while individuals who participated in the construction were paid by the project budget. Citizens of the community have been involved in the construction, provided in kind contribution (principal materials) and solved one of the acute problems prioritized by citizens of communities. The less successful cases resulted largely from when an outside donor was in charge of a project and the community contribution was minimized. In these cases, the community did not take ownership of the project, i.e. it was not required to co-finance the project, and instead the contribution has been provided by the other donor, for example the World Food Programme (WFP). As a result, the community took very passive role. The case of Pemzashen Village Potable Water Project is a visual example. The community did not provide matching funds and WFP provided food for temporary project workers and actually paid off the community labor, which was considered to be (or viewed as) community contribution. Thus the municipality was attributed very passive role in the project. It did not participate in the selection of the contractor (the mayor even was not invited to participate). Partially the reason of the weak involvement of the community was connected with the poor provision of information and the use of community awareness and transparency mechanisms. The municipality did not take control over the contractor, was not aware of the project details and as a result the contractor did not complete works timely and completely. In the case of Pemzashen Village, the chlorination station allotted by the project was not completed, which will impact on sustainability of the project. In addition the project partner (RAEDP) did not provide material delivery for the project and the project was completed with delay. Overall, when there was a lack of transparency in project implementation, obstacles were created for collaboration and trust of the implementing partners diminished leading to failure of full completion of the projects. The types of projects implemented at the community level were diversified and consisted of four main categories: water, gas, pre-schools, and other construction projects. Water system rehabilitation projects accounted for 40% of total. Thirty-eight percent of the projects were in the construction category (i.e. community centers, sport halls for youth, bridge, computer classes, etc.) Preschool and kindergarten renovation projects accounted for 12% of total. In the villages, gas projects totaled 10% of the total and generated an important issue for the SSIP project that was not foreseen. Even though gas is now available to villagers, many have refused to connect to the gas because it is too expensive and they refuse to pay for it themselves, as is the case with Shenayan village. Hence, there are technical tasks with the gas projects that SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 11 are incomplete. however the gas connection is under the question and by the completion of the project there still remain incomplete technical tasks. Another issue that arose with the gas project is that the major beneficiary of the gas was the private company HAYRUSGAZARD rather than the community. Even though communities can now access gas when they could not before, it is too expensive and the private company would profit from it more than the villagers. • What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? • What “lessons learned can be provided for future USAID programming in this area? • If a similar activity is considered in the future, what should be changed in the design and technical approach?3 In the future it is recommended to use different two approaches for infrastructure projects in community selection and implementation: 1) the community provides in-kind contributions, and 2) the community can co-finance the project or provide matching funds. In both cases the community involvement must be required without substituting it with other donor contribution. It is also possible to apply both approaches together. It is evident that communities are willing to improve their infrastructure and service delivery. Communities which do not have enough resources and have budget constraints may participate in the project by contributing in-kind (materials, labor). This will work when the type of construction needed does not involve a specialized professional company to meet state requirements (current regulations of Ministry of Urban Development), such as street lighting, renovation (not new construction) of potable water pipeline, renovation of internal spaces, etc. For these types of projects, the participatory approach can be applied, which will less costly, more efficient and transparent, avoiding involvement of contractors. When a community is more financially capable and can provide cash contribution from the budget, the community should be required co-finance the project. The principle drawback of the SSIP project design in creating employment in rural areas is discrepancy of payments that were offered to employees. The employees of the CHF projects were paid in cash and those for the WFP projects were paid food delivery. These two types of remuneration are principally different. Employees receiving cash payment for works are in more priviliged position than those receiving foods for works. The communities should have been made aware of how the employees were going to be remunerated before the projects were designed. The design of the future similar projects should take into consideration that the municipality should have the role as the principle partner and coordinator of the projects and be required to provide co-financing, matching funds or in-kind contributions as a precondition for receiving the project. 3 Th is is general qu estion but is in terrelated with sp ecific questions above; therefore we broug ht it tog ether with specific questions provided in SOW. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 12 Cost share and collaboration of various partners should be designed in the way the involvement of a partner will be less dependent (if possible) on the activity of other partners. For example, if the construction is provided by one partner and furniture is provided by others, the project will be more manageable and coordination will be easier, while delivery of one type of construction materials can hinder the whole process of construction and consequently cause delay of project implementation. Generally while scheduling the project timeline, it is important that it is realistic for each of the contributing partners and take into consideration seasonality of construction. It is also important to take into account the difficulty of management and coordination of several different partners (with different policy and operation style, etc.) the community/local government /municipality should be attributed the principle role: a. requiring community match/co-financing (in kind or in cash) as a precondition for funding In case of designing similar projects more attention should be paid to transparency of the project, and conducting community outreach and awareness (flyers, brochures, community meetings, periodical meetings with CAG representatives, councils) that encourages wide citizen involvement in the decision making. This will help in both establishing transparency, creating good collaboration between partners and consequently establishing trust of community members towards partners. These comments stem from the outcomes of some of the projects. For example, some projects were not completed on time due to the season and the geographic area, especially with the water and gas projects that require trench digging (especially by hand) because it is very difficult to do when the soil surface is frozen. Because the findings from gas system development projects indicate that currently only 13% of interviewees were buying natural gas, SSIP should have had better, advanced communication with the gas delivery company before starting in order to assess the ability and willingness of the villagers to buy the gas and the service. B. USAID Evaluation Questions Specific to SSIP • What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)? • In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? Findings: • The number of individuals who received short-term employment generated by the CHF project was 505 as compared to the projected number of 823. Combined with the jobs created by WFP, 415, the total number of short-terms jobs is 920, which exceeds the target of 823. WFP’ • WFP paid 415 temporary jobs not by cash, but by food. • In two cases contractors cheated and either did not pay for professional works (Pemzashen) or did not complete their tasks properly (Malishka). SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 13 • According to Shenavan village mayor, it was very difficult to convince local short￾term employees to agree to work during severe winter and dig frozen soil for standards of gas pipelines for an obviously low rate (1,000 drams per pit for the one standard). • As a rule, the salary rate paid to short-term employees by CHF projects varied from 3,000 to 5,000 drams per day. Analysis: The total number of short-term employment opportunities generated should be broken up into two sections: 1) jobs created by CHF and paid by cash; and 2) jobs created through WFP paid by food. It is unclear from expected result as laid out in the program design, or 823, whether it was envisions that CHF should be directly responsible for generating all 823 jobs or whether it was planned that other contributing partners would also contribute to the number. Therefore is it difficult to attribute the success of exceeding the number of short-term jobs created solely to CHF. It also suggests that CHF had to bring in WFP in order to successfully achieve the expected result. Additionally, the jobs created by CHF and WFP projects differ in that employees for WFP projects were paid by food and could not as a result pay for the communal services they were building or provide for daily expenses. Alternatively, the payments made by CHF projects were quite sufficient and tangible to take care of households needs, especially in rural areas. • How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? • How did the partnerships impact the project (timeliness, quality, and cost-benefit)? Findings: • Several organizations (RAEDP, Shen, WFP, WV, UNDP, CoAF) that participated in the program shared the cost of inputs (see the table above). • The partnerships became too complex making project management and material delivery difficult and untimely. • Parnerships made it difficult for mayors/communities to participate or monitor the process of project implementation. • Three village mayors out of 21 interviewed indicated that the community was not invited to participate in the project tendering process. • Three mayors indicated that the community was left out of budgetary related process. • Local government leadership was not systematically involved in the process of implementation, hence local capacity building was not enhanced. • Around 50% of observed projects were not completed on time and CHF has partners in those projects. • In 13 cases out of 21, the mayors could not answer the budget related questions (total budget), although they remembered the amount of their community’s contribution (if any). • 26 out of 58 projects did not have community matching contribution. Instead of those communities matching funds were provided by other donors (WFP, etc.). SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 14 Analysis: The involvement of well known international and local organizations (RAEDP, Shen, WFP, WV, UNDP, CoAF) provided a chance to expand project opportunities and enables more projects to be carried out. However, these partnerships created difficulties in the management of projects, specifically resulting in the slow delivery speed of materials. According to the majority of mayors interviewed, these delays possibly occurred due to a lack of understanding among partners of their responsibilities, which was a source of embarrassment. This occurred in the following cases: Pemzashen, Shenavan, Malishka, and Kornidzor. Additionally, when these other organizations were involved, the significance of community involvement and the community as a project partner was diminished. • Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented? Findings: • In some rare cases mayors do have documents on file to support the figures in their budgets, but as a rule they did not show any. • The total contribution of partners mentioned above is significant (see the Table 5 below). This cost sharing extended the budget for the program, which enables either to increase size of project, or extend project areas. Table 5: SSIP partners and matching contributions Project partners Matching contribution (as of June 30, 2011) Shen NGO $296,637 Community contribution $180,102 RAEDP $ 673,533 World Vision Armenia $57,286 WFP $ 172,459 UNDP $4 8,086 COAF $ 104,670 Other $2 9,825 TOTAL $ 1,562,598 Analysis: SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 15 Cost sharing among partners is generally in place. The scarcity of documentation can be explained by either the reluctance of officials to show documents or lack of documentation. • How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? Findings: • The majority of mayors interviewed (17 out of 21) are allocating budget money for future maintenance of the SSIP infrastructure facility. • In three out of seven water project evaluations, the issue of chlorination and disinfection of the water supply remains unsolved (in one case funds were allocated to construct a disinfection facility, but the contractor did not do so). Since these are not completed, it is less likely that these projects will be sustainable. • Some facilities allotted by the design and project budget have not been constructed (Chlorination in Pemzashen, a Community Center bathroom in Dprabak). Analysis: The maintenance of the projects has been well planned as revealed through interviews with mayors. However in some cases facilities allotted by the project budget have not been constructed and completed by the contractor. In those cases, additional resources need to be invested by the community to maintain the project and ensure future sustainability. The lack of a chlorination station definitely will have a negative impact on future maintenance of the water supply system, as the community will be required by the State agency to abide by regulation requiring disinfection of water. The same is related to other projects where the facilities planned by the project have not been completed by the contractor/construction company. This is because the community could not provide oversight to the process. The likelihood of sustainability would increase if communities played a more significant role in the process of tendering and monitoring of implementation and would better arrange the project after implementation. • How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.? • What are social, economic and other impacts of the project on target communities and beneficiaries? Findings: • Over 50% of mayors interviewed considered that the rehabilitation/construction of infrastructure was the principle and most important outcome of the project. • Those mayors who highlighted the importance of both outcomes, i.e. rehabilitation/construction and creation of jobs, gave preference to infrastructure rehabilitation. • All mayors were pleased with the project implemented in their community and consider the project either successful or very successful. Analysis: SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 16 The projects have considerably improved the quality of life in target communities. Communities overwhelmingly considered the SSIP implemented projects critical and solved an important problem facing their community. Over 95% of beneficiaries from the water system construction projects have indicated that the quality of their life has changed due to the access to cleaner water. The beneficiaries of the civic construction projects are overwhelmingly (about 90%) satisfied of the services and quality of these projects. Direct impacts on beneficiaries and end-users: An important emphasis of this evaluation is the impact of the projects on end-beneficiaries or end-users of the newly developed or renovated infrastructures. The beneficiaries were asked various questions to help understand the overall impact on the local communities. The sections below highlight the major findings of these surveys by type of project. For gas system delivery projects: • 100% of beneficiaries indicated that their household currently has the potential to directly benefit from the new system because the gas pipeline reaches their street, however only 13% of beneficiaries indicated that they have connected and use the gas. The remaining 87% of the households have not connected because they cannot afford the cost of the hook up and the cost of the gas. Survey results also identified that current high prices for natural gas and high costs of installations were the major reasons behind such a low percentage of current use. • 22% of respondents thought that the new source of energy (i.e. natural gas) will result in financial savings for their households. 11% indicated that it will not save money, and 15% indicated that the use of natural gas will result in higher energy expenditures. 52% were not able to answer the question. • In spite of the gas rehabilitation projects 43% of respondents indicated problems with the supply of natural gas. • Many people refuse gas consumption due to the high price of gas. For kindergarten renovation projects: • 98% of respondents indicated that the renovated kindergarten had a high importance for their household. • 90.3% indicated that they were highly satisfied of the services provided by the kindergarten. • 15% reported some problems with the renovated kindergarten that were mainly related to the lack of proper/enough furniture and rooms for extracurricular activities such as sports. • 16% indicated that they do not pay any tuition for the kindergarten and the remaining interviewees, on average, indicated that they make monthly payments for the kindergarten services in the amount of 3,200 drams. • When asked a hypothetical question if they would be willing to make a financial contribution for future maintenance and upkeep of the facilities, 61% of respondents indicated a willingness to pay 500 or less drams monthly; 28% were willing to pay over 1000 drams, and those remaining could not answer the question. For long-term job creation: SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 17 • The average salary reported was 31,656 drams per month. • 63% of employees were satisfied with the amount of the salary, though many indicated that high unemployment and lack of alternative employment opportunities considerably decreased their bargaining power. • 83% indicated that the amount of the current salary was very important for their household. For other projects: Other projects were primarily renovation/construction of facilities of public importance and use, such as projects to develop computer rooms for public access or sport centers, etc. • 100% indicated that they or their family directly benefits from the project. • 92% reported dissatisfaction of the previous quality. • 92% reported satisfaction of the current quality. • 60% indicated that the new system/project has positively impacted their quality of life. For short-term employees: • 88% indicated satisfaction with the amount of salary. • 97% indicated that the salary was very important for their household. For water delivery systems: • 100% indicated that their household was a direct beneficiary. • 99% indicated that they currently receive water through the new system. • 95% indicated that their quality of life has been positively impacted. • 93% and 98% indicated high satisfaction with the current quality of water for cooking and bathing purposes, respectively. • 87 and 74% indicated high satisfaction with the previous quality of water for cooking and bathing, respectively. IV. BEST PRACTICES AND LESSONS LEARNED The SSIP evaluation team has identified best practices and lessons learned that can be taken into consideration when designing and/or implementing projects of similar scope in the future. Two best practice examples and two lessons learned examples are described. 0BA. Best Practices Teghut Village Street Lighting Project An example to illustrate this best practice took place in Teghut village which is adjacent to Dilijan. Long-time village habitants have not had street lighting in the village territory, which has caused much inconvenience and people had difficulties to move from one place to another. Since Dilijan is considered to be one of the resorts centers in Armenia, the lack of lighting is an obstacle for tourism development as many of Yerevan’s inhabitants vacation there during the SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 18 summer. The lack of lights in the streets of Teghut caused problems not only for inhabitants of the village, but for cars passing through the village. People are afraid to walk along the street during late hours and there is also the risk of car crashes because of wandering cattle on dark streets. Street lighting is a mandatory responsibility of local governments in Armenia and therefore the village leadership was very interested in rehabilitation and construction of street lighting. The existing network was damaged in 1990 and since then there has not been any lighting on the main street of the village. The proposed solution was to renovate 1,400 meters of the existing network which was salvageable, and added another 600 meters of lighting (to each side of the original 1,400 meter line). The middle part was renovated where the power lines went through trees, which resulted in short circuits. The direct beneficiaries are those residents living on the main street (approximately 200 people), while the number of indirect beneficiaries includes the entire population and people traveling through the village at night. NGO Shen has been involved in the work organizing communities and delivering materials (paints, cable, etc) for implementation, while the main materials, including the metal pipes, have been contributed by the village itself. The project did not hire a contractor because the character of work was not overly difficult so villagers could carry out the activities themselves. Up to eight temporary jobs (unskilled and skilled works) have been created and were paid by CHF, and two long-term jobs have been created. As the local government is responsible for the operation and maintenance of the lighting system, the village municipality is paying for maintenance of the new lighting system. This project is a classic example of promoted, participatory self-help. It responded to a very important village need, thus solving one of the important infrastructure issues in the village. The villagers participated in all of the processes and they provided self-help to the project. The project succeeded by not having to deal with a contractor-organization and completed the works in a cost effective way. It is evident that not all types of projects can be implemented like this because in majority cases state agencies require certification of professional works in construction. Also, the project satisfied all the required indicators: construction of an important village infrastructure, creation of work places for vulnerable families, and creation of long-term jobs for the maintenance and sustainability of the lighting system. Sevkar Village Kindergarten Reconstruction Project Sevkar kindergarten building was built in 1973 and served 120 children annually. The kindergarten has been closed since 1989, when the war began with Azerbaijan. Soldiers were located in the facility, which led to its poor condition. Before the reconstruction, the building was structurally sound even though it had not been used since 2000. The reconstruction of the kindergarten was a top priority for the community, as the project was designed to completely reconstruct the building and landscape the site. Due to the importance to the citizenry, they decided to provide a significant contribution and to sustain the facility after completion. World Vision agreed to provide all of the new furniture. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 19 SSIP completely reconstructed the kindergarten, including a new design of the interior, installation of windows and doors, replacement of the floor, plastering and painting walls, electric wiring, hot and cold water system and a new roof. CHF installed a heating system (gas is located next to the kindergarten building) and addressed landscaping at the site. In addition to the USAID assistance to the village, the UNDP/GEF Small Grants Program supported the construction of a solar heating system for the kindergarten and solar accumulators that provide environmentally friendly energy to the system, and then Shen NGO, World Vision Armenia, and Bridge of Hope NGO provided furniture, kitchen appliances and a piano for the kindergarten. The community contributed significant financial resources to the project as the cost share was about one-third of the total cost. The project has been completed on time, and now the kindergarten serves 70 children per year. According to the Director of the kindergarten, there are still children who cannot attend as they have reached capacity limits. Now the kindergarten employs 16 people (director, fulltime cook, accountant, medical nurse, five teachers, six nurses, and a part-time musical leader. It exceeds the usual number of long-term jobs created in other projects by five to ten fold, as normally it is not more than two to three jobs (if any). The project implies a new and environmentally friendly technology of solar heating system, which is extraordinary for SSIP. Finally, the project involves several partners (CHF, SHEN, WV, UNDP, Bridge of Hope) offering significant contributions and solving the issues associated with the kindergarten in Sevkar. 1BB. Lessons Learned Pemazashen Village Water Supply Renovation Through SSIP, the Pemzashen village water supply was renovated by CHF through the construction of a 500 meter, 160 mm pipe which stretched from the reservoir to the first water distribution junction. This resulted in a total of three pipes along this 500 meter stretch: 150 mm, 200 mm and a new 160 mm. At this first water distribution junction, SSIP built a 1.0 kilometer, 200 mm pipeline which re-routed the cemetery, and brought water to the existing second water distribution junction. Interviews with the mayor, coordinator of the project, community members and short-term employees indicated that despite the project improving conditions of the potable water supply, it suffered from several shortcomings. According to the main objective of the program, eight people were hired from the village for short-term employment from the village, two of whom were highly skilled workers (a welder and a valve installer). Until now, they did not receive payment for their work and were cheated by the engineering firm CJSC. At least one short-term employee indicated during an interview that they received a bonus for finishing the work on time. Five others indicated that the payments/transfers were considerably late; two indicated being told that there was no money. Three additional workers employed by WFP indicated they received only food (flour, cooling oil) instead of monetary compensation. The two workers who did not receive payment for their work was because they did not have a contract with CJSC and CJSC took advantage of that. This undermined the community’s trust towards CHF and USAID. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 20 Additionally, during the tendering and bidding stage of the project the mayor was assured that he would be involved in the award process, but when this took place he was not consulted. Instead, CHF worked directly with WFP who worked directly with the contractors. With multiple levels of partners, the construction and planning processes were impeded. One of the project partners, RAEDP, did not send their materials on time which led to a work delay of almost four months. This delay forced villagers to work during a severe winter. The project planned start date was decided to be July 01, 2010, with an end date of September 9, 2010. Due to the delays, the actual date of completion was December 31, 2010. WFP was responsible for the bulk of the labor that was allotted in the budget (with the exception of eight workers who were supposed to be paid from CHF budget) and paid off the community contribution. In hindsight, the community should have been required to contribute this amount themselves since this responsibility often triggers a sense of ownership among citizens and enhances the likelihood of sustainability. Co-financing is also a useful tool to ensure sustainability. Ttujur Village, Ghegharkunik Marz – Ttujur Bridge Reconstruction The Ttujur Bridge crosses the Gedik River and connects the village to approximately 60% of the community’s hayfields and agricultural land located on the other side of the river where there are approximately 400 plots of land. As the result of poor construction, the bridge was no longer able to carry heavy loads of agricultural machinery and villagers were forced to go through the river to reach their fields. This caused problems with the brake pads and increased the risk of automobile accidents. Sometimes, when water levels rose and the conditions were turbulent, villagers could not cross the river and were forced to use the bridge. These realities contribute to the fact that only 10 % of the lands in Ttujur community are currently cultivated. Under the framework of SSIP, the 12-meter span of the Ttujur Bridge has been re-constructed, including new foundations, railings and protective walls along the base to protect the bridge from the river’s flow. The new bridge will ensure that agricultural machinery and equipment do not have to traverse the river. While there were no long-term jobs created due to the nature of the project, the bridge is of high significance to the direct beneficiaries of the project: the 1,103 inhabitants of Ttujur community. The pitfall in this project was the lack of communication between the community and the implementing organization. This was illustrated by two major omissions during the design and construction phases, which resulted in a less functional bridge (despite the fact the bridge was built according to Armenia’s standards). The first omission was that the bridge was constructed too narrowly for certain agricultural machinery, such as combine harvesters, to safely pass. Second, the bridge railings were installed in a way that they created physical obstacles when trucks attempted to cross the river with heavy hey loads. During interview with the community head, it became apparent that during the design and construction phases the community was not properly involved. Even though the community indicated the issue with the width of the bridge and potential rail obstacles early on, these concerns were ignored and not incorporated into the overall design and construction process. Of SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 21 these failures, the railings can be modified to allow safe and easy passage, although this corrective action will require a significant financial investment from the community. This story highlights the importance of a participatory approach to these types of development initiatives. Communities are in a unique position to assist implementing and donor organizations to better plan and implement vital development projects such as the Ttujur bridge project. In the future, it is highly recommended that local communities and stakeholders are consulted at every stage of the project design, and that they remain involved during project implementation and monitoring. IV. CONCLUSIONS SSIP was designed to create temporary jobs for vulnerable Armenians. The targeted amount of short-term jobs (823) was reached as a result of other donor (i.e. WFP) involvement in the projects. However, while this donor participation allowed the project to exceed the targeted number of jobs created (920), the methods of payment were quite different – CHF paid with cash while WFP paid with food. Thus if the objective of creation of short term jobs implies payment of short term employees by cash, the project did not reach its objective. Conversely, if the preliminary objective included any type of employment including those paid by food, it did reach its objective. In spite of the significant creation of jobs through SSIP (considered to be the main objective of the program), the results created by the temporary jobs are viewed by mayors to be less important than the proper construction and renovation of small scale infrastructure, which enabled local governments to improve their services to citizens. It is possible that this perspective has been created due to the mayors’ dissatisfaction discrepancy with the payment methods for short-term job recipients. CHF small scale infrastructure projects in Armenian communities were very helpful and solved important community problems, especially for those villages situated in remote areas and distant from regional centers and capital Marzes. Improvements in the infrastructure of several villages were considered to be one of the most important accomplishments of SSIP. This success leads to the achievement of one of the important objectives of the program: improved quality of life for Armenians through the rehabilitaion of small scale community infrastructure. SSIP was also an important step to decrease the extent to which disparities in development exist between capital and regional centers, and remote areas. SSIP is a valuable contribution by USAID and is necessary for the further development of Armenia. In terms of building vital partnerships with other organizations that extended the reach of the assistance to Armenia’s vulnerable communities, it is considered a successful practice of efficient and reasonable use of USAID resources in Armenia. Some projects were not completed in time due to the season and the geographic area, which leads to the suggestion that seasonality be considered during the design of water and gas projects. This includes trench digging (especially by hand) for water lines and soil digging for gas lines, as both are become increasingly challenging when the soil surface is frozen. In other construction SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 22 projects, the provision of works during winter has had a negative effect on the overall quality of construction (tiling, painting, cement works). As in any other development project, sustainability is crucial to ensure the continued benefit and development for the affected communities. This is especially vital for vulnerable communities faced with limited financial resources. The results obtained by the evaluation team indicate mixed sustainability of SSIP projects partially due to the minor roles played by the communities, as well as the community contribution coming from other donors. These factors erode the sense of ownership among communities, and negatively impact the future maintenance of constructed/renovated infrastructure. In some cases, the tendering process for construction work was decided without the participation of community representatives. This damaged the trust towards implementing partners while also diminishing the role of community in project implementation which has a negative impact on future implementation and sustainability. V. RECOMMENDATIONS 1. SSIP implementation provided excellent support to villages, especially in remote areas, to solve their infrastructure problems and improve local government services. It also helped diminish the disparity between rural communities and regional centers and capital. It is recommended to continue SSIP types of work in Armenia on a “demand driven” basis. 2. For the future design short-term job creation programs similar to SSIP, only one method of payment should be provided (either in cash or food) in order to ensure equal social impacts. 3. In order to create more transparent practices and stable supervision over construction works, SSIP should emphasize and seek to rely on local government and local communities with wide citizen involvement for both voluntary and paid works. 4. Construction projects which do not require highly skilled workers should be implemented without tendering and calling on construction firms from outside of the community. Instead, participatory methods are more cost efficient and transparent when applying community development practices. 5. The communities/local governments should be key partners in the design of SSIP-type projects. Specifically, the following should be taken into consideration: a. Contribution (co-financing) from the community should be the principle prerequisite for project funding and a key project selection criterion; b. Community contribution should be provided by the community rather than other donors (either exclusively or mixed) as this is a tool that triggers a strong sense of ownership and promotes sustainability; c. The community should have greater participation in the design and planning processes, the selection of contractors, and project management and monitoring of SSIP-type activities. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 23 6. In order to improve quality and timeliness of future programming, partnerships should ensure the involvement of local communities in the decision-making process as they can provide valuable information regarding seasonal constraints for construction. 7. The findings from gas system development projects indicate that currently only 13% of interviewees are buying natural gas. This leads to the recommendation that future SSIP￾type activities ensure earlier and more thorough communication with gas delivery companies prior to initiating work in order to assess the ability and willingness of local citizens to purchase and use this service. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 24 ANNEX I: EVALUATION SCOPE OF WORK Evaluation of USAID/Armenia Armenia-Turkey Rapprochement (SATR), Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA￾ROLI), and the Small Scale Infrastructure Program (SSIP) Summary: USAID/Armenia requires evaluations of the following activities: Armenia-Turkey Rapprochement (SATR), IFES and NDI electoral and political process Associate Awards under the Consortium for Elections and Political Process Strengthening (CEPPS), Rule of Law Initiative (ABA-RO LI), and the Small Scale Infrastructure Program (SSIP). The purposes of this Task Order are to evaluate the success of these projects in their relevant areas and assess effectiveness of these in achieving set programmatic goals and the USAID/Armenia’s strategic objectives. Two of the five planned evaluations are designed as midterm evaluations (Armenia￾Turkey Rapprochement, and ABA-ROLI), while the other three are designed as end-of-project evaluations (IFES, NDI and SSIP). In the case of mid-term evaluations the findings will be used to inform USAID’s determination whether they are on track achieving their set programmatic goals and targets and whether the initial design of the projects still leads them to the set objectives. They will feed information into the future work plans. In the case of end-of-project evaluations, the findings will be used to inform design and development of future projects. Therefore, the evaluations will identify “lessons learned”; assess strengths and weaknesses of strategies and activities performed under these projects; and provide recommendations to USAID for project planning purposes for the next three to five years. The Contractor will seek to capture effective approaches; analyze the utility of performance monitoring efforts; consider respective outcomes and results; and assess the influence of internal and external changes on the achievement of results. Contractor Responsibilities and Projects: The evaluation should measure and analyze the accomplishments or the progress toward achievement of the results of the activities, including an “effectiveness and efficiency assessment” that looks at how successful the programs have been in achieving their set targets, and how effectively USG resources have been used. Additionally, USAID/Armenia would like to measure the sustainability of the project results on respective beneficiaries where applicable and possible. The Contractor shall review each project’s implementation methodology and to the degree possible, verify the results achieved, the relevance of the project in addressing USAID priorities, and to what extent USAID can be accountable for achieving those objectives. Final recommendations to USAID will help improve program outcomes, weigh sustainability factors, and address program relevance as well as cost efficiency and effectiveness. The evaluations will serve to guide how similar projects, approaches and/or work plans can be improved. The evaluations will also validate (or not) the feasibility of the initial designs of the projects and of their respective development hypotheses. SATR SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 25 This two-year activity is in its first year of implementation. The main objective is to promote improved Armenia-Turkey relations by engaging civil society in the reconciliation processes; establishing and developing business partnerships and regional professional networks; and facilitating government-to government dialogue. The activity is being implemented in a rapidly changing environment of Armenia-Turkey relations. Therefore the appropriateness of the design of the project needs to be explored in addition to the actual project implementation issues. Thus, this evaluation should contribute to the Mission’s understanding of whether or not adjustments are necessary in the approach and in the future planning of this project. The evaluation should aim at obtaining feedback from stakeholders and project partners both from Armenia and Turkey. CEPPS These are two separate Associate Awards under CEPPS III, implemented by IFES and NDI, both of which are in their third year of implementation and are expected to end in FY2012. Both activities are primarily aimed at improving political processes. The IFES activity supports the strengthening the administration of electoral processes and assists the electoral administration to meet international standards for free and fair elections through its support to the Central Election Commission and the Passport and Visa Department of the Police of the Republic of Armenia (OVIR). The NDI activity aims to improve the ability of citizens to effectively participate in political processes as members of political parties and helps political parties develop their ability to take part in parliamentary and presidential elections scheduled for 2012 and in 2013 respectively. The findings and recommendations from this evaluation will be reviewed for results achieved, and effectiveness of program approaches. Best practices and lessons learned will be identified which will contribute to the Mission’s decisions on future election-related activities. ABA-ROLI This is in its third year of implementation and will end in FY2012. The main objectives are to support curriculum reform in the Judicial School; cooperate with the Judicial Department of the Republic of Armenia for enhancing the judicial reforms; and provide greater access to justice through Law School legal clinics with a particular focus on protection of human rights. The purpose of evaluation is to assess the results and outcomes of this rule of law initiative and the sustainability of the achievements of the project to inform future USAID/Armenia decision￾making with regard to similar undertakings. SSIP This is a 27-month project ending in FY2012. The goal of the project is to mitigate the consequences of Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. This project deals with multiple communities and partners in jointly carrying out construction and work with target communities. Executing water projects has been challenging due to different factors, including multiple construction partners, delayed contribution from other donors, weather conditions, etc. However, civil construction has been progressing timely and with very good quality. Project has been modified twice to increase the number of projects (from SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 26 48 to 58), labor days and expand the geographic coverage. The final evaluation will look at few aspects of the program: how the targets have been met vis-à-vis original targets and benchmarks; how the partnership between all parties (IFAD, local NGO Shen, local construction companies, village Mayors, Local Supervisors, etc.) worked; what are mechanisms that would help communities sustain project gains after its completion; did the project leverage contributions or matching funds from other donors or stakeholders as anticipated originally. Evaluation Questions: The Contractor shall review and summarize the implementation and results achieved by all five activities to answer the following evaluation questions and additional questions that may be developed by the Contractor after reviewing the provided materials. For ALL projects: • To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? • Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? • What were the main achievements of the program? • How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? • How did the implementer perform in terms of project management and how effective was the project leadership? • What lessons learned‟ can be provided for future USAID programming in this area? • What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? • Did the agreement provide clear and achievable results against which progress and impact could be measured? • Analyze attribution of project successes to USAID involvement. • Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. Project-specific questions/Tasks: For SATR: • Are the project implementation approaches relevant and feasible in the current state of affairs in the Armenia-Turkey relations? • Which specific aspects of the project are or are not working in the given political situation? [This question should be explored in the Armenian and Turkish contexts separately, because some approaches/activities may be still feasible in Armenia but not in Turkey and vice versa.] • How flexible is the project in terms of adjusting activities to the changing political context? • How has this activity been able to build on the advances made under the previous “Days 2 and 3”project? SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 27 • What signs exist that the project is having impact, anecdotally? Why? What concrete examples of impact (or lack of it) are given? • What is the external perception of the project’s role and impact both in Armenia and Turkey, according to the key stakeholders not involved in the project? • What recommendations can be provided to improve impact? What are the key obstacles and what recommendations can be made to minimize their effect? • How appropriate is the decentralized project implementation approach (four local partners with their Turkish counterpart organizations) to the project objectives? • How effective is the implementing partner as a consortium of local organizations: responsiveness to the donor, responsiveness to project stakeholders, information flow. What recommendations can be made to improve consortium management and operations? For CEPPS: • How can the projects be better designed in the future to measure impact, given the political situation faced in Armenia? NDI • Have NDI youth and women’s Leadership Training Academies, the NDI Women’s Candidate Schools, and the National Conference of Women in Politics successfully resulted in increased social and political involvement (party memberships and activities, political engagements, political public gatherings, etc.) among youth and women? • Have Leadership Training Academy graduates successfully used the new skills that they have developed to: a) implement social and political projects, or b) achieve political and/or policy goals? Identify examples of specific projects and/or political and policy activities. • To what extent have leadership training graduates conducted follow-on trainings for members of their respective political or civil society organizations? • Have NDI/Yerevan Press Club public debates had value for participating political parties? • What are the key obstacles and what recommendations can be made to minimize their effect? Given the obstacles identified, does the approach of the program take these obstacles into account and mitigate them? How successfully? IFES • Was the IFES International Symposium on Election Codes successful in bringing about meaningful compromise and encouraging public debate in the process of amending the RA Electoral Code? • Has the IFES workflow analysis, conducted for the Central Election Commission (CEC), been successful in identifying areas to improve the administration of elections? Has the CEC implemented, or committed to the implementation of administrative reforms resulting from IFES recommendations? SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 28 • Has IFES successfully leveraged US material assistance related to CEC equipment requests with other donors? Has IFES successfully linked such material assistance to electoral administration reforms? • Has IFES created a successful and sustainable partnership between the Police Department of Passports and Visas (OVIR) and the Voter Lists Advisory Committees (VLACs)? To what extent have these partnerships resulted in improvements in completeness and accuracy of the OVIR voter rolls? For ABA-ROLI: • What are the most notable accomplishments of the project in the ROL area since the inception of the current agreement in 2009? • What are the factors hindering judicial independence in Armenia and what has the project done to strengthen the judiciary? • Has the project been able to increase the capacity of the Armenian judiciary to play a role in making the government more accountable? • What did the project do to ensure proper implementation and enforcement of new or existing laws? • What are the activities aimed to establish mechanisms for oversight of court proceedings o How do these mechanisms affect judicial performance? o Are the existing oversight mechanisms sufficient to activity build on the existing efforts to promote accountability and transparency in the justice sector? o If yes, please name, if not, please mention gaps. • Is there capacity or interest within the legal professionals to promote reform in ROL? If yes, what does the project do to support that interest? • How does the project support the country’s only Bar Association in helping to play a balancing role within the justice sector? • What has the project accomplished in regard to defending human rights through legal protection? • How did the project affect legal education in law schools? • Where is the most viable stakeholder support for rule of law reform likely to be found? • What targeted activities could be proposed to address Armenia specific deficiencies in the justice sector? • How successfully does the project coordinate with the international community to promote human rights? • What recommendations can be made for a more effective, integrated project design? For SSIP: • What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)? • What are social, economic and other impacts of the project on target communities and beneficiaries? • How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 29 • How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? • How did the partnerships impact the project (timeliness, quality, and cost-benefit)? • In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? • How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.? • Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented? • What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies? • If a similar activity is considered in the future, what should be changed in the design and technical approach? USAID’S Role in the Evaluation The USAID Mission in Armenia will: • organize a small USAID advisory group to support the Contractor in the implementation of this scope of work; • provide relevant programmatic and budgetary information to the Contractor (some relevant portions of contracts and assistance agreements are attached); • provide project documents and evaluations to the Contractor; • facilitate obtaining USAID/Mission input; and • arrange USAID/Armenia meetings. In some instances (although the Contractor should not depend on this), an additional USAID staff person may join the Contractor during the field visits/stakeholder interviews in Armenia. USAID Mission staff and/or the USAID team members will be available to assist the Contractor in providing in-depth knowledge of the various projects and activities that are being evaluated. Methodology The Contractor will: 1. Conduct a comprehensive review of performance reports and other materials and identify data gaps. 2. Develop additional research and evaluation questions as needed based on the development hypothesis and on the above-mentioned evaluation questions; identify informants and stakeholders, samples and/or other relevant data sources. 3. Develop data collection tools based on the best possible methodology in accordance with the evaluation questions and feasibility considerations and provide to USAID prior to commencing field work. 4. Prepare a field work plan. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 30 5. Conduct field research in Armenia. 6. Analyze data and compile key findings, conclusions and recommendations. 7. Revise the draft reports as requested by USAID and submit final reports to USAID/Armenia for acceptance. The proposed methodology should address the need for data collection from qualitative and quantitative sources; and provide the best possible combination of methods, given the evaluation questions and the available resources and timeline. There is no preference for any particular method. The ability of particular method(s) to properly answer the evaluation questions is important. To the extent possible, data should come from facts, rather than be based on anecdotal evidence, and conclusions should be based on findings received from multiple sources. Clear, standardized data collection methodology should be described in detail to ensure reliability and consistency of the evaluation findings. Deliverables The Contractor’s deliverables shall include: 1. A written methodology plan (research design and operational work plan). 2. Provide a verbal debriefing at the end of the field work to Mission management and technical teams. 3. Prepare draft evaluation reports (electronic and hard copy) which will analyze data and summarize key findings, conclusions and recommendations. The Evaluation Report shall at a minimum contain1) an Executive Summary; 2) a brief description of the project; 3) a section on the purpose and the methodology of the evaluation; 4) a section on clearly defined findings, conclusions and action oriented recommendations. This section should be organized around the evaluation questions defined for each project. 5) Annexes, including the Scope of Work, all evaluation tools, all sources of information. Submit these to USAID/Armenia within three weeks after completing the fieldwork. USAID will be responsible for compiling Mission comments for inclusion and submission to the Contractor. USAID/Armenia will provide the Contractor with a summary of such written comments within three weeks of having received the draft reports. 4. The Contractor shall submit final reports to USAID/Armenia within two weeks after USAID's comments are provided. The final reports will meet the following quality standards: a) The reports will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; b) The reports shall address all evaluation questions included in the scope of work; c) The report shall include the scope of work as an annex; d) Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; e) Evaluation findings will assess outcomes and impact on males and females; f)Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology; g) Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, or the compilation of people’s opinions; h) Sources of information need to be properly identified and listed in an annex; i) Recommendations need to be supported by a specific set of findings; j) SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 31 Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 5. The Contractor will submit Evaluation data to USAID/Armenia along with the final reports for warehousing and future use by the Mission. The data will be in easily accessible format, such as MS Word documents for qualitative data, and SPSS or Excel files for quantitative data. Evaluations Timeline 1. Evaluations should be initiated around September-October 2011. The preliminary findings of the evaluations should be submitted to USAID/Armenia immediately after the completion of the field work. 2. The first drafts of the five evaluation reports shall be submitted to USAID/Armenia no later than three weeks following the completion of the field work. (For CEPPS USAID expects to have two separate reports – one on the IFES activity and one on the NDI activity). 3. The final Evaluation Reports shall be submitted no later than two weeks after final comments on the draft evaluation reports are submitted by USAID/Armenia. 4. Once finalized, the contractor is responsible for ensuring that the final approved reports are also submitted to USAID’s Development Experience Clearinghouse within three months of the completion of the reports. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 32 ANNEX II: EVALUATION WORK PLAN Project Purpose SSIP was designed to mitigate the consequences of the Global Economic Crisis through the creation of temporary employment opportunities in vulnerable rural communities by means of implementation of small scale infrastructure projects prioritized by communities such as rehabilitation/renovation of kindergartens, pre-schools, community centers, sport halls and drinking water supply systems. Evaluation Purpose The objective of the field work is to collect necessary data and information for properly evaluating the effectiveness, efficiency, and sustainability of SSIP implemented by CHF International. The evaluation team, Dr. Vahé Heboyan and Dr. Mayis Vanoyan, will (1) hold meetings with key stakeholders and implementing partners to gain inside knowledge of project implementation, challenges and lessons learned, (2) conduct an extensive data collection4 F through representative sample of all potential beneficiaries (short-term employees, long-term employees, communities, and contractors, and (3) interviews with CHF and USAID/Armenia personnel. Work formats The field work will be conducted by Dr. Heboyan and Dr. Vanoyan with assistance from the Civil Development and Partnership Foundation (CDPF) based in Yerevan, Armenia. CDPF’s role is to assist in key beneficiary surveys. CDPF’s team will work alongside the SSIP Evaluation Team (SSIP-ET) during its field visit and will assist in the data collection and database storage efforts. This format allows for (a) constant supervision of the CDPF-team by the SSIP-ET, (b) guidance during the survey process, and (c) logistical support on the ground by the SSIP-ET. Field-work dates The fieldwork will commence on Nov 21, 2011 and end on Dec 8, 2011. A detailed preliminary daily activity schedule is provided in Annex VII, and is subject to confirmation with third parties on meetings. Desk Review in Armenia The SSIP-ET will start its activities by studying the documents that contain information on the job creation component regarding short-term employment during SSIP implementation. This information is vital to be able to evaluate CHF International’s achievement of the major goals5 F towards job creation set forth in the USAID SSIP project (see page 6, Attachment B of AID-111- A-09-00005 provided by USAID Armenia)F 6 F. 4Assisted by CDPF. 5 (a) 48 infrastructure projects completed, benefiting 37,900 Armenians; (b) 47,175 persons days of employment generated; (c) 823 vulnerable Armenians obtaining short-term employment; and (d) 142 long-term jobs created. 6 This needs to be completed in Armenia, since SSIP-ET was not able to obtain such information from CHF. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 33 Key Informant Interviews Key informant interviews will be used to answer questions about the process of the program, the CHF/SHEN interface; the CHF USAID coordination; the Contracting process; the interaction with the mayors and managers of the local utility services and so forth. Annex I is an initial list of key informants. The relevant interview guides in Annex II will be used for these interviews SSIP Field Work Surveys During the field work, four different surveys will be carried out. They are designed to obtain comprehensive information and data in the effectiveness, efficiency, and sustainability of the SSIP. Survey categories and brief descriptions are provided below. The draft survey instruments are in Annex II and subject to further development and translation. A. Short-Term Employment This survey is designed to assess the impact of SSIP on local vulnerable households and communities through short-term employment opportunities. The survey will cover a representative sample of people employed by the 58 SSIP projects. The sample will be done by region and project type.7 B. Long-Term Employment This survey will assess the impact of SSIP on creating long-term employment opportunities in local communities and show the impact on the livelihoods of their households and sustainability of project results in local communities. C. Communities This survey will assess the overall impact of SSIP outcomes on the local communities. For example, we will explore the cost savings associated with an access to natural gas and water systems versus the prior alternatives. In addition to monetary benefit, we will assess the implications of SSIP projects on the overall quality of life of the end￾beneficiaries, such as the impact of cleaner fuel (i.e. natural gas) on the environment and community health. Additionally, the survey will ask what the potential implications of the access to cleaner water for the health of local communities are. D. Contractors The fourth survey will focus on the impact of SSIP on businesses involved in the project. It aims at documenting the benefits to contractor firms (income, jobs) as well as the challenges and issues faced by contractors during the implementation process. Complementary to the surveys, groups and individuals who have benefited from SSIP will be interviewed to ascertain the results of the infrastructure projects in terms of the services and utility to the residents. 7 It should be noted that the SSIP-ET has not been able to obtain data from CHF on the number of employees for each project which is vital for most accurate sample selection. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 34 Sampling A purposeful sample of the communities will be drawn so that the types of projects (water, electricity, gas, schoolrooms, sports and public place infrastructure) will be covered (See tables below). In each community the key informants such as mayors, managers of the utilities, school teachers or directors, managers of the computer or sports centers will be chosen and not sampled. The contractors will be sampled so that they cover the range of types of infrastructure. In the sampled communities a sample of people who gained short-term employment will be drawn from the contractor lists to ascertain the importance of the employment to them. Permanent employees, who resulted from expanded infrastructure, will be interviewed. In this draft the numbers in the following tables are tentative and will depend upon the initial desk review work and will dependent upon weather conditions for travel. Nevertheless, these are the goals for the sample. No. Task/Survey Actual Sample Size 1 Desk Review 58 projects 58 (not actual sample size) 2 Short-term employment 823 people (target) 200-250 3 Long-term employment 142 people (target) Approx. 70 4 Communitie s 58 25-32 5 Contractors 26 26 6 SSIP outcome beneficiaries TBD 250-300 Table 2: Sampling by Types of Projects and Locations Types of projects Total number of projects Sample size Number of projects in Marzes Samples pro rata to number of projects in Marzes Water projects 40% 23 8 Gegharkunik3, Shirak4, Lori 3,Tavush 7, Kotayk1, Syunik3, Aragatsotn2 Gegharkunik1, Shirak1, Lori 1Tavush2, Kotayk1, Syunik1, Aragatsotn1 Gas projects 10% 6 2 Lori 1, Shirak3, Aragatsotn2 Shirak1, Aragatsotn1 Preschool renovation 12% 7 2 Lori 1, Aragatsotn4, Tavush1, Gegharkunik1 Aragatsotn1, Gegharkunik1 Other projects (38%) including: Ventilation, Lighting, Computer classes, 22 8 Ventilation 1 (Shirak), Lighting 4 (Tavush3, Pampak1), Computer classes 3 Ventilation1 (Akhurian/Shirak) Lighting1 (Teghut/Tavush) Computerclasses1 (Aragatsotn) CommunityCenters2 SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 35 Types of projects Total number of projects Sample size Number of projects in Marzes Samples pro rata to number of projects in Marzes Community Centers, Sport halls, Bridge (Aragatsotn2, Gegharkunik1), Community centers 7(Gegharkunik3, Tavush1, Shirak1, Aragatsotn1, Lori 1), Sport halls6 (Syunik2, Tavush1, Shirak1, VayotsDzor1, Armavir1), Bridge 1 (Gegharkunik), (Avazan/Gegharkunik1, Shahumian/Lori 1) Sporthall2, (Myasnikyan/Armavir1, Malishka/VayotsDzor1) Bridge 1 (Ttujur/Gegharkunik1) Table 3: Excerptions by Marzes and Types of Projects in Marzes Marzes Number of implemented projects Number of projects to be evaluated by marzes Breakdown by types of projects Water Gas Preschool Other projects/note Gegharkunik 12 4 1 1 1 (Community Centre) 1(Bridge) Shirak 11 3 1 1 1 (Ventilation) Lori 6 2 1 1 (Community center) Tavush 11 3 2 1 (Lighting) Syunik 5 1 1 Aragatsotn 11 4 1 1 1 1(Computer classes) Armavir 1 1 1 (Preschool) Kotayk 1 1 1 Vayotsdzor 1 1 1 (Sport hall) SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 36 Reporting The format of the evaluation report will follow the standard format for USAID evaluation reports8 and will not exceed 25 pages, excluding annexes. Deliverables and Key Tasks Related to Their Delivery No. Deliverable Deadline Responsible Party 1. USAID initial briefing Nov 21, 2011 Vahé and Mayis 2. USAID out-briefing Dec 6, 2011 Vahé and Mayis 3. Draft report to IBTCI Dec 19, 2011 Vahé and Mayis 4. Draft Report to USAID Dec. 30 IBTCI 5. Comments from USAID Jan 20, 2012 USAID 6. Final Report to IBTCI Feb 2, 2012 Vahé and Mayis 7. Final Report to USAID Feb 3, 2012 IBTCI Minimum Set of Questions to be Answered (from the RFP) 1. To what extent has the project been successful in achieving its expected results? If not, or in some particular areas, why? 2. Are the processes, innovations, institutions, partnerships, linkages introduced sustainable? 3. What were the main achievements of the program? 4. How relevant was the intervention? How well designed or developed was the theory of change/development hypothesis? 5. How did the implementer perform in terms of project management and how effective was the project leadership? 6. What “lessons learned can be provided for future USAID programming in this area? 7. What strategies should be promoted and/or abandoned to more cost-efficiently or effectively achieve objectives and measure impact? 8. Did the agreement provide clear and achievable results against which progress and impact could be measured? 9. Analyze attribution of project successes to USAID involvement. 8 IBTCI evaluation teams refer to USAID’s “EVALUATION Learning from Experience,” Bureau for Policy, Planning, and Learning and “Checklist for Assessing USAID Evaluation Reports” for guidance. SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 37 10. Analyze and evaluate the relative effectiveness of alternative activities, approaches and strategies for future programming. 11. What is the ratio of “planned” or projected and “actual” jobs generated throughout the project? How do you explain the gaps (if any)? 12. What are social, economic and other impacts of the project on target communities and beneficiaries? 13. How do you think the completed projects will be maintained by the communities? What are the grounds for their sustainability? 14. How did the partnership work within the project between different partners (IFAD, Shen, local contractors, communities, etc.)? What worked, what did not, why? 15. How did the partnerships impact the project (timeliness, quality, and cost-benefit)? 16. In terms of jobs generated, how did it affect the rural population? Was it tangible enough for them (days, income generated, etc.)? What is their perception on this? Were their expectations met? 17. How does success in two main areas that the project tackled - water and civil construction compare in terms of progress made, achievements, implementation challenges, etc.? 18. Has/will CHF meet its cost-sharing commitment? Are cost-sharing valuations reasonable, consistent and adequately documented? 19. What are the lessons learned? What are the strengths and weaknesses of this project, its approaches or strategies? 20. If a similar activity is considered in the future, what should be changed in the design and technical approach? SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 38 ANNEX III: INFORMANTS FOR FIELD INTERVIEWS USAID/Armenia COTR: Mariam Gevorgyan COTR/SSIP: Haikanush Bagratunyan (Yerevan/EGO) Project Implementers CHF Deputy COP Contractors (Owner/Manager) Community Mayor Utility Managers Teachers & School Directors Sport or Computer Center managers SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 39 ANNEX IV: ILLUSTRATIVE INTERVIEW GUIDES A. FOR ALL 58 PROJECTS (Desk review of CHF and contractor files) 1) List of all short-term employees and their information a. Name, contact phone number, gender, age b. Days worked and labor category c. Wage rate 2) Projected and Actual a. Start and end dates b. Budgets c. Cost sharing d. Long-term employment 3) Budget breakdown (% share) a. Labor b. Materials c. other Schools and other facilities 1) Size of facility renovated or constructed 2) Is it currently in use (operational)? If not why? 3) If existed before: a. What was the condition? b. How many were using it? 4) If new, was there an alternative before? If yes, describe condition and how many were using. What were the issues or problems with the alternative? 5) What was/is approximately monthly maintenance cost? a. Who was paying for it before? b. Who was paying for it now? c. What is the source of funding and how secure is it? d. Is there alternative source of funding? 6) Number of direct beneficiaries or potential users. 7) List primary and alternatives uses of the facility. 8) Did this make it possible/necessary to hire teachers, admin, janitors, cooks, etc? B. NATURAL GAS PROJECTS 1) Number of households (or people) benefiting in each project community SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 40 2) Have engineers, maintenance, etc. positions resulted for full time or part time jobs? 3) Status of project, i.e. are households currently using the natural gas for cooking, heating, bathing and other purposes? For how long? 4) Energy expenditures of the household: a. The month prior to having the natural gas connected to their appliances. b. The month after using natural gas. c. Note: this is total expenditure on energy for the household. Also need to account for seasonality during those 2 monthly energy bills. 5) Main source of energy for (wood, coal, electricity, propane, diesel, dung-cake {atar}, etc.): a. Cooking b. Heating c. Bathing d. Other purpose 6) How much was spent on each of these sources? 7) How their life improves due to having an access to the natural gas? 8) What were issues/problems/challenges associated with the previous fuel types? a. Health b. Comfort c. Cost d. Time e. Cleanness f. Other 9) How satisfied were you of the quality of the following items from using non-natural gas energy source? a. Cooking b. Heating c. Bathing d. Other 10) How satisfied were you of the quality of the following items from using natural gas energy source? Or, for just finished projects, how do you think an access to natural gas as a source of energy will change the quality of the following things? a. Cooking b. Heating SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 41 c. Bathing d. Other C. WATER PROJECTS 1) Number of households (or people) benefiting in each project community 2) Have engineers, maintenance, etc. positions resulted for full time or part time jobs? 3) Is this a new water pipeline/water tower for the community or renovation of an existing one? a. New Q4 b. Existing Q5 4) Status of project. i.e. Is water currently flowing? a. At each faucet at home? b. At a single faucet inside or outside house? c. For how long? 5) What was the condition of the old-existing system and what were problems associated with it? a. Water quality (color, smell, foreign matter) b. Frequency of water delivery system malfunction or breakdown 6) If NEW water system, how were you accessing water previously? a. Local community water source b. Purchasing (list price) 7) How their life improves due to having an access to new and/or cleaner water source? 8) What were issues/problems/challenges associated with the system put in place through SSIP? 9) Main source of water for: a. Cooking/Drinking b. Dishes c. Bathing d. Other purpose 10) How satisfied are you of the current quality of the following items from using updated/new water source? a. Cooking b. Heating c. Bathing d. Other SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 42 11) How satisfied were you of the previous quality of the following items from using water source? a. Cooking b. Heating c. Bathing d. Other D. SCHOOLS AND OTHER FACILITY RENOVATIONS 1) Were you or your household member using this facility before? a. How often and how many people from your household? 2) Are you or your household member using this facility now? a. How often and how many people from your household? 3) How satisfied were you of the quality and services of the facility before renovation? a. Overall quality b. Safety c. Cleanness and healthiness d. Usability 4) How satisfied were you of the quality and services of the facility after renovation? a. Overall quality b. Safety c. Cleanness and healthiness d. Usability 5) If you or your household were asked to pay a monthly nominal user-fee to use the facility, how much will you be willing to contribute per person using it? a. 0 drams b. 100 c. 200 d. 500 e. 1000 f. More than 1000 E. DEMOGRAPHIC INFORMATION FROM ALL RESPONDENTS 1) Age 2) SMarital status SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 43 a. Married b. Divorced c. Widowed d. No married 3) Household size and composition a. Children under 10 b. Children 11-18 c. Adults 19-35 d. Adults 35-60 e. Adults above 60 4) Employment status a. Unemployed b. Government c. Self d. Private company e. NGO/foreign government f. Student g. Retired h. Other 5) Highest level of education completed a. No education b. Some school c. High-school diploma d. University degree (diploma or bachelor’s) e. Post-graduate (MS, PhD, Kandidat Nauk, Doctor Nauk) f. Currently in degree program (specify: ______________ ) 6) Income for the month before you started to work in SSIP project. (applied to short-term employment survey only) a. Self-income b. Total household income SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 44 ANNEX V: LONG-TERM EMPLOYMENT SURVEY QUESTIONNAIRE SURVEY QUESTIONNAIRE Long-term employees 1. What is your current occupation? _______________________________________________________ 2. Is this full time or part time job? FULL PART 3. How many hours in average you work in a day? ________ hours 4. How long have you been working in this position? ________ months 5. What was your occupation before this job? ____________________________________ a) Unemployed b) Migrant work ֜see sub-question. c) Other (specify): _______________________________________________________ if (b), why did you chose to work in this project instead of going abroad? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. How much are you currently getting paid? ____________ drams per ___________ 7. Are you paid on time? a) Yes always b) Most of time c) No if (b) and (c), explain. ________________________________________________________________ ________________________________________________________________ 8. How satisfied are you with the amount of salary? a) Very satisfied b) Satisfied c) Neutral d) Unsatisfied e) Very unsatisfied f) … declined to answer 9. What is the significance of the financial benefit from this job for your household? a) Very significant SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 45 b) Significant c) Not significant 10. Do you feel that your compensation is fair compared to others for the same type of work? YES NO NOT SURE Comments: _________________________________________________________________ _________________________________________________________________ _________________________________________________________________ DEMOGRAPHICS 11. Age ________ 12. Marital status a) Married b) Divorced c) Widowed d) Not married 13. Household size and composition a) Total size (including yourself) ________ b) Children under 18 years of age ________ c) Students ________ 14. How many in your household currently have jobs? FULL time ___________ PART time ___________ MIGRANT ___________ 15. Highest level of education completed. a) No education b) Some school c) School diploma d) University degree (diploma or bachelor’s) e) Post-graduate (MS, PhD, Kandidat Nauk, Doctor Nauk) f) Currently in degree program (specify: _____________________________ ) 16. What was your TOTAL household income during the month before you started to work in this job? ________________ drams. OR use range below a) 0 – 50000 drams b) 50001 – 100000 c) 100001 – 150000 d) 150001 – 200000 e) 200001 – 250000 f) 250001 – more SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 46 17. Do you have any comments? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you for your cooperation! SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 47 ANNEX VI: SHORT-TERM EMPLOYEE SURVEY QUESTIONNAIRE SURVEY QUESTIONNAIRE Short-term employees 1. What type of work have you been involved during project implementation? ____________________________________________________________ 2. Did you previously work for this company? Yes No 3. How many days have you worked in this project? ________ days 4. How many hours in average you worked in a day? ________ hours 5. How were you getting paid? a) Hourly b) Daily c) Weekly d) Monthly e) Lump sum 6. How much were you getting paid for your work? ____________ drams per ___________ 7. How satisfied were you with the amount of salary? g) Very satisfied h) Satisfied i) Neutral j) Unsatisfied k) Very unsatisfied …declined to answer 8. What was the significance of financial benefit from this project for your household? d) Very significant e) Significant f) Not significant 9. Do you feel that your compensation was fair compared to others for the same type of work? YES NO NOT SURE Comments: ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 48 10. Were you paid on time? d) Yes always e) Most of time f) No if (b) and (c), explain. ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 11. Where the supplies and materials delivered on time? a) Yes always b) Most of time c) No 12. In your opinion, what was done well in this project? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 13. In your opinion, what can be changed to improve this project? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 14. What was the selection of the employees based on? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ 15. Did this work enable you to acquire or improve technical or professional skills? Yes No Explain: ___________________________________________________________________________ 16. After this work, were you able to find similar work? Yes No if Yes, was it with the same company? Yes No 17. Do you currently work (any)? Yes No SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 49 Demographics 18. Age ________ 19. Marital status e) Married f) Divorced g) Widowed h) Not married 20. Household size and composition d) Total size (including yourself) ___________ e) Children under 18 years of age ___________ f) Students ___________ 21. Highest level of education completed g) No education h) Some school i) High-school diploma j) University degree (diploma or bachelor’s) k) Post-graduate (MS, PhD, Kandidat Nauk, Doctor Nauk) l) Currently in degree program (specify: _____________________________ ) 22. Before this work, what were you doing? d) Unemployed e) Migrant work ֜see sub-question. f) Other: _______________________________________________________ if (b), why did you chose to work in this project instead of going abroad? ________________________________________________________________________ ________________________________________________________________________ 23. How many in your household currently have jobs? FULL time ___________ PART time ___________ MIGRANT ___________ 24. What was your TOTAL household income during the month before you started to work in this project? ________________ drams OR use range below a) 0 – 50000 drams b) 50001 – 100000 c) 100001 – 150000 d) 150001 – 200000 e) 200001 – 250000 SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 50 f) 250001 – more 25. Do you have any comments? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Thank you for your cooperation! SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 51 ANNEX VII: DRAFT WORK PLAN CHART Date Vahé Heboyan Mayis Vanoyan CDPF Week 1 (November 21 to 26 Mon, Nov 21 Evaluation Team Planning Meeting USAID briefing Evaluation Team Planning Meeting USAID briefing Tues, Nov 22 Meeting with CDPF Meeting with CHF, Shen Final evaluation preparations Meeting with CDPF Meeting with CHF Final evaluation preparations Meeting with IBTCI evaluation team Wed, Nov 23 Field work – Dilijan Field work – Dilijan Field work Thurs, Nov 24 Field work Field work Field work and data entry Fri, Nov 25 Field work Field work Field work and data entry Sat, Nov 26 Field work Field work Field work and data entry Week 2 (November 28 to December 3) Mon, Nov 28 Field work Field work Field work and data entry Tues, Nov 29 Field work Field work Field work and data entry Wed, Nov 30 Field work Field work Field work and data entry Thurs, Dec 1 Field work Field work Field work and data entry Fri, Dec 2 Field work Field work Field work and data entry Sat, Dec 3 Field work Field work Field work and data entry Week 3 (December 5 to 8) Mon, Dec 5 Prepare for USAID out￾briefing Work on draft report Prepare for USAID out￾briefing Work on draft report Field work and data entry Tues, Dec 6 USAID briefing Work on draft report USAID briefing Work on draft report Field work and data entry Wed, Dec 7 Work on draft report Work on draft report Data entry Thurs, Dec 8 Depart Yerevan Data entry SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 52 ANNEX VIII: CONFLICT OF INTEREST STATEMENTS SSIP Evaluation Report International Business & Technical Consultants, Inc. Page 53