1 Performance Evaluation of the Palestinian Community Infrastructure Development (PCID) Program FINAL REPORT 29 JUNE 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by QED, LLC Group. Performance Evaluation of the Palestinian Community Infrastructure Development (PCID) Program FINAL REPORT Prepared for the United States Agency for International Development, USAID Contract Number AID-294-C-17-00005 Implemented by The QED Group, LLC. 1820 N. Ft. Myer Drive Suite 700 Arlington, VA 22209 +1 703.678.4700 1 TABLE OF CONTENTS ACRONYMS.................................................................................... 2 I. PROGRAM CONTEXT AND PURPOSE ...................................... 12 II. EVALUATION PURPOSE AND OVERVIEW............................. 13 III. EVALUATION METHODOLOGY............................................. 15 Data Sources...........................................................................................................................................15 Data Analysis...........................................................................................................................................17 Limitations...............................................................................................................................................17 IV. EVALUATION FINDINGS AND CONCLUSIONS ..................... 18 EVALUATION QUESTION 1 ................................................................................................................18 EVALUATION QUESTION 2: ...............................................................................................................28 EVALUATION QUESTION 3 ................................................................................................................34 EVALUATION QUESTION 4: ...............................................................................................................37 V. EVALUATION CONCLUSION ................................................ 38 VI. LESSONS LEARNED AND RECOMMENDATIONS ................... 39 VII. LIST OF ANNEXES .............................................................. 42 ANNEX A: SCOPE OF WORK.............................................................................................................43 ANNEX B: DETAILED DESCRIPTION OF SURVEY METHODOLOGY.............................................58 ANNEX C: DATA COLLECTION TOOLS .........................................................................................67 ANNEX D: LIST OF DOCUMENTS REVIEWED................................................................................110 ANNEX E: LIST OF STAKEHOLDERS INTERVIEWED.....................................................................111 ANNEX F: EVALUATION MATRICES ................................................................................................114 ANNEX G: SELECTED BACKGROUND DOCUMENTS ..................................................................124 ANNEX H: ADDITIONAL SURVEY TABLES......................................................................................162 ANNEX I: STATEMENTS OF DIFFERENCES.....................................................................................169 2 ACRONYMS ANERA American Near East Refugee Aid CA Cooperative Agreement CMWU Coastal Municipalities Water Utility COP Chief of Party DCL Israeli District Civil Liaison Office DCOP Deputy Chief of Party EGI Economic Growth and Infrastructure Office (USAID/WBG) EQ Evaluation Question GRM Gaza Reconstruction Mechanism HIMII High Impact Micro-Infrastructure Initiative IR Intermediate Result KII Key Informant Interview MOEHE Ministry of Education and Higher Education MOU Memorandum of Understanding PA Palestinian Authority PCID Palestinian Community Infrastructure Development PCPO Palestinian Center for Public Opinion PMEP Performance Monitoring and Evaluation Project PWA Palestinian Water Authority RFA Request for Applications TAM Women Media and Development UN United Nations UNRWA United Nations Relief and Work Agency USAID/WBG US Agency for International Development Mission for West Bank and Gaza WASH Water, Sanitation, and Hygiene (USAID program) YDRC Youth Development Resource Center 3 EXECUTIVE SUMMARY The Palestinian Community Infrastructure Development (PCID) program is supported by the USAID Mission to the West Bank and Gaza (USAID/WBG) to increase access to water and sanitation services, and to other community services, through small and medium infrastructure projects in villages and other vulnerable and under-served communities throughout the West Bank and Gaza Strip. By responding to community needs for fundamental services, the program should contribute to social and economic development in the communities and improve the quality of life for the residents. The purpose of the evaluation, conducted as a standard practice during the final year of a large program like PCID, is to assess performance, highlight successes and challenges of the programmatic approach and implementation, and inform future program directions and designs. The external evaluation was done by the QED Group LLC as part of their support to the USAID/WBG Mission through the Performance Monitoring and Evaluation Project (PMEP). A team of four independent evaluators conducted the fieldwork in February and March 2018. The team was supported by the Palestinian Center for Public Opinion (PCPO) in data collection and analysis. The evaluation was organized around four evaluation questions (EQs), defined by USAID. OVERALL CONCLUSION The overall picture emerging from the evaluation is that the PCID program has been effective in addressing basic needs for infrastructure in Palestinian communities, channeling resources to small, vulnerable, and under-served communities and reinforcing the local advocacy and outreach efforts. The model also has been effective in engaging the community and in building a sense of achievement and progress. The projects are valued by the beneficiaries and have had a positive effect on their lives. The findings clearly suggest that residents see community projects as a focal point to improve their daily lives At the same time, given the unique circumstances in the Palestinian territories and the absence of baseline data, realistic indicators, and comparable activities, the evaluation has no basis on which to judge the objective, relative validity of this model of community engagement or its broader utility. EVALUATION METHODOLOGY The evaluation employed a mixed methods approach that drew on multiple sources of data to provide a balanced understanding of the activity from various points of view, utilizing, both quantitative and qualitative measures of results and process, and seeking inputs from a broad spectrum of stakeholders. EVALUATION QUESTIONS FOR PCID FINAL PERFORMANCE EVALUATION EQ 1: To what extent did the implementation of PCID activities (infrastructure projects) support the project development hypothesis and intended results? EQ 2: To what extent did the project succeed in identifying and addressing community infrastructure needs in various sectors? EQ 3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? EQ4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? 4 The data sources for the PCID evaluation included: • Document Review. PCID program documentation provided by USAID and ANERA include the quarterly and annual reports, annual workplans, annual Performance Monitoring Plans, and the GEO-MIS program database. • Key Informant Interviews in stakeholder organizations, including ANERA, TAM, ANERA partner engineering firms, USAID, and the principal government counterpart organizations, PWA (the Palestinian Water Authority, MOEHE (Ministry of Education and Higher Education, and, in Gaza, CMWU (Coastal Municipalities Water Authority). • Field Visits to Beneficiary Communities. Data was collected in a sample of 18 communities where PCID projects have been completed (14 in the West Bank and 4 in Gaza) through key informant interviews with community leaders, group discussions with beneficiaries, and observations and inspections of the project sites. • Survey of Beneficiary Households. A random sample of approximately 30 households in each of the sample communities was selected to respond to a brief fixed-response questionnaire through face-to-face interviews, yielding a total of 619 respondents across the targeted communities in the West Bank and Gaza. The analysis was an iterative process built around the evaluation questions, whereby evidence was drawn from the various sources to articulate and corroborate findings. The findings were used to establish conclusions, which in turn served as the basis for extraction of recommendations. The key limitations of the methodology were (1) without baseline measures or a comparison group, there is no basis on which to measure degree of change or cause, and the diversity of projects and communities limits the reliability of generalizations from the sample of communities; (2) the necessity of having separate teams collecting data in Gaza and in the West Bank presents a potential for a systematic bias in collection or analysis that could have distorted comparisons across the two areas. FINDINGS AND CONCLUSIONS Evaluation Question 1: To what extent did the implementation of PCID activities support the project development hypothesis and intended results? The PCID results framework has two Intermediate Results (IRs), focused on increased access to water and sanitation services, and increased access to community infrastructure facilities. In quantitative terms, both results are measured by the number of infrastructure projects completed. In the case of water, increased access should result in increased availability and affordability; for the other community services, the infrastructure should contribute to increased access to community services like health and education. In addition, the program should enhance the services provided by other USAID offices, focus particularly on vulnerable and under-served communities, and provide a rapid response for unforeseen infrastructure and other emergency needs. In terms of program outputs, PCID produced quantifiable results in all the results and objective categories. On the other hand, when the results are measured against the program indicators and the targets for achievement, the achievements are almost uniformly deficient. The consistency in this finding may reflect unrealistic indicators and targets, or inadequate performance by the implementers. Most of the key informant interviews focused on the former. The evaluation also examined the program achievements and results in qualitative terms, utilizing the findings from the interviews with community leaders, the group discussions with beneficiaries, and the 5 household survey in the communities visited. How do the beneficiary communities rate the results of the projects in terms of increased access to services and the effect on the quality of their lives (i.e., the development hypothesis)? In these terms, the findings are generally positive. • For the water/sanitation projects included in the sample, community leaders said the project had resolved a critical issue in the community. They cited positive changes in their communities as a result, including increased access to water, reduction in cost, and improved quality of life. Among survey respondents in these communities, 92% and 96% respectively, said the projects were important to their families and to their communities. • For IR2, increased access to community infrastructure facilities and services, the responses of community leaders and beneficiaries (in the household survey) showed a similar affirmative pattern. Across all communities in the sample, the interviews with community/NGO leaders and the household survey showed a consistent positive assessment of the projects in terms of responsiveness to needs, and resolution of a critical issue. Both the leaders and the beneficiaries agreed that the services were being utilized. Overall 84% of the survey respondents rated the project as important to the community. In addition to the effectiveness of the results in terms of increased access to community services and improved quality of life, the evaluation also examined the results from the point of view of technical quality and of gender sensitivity/social inclusion. Based on observations of the project sites and discussions with community leaders, in general the technical quality of construction is good. In all of the sampled communities except two, the physical structures were considered to be “as planned,” and, observers judged the projects to be in good shape. At the same time in the majority of projects, community leaders and discussion group participants cited specific, relatively minor, shortcomings in the design or, in particular, in the work of the contractors. The concerns about design are complex and difficult to assess as external evaluators. The issues raised about the contractors and the construction process were more widespread, among community leaders, design engineers, and the ANERA field engineers. The quality of the work, inappropriate and unauthorized changes in design or materials, and the use of inexperienced and unqualified workers were raised as issues, also pointing to continuing concerns with oversight and quality control. 1Water Reservoir and installation of main water lines, Attil, West Bank 6 Finally, an important factor in the PCID program, which is not usually associated with infrastructure activities, is attention to gender sensitivity and social inclusion. The staffing of the program and the outreach model to engage women and other marginalized groups in the project process demonstrated an active commitment to social inclusion. TAM designed and implemented the gender inclusion aspect of the program through a four-year subgrant (July 2013 – June 2017). TAM gender outreach staff worked as a part of the ANERA team in the PCID field offices (except Gaza) and in Jerusalem and contributed to all PCID projects during the sub-grant period. In this capacity, they reviewed the project designs to identify and current issues for women/girls and other marginal groups, and worked with the village councils to encourage the participation of women, persons with disabilities, youth and elderly in community project meetings, and in all phases of the implementation process. In qualitative terms, the assessments by men and women of community problems and of the value of the PCID projects were essentially the same. EQ 1 Conclusions • PCID completed small and medium scale infrastructure projects in support of all intermediate results and objectives. • PCID did not achieve the expected quantitative results as measured by the performance indicators. The gap between the actual and targeted results may reflect unrealistic targets, contingencies in the operating environment, or PCID underachievement. As a result, the evaluation has no objective/quantitative basis on which to assess the adequacy of the PCID performance of expected results. • As assessed qualitatively by the communities and beneficiaries, the PCID program was considered to be highly effective relative to the expected results in terms of responsiveness, utilization, technical quality, inclusiveness, and overall satisfaction among the beneficiaries. To the extent that the development hypothesis refers to change in level of living or quality of life, beneficiaries affirm that the PCID program has made positive changes in their lives. • Despite the strong level of support and satisfaction for the completed projects, the construction process itself generated widespread complaints about contractor performance and oversight. 7 Evaluation Question 2: To what extent did the program succeed in identifying and addressing community infrastructure needs in various sectors? This question focuses on the project selection process and community outreach. Program documentation describes four channels for identification of potential PCID projects: • Water/sanitation projects (60% of PCID expenditures) are identified through the PWA in the West Bank, and CMWU in Gaza. The selection process is managed by a Steering Committee composed of PWA, ANERA, and USAID. • Other community service projects may come to ANERA through the Palestinian Authority (PA) line ministries. This category was principally applied to school construction projects, selected in collaboration with the MOEHE, with priorities set by the MOEHE directorates. • USAID priorities, and particularly community infrastructure associated with activities in health, education, youth, and democracy and governance may be directed to ANERA for implementation. • Projects may be identified by ANERA through its ongoing contacts with communities. This avenue has been particularly important for PCID in Gaza. Each potential project is assessed by ANERA engineers through a site visit and meeting with community leaders (except in Gaza), according to a defined set of technical and social selection criteria, specific to each type of project. Potential projects are then prioritized and selected by an ANERA internal committee and included in the Annual Work Plan submitted to USAID for approval at the start of each fiscal year. Projects presented by USAID technical offices and others (e.g., humanitarian assistance, High Impact Micro-Infrastructure Initiative (HIMII) projects) are given priority and are automatically included. While the formal selection process describes a top down approach, the interviews with community leaders and residents suggest a parallel process of advocacy by the community that intersects with the formal process and is important in communicating community needs. When community leaders were questioned about the source of the project, 87% replied that the village council had placed a formal request for the project, sometimes to multiple sources including ANERA, USAID, and government agencies. Only 13% said the project was presented to them as an option by another organization. In addition to these advocacy activities by the village council, the PCID community outreach model also may be a factor in the communities’ confirmation that PCID projects respond to critical needs. The outreach model has been applied across all communities and types of projects except in Gaza. Once 2Construction of additional rooms and external works, Secondary School for Girls, Bil'In 8 potential projects are identified, the implementation process is developed through and in collaboration with the village council by the PCID outreach and gender workers. Outreach begins with a series of meetings, including a community meeting with broad attendance to present the project design concept and elicit feedback. During construction, the outreach workers visit each project on a weekly basis to respond to residents’ concerns. At the end of the project, a kick-off event is held, and workshops are organized to raise awareness about the proper use of the service and responsibilities for maintenance. Follow-up meetings may continue for up to a year after completion. Another issue raised under EQ 2 is the utility of the PCID model of community engagement as a mechanism for implementing community infrastructure projects under other USAID programs, as anticipated in the results framework. Two of the four projects that came to PCID from other USAID offices were included in the evaluation community sample. These projects, compared to those identified by ANERA, show distinct approaches to the community in terms of needs and involvement. This difference was reflected in tensions and delays throughout the implementation process as reported in key informant interviews with both the PCID implementers and with the USAID project managers. In both cases, however, at the end of the process, all parties praised the completed project. EQ 2 Conclusions • PCID has responded to basic community needs that are recognized and articulated as priorities by community leaders and citizens. The PCID method for identification and selection also has been successful in reaching under-served communities. On the other hand, the evaluation methodology does not provide any basis for assessing whether these particular communities are more or less needy than others, or whether the selection process somehow gives some communities a better chance of benefitting than others. • The community outreach model has been effective in community engagement. Interviews and discussions in the communities showed an understanding of the project process and its relevance to them, and the openness of the discussions and opinions, both positive and negative, reflected this engagement. • The projects brought to PCID through other USAID programs derive their priorities and design through a separate process of identification and engagement. In some cases, the ANERA community outreach model, which is geared to basic needs, may not apply and may even be antithetical to the USAID program approach. Differences in assumptions about the importance, nature, and timing of grassroots engagement may generate misunderstanding between the PCID and the other program. Further, the expected results and indicators may be different. Evaluation Question 3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? Utilization: Building on the findings presented in Evaluation Question 1, the level of utilization of the projects overall is strong, especially for the water/sanitation projects and schools. The PCID program also had two additional characteristics to strengthen utilization. • Implementers noted that in the water/sanitation projects, increased access and lower cost may not necessarily translate into utilization, particularly in poor households for whom the upfront cost of connecting to the network may be prohibitive. The gender staff worked with the village councils to identify and assist elderly or disabled persons, or women heading households who might not otherwise use the service. • The PCID program design also supported utilization through an allowance to provide basic furniture and equipment for schools and other buildings, if necessary, to ensure that the finished structure could be used as intended. 9 Maintenance and Sustainability: The final step in the implementation process is raising awareness about proper use of the resources/services, and personal/community responsibility for maintaining the project. Workshops are sometimes held to train residents for simple maintenance and repairs. The kick-off meeting at the completion of the project is intended to reinforce ownership of the project, and brochures and other materials are distributed. At the same time, ANERA staff are clear in stating that PCID does not have a maintenance component and its responsibility and involvement in maintenance and repairs ends one year after completion. For the evaluation, each community visit included a site inspection to observe the state of repair and maintenance of the project, utilization, and presence of awareness materials. • In 20 projects, observers reported that the project is in good shape; problems were observed in two cases (storm water drainage; wastewater system.) Three other projects reported problems that already require on-going maintenance.1 In the interviews, community leaders were questioned about ownership of the project and responsibility for maintenance. Nearly all attributed the responsibility to the village council or municipality (the NGOs, said the responsibility is their own). When asked about resources available for the task, the leaders said they have manpower and technical capacity but lack financial resources and machinery. EQ 3 Conclusions • In congruence with the qualitative evidence of the priority attached to the projects and satisfaction with the results, the community responses from both residents and leaders indicate high levels of utilization for the core ANERA projects. • The communities and leaders recognize and accept responsibility for maintenance of the completed projects. The awareness-raising activities included in the community outreach model contribute to the sense of responsibility and stewardship for the project. Given that the projects are relatively new, observation of the condition of completed projects in the sample communities indicates on-going care and maintenance in terms of cleanliness and condition, as well as identification of the individuals managing these tasks. 1 These cases are discussed in the text. The circumstances in each are unique. In one case, the wastewater drainage is not draining properly; diverse explanations were heard from all parties involved. In the case of the sanitation network, the sub-standard operation seems to be due to the need for cleaning, as well as improper use by some residents. The three cases of ongoing maintenance reflect problems with materials. 3Construction of a Water Reservoir, Rantis, West Bank 10 • Community leaders recognize potential constraints on the ability of the village to provide maintenance in the future as the projects age, in terms of lack of finances and machinery. Evaluation Question 4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? The intent of this question was to illicit feedback from communities on potential opportunities for future support in water and sanitation and community infrastructure. Responses had no bearing on the assessment of the overall performance of the PCID. The question was posed as an open-ended question to community leaders, discussion groups, and survey respondents. EQ 4 Conclusions • Both leaders and residents identify clear infrastructure needs at the community level, suggesting that residents see the community as a focal point for projects to improve their daily lives. • The needs identified are consistent with those included in the PCID programs and in most cases, are basic needs related to daily lives. • Broadly, community leaders and residents agree on the needs and priorities. LESSONS LEARNED AND RECOMMENDATIONS • Lesson Learned: Outcome indicators and realistic targets are necessary to evaluate objectively and quantitatively whether the program produced the expected results and contributed to the development objectives. Most infrastructure projects are fixed-price contracts judged by standards of quality, timeliness and budget. As a cooperative agreement with social objectives and results, these output indicators, which are measured by the current project tracking system, give an incomplete picture of the expected outcomes. • Recommendation: In the future, performance indicators should be designed to link to the decision process and the expected results so that they can be used to guide management decisions and measure the expected outcomes. For example, the Performance Management Plan should include measures of access or utilization, with targeted levels of change, for other services besides water, such as school enrollment/attendance, or frequency of visits to the clinic. Performance indicators should be defined as a part of project implementation and not pre￾determined in the RFA. (Request for Application). In addition, in line with USAID policy, indicators and targets should be reviewed and revised annually, both to identify whether the program is performing as expected and to project realistic expectations about the achievements for the coming year. For example, if the actual performance is either far below or far above the target, the reasons for the discrepancy should be examined and explained, and the target for the following period should be revised to match realistic performance expectations. • Lesson Learned: In terms of achievement of results, in both quantitative and qualitative terms, the weakest link in the entire project process, based on commentary from community leaders, project design partners, and ANERA construction engineers, is the selection, performance, and supervision of contractors for project construction. ANERA acknowledges the presence of these problems and has built procedures into their operations (including some aspects of the first three items in the recommendation below) to counter these problems. Interviews in some communities and with other key informants, however, gave evidence that enforcement is uneven and the problems persist. 11 • Recommendation: Four main steps should be taken to counter the weaknesses observed in the implementation process. (1) Unified and clear specifications, standards and guidelines for materials and items used in the project construction should be included in the project bid documents, ensuring that the contractor cannot make changes without ANERA’s concurrence. Attention should be given to availability of specified materials in the local market. (2) Undertake a critical examination of the process for selection of contractors and the relative weight given to cost as a decision factor relative to experience and quality of work. (3) Review the adequacy of the quality control procedures and personnel in the field, including the experience and deportment of the engineers contracted by ANERA to provide full-time oversight of the construction process. (4) Finally, given the extent of USAID/WBG investment in infrastructure and construction, USAID may consider investment in training and information dissemination to strengthen the skills and knowledge base of contractors about products and technology that may affect design and implementation errors throughout the process. • Lesson Learned: The PCID focus on responding to community priority needs through small￾and medium-scale projects, in practice, tends to narrow the scope of the community needs assessment and the technical feasibility study to the particular “small” problem, and thereby limit attention to factors in the broader social and physical environment that may affect future activities. For example, in two communities, evaluators noted inadequate coordination with other projects (sanitation network), little consideration of next steps (drainage projects), or the contingencies presented by Area C boundaries. Recommendation: The feasibility study for the design concept should be broad enough to map the place of this small project within the complexities of the community social and environmental context, particularly in considering links and relationships to other current and future projects in the community. In addition to the standard studies that are required for USAID project approval, the engineers also should be required to demonstrate that they have consulted with the village/municipal council about of planned projects, and with USAID and other donors, and PA ministries working in the community. They also should present an analysis of the costs/benefits of omitting parts of the community in area C, and this analysis should be discussed and understood by the village/municipal council. • Lesson Learned: A key issue for the PCID evaluation has been the capacity and experience of the PCID program to serve as a flexible and rapid response mechanism for implementation of small- and medium-scale infrastructure in support of other USAID programs. On paper, the connection between the two programs seems straight forward since the PCID has a structure in place for rapid response to community infrastructure needs; in practice, the collaboration has not been smooth, due primarily to differences in the terms for engagement with the community and measures of achievement. At the same time, the construction has been successful and praised by both parties. Recommendation: The successful completion of these projects suggests that the utilization of the PCID program as a mechanism for other USAID programs is feasible and realistic. The expectations, terms, and responsibilities for these projects should be negotiated, understood, and agreed before the project process begins. For example, topics to be resolved include the extent to which the design concept and components are complete, the role of the other USAID program in oversight and control during the project design phase, selection of contractor, and construction, the extent of village/municipal council participation, input, and approval, and the role/control of the other USAID program in handoff and approval. 12 • Lesson Learned: While the evaluation documented an acceptance of responsibility for maintenance on the part of the village councils and municipalities, carrying out that responsibility in the future, as equipment ages, may be a problem because of lack of financial and mechanical resources. The gap is likely to be a problem particularly in the villages (compared to municipalities), because they do not have a revenue source or paid staff to handle these tasks. The PCID program does not have a maintenance component. At the end of one-year after completion, both the implementers and the contractors are separated from the project. • Recommendation: Additional attention to arrangements and sources of finance for maintenance should be built into the PCID project process. Several approaches to this issue have been suggested. First, simplify the design and materials. For example, in water projects, maintenance problems are most likely to occur in pumps rather than in the network. While elevators are important for persons with disabilities and elderly, the lack of parts, technical expertise, and reliable electricity cause constant problems in utilization and functionality. Second, ensure that contractors provide a manual for operation and maintenance before closure, including on-site training for key village personnel. Third, require a maintenance plan as part of the close-out of each project including institutional responsibility and source of finance. I. PROGRAM CONTEXT AND PURPOSE Within the framework of USAID’s goal to end extreme poverty and promote resilient, democratic societies, the Economic Growth and Infrastructure (EGI) Office of USAID’s Mission in the West Bank and Gaza (USAID/WBG) seeks to generate tangible benefits in the lives of Palestinians by increasing access to clean, reliable and affordable water and energy, and stimulating the growth of the Palestinian economy. The Palestinian Community Infrastructure Development (PCID) program is currently the only USAID/WBG infrastructure activity explicitly directed to community-level needs, first in terms of access and affordability to water and sanitation, and second, other types of community infrastructure to improve access to public services. It is focused on small and medium infrastructure with particular attention to rural, vulnerable or under-served communities. In this way, it is distinct from other infrastructure programs in the USAID/WBG EGI Office, which are directed to large scale regional activities. It is implemented in the West Bank and in the Gaza Strip. The underlying development hypothesis is that improved access to basic public services in the community will contribute to social and economic development. The PCID program is implemented through a cooperative agreement, (CA) 294-A-13-00005-00, which was signed on March 6, 2013, with the international non-governmental organization (NGO), American Near East Refugee Aid (ANERA), which has been active in the region since 1987. The agreement was for a US$115,000,000 five-year program, with USAID providing US$100,000,000 in funding. In 2017, PCID was extended for a sixth year, and is scheduled to end March 5, 2019. PCID is implemented by ANERA through four offices, the central ANERA headquarters in Jerusalem, two field offices in the West Bank, (in Nablus in the north, and Hebron in the south), and a technical office in Gaza. In the program ANERA, with the sub-grantee Women, Media, and Development (TAM) (sub-grant ended July 2017), managed the entire infrastructure process from project and community selection, community outreach and engagement, project design, construction, and hand-off to the community. The program was designed not only to prioritize small, vulnerable, and marginal villages but also social inclusion through mainstreaming gender in all phases of the program in terms of participation, decision-making, recognition and resolution of needs, and employment. Gender sensitivity focuses 13 particularly on women, but also includes other marginalized groups such as youth, elderly, and persons with disabilities. At the time of the evaluation (GEO-MIS, 5 Feb 2018), PCID had completed 39 projects in the water, sanitation, education, health and community infrastructure sectors, affecting nearly 340,000 people living in 33 communities in the West Bank and Gaza. These projects created nearly 170,725 person-days of employment (equivalent to 8,536 short-term jobs). An additional 13 projects are ongoing, while another five projects are under the process of design and/or approval. The total USAID obligation to date has been US$83,454,9464. Two-thirds (n=26) of completed infrastructure projects are in the West Bank, and most are in the water and sanitation sector. Completed education infrastructure projects (7) represent about 18% of all completed projects, and all are in the West Bank. Road projects (4) represent about 10% of all completed projects, and they are mostly in the Gaza Strip. Other infrastructure projects completed by PCID include a health clinic, a rehabilitation center, a youth center, a sports playground, three public spaces (parks and playgrounds), and a municipal marketplace. The regional distribution of completed projects is shown in Table 1 below. Table 1: Distribution of Completed Projects by Region and Sector II. EVALUATION PURPOSE AND OVERVIEW In accordance with global USAID evaluation policy, large activities like the PCID program are subjected to an independent external evaluation in the final year of the activity to assess program performance, highlight the successes and challenges of the programmatic approach and implementation, and to inform future directions and designs in this area. The primary audience for this evaluation is USAID/WBG, especially the EGI Office, and ANERA. The secondary audiences include other stakeholders in the infrastructure, health, education and governance sectors, and potentially USAID on a global level. The PCID final performance evaluation is structured around four evaluation questions (EQs), identified by USAID/WBG to serve as the basis for discussing achievements, challenges, lessons learned and recommendations. The questions look at the quantitative and qualitative results and impacts of the program, and the viability and utility of the community outreach model. Sewage Network Reservoir Drainage Network North 10 10 1 3 2 0 3 1 0 0 0 0 Middle 5 6 0 2 1 0 2 0 1 0 0 0 South 11 12 0 0 2 5 2 1 0 1 1 0 WB 26 28 1 5 5 5 7 2 1 1 1 0 GS All 7 11 0 0 0 4 0 1 1 1 3 1 WBGS Total 33 39 1.0 5.0 5.0 9.0 7.0 3.0 2.0 2.0 4.0 1.0 No. Communities Roads Markets WB Region Sub￾Region No. of projects Disribution of Projects Water & Sanitation Education Public Spaces Youth Centers and Facilities Health 14 EQ1: To what extent did the implementation of PCID activities (infrastructure projects) support the project development hypothesis and intended results? There are two aspects to this question: the extent to which the program achieved the expected results and targets; and, ii) in terms of the development hypothesis, the extent to which the results contributed to social and economic development. The evaluation examines this question first in terms of the objective, quantified results spelled out in the results framework, and the associated indicators. Second, the team examines the results from the point of view of the communities, in qualitative terms, in relation to well-being and prospects for the future. EQ2: To what extent did the project succeed in identifying and addressing community infrastructure needs in various sectors? Building on EQ 1, the focus here is on the framework PCID used to select projects, and the effectiveness of this process together with the community outreach activities in addressing infrastructure needs identified by the community leaders and residents. The second part of this question concerns the effectiveness of the PCID approach as a mechanism to support and enhance the programs of other USAID technical offices through infrastructure projects. EQ3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? The question of maintenance and sustainability is crucial in assessing the potential for infrastructure projects in small communities to have an extended effect on level of living. The evaluation examines the relevance of the community outreach model to this issue. EQ4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? Information about continuing needs for community water, sanitation and infrastructure is important for assessing the viability and need for USAID support for small and medium infrastructure projects in the future, and also provides another angle for the question of whether PCID has addressed the types of issues identified by the communities. EVALUATION TEAM AND REPORT STRUCTURE The evaluation report describes the sources of information and the analysis process to generate the evidence brought to bear in responding to each of the four questions. A series of conclusions are presented based on this evidence, and from these, the report extracts a series of lessons learned and recommendations for the PCID program and for future programs of this type. The evaluation was implemented by the QED Group LLC as part of their support to the USAID/WBG Mission through the Performance Monitoring and Evaluation Project (PMEP). A team of four independent evaluators conducted the fieldwork and analysis in February and March 2018. The team was supported by the Palestinian Center for Public Opinion (PCPO) in the data collection and analysis process, and the three Palestinian team members were recruited and contracted by PCPO. One of the four evaluators is a resident in Gaza and carried out the interviews and community visits there. The other evaluators were responsible for data collection in the West Bank. The same data collection instruments were used in both locations. 15 III. EVALUATION METHODOLOGY As a performance evaluation (rather than an impact evaluation), without a baseline or a comparison group against which to assess the magnitude of results or change, a mixed methods approach to data collection and analysis is used. In this approach, multiple sources of data are tapped to draw a balanced understanding of the activity from various points of view, utilizing, for example, both quantitative and qualitative measures of the results and process, and seeking inputs from a broad spectrum of stakeholders. This section provides an overview of the data sources and analytic methods used to address the evaluation questions. Data Sources The evaluation findings draw on four main sources of data: 1. Document Review. PCID program documentation provided by USAID and ANERA include the quarterly and annual reports, annual workplans, annual Performance Monitoring Plans, and the GEO-MIS program database. These documents are the principal source of quantitative data about program outputs and outcomes. They also track changes in the program over time and the flow of projects over the five-year span. In addition, the team also consulted other contextual information about the USAID/WBG infrastructure program. The complete list of documents consulted is in Annex D. 2. Key Informant Interviews (n=28) with stakeholder organizations. Individual interviews were conducted, primarily by the Team Leader, with: • ANERA staff in the PCID central office, and in the Hebron field office; • TAM, the ANERA sub-grantee for gender, social inclusion, and outreach; • ANERA partner engineering firms for project design and proposals; • USAID technical, program, and management officials; • Palestinian Water Authority (PWA) in the West Bank; • Coastal Municipality Water Utility (CMWU) in Gaza; and, • Ministry of Education and Higher Education (MOEHE). The list of people and organizations interviewed by the evaluation team is at Annex E. 3. Field Visits to Beneficiary Communities Sample: A purposive sample of 18 of the 39 communities in the West Bank and Gaza with projects completed by PCID by February 5, 2018. The sample was selected to represent the diversity of types of projects (water and sanitation, school construction/rehabilitation, storm water drainage, internal roads, and social assistance), geographic location (Gaza, 4, north/central West Bank, 8, and south West Bank, 6), year of completion, and size in terms of cost. The communities included in the sample are shown in Table 2 below. Data collection in each community: The same data collection activities were done in each community (to the extent possible), using uniform data collection tools to allow aggregation and comparison of communities. 16 • Key informant interview(s) (KII) (n=23 interviews), with the local government unit (village council or municipality) and, in some communities with other individuals involved with the project (e.g., the school principal, the clinic administrator). For the NGO projects in Gaza, the interview was held with the head of the NGO board. • Group discussions with residents and beneficiaries (n=16 groups with an average of 7 participants/group, including 53 women and 65 men.) In most cases, these groups were called together by the village council leaders. • Inspection of the project site and completion of an observation checklist. (n=22) 4. Survey of Beneficiary Households Sample: A random sample of households in each of the 18 communities in the area surrounding the project, targeting completed interviews with 30 households per project (i.e. more people were sampled in communities with more than one project.) Interviews: The questionnaire was prepared by the evaluation team, pre-tested in selected communities, and administered by trained enumerators identified by the PCPO, which was responsible for the survey. Responses were recorded electronically and transferred directly into a single database. Each interview took approximately seven minutes. The timing of survey administration was separate from the schedule for community visits. Respondents: A total of 619 interviews were completed. In the analysis, the responses were weighted to ensure equal representation across communities/projects. The sample was 53% female and 47% male, with an average age of 37. The average household size was six, and 76% of the respondents were the head of household or spouse of the head. A detailed description of the survey methodology is included in Annex B. All of the data collection tools, for the KIIs, community data collection, and the survey questionnaire are included in Annex C. 17 Table 2: Sample Communities Data Analysis The analysis in a mixed methods evaluation is an iterative process built around the evaluation questions. The analytic process involves drawing on evidence from the various data sources to articulate and corroborate findings relevant to these questions, with cross-checks across sources. This analytic process, referred to as triangulation of the data, is continuous throughout the data collection period as new pieces of evidence are brought to bear. The findings relative to each question are used to establish conclusions, which in turn serve as the basis for extraction of lessons learned and recommendations. The key informant interviews and group discussions were administered as open-ended questions to elicit discussion and unanticipated insights. In the analysis, the responses were coded by question on excel sheets to facilitate analysis across communities and to identify patterns in the responses. These patterns are often reported in terms of the percentage distribution of responses among groups or community leaders. The analysis of the household survey was done using SPSS and relied primarily on simple distributions and cross-tabulations. Comparisons across data sources reported in terms of percentages show relative but not absolute differences. Limitations As noted in the introduction to the Methodology section, the absence of baseline measures or a comparison group means that it is not possible to assess the degree of change and impossible to attribute cause. Likewise, in the PCID program, with a diversity of projects and communities, generalizations made on the basis of analysis of a sample may not apply precisely to any specific case. A more serious limitation in terms of generalization for the PCID program as a whole is the fact that community data was collected by different teams in Gaza and the West Bank. According to all reports, the situation in Gaza is very different from the situation in the West Bank. Although the same tools were used for data collection in both areas and the entire team was engaged in discussions about the results, it is possible that there were some systematic differences in the two sets of data that were not 18 reflective of the differences on the ground. The problem of separating actual differences between the PCID program in Gaza and in the West Bank from differences in interpretation across the two teams is particularly difficult because the two teams could not work face-to-face. Moreover, the absence of reliable indicators to measure project results means that the assessment of project effects on level of living or social/economic development is largely subjective. IV. EVALUATION FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: To what extent did the implementation of PCID activities (infrastructure projects) support the project development hypothesis and intended results? FINDINGS The Results Framework for the PCID program has two Intermediate Results (IRs) focused on availability and affordability of water and sanitation, and access to community infrastructure facilities. The sub￾objectives under IR 2 point to two additional characteristics of PCID, (1) to serve as a mechanism for small and medium infrastructure projects, and (2) to respond to unforeseen urgent infrastructure needs. The two aspects of the program, executing internally defined priorities in water and sanitation and other community infrastructure projects, and as a mechanism for other USAID and emergency infrastructure needs, affect both the delineation of “intended results” in terms of targeted indicators, and achievements within a particular timeframe. The development hypothesis underlying this framework is that increased availability and affordability of water and sanitation and access to other community services will contribute to improved social and economic development. 2 2 Improved social and economic development is not defined either in the narrative or in terms of specific indicators. Measurable changes in terms of increased investments, business development, or population growth or stabilization have not been tracked by the Mission or the project, and are unlikely to be observed, except in isolated anecdotal cases during the five￾year span of the PCID program. In this evaluation, social and economic development is operationalized in terms of improved level of living or quality of life, as assessed by the beneficiary. Figure 1. PCID Results Framework IR1: Availability and affordability of water and sanitation increased. 1.1 Increase access to water and sanitation systems and other public infrastructure needs in the West Bank and Gaza Strip. IR2: Access to community infrastructure facilities increased. 2.1 Contribute to increasing the impact of USAID-funded programs in health democracy and governance, education and private sector needs. 2.2 Enhance access to health, schools, youth center and community infrastructure facilities in rural, urban, vulnerable underserved communities. 2.3 Enhance the lives of Palestinians’ well-being by responding to unforeseen infrastructure and other urgent needs in the West Bank and Gaza Strip. 19 Quantitative Outputs and Achievements In quantitative terms, according to the USAID GEO-MIS database of 5 Feb 2018, PCID had completed 39 infrastructure projects and 13 additional projects were underway. (IRs 1 and 2) Twenty of the completed projects, were water/sanitation or drainage activities. Seven were education projects. Six new projects are currently under review. Included in these projects are three projects carried out for other USAID programs, (IR2.1) building a youth center in Jericho (Partnership with Youth Activity in the Education and Youth Office), a health clinic in Walajeh (Health Office), and the Old Market in Bethlehem (EGI Office). A fourth project for school construction in Area C has been dropped recently because the necessary permits have been declined.3 In addition, in July 2014, through CA Modification 3, up to US$5,000,000 were reprogrammed for humanitarian assistance in the Gaza Strip. (IR2.3) According to the modification, these funds may be used “to provide rapid response humanitarian assistance of goods, services, and infrastructure repair.” This assistance may include “repair of buildings, repair of water and sewage systems, debris/waste removal, and storage/distribution of humanitarian items.” The PCID Annual Report I (September 2014) details the assistance provided in that year under a series of Gaza Rapid Response Action Plans, including procuring, shipping and distributing bottled water and hygiene kits to displaced families in UNRWA (United Nations Relief and Works Agency) shelters in Gaza, establishing a mechanism for locally procuring desalinated tankered water for UN schools and three communities in Gaza - Abassan Al-Kabira, Al-Mossadar and Al-Maghazi, and suppling diesel fuel to CMWU’s (Coastal Municipalities Water Utility) water and sanitation facilities. All activities were coordinated with UNRWA, the WASH (Water, Sanitation, and Hygiene) Cluster and the CMWU. Other infrastructure activities in Gaza have continued throughout the PCID program, albeit with extended delays due to Israeli security measures related to shipment of building materials. The support of infrastructure and access to community services in vulnerable or underserved rural and urban communities (IR2.2) is frequently cited as an important characteristic of the PCID program, but it is not systematically tracked. The PCID program is unique in USAID/WBG infrastructure activities in its work with village councils as well as municipalities. None of the project databases distinguishes villages from municipalities but a review of the completed projects (including those associated with NGOs in municipalities) shows that slightly more than half of all projects have been in villages. The community visits also verified, through observation, that some, but not all of the communities receiving services are clearly isolated and underserved, both in terms of the services available (e.g., health clinic, village council meeting space, road conditions) and the accounts of council members about their outreach efforts. According to ANERA, the focus on under-served, vulnerable, and rural communities is applied as a secondary selection factor, i.e., if two communities require water networks, the choice would be made to work with the more under-served of the two. 3 In the Oslo Accords (1995), all lands in the West Bank were designated as either Area C, under full Israeli administrative control, Area B, under Israeli security control, or Area A, under full administrative control of the Palestinian Authority. Nearly 70% of the West Bank territory is designated as Area C. Construction in this area requires permits issued by the Israeli District Civil Liaison Office (DCL). 20 PCID Performance Indicators In USAID, program outputs and outcomes are measured in terms of indicators, intended to track progress and achievements and to provide guidance for management and adjustments in implementation. Indicators are defined and refined in the Annual Performance Monitoring Plan (PMP), as a part of the Annual Work Plan (AWP). Most indicators tracked by the PCID program are output indicators, meaning they are a simple count of tasks completed (A full list can be found in Annex A). Table 3 below shows key illustrative PCID indicators and targets relative to actual achievements for each year of the program, as reported in the Annual Reports and Annual Performance Monitoring Plans. Measured in these terms, with only a few exceptions, the PCID has under-performed and failed to meet expectations across the board. The data also show large variations in project completion across years. The targets cited in the Request for Applications (RFA) and the CA for the five years of the program were 60 communities with improved potable water, 20 communities with improved sanitation, and 50 education health and other public facilities constructed or rehabilitated. As documented above, actual PCID projects did not approach these global targets. Further, the total obligated amount as of September 30, 2017, was US$83,454,946.49 rather than the expected US$100,000,000. Table 3: Illustrative Indicators, Targets, and Achievements In key informant interviews with implementers and with USAID, various explanations were offered for the discrepancy between the targets and the actual achievements throughout the course of the program. 21 • The source and reasoning for the original targets are not clear. The original targets in the RFA were unrealistic relative to the resources. The annual review process did not produce substantial revisions in the indicators or the targets. • Many of the funds were earmarked and non-transferable. Funds allocated for water and sanitation could not be spent on social assistance infrastructure projects. Funds allocated to Gaza could not be used in West Bank communities. In some cases, (usually projects from other USAID offices), funds were targeted to specific buildings. Because of the lack of flexibility delays in a particular area, such as those associated with Gaza approvals for shipping materials,4 meant that funds could not be spent and targets could not be met. • USAID allocations were often not received in a timely manner. ANERA could not move forward with contracting designs or construction until the funds were in-hand. These delays, as well as those associated with problems in securing licenses and permits, especially in Area C in the West Bank, meant that projects were not begun or completed within the targeted fiscal year. Delays within the project process itself also were discussed, such as USAID approvals and required revisions at various stages. Although PCID is a cooperative agreement/grant, which contributes to flexibility, because it is an infrastructure program, it has “substantial involvement of USAID,” meaning that it is subject to engineering and spending approvals by USAID at each stage in the process (i.e., the original identification as a potential activity, the design concept, the engineering project proposal,). • Another factor may be the lack of predictability in the unique operating environment of West Bank/Gaza, including political priorities or requirements for rapid response that are superimposed on the annual plan. For example, the High Impact Micro-Infrastructure Initiative (HIMII) was activated by Secretary Kerry in September 2013, and the selected PCID projects were immediate priorities for implementation. The call for humanitarian assistance in Gaza in 2014 is another example. • More broadly, the utility of the indicators and attention to targets may be compromised by the separate components of the agenda, PCID as a community infrastructure program with an annual plan and priorities, and PCID as a mechanism for other Mission programs. The evaluation has no basis on which to judge the relative importance or validity of these various explanations. Given the available information, however, it is clear that performance indicators were not considered to be reliable quantitative measures of results or achievements. Qualitative Assessment of the Community Projects The methodology for the evaluation was designed to examine the program outcomes and results in qualitative as well as quantitative terms. As stated in the development hypothesis: Did the program make a difference for the quality of life and future development in the communities? 4 The Gaza Reconstruction Mechanism (GRM) is a temporary agreement created by the United Nations and agreed between the Palestinian Authority and Israel in September 2014. The mechanism is designed to address Israel’s security concerns while allowing the entry of construction materials into the Gaza Strip. Materials defined as “dual use” by Israel, meaning they could be used for construction or for weaponry are subject to surveillance and approval by Israeli security, restricting their availability and causing severe delays in entry and delivery. 22 Achievement of Expected Results For IR 1, increased availability and affordability of water and sanitation, in interviews with village council and municipal leaders in the seven sampled communities with water/sanitation projects, five of the seven leaders said that the project had resolved a critical issue in the community. In two communities, the leaders said it had resolved the issue to some extent. In one case, the response reflected the absence of coverage for the households in a part of the community in Area C; in the other, a sanitation project, the leader said that some problems remained because the system was not being used correctly by everyone and the network was clogging and in need of maintenance. He also was concerned about some remaining areas without access to the network. All these leaders cited positive changes in their communities as a result of the project including increased access to water, reduction in cost, and improved quality of life. 5 Among survey respondents in the communities with water/sanitation projects (n=210), 92% and 97% respectively, said that the water projects implemented in their communities were important to their families and to their communities. Further, the average level of satisfaction with the project (on a scale of 1 to 10, where 10 is very satisfied) among survey respondents in the communities with water/sanitation projects was 9.0. While the assessment of the water/sanitation projects (IR 1) by leaders and beneficiaries was generally very positive, for IR 2, increased access broadly to community infrastructure facilities and enhanced quality of life in the community, the responses of community leaders and beneficiaries (in the household survey) showed a similar affirmative pattern but with more variation. With 82% of survey respondents in communities with non- water and sanitation projects indicating projects were important to their communities. Across all communities in the sample, including both the West Bank and Gaza, and water/sanitation and other facilities, the interviews with community/NGO leaders and the household survey showed a consistent positive assessment of the projects in terms of responsiveness to the needs in the community for access to services.6 Among leaders, 82% said the project is fully utilized, 60% said it met their expectations to a great extent, and 81% said it had resulted in change in the community. (For the breakdown of responses by type of project, see Annex H.) In the survey of beneficiary households, across all projects (n= 619), 82% said that someone in the household is using the services, 77% said the project filled an urgent need in the community, and 84% agreed that the project is important to the community. More than half (54%) said the project met their expectations to a great extent, while the remainder said it met their expectations to some extent. In the 5 In Gaza communities, the water projects affect access and availability but not affordability because of the CMWU unified tariff on water per unit. Individuals may not be able to afford more water even though water is more available to them. 6 The responses from the discussion groups of beneficiaries are less relevant to the question of responsiveness to community needs because the groups were composed of those utilizing the services of each particular project (e.g., users of the youth club or the rehabilitation center, and parents of students in the school.). 97% 82% 84% Water & Sanitation Projects [N=210] Other Infrastructure Projects [N=409] All PCID Projects [N=619] Survey Respondents felt PCID Projects were Important to their Communties Figure 1: Importance of PCID Projects 23 interviews with community leaders cited above, leaders in seven communities said that the project resolved an urgent problem in the community “to some extent.” The reasons for this response are instructive, as the most consistent reason given was that the water or sanitation network did not cover the entire community because the households in Area C were excluded. Likewise, the improvement of the internal road was considered only partial because the road extended into Area C and was not changed. Overall, although the project was needed and effective, its impact was compromised by the unresolved jurisdictional barrier. In terms of the changes that leaders have seen as a result of the project, the leaders in communities with water projects cited greater and more reliable quantity, better quality, and lower costs, and in one case, better agriculture. For sanitation and waste water, leaders mentioned improved health and community environment, as well as lower costs and time￾savings. For school projects, the factors mentioned were fairly consistent across communities: improved attendance, lower dropouts, especially for girls, and improved morale. In at least two small villages, leaders referred to a change in attitude and outlook, paraphrasing, “now that this has been achieved we can look forward to other improvements in the future.” Neither community leaders nor participants in discussion groups mentioned any evidence of emerging effects on social or economic development in the community, (e.g., new businesses or investments) but the timeframe is short since none of the projects has been in place for more than four years. Paraphrasing an observation expressed in at least one discussion group in relation to a water project, “Now that water is no longer the central daily concern, it is possible to start thinking about other possibilities.” 4Computer Lab, Youth Development Resource Center, Jericho 24 The situation in Gaza is somewhat different. Because of the no-contact policy and constraints on community engagement, projects developed through local NGOs tend to have a strong impact on the immediate clientele but a less defined effect on the community as a whole. Technical Quality In addition to the results in terms of responsiveness and effect on well-being in the community, the cooperative agreement cited other key principles to guide the PCID technical approach including technical excellence in design and implementation, and gender sensitivity, which are important in evaluating the extent to which the program achieved the expected results. Based on observations of the project sites and discussions with community leaders, in general the technical quality of construction is good, utilizing appropriate quality of materials and adhering to the standards of the International Building Code, as mandated by the USAID mission. Observations at all the sites in the sampled communities, except two, showed that the physical structure is entirely or mostly “as planned,” and, except for these same two sites, observers judged the projects to be in good shape.7 At the same time in the majority of projects, community leaders and discussion group participants cited specific shortcomings in the design or, in particular, in the work of the contractors. The concerns about design are complex and difficult to assess as objective external evaluators. The design process for each individual project begins at ANERA (or at times, PWA, MOEHE, or the community) with a design concept. The detailed design/project proposal is done by a private design firm (i.e., ANERA partner engineering firm), which ,ideally, is experienced/specialized in that type of project (e.g., water networks, drainage, rehabilitation, roads.) The project proposal is reviewed and approved by the USAID engineering department, and in the case of water/sanitation projects by PWA, and schools, by the MOEHE before construction begins. Based on key informant interviews, all engineers involved in the design and approvals visit the project site.8 Two issues related to design stand out. In at least three communities in the sample, leaders pointed to issues related to the narrowness of the design, whereby the design of the small project being constructed under the PCID did not take account of the other concurrent projects or future projects related to the same issue. For example, PCID may repair a section of an internal road in Area A or B without attention to the continuation in Area C.9 Examples also were cited in storm water drainage projects visited by the evaluation team. A second issue concerns the specifications for materials, both in terms of availability of products in the local markets and definition of the requirements for substitution. 7 One of these two projects (storm water drainage) was judged by the community leaders (and by the evaluation observer) to be inadequate. The community leaders cited design issues. In the other case, the mayor complained of continuous high maintenance costs, which he attributes to poor work by contractors. 8 The ANERA design department engineers visit the site at the beginning of the process and sometimes at the initiation of the construction. The design engineering firms visits the site when they do the design. The USAID engineers visit each site at least once, when they review and approve the project proposal. 9 The issue of lack of services in Area C was raised frequently in interviews and discussion groups. ANERA is not permitted to do any construction in Area C without permits from the Israelis. Securing the permits is an arduous and frequently unsuccessful process. According to ANERA, an analysis is made of the impact of the Area C restrictions on the operational integrity of the project before going forward with the design. 25 A further issue raised in Gaza is the need to focus on materials that are less subject to dual-use restrictions.10 The issues raised about the contractors and the construction process were more widespread. Community leaders complained of ways in which contractors cut corners by substituting materials or failing to comply with standards. In addition, while the PCID design includes open channels of communication with the community, community leaders frequently said they felt that, in contrast to the planning process for the project, at the time of implementation their observations were no longer welcome. Various leaders quoted the phrase from contractors and on-site engineers that the project was going forward “as per design,” without further discussion. Although, by plan, the village council or municipality is supposed to sign off on the completed project, in at least four communities, community leaders said they did not sign off.11 The concerns about contractors were not limited to the community leaders but also were voiced by the design engineers and the ANERA field offices. The quality of the work, inappropriate and unauthorized changes in design or materials, and the use of inexperienced and unqualified workers were also raised as issues. According to these key informant interviews, these problems are not unique to ANERA, and in fact, ANERA has taken steps to increase supervision and oversight in the field, but the frequency of the discussion points to continuing problems with oversight and quality control. Gender Sensitivity and Inclusiveness Finally, an important factor in the PCID program, which is not usually associated with infrastructure activities, is attention to gender sensitivity and social inclusion. The staffing of the program and the outreach activities to actively engage women and other marginalized groups in the project process demonstrated an active commitment to social inclusion. TAM, designed and implemented the gender inclusion aspect of the program through a four-year sub-grant (July 2013 – June 2017). According to the TAM Final Report, the overall purpose of the sub-grant was to promote gender equality in the Palestinian community by giving gender groups equal opportunities to access resources through effective participation at the community level, identification and voicing of their needs and priorities in infrastructure services, and by participation in decision-making processes. Under the sub-grant, TAM gender outreach staff worked as a part of the team in the PCID field offices (except Gaza) and in the central office in Jerusalem. They contributed to all PCID projects during the sub-grant period. In this capacity, they reviewed the project designs and made recommendations to support gender equality, based on criteria of accessibility, privacy, safety, security, and compatibility. They also worked with the village councils to encourage the participation of women, persons with disabilities, youth and elderly in community meetings related to the project, in the planning/design phase, during implementation, and in awareness-raising activities at the conclusion of the project. Gender outreach workers also went door-to-door in the villages to identify gender-specific needs and conducted follow-up visits in some communities to hear and address remaining social issues. TAM also produced a gender checklist to identify the characteristics and needs of potential beneficiaries of the project (See Annex G), and awareness-raising tools including posters and billboards, brochures, coloring 10 According to ANERA, “Wherever applicable and whenever the quality and durability of the finished produce allow for alternative non-dual use material. ANERA is making this substitution. 11 ANERA explained that community members were not admitted to the construction sites primarily because of safety requirements. Lack of relevant expertise also was cited as a reason. ANERA said that at times, a community engineer was invited to join the construction oversight team. The evaluation team did not hear about this situation in the community visits, even in the municipalities that are staffed with trained engineers 26 books and pencils, an educational animation film, and hygiene kits that were distributed during the awareness sessions and inauguration events. Finally, TAM worked closely with the ANERA Monitoring and Reporting Unit and documented a before/after time use survey for women in a sample of households in six communities with water or sanitation projects. They found consistent time savings for women as a result of the projects, both in terms of securing water and in carrying out routine household tasks, which allowed women to engage more in productive activities and self-care. (See the TAM Final Report in Annex G.) The extent, and quantitative results of TAM’s gender-based activities are shown in the following table extracted from their Final Report: Table 4: TAM Participation in PCID Activities (TAM Final Report, 2017, Exec. Summary) In the household survey, comparison of the responses for men and women showed few differences in their assessment of their involvement in the project and of its importance or success. (See examples in Annex H.) Both men and women said the project met their expectations, that they or someone in their household uses the services, and that the project met an urgent need. Both men and women were satisfied with the projects. The only notable difference is that women reported they were less likely than men to be active participants in the meetings they attended about the project (active participants: men 53%, women 27%; mostly a listener: men 47%, women 73%). Gender Community Participation Total Activities Total Participants Males Females PWD Awareness Raising Workshops 9 447 175 272 14 Educational Open Days 6 2,123 945 1,178 4 Teachers Workshops 2 35 20 15 0 Health Day 1 112 42 70 0 Impact Assessment Study 1 201 109 92 3 Coordination Meetings 119 577 284 293 8 Field Visits 271 1503 703 800 0 Participated in Community Gatherings 34 1764 1,024 740 54 Total 6,874 3,344 3,530 87 Percentage 48.6% 51.4% 1.3% 27 Satisfaction Finally, as a broad summary measure in qualitative terms of how beneficiaries in the communities rated the PCID projects, survey respondents were asked “On a scale of 1 to 10, where 1 is not satisfied at all and 10 is very satisfied, how satisfied are you with these services?” The results, by type of project, are shown in the graph below. (Results by community are shown in Annex H.) While beneficiaries are positive about all the projects, the results for the core ANERA projects, especially water/sanitation and schools, which focus on basic needs, are nearly unanimous, while the projects with the more limited clientele are scored lower, even by those clients. (The exception is the Rehabilitation Center in Gaza, which received very high scores from clients,) There also is a rough correlation between the reported level of utilization and satisfaction. Figure 2: Satisfaction with Services by Type of Project (n=619) CONCLUSIONS • PCID completed small- and medium-scale infrastructure projects in support of all intermediate results and objectives. • PCID did not achieve the expected quantitative results as measured by the performance indicators. The gap between the actual and targeted results may reflect unrealistic targets, contingencies in the operating environment, or PCID underachievement. As a result, the evaluation has no objective/quantitative basis on which to assess the adequacy of the PCID performance of expected results. • When assessed in qualitative terms by the communities and beneficiaries, the PCID program was highly effective in terms of the expected results, as measured by responsiveness, utilization, technical quality, inclusiveness, and overall satisfaction among the beneficiaries. To the extent 5.88 6 6.6 7.36 7.82 8 8.18 8.34 8.85 8.99 10 Mean value of satisfaction with the services provided by the projects. 28 that the development hypothesis refers to change in level of living or quality of life, beneficiaries affirm that the PCID program has made positive changes in their lives. EVALUATION QUESTION 2: To what extent did the program succeed in identifying and addressing community infrastructure needs in various sectors? FINDINGS Project Identification This question focuses on the project and community selection process and community outreach. According to the program documentation, potential PCID projects are identified through four channels. • First and foremost, water and sanitation projects, which are to account for 60% of PCID expenditures, are selected through the PWA and in Gaza, through the CMWU. The selection process is managed through a Steering Committee of representatives from PWA, ANERA, and USAID, chaired by USAID.12 In February 2014, ANERA and PWA also signed a Memorandum of Understanding (MOU) spelling out the procedures, requirements, and responsibilities of each party. • Second, non-water projects may come to ANERA through the Palestinian Authority (PA) line ministries. This category was principally applied to school construction and rehabilitation projects, selected in collaboration with the MOEHE. Priorities are set by the MOEHE directorates. ANERA signed an MOU with the MOEHE in 2014. Both the PWA and the MOEHE present ANERA with a prioritized list of projects, based on their own internal criteria, from which to choose. • The other two channels are less formal: USAID priorities, and particularly community infrastructure associated with activities in health, education, youth, and democracy and governance; and, ANERA, through its contacts with communities based on decades of humanitarian assistance activities in Palestine. The latter has been particularly important to PCID activities in Gaza because of the no-contact policy with local governments, and in identification of marginalized, vulnerable, or under-served communities. Project Selection Each potential project is assessed by ANERA Design Team engineers through a site visit and meeting with community leaders, according to a defined set of technical and social selection criteria, specific to each type of project. These criteria include technical feasibility and viability as well as availability of land, and community commitment to the project both now and in the future. (The list of selection criteria is included in Annex G.) Potential projects are prioritized and selected by an ANERA internal committee composed of the Chief of Party, Deputy Chiefs of Party, Chief Design Engineer, and the Chief Construction Engineer in Hebron. The selection process is made at the beginning of each fiscal year as 12 There is a sub-committee for Gaza, which also includes CMWU. 29 part of the Annual Work Plan process and updated based on availability of funds.13 All selections are approved by USAID. According to ANERA, and confirmed by PWA and MOEHE, this formal selection process is applied to potential projects presented through formal government channels. ANERA affirms that it also is applied to projects and communities identified through ANERA informal contacts, particularly in marginal or under-served areas. In Gaza, where ANERA does not work with local communities, the selection criteria are applied to the NGO partner organizations. In the case of water projects, ANERA moves forward with CMWU, based on its priorities. On the other hand, projects presented by USAID technical program offices and others (e.g., humanitarian assistance, HIMII) are given priority and are not subjected to the same selection process. While the formal decision process describes a top down objective approach in the selection of projects and communities, emanating primarily from the PA, the interviews with community leaders and residents suggest a parallel process of advocacy in the community that intersects with the formal process, and is important in communicating the basic needs of the communities. The key informant interviews with community leaders (the head of the village council, mayor of the municipality, or in Gaza, the president of the NGO board), included a series of questions about the source of the project. The responses were similar across communities. Two-thirds of the leaders said that they have prioritized the needs of the community in a strategic plan. (Others said they don’t have a strategic plan but they do have a list of priorities.) When asked how the idea for the project came up, almost all (87%) said that the village council placed a formal request for the project. They had placed their requests with ANERA, PWA or MOEHE, USAID, and other donors or ministries. Some made requests to multiple organizations. Affirming that many of these communities were under-served, on average they had waited five years for a response. The role of this advocacy was further confirmed by comments during the community visits such as a current mayor giving credit to his predecessor for bringing the water to the community, or parents lauding the school principal for her activism in bringing the improvements to the school. The interviews did not provide a complete picture about how this advocacy works. In some cases, particularly in Gaza, past experience with ANERA was an acknowledged factor. In others, leaders said they contacted ANERA because they heard through neighboring villages that ANERA had funds for particular types of infrastructure. In some, leaders seem to have simply made wide inquiries across numerous NGOs and donors. Community Outreach In interviews at ANERA, the relationship with the community was described as a basic operating principle. The model for community outreach was developed as a part of the ANERA response to the PCID RFA. In 2016, steps were taken to strengthen this aspect of the program through a reorganization 13 According to USAID, availability of funds depends not only on obligations from USAID but also the implementation process. The intention is that at any given time, all funds are accounted for, but are not necessarily reflected in the pipeline. For example, the contracting process and commitment of funds begins when the funds are obligated, but contractors are paid only after the satisfactory completion of their work. Some projects, which have funds committed to them are delayed in implementation because of difficulties in securing licenses. particularly in Area C, or in approval and delivery of dual-use materials to Gaza through the GRM system (Gaza Reconstruction Mechanism.) Further, funds obligated for water and sanitation projects cannot be transferred to other social projects; funds for Gaza cannot be spent in the West Bank. Further, as a Cooperative Agreement, PCID is able to absorb excess end-of-year funds from other USAID programs. 30 of the PCID management structure from a single Deputy Chief of Party (DCOP) position to two parallel DCOP positions, one for technical (engineering) matters and the other for non-technical programming. The PCID community outreach model has been applied across all communities and types of projects except in Gaza (due to the no-contact policy.) While potential projects were identified through the external channels described above, the implementation process was initiated and developed through and in collaboration with the village council or municipality. Community outreach workers and gender outreach workers (until July 2017) are located in the ANERA field offices and work in all communities and projects.14 The following description of the outreach model is based on interviews at ANERA and key informant interviews and group discussions in the communities. A series of community meetings are held at the beginning of the project. All meetings were arranged through and with the village council (or municipality) by the outreach worker. The initial meeting in the community was with the village council as a part of the ANERA assessment of the proposed project.15 After the decision to move forward with the project, the outreach workers with the village council sets up a meeting with broad attendance to present the design concept and elicit community feedback. According to TAM and ANERA interviews, an explicit effort was made to encourage attendance by women and other under-represented groups.16 The outreach workers and other ANERA and TAM non-engineering staff also discussed their work with the engineers to help them present the project to the community in non-technical terms and to use three-dimensional models rather than engineering drawings. When questioned about these meetings in the group discussions, in 14 of the sampled communities (and NGOs) participants voiced a sense that ANERA was open to their feedback at least to some extent, but there was clearly no consensus about whether this feedback resulted in any change in the design. In interviews with community leaders, all confirmed that ANERA held several meetings with the community during the course of the project but a majority (13 of 22) said that they had little role/input in either the design or the construction. The community outreach and gender workers from the field offices visit each project more or less on a weekly basis during the implementation process, traveling with the engineers responsible for oversight and quality control. They assist in gathering survey information about the needs for and use of the services, identify individuals with special needs (e.g., persons with disabilities, elderly, women heading households), and meet with community-based organizations and others in the community to listen to their concerns, and communicate them to the engineers. At the end of the project, the outreach workers organize a kick-off event and hold meetings and workshops to raise awareness about the proper use of the service and responsibilities for maintenance and sustainability. These sessions also involved the distribution of the awareness-raising materials designed by TAM. According to the non-technical staff in ANERA and TAM, follow-up meetings and visits in the community often continue for up to a year after project completion (during the guarantee 14 Again, the exception is Gaza, where there were no community outreach activities or outreach/review related to gender. Also, while TAM had provided training on gender sensitivity for ANERA staff in the West Bank, no training was delivered in Gaza. 15 In interviews with community leaders, 86% describe the purpose of the first meeting as an assessment of community need or a discussion of the concept of the project. From ANERA’s point of view, the purpose of the meeting was to assess both need and community interest and commitment. During this visit ANERA engineers also visited the project site for a technical assessment of feasibility and viability. 16 See the TAM final report for numbers showing the diversity of participation in these meetings. 31 period) to collect feedback on problems or issues, to continue with awareness-raising meetings, and to collect outcome data about utilization and affordability. Returning to the central question on the extent to which the PCID program succeeded in identifying and addressing community infrastructure needs in various sectors, as discussed under EQ 1, the qualitative assessment of the program results at the community level by the leaders and the residents, as well as their description of their experience with ANERA during the project process, show a high level of satisfaction with the results and positive statements about the responsiveness of the program to the needs of the community. When asked how they would rate the importance of the project to the community, there was unanimous agreement across the group discussions that the project was important (as compared to somewhat or not important). When asked why it was important, participants focused on the fact that the project resolved a critical issue in the community, and secondarily on its potential to contribute to social and economic development.17 Further, the PWA, CMWU, and the MOEHE reported that, from their point of view, the project identification process has worked well. In interviews, the leaders and beneficiaries also reported that the projects matched the community priorities. The part of the process that was most often cited as problematic or unsatisfactory was lack of provision of services to the parts of the communities that are in Area C. In four of the 14 communities in the evaluation sample in the West Bank, the lack of coverage to the Area C part of the community was cited explicitly by community leaders as a shortcoming. One of the selection criteria is that the project is primarily in Areas A or B, and PWA also mentioned this as a requirement for recommended water projects under PCID. In a situation like the Walajeh health clinic, when PCID and USAID were successful in completing a project in Area C, USAID was praised by the recipients as the donor who could best obtain the necessary permits from Israel. PCID AS A MECHANISM TO ENHANCE THE IMPACT OF OTHER USAID PROGRAMS PCID is unique among the USAID/WBG infrastructure programs in several ways. It is currently the only infrastructure program operated as a cooperative agreement (CA),18 and the only infrastructure program focused on small and medium projects. As a cooperative agreement, the PCID has an in-house implementation framework (i.e., management, design team, field oversight, fixed overhead costs, etc), 17 The specific question was: How would you describe the importance of this project for the community? (probe in terms of well-being, social and economic development, did it resolve a critical issue?) 18 Globally, USAID has mandated that infrastructure activities should be financed through contracts rather than grants because USAID has direct control over all aspects of implementation, including costs and quality control. USAID/WBG has a special waiver to also use cooperative agreements, grants with substantial oversight by USAID. Nonetheless, the CA format increases the risk for USAID 32 which allows for rapid response and mobilization, and lower marginal costs as new projects are added. For this reason, the CA is particularly suited to small and medium community-level projects. The PCID results framework anticipated that the PCID would collaborate with the Mission education, youth, health, private sector development, and democracy and governance offices to implement small and medium scale infrastructure projects in support of these programs. These projects therefore came to PCID through a separate channel. They were not subjected to the PCID program selection criteria but rather the selection criteria of the other USAID programs and they were given priority in implementation. Four projects of this type were presented, building the Youth Development Resource Center (YDRC) in Jericho, the health clinic in Walajeh, the restoration of the old market in Bethlehem, and a school construction project in Area C, that was ultimately dropped because of denial of permits. Two of these projects, the youth center and the clinic, were included in the evaluation sample of communities. The data collected in these communities through key informant interviews and the survey of beneficiaries, which focuses on the PCID indicators of utilization and response to community needs, paints a picture of distinct approaches to the community in terms of needs and outreach/involvement. In addition, they were not designed to be assessed by the same indicators and results as the PCID projects. This disjuncture was reflected in tensions and delays reported in key informant interviews with program implementers and USAID, although in both cases all parties praised the final building itself. Figure 4 is 5Reconstruction of Zeita Recreational Park 33 drawn from the survey of beneficiary households illustrate these differences.19 In addition, 80% of survey respondents in communities with ANERA-selected projects reported that these projects met an urgent need for their communities, while 46% of survey respondents in communities with USAID-selected projects reported that their communities had higher priority needs that should have come first. Figure 3: Survey Respondents rated ANERA-selected projects as Important to their Communities Based on key informant interviews, the village council in Walajeh was active in soliciting the clinic (i.e., it was cited as a community priority and the council reached out to ANERA, USAID, and other potential donors). In Jericho, the role of the municipality was limited to approval of the construction design. By all accounts (implementers, community, and clinic and club administrators), the level of utilization in both sites is low and administrators discussed steps being taken to increase their clientele. Likewise, based on the survey of beneficiaries, the level of satisfaction with these services is relatively low compared to other projects. (See the table of satisfaction scores by project in EQ 1). CONCLUSIONS • PCID has responded to basic community needs that are recognized and articulated as priorities by community leaders and citizens in the selected communities. The PCID method for identification and selection also has been successful in reaching under-served communities. On 19 In these graphs, the USAID projects are the Walajeh clinic and the Jericho youth club. The ANERA projects are the other 16 communities in the sample, including both West Bank and Gaza. In the survey, the average findings for the youth club may be biased. Because the club serves a wide area, not necessarily including residents of the immediate neighborhood, the enumerator attempted to identify potential or actual clients to respond to the survey. 87.8% 59.4% 9.8% 36.2% 1.1% 4.3% 1.3% 0.0% ANERA-selected projects USAID-selected projects Not Important Somewhat Unimportant Somewhat Important Important 34 the other hand, the evaluation methodology does not provide any basis for assessing whether these particular communities are more or less needy than others, or whether the selection process somehow gives some communities a better chance of benefitting than others. • The community outreach model has been effective in engaging community leaders and some residents in the project process. Interviews and discussions in the communities showed an understanding of the process and its relevance to them, and the openness of the discussions and opinions, both positive and negative, reflected this engagement. • The expression of satisfaction, utility, effectiveness in resolving a critical issue, involvement, and ownership (EQ 3) on one level are testimony for the effectiveness of the community outreach model. At the same time, because the evaluation is looking only at this model, without a comparison to other approaches, no assessment can be made of the value of the various components of the model or of the results of this model per se. • The projects brought to PCID through other USAID programs derive their priorities and design through a separate process of identification and engagement based in the other program. In some cases, the ANERA community outreach model, which is geared toward projects associated with basic needs, may not apply and may be antithetical to the USAID program approach. Both the YDRC building and the Bethlehem market illustrate this point, In these cases, seeking to apply the PCID community outreach model may result in delays, and tension between the PCID and the other program. Further, the expected results and indicators may be different. EVALUATION QUESTION 3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? FINDINGS Utilization Building on the findings presented in Evaluation Question 1, the level of utilization of the projects overall is strong, especially for the water and sanitation projects and the schools. (ANERA does not track utilization quantitatively for other types of projects.) Reports from the sample of communities confirm this general pattern. • Among the community and NGO leaders interviewed, respondents in only three communities said that, in their opinion, “people are not using the project services as much as they could.” As noted above, this assessment applies to the youth center in Jericho, the clinic in Walajeh, and to the storm water drainage system in Nahalin. 20 20 In the evaluation sample of communities, Nahalin is a singular case in which the village council has voiced a strong level of dissatisfaction with implementation of the project. Observation at the site confirmed that the system is not draining effectively. Various explanations were given by ANERA, USAID, and the village council about what went wrong in terms of design and implementation problems. The evaluators have no basis on which to assess the validity of the arguments. A second important caveat is that the assessment of utilization for the NGO projects in Gaza (YMCA, Sport Club, and the Rehabilitation Center, came from the leadership of the NGOs rather than the community. 35 • Participants in group discussions also felt that project services were being fully used (although there were no group interviews at the youth club or the clinic.) • Among survey respondents, 82% said, “yes,” when asked, “Do you or anyone in your household use the service provided by the project?” The percentage was lower in Gaza (59%) than in the West Bank (88%), however, presumably as a reflection of the NGO projects that have a limited service group.21 Increased access and lower cost may not necessarily translate into utilization, particularly for marginalized groups for whom the cost of connecting to the water or waste water network, may be prohibitive. According to the program implementers, a part of the function of the gender outreach workers was to identify elderly, persons with disabilities, or women heading households and arrange for the village council, or the PCID program to cover these costs. In some communities, the local government assumed this cost for all households. The PCID program design also supported utilization through an allowance to provide basic furniture and equipment for schools and other structures (e.g., the women’s center at Dar Salah), as necessary. Likewise, the MOUs with the PWA and the MOEHE included agreement that these agencies would provide furnishings, supplies and staff to the structures to ensure continued availability of the intended services. Maintenance and Sustainability The final step for ANERA when the construction is complete is to organize various events to raise awareness about proper use of the resources/services, and personal/community responsibility for caring for and maintaining the infrastructure. In addition, workshops are held to train the local community about maintenance and repairs. This awareness-raising was primarily the responsibility of community and gender outreach workers. TAM in particular prepared numerous tools, brochures, posters, and billboards that were distributed and displayed at all the project sites. At the completion of each project, an event is held for the community to celebrate the new project, to convey the awareness messages, and to distribute the educational materials. These events were frequently cited in the interviews, group discussions, and surveys as memorable both in terms of raising awareness about the project but also as events in and of themselves. Over time, these events became more specialized, based on the type of project and the needs of the community. At the opening of the clinic, for example, ANERA and TAM organized a health fair whereby nurses from a nearby training centers volunteered to provide blood pressure checks, physical exams for the children, and training sessions. The opening of a new playground included basic training on first aid and accident prevention, and the opening of the school was a fun day with games, coloring books, and videos. For the evaluation, each community visit included a site inspection to observe the state of repair and maintenance of the project, utilization, and presence of awareness materials. An observation check list was completed for each site (n=22). • In 20 of the 22 projects included in the sample, observers reported that the project is in good shape (e.g., clean, new). Problems were observed in two cases (i.e., a storm water drainage project not draining as initially designed; wastewater system in need of maintenance to correct 21 It also is important to repeat the caution mentioned above that the utilization rate for the youth club may not be comparable to the other projects because the survey enumerator made an effort to find people to interview who are familiar with the center. 36 an emerging problem with clogging. Three other projects reported problems that require on￾going maintenance (e.g., a faulty float in a reservoir, damage in some joints in the network pipes, and problems due to settling in a school patio). • Observers noted the presence of awareness billboards or posters in all locations, at a minimum the billboard announcing the date of construction and the support of USAID and ANERA. In the schools, additional posters were observed reminding students that they have a responsibility to care for their school, etc. The interviews with community leaders (n=21) included questions about the responsibility and availability of resources for maintenance. • “Who or what institution is responsible for maintenance?” The majority (86%) said the village council/municipality is responsible, while some also mentioned contractors (during the first year), and leaders associated with NGO projects said the NGOs are responsible. • When asked about the resources for maintenance, respondents said they have human and technical resources, but face gaps in terms of financial resources and machinery. • In many communities, specific individuals identified themselves or were introduced as the people carrying out or supervising the maintenance. Since these projects are relatively new, in most cases maintenance has not been a major problem so far. The gaps in resources may prove to be a factor in the future as the projects age. This possibility is illustrated by the case of a waste water network, which was completed in 2014, and is now facing the problem that many of the lines are becoming clogged and need to be cleaned and re-opened. At this point they are seeking assistance from the municipality to train workers to do these repairs but they also need to rent the truck (mobile unit) used in the cleaning operation, which is expensive. They cannot afford to purchase a truck and in addition to the cost, the truck is not always available and in good repair when needed. The community leaders also note a continuous and on-going need to raise awareness in the community about proper use and management of the system to reduce the costly repairs. CONCLUSIONS • In congruence with the qualitative evidence of the priority attached to the projects and satisfaction with the results, the community responses from both residents and leaders indicate high level of utilization for the core ANERA projects. • The communities and leaders recognize and accept responsibility for maintenance of the completed projects. The awareness-raising activities included in the community outreach model contribute to the sense of responsibility and stewardship for the project. Given that the projects are relatively new, observation of the condition of completed projects in the sample communities indicates on-going care and maintenance in terms of cleanliness and condition, as well as identification of the individuals managing these tasks. • Community leaders recognize potential constraints on the ability of the village to provide maintenance in the future as the projects age, in terms of lack of finances and of access to machinery. In discussions of maintenance, leaders did not mention or anticipate future support from PWA or other government bodies. 37 EVALUATION QUESTION 4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? According to the evaluation scope of work, the intent of this question is to illicit feedback from communities on potential opportunities for future support in water and sanitation and community infrastructure for USAID. Responses had no bearing on the overall performance of the PCID. The question was posed as an open-ended question to: • community leaders/KIIs (n=22) (What do you see as the most urgent need in the community at this time?); • discussion groups/group meetings (n=16) (What are the most pressing needs now in terms of infrastructure and services?); • and, in the survey of beneficiary households (n=619) (In your opinion, what are other pressing Infrastructure projects?). The responses subsequently were coded into uniform categories. The following graph shows the priority given to various types of projects across the three categories of respondents, and their priority, based on the number of responses. (In all cases, multiple responses were permitted.) Figure 4: Other Priority Community Needs Identified by Evaluation Respondents 196 134 99 78 69 43 17 12 11 11 10 8 8 6 2 Electricity/energy infrastructure Health care services/infrastructure Sewage and sanitation Electricity infrastructure Education services (Schools/additional rooms) Public spaces (parks, gardens, playgrounds) Water for domestic purposes Drainage systems Rehabilitation centers for the disabled Transport (e.g., Roads, streets, transportation) Care facilities for the elderly Improved accessibility of public buildings Sports facilities Water for agriculture There are no other projects that should have come first. Survey KIIs Group discussion 38 CONCLUSIONS: • Both leaders and residents identify clear infrastructure needs at the community level, suggesting that residents see the community as a focal point for projects to improve their daily lives. • The needs identified are consistent with those included in the PCID programs and in most cases, are basic needs related to daily lives. • Broadly, community leaders and residents agree on the needs and priorities. • Water projects appear to have a lower priority than expected, based on previous experience, the observations of implementers, and the high levels of satisfaction with water projects under PCID. This result may reflect the fact that many of the communities included in this sample have already benefitted from water projects. Sewage/sanitation remains a key concern.22 V. EVALUATION CONCLUSION The overall picture emerging from the evaluation, through the analysis associated with each of the evaluation questions is that the PCID program has been effective in addressing basic needs for infrastructure in Palestinian communities. In water and sanitation, the program has operated at the interface between the PWA and the village councils, channeling resources to small, marginal, and under￾served communities, and reinforcing the local advocacy and outreach efforts. The model also has been effective in engaging the community in the process, and in building a sense of achievement and progress. The core ANERA projects are valued by the beneficiaries and have had a positive effect on their day-to￾day lives. The consistency and agreement seen in the list of urgent community needs for the future (EQ 4) clearly demonstrate a demand in the communities for a continuation of these types of community￾level infrastructure activities. At the same time, given the unique circumstances in the Palestinian territories and the absence of baseline data, realistic indicators, and comparable activities, the evaluation has no basis on which to judge the objective, relative validity of this model of community engagement or its broader utility. It also is not possible to assess the relative value of the various components of the model. For example, clearly there is a value in terms of community engagement in arriving with a potential project in hand, and a value in focusing on basic needs, but the evaluation has no point of comparison to say whether these are make or break components. The evaluation also cannot address the question of whether the community/project selection process was effective in addressing broad social and economic development needs at the community level, or filled gaps that would not otherwise have been addressed. 22 Few sanitation projects were implemented under PCID because a community -level sanitation project was only viable if it could be linked to a waste treatment facility. Waste treatment facilities are too costly to fit into the PCID small and medium infrastructure cost parameters. 39 VI. LESSONS LEARNED AND RECOMMENDATIONS 1. Lesson Learned: Outcome indicators and realistic targets are necessary to evaluate objectively and quantitatively whether the program produced the expected results and contributed to the development objectives. Should the reach have been broader? Were the achievements in line with the funds expended? Did the availability and adequacy of community services and quality of life improve? Most infrastructure projects are fixed-price contracts judged by standards and outcome indicators to measure quality, timeliness and budget against. As PCID is a cooperative agreement with social objectives and results, these output indicators, which are the type currently measured by the project tracking system, give an incomplete picture of the expected outcomes. Recommendation: In the future, performance indicators should be designed to link to the decision process and the expected results so that they can be used to guide management decisions and measure the expected outcomes. For example, the Performance Management Plan should include measures of access or utilization, with targeted levels of change, for other services besides water, such as school enrollment/attendance, or frequency of visits to the clinic. Performance indicators should be defined as a part of project implementation and not pre￾determined in the RFA. (Request for Application). In addition, in line with USAID policy, indicators and targets should be reviewed and revised annually, both to identify whether the program is performing as expected and to project realistic expectations about the achievements for the coming year. For example, if the actual performance is either far below or far above the target, the reasons for the discrepancy should be examined and explained, and the target for the following period should be revised to match realistic performance expectations. 2. Lesson Learned: In terms of achievement of results, in both quantitative and qualitative terms, the weakest link in the entire project process, based on commentary from community leaders, project design partners, and ANERA construction engineers, is the selection, performance, and supervision of contractors for project construction. ANERA acknowledges the presence of these problems and has built procedures into their operations (including some aspects of the first three items in the recommendation below) to counter these problems. Interviews in some communities and with other key informants, however, gave evidence that enforcement is uneven and the problems persist. Recommendation: Four main steps should be taken to counter the weaknesses observed in the implementation process. (1) Unified and clear specifications, standards and guidelines for materials and items used in the project construction should be included in the project bid documents, ensuring that the contractor cannot make changes without ANERA’s concurrence. Attention should be given to availability of specified materials in the local market. (2) Undertake a critical examination of the process for selection of contractors and the relative weight given to cost as a decision factor relative to experience and quality of work. (3) Review the adequacy of the quality control procedures and personnel in the field, including the experience and deportment of the engineers contracted by ANERA to provide full-time oversight of the construction process. (4) Finally, given the extent of USAID/WBG investment in infrastructure and construction, USAID may consider investment in training and information dissemination to strengthen the skills and knowledge base of contractors about products and technology that may affect design and implementation errors throughout the process. 40 3. Lesson Learned: The PCID focus on responding to community priority needs through small￾and medium-scale projects, in practice, tends to narrow the scope of the community needs assessment and the technical feasibility study to the particular “small” problem, and thereby limit attention to factors in the broader social and physical environment that may affect future activities. For example, in two communities, evaluators noted inadequate coordination with other projects (sanitation network), little consideration of next steps (drainage projects), or the contingencies presented by Area C boundaries. Recommendation: The feasibility study for the design concept should be broad enough to map the place of this small project within the complexities of the community social and environmental context, particularly in considering links and relationships to other current and future projects in the community. In addition to the standard studies that are required for USAID project approval, the engineers also should be required to demonstrate that they have consulted with the village/municipal council about of planned projects, and with USAID and other donors, and PA ministries working in the community. They also should present an analysis of the costs/benefits of omitting parts of the community in area C, and this analysis should be discussed and understood by the village/municipal council. 4. Lesson Learned: A key issue for the PCID evaluation has been the capacity and experience of the PCID program to serve as a flexible and rapid response mechanism for implementation of small- and medium-scale infrastructure in support of other USAID programs. On paper, the connection between the two programs seems straight forward since the PCID has a structure in place for rapid response to community infrastructure needs; in practice, the collaboration has not been smooth, due primarily to differences in the terms for engagement with the community and measures of achievement. At the same time, the resulting buildings have been successful and praised by both parties. Recommendation: The successful completion of these projects suggests that the utilization of the PCID program as a mechanism for other USAID programs is feasible and realistic. The expectations, terms, and responsibilities for these projects should be negotiated, understood, and agreed before the project process begins. For example, topics to be resolved include the extent to which the design concept and components are complete, the role of the other USAID program in oversight and control during the project design phase, selection of contractor, and construction, the extent of village/municipal council participation, input, and approval, and the role/control of the other USAID program in handoff and approval. 5. Lesson Learned: While the evaluation documented an acceptance of responsibility for maintenance on the part of the village councils and municipalities, carrying out that responsibility in the future, as equipment ages, may be a problem because of lack of financial and mechanical resources. The gap is likely to be a problem particularly in the villages (compared to municipalities), because they do not have a revenue source or paid staff to handle these tasks. The PCID program does not have a maintenance component. At the end of the one-year guarantee period after completion, both the implementers and the contractors are separated from the project. 41 Recommendation: Additional attention to arrangements and sources of finance for maintenance should be built into the PCID project process. Several approaches to this issue have been suggested. First, simplify the design and materials. For example, in water projects, maintenance problems are most likely to occur in pumps rather than in the network itself. While elevators are important for persons with disabilities and elderly, the lack of parts, technical expertise, and reliable electricity contribute to constant problems in utilization and functionality. Second, ensure that contractors provide a manual for operation and maintenance of the project before closure, including on-site training for key village personnel. Third, require a maintenance plan as part of the close-out of each project including institutional responsibility and source of finance. 42 VII. LIST OF ANNEXES ANNEX A: Scope of Work ANNEX B: Detailed Description of Evaluation Methods ANNEX C: Data Collection Tools ANNEX D: List of Documents Reviewed ANNEX E: List of Stakeholders Interviewed ANNEX F: Evaluation Matrix ANNEX G: Selected Background Documents ANNEX H: Additional Survey Tables ANNEX I: Statements of Differences ANNEX J: Signed Disclosure of Conflict of Interest ANNEX K: Summary Information on Evaluation Team 43 ANNEX A: SCOPE OF WORK EXPANDED SCOPE OF WORK AND EVALUATION DESIGN Performance Evaluation of the Palestinian Community Infrastructure Development (PCID) Program INFORMATION AND BACKGROUND Name Palestinian Community Infrastructure Development Program Implementing Partner ANERA Cooperative Agreement No 294-A-13-00005-00 Budget Total Program Value: US$ 115,000,000 USAID Contribution: US$ 100,000,000 Cost Share (Non-Federal): US$ 15,000,00 Duration 5 years (March 6, 2013 – March 5, 2019) Table 1: Basic Project Details Within the framework of USAID’s mission to end extreme poverty and promote resilient, democratic societies, the Economic Growth and Infrastructure Office within USAID’s Mission in the West Bank and Gaza generates tangible benefits in the lives of Palestinians by increasing access to clean, reliable and affordable water and energy and stimulating the growth of the Palestinian economy. Since 1994, USAID has been one of the largest donors in the infrastructure sector in the West Bank and Gaza, with projects that include upgrading water and sanitation systems and constructing and rehabilitating roads, schools, clinics, and other community infrastructure. USAID investments in the water sector aim to increase access to water supplies and sanitation systems and facilitate the sustainable use of limited resources. Activities include rehabilitating wells, installing pipelines, constructing reservoirs, upgrading and expanding household networks, and expanding small￾scale desalination plants. These efforts lead to improvements in the safety, quality, and reliability of water services for 1.4 million Palestinians. There is a pressing need in the West Bank and Gaza to connect rural, vulnerable or under-served communities without piped water systems to existing bulk water supply systems and to rehabilitate existing deteriorated water distribution networks. New connections dramatically improve household budgets in these communities, improve health and well-being, and contribute to the achievement of Palestinian Authority goals. Water loss reduction through improved infrastructure has the potential to save as much as 10 billion gallons per year and help address the growing water supply/demand deficit. Additionally, in the wastewater sector, few communities have proper sewage collection systems, with most of the sewage produced being discharged untreated directly into the environment. New collection networks help eliminate household reliance on unregulated cesspits and leaky septic tanks and will facilitate future connection to planned wastewater treatment plants. 44 The PCID program was conceived in 2013 to pressing needs in the community water and sanitation infrastructure, thereby supporting USAID/West Bank and Gaza (USAID/WBG) infrastructure strategy at the time. This strategy sought to improve social and economic development through increased access to new and improved public infrastructure facilities and services. The program’s development hypothesis is grounded in the idea that USAID can improve social and economic development through increased access to new and improved public infrastructure facilities and services. The PCID program is one of several programs linked to this strategy, including large-scale as well as village-scale activities. PCID was specifically designed to increase access to water and sanitation, and other public services at the community level through small- and medium-scale community infrastructure projects, with particular attention to rural, vulnerable or under-served communities, as well as complement other USAID activities in selected locations. PCID will terminate in March 2019, at which point it will have invested US$115,000,000 in infrastructure projects in water and sanitation, education, health, and other social sectors in the West Bank and Gaza. PCID is focused on increasing Palestinian access to water, sanitation and other small and medium scale community infrastructure in the West Bank and Gaza Strip through increased availability and affordability of water and sanitation, as well as increasing access to community infrastructure facilities. To achieve these results, the program is structured to address the following two intermediate results and four objectives (see also detailed results framework below): IR 1: Availability and affordability of water and sanitation increased 1) Increase access to water and sanitation systems and other public infrastructure needs in the West Bank and Gaza Strip. IR 2: Access to community infrastructure facilities increased 2) Contribute to increasing the impact of USAID funded programs in health, democracy and governance, education and private sector needs in the West Bank and Gaza Strip. 3) Enhance access to health, schools, youth centers and community infrastructure facilities in rural, urban, vulnerable and under-served communities in the West Bank and Gaza Strip. 4) Enhance the lives of Palestinians’ wellbeing by responding to unforeseen infrastructure and other urgent needs in the West Bank and Gaza Strip. The PCID Program receives lists of projects from three main sources: PA Line Ministries (i.e. the Palestinian Water Authority (PWA) and Coastal Municipality Water Utility (CMWU) in Gaza), USAID inter-offices (where requested projects complement other, USAID-funded program activities), and finally, through ANERA’s interactions with communities and local organizations/own reconnaissance (No-contact policy and community clearances are strictly observed and adhered to). Once received, the projects go through a rigorous assessment (including, against pre-determined selection criteria23) to ensure that they fit the Program’s scope of work, objectives and requirements. PCID also includes a community outreach component used to promote ownership, where the project team works with communities to get feedback on designs for identified projects, and after implementation to discuss maintenance and sustainability. Results to Date 23 The project selection criteria are found in a document titled: PCID Project Selection Criteria 45 The majority of the project’s work falls under IR1. To-date PCID has completed 39 projects in the water, sanitation, education, health and community infrastructure sectors, impacting nearly 339,452 people in 33 marginalized communities in the West Bank and Gaza. These projects created nearly 170,725 person-days of employment (equivalent to 8,536 short-term jobs). An additional 13 projects are ongoing, while another five projects are under the process of design and/or approval. The majority (26, or two thirds) of completed infrastructure projects are in the West Bank, and most are in the water and sanitation sector. Completed education infrastructure projects (7) represent about 18% of all completed projects, and all are in the West Bank. Road projects (4) represent about 10% of all completed projects, and they are mostly (75%) in the Gaza Strip. Other infrastructure projects completed by PCID include a health clinic, a rehabilitation center, a youth center, a sports playground, three public spaces (parks and playgrounds), and a municipal marketplace. The regional distribution of completed projects is shown in Table 1 below. Table 2: Distribution of completed projects by region and sector Sewage Network Reservoir Drainage Network North 10 10 1 3 2 0 3 1 0 0 0 0 Middle 5 6 0 2 1 0 2 0 1 0 0 0 South 11 12 0 0 2 5 2 1 0 1 1 0 WB 26 28 1 5 5 5 7 2 1 1 1 0 GS All 7 11 0 0 0 4 0 1 1 1 3 1 WBGS Total 33 39 1.0 5.0 5.0 9.0 7.0 3.0 2.0 2.0 4.0 1.0 No. Communities Roads Markets WB Region Sub￾Region No. of projects Disribution of Projects Water & Sanitation Education Public Spaces Youth Centers and Facilities Health 46 Figure 1: PCID Results Framework EVALUATION PURPOSE AND AUDIENCE The purpose of this performance evaluation is to help USAID/WBG assess project performance as well as highlight the successes and challenges of the PCID programmatic approach and implementation in terms of achieving community-level results. A second purpose is to inform future directions and possible new designs in this area of USAID/WBG support going forward. The primary audience for this evaluation will be USAID/WBG (all offices), especially the Economic Growth and Infrastructure (EGI) office as well as the implementing partner, ANERA (American Near East Refugee Aid). Secondary audiences will be other stakeholders in the infrastructure, health, education and governance sectors and potentially USAID on a more global level. EVALUATION QUESTIONS 47 USAID identified four key evaluation questions (EQs) to frame the evaluation, and to serve as the basis for identifying achievements, challenges, lessons learned and recommendations. The evaluation team’s approach to addressing these questions is outlined in the four evaluation matrices included in Annex B. Each matrix identifies the sub-questions and topics to be addressed in the response to the evaluation question and the types and sources of data used to support the response. Importantly, the matrices are based on the team’s initial review of program documents, and may be refined on the basis of the discussions with USAID in the kick-off meeting (see below). The detailed Methodology (Section V) explains the team’s approach to acquiring and analyzing this data, and the process of triangulation through which the evidence from multiple sources is used to corroborate findings and conclusions. The matrix also identifies the data collection tools to be used in implementing the methodology. The draft tools are found in Annex C. EQ1: To what extent did the implementation of PCID activities (infrastructure projects) support the project development hypothesis and intended results? There are two aspects to this question: the extent to which the program achieved the expected results and targets; and, ii) in terms of the development hypothesis, the extent to which the results contributed to social and economic development. The evaluation team will address this question first through a quantitative mapping of the expected results and targets to the actual outcomes over the five years of the program, utilizing PCID program documents, the GEO-MIS database, and the performance indicators. Particular attention will be given to patterns and changes over time, to identify areas of success and challenge in achievement of results as well as differences in the outcomes of the approach for different types of projects and communities. The team also will draw on key informant interviews with the implementers and other stakeholder counterpart organizations in interpreting the results. In examining the second part of this question, the team also will engage leaders and residents in a sample of the communities with PCID projects to understand the qualitative as well as the quantitative results. The interviews and survey in the communities will address questions such as how access and use of community infrastructure changed due to PCID activities and the value they attach to these changes, the importance of the employment opportunities created by the PCID, and their assessment of the project’s contribution to any other social and economic benefits in the community, now and in the future. EQ2: To what extent did the project succeed in identifying and addressing community infrastructure needs in various sectors? This question will draw on findings and conclusions from Question 1, including the assessments identified in the community visits, to consider the framework PCID used to select projects, assessing both inputs and final decisions made on community selection, project selection and implementation. The team will assess the project methodology for selecting targeted communities; examine the effectiveness in identifying communities, and trends in geographic distribution. In particular, the evaluation team will assess the project’s performance in addressing the needs of municipalities, village councils, rural areas, and communities with a large percent of their lands in Area C. To the extent possible, the team also will seek feedback from community leaders and counterpart organizations in assessing the alignment of the projects with local and national development plans. 48 The team will build their data collection and analysis around the documented selection criteria provided by the PCID team to determine if the model for selection was efficient and flexible enough to provide community level results. Finally, the team will also look at the potential added value of the project’s modality for the Mission as a whole, looking in particular at its community outreach model, and its ability to address other USAID Office needs. To do this the team will use the lens of the USAID/WBG priorities under the EGI office to improve social and economic development through increased access to new and improved public infrastructure facilities and services as well as lower level objectives. The team will identify successes and challenges for PCID in supporting this goal and provide some comparison to other USAID infrastructure projects. EQ3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? The team will assess the successes and challenges of the community outreach model in supporting recipient institutions to utilize and maintain completed infrastructure projects. This assessment will draw on several types of data that together, will examine the extent of utilization and maintenance for the PCID projects and the role of the PCID community outreach activities in explaining these results. During the evaluation team’s community visits, project sites will be observed to assess their condition and utilization. During these site visits, the team will note evidence of PCID’s outreach activities such as posters or community meetings. The visits will allow some assessment of maintenance and utilization over time since the sample includes projects completed several years ago, as well as recent projects. In the communities, the team also will discuss maintenance and utilization with the community leaders and beneficiaries, as a part of the key informant interviews. This is to gauge both the level of involvement in maintenance, and the sense of ownership and commitment to sustainability. The team also will interview the PCID implementers involved in community outreach to understand their approach and methods, and particularly, changes in this approach over time and lessons learned. EQ4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? The evaluation team will include questions in the various data collection tools that are designed to illicit feedback from communities on what potential opportunities exist for further support in community infrastructure, water and sanitation by USAID. The identification and documentation of unmet needs for this evaluation question will not be linked to PCID performance but is rather, to inform USAID’s future programming. Recommendations and lessons learned will stem from the evaluation team’s work in addressing the above questions and will address how other USAID offices could address remaining gaps, challenges and any unmet needs identified as well as apply successful practices identified. The evaluation aims to provide recommendations and lessons learned in terms of future program focus or targeting (geographic, urban/rural, needs of local actors) and in consideration of other USAID/WBG interventions. MEMBERS OF THE EVALUATION TEAM The evaluation team has three evaluators and additional support personnel. Together, the three evaluators bring a range of experience and expertise required in a performance evaluation that draws on multiple sources and types of data. 49 The Team Leader has overall responsibility for the evaluation, including oversight of the development of the evaluation design and methodology, work plan and data collection tools, protocols, and processes, leadership of the analysis of data collected, and presentation of de-briefings for USAID and QED. She will be primarily responsible for undertaking the evaluation meetings and interviews with USAID, ANERA, PA counterparts, TAM, and engineering firms. She will also lead site visits and community-level meeting whenever possible. In addition, she will produce the draft and final evaluation report. The team leader will ensure integrity, quality control, conformance with the USAID Evaluation Policy, and the application of evaluation best practices. The Evaluation Manager will oversee the data collection and data processing activities. He will direct and implement all phases of the survey of beneficiary households in both the West Bank and Gaza. He also will support the team leader in analysis and report writing. The third evaluator, the Technical Specialist, will have a lead role in undertaking the community site visits in the West Bank, including key informant interviews, facilitated discussion groups, and site visits including specific technical input related to water and engineering works. Both the Evaluation Manager and the Technical Specialist are employed for this evaluation by the Palestinian Center for Public Opinion (PCPO), a local subcontractor, which also will provide logistical support, support personnel, and the survey team. PCPO, under the supervision of the Team Leader, will also be responsible for the data collection process in Gaza through a team of two consultants with demonstrated research and sub-matter expertise. The first of these consultants will oversee the day-to-day survey implementation in the Gaza communities sampled for this evaluation, and will report to the Evaluation Manager. The second consultant –a technical specialist- will be responsible for undertaking the site visits and primary data collection at the community level in Gaza. He will report directly to the Team Leader. EVALUATION DESIGN: METHODOLOGY The final performance evaluation for the PCIP will utilize a mixed-methods design to respond to the evaluation questions, with a reliance on evidence-based results, including both achievements and challenges. As an external, independent evaluation, it will be carried out impartially, rigorously, objectively, responsibly and transparently. Wherever possible, information from various sources will be triangulated with data from other informants and sources in order to confirm its accuracy. Sources of data and findings will be clearly documented. Findings and conclusions emerging from triangulated individual and group interviewees –including quotes- will be cited as evidence when appropriate, but these will not be attributed to specific individuals, to ensure anonymity. Both positive and negative findings will be presented. Where findings indicate weaknesses in the strategy or performance of PCID, these weaknesses will be identified clearly but constructively, with the intent of using the evaluation to enhance future USAID/WBG programming. Evaluation conclusions will be clearly linked to findings. And recommendations will be clearly linked to conclusions. The evaluation process will be consultative and participatory. It will seek to engage as wide a range of informants as possible at all levels, from beneficiaries to USAID and ANERA and the PA, through methods that not only extract information, but also stimulate debate and the exchange of ideas about past experience and future prospects. Overall, the team’s approach will be shaped by the need to build an interlocking, cross-checked set of information and ideas from as wide a range of informants as possible. 50 Types and Sources of Data Broadly, the evaluation will draw on three types of data, which will be used together to address the evaluation questions. 1) PCID background and program documents provide an overview of the activity from the signing of the cooperative agreement to the present. The team began with a systematic review of these documents, including the cooperative agreement, program monitoring reports, and records of modifications of program implementation and funding through 2017. The list of documents provided to the evaluation team by USAID is in Annex A. The focus of the initial review was to provide the basis for development of the design matrices for each Evaluation Question (Annex B), and the methodology and data collection tools. The review also focused on providing an initial mapping of potential evaluation stakeholders. The documents also will be included in the data analysis process, in the interpretation and assessment of findings from the field data collection. 2) Primary qualitative data will be collected from two primary sources: key informant and group interviews with PCID implementers, counterpart organizations, and other stakeholders, including USAID program offices; and, in a representative sample of communities in which PCID projects have been completed. Each community site visit will include key informant interviews and group discussions with community leaders, including the head of the local government unit, other people in the community involved directly with the project, and representatives of local community organizations. (The no￾contact policy and community clearances will be strictly observed and adhered to.) Evaluators also will visit the project sites to observe the condition of the projects and extent of utilization. The draft data collection tools, including key informant and group interview protocols and the guide for project site visits are attached in Annex C. The use of standard interview protocols across communities, types of projects, and organizations is the basis for drawing general findings and meaningful comparisons for the PCID as a whole. Table 3 illustrates the categorization of PCID’s internal and external stakeholders that the team has identified during the desk review. These stakeholder categories are also used to structure the team’s proposals for meetings during the evaluation mission. 51 Table 3: Stakeholders for the evaluation Stakeholders Interest in the evaluation Participation in the evaluation A. Internal (USAID) stakeholders USAID WBG Mission Primary stakeholder of this evaluation. Responsible for country level planning and operations implementation, it has a direct stake in the evaluation and will be a primary user of its results in the development of future interventions. USAID WBG Mission staff will be further involved in workshops/ feedback sessions at the beginning and end of the evaluation mission, and they will have an opportunity to comment on the draft evaluation report. Mission staff in the EGI Office will be primary key informants who will be interviewed and consulted by the evaluation. Other Offices with project portfolios that the PCID was intended to complement will also be interviewed. The team will seek information on program approaches, standards and success criteria from all units linked to main themes of the evaluation with a particular interest in improved reporting on results. B. External stakeholders B1. Palestinian Authority The PA spans several stakeholder categories (see below). Overall, it has a direct interest in knowing whether PCID’s activities in the West Bank and Gaza achieve the expected results; and furthermore the extent to which PCID has aligned with government priorities and what might be needed in the future to ensure greater alignment and contribution to national- and local-level plans and priorities. Various ministries are direct partners at the program level (reviewed in detail below). Dissemination strategy will ensure that evaluation findings are actively shared with key PA bodies, as directed by USAID. B2. Direct Partners Palestinian Water Authority (PWA) The PWA is an independent semi￾governmental entity that is responsible for regulating and overseeing the development of the water sector in the West Bank and Gaza Strip. PWA has been heavily involved in the implementation of PCID. Will be involved in the evaluation process through interviews and field visits. Coastal Municipalities Water Utility (CMWU) CMWU is responsible for providing the Gaza Strip ’residents with water and sanitation services. PCID coordinated closely with CMWU. Will be involved in the evaluation process through interviews and field visits. 52 Stakeholders Interest in the evaluation Participation in the evaluation Ministry of Local Government (MoLG) MoLG is the government entity responsible for overseeing and co￾ordination of development activities in the local government sector, and to which USAID contributes through PCID and other programs. Will be involved in the evaluation process through interviews and field visits. Ministry of Education Through its Directorates in the West Bank and Gaza, this is PCID’s government partner for all education sector infrastructure projects. Will be involved in the evaluation process through interviews and field visits. Ministry of Health Through its Directorates in the West Bank and Gaza, this is PCID’s government partner for all health sector infrastructure projects. Will be involved at in the evaluation process through interviews and field visits. ANERA USAID’s implementing partner the West Bank and Gaza. Will be involved at both policy and technical level in the evaluation process through interviews and feedback sessions. B3. Ultimate Beneficiaries Beneficiary Groups (residents of targeted communities) As the ultimate users of infrastructure projects, beneficiaries have a stake in PCID determining whether its assistance is appropriate, responsive and effective. . Community Based Organizations This group of stakeholders includes the organizations that have benefited from the infrastructure projects (such as YMCA in Gaza), as well as organizations in the targeted communities that are working towards community development objectives. Representatives of these organizations will be consulted during the field missions. 3) Quantitative data will be drawn from two sources, monitoring and indicator data provided by the implementing organizations, and a fixed-response, in-person survey of beneficiary households in the targeted communities. The draft survey questionnaire is in Annex C. All three types of information will used in the analysis and responses across the four Evaluation Questions, as detailed in the four evaluation design matrices included in Annex B. The matrix for each Evaluation Question identifies the sub-questions and topics pertinent to analysis for that question, as well as the type and source of the required data and the tools to be applied for data collection. 53 Data Analysis Plan The analysis in a mixed methods evaluation is an iterative process built around the Evaluation Questions and sub-questions. The analytic process involves drawing on evidence from the various data sources to articulate and corroborate findings relevant to these questions, with cross-checks across sources. The type of analysis will vary by type of data. The document review will stress content analysis. The quantitative data will use basic statistical tools, aggregating across communities in some cases and drawing comparisons among communities in others. The qualitative data will be coded and mapped by themes to identify (and document) patterns across communities and types of projects, and over time. Because it is an iterative process, the analysis is continuous throughout the data collection period as new pieces of evidence are brought to bear. In keeping of one of QED’s best practices in evaluation, the team will have dedicated time to do this throughout the evaluation instead of just a block of time at the end- (i.e. day six of every work week). The findings relative to the Evaluation Question or sub-question are used to establish conclusions, which in turn serve as the basis for extraction of lessons learned and recommendations. In the PCIP evaluation, particular attention will be given not only to the extent to which the program met expectations and targets, but also to trends over time in characteristics of the community projects, and to the patterns of interaction and involvement with the communities. Additional discussion of the analysis for the Evaluation Questions is included in the matrices. With the exception of the West Bank Technical Specialist who was previously employed by ANERA, all members of the evaluation team will be involved in the analysis through examination of the findings and statement of conclusions, and in the definition of the lessons learned and recommendations. Consensus among team members is required for the evaluation recommendations. Sample Selection - Communities Eighteen of the 39 communities where PCID projects have been completed between 2014 and February 2018, have been selected through a purposive sample, designed to be representative of the spectrum of types communities and infrastructure projects in the PCIP. The 18 communities include 22 of the 39 completed projects. The criteria used in selection are: - Type of project: water and sanitation, education, and social assistance, also recognizing sub￾types within each category - Size of project: cost - Location of the community: north, central, and south West Bank, and Gaza - Date of project completion. The number of communities was based on the need to collect enough cases for analysis of the different types of projects and beneficiaries, as well as by limitations of time and costs of data collection and processing. The list of communities and distribution by selected characteristics can be found in Annex D. Sample Selection - Household Survey A household-level survey in areas that are most likely affected by the completed projects in the 18 sampled communities will be undertaken. This survey will focus on capturing the satisfaction levels of 500 women and men in sampled communities with the projects, as well as gauging their level of involvement in the projects’ design and implementation. The survey questionnaire is included in Annex C, while the sample design and selection process is presented in Annex E. 54 Social Inclusion The PCID design emphasized social inclusion as an essential consideration in the community outreach model for identifying and prioritizing community infrastructure needs, and for assessing utilization and sustainability. The evaluation will address directly the extent to which the PCID achieved this inclusion through the community site visits. Specifically, evaluators will identify and interview community-based organizations for women and youth. The topic also will be addressed in the beneficiary survey, and in key informant interviews with implementers and community leaders. Additional evidence will be provided by the PCID program monitoring data, which includes disaggregated numbers on attendance at community meetings and other events, and by data on the gender balance within PCID staff and other, indirect, employment generated during program. Finally, in the conduct of the evaluation itself and application of the various tools, steps will be taken to ensure a balanced representation of women and youth among interviewees and in the selection of survey respondents. The overall analysis of performance will include findings and conclusions on the differential impacts of PCID on males and females. Methodological Limitations Lack of baseline measures The evaluation is limited in the ability to measure the effectiveness of the projects by the absence of baseline measures for the performance indicators and for the key variables of access to and utilization of services. In lieu of quantitative measures of these variables, the evaluation will include questions in the beneficiary survey and in the community interviews about their perceptions and memory of their situation before and after the project. In addition, to complement the findings from the beneficiaries, the evaluation team will pursue any quantitative measures of access to services that implementers may have used in project selection and design, and in post-construction follow-up, for the 22 projects included in the evaluation’s sample communities. Separate data collection in West Bank and Gaze communities A second potential limitation is the fact that community data will be collected by different teams in Gaza and the West Bank. Although both teams will use the same tools for data collection, it is possible that some systematic differences in the two sets of data. The team will be alert to this possibility in the data analysis and seek to assess the extent of any bias and take account of it in the analysis. For example, the evaluation team will have more direct contact with stakeholders in the West Bank than in Gaza, including counterpart organizations and community leaders. EVALUATION DESIGN: WORKPLAN The evaluation will be conducted in three phases: the inception period, data collection and analysis, and reporting. (See the preliminary calendar in Annex F) 1) Inception The inception period, which began on/about January 30, 2018, included document review and team discussions (via Skype) to reach a common understanding of the purpose and process of the evaluation. The document review at this stage focused on development of the detailed evaluation design and methodology, and the workplan for the expanded Statement of Work (Deliverable 1). The review of documents and of the PCID quantitative monitoring and indicator data will continue throughout the 55 evaluation, however, particularly in the analysis stage. The stage will conclude with in-person team planning meetings, the Kickoff Meeting with USAID and ANERA (16 February 2018), and approval of the data collection tools and workplan. 2) Data collection and analysis The second stage, which is expected to commence on/about 22 February 2018, will continue for approximately three weeks, with the first two weeks dedicated to collection of field data. The data collection will include: - A series of key informant interviews in the offices of implementers (essentially ANERA and TAM, including the three PCID field offices), USAID, and counterpart organizations. These interviews will be conducted primarily by the Team Leader, with potential participation of the Evaluation Manager in meetings with the counterparts. These interviews will require two to three working days. - Key informant interviews, group discussions, and project site visits in the 18 selected communities. Each visit is expected to take approximately one day (including reporting and coding). Community-level data will be collected simultaneously in the West Bank and Gaza. The community visits in Gaza will be conducted by the Technical Specialist in Gaza. Community visits in the West Bank will be split between the Evaluation Manager and the Community and Infrastructure Technical Specialist. The Team Leader will accompany one of them to the extent possible. Each team member will be responsible for documenting the interviews and group discussions he/she conducts, according to the interview protocols. All notes will be in English and will be coded and entered into a common Excel database on an ongoing basis. The interview notes and the database will be accessible to all team members. The team will meet at the end of each week to review the evidence and discuss emerging findings. - The survey of beneficiaries will be conducted by PCPO and managed by the Evaluation Manager. PCPO will pre-test and finalize the questionnaire, conduct the face-to-face interviews, prepare the database, and present the tables/analysis requested by the evaluation team. To the extent possible the survey will be administered in each community at the same time as the interviews and discussion groups. Analysis will be continuous throughout the data collection period. Interview and discussion group notes will be shared among team members through on-going access to the electronic files. In addition, the team will meet together on Saturday each week to review progress, challenges, and findings. At mid-point of the data collection period, the Team Leader will present a mid-point debriefing with USAID on preliminary findings and conclusions, and will provide directions of possible recommendations. She will also provide a progress report on field activities, including challenges and any unanticipated field course corrections in scheduling or in the overall implementation plan. (Deliverable 2) The third week beginning on/about March 11, will include finalization of the data collection and data files (including the coding of qualitative data and processing of the survey data), team discussion to reach consensus on conclusions and recommendations, and preparation of a PowerPoint presentation on findings, conclusions and recommendations for the formal End-of-Evaluation Oral De-brief to USAID by the Team Leader. The Team Leader will depart on March 19, 2018 (Deliverable 3). Oral comments and feedback from USAID during the debriefing will be incorporated into the draft report. 3) Reporting The Team Leader will be responsible for writing the draft final report and a stand-alone Executive Summary for review and comment by USAID (Deliverable 4) on 12 April 2018. The Team Leader will 56 incorporate USAID comments into the final version of the evaluation report within five working days after receipt of comments for delivery to USAID by QED. (Deliverable 5) An outline for the 30-page evaluation report, based on USAID requirements and guidance, is included in Annex G. (Note: The remaining deliverables in Section VI under the reporting phase of the evaluation, will be completed by QED after the completion of the work of the evaluation field team.) DELIVERABLES Deliverable 1: Final SOW At the Kickoff Meeting, the evaluation team leader will present a final SOW which includes a) a detailed work plan/timeline, b) methodology, c) data collection tools/protocols and informed consent forms, d) data analysis plan and e) any modifications to deliverables. Feld work will not begin until the evaluation team receives written approval of the proposed data collection instruments. Deliverable 2: Mid-Evaluation Debrief to USAID The Evaluation Team Leader will conduct a debrief to USAID to provide a progress report on field activities, including challenges and any unanticipated field course corrections in scheduling or in the overall implementation plan. Deliverable 3: End-of-Evaluation Oral De-brief to USAID on Preliminary Findings, Conclusions and Recommendations After completion of the fieldwork and prior to the submission of the draft evaluation report, the Team Leader will provide a formal debriefing to USAID points of contact. The debriefing will include a discussion of findings (accomplishments and challenges), conclusions and preliminary recommendations. The evaluation team will consider all oral comments and will incorporate feedback into the draft report as it is finalized. Deliverable 4: Delivery of the draft evaluation report and the draft stand-alone executive summary QED will submit a draft version of the 30-page evaluation report with associated annexes to the USAID technical team for review and comment. The report will describe findings (accomplishments and challenges) and recommendations. The 6-8-page executive summary will be included in the body of the text of the draft report prior to the page count and will also be delivered as a stand-alone document for public dissemination. The USAID technical team will provide consolidated written comments on the draft report/executive summary within 10 working days of receiving the draft documents. Deliverable 5: Final Evaluation report and the stand-alone final executive summary Upon receipt of consolidated comments24, the team leader will address said comments in a final revision of the report/executive summary within 5 business days. QED will resubmit the revised versions to USAID for review and approval. Deliverable 6: Infographic The QED team will engage our in-house designer to produce a two-page full color infographic which illustrates key findings, recommendations and insights. This deliverable will be geared toward one particular audience as determined during the Kickoff Meeting. The draft and final infographic will be 57 based on the USAID approved executive summary. QED will submit to USAID an infographic in English and a translation of the infographic into Arabic in electronic format (PDF), if required. Prior to submission, the Arabic translation will be proofread by a professional from within the technical field to ensure technical terms are appropriately and accurately translated. Deliverable 7: Post the USAID approved final report, the stand-alone final executive summary and the infographic to the USAID Development Experience Clearinghouse (DEC) Within 30 calendar days of USAID written approval of the final report, the stand-alone final executive summary and the infographic, QED will post said documents to the USAID Development Experience Clearinghouse (DEC) for public access and use. Deliverable 8: Post USAID-approved anonymized public-release datasets and associated supporting technical documentation to the USAID Development Data Library (DDL). Within 30 calendar days of USAID written approval of the final report and stand-alone executive summary, QED will submit anonymized “public release” quantitative and qualitative datasets and associated supporting technical documentation for written approval. Within 30 calendar days of USAID written approval of anonymized “public release” datasets and associated supporting technical documentation, QED will post said materials to the USAID DDL for public access and use. In addition, QED will submit said materials to the task order COR on a secured flash drive; all materials will be in an unlocked and editable format. 58 ANNEX B: DETAILED DESCRIPTION OF SURVEY METHODOLOGY Selection of the Sample of Communities A total of eighteen communities with completed infrastructure projects will be sampled by the evaluation. The number of communities to sample was determined on the basis of the resources (human and time) available to the evaluation, as well as needs to maximize opportunities for beneficiaries and users of completed infrastructure projects to participate in the evaluation. These communities are depicted in the following table. This sample was selected purposively on the basis of the following analysis: • Representation of the proportional distribution of projects between the West Bank and Gaza. • Representation of the proportional distribution of completed projects by sub-regions (north, middle, and south West Bank, and Gaza). • Representation of the different types of projects completed by PCID (sectoral distribution). • Representation of the different sizes of investment made by USAID. The process followed by the team in undertaking this analysis is explained below. 1) Analysis of completed projects: 2) Determine the number of communities that could be visited given resources: Within the program, and given the additional human resource made available to the team in Gaza, Sewage Network Reservoir Drainage Network N 5 Habla Attil Qabatia - Qaffin Zeita - - - - M 3 - Bil'in, Rantis - - Bili'n - Jericho - - - S 6 - - Nahalin Imneizil (2), Hitta - Al Walajeh Wadi Fukin Total 14 GS All 4 - - - Kuza'a - - Al Musaddar, Gaza - Al Musadd ar (2) Abasan al Kabira WBGS Total 18 1 3 2 4 3 1 3 1 3 1 Youth Centers and Health Roads Markets WB Region Sub-Region No. of communities to be sample Projects to sample within selected communities Water & Sanitation Eduation Public Spaces Region Sub-Region No. Communities % of Sub￾Region No. of Projects % of projects N 10 38% 10 36% M 5 19% 6 21% S 11 42% 12 43% Total 26 100% 28 100% GS All 7 27% 9 32% WB 59 examined WB and GS independently. Determined that 14 communities in the West Bank could be visited within the timeframe available for the field mission. This represents about 53% of all WB targeted communities. So, it was decided to also have 54% of communities in Gaza, i.e. 4 communities. 3) Analysis of sub-regional distribution of projects: 4) Determine number of projects that need to be visited to reflect the actual distribution of projects by sector within each sub-region 5) Purposively approximate the number of projects on basis of above an align with sub-regions, while also keeping in mind sampling the wide range of project typologies: Sewage Network Reservoir Drainage Network North 10 1 3 2 0 3 1 0 0 0 0 Middle 6 0 2 1 0 2 0 1 0 0 0 South 12 0 0 2 5 2 1 0 1 1 0 WB 28 1 5 5 5 7 2 1 1 1 0 GS All 11 0 0 0 4 0 1 1 1 3 1 WBGS Total 39 1.0 5.0 5.0 9.0 7.0 3.0 2.0 2.0 4.0 1.0 Sewage Network Reservoir Drainage Network N 36% 10% 30% 20% 0% 30% 10% 0% 0% 0% 0% M 21% 0% 33% 17% 0% 33% 0% 17% 0% 0% 0% S 43% 0% 0% 17% 42% 17% 8% 0% 8% 8% 0% Total 100% 10% 50% 50% 50% 70% 20% 10% 10% 10% 0% GS All 11 0% 0% 0% 36% 0% 9% 9% 9% 27% 9% Youth Centers and Health Roads Markets WB Roads Markets WB Region Sub-Region No. of projects Percentage Disribution of Projects Water & Sanitation Eduation Public Spaces Region Sub￾Region No. of projects Disribution of Projects Water & Sanitation Eduation Public Spaces Youth Centers and Facilities Health Sewage Network Reservoir Drainage Network N 5 0.5 1.5 1.0 - 1.5 0.5 - - - - M 3 - 0.8 0.4 - 0.8 - 0.4 - - - S 6 - - 0.9 2.3 0.9 0.5 - 0.5 0.5 - Total 13 0.5 2.3 2.3 2.3 3.3 1.0 0.4 0.5 0.5 - GS All 4 - - - 1.3 - 0.3 0.3 0.3 1.0 0.3 WBGS Total 17 0.5 2.3 2.3 3.6 3.3 1.3 0.7 0.8 1.4 0.3 WB Eduation Public Spaces Youth Centers and Health Roads Markets Region Sub-Region No. of communities to be sample Disribution of Projects Water & Sanitation 60 6) Select communities with projects as per above table, taking into consideration size of investment: Survey of Beneficiary Households (The following summary was prepared by Elias Kukali at PCPO, which was responsible for implementation and analysis of the household survey.) Overview In the beginning, we allocated the goals anticipated from making the questionnaire in the light of the evaluation goals and questions, and then determined the data and the information to be collected, and after that compiled and converted these objectives into a set of questions and inquiries. After having completed the design and selection of the questionnaire, we conducted a pilot test on a limited group of respondents identified in the research sample as to give me a feedback about their opinion on the quality of the questionnaire in terms of their understanding, inclusiveness and significance, as well as their quantity and adequacy to collect the required information on the topic of the research and its problematic. In the light of the observations and comments obtained, we modified the questions and compiled the questionnaire in its final form as to obtain the required outputs. Training Since its establishment in 1994, the center has been interested in carrying out training programs for both the researchers and supervisors all throughout the year. Many of those working with the center have already participated in several training programs, which aimed at developing their capacities to Sewage Network Reservoir Drainage Network N 5 1 1 1 - 1 1 - - - - M 3 - 2 0 - 1 - 1 - - - S 6 - - 1 2 1 - - 1 0 - Total 14 1 3 2 2 3 1 1 1 1 - GS All 4 - - - 1 - - 1 0 1 1 WBGS Total 18 1 3 2 4 3 1 2 1 2 1 Roads Markets WB Region Sub-Region No. of communities to be sample Number of of projects to sample Water & Sanitation Eduation Public Spaces Youth Centers and Health Sewage Network Reservoir Drainage Network N 5 Habla Attil Qabatia - Qaffin Zeita - - - - M 3 - Bil'in, Rantis - - Bili'n - Jericho - - - S 6 - - Nahalin Imneizil (2), Khallet Salih Hitta - Al Walajeh Wadi Fukin Total 14 GS All 4 - - - Kuza'a - - Al Musaddar, Gaza - Al Musadd ar (2) Abasan al Kabira WBGS Total 18 1 3 2 4 3 1 3 1 3 1 Youth Centers and Health Roads Markets WB Region Sub-Region No. of communities to be sample Projects to sample within selected communities Water & Sanitation Eduation Public Spaces 61 collect data and at making them familiar with the required mechanism, scientific methodology, scientific sincerity and ways of dealing with the public. When the questionnaire was ready for distribution, our center held training sessions for field researchers during which various issues were demonstrated including the objectives of the survey, data collection, confidentiality, dealing with respondents, the appropriate time to fill the questionnaire and resolving field problems. The training session also included an explanation of the terms, expressions and abbreviations used, general concepts, scrutinizing data in field, and way of selecting respondents. • It is PCPO standard to pilot any questionnaire prior to fieldwork to ensure that the questions are relevant, unambiguous, smooth and efficient. The pilot involved 6 interviews were conducted by each of the interviewers prior to fieldwork. • We have a quality policy: “No interviewer will go into the field if he/she does not prove full mastery of the interviewing process else they will be replaced”. The pilot interviews served also as a pre-set for the questionnaire, so any required adaptation of its wording was implemented prior to the launch of the field process. The same training took place for our enumerator in Gaza on Skype due to mobility restrictions imposed by the political status quo, and inability to travel to Gaza. Development of Training Materials and Protocols • PCPO prepared all training protocols and materials, including: • Training on the research process and research ethics. • An outline of the project and its objectives. • Written field instructions for all interviewers and supervisors for their reference. As per our usual practice, in addition to a personal briefing from a PCPO project manager, written instructions were provided to all interviewers during their training sessions. Among other items, written instructions consisted of: information about the purpose of the project (but at all times respecting client confidentiality); how respondents are to be selected; explanations and definitions of any atypical terminology; instructions such as skip rules, respondent selection, recording open-ended questions and the collection of information on non-response rates. Pilot The pilot included 32 interviews: • 16 men, 16 women; • 8 per age category 18-24, 25-39, 40-54, 55+. The main objectives of the pilot were to: • Test the translation into the local language. • Check the questions wording, if they were fully understood by the interviewee, and if there were any “local expressions” that could fit better into the translation. • Verify the total length of each section and of the whole questionnaire. • Identify the questions which were particularly difficult to understand, check the survey’s flow, etc. • Verify skip patterns. Results of the pilot test were the following: • The questions were clear and understandable to the respondents. • The length and time of filling the questionnaire were reasonable and we received no complaints at all in this regard. 62 • The fact that the preamble sustains that the aim of the questionnaire was for evaluation of USAID project in the Palestinian Territories has facilitated completing the questionnaire. • Validity was high and Cronbach scale (of 0.91) was used. Questionnaire Design We made sure when designing the questionnaire to write in an understandable and clear manner that did not bear multiple interpretations to avoid any confusion or ambiguities by the respondents, or obtain inaccurate answers from them. Furthermore, we have taken into account the time available to the respondents and, therefore, was keen not to have as much as possible long questions, which might lead to the rejection of the interviewees to respond to the questionnaire or give fast and inaccurate answers. The wording of the questionnaire was characterized by simplicity, clearness and consistency across questions. Moreover, the sequence of questions was carefully studied to avoid dependency of responses on previous relevant answers. Afterward, the questionnaires were proofread and revised thoroughly. Translation The master copy of the questionnaire was developed in English and then was translated into Arabic. The translation was done by an experienced and qualified translator, who already translated books from various languages, particularly from English into Arabic. The translation was subject to possible modifications, since some expressions, terms and ideas were culturally grounded. The translation was reviewed by the evaluation manager and the technical specialist before sending it into the field for the pilot. Sampling We are looking for residential units. By that, we mean a residential unit called "family living space". Each unit must have its kitchen. In the vast majority of cases, the residential unit is characterized by a doorbell or a post office box. Enumerators should go to a specific area and apply the method of selection of housing units (households) and interviewees in the manner identified during the training period. The supervisor will provide the enumerator with the community name, the specific address of project’s facility/location, and the route. Below are the phases that enumerators will follow before they start interviewing: 1. Enumerators will go to the sampling point (community) as determined by the district’s supervisor. 2. Enumerators will go to the project’s location/facility in that community as the starting point to start conducting the survey. 3. Enumerators will randomly define a direction (North, South, East, West), using a simple random method (e.g., Lottery method of sampling). 4. Enumerators will move from project’s location (center point) into the defined direction by approximately 150 meters. 5. Once in the street, either side of that particular street is selected (i.e., left or right houses). Then every other house (i.e., those with odd or even numbers will be interviewed). The side (left or right) and the system (every odd or even number) will be pre-determined by the researcher using a simple random method and will be applied during the conducting of the survey in that community. In the case of a multi-apartment building, each apartment is considered as a separate household. 6. Last Birthday Method: this is the last stage, which involved selecting one individual in each household using the Last Birthday Method. Face-to-face interviews are then conducted with the selected individuals. 7. Enumerators will conduct a maximum of 4 interviews in that street and will move a circle clockwise to the next street and continue interviewing using the same system. The researcher 63 will complete a circle clockwise, entering all the streets that are encountered in the path. - At least 7 streets should be included per community. The main idea is to cover as much as possible from the designated area. - Enumerators can move to the next street even if the 4 interviews were not completed in certain cases; A) If for a reason or another, the street was not found. B) If the street was found to be deserted or slightly inhabited. C) If the street was found to be commercial or industrial. 8. Once the first circle has been completed, further move by 50 meters in the same direction and complete the second circle clockwise (wider circle). - The term ‘circle’ implies walking clockwise with the aim of ending up where you started; - a circle, once started, must be completed (must not have incomplete circles); Example: Move from Starting Point (the “project’s location) into the defined direction by 150 meters. Complete a circle clockwise… …entering all the streets that are encountered in the path, in the direction of the “project’s location” center. 64 Once the first circle has been completed, further move by 50 meters… …in the same direction and complete a second circle clockwise. If a certain house or apartment was found to be uninhabited or the inhabitants refrained from receiving the interviewers, work would continue as normal, i.e., every other house. Only one member of any one household will be interviewed. For the selection of the respondent, all 15+ residents in a household will be listed by name and dates of birth according to their ID cards. After listing the household members, the designated respondent will be the person who had ‘The Last Birthday’ among all listed individuals before the date of the interview. In cases where the designated respondent is not at home after three visits to the same household at different times of the same day or different days of the field period, or in cases of definite refusals, or when the interview could not be completed with the designated respondent, the interviewer continue going ahead to the next household on the route from the same sampling point. The main benefit of the simple random sample is that each member of the community (mainly beneficiaries) has an equal chance of being chosen for the study. This means that it guarantees that the sample chosen is representative is selected in an unbiased way. In turn, the statistical conclusions drawn from analysis of the sample will be valid. A representative sample of 540 Palestinians (18 years and older) will be selected for this survey from Gaza Strip and the West Bank. This sample was based on 40 communities that benefited from the PCID program. The total number of sampled communities is 18 with 30 questionnaires to be allocated for each community, as shown in the Table hereunder: 65 The general rule is a sample size of 30 would allow us an adequate observation to take the benefits of the Central limit Theorem, i.e. at n = 30, we start to see the bell shape curve if the data is normally distributed. It's that you need at least 30 before you can reasonably expect an analysis based upon the normal distribution (i.e. z test) to be valid. That is it represents a threshold above which the sample size is no longer considered "small". Furthermore, the sample 45.0% of total number of communities that benefited from PCID program, and in which we will survey at least 126 streets. Conducting interviews We in the PCPO follow a policy of conducting some extra interviews in each community just in case. We ended up conducting 619 interviews with a least of 30 interviews in each community as shown above. The interview results were then weighted accordingly. Some additional tables and sample characteristics Descriptive Average Minimum Maximum Duration of interview 6 minutes and 52 seconds 4 minutes and 22 seconds 9 minutes and 42 seconds Age 37.1 years 18 years 75 years Gender Frequency Percent Male 290 46.8 Female 329 53.2 Total 619 100 Which of the following best describes the highest level of education you completed? Frequency Percent Elementary or below 63 10.1 66 Intermediate 113 18.3 Secondary 200 32.4 Diploma 72 11.6 University 159 25.6 Graduate studies 11 1.8 Refused 1 0.2 Total 619 100 Which of the following best describes your marital status? Frequency Percent Married 466 75.3 Single, never married 138 22.2 Divorced/widowed/separated 16 2.5 Total 619 100 A note on weighting: As stated above, the agreed sample size for each community is 30 respondents. Nevertheless, the PCPO have done some additional interviews in some communities like Al-Mosaddar, Imneizel, and Bili’n because they had multiple PCID projects. As a result, weighting was applied so all communities had the same contribution to the conclusions of this evaluation. 67 ANNEX C: DATA COLLECTION TOOLS Questionnaire for Beneficiary Household Survey - English version - Arabic version Community Tools - Checklist for Project Site Inspection - Key Informant Interview Protocol for Community Leaders - Interview Protocol for Group Discussion with Beneficiaries/Residents Key Informant Interviews with Implementing Partners, USAID, and Counterpart Agencies - ANERA - TAM - Partner Engineering Firms - USAID - PWA - CMWU - MOEHE 68 QUESTIONNAIRE FOR SURVEY OF BENEFICIARIES PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to evaluate one of the programs it supports, called the PCID (the Palestinian Community Infrastructure Development program). The evaluation team is visiting 18 of the communities where the PCID program has worked, to learn about the successes and challenges PCID has encountered. The study is especially important to USAID for designing future community programs in Palestine. We would appreciate your collaboration to respond to a short survey about the PCID activity in this community. We have selected your household randomly. Everyone’s answers and perceptions are important to the study. It will take no more than 10 minutes of your time. Please be confident that your responses will remain confidential and anonymous and that your answers will be used for statistical purposes only, in combination with those of others in the communities we are visiting. No personal information or individual responses will be reported. Do you agree to allow the PCID evaluation team to use your responses to the survey for the purpose of the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Demographics D1) Gender 1. Male. 2. Female. D2) What is your age? _______________ years. D3) Which of the following best describes the highest level of education you completed? 1. Elementary or below. 2. Intermediate 3. Secondary. 4. Diploma. 5. University. 6. Graduate studies. 7. Refused (Do not read) D4) Which of the following best describes your marital status? 1. Married. 2. Single, never married. 3. Divorced/widowed/separated. D5a) Number of individuals living in this house (including all ages): _______persons. D5b) You are: 1. Head of the household? 69 2. Spouse? 3. Son/daughter? 4. Grandfather/Grandmother? 5. Other? D6) For this property, does the head of household: 1. Own? 2. Rent? 3. Other? D7) How long have you been living in this community? 1. Less than one year. Terminate interview 2. From one to five years. Continue to Q1 3. More than five years. Questions: Q1) Are you familiar with “name of the project” that was implemented in your community? 1. Yes. 2. No. Q2) Do you know the name of the entity responsible for implementing these/this project? • Please do NOT read the options, keep it spontaneous. • Multiple answers are allowed. 1. USAID. 2. ANERA. 3. Village Council. 4. Palestinian Authority. 5. Other, please specify: ______________. 6. I have no idea. Q3) How did you first hear about this project? • Multiple answers are allowed. 1. The media (radio, TV, newspaper, social media). 2. Other people in my community. 3. Relatives and friends. 4. Visits by outside officials. 5. LGUs (Municipality, Village council, Joint Services Council). 6. Brochures, posters, billboards, etc. 7. I saw it during construction. 8. Other (Please do not read). Q4) When did you hear about the project? • Please read all options. 1. Before the implementation of the project. 2. During the implementation of the project. 70 3. After the implementation of the project was completed. 4. Do not know (Please do not read). Q5) How interested were you in this project when you first heard about it on scale from 1 to 10, where 1 means “not interested at all”, and 10 means “extremely interested”: Not interested at all Very interested 1 2 3 4 5 6 7 8 9 10 Q6) Did you or anyone else in your household attend any meetings for this project? • Multiple answers are allowed 1. Yes, I did Continue to Q7 2. Yes, a household member did Skip to Q11 3. None Q7) What was the purpose of the meeting? • Please read all options. • Multiple answers are allowed. 1. Selection of the project. 2. Design of the project. 3. Construction of the project. 4. Awareness about sustaining the project. 5. Other, please specify: _________________________. Q8) How would you describe your participation in these meetings? 1. I was an active participant. 2. I was mostly a listener. Q9) How did you know about these meetings? 1. I got an official invitation. 2. I heard about it from someone in my community. 3. I heard about it from a public announcement (radio, TV, newspaper, social media, mosques, etc.). 4. I attended it by coincidence. Q10) To what degree do you feel that the organizers were responsive to the participants in the meeting? 1. They were very responsive. 2. They were somewhat responsive. 3. I do not think that their input considered. 4. I do not know (Do not read). Q11) Did you or anyone else from your household have work on the construction of the project? • Multiple answers are allowed. 1. Yes, I did.  For how many months? _________ months. 71 2. Another member of the household did.  For how many months? _________ months. 3. No one from the household was employed in the project. Q12) How would you describe the importance of this project to: • Please read all options. Important Somewhat important Somewhat unimportant Not important a. Your family? 1 2 3 4 b. Your community? 1 2 3 4 Q13) How would you describe the duration of the project? • Please read all options. 1. It took too long to finish. 2. It took a reasonable amount of time. 3. It was completed more quickly than I expected. 4. I have no idea (Do not read). Q14) In general, did the project meet your expectations? 1. Yes, to a great extent. 2. Somewhat met my expectations. 3. No, to a low degree. 4. No opinion / had no expectations. Q15) Do you or anyone in your household use the services provided by this project? 1. Yes (Continue to Q16) 2. No (Skip to Q19) Q16) On a scale from 1 to 10, how satisfied are you with these services? Not satisfied at all Very satisfied 1 2 3 4 5 6 7 8 9 10 Q17) Since the project was finished, did the quality of services provided by this project change over time? What kind of change? 1. A positive change Skip to Q19 72 2. Stayed the same 3. A negative change Continue to Q18 Q18) Why do you think the quality of service has declined? • Multiple answers are allowed. 1. Lack of maintenance. 2. Improper use by the community. 3. The quality of the construction. 4. Acts of vandalism. 5. Lack of human/financial/technical resources to operate well. 5. Other. Q19) Did the project fulfil an urgent need for your community or are there other projects that should have come first? 1. Yes, this project fulfilled an urgent need for my community. End the interview and thank the respondent 2. No, there are other projects that should have come first. Continue to Q20 Q20) In your opinion, what are the other pressing infrastructure projects that should have come first: • Multiple answers are allowed. • Please do not read the options, keep it spontaneous. ECONOMIC 1. Electricity/energy infrastructure SOCIAL 2. Health care services/infrastructure 3. Education services (Schools/additional rooms) 4. Public spaces (parks, gardens, playgrounds) 5. Care facilities for the elderly 6. Sports facilities 7. Improved accessibility of public buildings/facilities to the disabled 8. Rehabilitation centres for the disabled INFRASTRUCTURE 73 9. Sewage and sanitation 10. Drainage systems 11. Water for agriculture 12. Water for domestic purposes 13. Electricity infrastructure 14. Transport (e.g., Roads, streets, transportation) OTHER, please specify: _________________________________ End the interview and thank the respondent إستبانة إستطلاع للمستفیدین تقییم الأداء النھائي لمشروع تنمیة البنیة التحتیة للمجتمع الفلسطیني آذار / مارس 2018 مرحبا / صباح الخیر / نھارك سعید ! اسمي _____________ من المركز الفلسطیني لاستطلاع الرأي؛ وھي شركة أبحاث تسویقیة مستقلة، ونحن بصدد تقییم أحد ّة للمجتمع الفلسطیني. البرامج التي تدعمھا الوكالة الأمریكیة للتنمیة الدولیة، ویدعى برنامج تنمیة البنیة التحتی سیزور فریق التقییم 18 ّ تجم ّ ع سكني تم تنفیذ مشاریع فیھا ضمن ھذا البرنامج وذلك للإطلاع على النجاحات والتحدیات التي ّة مجتمعیة في ّة بھدف المساعدة في تصمیم برامج مستقبلی ّ واجھھا. الدراسة مھم ّ ة بشكل ٍ خاص للوكالة الأمریكیة للتنمیة الدولی فلسطین. ّ ّ ة للمجتمع الفلسطیني في ھذا التجمع السكني، فقد نقدّر لك تعاونك باستجابتك لمقابلة قصیرة حول نشاط مشروع تنمیة البنیة التحتی ّ اخترنا أسرتك بصورة عشوائیة. وتأكد أن أجوبة وأفكار كل شخص تعتبر مھمة للدراسة ولن نأخذ من وقتك أكثر من عشرة دقائق. ّة وأنھا ستستخدم فقط لأغراض إحصائیة مقترنة مع ّا ً وواثقا ً بأن إجاباتك ستبقى سریة ومجھولة الھوی نرجو أن تكون مطمئن ّة. ّة أو إجابات فردی إجابات أناس آخرین في التجمعات السكنیة التي سنزورھا. ولن نبوح في ھذا السیاق بأیة معلومات شخصی ّ فھل توافق على السماح لفریق التقییم باستخدام إجاباتك لغرض إعداد تقریر التقییم؟ _____ نعم ______لا (للباحث: نرجو تسجیل الردّ الشفھي للشخص). 74 ّـــــة: أسئلة دیمغرافی د :1 الجنس 1 ذكر 2 أنثى د :2 عمرك ____ سنة د :3 ما ھو أعلى مستوى دراسي أكملتھ مما یلي ؟ 1 ّ إبتدائي أو أقل من ذلك 2 إعدادي 3 ثانوي 4 دبلوم متوسط 5 جامعة 6 دراسات علیا 7 یرفض ( لا تقرأ) أي د :4 مما یلي ینطبق على وضعك العائلي ؟ ّ 1 متزوج 2 أعزب، لم یسبق لھ الزواج ّق / أرمل / منفصل 3 مطل د 5 أ: عدد الأفراد الذین یعیشون في ھذا البیت (شاملا ً جمیع الأعمار): _______ شخصاً د 5 ب: ھل أنت : 1 رب الأسرة ؟ 2 الزوج / الزوجة ؟ 3 إبن / إبنة ؟ 4 جدّ / جدّة ؟ 5 غیر ذلك د :6 ّ ھل رب الأسرة بالنسبة لھذا العقار: 1 ھو المالك ؟ 2 مستأجر ؟ 3 غیر ذلك ؟ ّ منذ متى تعیش في ھذا التجم ّ ع السكني / الحي د :7 ؟ 1 ّ منذ أقل من سنة (للباحث: إنھ ِ المقابلة !) 2 من سنة إلى خمس سنوات (للباحث: إنتقل للسؤال س 1) 3 أكثر من خمس سنوات (للباحث: إنتقل للسؤال س 1) أسئلة الدراسة: س :1 ّ ھل أنت على درایة "بإسم المشروع" الذي تم تنفیذه في تجمعك السكني ؟ 1 نعم 2 لا (إنھ ِ المقابلة) 75 س :2 ھل تعلم إسم الجھة المسؤولة عن تنفیذ ھذا المشروع ؟ ّة) • ّ (للباحث: الرجاء لا تقرأ الخیارات ! دع الإجابة عفوی ُسمح بأكثر من إجابة) • (للباحث: ی 1 الوكالة الأمریكیة للتنمیة الدولیة (یو.إس.إید) 2 أنیرا (المساعدة الأمریكیة الشرق أوسطیة للآجئین) (ANERA( 3 المجلس القروي ّة 4 ُ السلطة الفلسطینی 5 ّ غیر ذلك، الرجاء حدّد: _________________ 6 ّ لیس لدي أیة فكرة س :3 ّ كیف سمعت لأول مرة عن ھذا المشروع ؟ ُسمح بأكثر من إجابة) • (للباحث: ی 1 من وسائل الإعلام (الرادیو، التلفاز، الجریدة، وسائل التواصل الإجتماعي) 2 من أناس آخرین في التجمع السكني الذي أعیش فیھ 3 من أقارب وأصدقاء 4 من زیارات مسؤولین من الخارج ّي (البلدیة، المجلس القروي، مجلس الخدمات المشتركة) 5 ُ من سلطة الحكم المحل ّة وما شابھ 6 من منشورات، ملصقات جداری 7 شاھدتھ أثناء التنفیذ 8 غیر ذلك (لا تقرأ) س :4 متى سمعت عن المشروع ؟ • ّ (للباحث: الرجاء قراءة جمیع الخیارات) 1 قبل تنفیذ المشروع 2 خلال تنفیذ المشروع 3 بعد إتمام تنفیذ المشروع 4 لا أدري (لا تقرأ) ّم من (1) إلى (10) حیث س :5 ّ ما مدى إھتمامك بھذا المشروع عندما سمعت عنھ لأول مرة ؟ أرجو بیان ذلك على سل ّ ) یعني "مھتم للغایة". ّ ) "غیر مھتم 10 یعني (1 على الإطلاق" و ( ّ غیر مھتم ّ مھتم للغایة على الإطلاق 10 9 8 7 6 5 4 3 2 1 ُسمح بأكثر من س 6 ّ : ھل شاركت أنت أو أي شخص من أفراد أسرتك في أیة إجتماعات تخص ھذا المشروع ؟ (للباحث: ی إجابة) ً (للباحث: إستمر بالسؤال رقم 7) 1 نعم، شاركت أنا شخصیا 2 نعم، شارك أحد أفراد أسرتي (للباحث: إنتقل للسؤال رقم 11) 76 3 لا أحد (للباحث: إنتقل للسؤال رقم 11) س :7 ماذا كان الغرض من الإجتماع ؟ • ّ (للباحث: الرجاء قراءة جمیع الخیارات) • (للباحث: یسمح بأكثر من إجابة) 1 إختیار المشروع 2 تصمیم المشروع 3 إنشاء المشروع 4 التوعیة حول الحفاظ على المشروع 5 غیر ذلك (الرجاء حدّد: ________________________) س :8 كیف تصف مشاركتك في ھذه الإجتماعات ؟ 1 كنت مشاركا ً فعالاً 2 كنت مستمعا ً في معظم الأوقات س :9 كیف علمت عن ھذه الإجتماعات ؟ 1 إستلمت دعوة رسمیة 2 سمعت عنھا من شخص ٍ ما في التجمع الذي أسكنھ. 3 سمعت عنھا من إعلانات عامة (الرادیو، التلفاز، الجریدة، وسائل التواصل الإجتماعي، الجوامع، وغیرھا). 4 حضرتھا عن طریق الصدفة. س 10) لأیة درجة تشعر بأن المنظمین كانوا مستجیبین لمقترحات المشاركین في الإجتماع ؟ 1 كانوا مستجیبین جدّا ً 2 كانوا مستجیبین نوعا ً ما 3 لا أعتقد بأن مداخلاتھم/اقترحاتھم أخذت بعین الإعتبار 4 لا أدري (لا تقرأ) س :11 ھل عملت أنت أو أي أحد من أفراد أسرتك في المشروع أثناء تنفیذه؟ • (للباحث: یسمح بأكثر من إجابة) 1 نعم، لقد عملت في المشروع ولمدة كم شھر ؟ ________ شھرا ً 2 نعم، عمل أحد أفراد أسرتي في المشروع ولمدة كم شھر ؟ ________ شھرا ً 3 لم أعمل أنا ولا أحد من أفراد أسرتي في المشروع. س :12 كیف یمكن أن تصف أھمیة ھذا المشروع لـ: ّ (للباحث: الرجاء قراءة جمیع الخیارات) ّ غیر مھم ّ غیر مھم نوعا ً ما ّ مھم نوعا ً ما ّ مھم أ. عائلتك 1 2 3 4 ب. التجمع الذي تعیش فیھ 4 3 2 1 س :13 ُ كیف تصف مدّة تنفیذ المشروع ؟ ّ (للباحث: الرجاء قراءة جمیع الخیارات) 77 1 ُ إستغرق مدّة طویلة جدّا ً لإنھائھ 2 ُ إستغرق مدّة معقولة من الزمن ّع 3 إنتھى أسرع مما كنت أتوق ِكرة (لا تقرأ) 4 ّ لیس لدي أدنى ف س :14 ّ بشكل ٍ عام، ھل لبى المشروع توقعاتك؟ 1 نعم، بدرجة كبیرة ّى نوعا ً ما توقعاتي 2 لب 3 لا، لدرجة منخفضة 4 ّ لیس لدي رأي / لم ّ یكن لدي توقعات س :15 ھل تستفید أنت أو أي فرد ٍ من أفراد أسرتك من الخدمات التي یوفرھا ھذا المشروع ؟ 1 نعم ( للباحث: ّ إستمر بالسؤال 16) 2 لا (للباحث: إنتقل إلى السؤال 19) ّم من (1) إلى (10) حیث یعني (1) "غیر راض ٍ على س :16 ما مدى رضاك عن ھذه الخدمات ؟ أرجو بیان ذلك على سل الإطلاق" و (10) یعني "راض ٍ جدّا". غیر راض ٍعلى الإطلاق راض ٍ جدّاً 10 9 8 7 6 5 4 3 2 1 س :17 ّ منذ إنتھاء العمل بالمشروع، ھل تغیرت نوعیة الخدمات التي یوفرھا ھذا المشروع مع الوقت؟ وأي نوع ٍ من التغییر؟ 1 تغییر إیجابي (للباحث: إنتقل إلى السؤال 19) 2 بقیت كما ھي (للباحث: إنتقل إلى السؤال 19) 3 تغییر سلبي ّ (للباحث: إستمر بالسؤال 18) س :18 لماذا تعتقد بأن نوعیة الخدمات قد تراجعت؟ (للباحث: یسمح بأكثر من إجابة) 1 بسبب عدم وجود صیانة 2 بسبب الإستخدام الغیر ملائم من قبل التجمع السكني 3 بسبب نوعیة الإنشاء / التنفیذ 4 بسبب أعمال التخریب ّة لتشغیل المشروع بصورة جیدة 5 بسبب عجز في الموارد البشریة / المالیة / الفنی 6 غیر ذلك س :19 ّ ھل تعتقد أن المشروع لب ّ ى حاجة ماس ّ ة لتجمعك السكني، أم أن ھناك مشاریع أخرى كان ینبغي أن تأتي أولا ً؟ ّى ھذا المشروع حاجة ماسة للتجمع السكني الذي أعیش فیھ 1 نعم، لقد لب 2 لا، ھناك مشاریع بنیة تحتیة أخرى كان ینبغي أن تأتي أولا .ً 78 ّ ّ ة الأخرى الملحة التي كان ینبغي أن تأتي أولا ً؟ س :20 برأیك، ما ھي مشاریع البنیة التحتی ُسمح بأكثر من إجابة) • (للباحث: ی ّة) • ّ (للباحث: الرجاء لا تقرأ الخیارات ! دع الإجابة عفوی الناحیة الإقتصادیة: ّة للطاقة .1 الكھرباء / البنیة التحتی الناحیة الإجتماعیة: ّة ّة / البنیة التحتی .2 خدمات الرعایة الصحی .3 الخدمات التعلیمیة (مدارس / غرف صفیة إضافیة) .4 ّ مساحات عامة (منتزھات، حدائق، ملاعب) .5 ّ مرافق رعایة لكبار السن .6 مرافق ریاضیة .7 تحسین إمكانیة الوصول للمباني العامة / مرافق لذوي الإحتیاجات الخاصة (المعاقین) .8 مراكز لتأھیل المعاقین ّة: البنیة التحتی .9 شبكات الصرف الصحي/المجاري .10شبكات تصریف میاه الأمطار .11میاه للزراعة .12میاه للاستخدام المنزلي ّة للكھرباء .13البنیة التحتی .14 النقل (مثلا :ً الطرق، الشوارع، نظام النقل). .15 لیس ھناك مشاریع أخرى ذات أولویة (لا تقرأ). غیر ذلك، الرجاء حدّد: ___________________________ إنھ ِ المقابلة واشكر المستفتى ! 79 GROUP DISCUSSION WITH BENEFICIARIES PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Total number of interviewees, male/female ___ female ___ male ___ total Group type Community Type of project Verbal response to consent 1.Yes 2.No 3.Didn’t ask (Community Outreach in Design) 1. How did you hear about the ANERA project? And when did you hear about it? - Was this project something that had been discussed in the community? Would you say that it was a priority for improving the living conditions in the community? - How much time passed between when the idea for the project was first discussed and its completion? Why do you think it took so long? If yes, what do you think caused the delay? (Outreach - Follow-through and involvement with the community/beneficiaries) 2. Did you attend any meetings or discussions with the project team about the design – before the construction began? Did you know about meetings? Were you invited/encouraged to attend? - Were there any changes that you or others in the community wanted to make in the project design? What were they? Were these changes included? Why or why not? - What about after the construction started? Did people in the community request any changes that you know about? - Did you feel like ANERA wanted to hear from you and others in the community, or were the meetings mostly to inform you about the project plans? 80 3. Did you respond to any surveys about the project? Before, during, after? What was your understanding of the purpose of the surveys? Do you think ANERA used any of the information you provided? 4. Did you or any one you know work in this project? For how long? What kind of work? (End of project and utilization) 5. Did you attend any of the project activities? (e.g., fun days, meetings about how to use the project, health day at the clinic)? How would you describe this event/training meetings? If no, why not? If yes: Were they well attended? Do you think they were useful? Explain. (people learned from them, they were entertaining and showed the usefulness of the project) 6. How would you describe the importance of this project to you and your family? How would you describe its importance for the community? (probe in terms of well-being, social and economic development, did it resolve a critical issue) 7. Do you think the community is using the project as much it could? Why or why not? - How often do you use the project? Why? - In general, did the project meet your expectations? 8. [For water projects only: What benefits have you seen from the project (e.g., availability, savings, quality, quantity)? [for networks] Are people in the community using the new system? Why or why not? [probe in terms of affordability of connection and of the service itself] (maintenance, sustainability, ownership) 9. In your opinion, do you think the project is being maintained properly? By whom? 10. Did you participate in training (maintenance and sustainability) during or after the construction? - Was it useful? Who else attended? Is there a community plan for maintenance? - Do you believe that your community has the resources (human/technical/financial) to maintain this project? What about the village council? - How long do you expect the project to stay operational? Why? Do you have ideas about how to improve it? (EQ 4: other priority needs) 11. You said XX about the importance of this project for the community. What are the most pressing needs now in terms of infrastructure and services? 12. Do you have any other messages for ANERA and USAID about projects like this? 81 OBSERVATION CHECKLIST FOR PROJECT SITE VISITS PCID Final Performance Evaluation March 2018  Community ___________________  Observer ___________________  Project title ___________________  Date ___________ ANSWER COMMENTS AND EVIDENCE 1 The project is in good shape (clean, new, etc.) 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 2 The project is being regularly maintained 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 3 The project is utilized at the moment 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 4 The project addresses the priorities and needs of gender and vulnerable groups 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 5 The project utilized locally supplied or manufactured construction and other materials 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 7 “Green building” ideas have been incorporated in the construction of the project 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 8 The project is well-operated and managed 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 9 The physical structure is as planned 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 10 Billboards and awareness campaigns exist around the project’s area 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 11 The quality of services delivered by the project are as intended 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 12 Beneficiaries are satisfied with services delivered by the project 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 13 Beneficiaries are taking care of the facility 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 82 14 Needed resources for maintenance are available 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 15 The location of the project is appropriate for the intended use 1. Yes completely 2. Mostly 3. Partly 4. No - or no evidence 16 The project has no significant impact on the surrounding environment 1. Yes completely 2. Mostly 3. Partly 4. No –or no evidence 17 The project is well-equipped with needed facilities 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 18 The project appears to be incomplete in some way. 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence 19 The project is being misused 1. Yes completely 2. Mostly 3. Partly 4. No – or no evidence Any additional observations or comments: 83 INTERVIEW PROTOCOL FOR ANERA PCID DEPUTY CHIEF OF PARTY CHIEF CONSTRUCTION ENGINEER PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Was the session recorded? 1.Yes 2. No Number of years with PCID 1. According to the organizational structure (AWP IV), you are responsible for the technical/engineering aspect of the projects. What is the relationship between your unit and the communities? - Is anyone from the community involved or consulted in the design and proposal stage? Does the village council (or the municipality) have to agree to move forward with the project before the design begins? What unit in PCID is responsible for interaction with the community at this stage? - Are any community-based/social factors included in the design decision process? (e.g., vulnerability, underserved, cost-share, rural vs urban, availability of labor for construction) - How are the communities selected? What are the criteria? (Is the community always selected on the basis of the proposed project? Has there been a case when the community was the target, and the project came about to meet that community’s need?) 2. Has PCID ever had to make a choice between two (or more) potential projects? If yes, what was the basis of that choice? (technical feasibility, cost, project source (PWA/USAID/ANERA), community vulnerability) 84 - Do water and sanitation projects have priority over other types of projects in the selection process? What about HIMII projects? - What is the main reason(s) that projects are not selected by PCID? [technical vs social] 3. Is it correct to say that the function and operation of the Design Development Team is primarily technical? Does this team have contact with the community? If yes, what is the nature of the contact? - In reality, is consideration given to gender or other social factors (youth, disability) at this point? - How and to what extent does the design team work with the Monitoring and Reporting Unit? 4. The construction stage is managed by the field offices and the Gaza technical offices. - The chief construction engineer is responsible for quality control, maintenance of construction standards, contract compliance etc. Does the oversight role include any examination of gender or other social factors? How does the chief engineer work with the other personnel in the field office (community outreach coordinator, gender coordinator) - What is the nature of the engineer’s interaction with the community during construction? - Who or what entity has the final approval for the completion and hand-off of the project? (PCID, line ministry, community) 5. Do you work directly with the DCOP Non-technical (e.g., joint planning meetings, joint approvals, daily interaction about specific projects during implementation)? - What advantages/improvements have you observed since these two roles were separated in the reorganization? 85 INTERVIEW PROTOCOL FOR ANERA PCID CHIEF ENGINEER FOR DESIGN DEVELOPMENT PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Was the session recorded? 1.Yes 2. No Number of years with PCID 1. At what point in the project selection process does the work of the Design Team begin – before or after the selection is finalized? 2. I am interested in knowing more about your interaction with the community during the design phase? - When you or a member of your team first visits the community . . . What is the purpose of the visit (technical feasibility, social feasibility)? Do you meet with the Village Council and/or other residents (to explain, to get their buy-in, for them to participate in the decision)? What consideration is given to maintenance and sustainability at this stage? 3. At what point in the design process do the formal project selection criteria come into play? - In the Selection Criteria, what is meant by the statement “Target/take into consideration Gender groups and needs (priority given to women and youth)?” How is it applied? - What is the main reason(s) that projects are not selected? [technical vs social] 86 4. Describe how you work with the partner engineering firms and what they contribute to the project design and the project proposal. 5. Describe who (or what units) are included in the Design Unit and their role/responsibilities. - Do the same individuals work on all the project designs? - Has gender been a consideration in staffing the Design Team? 6. Describe how the Design Unit works with the Monitoring and Reporting Unit. - When, if at all, does the Design Unit consult monitoring or tracking data provided by the Monitoring and Reporting Unit? 7. What is the main reason that USAID may question or reject a design? - In your view, is this approval process an important point of delay in getting the project underway? Are there differences among the types of projects (e.g., water and sanitation, roads, schools, youth centers)? - What is the relationship to costing? (Where do you make cuts when necessary – materials, labor, maintenance, extent of project, etc.) 8. Describe how the Design Unit works with the engineers involved in construction. At what point is the work of the Design Unit on a project complete? 9. About a year ago, PCID revised the organizational structure? - Were there any changes in the work of the Design Unit after this change? 87 INTERVIEW PROTOCOL FOR ANERA PCID CHIEF OF PARTY PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Was the session recorded? 1.Yes 2. No Number of years with PCID 1. What was ANERA’s understanding of the purpose of this program when it started, in terms of infrastructure, community needs, links to other social and economic challenges in Palestine, etc.? What is the relationship between the PCID and other USAID infrastructure projects? 2. In your opinion, has this purpose (or the emphasis) changed over time? [If yes, in what way and why?] - Initially, in the cooperative agreement, water and sanitation projects were emphasized as well as HIMII projects. In what ways, if any, has this priority changed? - In implementation, to what extent/in what way has the PCID program been guided by the results framework? [increased affordability and availability of water and sanitation; increased access to community infrastructure services, including support to USAID programs, focus on vulnerable and underserved communities, and rapid response to unforeseen urgent needs] - On a different note, how does PCID fit into the ANERA strategy/mission? Has that changed? 3. What do you see as the principal achievement/success of the PCID program? Are there other important achievements? 88 4. What steps did ANERA take to respond to the requirement for attention to gender and to social inclusion, within the program itself and in the communities? (systematic vs ad hoc) What, if any, effect did you observe as a result? What more could have been done? 5. What was the role of TAM in PCID? Describe the way in which TAM and its field staff worked with the ANERA team? At what point did TAM enter the identification, selection, and implementation process for projects? 6. USAID refers to PCID as a community infrastructure program, and the evaluation questions refer to the “community outreach model.” How do you describe this model – its components and how it has functioned in practice? What is the relationship between the project selection process and the community selection process? What changes, if any, would you recommend for the outreach model? (Is this a standard ANERA model?) 7. The PCID reports provided by USAID note that ANERA conducted an internal mid-term evaluation of PCID, which was followed by changes in the management and organizational structure of the program. What findings/conclusions led to these changes? What has been the effect of the changes in terms of operations and results? 8. In your opinion, what was the principal unanticipated challenge ANERA encountered in implementing the PCID program? Have there been other significant challenges? - Challenges ANERA was able to overcome and challenges not able to overcome 9. What is the main reason that USAID may question or reject a design? - Are there differences among the types of projects (e.g., water and sanitation, roads, schools, youth centers)? - What about cost and budget? 10. According to the Performance Indicator database, across all five years, the program has failed to meet the performance indicator targets, in some cases by wide margins. What is your explanation for and response to this finding? Did it matter in terms of the project purpose or in the relationship with USAID? Why or why not? 11. What was the rationale for the one-year PCID extension? 12. How do you assess PCID’s success in countering historic problems with community infrastructure maintenance and sustainability? What, if any, potential issues or problems have you identified going forward in terms of maintenance and sustainability of the PCID community projects? 89 INTERVIEW PROTOCOL FOR ANERA PCID MONITORING AND REPORTING UNIT PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Was the session recorded? 1.Yes 2. No Number of years with PCID 1. How many people are in the M&R Unit? Is everyone located in the Jerusalem office? What are the positions/roles? - Has the structure or responsibilities changed since 2013? If yes, in what way? 2. Is this unit consulted when a new project is being considered? - Do you ever provide background data such as monitoring information from other similar projects? - What is the role of this Unit during the design and project proposal stage? - Does anyone from this Unit have direct interaction with the Village Council or others in the community in setting up the monitoring for a new project? 3. According to the PCID reports, baseline data is collected for each project? Please describe the types of baseline data that you collect. - When was the baseline data collected for the USAID PCID indicators? (in most cases, the value is 0) Was it done by this Unit? Have the indicators or targets changed since 2013? Was this Unit involved in setting the targets? - Describe how this Unit collects the data for the quarterly and annual reporting on the USAID indicators. (process and checks on data quality) 90 - Do you collect baseline data for the projects that is not reported to USAID (e.g., for PCID monitoring and management of individual projects)? How often do you measure these indicators and what Unit(s) use this information? - [The work Plan 2016-2017 document says baseline data is collected for WATSAN projects only and for three outcome indicators only, why? What about other projects?] 4. According to the PCID documents, the Monitoring and Reporting Unit carries out mini-surveys to monitor the effects of the community outreach aspects of the project, and to measure any change in the residents’ utilization of the services. Please describe the baseline and follow-up monitoring for these activities. How is this information used in the PCID (i.e., what Unit, where in the process)? (quality – comparability over time – sample selection - numerators) - Does the Monitoring and Reporting Unit coordinate with TAM at all? - Are the findings of these surveys always shared with the design and/or the construction units? (in written documents, in meetings or consultations?) How are these surveys used by the technical units? (in decisions, in work with the community?) 5. Do you report on all the USAID indicators quarterly and/or annually? (Have the indicators or the targets changed since 2013? How are/were the targets set? (who, on what basis?) - In some of your quarterly and annual reports you cite survey data to show the impact the impact of some (mostly water) projects on individuals – for example, women’s time use. What was the process for collecting and analyzing this information? 6. I understand that the outreach to the community continues after the construction is complete to focus on maintenance and sustainability. Does this apply to all types of projects? Does this Unit monitor this outreach to measure whether it is effective? Is there a 12-month follow-up survey in all the projects or only in the water and sanitation projects? 7. How would you describe the place of the Monitoring and Reporting Unit in PCID? - Does the unit participate in decisions about the selection or design of projects? - Does the Unit provide critical information for the technical teams about the community and the residents? - Does the Unit primarily support ANERA in its interactions with USAID and the PA line ministries? - Have you observed any changes in the role of the Unit during the course of the project? (or after the reorganization). 91 INTERVIEW PROTOCOL FOR ANERA FIELD OFFICE, HEBRON PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Community Outreach 1. At what point in the project development process do you first visit the community? - Are there differences in your approach to the community for different types of projects? (water, sanitation, drainage, school, social assistance). - Is your approach to the community always the same? [What about a situation like the clinic in Walaja, where the request came from the health office in USAID?] Does the approach differ in high priority projects (e.g., HIMII, or emergency/humanitarian response projects)? [Bethlehem market] - Who is your point of contact in the community? What is the purpose and agenda of this first meeting? 2. How often do you normally visit a community during the project process? (during the design phase, during construction, after completion) - What do you do in each visit and for what purpose? - Who do you contact? Do you go door-to-door, have meetings with local organizations? Examples? 92 - Can you give examples of specific issues you have encountered? - Do you work directly with the gender outreach person in the field office? (Did all the tools for the community outreach come from TAM, or did you develop some independently? Examples?] - What is your normal daily work schedule? (What transport do you have?) 3. Do you have a role in monitoring and reporting? Please explain. 4. How many community outreach workers are there in the Hebron Office? (Has the number increased or decreased over time?) How many communities do you personally work with at the same time? 5. What is your role in the project hand-off process? (inauguration ceremony, fun day, household surveys about utilization, satisfaction/complaints, households with special needs or very poor, training, awareness posters/sessions.) Examples? Do you have a specific pattern you follow or specific responsibilities at this point in the process? 6. Based on your experience, how do you assess the importance or value of the community outreach aspect of the PCID program? Can you give examples of situations where you think your work was essential? [Is there an example where no matter what you tried, it didn’t work?] 7. How could this role be improved? Infrastructure Engineer 1. What is your main responsibility on a project? Have you worked on various types of infrastructure or are you specialized for just one type (e.g., water networks?) 2. How many infrastructure engineers work out of this office? [Are you originally from this area? What kind of work did you do before joining PCID?] 3. At what point do you become involved with a project and when do you finish? - Do you review the design before it is finalized? - Are you involved in the tendering process? - Do you have any role in recruiting people to work on the construction? - Do you have any role in collecting data for monitoring and reporting in the PCID main office? - What is your role in the hand-off of the completed project? 4. Do you interact with the local officials or people in the community in any formal way? - I understand that often someone is identified by the community to provide oversight or quality control during construction? Is that your experience? Do you collaborate with this person? Who else works with you during construction? 93 - Are you at the construction site full time? Do you live on-site for the duration of construction? (If no – has that ever happened?) 5. What action do you take if you see a problem or an issue in the work of the contractor? (For example, what if the contractor is not working exactly as per design, e.g., using a different standard of materials?) 6. How do you deal with community concerns or complaints? - For example, some people have said they aren’t allowed to go near the construction site or to talk to the contractors. - What about complaints about noise or dirt, etc.? - [We have heard, for example, that no matter what they say, ANERA always sides with the contractor and ignores them?] 7. Clearly some projects take longer than others. What is the average time that you work at a project site? (longest/shortest) - Do you think that the PCID process is as efficient as other projects in getting the job done, or are there additional delays? Explain. 8. At what point do you finish with the project? Are you involved during the one-year after the building is finished? In your experience, are the contractors responsive in making the necessary corrections? 9. Do you have any other observations or comments about your experience with PCID? Any recommendations for improvements? Gender Outreach (TAM) 1. When I met with Suheir and May in Bethlehem, they suggested I talk with you about your on￾the-ground experience in gender outreach. Have you continued to fill this same role since the TAM sub-grant ended or have you moved to other activities? 2. When you were working with TAM, did you visit the communities and projects on a daily basis? Were you involved with all the PCID communities in the Hebron region? - Who did you work with most closely from the ANERA team? - Did you go with the ANERA team in their first visit to the community? What was your role in this visit? Who did you contact? What were you trying to learn? - To what extent, were you involved in the review of the project designs and the application of the gender checklist? Were the engineers open to your observations and suggestions? - In the later community visits, did you work through the Village Council to connect with women and persons with disabilities in the community, or did you seek out other contacts on your own (i.e., people who are not usually active in community decisions?) 94 - Did you find that the community leaders were open to your ideas about involving women in the project discussions and decisions? 3. One of the main tasks of TAM was awareness-raising. What did this involve? What tools were most effective – please give examples of how you used them? 4. In what ways (if any) did you work with the ANERA monitoring and reporting unit? Did you collaborate with them in collection of information on an on-going basis? What can you tell me about the TAM household surveys and the impact studies? 5. I understand that TAM had a big role in the activities at the time the project is finished and turned over to the community (open days, fun days, health awareness, training sessions about maintenance)? How did you go about organizing these days? In your opinion, what was the importance or influence of these activities? 6. In your opinion, what was the most important or effective result of TAM’s part in the PCID program and your work with the engineers and the communities? Explain and give examples. 7. From the other side, what aspects of TAM’s participation did you find disappointing or ineffective? Explain and give examples. How could these things have been improved? 8. Since the TAM sub-grant ended, do you think that ANERA has continued to apply the lessons learned from the partnership with ANERA? Explain. Do you think this approach should be a part of future infrastructure projects in the communities? 9. Do you have any other comments, observations, or recommendations? 95 INTERVIEW PROTOCOL FOR ANERA CHIEF CONSTRUCTION ENGINEER PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Was the session recorded? 1.Yes 2. No Number of years with PCID 1. At what point do you, as Chief Construction Engineer in Hebron, become involved with a particular project and the community? - Do you participate in the selection of projects or communities? (others in the field office?) - Is anyone from the community involved or consulted in the design and proposal stage? Does the village council (or the municipality) have to agree to move forward with the project before the design begins? What unit in PCID is responsible for interaction with the community at this stage? - Are any community-based/social factors included in the design decision process? (e.g., vulnerability, underserved, cost-share, rural vs urban, availability of labor for construction) 2. As Chief Construction Engineer, who is your main point of contact during the construction process? ANERA, PWA/MOEHE, design firms, contractors, community leaders 3. In your opinion, what is the main reason(s) that projects are not selected by PCID? [technical vs social]? - Why have projects been dropped after they have been included in the annual workplan? 96 4. The construction stage is managed by the field offices and the Gaza technical offices. The chief construction engineer is responsible for quality control, maintenance of construction standards, contract compliance etc. - Who else works in the field office and what is the role of each person? - - Does the oversight role include any examination of gender or other social factors? How does the chief engineer work with the other personnel in the field office (community outreach coordinator, gender coordinator)? - What is the nature of the engineer’s interaction with the community during construction? Who is responsible for managing complaints from the community (or another client), of contractors - Who or what entity has the final approval for the completion and hand-off of the project? (PCID, chief construction engineer, line ministry, community) 5. In our community visits, we encountered a difficult situation in Nahalin, in which the community is unhappy with the drainage project and the results. - From your point of view, what is the source of this dissatisfaction? - What steps did you/PCID take to deal with or correct this situation? - Where did the problem originate – original project request, design, construction? How did this happen? 6. Any other observations, recommendations, key factors I have missed? 7. Would it be possible to arrange a visit to the field office for Monday, 12 March? - Contact with site engineers, outreach workers, former TAM employee (Heldan) - Timing - Directions 97 INTERVIEW PROTOCOL FOR TAM (Women, Media, and Development) PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask Opening – background on TAM and the interviewee(s) Questions 1. TAM worked with ANERA on the PCID program for approximately four years. (Contract ended in June 2017.) How do you describe TAM’s role in the program? - At what point did you begin working with ANERA on PCID? - in the response to the RFA (i.e., program design, staffing design); project start-up (i.e., opening the field offices); added in when community interaction began; on separate tracks, where TAM made inputs to ANERA operation; integral/lead on non-technical aspect of program. - What unit or person was TAM’s principal contact/counterpart at ANERA? - Did TAM’s role change over the course of the program? - Did TAM encounter any limitations in working with ANERA? (things thought were important and couldn’t complete). Explain. (any due to lack of funds?) - What is your understanding for why TAM’s sub-grant was ended when the extension was granted? 98 2. In your mind, what were TAM’s principal achievements in the PCID – or contributions to the PCID design and implementation? 3. PCID adopted a very broad definition of “gender,” including all vulnerable or underserved groups under this term. Gender inclusion is used as the equivalent of social inclusion. How was this definition developed? Did it affect the program implementation or was it used mainly for reporting purposes? What effect did it have on attention to gender relations in the more narrow/traditional sense? 4. One of the topics of interest to USAID in this evaluation is the PCID community outreach model. I understand that TAM had a central role in the development of this model. (Is this true, or is it common to ANERA projects?) - In your view, what are the critical elements of this model? What was the role of TAM (including field office staff) in developing and implementing this model? [TOOLS – examples￾documentation] - Did all projects start with a needs assessment? How was it done? Was it effective? - At what stages, did TAM have direct contact with the communities? What was the purpose and content? Was the pattern the same for all types of projects? [needs assessment – design – construction – maintenance – community ownership and sustainability] - Did you encounter resistance to this objective or other problems in the communities related to attention to vulnerable groups? - Was TAM involved in community data collection, monitoring, and measurement of before and after change? Discuss the methods of data collection and reporting. - To what extent were the findings and recommendations that resulted from this process applied in the projects? examples 5. The PCID is distinguished from other USAID infrastructure programs as a community-level program for vulnerable and underserved communities through small and medium projects to enhance availability and accessibility of public services. - In your opinion, to what extent do these projects help in resolving critical issues in the region? What can other potential activities be integrated into the process in general to enhance anticipated results? 6. Any other comments on PCID programs and the process? 99 INTERVIEW PROTOCOL FOR THE PARTNER ENGINEERING FIRMS PCID FINAL EVALUATION March 2019 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent 1.Yes 2. No 3. Did not ask 1. Background on the firm (when started, size, scope of activities, work done on PCID with ANERA, types of projects). Normally, does this firm specialize in a particular type of project (e.g., water, public buildings, public spaces)? 2. What is the role of this firm on PCID projects? - Who is your principal contact at ANERA? - What does this firm receive from ANERA (e.g., concept design, design specifications, etc)? - What is the product that the firm is contracted to complete? What is included in the design proposal? - Does the firm interact with the communities receiving the projects? If yes, when and in what way? - Does the firm interact directly with the USAID engineers? If yes, when and in what way? - Describe the situations in which the firm may be required to make revisions to its design? - When does the firm’s role in the project end (i.e., their contracted obligations; the point at which they sign off and don’t make anymore revisions)? - Does the firm have any role in quality control or oversight of the actual construction? Describe. 3. In your opinion, is the work of the partner engineering firm fully utilized by ANERA? Do you have recommendations for how this process could be improved, especially in terms of the quality and functionality of the final infrastructure product? 100 4. Tell us about major challenges/problems(s) you have faced in your work on the PCID Program during: - Design preparation? - After the final design is submitted? - During project implementation/construction? - When the project is commissioned? - After the project is operational? - How did you deal with these problems? Were they fixed? Did ANERA help in this? 5. Were there any already available designs for the PCID projects that you received from ANERA or the local government unit or others? Did you use them? Why or why not? What was the quality of these designs? 6. Tell us more about the total number and type of projects you designed for the PCID? Was the design prepared completely by your staff or did you have support from other firms to complete the design? Were they all implemented? 7. Did you receive any report from ANERA or the local government about major issues in the implemented projects after they were done? Were the issues serious, in your opinion? We they resolved? 8. Do you have any other comments about your experience in preparation of infrastructure project in the PCID program – in general, with ANERA, the communities? 9. Tell us about any lessons learned during your contract with ANERA. 101 Questions for USAID PCID Evaluation March 2018 Program Office: - Expectations from evaluation in terms of: o Central themes – community outreach model; importance of this kind of infrastructure; value of the PCID as a mechanism o How do you want it to be used by the Mission? Are there other audiences besides Program and EGI Offices? o Purpose/audience for stand-alone executive summary - Indicators: o PCID met few of its targets, and yet continued to receive funds and a one-year extension o How were the indicators/targets developed? Were they reviewed and revised annually? o Why is “full-time equivalent employment days” tracked and reported as an indicator? It does not seem to have any relationship to the PCID results framework? - HIMII – How did PCID end up with the HIMII projects? [was it created, in part, as a mechanism for this purpose?] - Links to be made to Mission strategy? What is the strategy and where is it documented? - Special issues in implementation in Area C and Gaza: cost, time, community involvement - What is a vulnerable community and/or marginalized area? To what extent was this a distinguishing factor in the PCID community selection, particularly compared to other EGI activities? - What was the process for identification of projects, by sector? What was the relative importance of source of request, cost, sector, geographic location, ease of approvals, cost￾sharing, etc? What was process for inclusion and for setting priorities. [e.g., Did PCID make choices among projects and/or communities?] Health Office: - Part of PCID results framework is to serve as a vehicle for other USAID programs: How did interact with PCID (process)? Did it work - strengths and weaknesses? Specific examples. - What do you want to learn from the evaluation? Suggestions for specific things we should consider/pay attention to? - - Discuss the experience with the clinic in Al Walaja. (origins of project; issues with community, MoH, problems now with staffing and utilization) 102 - - Relationship with MOH – communities talk about the need for clinics – PCID worked closely with MOEHE but not MOH Education and Youth: - Part of PCID results framework is to serve as a mechanism for other USAID programs? How did interact with PCID? Did it work - strengths and weaknesses? Specific examples. - According to ANERA, strong direct interaction with MoE. Where does Education and Youth fit into this? - Getting ready to visit the youth center in Jericho. How did that come about – process - assessment – repeat the experience? Was the broader community/municipality involved at all? - What do you want to learn from the evaluation? Suggestions for specific things we should consider? Contracts Office - Key issues for contracts in infrastructure project? Are there additional issues, unique to community infrastructure? - Proportion of costs to administration vs infrastructure projects – how does PCID compare to other infrastructure projects – justification – Will this type of infrastructure program be less viable with a reduction of/curb on USAID/WBG funding? - Does contracts office have a role in funding of community-level projects? What is that role? What kind of over-sight or audit is done of the community projects? - ANERA was to contribute $15,000,000 in counterpart funds – in funds, in-kind? What happens if this obligation is not met? - Effect of earmarked funds on PCID program in terms of pipeline and overall achievements? (Gaza, water, specific projects transferred from another program) Engineers 1. Describe your role in the PCID community infrastructure projects. - Have you been involved in all the projects? (types of projects, costs) - When do you first enter the process? (concept design, negotiations with PWA, project approval, design approval, tendering, hand-off? - What are the main things you examine in your review? - Do you visit the project sites? (differences between West Bank and Gaza?) 103 - Do you take budget and costs into consideration in your review? - On average, how long do you spend on your review of the project? On average, how much time passes between when you receive the project and when you take your final action? - Is it the same as under the previous community infrastructure program with CHF/Global Communities? If no, what were the differences 2. Do you have the power to reject/kill a project? Did you ever kill a PCID community infrastructure project? (explain, example) 3. How does your role for PCID community projects differ from your role in other USAID infrastructure activities? - What is the difference in terms of your role and authority for infrastructure projects implemented under a cooperative agreement and those implemented under contracts? - Which process takes longer, on your part? 4. Where are you located in the USAID/WBG organizational structure? 5. In our visits to completed project sites we have encountered several cases where there appear to have been flaws in the project design (e.g., a drainage project that does not drain.) We have been told that the design comes from a registered engineering firm, it is approved by ANERA engineers, and by USAID engineers. Where is the breakdown in this process? 6. Recommendations (efficiency, community satisfaction, quality, cost, other) PCID AOR 1. Based on the four evaluation questions, the main focus of this evaluation is the community aspect of the program. It seems that PCID is distinguished from the other Mission infrastructure activities by the fact that it is community-based and limited to small and medium projects. From your point of view, are there other important differences between this activity and others that should be taken into account? - What do you, as AOR hope to clarify or learn from the evaluation? - Strengths is-à-vis other infrastructure activities? 2. PCID is a cooperative agreement and therefore ANERA has a cost-share. - To what extent has ANERA complied with this requirement? - Do you see any value of the cost-share factor for PCID programmatically? - What is your assessment of the value of the cost-share of the communities? Have they complied? 3. Indicators: - PCID met few of its targets, and yet continued to receive funds and a one-year extension - How were the indicators/targets developed? Were they reviewed and revised annually? 104 - Why is “full-time equivalent employment days” tracked and reported as an indicator? It does not seem to have any relationship to the PCID results framework? 4. PCID has not and will not spend the total budgeted amount of $115,000,000. What are the most important factors in explaining this shortfall? Have the other infrastructure projects faced these same issues? 5. Proportion of costs to administration vs infrastructure projects – how does PCID compare to other infrastructure projects – justification – Will this type of infrastructure program be less viable with a reduction of/curb on USAID/WBG funding? 6. The PCID Results Framework includes both infrastructure activities developed through ANERA’s relationship with the PA ministries and activities developed through other USAID program areas, especially education, youth, and health. PCID is an infrastructure activity for Water and Sanitation and Social Assistance, and a mechanism for the Mission to use to achieve other objectives. - How do you assess the effectiveness of the working relationship between ANERA and the PWA/CMWU? What about the Ministry of Education and Higher Education (identifying needs and priorities; maintenance and sustainability)? Have changes been made in these relationships during the course of PCID? - How do you assess the relationship between ANERA/PCID and the other offices in USAID? (e.g., Youth Center in Jericho; Area C projects) - What is the importance of PCID operations in Gaza, when there are no possibilities for community involvement? - Do USAID projects take precedence over other projects? What about things like HIMII? Can you talk about any situations in which USAID allocated funds to PCID that needed to be spent in the short-term for some reason? (Randy – PCID as a flexible mechanism for the Mission.) 7. In terms of the process of design, implementation, and handover of projects, my impression is that overall the communities are pleased with the results. However, in several situations we have been told about problems in project design, and problems with the construction contractors. - In our visits to completed project sites we have encountered several cases where there appear to have been flaws in the project design (e.g., a drainage project that does not drain.) We have been told that the design comes from a registered engineering firm, it is approved by ANERA engineers, and by USAID engineers. Where is the breakdown in this process? - Don’t have actual numbers yet but impression is that communities were unhappy with their experience with the contractors – lack of openness, transparency – ANERA took the side of the contractors. Does ANERA have the responsibility and capacity to make sure the contractors are performing well? - In your opinion, what effect has the adoption of the IBC (International Bldg Code) had on the quality of the PCID projects? 105 106 INTERVIEW PROTOCOL FOR PWA – PCID STEERING COMMITTEE PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent Yes No Did not ask Questions 1. Please describe how the PWA and ANERA work together on PCID and the community water and sanitation projects. - Does PWA have a list of projects that should be addressed in specific communities? - Where does the list of community needs and projects originate? (with the communities? PWA? other?) How does the PWA prioritize the community needs? What criteria are used in this process? - We understand that PWA, ANERA/PCID, and USAID have a steering committee that meets formally on a regular basis. Please describe the purpose and format of these meetings. - What types of decisions are made by the steering committee? What role does the PWA play in the selection of a particular project/community for PCID action? - Does PWA have a similar relationship with other donors besides USAID? - - Typically, how much time passes between when the idea for a project is first proposed (by a community or PWA and its completion? 107 2. What is the role of the PWA in the implementation of projects? (Design, site selection, legal arrangements, license, construction oversight…)? 3. What, if any, problem/s has PWA encountered during the selection/design/implementation/maintenance of ANERA/PCID projects? Please explain. 4. Does PWA follow-up with the communities after completion of the projects? Have the ANERA/PCID projects met the expectations of PWA? Are they fully utilised? How do you rate the water and sanitation projects completed under PCID? Please give examples. 5. How do you assess the importance of these community water and sanitation projects relative to other water/sanitation needs and projects in the West Bank? Do they meet critical needs? Are they sustainable under current circumstances? Do you think there are more critical projects that should come first? What about the situation in Gaza? - What is another urgent need in the water sector? 6. For the PCID projects, does the PWA have a role in the operation and maintenance? What about the resources (water, financial, human, information, technical, etc.) needed for maintenance? 7. What other activities or groups could be integrated into the process for selection, design, and construction of PCID projects? For example, what is the role of the local officials and project beneficiaries in the communities? To what extent is the community responsible for utilization and maintenance of the new infrastructure? (Does PWA do any training for this purpose?) 8. Any other comments on PCID programs and the process? 108 INTERVIEW PROTOCOL FOR MoEHE PCID Final Performance Evaluation March 2018 Hello. My name is ___________________ and I am a member of an independent research team contracted by USAID to carry out the end-of-program evaluation for the PCID program. This type of external evaluation is a standard practice for USAID during the final year of a program. The purpose is to identify the successes and challenges PCID has encountered as well as the strengths and weaknesses of the program’s approach to community infrastructure. The evaluation is especially important to USAID for designing future community programs in Palestine. I appreciate this opportunity to meet with you. Please be confident that your responses will remain confidential and anonymous. The content of this interview will be used only in combination with those of other stakeholders. You will not be identified or quoted. No personal or identifying information will be reported. Before we start, do you agree to allow the PCID evaluation team to use the contents of this interview for the evaluation report? ____ yes. ____ no. (Interviewer, please record the person’s verbal response.) Date of interview Interviewer Title of position Gender of interviewee 1. Male 2. Female Verbal response to consent Yes No Did not ask Questions 9. Please describe the way in which the Ministry has worked with ANERA and USAID on the PCID program. - What was the process for selection of the schools for construction or rehabilitation under PCID? What were the criteria? - What is your understanding of why the selection only occurred once? - Did the MOEHE select the communities for PCID or was it a mutual process? What were the steps and the criteria? How long did it take to reach agreement? - How does this process compare to the way MOEHE works with other donors? 10. Please describe the MOU (Memorandum of Understanding) between the MOEHE and ANERA/PCID. - Was this signed at the beginning of the PCID program or later? - What is the MOEHE contribution under this agreement (e.g., finance, services, staffing and supplies, maintenance)? - What is the ANERA/PCID contribution? - Does the MOEHE have similar agreements with other donors supporting school construction? 109 - Has the MOU been effective? Has it resulted in any problems for the MOEHE? Has it ever delayed school construction or rehabilitation under PCID? 11. How does the MOEHE create the list of schools? Does the MOEHE prioritize the needs and communities according to a specific strategic plan? What are the criteria? - Typically, how much time passes between when the idea for a project is first proposed and its completion? 12. What is the role of the MOEHE in the implementation of projects? (Design, site selection, legal arrangements, license…)? 13. As far as you know, is there any direct coordination between the MOEHE collaboration with the USAID Education Office (quality of education, training) and the school construction? 14. What can you tell me about significant problem/s that the MOEHE faced during the selection/design/implementation/maintenance of projects? Please explain. Were these issues satisfactorily resolved from your point of view? 15. Generally speaking, have the ANERA/PCID projects met the expectations of MOEHE? Are they fully utilised? Please give examples. - Has your experience with PCID and USAID been similar to your experience in working with other donors? 16. To what extent do these projects help in resolving critical issues in the West Bank? - How do these projects to compare to others in terms of critical needs of the communities or location (i.e., are they more likely to be in remote or vulnerable regions?) 17. Who or what organization is responsible for maintenance of the schools – the school itself, the village council, the MOEHE, other? - Please give examples of situations in which maintenance has been a problem and others when it has been done well. - What about the availability of financial resources, parts/equipment, technical knowledge? 18. Do you think there are more critical projects that should come before school construction and rehabilitation? Please give an example of another urgent need in the education sector? - During the past five years, has the MOEHE worked with other donors or its own resources to confront these other needs? Examples. 19. Do you have any other comments on PCID programs and the process, or recommendations for potential future collaboration? 110 ANNEX D: LIST OF DOCUMENTS REVIEWED USAID/PCID Program Documents • Annual Reports: FY 2014, FY 2015, FY 2016, FY 2017 • Annual Work Plans: March 2013, October 2014, October 2015, October 2017 • Annual Performance Management Plans (PMPs): March 2013, October 2014, October 2015, October 2016 • Quarterly Reports: No. 15, 16, and 17: October 2016 – June 2017 • Quarterly Report No 19: December 2017 Cooperative Agreement No. AID-294-A-13-00005-00 Palestinian Community Infrastructure Development (PCID) Program Cooperative Agreement Modifications No. 1 – 17 Office of the Inspector General, Audit of USAID/West Bank and Gaza Construction Programs, Audit Report No. 8-294-16-001-P, February 22, 2016 Memorandum of Understanding between ANERA and MOEHE, March 5, 2014 Memorandum of Understanding between ANERA and PWA, February 18, 2014 Gender Integration in the Palestinian Community Infrastructure Development (PCID) Program, Women Media and Development (TAM) USAID GEO-MIS List of PCID Sub-Activities, February 5, 2018 USAID Program Indicators Report for PCID, February 5, 2018 PCID Project Selection Criteria USAID/West Bank and Gaza Revised Mission Order No. 202, Subject: Review and Oversight of Construction Activities under Mission Awards, September 28, 2016 111 ANNEX E: LIST OF STAKEHOLDERS INTERVIEWED KEY INFORMANT INTERVIEWS ORGANIZATION POSITION ANERA Chief of Party Deputy Chief of Party Technical; Chief Construction Engineer Deputy Chief of Party Non-Technical Chief Engineer/Design Development Chief Construction Engineer Southern West Bank Gaza Technical Manager Reporting and Monitoring Manager Reporting and Monitoring Specialist Infrastructure Engineer, Hebron Office Community Outreach and Gender Worker, Hebron Office Women Media and Development (TAM) Executive Manager Programs Manager USAID/West Bank and Gaza Program Office (4 officers) Democracy and Governance (1) Education and Youth Office (1) Engineering (1) Contracts Office (2) PCID AOR and Deputy, Economic Growth and Infrastructure Office Palestinian Water Authority (PWA) Representative to PCID Steering Committee Ministry of Education and Higher Education Engineer for School Construction Unit Coastal Municipalities Water Utility (CMWU) Deputy Head of CMWU ANERA Partner Engineering Firms Center for Engineering and Planning (Gaza) Habash Consulting Engineers Diyar Consultants Co. 112 COMMUNITY KEY INFORMANT INTERVIEWS AND DISCUSSION GROUPS COMMUNITY TYPE OF PROJECT INTERVIEWS DISCUSSION GROUPS Bil’in Water reservoir and rehabilitation of network Deputy Head of Village Council (M) Beneficiaries Reservoir (3 M, 3 F) Girls’ School School Principal (F) Teacher and Parent (F) Qaffin Girls’ School Mayor and Former Mayor (M) Beneficiaries/Parents (2 M, 6 F) School Principal (F) Zeita park Mayor (M) Beneficiaries (1M, 3 F) Municipal Engineer (F) Attil Water reservoir Mayor (M) Beneficiaries (7 M, 4 F) Habla Waste water main line and connections Mayor (M) Beneficiaries (2 M, 4 F) Head of Projects and Engineering Unit (F) Jericho Youth Center Board Chairman (M) Former Board Chrm (M) Administrator (F) Municipality Engineer (M) Qabatia Rainfall drainage system Mayor (M) Beneficiaries (1M, 5F) Municipality Engineer (M) Nahalin Drainage system Mayor (M) Beneficiaries (4M) Municipality Engineer (M) Hitta School and Kindergarten Head of Village Council (4) Members of Village Council (4M) School Principal (F) Al-Walaja Clinic Hear of Village Council (M) Member of Village Council (M) Administrator of Clinic (F) Khallet Salih Water network Mayor (M) Beneficiaries of the network (6M) Member of Council (M) COMMUNITY TYPE OF PROJECT INTERVIEWS DISCUSSION GROUPS Al-Mosaddar Rehabilitation of Sports Club Board Chairman Sports Club (M) 113 Board Chairman Association for Rural Development (M) Board Members (5M) Local community leader (M) Upgrading of Roads Head of Engineering Department (M) 2 school girls, users of road (@F) Mukhtar (M) Khan Younis PRCS Rehabilitation Center Director of Center (M) Beneficiaries, people with physical disabilities (3M, 4F) Head of Engineering Department (M) Beneficiaries with disabilities (hearing; learning) (4M, 3F) Board Member (M) Administrative Director (M) Wadi Fukin Upgrading internal road Head of village council Imneizel Construction of water reservoir and network Mayor and one other person (2 M) Rantis Elevated water reservoir Head of village council (new) and council technical person (2M) Beneficiaries (representatives of a CBO, School Principal, LGU members, farmer) (2F, 3M) Khoza’a Rehabilitation of water network Engineers from CMWU (2M) Muktars from the community (2M) Beneficiaries/users (4M, 2F) Gaza City YMCA – construction of a playground and rehabilitation of the youth facility YMCA General Secretary (M) Chairman of YMCA Board (M) Youth YMCA members (8M) 114 ANNEX F: EVALUATION MATRICES The following matrix of methods for data collection and analysis for the four evaluation questions was included in the Evaluation Scope of Work. It was used as guidance for the following set of matrices by evaluation question, developed by the evaluation team in preparing the Expanded Scope of Work (deliverable 1) and the data collection tools. Evaluation Question 1: To what extent did the implementation of PCID activities (infrastructure projects) support the project development hypothesis and intended results? Sub￾questions/ topics Data source Data collection method Sampl e Tool required Analysis and [pending thoughts] Developme nt Hypothesis: Improved social/econ omic developmen t through increased access to water and sanitation [affordabilit y, availability] and to community services, through infrastructu re projects. Analysis will look at PCID results relative to USAID/WBG infrastructure strategy, with focus on community-level projects. IR1: availability and affordability of water and sanitation PCID Indicator tracking sheets and related PCID documents . USAID/PCI D files Entire project Document review Comparison of actual results to targets. Examination of trends over time and across type of projects; underlying complicating factors and explanations. 115 PCID database (M&E manager and staff) Source of request; size; location; pre- and post￾project utilization data All commu nities Spread sheet Factors in decision for PCID implementation vs other USAID (or other entity) infrastructure activities, i.e., why PCID? Perceptions of increased availability and affordability for the community and for individual beneficiaries [Relationship to community cost-sharing.] ANERA/P CID staff and partner engineering firms. ANERA/P CID and TAM staff concerned with post￾completion assessment KII and/or group interviews ANERA and enginee ring staff involved in design and approval . ANERA and TAM staff for post￾complet ion activitie s ANERA/enginee rs design interview protocol Interview protocol for post-completion staff Communit y interviews and household survey Interviews with leaders; focus groups with CBOs, women, youth, etc. Survey with sample of households across the community TBD 10-12 commu nities West Bank and Gaza Comple ted projects -Interview protocol for community leaders -Focus group protocol for community reps. -Household survey questionnaire IR2: Access to community infrastructu re facilities Results compared to targets. Factors in requests that were declined. 116 -support USAID programs PCID database; Project source of request, location, size, status, evidence of utilization; requests declined and why All commu nity projects Spread sheet Comparisons to other sources of request, e.g., ANERA contacts, direct requests from communities or other official entities CORs/AO Rs in health, education, DG, EG Individual/g roup interviews (or questionnai re) – request process, PCID responsive ness, result for USAID program, wellbeing, gender+ Interview protocol or questionnaire -access [utilization? quality? location?) to community infrastructu re facilities Communit y interviews, focus groups, household survey. Observatio n and interviews in the facilities. Same content to community visit as above. Visit to the project site. 10-12 commu nities Project sites in the selected commu nities (non￾water and sanitatio n) Same as above Protocol for site visit observation and discussions PCID database Source of request; size; location; pre- and post￾project All projects 117 utilization data ANERA/P CID staff perception s KIIs/group interviews All projects Interview protocol for top leadership -enhanced well-being through response to urgent needs; rapid response Perception s in the communiti es KIIs, focus groups, beneficiary survey Responses/analysis here are also related to EQ2 Was the PCID rapid and responsive to urgent needs? What types of projects were most effective in enhancing well-being USAID, ANERA/P CID staff, TAM – perception s of changes in well-being; contributio n to social/econ omic developme nt KIIs and group interviews PCID as a whole, Interview protocols for KIIs PCID database Duration of process to complete community project Employme nt generated as a benefit 118 Evaluation Question 2: To what extent did the project succeed in identifying and addressing community infrastructure needs in various sectors? Sub￾questions/topics Data source Data collection method Sample Tool required Analysis and [pending questions] To what extent did the project selection criteria contribute to achievement of community-level results? [By sector and changes over time] At what point in the project process were the criteria applied? PCID document review/project database Measure time from start to completion by sector PCID as a whole spreadshee t Examine the use of the selection criteria from the point of view of efficiency (rapid response) and flexibility (responsivenes s) in providing community￾level results. Implementers, especially design team In-person individual and group interviews Interview protocols Community leaders and CBOs, and beneficiaries In-person individual/ group interviews; household survey Interview and focus group protocols questionnai re What was the process for identification of projects, by sector? What was the relative importance of source of request, cost, sector, geographic location, ease of approvals, cost￾sharing, etc? What was process for inclusion and for setting priorities. [e.g., Did PCID make choices among projects and/or communities?] Implementers/ USAID In-person interviews PCID as a whole Interview protocols What were the priorities and to what extent were they achieved? Changes/trend s over time. At what point did the project selection criteria come into play? PCID document review Content analysis Citations/r eferences What was the process for selection of the communities? What was the relative importance of the source of the Implementers/ USAID In-person interviews PCID as a whole Interview protocols Identify the community selection process, with comparisons between the ideal and actual, and Observations based on mapping of projects; PCID document review Content analysis Citations; maps 119 request, the location of the community, the degree of marginality, the presence of other assistance activities? Other considerations? identification of intervening factors and their effect on results. Trends/change s over time How effective was the PCID in addressing community needs in Area C? What factors explain these results? Project documents Extract from project/communi ty database PCID as a whole Spreadshee t Achievements, by sector, in terms of number of communities, projects, beneficiaries, work days. Ratio of actual projects to requests. Implementers and counterpart organizations (i.e., PWA, ministries) In-person and group interviews Interview protocols What is a vulnerable community and/or marginalized area? To what extent was this a distinguishing factor in the PCID community selection, particularly compared to other EGI activities? Socio￾economic secondary data Extraction from public secondary sources PCID as a whole Spread sheet Define quantitative metric for vulnerable and marginalized and rank PCID communities. Comparison to other USAID activities. Qualitatively, relative weight given to this factor. Differences by sector Trends/change s over time. Implementers/ USAID In-person interviews Interview protocols What are the key components of the PCID community outreach model? What are the strengths and Document review Content analysis PCID as a whole citations Describe the outreach model in theory and in practice. Implementers, especially in field offices; TAM In-person individual and group interviews Interview protocols 120 weaknesses of the model and components? What was the effect of the outreach model on results? Community leaders and CBOs In-person interviews and focus groups Selected communiti es KII and focus group protocols Identify and analyze the on-the-ground factors that affected the application of the model and its impact on results. Changes/trend s over time Beneficiaries Household survey questionnai re To what extent did the PCID respond to and add value for other USAID programs in health, education, etc.? What, if any, was the utility of the PCID community outreach model in achieving these results? Document review Extract information on number, size, location of projects linked to USAID program requests PCID as a whole Spread sheet Was the PCID impact significant – necessary/suffi cient – in achieving other results? How would these infrastructure requirements have been met without the PCID? In what ways are these projects different from other PCID projects? USAID In person interviews to discuss cases, process, perceived effect of PCID activities for other USAID objectives Interview protocols Community visits Interviews/focus groups with leaders and CBOs; household survey Interview, focus group protocols; survey questionnai re 121 Evaluation Question 3: To what extent have recipient institutions utilized and sustained the completed infrastructure projects under PCID? Sub￾questions/topics Data source Data collection method Sample Tool required Analysis and [pending thoughts] What entity (and individual) is responsible for maintenance, under each type of project? Are there differences by geographic location, and/or West Bank and Gaza? What is the source of funds for maintenance and repair? Community visits Individual and group interviews with community leaders and PCID regional staff 10-12 communities and PCID regional offices Interview protocols To what extent is maintenance and sustainability systematic and applied? Differences across communities and types of projects? PCID regional office files and/or interviews What is the actual state of maintenance? Observation of project sites and completed infrastructure Visit to project sites and conversations with on-site staff. 10-12 communities Observation checklist Evidence of maintenance and sustainability – comparisons across types of project and time since completion PCID regional offices and follow-up reports Individual and group interviews in regional offices (including TAM) Interview protocols To what extent does the actual level of utilization match to projected (targeted) utilization in project planning? PCID project files, especially project planning and design documents; baseline utilization measures if available Document review and interviews in PCID regional offices PCID regional offices Spreadsheet and interview protocols Adequacy of the response to the documented need for the infrastructure; other factors that may be affecting actual use; trends in use over time. Observation of utilization at project site Community site visits 10 – 12 communities 122 From the point of view of the beneficiaries, how much do they use the completed project and how do they rate its importance for them and for the quality of life in the community? Community site visits Household beneficiary survey 10 -12 communities Survey questionnair e Quantitative and qualitative measures of the value of the infrastructure for the community. Trends over time Focus group discussions in the community Focus group protocols What was the extent of application of the community outreach model in terms of a focus on sustainability? PCID documentat ion of outreach strategy and application Accounting of events and other activities from the monitoring reports, especially from TAM PCID as a whole spreadsheet Examination of the community outreach model activities for the PCID overall. Actual evidence of the model on the ground and in perceptions of the beneficiaries Observatio n of posters, etc Number of meetings, other activities in community Community site visits, beneficiary household survey 10 -12 communities Focus group protocols; Survey questionnair e To what extent does the community and beneficiary households exhibit a sense of ownership of the infrastructure? Community site visits Individual and group interviews, focus groups, and beneficiary survey 10 -12 communities Interview and focus group protocols; Survey questionnair e Comparative analysis of communities across time, and type of project To what extent does the cost-share component affect the sense of ownership and PCID project files on maintenanc e. Documentation of cost-share by project (actual vs target) PCID as a whole spreadsheet Comparison of level of cost￾share to actual outcomes in terms of maintenance 123 project sustainability? Community site visits Individual and group interviews, focus groups, and beneficiary survey 10 -12 communities Interview and focus group protocols; survey questionnair e and utilization, and to perceptions expressed by the community an beneficiary households. Evaluation Question 4: What community water, sanitation and infrastructure needs exist that could be supported by USAID in the future? Sub￾questions/topic s Data source Data collection method Sample Tool required Analysis and [pending thoughts] What other potential activities are identified to support water, sanitation and other infrastructure at the local level? Community leaders, counterpart orgs, CBOs (especially women, youth, marginalized groups), PCID implementer s Individual and group interviews, community -level focus groups, Beneficiary household survey 10-12 communities for community￾level interviews and survey; West Bank and Gaza as a whole for implementer s and counterpart organization s Interview and focus group protocols; survey questionnair e Trends in terms of areas of need and types of projects Differences/similaritie s among sub-groups in terms of areas of need. Geographic variations To what extent is there evidence of potential community-level commitment and support (include cost-sharing) for various types of projects? Community leaders, counterpart orgs, CBOs (especially women, youth, marginalized groups), PCID implementer s Individual and group interviews, community -level focus groups, Beneficiary household survey 10-12 communities for community￾level interviews and survey; West Bank and Gaza as a whole for implementer s and counterpart organization s Interview and focus group protocols; survey questionnair e Need for/support for community-level support as opposed to other levels (e.g., municipality, governorate). Is there any perceived clear value-added in working at the community level? 124 ANNEX G: SELECTED BACKGROUND DOCUMENTS The following documents are attached as resources and references for the evaluation report. PCID Selection Criteria by Sector PCID Final Report from Gender Media and Development (TAM) 125 Gender Integration in the Palestinian Community Infrastructure Development (PCID) Program Women Media and Development (TAM) Final Report For the Period of July 2013 – June 2017 126 Executive Summary Due to the significance of achieving gender equality as a requirement for sustainable development in recent years, gender integration became mandatory in designing development programs and projects. ANERA has collaborated with TAM in order to integrate gender needs in its Palestinian Community Infrastructure Development program (PCID). Following this partnership, TAM and ANERA worked together to integrate a gender-relevant methodology to confront and fill the “gender gap” in infrastructure. This methodology contributed to “promoting gender equality and empower women”, according to the United Nations Third Millennium Development Goals (MDG). 25 Moreover, such interventions will ensure fulfilling the needs of different gender groups and ensure safe accessibility to water and sanitation as specified the United Nations Fifth and Sixth Sustainable Developmental Goals (SDGs). 26 The overall objective for integrating gender into the PCID Program is to promote gender equality in the Palestinian community. This objective was achieved by giving gender groups equal opportunities to access to resources and by having effective participation at the community level and in the decision-making process in order to identify their needs and priorities with regards to infrastructure services. During the planning, design and implementation phases of the project, TAM team provided gender integration recommendation and followed up on 28 infrastructure projects to ensure integrating gender into the project through gender recommendations, by observation and taking notes from beneficiaries in the planning phase. Many of these recommendations were taken into account in the design and implementation of the projects, with continuous follow up by the gender specialists with the design unit and on the site. This report includes a description of project successes and measurable impacts based on the project methodology, immediate results and indicators, as well as a summary of all activities, lessons learned, challenges and recommendations. Based on the measureable indicators of the project and the Impact Assessment research that was conducted in the late stages of the sub-grant agreement, TAM has significantly succeeded in achieving gender integration into PCID infrastructure projects. This success was proved by the change in gender roles for women and PWD’s in all project intervention areas, gender groups’ empowerment to access to resources, and the increased participation of gender groups, especially women, at the community and decision-making levels. Moreover, the awareness activities enabled women to raise their voices and express their needs and wants, and raised the communities’ awareness about the importance of rationing water consumption and being more socially responsible for maintaining project facilities. Chapter 1 explains the gender integration methodology utilized for the PCID Program. Chapter 2 gives a more comprehensive discussion of how the project successes and measureable impacts were accomplished. Chapter 3 presents a summary of the conducted 25 http://www.undp.org/content/undp/en/home/sdgoverview/mdg_goals.html 26 http://www.undp.org/content/undp/en/home/sustainable-development-goals.html (SDG 5: Achieve gender equality and empower all women and girls. SDG 6: Ensure availability and sustainable management of water and sanitation for all) 127 activities that significantly contributed to reaching the project overall impact. The report concludes with challenges and recommendations. The following table depicts all activities conducted during the project period, and the total number and disaggregation of participants. Gender Community Participation Total Activities Total Participants Males Females PWD Awareness Raising Workshops 9 447 175 272 14 Educational Open Days 6 2,123 945 1,178 4 Teachers Workshops 2 35 20 15 0 Health Day 1 112 42 70 0 Impact Assessment Study 1 201 109 92 3 Coordination Meetings 119 577 284 293 8 Field Visits 271 1503 703 800 0 Participated in Community Gatherings 34 1764 1,024 740 54 Total 6,874 3,344 3,530 87 Percentage 48.6% 51.4% 1.3% 128 1. CHAPTER ONE: GENDER INTEGRATION METHODOLOGY TAM believes that the project concept of integrating gender in infrastructure was the first of its kind in Palestine; hence, there was there was no baseline information to develop or previous experiences to rely on. TAM and ANERA developed the project concept and methodology, by conducting regular meetings and brainstorming sessions to develop the most effective method of intervention. Moreover, TAM has conducted a capacity building training for ANERA’s staff about gender equality and integration, in order to ensure mutual understanding and raise the awareness of the staff about gender equality. During the preparation stages of the project, TAM worked tirelessly to build the project baseline data in order to set the project methodology, by conducting the following activities: • Gender gaps identified • Building indicators based on gender gaps • Review gaps, assessment and indicators • Gender groups needs assessment in field • Gender Checklists during the preparation of the project • Building capacities and create a mutual understanding • Developing gender tools for assessment in the design and implementation phases. • Data collection and analysis based on indicators • Gender recommendation raised by gender specialists to the design unit • Follow up on gender recommendation during the implementation phase. • Raise the awareness of targeted community on the social responsibility in maintaining public property and resources 129 As a result, and based on the outcomes of the activities mentioned above, TAM set the project methodology based on the definition of gender integration, as shown in the figure below: 2. Chapter Two: Project Results 1. Overall Impact: To achieve the project overall impact “Gender Integration Equality within the Palestinian Community”, and based on the project methodology, TAM have set a number of performance indicators and assigned immediate results to each indicator. The immediate results enabled project staff to monitor and evaluate activities in order to measure whether TAM was able to achieve the project impact. The figure below shows the project impact and its two corresponding performance indicators: Gender Integration In the PCID Equal access to resources (economic, political, education, information, internal or personal, social, and time) Equal opportunity in decision making Equal participation (Identification of needs, plans and monitor) To use/access resources 130 2. Performance Indicators: 2.1 Performance Indicator 1: 40% of PCID implemented projects offer equal opportunities and benefits for males and females in infrastructure projects design, data collection, and implementation. TAM’s baseline data showed that women’s participation in infrastructure project processes was zero prior to project implementation (based on information provided by municipalities at the beginning of the program). However, during project implementation (preparation) period and as a result of TAM’s gender integration strategy to increase women’s participation in community and awareness gatherings, 119 municipality and local councils meetings (coordination meetings) were conducted, with the participation of 577 of local government unit representatives, of whom 293 were women. Moreover, 271 field visits were conducted with the participation of 1,503 people, of whom 800 were women. Naja’ Abu El-Shaar (aged 53) from Jinsafut said: “The activities that TAM conducted gave us the opportunity to participate in events such as the community gathering, especially that women are main responsible for the household. She should be the one participating in such activities. Before TAM’s intervention, women were neglected and never called on to participate in community activities.” Overall Impact Gender Integration within the Palestinian Community Performance Indicator 2: 30% of PCID projects (water and sanitation) save time for the welfare of gender groups (change in gender roles) Performance Indicator 1: 40% of PCID implemented projects offer equal opportunities and benefits for males and females in infrastructure projects design, data collection, and implementation 131 Thus, after project implementation and gender integration, female participation at decision-making levels increased from 0% to 51%, and women participation in field visits increased from 0% to 48%. Heiam Hrezat (aged 40) from Imneizel village said: “I participated in all the activities, now that there are people who care for me and gave me the space to participate and express my opinions.” She added: “Now I have the courage and the will to have what I want. TAM promoted me as a woman in my own household and community.” Samia Salamat (aged 37), a member of the local village council of Al-Walajeh Village, said: “Now I have the ability to express my opinions, even though the majority of the members are men.” 2.2 Performance Indicator 2: 30% of PCID projects (water and sanitation) save time for the welfare of gender groups – change in gender roles During the project period, 121 women filled time-use surveys in 5 projects (Imneizel, Khallet El￾Mayyeh, Rantis, Habla and Jinsafut). The time-use survey was used to compare the time saved before and after project interventions, and to measure the effect of intervention and gender recommendations on the life roles of female beneficiaries. The results of the surveys indicated that women were able to save more time after the implementation of the water and sanitation projects (clarified in the chart below), and were granted easier access to water. As a result, they now have more spare time for their welfare, which led to 284 703 293 800 8 0 0 200 400 600 800 1000 Coordination Meetings Field Visits Gender Groups Participation Male Female PWD 132 a significant change in gender roles. Moreover, results indicated an increase in the time used for women’s productive and social roles, and in the time used for self-care. For example: Women in Imneizel are saving 3.3 hours per a day, equal to 1,188 hours per year. As quoted by Mariam Salem (aged 36), “I used to spend at least 5 hours on housework; now with the availability of water, I finish the housework in one hour and I have found the time to produce and sell cheese.” The following table shows the results of the time-use surveys conducted in the aforementioned locations, in terms of time spent in procuring water, changes in reproductive, productive and social roles, as well as changes in the time women spent on self-care. Location Time spend in procuring water Reproductive role Productive role Social Role Self-care Pre Post Pre Post Pre Post Pre Post Pre Post Jinsafut 4% 1% 80% 67% 11% 15% 4% 12% 5% 6% Habla 5% 0% 76% 65% 5% 11% 7% 11% 12% 13% 4% 5% 10% 2% 24% 1% 0% 0% 0% 0% 0% 5% 10% 15% 20% 25% 30% jinsafut Habla Khallet Al￾Mayyeh Rantis Imnizel Time Spent Procuring Water Pre Post 133 The implementation of water and sanitation projects in the targeted areas shows significant change in women’s roles. Time spent procuring water decreased to almost zero percent. Time spent on reproductive roles such as housework, cleaning, and children care dramatically decreased. The decrease of time spent procuring water and in reproductive work allowed women to spend more time on their reproductive and social roles, and taking care of themselves, thus explaining the increase in the post time-use survey percentages. Amneh Shahwan (aged 48) who now works as a health educator in Habla said: “Sewage problems used to consume all my time. Now I have enough time to work as a health and cultural educator. The sewage network allowed other women to attend social event and participate in community activities.” Khallet Al￾Mayyeh 10% 0% 83% 78% 9% 12% 5% 6% 3% 4% Rantis 2% 0% 68% 54% 9% 19% 11% 7% 12% 20% Imneizel 24% 0% 89% 29% 7% 23% 4% 15% 0% 33% 134 1. Immediate Results: This section shows project immediate results and their corresponding sub-indicators: 3.1 Immediate Result 1: Gender groups have equal opportunity access to resources: The following performance indicators contributed to granting gender groups equal opportunities to accessing resources: 3.1.1 Performance Indicator 1: 20% of community infrastructure facilities of PCID implemented projects (schools, clinics, centers) integrated gender criteria of accessibility, privacy, safety and Compatibility Based on the recommendations of the TAM team, gender equality criteria were integrated in most of the infrastructure projects, including criteria of accessibility, privacy, safety and security and compatibility, as shown in the following examples: a) Health Sector: Gender Criteria for the Al-Walajeh Clinic focused on ensuring privacy and accessibility for women, such as locating the “mother and child” health care room on the first floor (based on gender needs and TAM’s gender recommendations) to make mothers feel more comfortable while using the clinic. b) Educational Sector: Gender Criteria for the Educational Sector focused on and succeeded in ensuring safety and security, accessibility features and privacy for students, which was vividly apparent in the quotes27 of some teachers of the targeted groups. 27 Quotes were taken from focus group sessions, during the impact assessment research. Therefore, we have no reference for whom each quote belongs. 135 To reflect on Jalqamous School, a teacher explained: “The school was in chaos, and students would run away all the time, especially during playtime because of the low surrounding walls and the crowded school. Now all of this has ended.” A teacher from Deir Qeddis School ensured that, “The environment of the school has a big impact on the psychological wellbeing of the students and the teacher, especially since the school garden was unsafe and full of rocks. The gardens were like a dessert, but after the completion of the projects, it became a colorful and a green space. We decided to start planting with the students to generate additional revenue for the school.” A teacher from Qaffin School said that, “The toilets are adequate, contrary to the previous school; teachers and students used the same toilets which was uncomfortable and never healthy. 136 A teacher from Al-Taqwa School said: “Before the project, sports classes often used to be canceled because the school playground was not suitable for play.” 3.1.2 Performance Indicator 2: 20% of public facilities and schools modified to allow access by persons with disabilities (PWD). Deir Qeddis School •Separated cars entrance to preserve the safety of students •Secured the sidewalk in front of the school gate to ensure that children do not go out directly to the street. Hatta School •Existing boundary walls were raised with fencing and protection fenders along the outside corridor of the newly constructed second floor to ensure gender group’s safety and security. •The pre-school kindergarten have separate entrance, and two toilet ( two seats and one shower), kitchenette and a separate play area (which was developed according to international standards) and a playground that is safe for students' use. Bil'in School Added a pedestrian path near the yard to ensure safety of students. School walls were painted in non￾stereotypical colors that reflect a non￾discriminatory culture Al-Taqwa School: Existing surrounding boundary walls were raised with fencing and protection fenders in order to prevent students from running away from school. 137 Gender integration criteria in infrastructure projects ensured granting PWD’s easier access to facilities such as:  All completed educational sector projects were modified to allow access for PWD’s: • Seven schools have ramps and PWD toilets. • One school has an elevator. • One school (Qaffin) allows for the rotation of classrooms to suit PWD’s movement and accessibility. • Two schools have Braille signage (Jalqamous and Seer El-Kfeer): “Omar, a blind student at Jalqamous, can easily access the bathrooms and other school facilities. He uses the Braille to direct him to classes. Of course other students now know about the purpose of the Braille and preserve them”.  Youth Center in Jericho: Followed the international codes related to design of spaces for PWD’stoilets, elevator; corridors were gender-convenient; ease of access and mobility for PWD’s; and Braille signage.  Al-Walajeh Clinic: • Toilets for PWD’s • PWD accessible entrance and accessible parking • Mobility and accessibility to all clinic facilities  Nahalin Drainage Project: TAM recommended widening the scope of the storm water drainage project. Moreover, following the request of the principal of a special needs school during the community gathering, TAM recommended modifying the design to include the school as it caters to persons with special needs. 3.2 Immediate Result 2: Gender groups have equal opportunity for participation. The following performance indicators contributed to granting gender groups equal opportunities for active and effective participation, as shown in the following chart: 138 3.2.1 Performance Indicator 1: Proportion of female to male participating effectively in Community Gathering Meetings and Awareness Raising Workshops Project activities enabled different gender groups to participate effectively in Community Gathering Meetings and Awareness-Raising Workshops. For example, the awareness workshop that was held in Rantis, where a fruitful discussion was led by two local female nurses volunteering from the Health Work Committee, defined responsibilities of individuals, community and village council for the best use of the water network and reservoir and cooperation by all parties to maintain and keep it working. Moreover, during Khallet El-Mayyeh Workshop, Ms. Lina Alghafi (aged 42), the head of kindergarten, presented the impact of implementing the water network project, explaining how they used to procure bad quality water which had a negative impact on children’s health. She had to continuously buy special filters to clean the water. However, after implementing the network, all these problems were resolved. Gender groups have equal opportunity for participation Indicator 1: Proportion of female to male participating effectively in Community Gathering meetings and Raising Awareness Meetings 1. Female participation increased from 0 % to 51% at decision making level 2. Increased Participation for gender groups in: Awareness-raising workshops: 60% were women, 3% were PWD's Community Gatherings: 42% were women, 3% were PWD's Total women participants who filled the questionnaires in the community meetings were 740 out of 1764, this is equal to 42% Indicator 2: 35% of women headed households have access to water and sanitation services and can afford them. 1,279 families headed by women directly benefited and have continuous access to water and sanitation services because of PCID projects 139 The figure below shows that the total number of women participants in 38 community meetings for infrastructure projects were 1,751 participants, of whom 1,024 were female (58%), 727 were male (42%), and 54 (3%) were PWDs. In addition, of 447 participants, the total number of female participants in awareness-raising workshops was 272 persons (60%) and 14 (3%) PWDs. Moreover, the total number of women participants who filled out the questionnaires in the community meetings was 740 out of 1,764, equal to 42%. 3.2.2 Performance Indicator 2: 35% of women-headed households have access to water and sanitation services and can afford them. Water and sanitation projects enabled households headed by women to have continuous access to water and sewage services at affordable costs. Based on project data, 1,279 families headed by women directly benefited from the PCID water and sewage projects. Per TAM’s recommendations, some local councils and municipalities provided those services free of charge for families headed by women. For example, 10 households headed by women in Habla town were exempted from sewage charges and some vulnerable families were offered discounts. 3.3 Immediate Result 3: Palestinian community knowledge, beliefs and practices have changed positively toward Gender groups, local resources and PCID projects. 175 1024 272 740 14 2 0 200 400 600 800 1000 1200 Awareness-Raising Community Gathering Gender Groups Participation Males Females PWD's 140 Awareness raising materials and brochures distributed during the awareness raising workshops and educational open days were gender sensitive - written in a language that addresses different gender groups equally, and avoiding stereotypical and cultural portrayals of men, women and PWD’s. The awareness materials also contained messages about social responsibility towards local resources and the implemented PCID projects. The awareness raising materials were distributed in schools, and brochures were distributed to the local public services centers and local, community based organizations. 3.3.1 Performance Indicator 1: Number of awareness-raising, gender-sensitive materials produced and distributed. The table below shows the number of awareness-raising, gender-sensitive materials produced and distributed throughout the sub-grant period: Palestinian community knowledge, beliefs and practices have changed positively toward Gender groups, local resources and PCID projects . Performance Indicator 1: Number of awareness raising gender-sensitive materials produced and distributed 1. 5,834 Awareness raising materials were distributed 2. Two success stories and one animation film were produced Performance Indicator 2: Number of brochures, designed and printed then distributed to public in the targeted areas 13,250 gender sensitive brochures were distributed to communities Awareness-Raising Material Produced Distributed Billboards 27 27 Educational Boards 54 48 141 Moreover, TAM produced 2 success stories and one animation film, and videotaped 10 impact assessment focus groups, 6 awareness raising workshops and 3 educational open days. Performance Indicator 2: Number of brochures, designed, printed and distributed to local communities in the targeted areas. The table shows the number of brochures designed and distributed in the targeted areas: Coloring Books 2,400 400 Puzzles 450 270 Coloring Pencil Boxes 600 400 Pencil Cases 400 400 Educational Animation Film 1 1 TV/Radio Spots 2 2 Hygiene Kits 1,900 1,900 Total 5,834 3,448 Location Number of Brochures Khallet Al- Mayyeh 1,500 Attil 1,600 Jinsafut 500 Rantis 500 Imneizel 250 Falamya 200 Al-Doha 1,500 Qabatia 3,500 Habla 1,300 142 3.CHAPTER THREE: ACTIVITIES This section provides a detailed description of activities implemented during the 4-year program period. All activities were designed and implemented to accomplish the project overall impact, immediate results and indicators. Awareness Raising Activities ANERA and TAM have developed an awareness campaign to ensure the sustainability of PCID projects in the five different sectors (water, drainage, sewage, health and education). Specific messages and activities were developed for interventions within each sector, that aimed to raise awareness of the target group and instill a sense of community ownership and belonging in each project, and therefore ensure its sustainability and continued effectiveness. Izzariyeh 1,500 Bil’in 400 Al-Walajeh 500 Total 13,250 143 This was achieved through the use of awareness tools and methodologies designed to educate and increase the knowledge of the communities about their roles and responsibilities towards the interventions. The awareness tools included awareness-raising workshops, educational open days, teachers’ workshops, and a health day. 1.1 Nine (9) Awareness-Raising Workshops were conducted throughout the project period in the following areas: Khallet Al-Mayyeh, Habla, Qabatia, Attil, Jinsafut, Falamya, Rantis, Imneizel and Al-Doha. The workshops allowed direct communication with residents in the targeted communities, engaging them in discussions about ownership and social responsibility towards the infrastructure projects, in addition to awareness raising about the conserving and rationing water consumption. The table below shows the date, number of participants and number of brochures distributed during each awareness workshop: The awareness workshops promoted the participation of the different gender groups, and allowed them to express their opinions and took into consideration their needs and wants. 28 Due to problems with the communities in Izzariyeh and Bil’in, we were not encouraged to conduct awareness activities. Project Date Number of Participants Brochures Total M F PWD Habla 1/12/2015 71 34 37 5 1,300 Qabatia 1/19/2015 50 24 26 0 3,500 Attil 10/3/2015 47 24 23 0 1,600 Aldoha 9/25/2014 36 15 21 4 1,500 Khallet Al-Mayyeh 17/2/2016 58 28 30 2 1,500 Jinsafut 3/16/2016 61 14 47 0 500 Falamya 3/23/2016 53 18 35 1 200 Rantis 5/11/2016 24 0 24 1 500 Imneizel 2/21/2017 47 18 29 0 250 Al-Walajeh 28/11/2016 112 42 70 500 Izzariyeh No awareness workshops held.28 1,500 Bil’in 400 Total 447 175 272 13 13,250 144 Coolers for Imneizel Village: TAM distributed 73 coolers in Imneizel Village during the awareness-raising workshop. Brochures: A total of 13,250 brochures were distributed during the awareness-raising workshops. The table below provides detailed information about each awareness workshop date, participants and number brochures distributed: Billboards: Twenty seven (27) awareness-raising billboards were installed for completed water projects as follows: Project Date Billboards Secondary Sewage Collection Network and House Connections in Habla Town, Qalqilya Governorate 21/6/2015 2 Construction of Storm Water Drainage System in Qabatia Town, Jenin Governorate 16/6/2015 2 Construction of On-ground water Storage Reservoir and Rehabilitation Part of Water Network in Attil Village, Tulkarem Governorate 21/6/2015 2 Construction of On-ground water Storage Reservoir and Rehabilitation Part of Water Network in Bil'in Village, Ramallah Governorate 18/6/2015 2 Installation of a Water Network with House Connections and Construction of a 250 M3 On-ground Water Tank in Imneizel Village, Hebron Governorate 19/2/2017 2 Construction of Al-Karmel Village Groups Water Networks, Yatta Town, Hebron Governorate 19/3/2017 2 Construction of Al-Hadideyya and Khallet Saleh Water Networks, Yatta town, Hebron Governorate 19/3/2017 4 Construction of a 500M3 Elevated Water Reservoir, Transmission Line and Rehabilitation of the Existing Water Network in Jinsafut Village, Qalqilia Governorate 23/2/2017 1 Construction of a 200 M3 Elevated Water Reservoir in Falamya Village, Qalqilia Governorate 26/3/2017 2 Construction of a 500 M3 Elevated Water Reservoir in Rantis Town, Ramallah Governorate 29/3/2017 2 Construction of a Storm Water Drainage System in Doha Town, Bethlehem Governorate 15/6/2015 2 145 1.2 Educational Open Days Six (6) Educational Open Days were conducted during the program period in the following locations: Bil’in, Deir Qeddis, Al-Taqwa, Qaffin, Jalqamous and Hatta. The open days included educative tools for the students presented in funny and exciting ways such as; presenting a short-animation awareness film (maintain school facilities, clean environment activity, personal and public hygiene), clowns, interactive theatre, youth initiatives, free drawing, recycling agriculture, behavior modification in school and a radio spot. The table shows the date and total number of participants in each of the open days. Goals of School Open Days: • Encourage students to maintain their school clean for a healthy and comfortable learning environment and to enhance their commitment towards their school. • Ensure the sustainability of the project implemented in the school. • Enhance students’ involvement through participation in topics related to school maintenance and hygiene (words, phrases, songs). • Enforce the role of school committees through participation in the open day activities. Construction of Khallet Al-Mayyeh Villages Group Water Networks, Yatta Town, Hebron Governorate 15/6/2015 2 Construction of Storm Water Drainage in Izzariyeh Village, Jerusalem Governorate 17/6/2015 2 Total 27 Project Date Number of Participants Total M F PWD Bil’in 1/12/2015 234 3 231 Deir Qeddis 2/12/2015 352 110 243 Al-Taqwa 6/5/2016 423 410 13 2 Qaffin 20/4/2016 519 0 519 1 Jalqamous 16/11/2017 280 280 0 1 Hatta 25/4/2017 314 142 172 Total 2,122 945 1,178 4 146 Open Days Activity Outcomes: • Introducing student to the project implementer and the funder. • Increasing the sense of ownership and public responsibility for the school to all of the beneficiaries, so they treat the school as their own home. • Enforcing the role of school committees including heath committees in open day activities. • Promoting social responsibility and motivating students to maintain school facilities. • Student participation in drawing messages about clean school environments. • Encouraging/activating the role and involvement of school parents’ councils/boards. • Giving students the chance to enjoy their time and break their routines, which was reflected in the smiles on their faces. • Support women associations by ordering the catering services from women’s association as support for local women’s production. Materials Distributed During Open Days: All materials distributed during the educational open day were gender-sensitive and carried educational messages that promote core values of social responsibility. Coloring books: TAM designed a gender-sensitive coloring book that includes 20 drawings, printed 2,400 and distributed 400 copies in Hatta School along with 600 coloring boxes and 400 pencil cases that carry a gender-sensitive image. Puzzles: TAM designed 10 gender-sensitive puzzles, printed 450 copies and distributed 270 copiesin Hatta School. Hygiene kits (including soap bars, towels, toothpaste and toothbrushes): TAM distributed 1,900 kits during open days in 5 schools. Awareness-Raising Material Produced Distributed Locational Coloring Books 2400 400 Hatta Puzzle 450 270 Hatta Coloring Pencil Boxes 600 400 Hatta Pencil Cases 400 400 Hatta 147 Short Animation Film for Schools: A short animation film was produced to be viewed in schools during the educational open days, in order to promote students' roles and responsibilities in maintaining the school facilities, to ensure a clean and healthy environment for the students and teachers. The film was produced with English subtitles and screened in schools during the educational open days for a total of 2,122 students and was discussed with the help of the facilitator. Discussions between students and facilitator focused on the mechanisms to keep the school clean and maintain its facilities, and how to maintain personal hygiene. The students committees were trained on how to display and discuss the movie in order to display it in future activities and for new students. Educational boards: Forty eight (48) educational boards were placed in the six schools before the implementation of the educational open days. Each board holds a different message and graphics to promote the communities’ roles and responsibilities to sustain and protect the projects constructed to serve their wellbeing. 1.3 Teachers Workshops: Two teachers’ workshops, with the participation of 35 teachers, were conducted as following: 1. Deir Qeddis School teachers’ workshop was conducted on 1/12/2015 with 15 female school teachers. 2. Jalqamous School teachers’ workshop was conducted on 29/9/2016 with 20 male school teachers. The teachers’ workshops aimed at addressing and decreasing teachers’ stress due to the increased number of school students, and to explore with them the skills and mechanisms that can be used with the students to maintain the school facilities and a clean school. 1.4 Health Day in Al- Walajeh Clinic TAM conducted an awareness-raising Health Day in Al- Walajeh Clinic with the participation of 112 persons (42 males and 70 females) from the Village Council, local community and the Department of Nursing in Bethlehem University. Hygiene Kits (including soap bars, towels, toothpaste and toothbrushes) 1900 1900 Bil’in, Deir Qeddis, Al￾Taqwa, Qaffin and Jalqamous 148 During the Health Day, the clinic was divided into five booths, in each booth there were two nursing students. Booths were divided as follows: 1. Blood pressure and diabetes screening 2. 1-min Osteoporosis test 3. Measuring weight and height and vision test 4. Breast Cancer self-examination and early detection 5. Awareness-raising for the importance of health insurance Supplies: The following supplies where provided during the health day: 1. Sphygmomanometer (blood pressure screening) 2. Blood Sugar Screening Device & Strips 3. Snellers Chart 4. Tongue Depressors 5. Band Aids and Alcohol Swaps 6. Medical Gloves 7. Cotton 8. Tissues, toilet papers and hand soap. Six (6) health information boards were installed at Al-Walajeh health clinic delivering awareness messages about the most important health issues and maintaining the clinic facility. Brochure: 500 brochures about sexual health, blood pressure and diabetes, personal hygiene and the brochures of the clinic were presented and distributed to people. Outcome of the Health Day: According to information obtained from the clinic: 1. The number of patients visiting the clinic increased since the implementation of the health day. An average of 15 patients visit the clinic per day, compared to 5 visitors per day before the health day was held. 2. Before the implementation of the health day, only 30 residents had health insurance. However, after the activity, the number increased to 150 (126 regular patients and 24 patients with chronic diseases). 3. The clinic makes 3 to 4 referral per months. The local community appreciated the health as such events allow for community interactions. One of the elderly said, “This clinic is comfortable and has all the facilities needed to serve our village and our people.” Another said, “I thank TAM and ANERA for such a nice day; we had the chance to come and check on ourselves and become familiar with the staff and clinic.” 2. Coordination Meetings: 149 TAM team conducted 119 coordination meetings with the participation of 577 (284 males, 293 females and 8 PWD’s), with the municipalities and local councils of the targeted project areas, to coordinate for and follow up on project activities. 3. Field Visits: TAM team conducted 271 field visits with the participation of 1,503 (703 males, 800 females), in order to follow up on gender recommendation during project implementation. 4. Gender Recommendations and Observations TAM team has provided gender recommendations for 27 infrastructure projects of infrastructure projects during the design and preparation period, as well as conducting gender observation through the implementation stages of the projects. The projects were: Water Networks: 1. Construction of a 1,000 M3 Ground Storage Reservoir, Booster Station, Transmission Pipeline and Rehabilitation of Parts of Attil Village Water Network, Tulkarem Governorate 2. Construction of a 500 M3 Elevated Water Reservoir, Transmission Line and Rehabilitation of the Existing Water Network , Jinsafut, Qalqilia Governorate 3. Construction of On-ground Water Storage Reservoir and Rehabilitation of Part of the Water Network in Bil'in Village, Ramallah Governorate 4. Construction of a 200 M3 Elevated Water Reservoir in Falamya Village, Qalqilia Governorate 5. Construction of a 500 M3 Elevated Water Reservoir in Rantis Town, Ramallah Governorate 6. Construction of Khallet Al-Mayyeh Villages Group Water Networks, Yatta Town, Hebron Governorate 7. Construction of Al- Hadidiya and Khallet Saleh Water Networks , Yatta Town, Hebron Governorate 8. Installation of a Water Network with House Connections and Construction of a 250 M3 On￾ground Water Tank in Imneizel Village, Hebron Governorate 9. Construction of Al-Karmel Village Groups Water Networks, Yatta Town, Hebron Governorate 10. Rehabilitation and Extension of Internal Water Networks in Al-Shuyoukh, Hebron Governorate Storm Water Drainage: 1. Construction of a Storm Water Drainage System in Qabatia, Jenin Governorate 2. Construction of a Storm Water Drainage System in Al-Doha, Bethlehem Governorate 3. Construction of Storm Water Drainage in Different Parts of Nahalin, Bethlehem Governorate 4. Construction of Storm Water Drainage in Izzariyeh, Jerusalem Governorate Sewage: 150 1. Secondary Sewage Collection Network and House Connections in Habla, Qalqilyia Governorate Educational Sector: 1. Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Qaffin Secondary Girls’ School in Qaffin Village, Tulkarm Governorate 2. Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Al-Taqwa Elementary Basic Boys’ School in Bani Na’im, Hebron Governorate 3. Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Jalqamous Secondary Boys’ School in Jalqamous Village, Jenin Governorate 4. Construction of Additional Educational Rooms, External Works and Rehabilitation of Existing Facilities at Deir Qeddis Elementary Girls’ School in Deir Qeddis, Ramallah Governorate 5. Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Bil'in Secondary Girls’ School in Bil'in, Ramallah Governorate 6. Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Seer el Kfeer Secondary Co- educational School in Seer El-Kfeer, Jenin Governorate 7. Construction of Additional Educational Rooms, External Works and Rehabilitation of Existing Facilities at Hatta Basic Co-Educational School in Hatta, Hebron Governorate Parks: 1. Rehabilitation of Zeita Recreational Park in Zeita, Tulkarem Governorate Centers: 1. Construction of a Women’s Center in Dar Salah, Bethlehem Governorate 2. Construction of a Youth Development Resource Center (YDRC) in Jericho, Jericho Governorate 3. Rehabilitation Works and Expansion of St. Nicolas Home for the Elderly in Beit Jala, Bethlehem Governorate Health Sector: 1. Construction of a Clinic in Al-Walajeh, Bethlehem Governorate Below are some of the most important gender recommendations: • Projects to be accessible by all gender groups, especially persons with disabilities, and integrate the needs of gender groups in the community. • Incorporate gender integration criteria for students of different age groups, and activate the role of the parents’ council/board in the schools. • Encourage local stakeholders to activate the role of women in the decision making process, especially municipalities and local councils. 151 • Increase women’s effective participation in all stages of the projects and increase the role of local women’s societies. • Increase volunteerism and social engagement for women. • Find young women leaders in all areas of project interventions. 5. Impact Assessment TAM conducted an Impact Assessment to assess and evaluate the impact of the project, awareness activities conducted with beneficiaries, and awareness materials distributed throughout the project period. To measure the impact in a concrete and effective way, the impact assessment research focused on assessing and evaluating the following indicators: (1) The effectiveness of awareness tools distributed and used in the program activities, (2) the change in gender roles and (3) the impact of the project on the village and local community. The methodology and assessment tools of the research were set to evaluate the abovementioned indicators as follows; (1) focus group and individual interviews for water and sanitation projects, (2) focus groups and questionnaire for schools. 5.1 Targeted Projects: The impact assessment research targeted the following projects: Seven water and sanitation projects in the following areas; Doha, Khallet El-Mayyeh, Imneizel, Habla, Qabatia, Attil and Jinsafut as clarified in the table below: Water and Sanitation Sector Projects Project Date Tools used Number of Participants Total F M PWD’s Northern Area Construction of a 1,000 M3 Ground Storage Reservoir, Booster Station, Transmission Pipeline and Rehabilitation of Parts of Attil Village Water Network, Tulkarem Governorate 21/2/2017 Focus Group 18 12 6 0 Interviews 5 5 152 Health Sector Projects: Construction of a Clinic in Al-Walajeh Village - Bethlehem Governorate: One focus group of 13 persons (10 females and 3 males), from members of the local council and local community, and four individual interviews, were conducted on 23/3/2017 Educational Sector Projects: Four Schools: Deir Qeddis, Jalqamous, Qaffin and Al-Taqwa, as indicated in the table below. 29 There is no local council in Jinsafut. 30 Currently, Deir Qeddis school has male students in elementary classes ( 1st to 3rd grades). The five males who filled the questionnaire are in third grade. Construction of a 500 M3 Elevated Water Reservoir, Transmission Line and Rehabilitation of the Existing Water Network in Jinsafut 23/2/2017 Focus Group No focus group29 Interviews 5 5 5 Construction of Storm Water Drainage System in Qabatia, Jenin Governorate 3/1/2017 Focus Group 13 2 11 Interviews 4 4 Secondary Sewage Collection Network and House Connections in Habla, Qalqilyia Governorate 5/3/2017 Focus Group 10 3 4 3 Interviews 5 5 Southern Area Installation of a Water Network with House Connections and Construction of a 250M3 On￾ground Water Tank in Imneizel Village, Hebron Governorate 21/2/2017 Focus Groups 32 2 1 11 Interviews 5 5 Construction of a Storm Water Drainage System, Al- Doha, Bethlehem Governorate 26/2/2017 Focus Group 8 5 3 Interviews 5 5 Construction of Khallet Al-Mayyeh Villages Group Water Networks, Yatta Town, Hebron Governorate 27/2/2017 Focus Group 16 7 9 Interviews 5 5 Project Date Tools Number of Participants Total F M PWD Construction of Additional Educational Rooms, External Works and Rehabilitation of Existing Facilities at Deir Qeddis Elementary 22/2/2017 Questionnaire 25 2030 5 Focus Group 10 10 153 5.2 Results of Impact Assessment: This section reflects the analysis of results generated using the abovementioned impact assessment tools (focus groups and interviews), in terms of (1) the impact of the project on the local community and (2) the change in gender roles for each sector, as following 1. Water and Sanitation Sectors: a) Water Networks: The results of the impact assessment for the water networks showed that: Impact of the project on the local community: constructing water networks in the targeted project communities have (1) significantly improved the lives of people, (2) granted them easy and continuous access to safe potable water and (3) reduced the cost of buying water tanks. Girls School in Deir Qeddis Village, Ramallah Governorate Construction of Additional Educational Rooms, External Works and Rehabilitation of Existing Facilities at Jalqamous Secondary Boys’ School, Jalqamous Village, Jenin Governorate 27/2/2017 Questionnaire 25 25 Focus Group 10 10 Construction of Additional Educational Rooms, External Works and Rehabilitation of Existing Facilities at Qaffin Secondary Girls’ School, Qaffin Village, Tulkarm Governorate 19/5/2016 Focus Group 20 20 Questionnaire 30 20 Construction of Additional Educational Rooms, External Works and Rehabilitation of Facilities at Al-Taqwa Elementary Basic Boys’ School, Bani Na’im, Hebron Governorate 22/5/2016 Focus Group 8 8 1 Questionnaire 32 32 154 Change in Gender Roles: implementing gender integration activities and awareness (1) enabled gender groups to have access to and control over resources, (2) saved time and reduced efforts while procuring water, (3) increased women’s effective participation and decision-making power, in addition to (4) improving the quality of life for women, PWD’s and elderly people. b) Storm Water Drainage: The results of the impact assessment for the storm water drainage showed that: Impact of the Project on the Local Community: the projects have (1) positively impacted and improved the quality of lives of the people in Qabatiya and Doha cities and their families, (2) vanished people’s fear and anxiety about the floods and reduced their physical losses, (3) and have significantly reduced physical damages that occurred due to the floods. Change in Gender Roles: The projects have (1) improved the life of PWD’s and elderly, (2) saved time and reduced efforts for women, and (3) improved the lives of families headed by women. c) Sewage Sector: The results showed in this section are for Habla Project: The impact of the project on the local community: (1) The sewage project improved the lives of PWDs and the elderly in Habla town through enabling them to move safely and helping increase their participation in society. (2) The project contributed to reducing the cost of sewage tanks and water bills, and reduced the use of pesticides which have a negative impact on the environment. Prior to project implementation, flooded sewage required large quantities of water to clean, especially when sewage flooded inside houses. It also caused odors which attracted mosquitos and insects, increasing the use of pesticides. (3) The sewage network is providing a clean and healthy environment for women and their families and PWDs. Change in Gender Roles: The project contributed to (1) enhancing women’s effective participation in the local community and (2) saving time and reducing efforts. Impact of Awareness Tools: The impact of the awareness-raising workshops conducted, brochures distributed and billboards installed was as follows: Awareness raising workshops: • Increased the community’s awareness on the mechanisms to protects and maintain the implemented projects and the rationalization of water consumption. • All segments of the community were able to participate in the awareness raising workshop; local women CBO’s, kindergartens, mothers and head of household, youth and elderly, in addition to youth CBO’s and persons with disability. • The awareness raising workshops conducted in the targeted locations were the first ever to include all gender categories. 155 Billboards: The billboards installed in targeted projects location represent important tools that remind people how to constantly protect the implemented projects and to rationalize their water consumption. • The locations where the billboards were installed are strategic, and information of the boards is easily readable by all segments of the society. However, most participants recommended installing more and larger-sized billboards. Brochures All brochures that were distributed during the awareness raising workshops contained important information that benefited all the participants and their families. The brochures were distributed to the local public services center, schools and local community based organizations. Moreover, the content of the brochures was presented in schools during the morning school radio programs. 2. Health Sector: Al-Walajeh Clinic This section provides an impact assessment for local clinic project in Al-Walajeh village on improving the lives of the local community and changing in the gender roles. Impact on the local community: • Reduced risk of movement: The clinic location is in the entrance of the village and it accommodates comfortable facilities suitable for all gender groups. Moreover, the clinic significantly decreased the difficulties and risks of movement when seeking healthcare from outside clinics or having to cross checkpoints. Gender Roles: • Enhanced Women’s Effective Participation: after participating in projects activities, women increased their participation in community activities and improved their decision-making abilities, became more confident of themselves, and were encouraged to have an active role in their communities. • Convenient Facilities for Gender Groups: clinic and its facilities are comfortable, safe and suitable for women, PWDs and elderly, in terms of the availability of ramps, bathrooms and wide and safe entrances and corridors. Moreover, the clinic includes closed rooms that take into account privacy of gender groups, especially women and young girls. 156 Effectiveness of the Health Day: The health day activity encouraged local citizens to visit the clinic on regular basis, and raised their awareness about the importance of having health insurance. The participants of the focus group said that the health boards are attractive, and readable by everyone who visits the clinic, and the brochures distributed were very beneficial, that they still have the in the homes. 3. Educational Sector The impact assessment of the educational sector targeted four schools as follows: Qaffin, Taqwa, Deir Qeddis and Jalqamous, to measure the (1) the impact of the projects on the lives of the community, (2) the change in gender roles, (3) the impact of the awareness tools used. Results of Impact Assessment: This section shows the results of the impact assessment on the school and students, and the effectiveness of the open days On the School and Students Level: • The school buildings and facilities became more comfortable and convenient for students and school staff. • The school building and playgrounds are safe, secure, and observable, which helped in reducing violence between the students. • Fewer students run away from schools. • Water faucets and toilets are adequate and appropriate for the number of students, their gender, and different ages. • The halls, labs and other facilities contributed to the implementation of interactive educational activities and improved students’ skills. • The new buildings had a positive impact on the academic achievement of the students. • The new school buildings are suitable for people with disabilities and their integration into educational activities. • Participation of the school staff in the preparatory stages of the project was incorporated. Effectiveness of Educational Open Days • Increasing the sense of ownership and public responsibility for the school to all of the beneficiaries, so they treat the school as their own home. • Enforcing the role of school committees including heath committees in open day activities. • Promoting social responsibility and motivating students to maintain school facilities. • Student participation in drawing messages about clean school environments. • Encouraging/activating the role and involvement of school parents’ councils/boards. 157 • Giving students the chance to enjoy their time and break their routines, which was reflected in the smiles on their faces. • The teachers’ workshop was very beneficial for the teachers, and it helped them in developing new techniques on how to deal with their students. Integrating gender comments and recommendations inside the project design and implementing them made a huge difference in the quality of infrastructure service, and how it fits with the needs and priorities of different gender groups, in order to give all of them equal opportunities to access these services. 4. CHAPTER FOUR: CHALLENGES AND RECOMMENDATIONS 4.1 Challenges: The main challenges that arose during the project implementation period were as follows: • The completion dates of some projects were delayed, which led to delaying TAM’s implementation of its activities. • TAM’s decision-making space was limited in regards to awareness raising activities. • The cultural background of the targeted areas: some areas were unaccepting of the values and concepts of gender equality, which presented a real challenge in conveying new concepts and conducting activities. • Some municipalities and local councils were uncooperative, which led to delays in project implementation. • TAM’s scope of work was defined and agreed to in the sub-grant agreement with ANERA; however, TA would have liked to work more on women’s empowerment and increasing their effective community participation, especially women’s roles in municipalities and local councils. Most of the work focused on addressing gender needs in infrastructure, rather than on the women themselves. 4.2 Recommendations: Based on TAM’s project results and work in the fields, we recommend the following: • A number of villages within the West Bank still need interventions in terms of infrastructure and awareness. For future programs, needs assessments should be conducted in marginalized and rural areas, and projects should be prioritized based on the needs assessment results. 158 • During community gatherings, projects should be presented in simpler ways that easily reach and can be understood by the local community, such as presenting infrastructure projects in 3D. • More flexibility should be given to design modifications especially after community gathering meetings and during implementation. • Future programs should focus on promoting the active and effective participation of women in their communities, as well as in decision making positions. • More sector budget flexibility would allow the Program to better meet the needs of people. 159 PCID PROJECT SELECTION CRITERIA The PCID Program receives lists of projects from three main sources: PA Line Ministries, USAID inter￾offices (where requested projects compliment other USAID-funded program activities), and finally, through ANERA’s interactions with communities and local organizations/own reconnaissance.* Once received, the projects go through a rigorous assessment to ensure that they fit the Program’s scope of work, objectives and requirements. The PCID Team will consider projects that fall under the PCID scope of work, are located in communities approved by USAID, and which will not violate ANERA’s no-contact policy. * No-contact policy and community clearances are strictly observed and adhered to. 1. Meet urgent needs in the water and sanitation and other related infrastructure sectors. 2. Respond to requests from USAID through inter-offices cooperation. 3. Address unmet needs in basic infrastructure in communities in the West Bank and Gaza. 4. Respond to emergency and unforeseen needs. 1. Projects are prioritized by the Line Ministries (i.e. the Palestinian Water Authority (PWA) in the West Bank, and the Coastal Municipality Water Utility (CMWU) in Gaza. 2. The projects are relevant to the Program’s scope of work. • Fit under Strategic Objective 1 (Water & Sanitation Infrastructure) • Proposed project activities are sustainable • Target/take into consideration Gender groups and needs • Target underserved/marginalized communities 3. Preliminary assessments of projects verify need, urgency and priority projects that seem to have all the components for reasonably smooth implementation. • Projects are located in and primarily serve areas A and B, and have adequate sources (in terms of quality and quantity) of water (West Bank). • There is an absence of water networks, and targeted residents depend on other, often more expensive, sources of water (e.g. tankering, rainwater cisterns, etc.) • Existing water systems are old, deteriorated, leak heavily, and cause health hazards through the spread of water-borne illnesses. I. WATSAN Sector Criteria (West Bank and Gaza) PCID Strategic Objectives 160 • Sewage: There is some form of a feasible sewage collection, disposal and treatment system for the sewage (in the event of an intervention). • Rainwater drainage: Communities suffer from annual flooding resulting in blocked streets, damaged property, economic losses, and limited mobility and access to public facilities such as schools and clinics. 4. Further verification confirms that these projects pass the following requirements: • Technical requirements (EIA, IBC, seismic requirements) • Necessary licensing/permits (i.e. JWC; Area “C”) • Availability of land and clear ownership (not rented or disputed) 5. Communities in the West Bank and Gaza are eligible to receive USAID funds. 1. Projects are prioritized by the Palestinian Ministry of Education and Higher Education (MoEHE). 2. The projects are relevant to the Program’s scope of work. • Fit under Strategic Objectives 2 or 3 • Proposed project activities are sustainable • Target/take into consideration Gender groups and needs (priority is given to girls’ schools) • Target underserved/marginalized communities 3. Preliminary assessments of projects verify need, urgency and priority projects that seem to have all the components for reasonably smooth implementation. • Projects are located in Areas A and B • Overcrowding/lack of space/location is a problem for the targeted schools • The schools lack adequate sanitation and educational facilities (e.g. laboratories, library, multipurpose halls, playgrounds, etc.) 4. Further verification confirms that these projects pass the following requirements: • Technical requirements (EIA, IBC, seismic requirements) • Necessary licensing/building permits (e.g. Area C) • Availability of land and clear ownership (not rented or disputed) 5. Communities are eligible to receive USAID funds. 1. The projects are relevant to the Program’s scope of work. 2. Fit under Strategic Objectives 2 or 3 3. Proposed project activities are sustainable II. Education Sector Criteria (West Bank) III. Community Infrastructure (West Bank and Gaza) 161 4. Target/take into consideration Gender groups and needs (priority given to women and youth) 5. Target underserved/marginalized communities 1. Preliminary assessments of projects verify need, urgency and priority projects that seem to have all the components for reasonably smooth implementation. 6. Projects are located in Area A or B (in the West Bank) 7. Lack of or deteriorated conditions of the facilities 8. Clear NGO partner that is eligible to receive USAID funding 2. Further verification confirms that these projects pass the following requirements: 9. Technical requirements (EIA, IBC, seismic requirements) 10. Necessary licensing/building permits (e.g. Area C) 11. Availability of land and clear ownership (not rented or disputed) 3. Communities in the West Bank and Gaza are eligible to receive USAID funds. 162 ANNEX H: ADDITIONAL SURVEY TABLES Q5) How interested were you in this project when you first heard about it on scale from 1 to 10, where 1 means “not interested at all”, and 10 means “extremely interested”: * Gender Crosstabulation % within Gender Gender Male Female Total Q5) How interested were you in this project when you first heard about it on scale from 1 to 10, where 1 means “not interested at all”, and 10 means “extremely interested”: 1 0.3% 0.2% 2 1.7% 1.2% 1.5% 3 2.8% 2.7% 2.7% 4 3.4% 3.0% 3.2% 5 7.6% 5.2% 6.3% 6 8.3% 7.3% 7.7% 7 10.7% 10.9% 10.8% 8 15.2% 14.5% 14.8% 9 13.4% 30.3% 22.4% 10 36.6% 24.8% 30.3% Total 100.0% 100.0% 100.0% Q13) How would you describe the duration of the project? * Gender Crosstabulation % within Gender Gender Male Female Total Q13) How would you describe the duration of the project? It took too long to finish 17.5% 26.4% 22.3% It took a reasonable amount of time 52.2% 50.2% 51.1% It was completed more quickly than I expected 21.3% 10.9% 15.8% I have no idea 8.9% 12.5% 10.8% Total 100.0% 100.0% 100.0% Q14) In general, did the project meet your expectations? * Gender Crosstabulation % within Gender Gender Male Female Total Q14) In general, did the project meet your expectations? Yes, to a great extent 55.9% 51.8% 53.7% Somewhat met my expectations 30.0% 29.0% 29.4% No, to a low degree 6.6% 11.6% 9.2% No opinion / had no expectations 7.6% 7.6% 7.6% Total 100.0% 100.0% 100.0% 163 Q15) Do you or anyone in your household use the services provided by this project? * Gender Crosstabulation % within Gender Gender Male Female Total Q15) Do you or anyone in your household use the services provided by this project? Yes 82.8% 80.5% 81.6% No 17.2% 19.5% 18.4% Total 100.0% 100.0% 100.0% Q16) On a scale from 1 to 10, how satisfied are you with these services? * Gender Crosstabulation % within Gender Gender Male Female Total Q16) On a scale from 1 to 10, how satisfied are you with these services? 2 0.8% 0.4% 4 2.1% 1.5% 1.8% 5 5.8% 4.9% 5.3% 6 6.3% 8.6% 7.5% 7 11.3% 13.2% 12.3% 8 21.3% 16.5% 18.8% 9 19.6% 25.2% 22.5% 10 33.8% 29.3% 31.4% Total 100.0% 100.0% 100.0% Q17) Since the project was finished, did the quality of services provided by this project change over time? What kind of change? * Gender Crosstabulation % within Gender Gender Male Female Total Q17) Since the project was finished, did the quality of services provided by this project change over time? What kind of change? A positive change 80.0% 83.0% 81.5% Stayed the same 15.4% 14.8% 15.1% A negative change 4.6% 2.3% 3.4% Total 100.0% 100.0% 100.0% Q19) Did the project fulfil an urgent need for your community or are there other projects that should have come first? * Gender Crosstabulation 164 % within Gender Gender Male Female Total Q19) Did the project fulfil an urgent need for your community or are there other projects that should have come first? Yes, this project fulfilled an urgent need for my community 77.2% 76.3% 76.7% No, there are other infrastructure projects that should have come first 22.8% 23.7% 23.3% Total 100.0% 100.0% 100.0% Q4) When did you hear about the project? * Gender Crosstabulation % within Gender Gender Male Female Total Q4) When did you hear about the project? Before the implementation of the project 56.2% 58.4% 57.4% During the implementation of the project 35.2% 30.4% 32.6% After the implementation of the project was completed 8.6% 10.0% 9.4% Do not know 1.2% 0.6% Total 100.0% 100.0% 100.0% Q8) How would you describe your participation in these meetings? * Gender Crosstabulation % within Gender Gender Male Female Total Q8) How would you describe your participation in these meetings? I was an active participant 53.2% 27.1% 42.0% I was mostly a listener 46.8% 72.9% 58.0% Total 100.0% 100.0% 100.0% Q9) How did you know about these meetings? * Gender Crosstabulation % within Gender Gender Male Female Total Q9) How did you know about these meetings? I got an official invitation 41.8% 32.8% 38.0% I heard about it from someone in my community. 40.5% 44.8% 42.3% I heard about it from a public announcement (radio, TV, newspaper, social media, mosques, etc.). 8.9% 12.1% 10.2% I attended it by coincidence 8.9% 10.3% 9.5% Total 100.0% 100.0% 100.0% 165 Q10) To what degree do you feel that the organizers were responsive to the participants in the meeting? * Gender Crosstabulation % within Gender Gender Male Female Total Q10) To what degree do you feel that the organizers were responsive to the participants in the meeting? They were very responsive 49.4% 43.1% 46.7% They were somewhat responsive 45.6% 43.1% 44.5% I do not think that their input considered 2.5% 8.6% 5.1% I do not know 2.5% 5.2% 3.6% Total 100.0% 100.0% 100.0% Q5) How interested were you in this project when you first heard about it on scale from 1 to 10, where 1 means “not interested at all”, and 10 means “extremely interested”: * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q5) How interested were you in this project when you first heard about it on scale from 1 to 10, where 1 means “not interested at all”, and 10 means “extremely interested”: 1 0.2% 0.2% 2 0.2% 5.8% 1.4% 3 1.2% 7.9% 2.7% 4 2.5% 5.8% 3.2% 5 4.8% 11.5% 6.3% 6 6.8% 10.8% 7.7% 7 8.7% 18.7% 11.0% 8 14.7% 15.1% 14.8% 9 25.3% 12.2% 22.4% 10 35.5% 12.2% 30.3% Total 100.0% 100.0% 100.0% Q13) How would you describe the duration of the project? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q13) How would you describe the duration of the project? It took too long to finish 26.2% 8.0% 22.1% It took a reasonable amount of time 55.9% 34.8% 51.2% It was completed more quickly than I expected 12.1% 29.0% 15.8% I have no idea 5.8% 28.3% 10.8% Total 100.0% 100.0% 100.0% 166 Q14) In general, did the project meet your expectations? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q14) In general, did the project meet your expectations? Yes, to a great extent 60.2% 30.9% 53.6% Somewhat met my expectations 29.0% 30.9% 29.5% No, to a low degree 6.8% 18.0% 9.3% No opinion / had no expectations 3.9% 20.1% 7.6% Total 100.0% 100.0% 100.0% Q15) Do you or anyone in your household use the services provided by this project? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q15) Do you or anyone in your household use the services provided by this project? Yes 88.1% 58.7% 81.6% No 11.9% 41.3% 18.4% Total 100.0% 100.0% 100.0% Q16) On a scale from 1 to 10, how satisfied are you with these services? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q16) On a scale from 1 to 10, how satisfied are you with these services? 2 2.4% 0.4% 4 1.4% 3.7% 1.8% 5 4.5% 9.8% 5.3% 6 7.1% 9.8% 7.5% 7 10.8% 20.7% 12.4% 8 19.1% 18.3% 18.9% 9 23.1% 18.3% 22.3% 10 34.1% 17.1% 31.4% Total 100.0% 100.0% 100.0% 167 Q17) Since the project was finished, did the quality of services provided by this project change over time? What kind of change? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q17) Since the project was finished, did the quality of services provided by this project change over time? What kind of change? A positive change 86.3% 55.6% 81.4% Stayed the same 11.1% 35.8% 15.0% A negative change 2.6% 8.6% 3.6% Total 100.0% 100.0% 100.0% Q19) Did the project fulfil an urgent need for your community or are there other projects that should have come first? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q19) Did the project fulfil an urgent need for your community or are there other projects that should have come first? Yes, this project fulfilled an urgent need for my community 81.9% 58.7% 76.7% No, there are other infrastructure projects that should have come first 18.1% 41.3% 23.3% Total 100.0% 100.0% 100.0% Q4) When did you hear about the project? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q4) When did you hear about the project? Before the implementation of the project 64.5% 32.1% 57.4% During the implementation of the project 30.3% 40.9% 32.6% After the implementation of the project was completed 5.2% 24.1% 9.4% Do not know 2.9% 0.6% Total 100.0% 100.0% 100.0% Q8) How would you describe your participation in these meetings? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q8) How would you describe your participation in these meetings? I was an active participant 42.3% 38.5% 41.6% I was mostly a listener 57.7% 61.5% 58.4% Total 100.0% 100.0% 100.0% 168 Q9) How did you know about these meetings? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q9) How did you know about these meetings? I got an official invitation 34.2% 53.8% 38.0% I heard about it from someone in my community. 48.6% 19.2% 43.1% I heard about it from a public announcement (radio, TV, newspaper, social media, mosques, etc.). 8.1% 15.4% 9.5% I attended it by coincidence 9.0% 11.5% 9.5% Total 100.0% 100.0% 100.0% Q10) To what degree do you feel that the organizers were responsive to the participants in the meeting? * Region Crosstabulation % within Region Region West Bank Gaza Strip Total Q10) To what degree do you feel that the organizers were responsive to the participants in the meeting? They were very responsive 49.1% 37.0% 46.7% They were somewhat responsive 46.4% 37.0% 44.5% I do not think that their input considered 1.8% 18.5% 5.1% I do not know 2.7% 7.4% 3.6% Total 100.0% 100.0% 100.0% 169 ANNEX I: STATEMENTS OF DIFFERENCES No statements of differences were reported 170