DISCLAIMER. This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. _ This publication was produced at the request of the United States Agency for International Development. It was prepared independently by IMPAQ International, LLC under Contract No. AID-167-I-17-00004, Order No. 72018219F00001. PERFORMANCE EVALUATION OF THE SUSTAINABLE WATER SECTOR CAPACITY DEVELOPMENT ACTIVITY FINAL EVALUATION REPORT May 2019 COURTESY OF SHOQATA UJËSJELLËS KANALIZIME E SHQIPËRISË (SHUKALB) 1 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV PERFORMANCE EVALUATION OF THE SUSTAINABLE WATER SECTOR CAPACITY DEVELOPMENT ACTIVITY FINAL EVALUATION REPORT May 2019 SUBMITTED TO: USAID/Albania American Embassy/Tirana Rr. e Elbasanit, Nr. 103 Tirana, Albania SUBMITTED BY: IMPAQ International, LLC 10420 Little Patuxent Parkway Suite 300 Columbia, MD 21044 www.impaqint.com USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 2 TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS ................................................................ 3 EXECUTIVE SUMMARY ....................................................................................... 4 1. INTRODUCTION............................................................................................ 6 1.1. DESCRIPTION OF THE OBJECT AND SCOPE OF THE EVALUATION......................................6 1.2. BRIEF OVERVIEW OF WATER SECTOR IN ALBANIA......................................................................7 1.3. EVALUATION METHODOLOGY.............................................................................................................8 1.4. LIMITATIONS AND MITIGATION MEASURES ....................................................................................9 2. KEY FINDINGS.............................................................................................. 10 2.1. QUESTION 1: WHAT HAS SHUKALB’S PERFORMANCE BEEN TO DATE AND WHAT HAS THE ACTIVITY ACHIEVED?.........................................................................................................................10 2.2. QUESTION 2: WHAT, IF ANY, OPPORTUNITIES WERE CREATED FOR THE SUSTAINABILITY OF THE CERTIFICATION PROGRAM? WHAT ARE THOSE OPPORTUNITIES AND WHAT, IF ANYTHING, NEEDS TO BE DONE TO ENSURE THAT THE OPPORTUNITIES ARE REALIZED? .........................................................................................................................................................18 2.3. QUESTION 3: WHAT, IF ANY, BEST PRACTICES AND LESSONS LEARNED CAN BE IDENTIFIED AS A LEARNED RESULT OF THIS ACTIVITY? ........................................................................23 3. CONCLUSIONS AND RECOMMENDATIONS ....................................... 25 ANNEX A. EVALUATION STATEMENT OF WORK.................................... 27 TABLE OF EXHIBITS EXHIBIT 1. EVALUATION QUESTIONS AND SUB-QUESTIONS ....................................................................9 EXHIBIT 2. NUMBER OF TRAINING COURSES DEVELOPED OR ADOPTED (PI 1)...............................12 EXHIBIT 3. PERSON-HOURS OF TRAINING RECEIVING TOT PROGRAM (PI 2)...................................12 EXHIBIT 4. PERCENTAGE BY WHICH TOTAL REVENUES FROM THE DELIVERY OF TRAINING COURSES EXCEED THE TOTAL TRAINING COST (PI 3)...............................................................................20 3 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV ACRONYMS AND ABBREVIATIONS ABC Association of Boards of Certification AKUM National Water Supply-Sewerage and Waste Infrastructure Agency BiH Bosnia and Herzegovina CA Cooperative agreement CSUS California State University, Sacramento FGD Focus group discussion GoA Government of Albania GIZ German International Development Agency IAWD International Association of Water Utilities in the Danube River Catchment Area KII Key informant interview LGU Local government unit M&E Monitoring and evaluation MIE Ministry of Energy and Infrastructure MoU Memorandum of understanding NEWEA New England Water and Environmental Agency NRW Non-revenue water O&M Operation and maintenance PI Performance indicator SHUKALB Albanian Association of Water Supply Companies SHUKOS Kosovo Association of Water Supply Companies SoW Statement of Work SWSCD Sustainable Water Sector Capacity Development ToT Training of trainers UACS University of Arkansas, Clinton School of Public Services USAID United States Agency for International Development WG Working Group WRE Water Regulatory Entity WS Water and sewerage USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 4 EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Albania contracted with IMPAQ International, LLC (IMPAQ) to conduct an independent performance evaluation of its Sustainable Water Sector Capacity Development (SWSCD) Activity. Under this contract, the evaluation team independently evaluated the processes, outcomes, and effectiveness of SWSCD’s activities and assessed the extent to which they achieved expected results. It also identified gaps in performance against Activity targets. Lastly, the evaluation team assessed whether the Activity created opportunities for the sustainability of its training program and sector certification program and identified best practices and lessons learned. USAID/Washington funded the Albanian Association of Water Supply Companies (SHUKALB) through USAID Cooperative Agreement (CA) No. AID-182-A-14-00001 to conduct the SWSCD Activity. The Activity started September 30, 2014 and finished at the end of March 2019, with total funding of $1,208,396. SWSCD supported the fulfillment of Mission Development Objective (DO) 1 “Strengthening the rule of law and improved governance” through intermediate results (IR) 1.3 “improved performance and management.” It also supported objective 4 of the Albanian Water Supply and Sewerage Services Sector Strategy (2011–2017), particularly Priority Action 4.a: “Establish a national program of training and certification, with training target requirements at all water utilities.” Fully in line with the stated sector priorities of the Government of Albania, as well as USAID’s development objectives, the objective of the SWSCD Activity was “to build a platform for training in Albania and the Western Balkans to be an example of sustainable workforce development while contributing to the increased professionalism in delivery of the essential public services of water supply and wastewater treatment.” The evaluation team based its evaluation of SWSCD on three questions stated in its Statement of Work: 1. What has SHUKALB’s performance been to date and what has the activity achieved? 2. What, if any, opportunities were created for the sustainability of the certification program? What are those opportunities and what, if anything, needs to be done to ensure that the opportunities are realized? 3. What, if any, best practices and lessons learned can be identified as a result of this activity? To answer these questions, the evaluation team conducted primary and secondary research and collected qualitative and quantitative data. During fieldwork, the evaluation team undertook 41 key informant interviews (KIIs) and one focus group discussion (FGD), reaching 53 people from 29 different stakeholders. THE SWSCD ACTIVITY WAS QUITE SUCCESSFUL, DELIVERING MOST OF ITS EXPECTED RESULTS AND ALL OF THOSE THAT WERE WITHIN ITS CONTROL. It also fully responded to the aforementioned Mission DO 1 “Strengthening the rule of law and improved governance” and Objective 4 of the Albanian Water Supply and Sewerage Services Sector Strategy (2011–2017), Priority Action 4.a. The key findings on activities implemented and results achieved under each of the SWSCD components, are listed below in order of importance. During the period of performance, SWSD: 1. Officially established the Test-Based Certification Program in Albania, governed by a Certification Board composed of key sector stakeholders. In September 2018, the Ministry of Infrastructure and Energy (MIE) issued Instruction No. 660, on “Establishing an Institutionalized National Training and Test-Based Certification Program for operational managers of Water Supply and Sewerage Utilities in Albania.” Based on the findings, the provisions of this instruction will be included in the new Water Sector Law. 5 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV 2. Developed the necessary materials, in terms of training curricula, course modules, and tests in Albanian to cover all the training needs of the water and sewerage (WS) sector, along with those of the Test-Based Certification Program. All materials are high quality and replicable. 3. Engaged and raised the capacity of a broad base of trainers to support the Test-Based Certification Program, as well as other trainings related to the WS sector. The Activity engaged 35 trainers with course material development and provided them with a train-the-trainers course cycle. 4. The Test-Based Certification Program, as emerged from SWSCD, is fully sustainable in terms of: (a) commitment by the Albanian authorities (based on point 1 above); (b) cost sustainability, as the MIE supports the option of cost-sharing between WS utilities and central government budget; (c) institutional setting, which is conducive to its sustainable implementation because of an adequate Board composition, the central role of the MIE in test administration, the outsourcing of the testing process to an independent outside organization, and the plurality of training delivery; and (d) a well-trained workforce with a high-quality set of course programs and materials. 5. Established successfully a training network with the Kosovo Water Association. A broader network including other countries in the region was not established due to factors outside of SWSCD’s control. This was the only area where the objective was not reached in full. 6. Created an opportunity for individuals from Kosovo’s WS sector to participate in the Certification Program by facilitating a ministry-level agreement between the two countries. 7. Enhanced the capacities of SHUKALB staff and maintained the association’s good management practices, through the development of business plans. 8. Initiated and piloted an innovative online training program. THE TEST-BASED CERTIFICATION PROGRAM, AS IT EMERGED FROM SWSCD, IS SUSTAINABLE IN ALL RELEVANT ASPECTS. The GoA, through the MIE, has formally committed to endorse the Program and is expected to include it in its new Water Sector Law. There is a feasible option to cover the costs associated with its implementation, in the form of cost sharing between the WS utilities and the central government budget. There is also a conducive institutional setting for the Program, including: a broadly representative Certification Board, the leading and exclusive role of MIE/AKUM in administering the testing process, the anticipated liberalization of WS utility trainings supporting the Certification Program, and a testing process outsourced to a reputable and specialized institution. SWSCD provided the full structure and roadmap to implement the certification program. It is now up to MIE/AKUM to undertake the procedural steps to initiate it. THE EVALUATION TEAM IDENTIFIED THE FOLLOWING BEST PRACTICES, which provide lessons learned for similar activities and programs: 1. The Activity established an Advisory Group and later, a Utility Advisory Group. These measures ensured Activity leadership, facilitated MIE action in establishing its Working Group, and attracted stakeholder support throughout the Activity’s lifetime. This aided in the completion of most Activity components, as many of these depended on the successful establishment of the Certification Program. 2. Inclusion of a broad base of professionals as future trainers. Although the exact number was neutral to the outcome of SWSCD, the broad base of trainers will ensure future training capacity when needed, thereby contributing to Activity sustainability. 3. SWSCD maximized stakeholder engagement by maintaining regular communication with key stakeholders throughout the Activity’s lifecycle. This allowed SWSCD to achieve key results (most notably the establishment of the Test-Based Certification Program) despite high staff turnover at the MIE, including at the highest levels. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 6 1. INTRODUCTION This report presents an independent evaluation of the Sustainable Water Sector Capacity Development (SWSCD) Activity. The SWSCD Activity (October 2014–March 2019) was initiated following an application by the Albanian Association of Water Supply Companies (SHUKALB1) at USAID Washington (USA), which selected and funded it through USAID Cooperative Agreement (CA) No. AID-182-A-14- 00001. The Activity was designed to support the fulfillment of Mission Objective 4 of the Albanian Water Supply and Sewerage Services Sector Strategy (2011–2017), and particularly Priority Action 4.a: “Establish a national program of training and certification, with training target requirements at all water utilities.” Fully in line with the stated sector priorities of the Government of Albania (GoA), and USAID’s development objectives, the overall objective of SWSCD was “to build a platform for training in Albania and the Western Balkans to be an example of sustainable workforce development while contributing to the increased professionalism in delivery of the essential public services of water supply and wastewater treatment.” The original financial agreement was based on a cost-sharing principle, with USAID contributing $1,208,396 and SHUKALB providing in-kind contribution of US$134,266 (i.e., 10%) amounting to a total of $1,342,662. The Activity timeframe, originally ending in September 2017, was extended by USAID through the end of March 2019. The final award value was $1,208,396, with the financing agreement on cost-sharing fully observed by SHUKALB. 1.1. DESCRIPTION OF THE OBJECT AND SCOPE OF THE EVALUATION SHUKALB’s work under SWSCD was based on a cooperative agreement (CA) that included various Activity components. The evaluation team, in agreement with USAID, conducted the evaluation based on these components, which were as follows: 1. Strengthen the capacities of the Water Supply and Sewerage Association of Albania through the training of its staff, and the development of a sound business plan and operational procedures for the sustainable conduct of the business of the Association. 2. Greatly expand the current inventory of fully documented training courses to meet the skills development and certification training needs in the water supply and sewerage sector of Albania. 3. Develop and train local national trainers to be able to deliver all courses in the training program. 4. Establish a not-for-profit institution, governed by a selected board of water sector stakeholders, that would guide the development of a sustainable water supply and sewerage skills and certification training program in Albania. 5. Support/advocate the establishment of test-based certification of water supply and sewerage system operators, where such certification would be a precondition of employment in the sector. 6. Promote formal water supply and wastewater training in the region, by working with Albania's neighboring countries under the Western Balkans Partnership of Water Associations. 7. Institute an innovative training delivery model.2 The purpose of this performance evaluation is to provide a rigorous, evidence-based, and independent analysis of SWSCD’s performance. Specifically, the performance evaluation will serve to examine the processes, outcomes, and the effectiveness of the project activities implemented, to determine the achievements of the expected results according to the Activity description/outcomes, and to identify gaps in performance against targets. The evaluation focuses on both management and technical aspects of the 1 SHUKALB is the acronym of the name of the association in the Albanian language. This notation will be used throughout the report. 2 Not in the CA; this component was added since the very start of SWSCD. 7 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV program and its implementation. It is intended to inform USAID of the success of the program and identify lessons learned to share with the implementing partner. This evaluation was based on three evaluation questions per the Statement of Work (SoW) (presented as Annex A of this report), which defined the scope of the evaluation as follows: 1. What has SHUKALB’s performance been to date and what has the Activity achieved? 2. What, if any, opportunities were created for the sustainability of the certification program? What are those opportunities and what, if anything, needs to be done to ensure that the opportunities are realized? 3. What, if any, best practices and lessons learned can be identified as a result of this activity? 1.2. BRIEF OVERVIEW OF WATER SECTOR IN ALBANIA During the past 25 years, the water and sewerage (WS) services sector in Albania has undergone important changes, resulting in a sector structure with separate legislative, regulatory, and service delivery functions, and a decentralized service delivery (since 2000). Until 2015, 57 WS utilities served urban areas, whereas numerous communes (komuna) provided WS services in rural areas. The GoA’s territorial and administrative reform of late 2015 reduced the number of local government units (LGUs) from 345 to 61, abolishing all communes, and expanded the jurisdiction of the existing WS utilities to both urban and rural areas of their respective municipalities. The WS utilities are joint stock companies, owned 100 percent by the municipalities that they serve.3 Of the 57 WS utilities, 53 are licensed to operate, 48 operate with tariffs approved by the national Water Regulatory Entity (WRE), and 32 provide wastewater services (five of which also perform wastewater treatment). Most capital investment in the sector comes from foreign donors and international financial institutions in the form of loans or grants and the state budget. These investments focus on construction and rehabilitation of WS systems, as well as construction of wastewater treatment plants. The operation and maintenance (O&M) cost coverage is partly realized through revenues generated by the utilities, and partly covered by government subsidies. Despite improvements, sector performance metrics remain below the strategic objectives of the GoA for the sector: water supply service coverage (80 percent) has grown but the wastewater service coverage (50 percent) continues to lag behind objectives, along with compliance with water quality and wastewater effluent standards. The main concern in the sector is the high level of water losses (65 percent). In terms of staff efficiency, the number of staff members per 1,000 connections is high, at 9.14. Finally, the average number of hours per day of water supply remains low, at 11.4.4 These indicators have not seen significant improvement in the past 25 years, despite infrastructure improvements. Among the underlying causes is the insufficient level of preparedness of the workforce, as recognized by the GoA. The GoA has produced several WS sector policies and strategies to improve utility performance and progressively streamline legislation with the EU Water Framework Directive. The most recent National Strategy for Development and Integration, 2014–2020 (NSDI II),5 outlines the necessary reforms Albania must undertake to significantly improve sector performance.6 One of its five mission objectives is to “invest in enhancing the capabilities of the sector workforce.” This objective calls for establishing a standardized, national program of WS sector workforce training and certification.7 3 In the 1 or 2 cases in which they serve more than one municipality, the ownership is shared. 4 WRE Annual Report 2017. Available at http://www.erru.al/doc/Raporti_vjetor_2017.pdf 5 https://shtetiweb.org/wp-content/uploads/2014/06/NSDI_2014-2020_version_JUne-2013.pdf 6 There is a new National Strategy draft being prepared at the time of this report. 7 http://www.erru.al/doc/NATIONAL_STRATEGY_OF_WS.pdf, p.28. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 8 1.3. EVALUATION METHODOLOGY In this section, the evaluation team briefly describes the three phases of the evaluation, along with the limitations of the evaluation and the mitigation measures the evaluation team applied. INCEPTION. After launching the evaluation, the evaluation team reviewed secondary data shared by USAID/Albania and SHUKALB. The team prioritized its review of these materials to further familiarize itself with the associated activities, elaborate the list of key informant interview (KII) and focus group discussion (FGD) participants, and develop detailed evaluation sub-questions tailored to the SWSCD evaluation context. The desk review informed the evaluation work plan, which USAID/Albania reviewed and approved. DATA COLLECTION. After USAID/Albania approved the evaluation work plan, the evaluation team conducted internal coordination meetings to ensure that all members shared a common understanding of the instruments and techniques for implementing the SWSCD evaluation. The team then began KIIs with the identified stakeholders in Tirana and with the selected WS utilities and their municipalities between March 16 and April 11, 2019. During this fieldwork, the evaluation team undertook 41 KIIs and 1 FGD, reaching 53 individuals from 29 stakeholders. The list of stakeholders included: 1. Implementing partners 2. Indirect stakeholders 3. Policy makers and regulatory institutions 4. Beneficiaries The evaluation team reviewed primary and secondary sources of information on the SWSCD Activity and the water sector. Primary sources of information included SWSCD reports, monitoring data, and other deliverables. Secondary sources of information included reports published by the GoA. The evaluation team delivered a final presentation at the USAID/Albania office on the final day of the data collection phase, focusing on its preliminary findings and recommendations. USAID/Albania provided initial comments, which the evaluation team took into consideration in preparing the current report. DATA ANALYSIS AND REPORTING. Data analysis involved a combination of qualitative and quantitative techniques. Most of the data collected and analyzed were qualitative data gathered through KIIs and the FGD, as well as desk review. Quantitative data reviewed related to the three quantifiable performance indicators (PIs) provided by USAID. The evaluation team triangulated all data, as appropriate, to address the evaluation questions, and where relevant, introduced contextual information (e.g., from the sector performance) to help with the interpretation of evaluation findings. The evaluation team structured this report to answer the evaluation questions by providing specific information on each component of the SWSCD, with the goal of enabling the reader to understand the context of the findings along with the conclusions and recommendations. The evaluation questions listed in the SoW, along with the corresponding evaluation sub-questions, are presented in Exhibit 1. 9 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV EXHIBIT 1. EVALUATION QUESTIONS AND SUB-QUESTIONS EVALUATION QUESTIONS EVALUATION SUB-QUESTIONS 1. What has SHUKALB’s performance been to date and what has the Activity achieved? 1.1. What project activities supported each component, and how do the results achieved compare against the expected outputs/results? 1.2. If outputs/results were not achieved in full, what were the reasons (differentiating between factors that objectively and reasonably, were within or outside of the Activity’s control)? 2. What, if any, opportunities were created for the sustainability of the certification program? What are those opportunities and what, if anything, needs to be done to ensure that the opportunities are realized? 2.1. What opportunities were created for the sustainability of the certification program in Albania? 2.2. What needs to be done to ensure that the opportunities are realized? What are the challenges and what measures need be undertaken to overcome them? 2.3. What challenges or opportunities were created for water supply and wastewater certification programs in the region? 3. What, if any, best practices and lessons learned can be identified as a result of this Activity? 3.1. What were best practices emerging from the Activity implementation that were particularly conducive to it reaching its objectives? 3.2. What aspects of Activity implementation could have been approached differently, leading to better results? 1.4. LIMITATIONS AND MITIGATION MEASURES The evaluation faced some limitations in conducting this evaluation; however, the evaluation team applied mitigation measures where possible. These limitations did not affect the robustness of the evaluation: SUBJECTIVITY AND SEMI-STRUCTURED INTERVIEWS. The semi-structured approach to the interviews created the opportunity to explore specific issues and themes in more depth. This increased flexibility but also introduced the risk of interviewer subjectivity. The evaluation team mitigated this problem by carefully triangulating KII data with data from other sources. LIMITED DATA. Although the team received a substantial amount of project documentation, it could not be sure that all relevant information was contained within the documents received. For example, some details of events related to SWSCD activities may not have been reflected in SHUKALB Annual Reports, but may be instead described in other documents, including SHUKALB’s “Burimi” newsletter. In addition, given the time lag (the Activity started in 2014), some interviewees acknowledged having forgotten relevant details. Finally, some of the utility managers were new in their position (1–2 years) and the former executive director of SHUKALB, Mr. Giantris, passed away in late 2017, potentially leading to lost information on the implementation of the SWSCD Activity. To mitigate these issues, the evaluation team: (1) reviewed all documents available, including all available editions of SHUKALB’s “Burimi” newsletter (2016, 2017, 2018), which included information on SWSCD-related activities unrecorded in its annual reports; (2) followed up by phone or email with interviewees who needed more time to recall events; and (3) conducted three in-person interviews with SHUKALB representatives and following up with phone calls and emails to clarify information. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 10 2. KEY FINDINGS 2.1. QUESTION 1: WHAT HAS SHUKALB’S PERFORMANCE BEEN TO DATE AND WHAT HAS THE ACTIVITY ACHIEVED? To provide for an all-compassing and thorough presentation of the findings, in this section the evaluation team discusses findings related to Evaluation Question 1 (and related sub-questions per Exhibit 1) separately for each component of the SWSCD Activity. Cross-cutting issues, which include the technical management by SHUKALB and gender considerations, are also addressed here, separately. COMPONENT 1: STRENGTHEN THE CAPACITIES OF SHUKALB THROUGH THE TRAINING OF ITS STAFF, AND IN THE DEVELOPMENT OF A SOUND BUSINESS PLAN AND OPERATIONAL PROCEDURES FOR THE SUSTAINABLE CONDUCT OF THE BUSINESS OF THE ASSOCIATION. This component was not linked to any measurable PI. The evaluation team therefore compares activities performed against the work plan and assesses their results based on stakeholder perceptions. ACTIVITIES PERFORMED. SWSCD addressed this component by (1) improving staff capacities of SHUKALB through trainings, study visits, and events, and working in close cooperation with international and local experts engaged by SWSCD; and (2) developing business plans to guide its activities. 1. Staff capacity building activities. These activities included (a) training SHUKALB staff, Board of Directors, Project Advisory Council, and Young Professional Advisory Group on Advocacy Policy (Annual Report Year 1); (b) conducting a study tour in the United States (Annual Report Year 2); (c) attending a conference on “Innovations in Certification” in Florida (US) organized by the Association of Boards of Certification (ABC) (Annual Report Year 3); and (d) participating in a training for trainers on non-revenue water reduction in Macedonia (Annual Report Year 2). Throughout the Activity, SHUKALB staff assisted various international and local experts with course development, preparing documentation specific to the certification, as well as updating/drafting business and strategic plans (Annual Report Years 2,3,4; interviews with SHUKALB; FGD feedback). 2. Business plan development. In 2016, SHUKALB staff updated the existing business plan of 2014. The Board of Directors approved the 2016–2020 business plan (Annual Report 2) along with a further update in 2017 (Annual Report 3). Assisted by an international expert, SHUKALB developed a strategic plan to address the new opportunities and challenges arising from the implementation of the Test-Based Certification Program (Annual Report 4). The strategic plan was finalized and presented to the Board of Directors for review and approval (SHUKALB email, 8 April 2019). In comparing the annual plans with realized activities, a planned training with IAWD/DWA on management issues was not pursued. The reason was that the training program with University of Arkansas Clinton School of Public Service (UACS) had already covered the topics intended for the above training, thereby making it redundant (Annual Work Plan 1; Interview with SHUKALB). Furthermore, the Activity also did not conduct internships for SHUKALB staff with the New England Water and Environmental Agency (NEWEA). Although SHUKALB staff visited NEWEA, the main reasons for not conducting the internships were time and staff constraints of SHUKALB: internships would have reduced SHUKALB’s capacity to deliver on schedule other outputs considered to be higher priority (Annual Work Plan 1, 2; interview with SHUKALB). RESULTS. Training institutions and international experts appreciated the SHUKALB staff’s professionalism, responsiveness, and engagement in the trainings (interviews with training institutions and 11 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV international experts; FGD). Meanwhile, SHUKALB indicated that the staff benefited greatly from the trainings and capacity-building activities, especially those related to the Certification Program. For example, the Technical Manual on the Certification Program, drafted by the SWSCD Activity, was the result of work performed by international experts assisted by trained SHUKALB staff. Regarding the development of business plans, SHUKALB and its staff had experience operating on business plans prior to 2016, under the leadership of its former executive director. Demonstrating the baseline capacity of the association’s staff, the international expert approached under SWSCD to develop the SHUKALB business plan and strategic plan did so in collaboration with SHUKALB staff. In SHUKALB’s view, the learning-by￾doing aspect of the capacity building was particularly beneficial to the staff. COMPONENT 2: GREATLY EXPAND THE CURRENT INVENTORY OF FULLY DOCUMENTED TRAINING COURSES TO MEET THE SKILLS DEVELOPMENT AND CERTIFICATION TRAINING NEEDS IN THE WATER SUPPLY AND SEWERAGE SECTOR OF ALBANIA. ACTIVITIES PERFORMED. The SWSCD Activity developed curricula based on a training needs assessment conducted two years prior to the start of the Activity. Given the sector, the training needs assessment remained relevant when the Activity started. In addition, four months into the period of performance, the Activity sent a Training Questionnaire to donors in Albania, to identify past, current, and future training topics8 (Annual Report 1; interview with SHUKALB; interview with former WRE official). After obtaining authorization, SHUKALB used training materials from California State University, Sacramento (CSUS) as the basis for developing the trainings to match the needs identified in the assessment. The Activity engaged various international and local experts to work on the training curriculum for the Certification Program and the adaptation of course materials. As a result, the Activity developed training curricula for certification courses at all levels and areas of certification (Annual report 4; Quarterly Report 4, 2018) and developed 46 training courses to address training needs for skills development for WS utilities’ staffs.9 Each course package contains a trainee’s guide, a trainer’s guide, and visual materials in editable formats, in both English and Albanian (Annual Reports, 1–4). Of these training courses, 15 are specific to the Test-Based Certification Program, covering all four WS utility functional areas: (1) water production and treatment; (2) water distribution; (3) wastewater collection; and (4) wastewater treatment, as well as the four levels of certification: • Level 1 – qualified workers • Level 2 – supervisors • Level 3 – managers • Level 4 – technical directors To maintain integrity in the selection of certification exam questions, the process of selecting and reviewing the questions was done using ABC’s secure online platform (Annual Report 4). RESULTS. The training courses are of a high quality, tailored to Albanian context. On the theory-practice ratio of the training modules, there is room for improvement: practical training is foreseen, but the equipment and facilities are lacking. To overcome this problem, the least-cost, feasible, and adequate solution would be to use the on-site facilities and equipment of various WS utilities, in agreement with 8 The donors drive all key infrastructure development in the sector; hence their training programs directly indicate mid- to long-term needs. 9 Forty-six courses were developed or adapted; five courses were existing. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 12 them. This approach was used successfully in previous SHUKALB trainings (FGD feedback; interview with donors; interview with WRE). This component was linked to one of the three PIs, namely “number of training courses developed or adopted.” The findings on this indicator (based on the Monitoring and Evaluation [M&E] Report Summary Table) are shown in Exhibit 2. EXHIBIT 2. NUMBER OF TRAINING COURSES DEVELOPED OR ADOPTED (PI 1) YEAR ANNUAL PLANNED TARGET ANNUAL ACTUAL ANNUAL PERFORMANCE ACHIEVED 2015 11 10 90.9% 2016 18 21 116.7% 2017 12 10 83.3% 2018 5 5 100% 2019 n/a10 n/a n/a Total 46 46 100% Based on Exhibit 2 and stakeholder perception, the SWSCD Activity achieved the outputs and results for this component in full. COMPONENT 3: DEVELOP AND TRAIN LOCAL NATIONAL TRAINERS TO BE ABLE TO DELIVER ALL COURSES IN THE TRAINING PROGRAM. ACTIVITIES PERFORMED. The training of trainers (ToT) program delivered in 2016–2017 included 35 local professionals.11 The objective of the ToT was to equip the future trainers, who will be supporting the Certification Program, with skills and knowledge on adult learning principles and techniques. The criterion used to select the participants to the ToT was their involvement with the development of the training materials for the Certification Program (Annual Reports 2, 3; FGD; interview with SHUKALB). The courses were developed and delivered by two Albanian experts in adult learning, advised by an international expert (Annual Reports 3, 4; FGD; interview with SHUKALB).12 In addition to the ToT, the 35 local trainers worked together with various international experts to develop and refine the WS utility training courses, contributing to their overall capacity development. RESULTS. The opinions of the local trainers who participated in the FGD for this evaluation were unanimous in their appreciation of the training and the trainers. They believed that they benefited a great deal from the training and felt that, as a result, their performance as trainers significantly improved. This component was linked to one of the three PIs, namely “person-hours of training received in the ToT program.” The findings on this indicator (based on the M&E Report Summary Table) are shown in Exhibit 3. EXHIBIT 3. PERSON-HOURS OF TRAINING RECEIVING TOT PROGRAM (PI 2) YEAR ANNUAL PLANNED TARGET ANNUAL ACTUAL ANNUAL PERFORMANCE ACHIEVED 2015 n/a n/a n/a 2016 720 270 37.5 % 2017 360 456 127 % 2018 360 359 99.7% 2019 n/a n/a n/a Total 1440 1085 75.3% 10 n/a indicates there was no activity planned in that year. 11 Of these, two were staff of Kosovo Regional Companies and eight of SHUKALB. 12 The course module included trainee guide, trainer guide, and the PowerPoint presentation. 13 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV Based on Exhibit 3, in terms of outputs, there was a discrepancy between target and realization observed in 2016. The reason for this discrepancy was a downward correction in the length of the ToT course, which was originally planned to be 5 days, but this was unfeasible for the participants (interview with SHUKALB). The factors leading to non-realization were within the Activity’s control to a good extent. Instead of proposing long courses, the Activity could have split the program into shorter courses and delivered them during its period of performance. However, in terms of results, the stakeholder feedback suggests that the Activity achieved its Component 3 objectives in full. COMPONENT 4: ESTABLISH A NOT-FOR-PROFIT INSTITUTION, GOVERNED BY A SELECTED BOARD OF WATER SECTOR STAKEHOLDERS, THAT WOULD GUIDE THE DEVELOPMENT OF A SUSTAINABLE WATER SUPPLY AND SEWERAGE SKILLS AND CERTIFICATION TRAINING PROGRAM IN ALBANIA. ACTIVITIES PERFORMED. The board of water sector stakeholders governing the Test-Based Certification Program was established through Instruction 660.13 However, the Activity did not undertake any other specific activities under this component, as there was no need because of the following reasons: 1. Since 2007, SHUKALB, not a separate or subordinate institution, delivers trainings to the WS utilities. Over the course of the SWSCD Activity, SHUKALB, which was already a not-for-profit institution,14 did not change its legal status. Therefore, it already fulfilled the requirement of establishing a “not-for-profit institution.” 2. Based on Instruction 660, the responsibility for the implementation of the Test-Based Certification Program is placed with the MIE (through the National Water Supply-Sewerage and Waste Infrastructure Agency [AKUM]). SHUKALB and other training institutions, based on criteria to be developed by AKUM, will deliver the trainings supporting the Test-Based Certification Program in Albania. COMPONENT 5: SUPPORT/ADVOCATE THE ESTABLISHMENT OF TEST-BASED CERTIFICATION OF WATER SUPPLY AND SEWERAGE SYSTEM OPERATORS IN ALBANIA, WHERE SUCH CERTIFICATION WOULD BE A PRECONDITION OF EMPLOYMENT IN THE SECTOR. ACTIVITIES PERFORMED. This component was key to the SWSCD and addressed through various types of activities, as follows: Preparatory work and engagement with implementing partners—key activities • Early in 2015, SHUKALB established an advisory council with key sector stakeholders to provide guidance and support to the Activity through its implementation (Annual Report 1). • On June 24, 2015 a memorandum of understanding (MoU) was signed between USAID and MIE, committing to cooperate in a mutual effort to develop a formal Test-Based Certification Program, to be institutionalized within the GoA (Annual Report 1). • In 2015, SHUKALB established a utility advisory committee whose members were all WS utilities’ representatives.15 Its role was to support the Working Group (WG) on technical aspects related to the Certification Program (Annual Report 1). • In 2015, SHUKALB signed a memorandum of agreement with UACS. UACS was engaged to provide training to the Board of Directors and staff of SHUKALB on the principles of public advocacy, and specific advisory services related to supporting and advocating for the establishment of test-based certification of WS operators in Albania (Annual Report 1). 13 Instruction No. 660, dated 28 September 2018, “Establishing an Institutionalized National Training and Test-Based Certification Program for operational managers of Water Supply and Sewerage Utilities in Albania.” Issued by the MIE. 14 SHUKALB Statute, 2008. 15 Tirana, Durres, Korca, Tepelena, Lezha, Vau i Dejes, Berat-Kucove, and Kamza. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 14 • In late summer of 2015, the MIE, supported by SHUKALB, established the WG to guide the Test￾Based Certification Program.16 The WG was based on the former Advisory Council (which was incorporated into it), with some changes and expansion of its membership. • In 2016, SHUKALB negotiated with ABC to develop the certification tests (Annual Report 3). These activities were fundamental to the implementation of the SWSCD Activity. Advocacy, awareness raising, information dissemination SHUKALB undertook various and continuous activities to raise awareness, inform, and ask for support in the establishment of the Certification Program. In 2015, two workshops were organized to raise awareness about the Certification Program, involving donors in Albania’s water sector, central government stakeholders, municipalities, and WS utility representatives (Annual Report 2). In the same year, UACS experts developed and presented three workshops on effective advocacy with various local stakeholders (interview with former WRE chairman, email interview with UACS). In early 2016, SHUKALB organized four regional meetings with all the WS utilities to present the first ten courses developed as part of this Activity. Several times during 2015–2018, international experts engaged by SHUKALB conducted meetings with key stakeholders and participated in various workshops to raise awareness and discuss technical details related to the Certification Program. In addition, SHUKALB managers met individually with key donors on various occasions to inform them about the SWSCD and ask their support to lobby for the Test-Based Certification Program (Annual Reports 1–4; interviews with various donors). Direct assistance to the MIE and Working Group with drafting supporting technical documents SHUKALB coordinated all seven WG meetings in 2015–2018. These meetings, which were small in number because of the limited degree of responsiveness by the MIE, were productive overall and enabled the stakeholders to effectively discuss the various options and drafts presented by SHUKALB, and render their finalization possible (Annual Reports 1–4; interview with MIE). In January 2017, SHUKALB delivered to the MIE the final draft of the “Technical Manual on Test-Based Certification Program in the WS Sector in Albania,” which formed the basis of the Certification Program17 (Annual Report 3). As part of the continued support provided by SWSCD to the MIE, during 2018, the Activity developed and submitted to the MIE the following documents to facilitate the future implementation of the Certification Program (Annual Report 4): • Draft internal regulation for the unit/department in charge of implementing the Test-Based Certification Program at the AKUM • A final draft set of regulations/standards for the Board of Certification • A manual containing example document templates for the implementation of the certification program (for use by AKUM) • A high-level road map describing the past and current progression of the Activity and highlighting important next steps for program implementation Direct assistance to the MIE in legal drafting to implement the Test-Based Certification Program SHUKALB provided support in drafting Instruction 660. Further, SHUKALB, through the Activity, has been contributing comments and inputs that have resulted in including a specific article in the draft water sector law on certification, as well as in elaborated details of implementation of certification, included in the draft water sector strategy (Annual Report 4). 16 The WG members represented MIE (chair), WRE, National Agency of Vocational and Professional Formation, the National Water Council, the Faculty of Civil Engineering (TU), and WS utilities. 17 The Technical Manual on Test–Based Certification Program is an annex of Instruction 660. It is compliant with the national Framework on Qualifications (SHUKALB email; review of Technical Manual on Test-Based Certification Program). 15 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV RESULTS. In September 2018, the MIE issued Instruction No. 660, “Establishing an Institutionalized National Training and Test-Based Certification Program for Operational Managers of Water Supply and Sewerage Utilities in Albania.”18 The objective of the program is “to train and raise the professional skills and operational performance of the personnel working in a utility by way of test-based certification requirements and procedures.” Instruction No. 660 stipulates the establishment of the Institutionalized National Training and Test-Based Certification Program, which fulfilled this component. Stakeholders were highly appreciative of the professionalism, flexibility, and goal-oriented approach taken by SHUKALB. They indicated that the tasks performed under this component were duly fulfilled and successfully reached their goal (interviews with MIE, WRE, most of the donors, various implementing institutions, and FGD). Although delays occurred because of MIE internal processes during 2016, 2017, and 2018, the Activity achieved its outputs and results related to Component 5 in full.19 COMPONENT 6: PROMOTE FORMAL WATER SUPPLY AND WASTEWATER TRAINING IN THE REGION, BY WORKING WITH ALBANIA'S NEIGHBORING COUNTRIES UNDER THE WESTERN BALKANS PARTNERSHIP OF WATER ASSOCIATIONS. ACTIVITIES PERFORMED. In 2016, SHUKALB and the Association of Water and Sewerage Companies of Kosovo (SHUKOS)20 signed a MoU to establish a formal arrangement between the two associations to achieve common objectives in the area of training and capacity development (Annual Report 2). That same year, SHUKALB cooperated with SHUKOS to host the Balkans Joint Conference and Exhibition titled “Water and Jobs,” with an audience of more than 400 participants from across the Balkan region (Annual Report 3; interview with SHUKOS). In 2017–2018, WS utilities from both Albania and Kosovo held 19 joint training courses. Overall, these training courses trained 146 personnel from Albanian WS utilities and 43 personnel from all seven regional water utilities of Kosovo (Annual Reports 3, 4; interview with SHUKOS). In addition, several trainers from Kosovo were participants to the ToT under the SWSCD Activity (list of ToT participants, provided by SHUKALB). In 2018, SHUKALB held the regional workshop “Establishing a National Training and Test-Based Certification Program for Operational Managers in the Water Supply and Sewerage Sector in Albania—A Pathway to Professional Management and Improved Performance.” All regional water associations21 participated (Annual Report 4). That same year, the deputy executive director of SHUKALB spoke on the Albanian Certification Program at the Water Conference organized by the Montenegrin Ministry of Sustainable Development and Tourism and Montenegro’s Water Association (interview with SHUKALB; interview with IAWD; “Burimi,” April 2018). RESULTS. The SWSCD Activity effectively established a training network between Albania and Kosovo. In 2018, Albania and Kosovo line ministries signed a MoU for a joint Test-Based Certification Program for WS staffs in both countries. SHUKOS appreciated the Activity, citing “excellent cooperation” in all areas. However, no effective cooperation materialized with the other associations in the region and no effective training network including all the countries in the region has been created (to date) or certification 18 The full text of the Instruction can be accessed (in Albanian only) at http://www.qbz.gov.al/Botime/Akteindividuale/Janar%202018/Fletore%20148/UDHEZIM%20nr.%20660,%20date%2028.9.2018.pdf 19 More on this can be found in the Cross-Cutting Issues section. 20 The acronym relates to its name in Albanian. It will be used throughout the rest of the report. 21 Kosovo, BiH, Montenegro, Macedonia, and Serbia. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 16 program started in other countries in the region. In addition to SWSCD, there were at least two other projects22 concerned with similar objectives. The reasons for not fully achieving this component lie outside of SHUKALB’s control: the creation of an effective training network and eventual certification programs are subject to the capacities of the associations concerned, their will to cooperate at that level, and, in relation to certification programs, decisions are subject to the sector policies in the countries concerned. Nevertheless, with regard to the role of SHUKALB in regional awareness raising, the International Association of Water Utilities in the Danube River Catchment Area (IAWD) stated that SHUKALB was key in raising awareness of the Certification Program at the regional level. COMPONENT 7: INSTITUTE AN INNOVATIVE TRAINING DELIVERY MODEL. ACTIVITIES PERFORMED. The innovative training delivery model under the SWSCD Activity utilized online learning. In 2016, SHUKALB outsourced a Feasibility Study for the Pilot Program for Distance Learning in Albania, to provide recommendations on options for the web-based platform, as well as a cost estimation (Annual Report 2). Following SHUKALB’s membership in the IAWD,23 the SWSCD Activity developed a pilot distance-learning training course and implemented it from September 10 to October 5, 2018, using the platform of the Danube Learning Partnership (D-LeaP) Academy. The Activity did not charge participation fees, as the e-course was intended to be a pilot. The course theme was “Practical Guidelines for the Administrative Council Members of Water Supply and Sewerage Utilities,” with an audience of 14 participants coming from various municipalities of Albania (7), WS utility directors (4), and independent professionals (3). RESULTS. Of the 14 participants, three did not attend sufficiently to qualify (i.e., above 75% of the time required). Of the remaining 11, three did not achieve the required score (i.e., above 75%). Of these, two were among the independent professionals who took the course. As a result, eight participants (57%) completed the learning requirements of the course and were awarded certificates (review of “Evaluation of the Pilot e-Learning Course” report). The feedback from the e-learning course participants, provided by SHUKALB, shows that the level of participant satisfaction was very high in all categories.24The average ratings (1 min–5 max) were: • Course objectives and content, 4.58 • Methodology and course materials, 4.76 • Perception on overall impact, 4.76 • Organization and logistics, 4.82 Overall, the feedback on the content relevance, design and methodology, logistics, and perceived impact was very positive. In addition, the feedback from IAWD on the success of the online platform performance was very positive. The outputs under this component were achieved in full and the results were very satisfactory, based on stakeholder feedback and the overall performance of course participants. Regarding future courses online, according to feedback from the IAWD, any of its members (SHUKALB among them) may use, at no cost, its platform for online courses (interview with IAWD). It is reasonable to expect that there will be further demand for online courses among Albanian professionals at the WS utilities, because of the time and cost saving advantages they offer. 22 GIZ Regional Capacity Building Network project, and IAWD–led Danube Hub operated under the D-LeaP Utility Benchmarking Program, both still ongoing. 23 During 2018, SHUKALB received and implemented two grants from the International Water Association of Water Utilities in the Danube Region (IAWD to develop (1) a “Utility Management Training Program Curriculum Framework for the Water and Wastewater Utility Managers in the Danube Region” and (2) a pilot “Distance Learning Platform for Training Delivery in the Water Sector in Albania” (Annual Report 4, p. 5). 24 “Evaluation of the Pilot e-Learning Course,” provided in Albanian language. This document also reflects the participation and results statistics. 17 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV CROSS-CUTTING ISSUES In addition to assessing the performance of SWSCD according to each of its components, the evaluation team assessed relevant cross-cutting issues related to the technical aspects of Activity management by SHUKALB and observance of gender issues. TECHNICAL ASPECTS OF ACTIVITY MANAGEMENT. Over the course of the SWSCD Activity, SHUKALB asked for two no-additional cost extensions: 1. The first extension, of one year, was requested and granted in mid-2016. REASON: support of training course development. CAUSES: (a) difficulties finding international and local experts; (b) delays in translation. 2. In 2018, SHUKALB asked for a second, 6-month extension, which was also granted. REASON: implementing a number of activities, for which MIE action was required. CAUSES: pending activities because of delays by MIE. The first extension was requested due to delays in factors within the Activity’s control. However, in light of the subsequent course of events happening at the MIE, which caused delays outside of the Activity’s control in delivering its main component (i.e. establishing the Test-Based Certification Program), it is likely that the first extension did not affect the end date of the Activity. In 2017, the MIE merged with another ministry, a process that significantly slowed down the implementation pace of SWSCD. During that period, some members of the WG, including the chairperson, moved (for more than six months) to other duties and were not replaced. In the course of 2015–2018 alone, the MIE changed ministers five times, which also had an adverse impact on the otherwise seamless progress of SWSCD. The approach and activities related to the six25 components that SWSCD addressed were overall conducive to its outcomes and objectives. At two moments, the Activity did not realize activities according to the work plan because of SHUKALB management.26 However, based on stakeholder feedback, these seem to not have affected the results. With regards to Activity monitoring, SWCSD was conducted in line with best project management practice: two months into the Activity, SHUKALB delivered an M&E plan to USAID (per the CA) and for each Activity year SWSCD prepared annual work plans with stated objectives. SWSCD implemented internal staff reporting and held staff meetings at least weekly. SWSCD undertook corrective measures as needed; the most important were the extensions it requested and the addition of the activity of preparing the certification tests for the first round of certifications (within the Activity’s budget). The other modifications were minor, and regarded mainly time re-allocation for various events, five staff replacements (in 2015, 2016, and 2017), involving two international experts and three SHUKALB staff. SWSCD reported progress on the performance objectives in the annual reports (Annual Reports 1–4; Annual Work Plans 1–4; modifications to CA; interviews with SHUKALB; feedback from USAID). SWSCD ended on March 31, 2019, in accordance with the final extension. The Activity reached all its objectives in a cost-efficient manner. In addition, SHUKALB managed to create synergies with other programs to enhance the impact of SWSCD on various components. More specifically, the Activity conducted the following activities: 25 Overall, there were seven components, but one was not addressed. 26 One is the missed internships for SHUKALB staff at NEWEA planned in 2015. Better internal planning and workload distribution by SHUKALB could have made these feasible, given the 4-year extension of the project. Another is the reduction of ToT in 2016: instead of planning for 5-day trainings, not feasible to the trainers, the program could have been modified to allow for more short￾term trainings. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 18 1. In 2016 and 2017, five trainers of SHUKALB attended a ToT course in Non-Revenue Water (NRW) in Macedonia (2016), organized by Aquasan Network and GIZ, and another in NRW in BiH management organized in the framework of the D-LeaP in Bosnia Herzegovina (BiH)27 (“Burimi,” May 2016 and April 2017). 2. SHUKALB organized a joint training with the USAID Planning and Local Governance Program for WS Administration Advisory Councils (email exchange with SHUKALB). 3. Through a competitive grant from IAWD, the Activity developed a specific utility management training curriculum for the utility managers (Annual Report 4). 4. In 2017, SHUKALB became part of the Regional Capacity Development Network project implemented by GIZ’s Open Regional Fund (ORF). This project addresses the development of a professional WS sector workforce in the western Balkans. SHUKALB is seen as one of the key contributors to the project because of the training course materials it produced and the established practice of sustainable training delivery (Annual Report 3). 5. In 2017, SHUKALB joined D-LeaP, which provided a grant to develop a training curriculum for the general directors of the WS utilities, and offered the use of the online platform for SWSCD component 7 (Annual Report 3; interview with IAWD). 6. In 2018, supported by the ORF Regional Capacity Development Network Project and IAWD, SHUKALB organized a Regional Workshop on Test-Based Certification Program for Operational Managers in the WS Sector in Albania, with the objective to build awareness and knowledge of such programs. Participants included representatives of WREs, line ministries, water associations, local government associations from Macedonia, Montenegro, Kosovo, BiH, and Serbia, and donor representatives from these countries (Annual Report 4). GENDER ISSUES. SWSCD had a limited scope for gender considerations, because of the nature of the Activity.28 However, the Activity reflected a gender-conscious approach. First, 90% of all Activity staff were female professionals. Second, SHUKALB maintained the gender balance in the selection of the pool of trainers to be included in the Certification Program. Based on the list provided by SHUKALB, 35 of 57 trainers (61%) involved with the SWSCD Activity and the trainings for the WS utilities were female. In addition, among the 35 participants in the ToT, 23 (66%) were female. 2.2. QUESTION 2: WHAT, IF ANY, OPPORTUNITIES WERE CREATED FOR THE SUSTAINABILITY OF THE CERTIFICATION PROGRAM? WHAT ARE THOSE OPPORTUNITIES AND WHAT, IF ANYTHING, NEEDS TO BE DONE TO ENSURE THAT THE OPPORTUNITIES ARE REALIZED? In answering this question, the evaluation team focused on the Activity results, following the findings under Question 1 related to SWSCD components and other findings specific to the sustainability obtained from the KIIs and FGD. The findings under Question 2 include the scope of SWSCD and what it was expected to achieve with regard to the sustainability of the Certification Program. However, the discussion of challenges related to the implementation of the Certification Program and the measures that need be undertaken to make it fully implementable relate to the broader context of the WS sector in Albania and future developments in the Certification Program. These areas are clearly outside the scope of SWSCD and can be handled only by the relevant Albanian institutions. 27 See http://shukalb.al/en/shukalb-merr-pjese-ne-trajnimin-e-trajnereve-per-programin-e-zhvillimit-te-kapaciteteve-uji-pa-te￾ardhura-ne-kuader-te-d-leap/ 28 As a side note: Regarding trainings with the WS companies’ staffs, an activity that was not part of SWSCD and not funded by it, but indirectly related through the third PI (on cost coverage), SHUKALB is not in a position to impose a gender balance: The WS utilities chose the courses to which to send their staff, who are selected based on availability and needs, as well as the profiles suggested by SHUKALB. Should there be a scope for gender considerations, that is within the discretion of the WS utilities. However, SHUKALB keeps evidence on the participants, and based on anecdotal evidence, the gender balance has been observed. 19 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV One finding related to all the interviewed stakeholders is that all of them acknowledged, beyond question, the importance of the Certification Program in the water sector in Albania. 2.2.1. WHAT OPPORTUNITIES WERE CREATED FOR THE SUSTAINABILITY OF THE CERTIFICATION PROGRAM? The opportunities related to the sustainability of the Certification Program, as it was conceived and emerged from SWSCD, reasonably concern the following aspects: (1) commitment by government; (2) [realistic] options for sustainable cost coverage; (3) appropriateness of the institutional setting; and (4) availability of material base and workforce. These aspects are discussed in this section. GOVERNMENT COMMITMENT. The GoA formally endorsed the Certification Program in September 2018. The MIE issued Instruction No. 660, “Establishing an Institutionalized National Training and Test￾Based Certification Program for Operational Managers of Water Supply and Sewerage Utilities in Albania.”29 The objective of the National Training and Test-Based Certification Program is to train and raise the professional skills and operational performance of the personnel working in a utility by way of test-based certification requirements and procedures. Based on findings from interviews held with high officials of the MIE, this Instruction is expected to be integrated into the new Water Sector Law, which is in the process of being drafted. The Certification Program is already mandatory by means of the aforementioned document, but its inclusion into a law will provide a stronger guarantee to that effect. Overall, the evaluation team believes that the GoA has a strong commitment to the sustainability of the Certification Program. COST SUSTAINABILITY. The information related to the cost sustainability of the Certification Program in Albania was derived from the following sources: (1) WS utilities; (2) respective municipalities; (3) MIE; (4) WREs; and 5) donors. At the time of the interviews, none of the WS utilities or the other relevant institutions had prepared any related plans for the Certification Program (e.g., estimated number of staff members to be trained and certified, or training/certification schedules). Therefore, the opinions expressed by the interviewees are based on their own knowledge and experience of the sector/work30 and not on robust calculations. The opinions on the Certification Program-related cost coverage by the WS utilities consisted of the following types of replies: 1. COSTS CANNOT BE COVERED BY THE UTILITIES THEMSELVES, NOR WOULD THE MUNICIPALITY ACCEPT TARIFF INCREASES FOR THIS PURPOSE OR SUBSIDIZE SUCH COSTS. This was the opinion expressed by three WS utilities31 and respective municipalities. Although they recognize the importance and potential benefit of having their staff trained and certified, their low operational cost coverage limits their capacity to fund this activity. 2. COSTS CAN BE COVERED BY THE UTILITIES THEMSELVES. Two utilities and their respective municipalities, and some of the donors interviewed, held this view. The WS utilities 29 The full text of Instruction 660 is published in the official Gazette and can be accessed (in Albanian only) at https://qbz.gov.al/eli/fz/2018/148/33b722d9-9d80-4853-b029-2942d603f5cc;q=certifikimi%20%202018 30 Note that all the interviewed WS utilities operate at a loss, according to WRE annual reports 2015, 2016, and 2017 (see http://www.erru.al/doc/Raporte). 31 Based on information on training participation, provided by SHUKALB, the selected WS utilities for this evaluation have the following record of respective staff participation to trainings organized by SHUKALB, per year, for the period 2016–2018: Based on SHUKALB’s feedback, the main reasons for the relatively limited participation of the WS utilities regard (1) difficulty to cover costs; and (2) shortage of adequate staff suitable for the trainings offered. Such feedback is informal, obtained during contacts with the various utilities (SHUKALB email exchange, 2 April 2019). USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 20 should be encouraged to provide for that in their business plans, being an operational cost. It may or may not imply a tariff raise, but it is an operational cost and should be included as such.32 3. COST SHOULD BE COVERED BY EMPLOYEES WHO WILL BE CERTIFIED, IN FULL OR IN PART. Two municipality representatives held that the employees of the WS utilities should self-finance their training and certification. One of the donors also supported cost sharing between the utility and the employees. 4. COST COVERAGE SHOULD BE INITIALLY SUPPORTED BY THE CENTRAL GOVERNMENT BUDGET, AS NECESSARY. This option implies provision of central government budget support to those WS utilities that cannot realistically cover, in full or in part, the cost of the Certification Program. This was the view of the MIE, WRE, and some donors. Over time, the costs would be gradually borne by the WS utilities, by including them in their operational costs, business plans, and tariff plans.33 Referring to these findings, the evaluation team believes that the scenario described in the fourth type of reply is the most feasible, as it resolves the cost coverage issue for all the WS utilities concerned. The first scenario described will not allow for utility certification, the second will allow only certain utilities to become certified, and the third is not realistic, given the cost of training and the relatively low salaries of WS utility personnel.34 The WRE and MIE share this opinion. The GoA already financially supports all WS utilities based on established criteria and the costs of certification are expected to be a small fraction of the current subsidies. This provides an encouraging premise for the cost-sustainability of the certification program. The third PI of SWSCD concerned cost-coverage of SHUKALB trainings, defined as by what percentage SHUKALB’s annual revenues exceeded annual full cost of WS trainings. The Activity exceeded its targets every year, recording full cost-coverage for the trainings. The results suggest sustainable cost management of the trainings by SHUKALB, which were due to economies of scale of serving a greater number of trainees. The results for this indicator were as follows: EXHIBIT 4. PERCENTAGE BY WHICH TOTAL REVENUES FROM THE DELIVERY OF TRAINING COURSES EXCEED THE TOTAL TRAINING COST (PI 3) YEAR TARGET REALIZED RESULT 2016 10% 20% 200% 2017 28% 32% 114.3% 2018 35% 48% 137.1% APPROPRIATENESS OF THE INSTITUTIONAL SETTING. The institutional setting of the Certification Program was established through Instruction 660. The Certification Program is managed by a Certification Board, with representatives from the MIE, AKUM, the WRE, municipalities, SHUKALB, and two faculties35 of Tirana University. SWCD fully developed the documentation supporting the institutionalization of the Test-Based Certification Program and made it available to the relevant stakeholders (Instruction 660; review of all SWSCD outputs submitted to MIE; interview with ABC). 32 Staff training is considered an operational cost. 33 As a side note, SHUKALB foresees that the costs related to the training and Certification Program will decrease compared with the current training costs. 34 The average cost of a training course provided by SHUKALB ranges from US$110 to 180. Given the need to attend more than one course, and the fact that the certification per se does not guarantee the job place, the incentive for an employee to fund, even partially, such activity is low or none; in addition, the average monthly salary of a WS utility professional is only about 1/3 above the cost of a single course. 35 Faculty of Civil Engineering and Faculty of Economics. 21 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV Although AKUM is in charge of the testing process, ABC will administer the tests through its online platform, based on a MoU between AKUM and ABC. Administering these tests through an external, neutral, and qualified entity is a strong guarantee of the integrity of the testing process. Based on Instruction 660, SHUKALB and other training institutions will train WS utility personnel in areas related to making them eligible for certification.36 AKUM is expected to issue eligibility criteria for the training institutions to support this component. However, it is not clear whether, under Albanian law and best international practice, SHUKALB and other training institutions will need accreditation.37 MIE/AKUM will need to address this. Overall, the institutional setting is conducive to the sustainable implementation of the Certification Program. The representation of the Board is appropriate, as is the role of MIE/AKUM. The approach taken by the MIE to outsource this important process to an outside organization shows its commitment to making the process rigorous. The plurality of training delivery, which is open to other training institutions besides SHUKALB, potentially enhances quality. AVAILABILITY AND QUALITY OF TRAINING MATERIALS AND TRAINERS. At the end of SWSCD, the training materials to support the Certification Program, as well as related tests, for the four fields of the Certification Program and the four professional levels were completed and made available in Albanian. The total number of courses developed (46) covers the full range of sector needs. With regard to the trainer base, SWSCD trained 35 local national trainers through a ToT; they are available to deliver the Certification Program. In addition, SHUKALB had engaged, for the various trainings with the WS utilities, another 22 trainers, achieving a total of 57 potential trainers. The WS sector workforce is estimated to be above 3,000 employees, of whom at least 1,500 will need to be certified based on the present scheme. More may be added as the certification areas expand. As such, the relatively large training workforce, ensuring availability of trainers when needed, is sufficient to cope with any reasonable training workload under the Certification Program. Further, the course materials produced cover all WS utility training areas, including the current certification areas and other areas that may be reasonably added in the future. Overall, in terms of training workforce availability and the availability and quality of training materials, the Test-Based Certification Program is sustainable. STAKEHOLDER PERCEPTION. Local national trainer participants in the evaluation FGD indicated that the training materials for the Certification Program were of good quality. However, they pointed out the need to better combine practical work along with theoretical aspects. Although that has been foreseen in the modules, the equipment and facilities are lacking for the technical/engineering aspects of the trainings. One effective way, in their view, to curtail this shortcoming would be the use of the assets and sites of selected WS utilities for the purposes of trainings, in agreement with them. It is realistic and, as was mentioned by the participants, it was practiced with success during the SHUKALB trainings on courses related to wastewater treatment systems. 36 The text of Instruction 660 bears a slight ambiguity that may lead to misperceptions, as it suggests that SHUKALB is already eligible to conduct these trainings, but it is clear that trainings are not vested with SHUKALB alone. 37 Regarding the process of accreditation for a training institution supporting it, in 2017 SHUKALB contacted the Office of Vocational Trainings and Accreditations at the former Ministry of Social Affairs; the response was that there is no legal basis for such accreditation for non-governmental associations. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 22 2.2.2. WHAT NEEDS TO BE DONE TO ENSURE THE OPPORTUNITIES ARE REALIZED? WHAT ARE THE CHALLENGES AND WHAT MEASURES NEED BE UNDERTAKEN TO OVERCOME THEM? Concerning foreseeable challenges, the evaluation team distinguished those related to the implementation of the Test-Based Certification Program and its impact, as follows: Challenges and measures related to the implementation of the Certification Program: 1. MIE/AKUM FAILS TO COMPLETE, OR DO SO ADEQUATELY, THE REGULATIONS FOR CERTIFICATION OR RELATED TRAINING REQUIREMENTS. In this case, the Certification Program will be delayed, and, when implemented, will face opposition. It is clear that AKUM needs to deliver these regulations to successfully implement the Certification Program. SWSCD has provided sufficient guidance to inform drafting of most of these regulations. 2. NONCOMPLIANCE BY WS UTILITIES AND MUNICIPALITIES. It is possible that some WS utilities will decline to certify their staff as requested, for reasons related to internal politics and interests. To curtail such behavior, the new water sector law should foresee appropriate sanctions (e.g., fines) and the Certification Board, through MIE/AKUM, should make sure to implement them. 3. WS UTILITIES’ STAFF QUALIFICATIONS MAY NOT MATCH THE CRITERIA FOR THEIR JOB POSITIONS AND THEREFORE ARE NOT ELIGIBLE FOR CERTIFICATION. This is already the case in several WS utilities. The measures include staff replacements with eligible candidates, which is the competence of the respective municipality and the WS general director. Should they fail to perform for reasons unrelated to human resource availability, sanctions38 should be applied. In the case when human resources are lacking and there are no replacements (small utilities/areas39), the longer-term (and definitive) solution is the regionalization of the WS in Albania.40 Challenges and measures related to the impact of the Certification Program: Potentially, some municipalities/WS general directors will delay/default in reflecting Test-Based Certification results in their staff appointments. As stated previously, the new water sector law should foresee sanctions against such occurrences and MIE/AKUM should ensure that they are implemented. A long-term solution would be a regionalized WS sector, where a much smaller number of WS utilities (6-12) would benefit from economies of scale that would significantly improve sector performance, including staff- and cost-efficiency. This would subsequently resolve the problem of human resource availability, as the salaries could increase and attract additional WS professionals. Further, these regionalized WS utilities could be jointly owned by several municipalities, which would provide additional scrutiny on the appointment of utility directors and key managers. 2.2.3. WHAT CHALLENGES OR OPPORTUNITIES WERE CREATED FOR WATER SUPPLY AND WASTEWATER CERTIFICATION PROGRAMS IN THE REGION? Albania is the first country in the region to have considered and, following SWSCD, committed at the highest level to proceed with the Test-Based Certification Program. Real opportunities for the Certification Program were created in Kosovo, with a first step being the signing, in November 2018, of a MoU between Albania and Kosovo line ministries for a joint Certification Program for Water and Wastewater Operators in both countries. Whereas SHUKOS will start test-based certifications for its 38 As above, through the new water sector law. 39 Staff hiring from another municipality may not be an option, as salaries offered may not be inducive to potential candidates located outside the areas concerned. 40 This is a priority of the GoA. 23 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV WS utilities’ staffs in Albania, the responsible ministry in Kosovo may consider setting up its own system at a later stage. Although SWSCD has greatly contributed to raising awareness for water sector certification by providing information and setting an example for other countries in the region, it appears that at the time this report is being prepared, no other country in the region has taken initiatives to start work on a similar program.41 2.3. QUESTION 3: WHAT, IF ANY, BEST PRACTICES AND LESSONS LEARNED CAN BE IDENTIFIED AS A LEARNED RESULT OF THIS ACTIVITY? 2.3.1. WHAT WERE BEST PRACTICES EMERGING FROM THE ACTIVITY IMPLEMENTATION THAT WERE PARTICULARLY CONDUCIVE TO IT REACHING ITS OBJECTIVES? WHAT ARE THE LESSONS LEARNED? The evaluation team outlines three BEST PRACTICES that helped SWSCD reach its objectives. These best practices offer LESSONS LEARNED for similar programs. They include the following: 1. CREATING THE ADVISORY GROUP (WHICH LATER BECAME THE WORKING GROUPS FORMED BY THE MIE) AND, LATER, THE UTILITY ADVISORY GROUP. The SWSCD Activity took these steps early on at its own initiative at the beginning of 2015, as part of preparatory work related mainly to the establishment of the Test-Based Certification Program. By doing so, SWSCD took leadership and greatly facilitated MIE action in establishing its Working Group and gained stakeholder support from the very start and throughout the Activity’s lifetime. This facilitated the completion of the majority of Activity components, as most of them depended on the successful establishment of the Certification Program. LESSON LEARNED: Taking leadership early on in an Activity and anticipating essential activities and steps that depend on the decision of the client is key to Activity success. 2. INCLUDING A BROAD BASE OF PROFESSIONALS AS FUTURE TRAINERS. The Activity trained 35 local national trainers in the ToT who were involved in the development of course materials. In addition, the WS sector benefited from 22 other trainers, who were already WS utility trainers working with SHUKALB. Therefore, the number of trainers who can deliver WS utility training in the WS certification areas and other fields is relatively large at 57. Although SWSCD could have developed fewer trainers and still achieved its Activity outcomes, the large number of trainers does encourage the future sustainability of the Certification Program and related WS utility training. The large pool of trainers is a guarantee that there will be a sufficient training workforce available when needed. LESSON LEARNED: To the extent possible, an Activity must take a long-term view and ensure the future sustainability of the institution or activities it creates, in all aspects that the Activity can address. 3. ENSURING STAKEHOLDER ENGAGEMENT THROUGH REGULAR CONTACT WITH KEY ACTIVITY STAKEHOLDERS. SHUKALB staff conducted formal meetings and workshops with central government stakeholders and donors from the beginning of the Activity to the point at which the MIE took legal steps to enact the Certification Program near the end of the Activity. By taking this approach, SWSCD maximized support from stakeholders and kept them engaged. This practice also mitigated the information gaps arising from high rate of staff turnover at the 41 Based on the interview with IAWD/Danube Learning Program. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 24 MIE and among WS utilities’ top management. Given this high rate of staff turnover at the MIE, including at the highest levels, SWSCD’s results related to its key component, the establishment of the Test-Based Certification Program, could have been compromised had SWSCD not maintained regular contact with Activity stakeholders. LESSON LEARNED: Stakeholder engagement is critical to delivering successful outcomes; one effective way of doing so is maintaining constant and regular contact with key stakeholders over the entire Activity period of performance. 2.3.2 WHAT ASPECTS OF ACTIVITY IMPLEMENTATION COULD HAVE BEEN APPROACHED DIFFERENTLY, LEADING TO BETTER RESULTS? The SWSCD Activity should have taken a longer-term view by researching whether it needed to address the accreditation of training institutions under the Certification Program in the context of Albanian laws and standards, based on international best practice. MIE and AKUM should have been provided with informed opinions and, eventually, suggested options to address this issue. This would have provided additional support to MIE and AKUM, potentially leading to a better-regulated institutional setting for the implementation of the Certification Program, prior to its start. However, this is certainly an aspect that MIE and AKUM can and should address in due time, and does not hinder, in any way, the start or credibility of the Certification Program. 25 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV 3. CONCLUSIONS AND RECOMMENDATIONS The following are the main achievements of the SWSCD Activity, from its inception to date: SWSCD ACHIEVED ITS EXPECTED OUTCOMES. The key findings on the SWSCD Activity’s performance on each of its components outlined in the Activity’s CA support the following conclusions, listed in order of importance. The SWSCD Activity: 1. Supported the process to establish the Test-Based Certification Program in Albania, governed by a Certification Board composed of key sector stakeholders. SWSCD developed the related instruments that will allow Albanian Institutions to smoothly implement the Certification Program in the short term. 2. Developed all the necessary materials in Albanian to cover all the training needs in the WS sector, as well as the specific needs of the Test-Based Certification Program, including training curricula, course modules, and tests. All these materials are high quality and replicable. 3. Engaged and improved the capacity of a broad base of trainers to support the Test-Based Certification Program and other trainings related to the WS sector. 4. Created a regional training network with Kosovo. However, the Activity did not create a broader network including other countries in the region (i.e., BiH, Montenegro, Serbia, and Macedonia) because of factors outside SWSCD’s control. This was the only component for which the Activity did not fully reach its objective. 5. Created an opportunity for individuals from Kosovo’s WS sector participate in the Certification Program by facilitating a ministry-level agreement between the two countries. 6. Enhanced the capacities of SHUKALB staff and maintained the association’s good management practices during its period of performance, mainly through the development of business plans. 7. Initiated and piloted an innovative online training program. SWSCD MADE STRONG PROGRESS TOWARDS ITS OVERARCHING DEVELOPMENT OBJECTIVE. With regard to the overarching SWSCD objective, “To build a platform for training in Albania and the Western Balkans to be an example of sustainable workforce development while contributing to the increased professionalism in delivery of the essential public services of water supply and wastewater treatment,” the SWSCD fulfilled it in most aspects. Although the Activity did not create a regional training network beyond Kosovo and Albania, this outcome was due to factors outside the control of the Activity. THE TEST-BASED CERTIFICATION PROGRAM, AS IT EMERGED FROM SWSCD, CAN BE SUSTAINABLY IMPLEMENTED. Any reservations on future sustainability of the Certification Program lie outside the responsibility of the Activity, relating to (1) its future management by MIE/AKUM/WRE and (2) developments in WS sector. The Test-Based Certification Program is sustainable in all the relevant aspects, specifically the following: 1. The GoA, through the MIE, has formally committed to endorse it. This endorsement is expected to be strengthened by including the Program into the new Water Sector Law. 2. There is a feasible option to cover the costs associated with its implementation: cost sharing between the WS utilities and the central government budget, supported by the MIE, WRE, and other key stakeholders who can ensure the implementation of this solution. 3. The institutional setting is conducive to its successful implementation. This is ensured by a broadly representative Certification Board; the leading and exclusive role of MIE/AKUM in administering the testing process; the anticipated liberalization of WS utility trainings supporting the Certification Program; and the guaranteeing of the integrity of the testing process by outsourcing it to a reputable and specialized institution. USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 26 The above findings demonstrate that SWSCD has accomplished the Activity objective of establishing a Test-Based Certification Program in Albania, as well as Mission Objective 4 of the Albanian Water Supply and Sewerage Services Sector Strategy (2011–2017), and particularly priority action 4.a: “Establish a national program of training and certification, with training target requirements at all water utilities.” The Activity has provided the full structure and roadmap to implement it. It is now at the discretion of MIE/AKUM to undertake the procedural steps to effectively start its implementation. SHUKALB’S APPROACH TO PROJECT MANAGEMENT WAS CONDUCIVE TO ACHIEVING ITS OBJECTIVES. In the course of the Activity, SWSCD correctly applied project management best practices. Some of the approaches in Activity implementation were highly appropriate and instrumental to ensuring the Activity’s success. Two instances of incomplete realization of planned activities were not significant and did not affect the results. Delays resulting from the management of SWSCD were minor, whereas other delays that the evaluation team noted were outside the Activity’s control. Activity stakeholders that the evaluation team interviewed were very appreciative of the professionalism, leadership, and flexibility shown by SHUKALB during all phases of the Activity. THE LONG-TERM SUSTAINABILITY OF THE TEST-BASED CERTIFICATION PROGRAM WILL DEPEND ON THE ABILITY OF THE ALBANIAN INSTITUTIONS TO MAINTAIN AND IMPLEMENT IT IN AN ADEQUATE AND SUSTAINABLE MANNER. To ensure that the successful implementation of the Certification Program, the evaluation team recommends that the GoA take the following actions: 1. AKUM should quickly undertake the specific measures stipulated in Instruction 660, in order to start the Test-Based Certification Program. These measures should include the provision of funding from the state budget to support cost coverage for the Certification Program. 2. In including the Test-Based Certification Program in the new Water Sector Law, the MIE, as part of the Drafting Working Group, should add sanctions to mitigate the occurrence of noncompliance with the requirements of the Certification Program, by both WS utilities and the municipalities. (The cases of noncompliance are explained in subsection 2.2.2 of this report.) 27 | PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY USAID.GOV ANNEX A. EVALUATION STATEMENT OF WORK The United States Agency for International Development (USAID) in Albania seeks the services of a Contractor to conduct a performance evaluation of the USAID-funded project “Sustainable Water Sector Capacity Development” (SWSCD). BACKGROUND The SWSCD is designed to directly support USAID’s good governance program in Albania. SWSCD aims at supporting the fulfillment of the Mission Objective 4 of the Albanian Water Supply and Sewerage Services Sector Strategy (2011-2017), and particularly Priority action 4.a: establish a national program of training and certification, with training target requirements at all water utilities. The overall program objective is to build a platform for training in Albania and the Western Balkans to be an example of sustainable workforce development while contributing to the increased professionalism in delivery of the essential public services of water supply and wastewater treatment. The program is designed to deliver the following outcomes/results: • Strengthen governance and development of the technical capacities of the overall WSSA as an institution, its staff, and its Board of Directors. • Establish a long-term contractual agreement with the Harry Fultz Institute by the end of the second year of the grant period. The agreement will foresee use of the Harry Fultz Community College facilities for the training courses. • Expand inventory of fully documented training courses; a total of 39 new technical courses delivered a minimum of once a year under the Associations training schedule. All new courses will be delivered by the end of year one of grant implementation. • Develop a certification training course to support the test-based certification program to be developed and administered by the Government of Albania. • Register and establish an official training institution under National Water Agency, based in Albania by the end of the second year of the grant. PURPOSE OF THE EVALUATION The purpose of this performance evaluation is to provide rigorous, evidence-based, and independent analysis of SWSCD’s performance. Specifically, the performance evaluation will serve to examine the processes, outcomes, and the effectiveness of the project activities implemented, to determine the achievements of the expected results according to the project description/outcomes, and to identify gaps in performance against targets. The evaluation will focus on both management and technical aspects of the program and its implementation. The evaluation will inform USAID of the success of the program and identifying lessons learned to share with the implementing partner. EVALUATION DESIGN AND METHODOLOGY The Contractor must conduct the performance evaluation and must utilize an evaluation design and methodology that addresses the overall goals of the evaluation and comprehensively answers the following evaluation questions: 1. What has Shukalb's performance been to date and what has the activity achieved? 2. What, if any best practices and lessons learned can be identified as –learned result of this activity? 3. What, if any opportunities were created for the sustainability of the certification program? What are those opportunities and what, if anything needs to be done to ensure that the opportunities are realized? USAID.GOV PERFORMANCE EVALUATION OF THE SWSCD ACTIVITY | 28 Where applicable, responses to the questions above and resulting data must be dis-aggregated by sex and gender. The Contractor will finalize the overall evaluation methodology in collaboration with USAID/Albania. However, the Mission expects that, at a minimum, the team will: • Use methods that generate the highest quality and most credible evidence that corresponds to the questions being asked; • Take into consideration time, budget, and other practical considerations; and • Include both qualitative and quantitative methods to yield valuable findings. Qualitative and quantitative data will be used to illustrate the processes and outcomes of the program. Data generated from the evaluation should be specific to each of the activities as well as USAID’s processes and systems for working with local partners. The Contractor will provide an accounting of performance against targets and, also an independent analysis on why targets were/not met or over/underachieved. Of particular interest will be the analysis of the reasons behind any gaps between expected and actual performance, including the identification of the likely source(s) of these gaps (design, project management, changes in operating environment, success factors, etc.). GENERAL CONTRACTOR REQUIREMENTS The Contractor must conduct a methodologically sound evaluation which must be fully compliant with USAID’s Evaluation Policy and Automated Directives System, and must utilize tested and scientific evaluation methodologies. TEAM COMPOSITION The Contractor must utilize a team with the experience and skills necessary to complete this task as specified in Section F.4. SCHEDULE AND LOGISTICS The schedule (with precise due dates for deliverables) will be defined in the work plan before the start of the field work. Prior to commencing the field work, all team members will review background program documents to gain a firm understanding of the situation in Albania and the SWSCD Program. The Contractor will be responsible for logistical arrangements for conducting the evaluation, including scheduling meetings, making copies of documents and drafts, making travel arrangements, etc. Field visit logistics will be organized and managed by the Contractor. This includes travel within country and in￾country transportation. USAID/Albania will provide the team with input and guidance in setting up a schedule of interviews and site visits, but the responsibility for the schedule for the evaluation resides with the Evaluation Team.