Le Fin an Af July Thi pre Photo cred earning nal Eva d Sani frica R y 2015 is publication epared indepe dit: ACWUA g Evalu aluatio tation Region was produce endently by In uation on of th Servic (IWSM ed at the requ nternational D and A he USA ces in MR) Ac uest of the Un Development nalysis AID M the M ctivity nited States A Group, LLC s Proje ME/TS Middle Agency for In (IDG). ect-II ( Impro East an ternational D (LEAP￾oving W nd No Development. -II) Water orth It was FINAL EVALUATION REPORT FINAL EVALUATION OF THE IMPROVING WATER AND SANITATION SERVICES IN THE MIDDLE EAST AND NORTH AFRICA REGION (IWSMR) ACTIVITY JULY 13, 2015 Learning Evaluation and Analysis Project-II (LEAP-II) Activity Order #7: Final Evaluation of the USAID ME/TS IWSMR Activity (AID-OAA-I-12-00042/AID-OAA-TO-14-00046) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACRONYMS ABC   Association of Boards of Certification ACWUA  Arab Countries Water Utilities Association BOD   Board of Directors BP   Business Plan CAQA     Jordan’s Center for Accreditation and Quality Assurance CIS     Certification Information System DWA Deutsche Vereinigung für Wasserwirtschaft, Abwasser und Abfall (German Association for Water Management, Wastewater and Waste) EQ   Evaluation Question   ESCWA   Economic and Social Commission for Western Asia EWRA   Egyptian Water Regulatory Authority FGD   Focus Group Discussion GIZ German Agency for International Cooperation (Deutsche Gesellschaft für Internationale Zusammenarbeit, formerly GTZ) GTZ   Deutsche Gesellschaft für Technische Zusammenarbeit HBRC     Housing and Building National Research Center ISSP   Institutional Support and Strengthening Program IWSMR   Improving Water and Sanitation Services in the MENA Region KII   Key Informant Interview LEAP II   Learn Evaluation and Analysis Project II MENA     Middle East and North Africa ME/TS   Middle East Bureau/Technical Services MOU   Memorandum of Understanding NUPAS   Non‐U.S. Organization Pre‐award Survey O&M   Operations and Maintenance OMEP   USAID Office of Middle East Programs OMT   Operations and Maintenance Training Project   PMP     Performance monitoring plan PwC   PriceWaterhouseCoopers RFP   Request for proposals ROCTP   Regional Operator Certification and Training Program SIDA   Swedish International Development Cooperation Agency) SOW   Scope of work TIA   Training Impact Assessment TOR   Terms of reference TOT     Training of Trainers USAID   United States Agency for International Development TABLE OF CONTENTS EXECUTIVE SUMMARY ......................................................................................................................... 1 Introduction ..............................................................................................................................................1 Arab Countries Water Utility Association (ACWUA)............................................................................. 1 IWSMR Activities...................................................................................................................................2 Evaluation Questions................................................................................................................................2 Methodology.............................................................................................................................................2 Key Findings, Conclusions and Recommendations................................................................................... 4 Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre‐award survey resulted in a more sustainable and self‐financing organization?.............4 Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? ...................................................................................................................... 5 Evaluation Question 3: To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power?......................................................................... 6 Evaluation Question 4: How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy‐in amongst the members?.............................. 7 Evaluation Question 5: What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs?....................................... 8 PART I. INTRODUCTION........................................................................................................................ 9 Evaluation Overview .................................................................................................................................9 Evaluation Report Structure ..................................................................................................................... 9 IWSMR Activity Description .................................................................................................................... 10 Context....................................................................................................................................................10 Evaluation Purpose and Audience .......................................................................................................... 11 Evaluation Questions.............................................................................................................................. 12 Methodology...........................................................................................................................................13 Document Review............................................................................................................................... 14 Key Informant Interviews.................................................................................................................... 14 Focus Group Discussions..................................................................................................................... 16 Web‐based Survey .............................................................................................................................. 16 Limitations...............................................................................................................................................17 PART II.  ARAB COUNTRIES WATER UTILITY ASSOCIATION (ACWUA) .................................................. 18 What are ACWUA’s mission and objectives?.......................................................................................... 18 When and why was ACWUA established?.............................................................................................. 18 What does ACWUA do? .......................................................................................................................... 20 Why has USAID supported ACWUA? ...................................................................................................... 21 How has USAID supported ACWUA? ...................................................................................................... 21 What has been accomplished through IWSMR to date?........................................................................ 23 Component 1: Build ACWUA Capacity................................................................................................ 24 Component 2: Regional Water and Wastewater Operator Certification Program ............................25 Component 3: Utility Twinning and Leveraging Best Practices.......................................................... 26 PART III. Evaluation Findings, Conclusions and Recommendations..................................................... 28 ACWUA’s Sustainability Following Pre‐award Survey Recommendations............................................. 28 Findings...............................................................................................................................................28 Conclusions.........................................................................................................................................32 Recommendations.............................................................................................................................. 33 ACWUA Sustainability and Growth Post IWSMR .................................................................................... 33 Findings...............................................................................................................................................34 Conclusions.........................................................................................................................................38 Recommendations.............................................................................................................................. 42 Impact of the Business Plan 2015‐2019 on ACWUA’s External Performance ........................................ 43 Findings...............................................................................................................................................43 Conclusions.........................................................................................................................................46 Recommendations.............................................................................................................................. 47 Impact of Training and Certification Programs on Utility Companies and Operators............................47 Findings...............................................................................................................................................48 Conclusions.........................................................................................................................................56 Recommendations.............................................................................................................................. 57 Utility Company Twinning Program........................................................................................................ 57 Findings...............................................................................................................................................58 Conclusions.........................................................................................................................................59 Recommendations.............................................................................................................................. 59 ANNEXES ........................................................................................................................................... 60 Annex 1. Statement of Work .................................................................................................................. 60 Annex 2. Evaluation Question Table – Summary of Findings, Conclusions and Recommendations......76 Annex 3. Online Survey for ROCTP Participants: Questions and Responses.......................................... 81 Annex 4. List of Key Informants Interviewed.......................................................................................... 84 Annex 5. Key Informant Interview Questionnaires................................................................................. 86 Annex 6. Focus Group Discussion Guide................................................................................................. 95 Annex 7. Bibliography of Documents and Sources Reviewed ................................................................ 96 Annex 8. Annex Pre‐award Survey (NUPAS) Recommendations and Actions........................................ 97 Annex 9. Disclosure of Any Conflicts of Interest................................................................................... 100 1 EXECUTIVE SUMMARY INTRODUCTION This report presents the evaluation findings, conclusions, and recommendations for the USAID Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR) activity under the Learning, Evaluation and Analysis Project‐II (LEAP II). IWSMR is a two‐year, $1.9 million water activity, managed by USAID’s Middle East Bureau/Technical Services (ME/TS), and implemented by Chemonics International, Inc. (Chemonics). The purpose of the evaluation is to generate insights into IWSMR’s success in building the Arab Countries Water Utility Association’s (ACWUA’s) “capacity to sustainably carry out its mandate to promote certification and accreditation, capacity development and information exchange among water and wastewater utilities and professionals during and beyond project completion.” The evaluation also assesses ACWUA’s “ability to sustain itself as a regional entity and manage funds given its future role in the region and its capacity at end of activity.” Arab Countries Water Utility Association (ACWUA) ACWUA is a non‐governmental organization that operates as a local, regional, and international platform offering opportunities to exchange knowledge, best practices in water and wastewater management, and experiences among stakeholders. As of June 2015, ACWUA had 330 members in 18 countries in the Middle East North Africa (MENA). Of these, the USAID Bureau for the Middle East supports 10 countries (see Table 1. for a breakdown). Table 1. ACWUA Members Supported by USAID Countries ACWUA Members USAID   Algeria  √  √  Bahrain √  Egypt  √  √  Iraq √  √  Jordan  √  √  Kuwait √  Lebanon  √  √  Libya √  √  Mauritania  √  Morocco √  √  Oman  √  Saudi Arabia √  Sudan  √  Syria √  Tunisia  √  √  2 Countries ACWUA Members USAID   UAE √  West Bank/Gaza (Palestine)  √  √  Yemen √  √  Source: ACWUA website: www.acwua.org; USAID website: http://www.usaid.gov/who‐we‐ are/organization/bureaus/bureau‐middle‐east   IWSMR Activities IWSMR was designed to work with ACWUA and members from USAID‐eligible countries to build capacity (both that of ACWUA’s and utility company members), and to develop training and certification programs.   EVALUATION QUESTIONS The evaluation addresses the following overarching questions: Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre‐award survey resulted in a more sustainable and self‐financing organization? Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? Evaluation Question 3: To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? Evaluation Question 4: How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy‐in amongst the members? Evaluation Question 5: What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs?   METHODOLOGY The evaluation team used the following methods to conduct the evaluation:   1. Document review 2. Key informant interviews (KIIs) (55 key informants, of which 48 were in person, six were by telephone, and one was by email) 3. Focus group discussions (FGDs) in Jordan and Egypt (two groups with six persons each) 4. Online survey of the Regional Operator Certification and Training Program (ROCTP) participants (13 responses out of 40 participants contacted) 3 Key informants were selected from a broad range of stakeholders (representing nine different groups) based on the desk review and recommendations from USAID, ACWUA, and Chemonics/IWSMR staff, in order to obtain diverse institutional perspectives. Given tight time constraints, and the fact that training‐ of‐trainer (TOT) participants were based in locations throughout the two countries, convenience sampling was used for selecting FGD participants in Jordan and Egypt, i.e. they were drawn from participants based in Amman and Cairo respectively, with the assistance of ACWUA and IWSMR.   Table 2. Types of Key Informants No. Key Informant Type Number Interviewed 1 Chemonics / IWSMR management and staff 4 2 ACWUA 5 3 Jordan certification agency 2 4 Utility companies – Jordan   19 5 Utility companies – Egypt 7 6 Utility companies – West Bank, Iraq 5 7 Regulatory / certification agency – Egypt 3 8 Development partners 3 9 Other (private sector, academia, other projects) 5 10 USAID staff 2 Total 55 The evaluation team spent seven days in Jordan and five days in Egypt, and traveled to seven cities for interviews with utility company management (Amman, Aqaba, Karak, Salt, and Tafila in Jordan; Alexandria and Cairo in Egypt). Jordan was selected because both the ACWUA Secretariat and Jordan’s Center for Accreditation and Quality Assurance (CAQA) are based in Amman; as are international donors (GIZ, SIDA, and JICA) who have the most knowledge of and experience working with ACWUA. Egypt was selected because it has the only other certification program in the region (Housing and Building National Research Center/Egyptian Water Regulatory Authority), a long history with ACWUA (the first ACWUA conference was held in Alexandria in 2006), the largest population in MENA, and a correspondingly large and important water utility sector.  Five key informants from two other countries – West Bank/Gaza1 and Iraq – were interviewed by telephone. The online survey generated responses from a further 13 individuals who took ROCTP courses in Lebanon, West Bank/Gaza, and Yemen.                                                              1 Referred to as Palestine by ACWUA. 4 KEY FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre-award survey resulted in a more sustainable and self-financing organization? Findings Under the terms of the IWSMR contract, ACWUA was assessed in July 2014 through a Non‐U.S. Organization Pre‐award Survey, or ‘Pre‐award Survey’ to evaluate ACWUA’s capability for receiving independent funding in the future. The key objectives of the pre‐award survey were to determine:   1. Whether the organization has sufficient financial and managerial capacity to manage IWSMR funds in accordance with U.S. Government and IWSMR requirements;   2. The most appropriate method of financing to use under the potential IWSMR award; and   3. The degree of support and oversight necessary to ensure proper accountability of funds provided to the organization.   The Pre‐award Survey assessed 13 financial risks and assigned scores (on a scale of one to four) to six areas, with accompanying narrative assessments. The overall score of 2.56 (and scores between 2.0 and 3.0 for five out of six criteria) puts ACWUA between ‘weak’ and ‘adequate’ with respect to control weaknesses and exposure to risk. ACWUA received the highest score (3.40) for ‘Legal Structure’ and the lowest score (2.15) for Financial Management and Internal Control System. Separate from the Pre‐award Survey, ACWUA has been implementing a Certification Information System (CIS). The system will enable ACWUA to track certification needs for member countries and give utilities access to training options.2 The CIS was originally planned to be fully operational and in use by ACWUA in November 2014. However, as of April 2015, the CIS was still being tested internally by IWSMR and ACWUA.   Conclusions ACWUA management viewed the Pre‐award Survey as an effective tool to develop an understanding of needed changes, including the tools to ensure ACWUA’s operational efficiency and effectiveness. ACWUA management stated that it believes that ACWUA’s Charter and Bylaws, together with the changes introduced in response to pre‐award survey recommendations, will serve to ensure the organization operates transparently. By adopting a more transparent accounting system, in line with the Pre‐award Survey recommendations which incorporates a breakdown of funding sources, ACWUA should be able to better manage member funds. However, it is not clear how and whether collecting member funds has been affected by IWSMR activities, as the key factors in collection rates appear to be   utility companies’ willingness and ability to pay, which are not management or accounting issues.                                                                2 ACWUA. (nd). Certificate Information System. Presentation. Amman, Jordan. 5 Recommendations  Conduct follow‐up Pre‐award Survey to assess in what areas and to what extent ACWUA has improved, prior to committing to additional funding.    Introduce an internal monitoring and evaluation system to measure performance, with a strong focus on outcomes. Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? Findings By codifying and documenting processes and separating functions (as recommended by the Pre‐award Survey), ACWUA is moving from reliance on personal/informal management to more institutionalized management. ACWUA appears to have a robust and well‐functioning governance framework including a General Assembly, Board of Directors, Secretary General, Secretariat, technical committees, and technical working groups. The area of concern most often mentioned by key informants was ACWUA’s internal capacity to sustain its activities. Feedback from several key informants reflects concerns that ACWUA is taking on too many activities for which it does not always have adequate human resources to manage. The issue of staffing constraints is well recognized, both within and outside ACWUA, and the Strategic Business Plan includes measures to boost capacity through an organizational restructuring.   Several key informants mentioned that ACWUA could be more responsive to utility member needs, for example, by basing training for their staff on a needs assessment.   Conclusions ACWUA’s leadership demonstrates an understanding of the importance of accountability, transparency and sustainability in its management and financial practices.  IWSMR has, despite leadership challenges, equipped ACWUA with financial and managerial tools to strengthen a foundation for ongoing reforms and improvements. Based on past performance and the fact that ACWUA was not dependent on IWSMR support for the period from its start in 2009 through the end of 2013 there is reason to believe the organization will continue to grow and show prudent financial performance after the IWSMR activity ends. ACWUA’s ability to continue acting as a service provider will most likely depend on the following factors: i) the quality of the services it provides; ii) the resources (staff, organizational capacity, etc.) it has to organize and provide services; iii) its reputation; and iv) demand for services. Recommendations  USAID should continue to provide support to ACWUA and associated institutions, through IWSMR‐type capacity building or similar activities, and possibly including direct budget support. Follow‐on assistance should focus on the following areas: developing management capacity, 6 building staff capacity, improving communications with members reducing the cost of training, expanding individual training programs, and benchmarking performance indicators appropriately.   Evaluation Question 3: To what extent has the 2015-2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? Findings ACWUA’s Strategic Business Plan (2015‐2019) outlines the organization’s priorities and its roadmap to growth and sustainability. It was developed in a participatory manner over several months, with input from ACWUA’s Board of Directors. However, the Business Plan was only finalized and approved in January 2015, and as a result, its impact on ACWUA’s regional status or influence could not be assessed. Nonetheless, the presentation of the Business Plan at Arab Water Week (2015) appears to have benefited ACWUA in terms of its credibility as an organization. In general, ACWUA’s revenue by source fluctuates significantly from year to year, reflecting issues with members not paying their fees, and the one‐off nature of some sources such as grants, and support from external parties and Arab Governments. ACWUA sees “continuing support from international organizations and funding agencies to develop the water and sanitation sector in the Arab region,”3 as a prerequisite for successful implementation of its Business Plan.   Conclusions Discussions with ACWUA management suggest that the financial projections included in the Business Plan do not play a strong role in guiding decisions. A number of factors may contribute to this including the regional turmoil, which has lowered utility members’ willingness and/or ability to pay their fees, and ACWUA’s general focus on raising revenues through marketing its training services and soliciting donor funding. The apparently weaker focus on internal management affairs may also be an explanatory factor. All of the above reinforces the value of ACWUA having a solid Strategic Business Plan. Recommendations  ACWUA should periodically update the financial tables in the Strategic Business Plan, and develop an Action Plan to accommodate potential fluctuations in revenue.    ACWUA should develop a plan to reduce non‐payment of membership fees among utility companies, which have dropped significantly since the start of the Arab Spring.                                                                3 ACWUA Strategic Business Plan 2015‐ 2019. 7 Evaluation Question 4: How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy-in amongst the members? Findings A core ACWUA activity is the provision and organization of training and testing for operators and water and wastewater sector personnel in member countries. ACWUA is accredited by CAQA to deliver a water operators certification program in the Arab region.4 CAQA administers the tests and issues certificates to participants who pass. Utilities select and nominate participants to receive training, based on their internal needs (at least in theory), while CAQA approves training nominees based on their qualifications. The total number of people trained under the IWSMR activity for ROCTP, TOT and CIS is 217, of which 46 were women (21.2 percent).   ACWUA organized three ROCTP courses with support from IWSMR in Lebanon, West Bank and Yemen. The evaluation team conducted an online survey of ROCTP participants generating 13 responses. When asked about their level of agreement with the statement, “Taking the ROCTP training was useful for my work” the average rating was 4.3 out of five (where five was equivalent to ‘completely satisfied’ and one was equivalent to ‘completely unsatisfied). Based on the rating questions used for the FGDs and KIIs, satisfaction levels among TOT participants were high, with average ratings for sufficient time, appropriate technical levels, and usefulness all above 4.5 and no individual rating of less than four. Buy‐in from utility companies appears to be good, based on interviews with utility company managers in Jordan and Egypt who spoke positively about the expected impacts of improved training capacity.   Conclusions The positive feedback from both trainees and utility companies on the training organized by ACWUA has strengthened the organization’s reputation among members who have benefited from it. The certification process is seen as worthwhile and valuable to both utilities and individuals as a mark of achievement and as an indicator of capability.   While the Training Roll‐out Plan could not be reviewed at the time of the evaluation, ACWUA will need to be careful to avoid letting training quality lapse as it increases the scope, number of programs, and countries targeted. The impact of training on utility performance has yet to be assessed. However, the fact that participants and government rate the program highly increases the chances that its impact will be positive. The broader impact of the TOT will depend in part on how many utility persons are trained and what the multiplier effect is, i.e., the secondary impact of how well those trained can train others.                                                                4 Under the Ministry of Labor, CAQA is Jordan’s government approved certification body, established in 2012. 8 Recommendations  ACWUA should build a larger and stronger cadre of trainers to reduce the need for training participants to travel which will reduce the cost of training per participant.  ACWUA should revise the TOT program to make it less generic and incorporate more material or examples relevant to the water and wastewater sector to the extent feasible. Evaluation Question 5: What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs? Findings At the time of the evaluation, the Twinning Program had been prepared, and possible twinning pairs identified, but the program had not yet come into effect. IWSMR provided support by reviewing literature on knowledge exchange and twinning programs, and then collecting information from member utilities on their needs to identify the focus for the twinning arrangements. As of the April 2015, IWSMR had produced three documents: a Draft Twinning Manual; a Twinning MOU template; and a Twinning Plan and Schedule.   Conclusions No conclusions regarding the effects of the Twinning Program could be drawn at the time of the evaluation. Recommendations No recommendations are made at time of the evaluation    9 PART I. INTRODUCTION EVALUATION OVERVIEW USAID requested International Development Group LLC (IDG) to conduct a final evaluation of the Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR) Activity under the Learning, Evaluation and Analysis Project‐II (LEAP‐II).5 This report presents the evaluation findings, conclusions, and recommendations. The statement of work (SOW) for this evaluation can be found in Annex 1. EVALUATION REPORT STRUCTURE Part I provides the context of both the IWSMR activity and the evaluation, including the evaluation purpose, audience and methodology. Since the Arab Countries Water Utilities Association (ACWUA) is both the primary recipient of IWSMR support and the main vehicle for implementing IWSMR activities, it is the main focus of the evaluation.   Part II describes ACWUA, its mandate, role and activities. It reviews the rationale for USAID support through IWSMR, and the nature of that support. This is followed by a brief overview of IWSMR’s achievements at the time of the evaluation (through the first quarter, January‐March 2015).   Part III covers evaluation findings, conclusions and recommendations. Five separate sub‐sections correspond to the five evaluation questions and their sub‐questions. For ease of reference, the sub‐ sections headings are streamlined versions of the evaluation questions:   1. ACWUA’s sustainability following pre‐award survey recommendations   2. ACWUA sustainability and growth post IWSMR   3. Impact of the Business Plan 2015‐2019 on ACWUA’s external performance 4. Impact of training and certification programs on utility companies and operators 5. Utility companies twinning program   On the first page of every sub‐section, all of the evaluation questions/sub‐questions are included in a box.    It should be noted that there is not a one‐to‐one alignment between the evaluation questions and the findings sections. This is because some evaluation questions could not be answered, given that certain activities (e.g., Business Plan) had been completed too recently for their effects to be evaluable, or had yet to be implemented (e.g., the CIS, the Twinning Program). On the other hand, the evaluators assessed some issues that were not explicitly targeted by the SOW evaluation questions, but which fall under the                                                              5 The evaluation team comprised Nils Junge (Team Leader), Azad Mohammadi (Utility Management Specialist) and Ehab Quran (Utility Specialist). The consulting firm, Al Jidara, provided logistics support during the mission in Jordan and Egypt. The evaluation was coordinated by Elly Preotle, LEAP‐II Chief of Party (IDG).  Amena Chenzaie (USAID Monitoring and Evaluation Specialist) joined the evaluation team for the first week of the mission. 10 general category of ‘Lessons Learned,’ and may be of interest to USAID (e.g., the overall impact of the IWSMR program on ACWUA, the Training of Trainers program, etc.).  Rather than covering them in a separate, stand‐alone ‘Lessons Learned’ sub‐section, they are integrated into the five sections listed above. Instead of simply leaving a blank space under findings for the eight (out of 24) evaluation questions and sub‐questions that are, strictly speaking, unanswerable, these sections discuss the issues in a broader sense as there is still information and analysis worth conveying.   IWSMR ACTIVITY DESCRIPTION IWSMR is a water activity, with an award budget of $1,991,240, managed by USAID’s Middle East Bureau/Technical Services (ME/TS), and implemented by Chemonics International, Inc. (Chemonics). The IWSMR activity began in October 2013 and was to be implemented over a two year period, ending September 30, 2015.   IWSMR was designed to work with the ACWUA and members from USAID‐eligible countries6 to build capacity (both that of ACWUA’s and utility company members), and to develop training and certification programs, through the Regional Operator Certification and Training Program (ROCTP). CONTEXT The Middle East and North Africa (MENA) is one of the most water scarce regions on earth. While average water availability globally is approximately 7,000 cubic meters/person/year, in the MENA region the average is just 1,200 cubic meters/person/year7 (and far below that in countries such as Jordan). The water sector in the Arab region suffers from chronic problems, such as water scarcity, weak water and environmental policies, high investment needs, lack of management and technical capacity, increasing demand due to growing populations, and conflicts. Population growth and increasing demand for food, housing, and jobs place extreme pressure on water resources. In addition, limited rainfall and declining groundwater make the region susceptible to the effects of climate change.   At the same time, policies which keep tariffs below cost‐recovery levels have starved utility companies of funds for operations and maintenance, rehabilitation and expansion. The lack of investment has led to deteriorating service quality and supply. As a result, many water and wastewater utilities are in need of major improvements to operational efficiency, reduction in non‐revenue water, and large investments in infrastructure rehabilitation and extension.    Utilities in the region also face significant capacity constraints with regard to operations and maintenance, management, and technical capacity. There is a significant need to increase skills at management, operator and technical levels. While international expertise is valuable, there are clear benefits from knowledge sharing, in particular among utility companies encountering similar problems and with a shared culture. This is where an organization such as ACWUA can play a significant role.                                                                6 Morocco, Tunisia, Algeria, Libya, Egypt, Jordan, West Bank/Gaza, Lebanon, Yemen and Iraq. 7 World Bank. 2010 Water Brief. Available at: http://go.worldbank.org/0Z2C3PFS70. 11 ACWUA is a non‐governmental organization (NGO) officially launched in 2009. It was conceived and supported since 2006 by the Economic and Social Commission for Western Asia (ESCWA) and the German Agency for International Cooperation (GTZ) as an initiative among key water sector representatives in the Arab Region. ACWUA describes itself as a local, regional, and international platform which offers opportunities to exchange knowledge, best practices in water and wastewater management, and experiences with different stakeholders.8 As of June 2015, it had 330 members in 18 countries in MENA. Of these countries, the USAID Bureau for the Middle East supports 10 countries (see Table 3. below for a breakdown). Table 3. ACWUA Members Supported by USAID   Countries ACWUA Members USAID Algeria  √  √  Bahrain √  Egypt  √  √  Iraq √  √  Jordan  √  √  Kuwait √  Lebanon  √  √  Libya √  √  Mauritania  √  Morocco √  √  Oman  √  Saudi Arabia √  Sudan  √  Syria √  Tunisia  √  √  UAE √  West Bank/Gaza (Palestine)  √  √  Yemen √  √  Source: ACWUA website: www.acwua.org; USAID website: http://www.usaid.gov/who‐we‐ are/organization/bureaus/bureau‐middle‐east   EVALUATION PURPOSE AND AUDIENCE The purpose of this evaluation is to generate insights into IWSMR’s success in building ACWUA’s “capacity to sustainably carry out its mandate to promote certification and accreditation, capacity development and information exchange among water and wastewater utilities and professionals during                                                              8 ACWUA website: http://www.acwua.org/node/1   12 and beyond project completion,” as per the evaluation statement of work (SOW). The evaluation was also specifically tasked with assessing ACWUA’s “ability to sustain itself as a regional entity and manage funds given its future role in the region and its capacity at end of activity.” The primary audience for this evaluation is USAID’s ME/TS to inform the potential design of any follow‐ on activities which could help improve management of water resources in the implementing countries. The secondary audience includes USAID’s partners and stakeholders.   EVALUATION QUESTIONS The specific evaluation questions which the evaluation was directed to address are: Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre‐award survey resulted in a more sustainable and self‐financing organization? a. How has ACWUA addressed deficiencies identified?  What have been the constraints? b. How has ACWUA’s organizational and financial management changed pre and post      IWSMR activity implementation? c. How has ACWUA developed its capacity and mandate to collect and manage member funds? d. How is the Certification Training Information System (CIS) being utilized? Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? a. What mechanisms are in place to ensure ACWUA’s financial sustainability? b. What mechanisms are in place to ensure ACWUA’s management sustainability? c. To what extent will ACWUA be able to maintain its current service provision? The same quality of service provision? d. What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? e. Is ACWUA’s management willing to accommodate stakeholder needs?   Evaluation Question 3: To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? a. How much success have the business planning exercises had in improving the reliability of ACWUA ‘s year to year revenue streams?   b. Were lessons learned from implementing the Bridging Memo incorporated into the 2015‐ 2019 Business Plan? c. Has ACWUA's operational capacity changed as a result of the Business Plan? 13 Evaluation Question 4: How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy‐in amongst the members? a. How have lessons learned from the pilot operator certification courses been incorporated into the ACWUA Training Roll‐Out Plan? Do they address needs in the respondent countries? b. How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long‐term structure of the certification program? c. To what extent do member country utilities value the certification of a Jordan organization (CAQA)?  Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? Evaluation Question 5: What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs?   a. To what extent has information and peer support between water and wastewater utilities been established through the regional twinning program? b. What lessons have been learned from the two pilot virtual twinning/exchange activities run by IWSMR? c. What has been the nature of exchange and collaboration amongst the twinning partnerships? Has it helped in building staff capacity? d. How effective are the existing indicators in measuring effective management of twinning partnerships? Based on answers to the above evaluation questions, the evaluation team was also to generate:    Lessons Learned: What are lessons learned, if any from implementation to date that ME/TS should take into consideration for design of future activities in this area?    Recommendations: What adjustments, if any, should be made to improve future activities' ability to help improve management of water resources in the implementing countries? As noted above under the Evaluation Report Structure: i) the evaluation findings are organized according to the evaluation questions; ii) only evaluable questions are answered in the report; ii) recommendations follow each section; and iii) lessons learned are incorporated into the conclusions and recommendations.   METHODOLOGY The methods used for the evaluation are primarily qualitative, enhanced by limited quantitative research. They include: 1. Document review 2. In‐depth interviews (55 key informants, of which 48 were in person, six were by telephone, and one was by email) 3. Focus group discussions (FGDs) (two groups with six participants in each group) 4. Online survey of ROCTP training participants (13 responses) 14 Document Review Documents were a key source of information, used to triangulate key informant sources. The document types reviewed included publicly available ACWUA material, printed and online, training material, internal ACWUA documents, IWSMR activity material, and other documents. A full list of documents reviewed is included in Annex 7.   Key Informant Interviews Following a desk review, which was used to inform the development of the evaluation methodology and interview questionnaires, the evaluation team spent two weeks in the field (seven days in Jordan and five days in Egypt), during which time it traveled to seven cities for interviews with utility company management (Amman, Aqaba, Karak, Salt, and Tafila in Jordan; Alexandria and Cairo in Egypt). Telephone interviews were conducted with five key informants from utility companies in the West Bank and Iraq. KIIs were a significant source of information underpinning the evaluation. The format was semi‐ structured in‐depth interviews, based on a list of question topics prepared in advance and tailored to the key informant type (see Annex 3).   Key informants were selected from a broad range of stakeholders (representing nine different groups), in order to obtain diverse institutional perspectives. To obtain a broad and varied perspective and validate findings from the key informant interviews, the evaluation team also sought the perspectives of the following groups: USAID activity contractors (IWSMR and Chemonics); ACWUA management; ACWUA member utility companies and their affiliates; training participants (TOT, ROCTP); other ACWUA members; and development partners. Key informants were selected from a broad range of stakeholders (representing nine different groups) based on the desk review and recommendations from USAID, ACWUA, and Chemonics/IWSMR staff, in order to obtain diverse institutional perspectives. A list of individuals and institutions contacted is available in Annex 4.   Table 4. Types of Key Informants No. Key Informant Type Number Interview 1 Chemonics / IWSMR management and staff 4 2 ACWUA 5 3 Jordan certification agency 2 4 Utility companies – Jordan   19 5 Utility companies – Egypt 7 6 Utility companies – West Bank, Iraq 5 7 Regulatory / certification agency – Egypt 3 8 Development partners 3 9 Other (private sector, academia, other projects) 5 10 USAID staff 2 Total 55 15 The evaluation team drafted questionnaires for interviews for the main stakeholder types: the contractor (IWSMR/Chemonics), ACWUA staff, and utility members.  Questions for other key informant types drew from the various questionnaires (see Annex 5). A short FGD guide was developed for trainers (see Annex 6). As appropriate and applicable, the following evaluation criteria were used to frame the interview questions (which informed answers to the SOW evaluation questions): ● Effectiveness of activities in achieving their objectives ● Relevance of activities to objectives ● Sustainability of outcomes linked to activities inputs and outputs ● Factors which impeded or stimulated progress toward the objectives ● Degree of attribution of observed outcomes to the activities   In order to introduce a quantitative element into the interview process, key informants were asked, when applicable, to rate various aspects the evaluation was assessing (such as training effectiveness, ACWUA performance, etc.) on a Likert‐type scale of 1 to 5. A more detailed description of the methodology is provided in Annex 1. In‐depth interviews lasted between 30 and 60 minutes on average. In a few cases interviews were brief because the key informant had minimal information to share or issues discussed were narrow in scope. In other cases, the interviews were far more in‐depth, and lasted several hours. Several interviews became ‘group interviews’ with more than one interlocutor. Confidence in the reliability and validity of the KII findings stems from two factors. First, the 55 key informants interviewed represented a wide range of backgrounds and institutions. The evaluators interviewed ACWUA staff, IWSMR/Chemonics staff, utility management representatives, utility company staff (training participants), external sector experts, and international donors. This reduced the probability that the views and interests would reflect a single viewpoint, while controlling for potential biases inherent to a particular stakeholder group. Second, responses to interview questions from the different key informants were relatively homogenous, largely confirming each other. There were few outliers, i.e., responses which contradicted one another, further strengthening the validity of the findings.   Interview summaries were produced to facilitate analysis. Issues of confidentiality were addressed by informing interviewees that their names would not be associated with their responses, unless they wished it, and that any interview transcripts or reports shared with USAID would be stripped of identifying information. At the same time, the evaluators relied on and were grateful for ACWUA’s assistance in arranging interviews with key informants and sometimes accompanying the evaluation team to introduce them to the key informants, but not participating in the interview itself.   In order to achieve some comparability between KII responses, as well as introduce a quantitative dimension, a select number of questions were accompanied by a rating or ranking question, using a 5‐ 16 point Likert‐type scale, a psychometric scale to gauge opinions. The respondent was asked to state a number, on a scale of 1‐5, and then explain the reasoning behind his/her choice. This approach has been used recently on another USAID evaluation conducted in Jordan, and found to be useful both in stimulating responses and for analysis.  While the sample of key informants interviewed was too small to allow for statistical analysis, the aggregate numbers are usually revealing and can be considered indicative. The qualitative data collected from key informant interviews was analyzed in depth for patterns that address the relevant questions and speak to effectiveness, efficiencies, relevance, potential sustainability, impeding/facilitating factors, and cause and effect.   Focus Group Discussions FGDs are rapid, cost‐effective data collection methods that use direct probing techniques to gather information from several individuals. FGDs represent a group situation in which participants share their perspectives on a limited number of topics under the guidance of a moderator to generate relevant information/data. Given tight time constraints, and the fact that TOT participants were located all over the country, convenience sampling was used for selecting FGD participants, i.e., they were drawn from participants based in Amman, with the assistance of ACWUA and IWSMR.   FGDs were conducted with TOT training participants and ACWUA staff. A total of 12 training participants took part in two FGDs, one held in Jordan with six Jordanian nationals, and one held in Egypt with six Egyptian nationals.   The evaluators facilitated FGDs using questionnaires and following standard USAID FGD protocols. There was a moderator and a note taker. Findings were summarized, and participant anonymity was protected. Focus group topics/questions (see Annex 3) are the same as those used to interview individual trainee key informants during visits to utilities. Web-based Survey An online survey (using SurveyMonkey software) was implemented targeting ROCTP participants. This allowed the evaluators to reach a greater number of training beneficiaries than could be reached through key informant interviews or focus group discussions.   The dual‐language English‐Arabic survey was short, consisting of 15 questions, with questions mostly closed‐ended. Potential respondents were identified using contact information from the training sessions provided by ACWUA.   The survey was distributed to the 40 persons who took the ROCTP training, and 13 individual responses were obtained. This number is too small to conduct any but the most rudimentary data analysis (i.e., beyond simple average on the quantitative questions) and is not statistically significant. However, the survey responses largely supported information obtained through KIIs and FGDs. 17 LIMITATIONS Evaluation limitations related primarily to timing and data. Because the research was conducted well before the end of a relatively short (two year) IWSMR activity, it was not possible to fully assess its impacts. At the time of writing, IWSMR’s work is ongoing, and several actions have yet to be fully implemented (e.g., Business Plan 2015‐2019, Twinning Program, CIS, etc.), and thus could only be partially assessed.   These activities are discussed in terms of early indicators relating to quality and stakeholder perceptions. Either insufficient time had passed for activity inputs to feed into (potential) impacts, or activities had not been implemented (as in the case of the Twinning Program). This limitation was addressed, to the extent possible, by assessing general factors known to contribute to sustainability as they relate to ACWUA, such as institutionalization of changes, demand for capacity, change in capacity, staff turnover rates, etc. The evaluation team focused on the direction in which early indicators pointed, and used expert judgment to estimate the probability that a given activity would be successful, and under what conditions.   The geographic scope of the fieldwork was a limiting factor. ACWUA engages closely with utility companies from at least 10 countries. However, given the budget and time constraints, fieldwork was only considered feasible in the two countries: Jordan (because of ACWUA’s location and the origin of the IWSMR activities) and Egypt (due to its certification program and size).   High turnover at IWSMR reduced the institutional memory on which the evaluators could draw.  IWSMR had gone through three Chiefs of Party (COPs) during its first 18 months, and over a period of several months just prior to the evaluation, two key staff had departed: the COP and Deputy COP (DCOP)/Training Specialist. However, the Chemonics manager, who had been responsible for IWSMR from the beginning was available and able to provide a wealth of information.   This   mitigated the turnover issue to some extent. As the activities are ongoing, extra caution in assessing effectiveness is called for. IWSMR is driven primarily by capacity building at the operational level with minimum interventions at the senior and executive levels to achieve broader objectives vis‐à‐vis sustainability. We interpret sustainability in this case to mean operational efficiency that can be transformed into cost effectiveness and cost reductions. These types of objectives are achievable only in the long term.   However, in part because of the timing of the evaluation (coming several months before the end of IWSMR activities), and in part because of delays in implementing some IWSMR components, the evaluation faced some data limitations. Impacts had either not been tracked or assessed yet, and in many cases it was too early to assess them. To address this, a probabilistic approach was used. Through review of a range of information sources, combined with assessment of past and current performance, the evaluation team used expert judgment to estimate probable effectiveness, sustainability, etc., of components whose impact cannot yet be measured.      18 PART II. ARAB COUNTRIES WATER UTILITY ASSOCIATION (ACWUA) WHAT ARE ACWUA’S MISSION AND OBJECTIVES? ACWUA was established as a forum to engage professionals in utilities from Arab countries in a dialogue that extends across borders within the region.   To achieve its goals, ACWUA seeks to bring together water experts and utilities in the Arab region to promote the common technical, managerial, legal, scientific and economic interests of water utilities in the Arab region.9    According to its charter and bylaws, ACWUA’s vision is “to be a self‐sustainable, strong, regional association of Arab water and wastewater utilities.” It views itself as a center of excellence in the Arab water sector (ACWUA, 2015) with the aim of supporting utility companies to provide “best practice service delivery to their customers.”10 WHEN AND WHY WAS ACWUA ESTABLISHED? ACWUA is a financially and administratively independent legal entity founded in 2009 as the result of a joint initiative of the United Nations Economic and Social Commission for Western Asia (ESCWA) and the German Agency for Technical Cooperation (“Deutsche Gesellschaft für Technische Zusammenarbeit”, GTZ, now GIZ) going back to 2006. It is registered in Jordan as a non‐governmental and non‐profit association. The stated aim of the organization was to improve water supply and sanitation services in the region (ESCWA, 2006).    GTZ contracted the German Association for Water Management, Wastewater and Waste (DWA) to support the foundation of ACWUA. The water utility association for the Arab region is thus broadly based on the DWA model.   A General Assembly meeting was held to elect the first Board of Directors on November 19, 2008 in Alexandria, Egypt and a decision taken to locate the permanent ACWUA Secretariat offices in Amman, Jordan.11 On July 30, 2009, ACWUA was officially launched in Amman at a ceremony attended by over 80 high ranking officials and donor representatives from across the Arab world and other countries, under the patronage of His Excellency, the Minister of Water and Irrigation of the Hashemite Kingdom of Jordan.                                                              9 ACWUA Strategic Business Plan 2015 – 2019. 10 ACWUA Charter and Bylaws. 11 ACWUA website: http://www.acwua.org/node/1 19 Figure 1. Timeline of ACWUA’s Establishment Source: ACWUA As of June 2015, ACWUA had a total of 330 members in various membership categories from 18 countries: Utilities Membership (106); Associated (Private Sector companies) (14); NGOs and Academic institutions (8); and Individuals (200). The membership number is 43.3 percent of ACWUA’s membership target (for the 2010–2014 planning period) of 762 members.   Figure 2. A Source: AC WHAT ACWUA a providing ● ● ● ● ● ● ● ● ● ● ● ACWUA Mem CWUA websit T DOES A aims to offer a wide range Arab Water Annual Best Technical W Training an E‐learning c Best Practic ACWUA new ACWUA we Studies and Studies and Workshops mber Countrie te. Available a ACWUA its members e of activities r Week Intern t Practice Con Working Grou d Certificatio courses ce manuals an wsletter ebsite and ACW d field visits to d Researches and Seminar es at: http://ww A DO? s, cooperation and services national Confe nferences ps n nd Operation WUA Wiki (e‐ o water and w rs ww.acwua.org n partners an (ACWUA, 20 erence and T al Guides ‐platform for wastewater tr g/membershi nd donors a 15) such as: Trade Fair r knowledge m reatment pla ip/who‐are‐o strong netwo management nts ur‐members orking platfor ) 20 rm by 21 Through its Working Groups, ACWUA is focused on the management of utilities, water resources management, management of water and health, utilities reform, benchmarking, and capacity building and training. Sixteen cooperative partners (including international donors, universities, and NGOs)12 support ACWUA.   WHY HAS USAID SUPPORTED ACWUA? USAID states that its Water and Development Strategy “steers water programs toward key themes consistent with two of the most important ways we rely on water: water for health and water for food."13 It focuses on four areas to pursue this goal: i) improving access to water supply and sanitation, ii) promoting better hygiene, iii) improving water resources management, and iv) improving water productivity in agriculture.14 Support to ACWUA thus addresses USAID’s first area indirectly and the third area directly.   Taking the first step in sound water decision‐making and management requires an accurate understanding of the location and availability of water resources. Obtaining verifiable water data remains a challenge in the region due to the costs of data collection and analysis, and the lack of uniform, comprehensive data management and decision‐support systems (USAID, 2015). Furthermore, improved water management can also be critical at the geopolitical level, given the potential for disputes over water to exacerbate political tensions.   At present, there are few regional institutions in MENA with a water and wastewater mandate.15 ACWUA is the only organization in the region considered to have a clear operational focus on building utility capacities and actively fostering information and knowledge exchange.   HOW HAS USAID SUPPORTED ACWUA? The IWSMR activity is designed to improve ACWUA’s capacity in order to ensure its financial sustainability.  Different IWSMR aspects are built around the results framework shown below.16 The USAID Office of Middle East Programs (OMEP)17 funded the two‐year IWSMR activity, building on previous activities promoting development of water and wastewater operator training and certification                                                              12 http://entire.collectiveleadership.com/wp‐content/uploads/2014/02/Presentation_ACWUA_Eng.‐Kashman.pdf 13 Water and Sanitation. USAID website. Available at: http://www.usaid.gov/what‐we‐do/water‐and‐sanitation   14 USAID. Water and Development Strategy 2013‐2018, p. 5 15 Other regional water organizations identified include: ) Inter‐Islamic Network for Water Resource Development and Management (INWRDAM) http://inwrdam.org.jo; 2)  Middle East and North Africa Network of Water Centers of Excellence (MENA NWC) http://www.menanwc.org/node/9#.VVtk‐vlViko; 3)  Arab Water Council http://www.arabwatercouncil.org; 4) CEDARE – Water Resources Management Council http://water.cedare.int/cedare.int/Main.aspx?code=481; and 5) Arab Water Academy (defunct) 16 IWSMR Year 1 Work Plan, November 2013. 17 Note that due to the evacuation in Cairo, OMEP was transferred out of Cairo and merged with Middle East Regional based in Washington, DC. 22 programs in Jordan and Egypt. IWSMR is implemented by Chemonics with a budget of almost $2 million. It provides technical expertise and assistance for the following three activities:18   1. Build the capacity of the ACWUA to enable it to achieve its mandate to promote certification and accreditation, capacity development, and information exchange among water and wastewater utilities and professionals during and beyond activity completion.   2. Evaluate the feasibility of, and develop and implement a mechanism to, most effectively scale existing water and wastewater operator certification programs in Jordan and/or Egypt to regional application.   3. Leverage regional USAID funded improvements in water and sanitation utility management and operation by exploring training opportunities (such as utility twinning) to encourage information exchange and disseminate best practices in the water and sanitation industry. IWSMR started up in October 2013, with procurement and outreach activities to make its “existence and mandate known to organizations, authorities, associations, NGO’s, water sector decision makers, international and regional agencies as well as the public in the region.”19 Implementation of activities to support the above objectives was divided over two phases: Assessment and Implementation. The assessment phase was initially planned to run from January to February 2014; however, it lasted until May 201420 due issues relating to the Work Plan which were resolved between the three parties (ACWUA, IWSMR and USAID). The implementation phase began in January 2014 (with preparatory activities overlapping with the Assessment Phase) and will run through September 2015.   ● Phase 1 (Assessment): a) review and update ACWUA’s strategic business plan; b) evaluate the feasibility of and identify approaches to scaling up regional certification programs; c) explore opportunities to improve the service delivery of utilities (e.g., utility twinning program).   ● Phase 2 (Implementation): working with ACWUA and member countries and utilities to carry out recommended actions.   Figure 3 below describes how IWSMR’s intermediate results and program results feed into the program objective.                                                                   18 Task Order: AID‐263‐TO‐13‐00002 IQC EPP‐I‐00‐04‐00020; Integrated Water and Coastal Resources Management (Water II IQC). 19 IWSMR 1st Quarterly Report October – December 2013, January 2014. 20 IWSMR 6th Quarterly Report January – March 2015, April 2015. 23 Figure 3. IWSMR Program Results Source: IWSMR IWSMR has focused on building ACWUA’s capacity and knowledge base to ensure it is financially sustainable and responsive to customer needs. By pooling information and resources to create more rigorous training programs, IWSMR has sought to help the region strengthen its delivery of water and wastewater services.   IWSMR’s support to ACWUA can be divided into two broad, interrelated areas: i) ACWUA’s internal capacity building, and ii) its externally‐oriented activities and services (training and certification, and twinning program). These two areas are symbiotic and have implications for sustainability. Good internal financial and organizational management is critical to the organization’s continued viability, while successful performance of external activities will strengthen its ability to promote positive change in the sector. Many of these latter activities generate revenue, which in turn bolster its financial health and sustainability.    ACWUA’s mandate and focus are very much in line with USAID’s goals for the region and the sector. The water sector in MENA faces pressure from population growth, urbanization, and faltering economies, and instability. Under these circumstances, the need and the demand for building regional capacity in efficiently and sustainably operating existing and future water utility assets is more pronounced than ever. By drawing on regional expertise and building on lessons learned, ACWUA has strong potential to emerge as a catalyst in promoting investments and more effective use of resources in the region. WHAT HAS BEEN ACCOMPLISHED THROUGH IWSMR TO DATE? Before assessing ACWUA and the IWSMR activity in terms of the key areas of interest (effectiveness, sustainability, and impact) the evaluation first summarizes the results of the IWSMR activity (at the time of writing, i.e., after 18 months of implementation) based on its annual and quarterly reporting. These results are reported to provide context for the evaluation findings, conclusions and recommendations which follow in Part III.   The information in this section is based on the most recent IWSMR Quarterly Report made available to the consultants (Number 6, covering the period January  ‐  March 2015), informed by key informant interviews.   24 The IWSMR Action plan to support ACWUA and the certification process is divided into three components: i) Build ACWUA capacity, ii) Regional Water And Wastewater Operator Certification Program, and iii) Utility Twinning and Leveraging Best Practices. Each component has a ‘program result’ (PR) and ‘intermediate results’ (IRs) assigned to it. Progress is reported quarterly by IWSMR to USAID.   After initial delays,21 the IWSMR activity appears to be on track to meet its goals. (Some modifications were made to the Twinning Program to reduce its cost by making it virtual.) Most activities have been completed, and the remaining activities are on track to be completed by the end of the activity.  No major problems or significant scaling back of activities was identified.   The main activities yet to be completed are the Twinning Program and CIS implementation. As of March 31, 2015, IWSMR had disbursed $1,376,671 out of $ 1,991,240, with $614,569 remaining to be disbursed. The quality of support provided by IWSMR, recommendations and capacity building was rated positively overall by ACWUA management and staff. A detailed breakdown of the status of IWSMR activities by component follows. Component 1: Build ACWUA Capacity Task 1.1 Update ACWUA strategic business plan.  A Strategic Business Plan 2015‐2019 was developed in a participatory manner and approved in January 2015. Task 1.2 Develop a Training and Certification Information System (CIS)  The CIS was launched and became operational internally in October 2014. Training on the CIS was held in April 2015. During Arab Water Week (January 2015) the CIS was demonstrated to ACWUA members, and members were encouraged to begin populating the system with the required training data. At the time of the evaluation it was still pending regional launched. Task 1.3 Establish Certification Unit within ACWUA Organizational Structure  ACWUA requested establishment of a Certification Unit within its organizational structure, which was deemed critical to effective and efficient delivery of ROCTP services and to program sustainability. The certification unit was planned to operate by October 2014. The unit was in operation by March 2015. The Quarterly report notes that Jordan’s Center for Accreditation and Quality Control (CAQA) is acting as an “interim testing and certifying agency, until country‐ specific certification frameworks are established. CAQA has agreed to certify non‐Jordanians who comply with CAQA requirements.” Task 1.4 Request a USAID pre‐award survey to obtain a responsibility determination of ACWUA management systems.                                                              21 According to both ACWUA and Chemonics’ sources, the work plan initially developed by IWSMR/Chemonics was not considered acceptable to ACWUA’s management. In addition, the fact that it was delivered to USAID without consultations with ACWUA led to problems, and ul*timately, delays to implementation lasting several months. 25  The pre‐award survey was conducted by an audit firm, and it proceeded ‘quickly and smoothly’22. It issued eleven actions and recommendations that ACWUA has already begun to address. At the time of the evaluation it had completed eight and was in the process of completing the remaining three (See Annex 8 for details). The recommendations/actions concerned the following issues:   o Bank Reconciliation Policies o Payments ‐ Segregation of Duties o Accounting Cycle ‐ Segregation of Duties   o Procurement policies and procedures o Anti ‐Terrorist Checking procedures o Staff Time Management   o Payroll System o Budget vs. Actual Analysis   o Allowable and Unallowable Costs o Direct and Indirect Costs o Chart of Accounts, General Ledger and Financial Statements   Component 2: Regional Water and Wastewater Operator Certification Program Task 2.1 Certification assessment and market analysis  The assessment is being conducted in two stages. The first stage focused on countries directly involved in implementation of the first three training pilots (West Bank, Yemen and Lebanon); while the second (ongoing) involves collecting data from other interested ACWUA members for consideration in the ACWUA Training Plan. Task 2.2 Formulate a market based, sustainable, regional training and certification approach  As of the sixth quarter, a Task Force, composed of the following stakeholders had been created and the terms of the MOU were being negotiated between Jordanian and Egyptian counterparts. The task force members were said to have agreed on the importance of the integration of the two operator certification programs for water and sanitation services that are applied in Jordan and Egypt. The role of the Task Force is to study each country’s program from a legal and technical perspective, and make proposals to modify both programs to enable regional application. The members of the Task Force are:   o The Holding Company for Water and Wastewater Egypt (HCWW)   o The Housing and Building National Research Center Egypt (HBRC)   o Egyptian Water Regulatory Agency (EWRA)   o The Center for Accreditation and Quality Assurance Jordan (CAQA)   o ACWUA   o IWSMR   Task 2.3 Implementation of training and proof of concept                                                              22 IWSMR Quarterly Report #4. 26 During the first two quarters, IWSMR organized a series of pilot training initiatives to test the approach and generate demand for ROCTP. By the last quarter, all Training of Trainer events (which are considered part of the ROCTP) were completed.   At the time of the evaluation, the following training events were held or are to be held:   Completed:   1. TOT Jordan (1st): May 18‐22, 2014   2. TOT West Bank: Aug 3‐7, 2014   3. TOT Yemen: Aug 17‐21, 2014   4. TOT Egypt (1st): Sept 28‐Oct 2, 2014   5. ROCTP West Bank 12‐23 Oct 2014   6. TOT Lebanon: Oct 27‐31, 2014   7. ROCTP Yemen 7‐18, December 2014   8. TOT Egypt (2nd): Jan 18‐22, 2015   9. TOT Jordan (2nd): Feb 15–19, 2015   10. ROCTP Lebanon: March 2‐6, 16‐20, 2015   11. Certification Information System, Jordan: April 21‐23, 2015          Planned:   12. Instructional Systems Design (ISD), Egypt: May 10‐14, 2015   13. Utility Management Training, Level 1, Jordan: May 2015   14. ROCTP Lebanon (2nd): TBD Task 2.4 Improved training materials and programs  A technical subcommittee (comprising the Egyptian Holding Company for Water and Wastewater, CAQA, IWSMR, and ACWUA) reviewed material from existing certification programs, documented gaps, and made recommendations for improvements. As of April 2015 the Egyptian Water Regulatory Authority (EWRA) and Housing and Building National Research Center (HBRC) had yet to sign MOUs. Component 3: Utility Twinning and Leveraging Best Practices The Twinning Program had been prepared, and possible twinning pairs identified, but the program had not yet come into effect at the time of the evaluation.   Task 3.1 Linkages to existing twinning programs  No linkages to existing twinning programs were established.   Task 3.2 Baseline assessments  To prepare the basis for the Utility Twinning program, IWSMR reviewed literature on knowledge exchange and twinning programs. This was followed by collecting information from member utilities on their needs to identify the focus for the twinning arrangements. As of the April 2015, 27 IWSMR had produced three documents: Draft Twinning Manual, a Twinning MOU template, and a Twinning Plan and Schedule. Task 3.3 Twinning arrangements  The Twinning program will initially focus on twinning Aqaba Water Company, considered the “most efficient and effective water utility in Jordan”23 with five water companies (all part of the Water Authority of Jordan) during the final two quarters of the activity (April‐September 2015). Task 3.4 Identification of additional funding streams  No additional funding streams for twinning programs were identified as of the latest quarterly report. Task 3.5 Identify and pilot knowledge‐sharing activities and promote regional good practices in utility finance, management, and service delivery.  The areas of focus for the upcoming Twinning Program have been identified. They will cover non‐revenue water, billing and collection, and data management among other things.                                                              23 IWSMR Quarterly Report #6. 28 PART III. EVALUATION FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Part III is divided into five sub‐sections, each of which includes Findings, Conclusions and Recommendations, which closely follow the five main evaluation questions and sub‐questions. It should be noted that because several themes are cross cutting, similar issues may be addressed in more than one section or sub‐section.  While the Findings are arranged according to the most relevant topics, the Conclusions are organized around the SOW Evaluation Questions (while occasionally including an ‘Addendum’ covering issues not directly related to the questions). Like the Findings, the Recommendations focus on what is deemed most relevant, and do not strictly follow the questions.   ACWUA’S SUSTAINABILITY FOLLOWING PRE-AWARD SURVEY RECOMMENDATIONS Issues related to the following evaluation questions are addressed in this section: Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre‐ award survey resulted in a more sustainable and self‐financing organization? a. How has ACWUA addressed deficiencies identified?  What have been the constraints? b. How has ACWUA’s organizational and financial management changed pre and post IWSMR activity implementation? c. How has ACWUA developed its capacity and mandate to collect and manage member funds? d. How is the Certification Training Information System (CIS) being utilized? Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? Findings The Pre‐award Survey Under the terms of the IWSMR contract, ACWUA was assessed in July 2014 through a Non‐U.S. Organization Pre‐award Survey (NUPAS), or ‘pre‐award survey.’ The survey was conducted (by audit agency PriceWaterhouseCoopers  ‐  PwC Jordan) in order to assess ACWUA’s capability for receiving independent grant funding in the future.   The key objectives of the pre‐award survey were to determine:   29 1. Whether the organization has sufficient financial and managerial capacity to manage IWSMR funds in accordance with U.S. Government and IWSMR requirements;   2. The most appropriate method of financing to use under the potential IWSMR award;24 and   3. The degree of support and oversight necessary to ensure proper accountability of funds provided to the organization. The Pre‐award Survey reviewed issues relating to: organization, management, financial management, accounting systems, internal control, travel, procurement, receipts, disbursements, fixed assets, audit processes, and documentation of policies and procedures.   The pre‐award survey assessed 13 financial risks and assigned scores (on a scale of 1 to 4) to six areas, with accompanying narrative assessments.25 Table 5 lists the criteria and accompanying scores.   Table 5. Pre‐award survey report table with scores for ACWUA (July 2014) No. Criterion Average score 1 Legal Structure   3.40 2 Financial Management and Internal Control Systems   2.15 3 Procurement Systems   2.33 4 Human Resources Systems   2.50 5 Project Performance Management   2.50 6 Organization Sustainability   2.50 Overall Score 2.56 Source: PriceWaterhouseCoopers Jordan. (2014) Improving Water and Sanitation Services in the MENA Region Non‐ U.S. Organization Pre‐award Survey (NUPAS) For Arab Countries Water Utilities Association (ACWUA), July 2014. The overall score of 2.56 (and scores between 2.0 and 3.0 for five out of six criteria) puts ACWUA between ‘weak’ and ‘adequate’ with respect to control weaknesses and exposure to risk. ACWUA                                                              24 Note that, according to communication from Chemonics, the word “potential” refers to IWSMR’s potential use of funds in support of pre‐award survey recommendations. In other words, the results helped identify ACWUA’s areas for internal improvements in finance and management. 25 ‘Inadequate’ (Scoring 1) is defined as: Significant control weaknesses could expose the organization to significant financial or other loss or otherwise significantly impair its ability to manage IWSMR funds. (Key deficiencies and significant weaknesses (SW) that are not remediable before the award or high risk).   ‘Weak’ (Scoring 2) is defined as: “Significant control weaknesses could expose the organization to unacceptable/inadequate levels of unmanaged risk. (Some deficiencies and SW that are not easily remediable before the award or moderate to high risk).” ‘Adequate’ (Scoring 3) is described as: “Although a control weakness was noted, compensating controls and other factors exist to reduce the residual risk within the organization to acceptable levels. (No deficiencies, SW [if any] is remediable before the award or low to moderate risk).” ‘Strong’ (Scoring 4) is defined as:  Overall, a strong control framework is in place given the inherent business risks. Some improvements may be recommended to routine detailed control activities. (No deficiencies or SW or low risk).   Source: PriceWaterhouseCoopers. (2014). The Improving Water and Sanitation Services in the MENA Region (IWSMR). Non‐U.S. Organization Pre‐award Survey (NUPAS) For Arab Countries Water Utilities Association (ACWUA). July 2014. 30 received the highest score (3.40) for ‘Legal Structure’ and the lowest score (2.15) for Financial Management and Internal Control System, reflecting a weak capacity to manage cash flow and absorb additional projects.   The pre‐award survey included 16 recommendations relating to financial and management issues at ACWUA. Key areas for improvements include: timesheet system, bank reconciliations, segregation of accounting duties, procurement and tendering procedures, fulfillment of the requirements of donor agencies, project management capacity, transparency, and organizational structure to ensure efficient and more transparent operation.   The pre‐award survey noted that although written procurement policies exist, no written detailed procurement procedures were in place. For example, the tender process did not identify the mechanism ACWUA uses to perform this activity, and the pre‐award survey report recommended that ACWUA formulate detailed procurement policies and procedures. The pre‐award survey rated ACWUA’s Absorptive Capacity as ‘adequate’, noting that ACWUA has the “ability to recognize the value of new information, assimilate it, and apply it to commercial ends".   The recommendations produced by PwC Jordan in the pre‐award survey were incorporated into the February 2015 Financial Manual Update, also produced by PwC Jordan. This document made additional recommendations to manage and control purchases (purchasing and control procedures) and tendering procedures (controls over the tendering process).   PwC Jordan also produced an Implementation Checklist, with clearly defined deadlines, and set a timeline for the completion of updating manuals and implementing the pre‐award survey recommendations. According to Chemonics, ACWUA was quick to respond to most of the recommendations, apparently taking action in many areas soon after they were received. As of this writing, out of the 16 recommendations, 11 have been addressed (through the implementation of new procedures, manuals, and hiring of a second accountant), and five are under preparation, based on a delivery status report provided by ACWUA (see Annex 8). ACWUA Responses to the Pre‐award Survey   Motivated by the pre‐award survey to improve its performance, ACWUA management reported that it welcomed the recommendations on all fronts. ACWUA’s positive response to the findings and recommendations was confirmed by IWSRM/Chemonics. The pre‐award survey was seen as an opportunity for ACWUA to review its institutional and financial sustainability, serving as a tool that ACWUA could use to improve its operational efficiency. Accounting for Member Funds The pre‐award survey noted that ACWUA’s “accounting records don’t show the accrued membership fees. This may impact the total revenue recognized [and] will lead to uncertainty in funds.”    In response to the pre‐award survey recommendation to “Create written policies on funding and revenue; membership fee collection; create plan for maintaining funding” ACWUA’s Business Strategy 31 2015‐2019 addresses the issue of funding sources (which covers membership fee collection) by introducing transparency into the accounting of inflows.   ACWUA’s Capacity to Collect and Manage Member Funds Although the number of ACWUA members continues to grow at a steady pace, a low level of commitment on the part of some member utilities to pay their membership fees has been an issue. Membership fees have been forecast to, on average, constitute 26 percent of ACWUA’s annual revenue streams for the 2015‐2019 planning horizon, which makes this an area of some concern. (In 2013, it was less than 10 percent.) ACWUA reports that it has also been patient with utilities who are not paying their membership dues, recognizing the constrained circumstances relating to the current geopolitical situation and counting on the goodwill extended during this period to result in members beginning to pay again. While this may be good for the organization’s reputation and long‐term goodwill, it has potential negative implications for revenue in the short‐term.   The Business Strategy 2015‐2019 (which took into account pre‐award survey recommendations) also identifies a range of performance indicators and mitigation options regarding membership including: stimulating active support from the Board of Directors to increase the number of members in all membership categories, establishing an incentive scheme to bring in new members from the private sector, and developing and implementing a marketing and a communication plan for ACWUA’s activities (e.g., regional water conferences, exhibitions, training courses for utility company managers on CIS or benchmarking, training courses for operators, etc.). ACWUA also expects to expand its marketing and communications department and to develop additional tools to collect member funds.   Certification Information System (CIS) The CIS is a system with eight main modules which will enable ACWUA to track certification needs for member countries, which also gives utilities access to training options.26 The CIS was originally to be fully operational and in use by ACWUA in November 2014. However, as of April 2015, the CIS was still being tested internally by IWSMR and ACWUA.     The ACWUA CIS is expected to “make a strong contribution to ROCTP sustainability by allowing member countries, utilities, and operators to efficiently monitor the credentials of technical employees and ensure full and long‐term compliance with certification policies at their water/wastewater treatment and network facilities.”27 Until a regional certification body has been established, it will be enforced by CAQA, as an interim regional certification authority.   ACWUA’s prior experience with training programs (under the Operations and Maintenance Training Project (OMT) 2008‐2012 project) suggests that there is strong demand among utility companies and among operators to obtain certifications which testify to their operational knowledge and capacity. Certificates act as a marker of achievement (and can be used as an indicator of staff competency and as a basis for promotions and pay raises). Certification builds professional confidence, while changing the                                                              26 ACWUA. (nd). Certificate Information System. Presentation. 27 IWSMR Quarterly Report #6, January – March 2015. 32 organizational culture within a utility by increasing accountability.   As certification was only recently incorporated into training, ACWUA anticipates more interest in certification as it trains more operators. Beyond the CIS, the certification program implemented by CAQA is accepted and seen by stakeholders in other countries as a valuable development in strengthening the professional cadres at utility companies. Insofar as operator certification in general is well regarded, CAQA certification is considered acceptable and a good proxy for regional certification.     “A certificate gives you confidence. Wherever I go [to assess other utilities or departments] I ask whether the person has a certificate.” – Female utility department manager, Jordan Key informants and focus group participants were pleased to receive certificates for their courses as a tangible indicator of achievement. Certificates can also be used as an assessment tool by utilities. For example, Aqaba Water Company management noted that they can now use certificates for merit‐based promotion, rather than having to rely on seniority alone. However, it is too early to assess the impact of the CIS, since it had not yet been rolled out externally for utility company members to use. Conclusions Evaluation Question 1: To what extent has the implementation of the recommendations coming from the pre‐award survey resulted in a more sustainable and self‐financing organization? The pre‐award survey was seen by ACWUA as an effective tool to develop a better understanding of financial and management changes necessary to raise its performance, including development of measures to ensure its operational efficiency and effectiveness. ACWUA management stated that it believes that the changes introduced in response to Pre‐award Survey recommendations (see below), and ACWUA’s Charter and Bylaws will serve to ensure the organization operates transparently. The evaluation concludes that these measures can only have a positive effect on the organization’s performance, although the extent of their impact could not be assessed. a. How has ACWUA addressed deficiencies identified?  What have been the constraints? In assessing ACWUA’s financial and managerial capacity between ‘weak’ and ‘adequate’, the pre‐ award survey demonstrated that ACWUA needed to make significant measures to improve in eleven specific areas. The findings indicate that ACWUA took the recommendations seriously and responded relatively quickly by implementing recommended actions (e.g. processes, manuals, and separation of accountancy functions) to address them. This strongly suggests that ACWUA management has taken the recommendations seriously and sees benefits in improving its internal management and accounting practices. Specific constraints were not identified; it is expected that these will only become clear over time, as ACWUA adapts its management and financial practices to fit with the new measures.   b. How has ACWUA’s organizational and financial management changed pre and post IWSMR activity implementation? 33 As a result of IWSMR activity implementation management reports that it has become more aware of the need for instituting stronger internal management and accounting procedures. Because the evaluation did not have audit powers and could not review internal financial and management documents or spend time observing its accounting practices, it was not possible to assess changes in financial management.   c. How has ACWUA developed its capacity and mandate to collect and manage member funds? By adopting a more transparent accounting system to incorporate a breakdown of funding sources in line with the Pre‐award Survey recommendations, ACWUA should be able to better manage member funds. However, it is not clear how and whether collecting member funds has been affected by IWSMR activities, as the key factors in collection rates appear to be utility companies’ willingness and ability to pay, which are not management or accounting issues. (Although this does not imply that nothing can be done.)   d. How is the Certification Training Information System (CIS) being utilized? Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? At the time of the evaluation, given the fact that the CIS had not yet been rolled out to utility members, it is too early to draw conclusions on the usage, effectiveness, or impact of the CIS on member performance.   Recommendations A follow‐up pre‐award survey should be considered to assess to what extent ACWUA has improved, prior to committing to additional funding. The evaluation did not have audit powers or an audit mandate, and therefore could only base its assessment of the pre‐award survey’s impact on circumstantial and secondary evidence   ACWUA performance can be strengthened by introducing an internal monitoring and evaluation of performance results. According to the Strategic Business Plan 2015‐2019, an organizational unit (Financial and Administrative Affairs Unit) will be responsible for monitoring institutional performance compared to preset plans. In developing this unit’s functions, it should be ensured that they focus not just on outputs but on outcomes. ACWUA SUSTAINABILITY AND GROWTH POST IWSMR The following evaluation questions are addressed in this section: Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? a.      What mechanisms are in place to ensure ACWUA’s financial sustainability? 34 b.      What mechanisms are in place to ensure ACWUA’s management sustainability? c.      To what extent will ACWUA be able to maintain its current service provision? The same quality of service provision? d.      What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? e.    Is ACWUA’s management willing to accommodate stakeholder needs?   Findings ACWUA’s Financial Sustainability The evaluation looked at three areas when assessing ACWUA’s financial sustainability: i) internal financial management and organization management; ii) ability to generate revenue through its services; and iii) ability to raise funding from sponsors (international donors and government sources).   Good internal financial management – the ability to manage resources, cash flow, revenues, expenditures, and other financial processes in a manner that is effective, prudent, transparent, and accountable – increases the likelihood that an organization will stay in business. While ACWUA’s financial management systems at the start of ISWMR were rated between weak and adequate (as per the pre‐award survey assessment), ACWUA has apparently embraced the resulting recommendations by implementing a series of financial and management controls. First, it suggests that the organization’s performance has likely improved as a result, and second, it signals that ACWUA’s management takes internal management issues seriously.   A second factor contributing toward sustainability demonstrated by ACWUA is the ability to generate growth in revenue which it does primarily through the services it organizes, particularly training programs (for which it charges overhead) and conferences and exhibitions, and membership fees. The last item has been the most difficult to increase, but ACWUA’s training programs and conferences are well regarded and, seem likely to continue enhancing ACWUA’s reputation and thus its ability to attract clients for its services. ACWUA Funds from international donors and government sources are another key source of revenue.  Since its founding, ACWUA has relied on international donors and the GoJ as a source of funding, both for capacity building and for programs. The Swiss International Development Agency (SIDA) and GIZ have expressed confidence that this assistance is a worthwhile investment in the sector and have spoken positively of the results. At present, ACWUA is partnering with GIZ on the WANT Capacity Development & Training project (2014‐2015).28   In its Strategic Business Plan 2015‐2019, ACWUA forecasts steady growth in revenues from these sources. As discussed below, however, revenue is unlikely to grow at an even and predictable pace, given the nature of its provenance.                                                              28 GIZ website: Strengthening the MENA water sector through regional networking and training (ACWUA WANT) https://www.giz.de/en/worldwide/28308.html   35 Mechanisms to Ensure ACWUA’s Management Sustainability Fulfillment of the pre‐award survey recommendations (discussed above), together with the new Strategic Business Plan 2015‐2019, should be the key drivers to help ACWUA maintain and improve its performance in the future. A planned monitoring and evaluation (M&E) and performance monitoring plan (see next section) should also contribute to its sustainability.   The actions undertaken by ACWUA in response to the pre‐award survey recommendations should make it a stronger organization. By codifying and documenting processes, separating functions, ACWUA is moving from reliance on personal/informal management to institutionalized management.   This will allow it to be less dependent on individuals, and therefore increase the organization’s sustainability following any management personnel changes that may take place in the future. The Strategic Business Plan takes into consideration both challenges and opportunities for sustainability including growth in membership and revenue streams through expanded ROCTP. Effective communication and marketing tools as well as expanded services are at the core of the prevailing strategy. Risk analysis and risk mitigation options form the basis for periodic review and improvements to the Business Plan.   The Business Plan highlights ACWUA’s institutional framework through organizational re‐structuring as the first step towards meeting institutional requirements. A key requirement involves increasing internal capacity and improving skills within ACWUA’s Secretariat. These are planned through the expansion of the organizational structure, including new units and additional staff, to respond to requirements of the next five years.   Finally, ACWUA appears to have a robust and well‐functioning governance framework. ACWUA operates in accordance with its Charter (which addresses objectives, membership affairs, its financial affairs, the bodies comprising the association, awards and grievance, amendments of the Charter, dissolution of the association, and resolution of disputes) and Bylaws which govern operations.   ACWUA’s governance structure includes a General Assembly, Board of Directors, Secretary General, Secretariat, technical committees, and technical working groups.29 Decisions within ACWUA’s Board of Directors are reported to be largely consensus‐based. The Bylaws require that board chairmanship is rotational, alternating every two years between the 18 country members. Feedback from key informants suggests that although ACWUA’s current organizational and management culture in general appears to be receptive and accommodating, the aim is to encourage more participation, inclusiveness, and diversification of the board of directors.                                                              29 ACWUA Strategic Business Plan 2015‐2019. 36 ACWUA’s Ability to Maintain its Current Service Provision at the Same Quality of Service   With respect to service quality, feedback from training participants and conference attendees is positive to strongly positive, as noted elsewhere in this report. ACWUA is also in the process of obtaining ISO 9001 certification30, which should further improve the quality of services it provides. With regard to demand for services, most training participants and utility managers would like to see ACWUA provide additional operator training levels, and engage in more activities, reflecting satisfaction with its performance to date, but also a significant need and desire among MENA water utilities for capacity building.   However, the area of concern most often mentioned by key informants was ACWUA’s internal capacity to sustain its activities. Feedback from several key informants reflects concerns that with ACWUA is taking on many activities but does not always have adequate human resources to manage them all. “ACWUA needs to be beefed up from the inside, they’re overstretched, lack focus and do a lot of firefighting.” – Program manager, international donor, Jordan The issue of staffing constraints is well recognized, both within and outside ACWUA and the Strategic Business Plan includes measures to boost capacity through an organizational restructuring. This involves expanding certain functions by establishing organizational units to meet its needs identified in the Business Plan. The units cover the following areas: Planning and Technical Support, Business Development, Project Management, Marketing and Communications, Capacity Building, Financial and Administrative Affairs, and a Secretariat Office.    ACWUA’s Ability to Manage and Grow Upon Completion of IWSMR When assessing the broader impact of ACWUA on the performance of water utilities, context matters. It is important to hold reasonable expectations of the organization’s potential impact.  In the present case, ACWUA’s potential impact is limited by several factors, including ACWUA’s resources relative to existing utility sector needs, the degree to which its activities and support address fundamental utility performance problems, and the fact that ACWUA is a young and expanding organization.   Resource limitations include human, financial, management, and technical resources. Given that ACWUA has more than 106 utility members (as of this writing), and there are an estimated 80,000 operators in MENA, the direct impact of an organization the size of ACWUA, with less than 15 staff and an operating budget of less than US$ 2 million per year, will by necessity be limited. Communication is an area where various key informants mentioned ACWUA can improve. (This refers to the organization rather than its leadership, which was given by several key informants regarding ability to communicate.) For example, according to a WAJ key informant, even in Jordan’s largest water company, WAJ, very few company employees reportedly know about ACWUA.  The same person also                                                              30 International Organization for Standardization’s 9000 family certifies Quality Management standards for organizations.   37 noted that it would be good if employees were more aware of ACWUA’s services. An Egyptian utility manager noted that very few employees were aware of ACWUA’s training program, an issue that the rollout of the CIS should at least partially address.   “There is no connection between WAJ office employees and ACWUA.”   ‐ WAJ subsidiary manager, Jordan. “ACWUA should have more frequent meetings, more training subjects. I would like ACWUA to issue a monthly newsletter” ” – Key informant, Egypt “Communication is not as I would wish, [it is] the main weak point. Communication with other utilities, and with lower level workers is needed.” Key informant, water utility, Egypt “They need to focus more on dissemination, be more visible with their final outputs.” Key informant, water utility, Jordan However, these perceptions must also be understood within the larger framework, which is that ACWUA has opened up communication channels which apparently did not exist before. “Before ACWUA, we worked alone, with no vision. Now there is a difference – training, collaboration, knowledge exchange between utilities.” – Utility company manager, Jordan Aspects of IWSMR Program Likely to be Sustainable Jointly, the new Strategic Business Plan, proposed organizational restructuring, the introduction of improved financial and management systems (in response to pre‐award survey recommendations), and the development of a new Financial Manual.  These efforts will form a set of planning and management tools which will make ACWUA stronger. Key informants from the various international organizations emphasized the importance of ACWUA and its continued support. The following statements come from four different international organizations:   “ACWUA is a worthwhile endeavor, very important.”  ‐ key informant, international organization, Jordan   “[ACWUA] is one of few organizations that has a regional focus and the convening power.” ‐ key informant, international organization, Jordan “I think the ACWUA concept is good – they can do a lot of things.” ‐ key informant, international organization, Jordan “ACWUA is…unique by its mandate and institutional set‐up ‐ and needs further support; the ACWUA secretariat needs further strengthening… if the donor community stops funding/co‐ funding who will replace ACWUA in such an instable political environment?” ‐ key informant, international organization, Jordan 38 It should be noted that IWSMR has faced some challenges during implementation including work plan development, cooperation with ACWUA, and personnel issues. The challenges may be related, in part, to IWSMR’s relatively small staff compared to ACWUA, weakening its influence. IWSMR’s co‐location with ACWUA staff was an advantage in facilitating communication and ease of access but also offered ACWUA an unusual amount of influence, to the extent that IWSMR staff appear to be reporting directly to ACWUA executive management, i.e. less independent than envisaged. The challenges do not appear to have significantly harmed IWSMR’s ability to meet its goals, largely due to strong back‐up support from Chemonics headquarters. Initial divergent priorities between ACWUA and USAID delayed the start of support.  However, these initial work plan issues were subsequently resolved through consultations. ACWUA’s Management Accommodation of Stakeholder Needs   Feedback from KIIs suggests that ACWUA’s current organizational and management culture is generally receptive and accommodating, although there is room for improvement.   “When ACWUA was formed, I thought it was a forum for high level people to talk to one another, but I was astonished with their work, and management.” – Manager, Jordan utility company The Alexandria Water Company (Egypt) noted that ACWUA can help in verifying its needs and that it would benefit from ACWUA’s knowledge sharing. Company managers reported that knowledge sharing would not only contribute to solving common technical problems faced by its operators, such as water reuse and filters backwash, but also in learning about best practices in the field of water and wastewater in Jordan and their applicability in Egypt. However, some key informants believed ACWUA could be more responsive to utility member needs, for example by basing training for their staff on a needs assessment.   “ACWUA offers many training courses, but doesn’t investigate what we need.” ‐ Manager, water utility, Egypt “ACWUA has been expanding very fast [and has] become involved in too many areas. You can’t just say ‘yes’ to everything, grow so rapidly. Staff is overstretched”  ‐  Program manager, international development partner, Jordan “[We] complained to ACWUA for not contacting its members and for not investigating…our Utility Company needs.” – Utility company manager, Egypt Conclusions Evaluation Question 2: What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? 39 ACWUA’s leadership demonstrates an understanding of the importance of accountability, transparency, and sustainability in its management and financial practices. Evidence for this lies in its response to the Pre‐award Survey. ACWUA management has addressed the recommendations and is updating its Financial Manual. The speed with which it hired a second accountant and began developing new standards, templates, and manuals following the pre‐award survey report is evidence of ACWUA’s responsiveness and willingness to improve its internal operations. Within the remaining life of the IWSMR activity, it is anticipated that ACWUA will implement all pre‐award survey recommendations and develop a plan of action to implement Business Plan activities.  This should provide a solid foundation to continue its path to growth and leadership in the industry.    Based on this past performance, and the fact that ACWUA was not dependent on IWSMR support for the period from its start in 2009 through the end of 2013 (although it did partner with USAID for the development of the Operations and Maintenance Training Project ‐ OMT preparation), there is reason to believe the organization will continue to grow and show prudent financial performance after the IWSMR activity ends.   ACWUA is still a relatively young and developing organization and significant changes in external conditions or management could still have adverse impacts on its sustainability. However, assuming major changes do not occur, and it continues to maintain its positive reputation, ACWUA is likely to be sustainable and also worth providing with continued support.    a. What mechanisms are in place to ensure ACWUA’s financial sustainability? ACWUA has also been proactive in identifying new revenue opportunities by expanding its services (e.g., an MOU with the Government of Jordan to train all water utility operators in Jordan – approximately 6,000), and identifying new funding sources. At the same time, ACWUA spends 55 percent of its budget on staff salaries, which is perceived as inflated. To the extent that the pay scale is known, there is a risk that willingness among members to join or to pay their membership fees could weaken, if they consider high salaries to be an ineffective use of ACWUA finances. However, implementation of a new salary scale beginning with the new Business Plan (in 2015), with adjustments based on skills and qualifications, is expected to address this issue.   b. What mechanisms are in place to ensure ACWUA’s management sustainability? ACWUA’s well‐developed governance structure (consisting of General Assembly, Board of Directors, Secretary General, and Secretariat), together with the new Strategic Business plan and higher standard accounting and managing procedures adopted following the pre‐award recommendations give confidence in its sustainability.   These mechanisms are coupled with ACWUA’s reputation, name recognition, proactive approach to identifying new revenue sources, and awareness in improving its internal management. The Business Plan reflects ACWUA’s plan for growth and sustainability and should be able to serve as a solid platform and a roadmap to address needed changes.   40 c. To what extent will ACWUA be able to maintain its current service provision? The same quality of service provision? Key areas for improvement concern internal management and staff capacity. ACWUA’s small staff is generally seen as competent and hardworking, but is also reportedly overstretched due to the many activities it manages. This could weaken responsiveness and quality of work if not addressed. ACWUA’s Business Plan includes measures to hire new managers, but this will in part be linked to its ability to generate sufficient revenue to cover their salaries.   ACWUA’s ability to continue acting as a service provider will most likely depend on the following factors: i) the quality of the services it provides; ii) the resources (staff, organizational capacity, etc.) it has to organize and provide services; iii) its reputation; and iv) demand for services. d. What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? IWSMR has clearly helped ACWUA increase its sustainability as an organization. IWSMR has equipped ACWUA with key tools and methods, such as its support in developing a Strategic Business Plan 2015‐ 2019, development of the Pre‐Award Survey, and development of Twinning Program material, all of which contribute to ACWUA’s sustainability. The sum of these activities should both strengthen ACWUA’s internal capacity and increase its credibility in the eyes of international donors.   Despite leadership challenges, IWSMR has equipped ACWUA with financial and managerial tools to build on past experience and strengthen its foundation for ongoing reforms and improvements. ACWUA now has the institutional framework to plan for and implement an expanded scope of services. e. Is ACWUA’s management willing to accommodate stakeholder needs? Based on a nearly consensus view of 55 key informants, ACWUA clearly enjoys a good reputation among member countries. ACWUA has emerged as a dependable organization for facilitating knowledge sharing within the water utility sector and highlighting the importance of investing in human and institutional capacities. Its role in promoting awareness of challenges facing the sector is still evolving, but signs are encouraging that it will continue to grow in a positive direction. No key informant suggested that ACWUA was ineffective or that the concept of having such an organization was not worthwhile. ACWUA is seen as a proactive organization bringing together MENA utilities and other institutions and individuals. This is important for several reasons. First, ACWUA’s ability to bring together sector participants for knowledge exchange and networking across organizations is highly valued.   Second, ACWUA is seen as an important catalyst for regional collaboration. This is welcomed because of the same (or similar) spoken languages and the shared culture across MENA countries. Third, utility companies across the region are grappling with many of the same issues. Finally, the difference in performance of utility companies in MENA is seen as more easily bridgeable than the difference in 41 performance between utilities that occupy the broader global spectrum (i.e. the gap between MENA‐ based and European utility companies is much larger). The relative impact of ACWUA may not be large, but key informants with sector knowledge report that it is making a welcome and positive contribution to water and wastewater services in the region.   At the same time, some utility company managers interviewed have stated that they would like ACWUA to tailor its support more closely to their specific needs. However, it can also be argued that utility companies will also need to take some responsibility for assessing their own needs and, as ACWUA members, should approach ACWUA directly with a plan for addressing them.   Addendum The impact and size of ACWUA’s contribution to the sector must be considered relative to: i) other forms of assistance/support that utilities receive; and ii) utility members’ needs and expectations, i.e., with respect to receiving direct assistance, capacity building, and advisory services. Some utility companies have received direct support from international donors, as in the case of Karak utility (Jordan), which was the recipient of a JICA project, or the Potable Water and Sanitation Investments project funded by USAID, which assists Egyptian utilities. This type of support is likely to have a more direct and visible impact than the benefits which ACWUA is promoting, which are less tangible and will likely take more time to filter through to utility operations. Utility companies with experience with direct assistance may expect more than an organization like ACWUA or an activity like IWSMR can reasonably offer (given that ACWUA has less than 20 staff to cover 18 countries, while IWSMR has a budget of less than $2 million). Furthermore, ACWUA’s key area of focus is the spread or exchange of knowledge and ideas (whether by bringing together utility staff at conferences, through twinning programs, in training courses or by other means).  The spread, and especially the utilization, of new knowledge can be notoriously slow as well as difficult to measure.   At the time of evaluation, no assessments had been done of whether training recipients had utilized or spread the knowledge they had gained.   Finally, the underlying assumption that critical utility performance issues, such as high Non‐Revenue Water (NRW) or below cost recovery tariffs, can be addressed by a regional association like ACWUA may not hold. It is possible that utility members may leave the organization, or stop paying their dues, because they have not seen expected performance benefits.      42 Recommendations Continue to provide support to ACWUA and associated institutions, through IWSMR‐type capacity building or similar activities, and possibly through direct budget support. The case for extending assistance to ACWUA is based on the following:   1. ACWUA plays a unique and valuable role in the MENA water sector. Continued support at this time will help the organization, which is still young, to increase its regional presence and impact. 2. There is high and growing demand among utility companies for ACWUA’s services, which have been rated positively by those who benefited from them. Meeting that demand requires further building ACWUA’s organizational and staff capacity. 3. ACWUA leadership and management are well regarded and have demonstrated strong commitment to promoting the organization’s mission. 4. ACWUA’s organizational and financial management has, by all accounts, improved since the IWSMR activity began, at least in part due to its good responsiveness to the pre‐award survey recommendations.   5. Many stakeholders, including other international organizations that support ACWUA, are in favor of continued USAID support. Follow‐on assistance can best be used to focus on the following areas: 1. Management capacity: Continuing to build management capacity and strengthening institutionalization of processes, in the key areas identified by a follow‐up pre‐award survey. 2. Staff capacity: Increasing staff capacity so that ACWUA can maintain the quality of its services and scale them up to meet potential demand. The end‐goal could be to reduce the organization’s dependence on external assistance (both donor funding and outsourcing of activities). 3. Communication: Improve and expand communication and marketing efforts in order to reach a broader audience within the water and sanitation services sector, specifically by extending awareness and capacity building deeper into utility companies’ ranks (i.e., below management level). 4. Membership: Develop a strategy for increasing membership and increasing membership fees.   5. Cost of training: Ways of reducing ancillary costs of training operators (i.e., travel and accommodation), e.g., by building up a larger cadre of ROCTP trainers throughout the region and enabling operators to train in their home countries instead of traveling.   6. Expand individual training programs: Consider developing comprehensive training programs for individuals. 7. Appropriately benchmarked performance indicators: The implications of utility member expectations for this and any future evaluations of ACWUA are that the benchmark (key performance indicators) for assessing performance must be set appropriately. Impacts that are difficult to observe and measure may nonetheless be present. 43 IMPACT OF THE BUSINESS PLAN 2015-2019 ON ACWUA’S EXTERNAL PERFORMANCE Issues related to the following evaluation questions are addressed in this section: Evaluation Question 3: To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? a. How much success have the business planning exercises had in improving the reliability of ACWUA‘s year to year revenue streams?   b. Were lessons learned from implementing the Bridging Memo incorporated into the 2015‐2019 Business Plan? c. Has ACWUA's operational capacity changed as a result of the Business Plan? Findings ACWUA’s Strategic Business Plan 2015‐2019   ACWUA’s Strategic Business Plan (2015‐2019), drafted by a local consultant subcontracted by IWSMR, outlines the organization’s priorities and its roadmap to growth and sustainability. Developed in a participatory manner over several months with input from ACWUA’s Board of Directors, the Strategic Business Plan covers, among other areas: ACWUA’s vision and mission; strategic objectives and approaches on achieving them; ACWUA’s institutional framework, including plans for a new organizational structure and new business units; a strategic financial plan, including expected revenues and expenses; and an M&E system. The Business Plan follows the previous Business Plan for the years 2009 – 2013 and Bridging Memo.   The Business Plan (2015‐2019) was only finalized and approved in January 2015 (just three months prior to the evaluation) and as a result its impact on ACWUA’s regional status or influence could not be assessed. As a key informant from an Egyptian water utility noted, “it is too early to tell whether the Business Plan is effective.” However, the presentation of the Business Plan at Arab Water Week (2015) appears to have benefited ACWUA in terms of its credibility as an organization with the aim of serving as a regional umbrella for facilitating and supporting development of human and institutional capacities of the water utilities. It was welcomed and praised by member countries as a well‐formulated guiding document which can serve ACWUA’s continued growth and development. Revenue Stream With respect to improving the reliability of year to year revenue streams, while the Business Plan includes revenue projections showing gradually increasing revenue, these are best considered benchmark targets, not probable returns. In fact, in discussions with ACWUA management it became clear that ACWUA managers themselves are not overly familiar with the Business Plan’s cost and revenue projections.  For example, the financial plan projects a small deficit (of approximately $17,000, not a cause for concern) in year 2019. When this was pointed out, the amount was plausibly described 44 by ACWUA management as insignificant and relatively far in the future, while it was noted that ACWUAs 2015 budget is already showing a surplus which far exceeds the deficit calculated for 2019.   ACWUA management’s unfamiliarity with revenue and expense forecasts is not necessarily a sign of low interest in financial aspects of the organization, since management appears to be very active in seeking out revenue‐generating opportunities. However, it does suggest that the financial planning element of the business was not given particular attention. This may be partly due to the fact that it was developed by an outside consultant and partly to the currently high degree of uncertainty relating to demand for services amid regional instability.   Nonetheless, as a benchmark, even highly uncertain revenue projections can play a valuable role as a motivating force. It is furthermore notable that the Strategic Business Plan includes an analysis of the previous business plan period (2009‐ 2013), which shows a relatively even growth dynamic on both the expenses and the revenues side. While revenue from membership fees peaked in 2010 and has declined ever since, revenue from training and consulting services grew steadily over the years, from zero in 2009 to becoming by far the largest source by 2013, exceeding all other revenue sources combined. This gives some confidence that revenue growth will continue into the current period of 2015‐2019, although it may not be as even.   In general, ACWUA’s revenue by source fluctuates significantly from year to year, reflecting not only the issue related to membership fees, but also the one‐off nature of some sources such as grants and support from external parties and Arab Governments. ACWUA sees “continuing support from international organizations and funding agencies to develop the water and sanitation sector in the Arab region,”31 as a prerequisite for successful implementation of its Business Plan. However, the most pertinent fact is that, to date, expenses and revenues closely track each other. Referring to the 2009‐ 2013 period, the  Business Plan 2015‐2019 states that maintaining the balance between revenue and expenses is due to “ACWUA’s policies to control costs in light of the low level of commitment of members to settle their membership fees, which are considered one of its main revenue streams.”32   Organizational Restructuring The Business Plan reflects ACWUA’s awareness and the need for organizational, financial and operational change, reform, and improvement so as to enable ACWUA to better support its members in facing water and wastewater utility emerging challenges vis‐à‐vis the Arab Spring that are most common to the MENA region. The plan reflects ACWUA’s past performance including lessons learned and analysis of its current business practices.   According to the Business Plan, the organizational restructuring is driven by the need to meet key strategic objectives, including establishment of organizational units that:33   1. Provide institutional and operational support within the Secretariat;                                                                31 ACWUA Strategic Business Plan 2015‐ 2019. 32 ACWUA Strategic Business Plan, p. 15 33 ACWUA Strategic Business Plan, pp. 27‐28. 45 2. Provide support to ACWUA’s role in terms of communication and knowledge exchange and dissemination and raising awareness to the services provided by ACWUA;   3. Are responsible for all of ACWUA’s technical activities and programs, which are considered the main knowledge producing mechanism within ACWUA; and   4. Provide support to achieve financial sustainability through annual financial planning, in addition to managing all of ACWUA’s accounting and financial affairs according to universally acknowledged best practices. External View of ACWUA’s Effectiveness   When asked about ACWUA’s effectiveness and management, virtually all key informants rated it either four out of five or five out of five, with five being the most positive rating. The ratings reflect many positive perceptions heard about ACWUA, e.g., that it has helped improve the water sector in Jordan, has strong management on the technical side, is a good platform for the water sector, and has a good ability to spread ideas, concepts and good practices throughout the Arab World. The following are illustrative responses referring to ACWUA’s positive impact: “Strategic planning wasn’t a priority of the Greater Cairo Water Company before joining ACWUA.” – Key informant, Greater Cairo Water Company, Egypt “ACWUA is one of few organizations with regional focus and convening power.” – Program manager, international development partner, Jordan “The good thing about ACWUA is that they take good experiences, concepts and spread the knowledge throughout the Arab World.” WAJ subsidiary key informant, Jordan   “ACWUA is a reputable organization and a valuable portal for knowledge sharing and transfer.” – Utility manager, Jordan Some key informants did voice criticism and concerns, often relating to the need for ACWUA to communicate more broadly and better, and a sense that it was taking on more activities than it could manage. “I don’t think it [ACWUA] is living up to expectations, but it could play an important role… the idea is good.” – WAJ subsidiary manager, Jordan “Communication [by ACWUA] was limited to the higher management or board level.” – Key informant, utility company, Egypt A general theme that came across in interviews is that ACWUA leadership has been less focused on internal management and more on external activities. However, critical comments such as these, while revealing, were in the minority and were typically made in the context of overall positive assessments.   46 Conclusions Evaluation Question 3: To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? Discussions with ACWUA management suggest that the financial projects included in the Business Plan do not play a strong role in guiding decisions. A number of factors may contribute to this, including the regional turmoil which has lowered utility members’ willingness and/or ability to pay their fees, as well as to ACWUA’s general focus on raising revenues through marketing its training services and soliciting donor funding. The apparently weaker focus on internal management affairs may also be an explanatory factor. All of the above reinforces the value for ACWUA in having a solid Strategic Business Plan, particularly should management change over the coming period.   It is too early to tell just what the impact of the new Business Plan will be. However, it should be noted that ACWUA’s ability to establish itself regionally as an influential actor in the utility sector goes beyond the Business Plan. ACWUA’s reputation among utility companies in Jordan and Egypt and beyond (based on a small number of key informant interviews)  ‐  particularly among managers who have attended ACWUA‐organized events and training, and operators who have taken ROCTP and TOT training experience ‐ is positive. ACWUA is becoming a recognized brand, and its leadership is well regarded.   a. How much success have the business planning exercises had in improving the reliability of ACWUA‘s year to year revenue streams?   While it remains to be seen what impact the business planning exercises have on revenue streams, review of the 2009 – 2013 business plan suggests the dynamic which can be expected in the future. Thus, while the sources of ACWUA’s year to year revenue stream fluctuate, in total they have been linked closely with expenses. At the same time, it should be emphasized that ACWUA is not a business that relies on what might be considered the traditional ‘customers’ of an association, (i.e. its members) for the bulk of its revenue, but rather has come to rely more and more on the services it provides (organizing training and conference events).   While it is understandable, given the regional political uncertainties (with negative revenue impacts so far) and possible large donor government assistance or contracts (with potentially positive revenue impacts) the five‐year financial projections for 2015‐2019 can quickly become rather theoretical the potential ‘lumpiness’ in revenue will make financial management more challenging.    b. Were lessons learned from implementing the Bridging Memo incorporated into the 2015‐2019 Business Plan? According to ACWUA management, the Bridging Memo did not have a direct influence on the Business Plan and is not mentioned in the latter document. c. Has ACWUA's operational capacity changed as a result of the Business Plan? 47 ACWUA’s operational capacity has not yet changed, but is expected to increase in line with the organizational restructuring envisaged in the Business Plan. Assuming the Business Plan is implemented and used as a roadmap to help ACWUA meet its goals, particularly the financial forecasts, ACWUA should be able to expand more effectively, reaching a wider customer base, and continue to strengthen its reputation. However, low collection of membership fees remains a concern. In the longer term, in order to be considered a regional power, ACWUA will need to get more buy‐in from members and be seen as less reliant on donors. Addendum ACWUA is not currently and is not likely to become a self‐financing organization anytime soon. It operates as a service provider (through organizing trainings, conferences, and exhibitions) as well as an association, and is thus able to generate a more reliable revenue stream than if it depended solely on membership fees in the current geopolitical climate. But as a proactive, well‐regarded regional institution, both international organizations (GIZ, SIDA, JICA) and utility companies have emphasized that it is worth continuing to support.   Recommendations Greater attention to the financial tables in the Strategic Business Plan is recommended, along with the development of an Action Plan to accommodate potential fluctuations in revenue, as noted above. The lumpiness in revenue means that these projections may quickly become obsolete, and could have implications for ongoing expense items which are steady and rising, particularly for salaries as staffing is increased.   ACWUA should develop a plan to reduce non‐payment of membership fees among utility companies, which have dropped significantly since the start of the Arab Spring. At present, it is likely that they will constitute far less than the projected one quarter of revenue for the foreseeable future. This may not pose a significant problem if this falling revenue stream is compensated by other sources. However, in order to stimulate buy‐in and a healthier ‘association’ culture among utility company members, more payment discipline is called for. Often, the fees are relatively small ($2,000 to $4,000 for utility company) and affordable, in theory at least. ACWUA should therefore develop and implement a plan to encourage a culture of payment by members. The plan might include incentives, such as recognition at conferences or on its website, as well as better promotion of the benefits of association membership.   IMPACT OF TRAINING AND CERTIFICATION PROGRAMS ON UTILITY COMPANIES AND OPERATORS Issues related to the following evaluation questions are addressed in this section: Evaluation Question 4: How effective have ACWUA’s training and certification program been in building a cadre of qualified operators as well as buy‐in amongst the members? a. How have lessons learned from the pilot operator certification courses been incorporated into the 48 ACWUA Training Roll‐Out Plan? Do they address needs in the respondent countries? b. How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long‐term structure of the certification program? c. To what extent do member country utilities value the certification of a Jordan organization (CAQA)?   Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? Findings Training and Certification Programs A core ACWUA activity is the provision and organization of training and testing for operators and water and wastewater sector personnel in member countries. The goal of the operators training and certification program is to enhance O&M performance and develop competent staff.   The total number of people trained under the IWSMR activity for ROCTP, TOT and CIS is 217, of which 46 were women (21.2 percent). While nine out of 47 ROCTP trainees were women, 30 out of 107 TOT trainees were women and six out of 12 CIS trainees were women. A breakdown of the gender distribution of the training program participants by country is as follows.   Source: ACWUA data IWSMR provided support to ACWUA in developing a regional operator certification and training course. IWSMR proposed the CAQA as an interim testing and certifying agency for the region, and a Regional Certification Task Force was created to develop such an institution. Many of the materials used for ROCTP training were prepared under the previous OMT USAID project, which was implemented in Jordan from 2008 to 2012 (also by Chemonics). Fourteen ROCTP training programs developed under the OMT were accredited by CAQA, while a further five have been developed and accredited. New courses can be developed within approximately three to six months.   49 At present, ACWUA offers three certification programs: TOT, ROCTP, and Geographic Information System (GIS) applications.34 Descriptions of ACWUA’s training and certification program offerings, including goals, benefits, methodology, program content, is available in a dual‐language booklet “ACWUA Water Operators Certification Program.” Training courses follow the Instructional Design System (ISD) approach.   ACWUA training courses have been certified by CAQA, which also monitors the courses (through both announced and unannounced visits). CAQA administers the tests and issues certificates to participants who pass them. Utilities select and nominate participants to receive training, based on their internal needs (at least in theory), while CAQA approves training nominees based on their qualifications.   Descriptions of ACWUA’s training courses are available in printed material, on its website, and soon will be integrated into the CIS. Training courses are well documented. They follow manuals, include handouts, and are summarized by Completion Reports. The Completion Reports describe the goals, training schedule, course materials, attendance, and pre‐post assessment test results, with scores for individual participants.   While the training pre‐post assessments measure short‐term changes in participant knowledge (comparing responses to the same questions given immediately before and immediately after the training course). Assessing impacts of training on utility performance requires a longer time horizon and different tools. ACWUA plans to implement Training Impact Assessments (TIAs) to further assess and evaluate the effectiveness of TOT trainings. The TIAs will generate feedback which ACWUA will then incorporate into its training strategy. Once the CIS is fully operational, the effectiveness of training for utilities will also be monitored through that system.   Organizing training courses represents a key revenue stream for ACWUA.  ACWUA is actively identifying new training opportunities and marketing its offerings. (It should be noted that ACWUA personnel do not actually deliver the training, but hire trainers to do so). There are two types of training – donor‐ funded/government‐funded and self‐financed, i.e., by utilities themselves (although their training budget may be financed by donors). In April 2015, ACWUA signed a MOU with the GoJ to provide training to all utility operators in Jordan (an estimated 6,000), and it plans to train 75 percent of them within five years. For this to be feasible it will need to significantly scale up its capacity to organize training. There are other water and wastewater training providers in the region besides ACWUA. They include Engicon, the only other certified organization in Jordan to provide such training (which has partnered with ACWUA in the past, although now tends to compete for the same training tenders), and HBRC, in Egypt, which offers training courses as well as certification.   ROCTP Program The ROCTP consists of separate courses for Water Treatment Plan operators (four levels) and for Wastewater Treatment Plan operators (four levels). ACWUA also offers courses for Water Distribution                                                              34 The GIS applications program was not evaluated.   50 operators (three levels) and Wastewater Collection Operators (two levels), but these courses were not part of the IWSMR program. The aim of the ROCTP certification program is to assist ACWUA in qualifying experienced individuals who meet Wastewater Treatment Operators’ selection criteria to become certified operators in their areas of expertise. The program also covers the needed qualifications and skills for operators who have responsibilities operating Water & Wastewater Treatment Plants. A total of 40 operators received ROCTP training with IWSMR support. Prior to implementing the courses, ACWUA/IWSMR prepared MOUs for ROCTP Pilots with the respective utilities “to ensure full understanding of the ROCTP approach and the specific roles and responsibilities of each participating pilot utility.”35 Three ROCTP courses were organized by ACWUA with support from IWSMR in Lebanon, West Bank and Yemen. Out of 68 trainees, 10 (14.7 percent) were women (see Table 6 below). Table 6. ROCTP trainees Country Participants Males Females Total Lebanon 20 1 21 West Bank 19 4 23 Yemen 19 5 24 Total 58 10 68 The evaluation team conducted an online survey of ROCTP participants, all of whom were from outside Jordan and Egypt (which allowed the evaluation to tap into a broader regional perspective), generating 13 responses.36 Asked on their level of agreement with the statement “Taking the ROCTP training was useful for my work” the average rating was 4.3 out of five (where five was equivalent to ‘completely satisfied’ and one was equivalent to ‘completely unsatisfied).   Asked to describe what impact, if any, the training had on their work, the following responses were given:   “I didn't know anything about treatment plants before having this training. Now, I can say that I can enroll this field, work in it, by preparing designs/studies after having more training.” “Improving quality, methods, and work performance.”                                                              35 IWSMR Quarterly Report #6, January – March 2014, p 18. 36 The survey was sent to the email addresses of 40 ROCTP participants. Out of 15 observations, one respondent reported that he did not take the ROCTP training and did not answer questions, and a second observation was a duplicate. The effective response rate was thus 32.5 percent (13 out of 40). 51 “Greater understanding of how to perform all work activities.” “The training was useful to me in preparing studies, and bids evaluation.” “Impact on safety issues and adding knowledge on other practical issues that were gained before the training, by experience. Also the program raised my knowledge on practical matters as well, helping me to give more in my work.” The ROCTP program is viewed positively by utility companies in the region. Utility companies welcome its further development and expansion. In conversations with the ROCTP trainees, four key benefits of the ROCTP program in particular were highlighted:   1. The very nature of the certification vis‐a‐vis impartial auditing and quality assurance oversight ensures the quality of training delivered. 2. The training, when certified by an outside agency with quality assurance oversight, fosters and promotes professional responsibility and accountability.   3. The program builds professional confidence among trainees (i.e., operators).   4. In the long‐term, the collective experience and knowledge from the ROCTP program will contribute to operational efficiency and effectiveness of utilities.    The Beirut Mt. Lebanon Water Establishment was reportedly highly satisfied with the ROCTP training in which its employees participated and requested additional operator training for more of its staff. The responses received from training participants were largely positive.   “[ROCTP] trainings are critical to operations as they change the culture of operation and management and foster awareness and a better sense of accountability and responsibility, which in turn will affect the operation efficiency vis‐à‐vis providing reliable water quality and quantity on a more cost effective basis.” – Female utility department manager, Jordan Although referring to OMT courses under the previous USAID project (which were very similar to the ROCTP training supported by IWSMR and thus gives a sense of the potential impact of IWSMR support in this area), Aqaba Water Company (AWC) management key informants noted that certification programs gave more confidence to its operators, while increasing the company’s credibility. A key informant at AWC reported that the company “felt the difference” after certifying operators and, in one instance, was now able to insource aerator maintenance, reducing costs by approximately 20,000 JD (approximately 28,118 USD) as a result.   Training of Trainers (TOT) Program   TOT programs are part of and complement ACWUA’s ROCTP program. They form a significant element of IWSMR’s support to ACWUA. TOT became one of the key training initiatives which ACWUA manages and which IWSMR supports. The purpose of the TOT program is “to build cadres of capable trainers in the pilot countries and other potential markets identified by ACWUA”37 in order to become certified trainers in their areas of expertise. The rationale from the utility company perspective is that TOT                                                              37 IWSMR Quarterly Report #2, January – March 2014, p. 8. 52 ensures sustainability and helps avoid high training costs (i.e., through outsourcing) because it then enables them to use their own in‐house certified trainers.38 IWSMR noted that TOT courses were not in their original work plan (although it is mentioned as a possible area of work). However, IWSMR wanted to be responsive to ACWUA’s needs, and therefore, agreed to allocate funding for them.   ACWUA describes its role as organizing and conducting TOT programs “for the sustainability of the certification program.”39 When properly designed, with the right participants selected, and the appropriate operational culture, including language and communication, taken into account, ACWUA management expects that the TOT courses will be a low‐cost way to ensure sustainability of the professional knowledge acquired. The TOT course is generic, i.e., not specifically designed for water sector professionals, although examples used during the training draw on participants’ background in the sector. The following outline describes the main elements of the TOT courses, as per ACWUA training description materials:40 ● Responsibilities of a trainer ● Training tools ● Instructions systems design and implementation ● Training preparation ● Training techniques and methods ● Trainees presentations With IWSMR support, seven week‐long (28 hours total) TOT courses were held in different countries, with numerous participants coming from countries other than where the training was held. A breakdown of participants, by gender, is provided in Table 3 below. Women participants represented 28 percent of all TOT trainees. The higher percentage of female TOT trainees compared with ROCTP trainees reflects the fact that operators are more likely to be male, while TOT participants comprised a broader range of positions, including managers and department heads. Table 7. TOT Training Participants (May 2014 – February 2015) Country Participants Males Females Total Algeria 31 1 32 Egypt 19 7 26 Jordan 26 13 39 Lebanon 9 3 12 Tunisia 3 1 4 West Bank 9 3 12                                                              38 Chemonics. Work Plan ‐ Year 1. November 2013. 39 ACWUA Water Operators Certification Program. Third Edition. December 2014, p 4. 40 ACWUA Water Operators Certification Program. Third Edition. December 2014, p 9. 53 Country Participants Males Females Total Yemen 10 2 12 Total 107 30 137 Effectiveness of ACWUA’s Training and Certification Program As part of assessing ACWUA’s effectiveness in building a cadre of qualified operators, the evaluation assessed the TOT program through interviews with utility managers and TOT participants, two focus group discussions with TOT participants, and training documentation. Documentation pertaining to the TOT training courses meets high professional standards and was found to be relevant, succinct, and thorough.   The task of assessing the impact of training and certification on utility company performance is likely to be complex. In the case of TOT it should take into account several key factors, including i) the degree to which internal training and knowledge sharing is valued and promoted within a company; ii) the amount of training which the TOT participant delivers and the number of staff who receive it; and iii) the added value of the training to the TOT participant (i.e., the more they benefited from the TOT, the greater the difference in their training ability should be). The broader impact of TOT will depend in part on how many utility persons are trained and what the multiplier effect is, i.e., the secondary impact of how well they train others. Feedback from utilities, including Aqaba Water Company, Greater Cairo Water Company, and Miyahuna Water Establishment, concerning training their employees received, has been highly positive. Menia Water Company reported that the knowledge gained from the TOT had been applied in internal training courses already. According to ACWUA management, enthusiasm for the TOT program has been ‘overwhelming’.  Nearly all FGD participants were enthusiastic about the TOT course, reporting that they found it valuable and expected it to be useful to them in their work.   “I directly used it. By the second week after the training, I used most of the tips [learned through the course]… It was one of the most successful workshops I ever attended.”  ‐  Female TOT participant, Amman “When ACWUA came with this professional tool to give you a license to certify you as a trainer, it was quite a jump for me as a professional.” ‐ Male TOT participant, Amman “I’m using all the tools with my technical people, such as how to evaluate them.”  ‐ Male TOT participant, Cairo “The TOT training was very useful in that it offers the opportunity to learn how to better communicate and how to convey and share information and knowledge.”  ‐  Female TOT participant, Cairo. 54 Based on the rating questions used for the FGDs and KIIs, satisfaction levels among TOT participants were high, with average ratings for sufficient time, appropriate technical levels and usefulness all above 4.5 and no individual rating of less than four. (See Table 8.)  While the small number of respondents and their non‐random selection means that the results should be considered not statistically significant, confidence in their validity is reinforced by the other information sources – the great majority of the 55 key informants and the 13 online survey respondents also positively rated ACWUA’s performance in the area of training.   Table 8. Evaluation of TOT training courses Level of agreement with the following statements* Average ratings Amman FGD (N = 6) Cairo FGD (N = 6) Combined The amount of training sufficient/insufficient 5 4.3 4.7 The technical level of the training appropriate 4.2 5 4.6 The training was useful 5 5 5 *On a scale of 1 to 5, where 1 = completely disagree and 5 = completely agree. Although as noted above the majority of responses were positive, some participants provided recommendations for modifying or strengthening the TOT course. Some participants found the timeframe of five days too compressed, and others would have liked to see the course tied more closely with the water/wastewater sector. This view was shared by a water company manager who also would have preferred the TOT course to be less generic. The importance of tracking the training impact was stressed.  Finally, the issue of the differences in Arabic dialect when trainees in one course came from different parts of MENA was raised.   “More detail on some subjects would have been better, e.g. how to use PowerPoint, data visualization, etc.” ‐ Female key informant, Jordan “The TOT could be more effective and improved if scheduled over a longer timeframe.” ‐ TOT participant, Cairo Finally, participants in the FGD sessions expressed a strong interest in taking additional technical training and seeing the development of more training courses. This underlines the general and widespread demand for ACWUA’s services which came through during key informant interviews. Training Roll‐out Plan At the time of the evaluation, ACWUA was in the process of finalizing a survey of water and wastewater utilities in countries currently involved in the ACWUA ROCTP (West Bank, Yemen, Lebanon, Jordan, Egypt) as part of IWSMR’s second assessment due to be delivered in June 015. This will feed into a re‐ design of the ROCTP to be implemented in 2015‐2017 and, according to the 6th quarterly report (January‐March 2015), “platform for scale‐up of ROCTP at a sustainable pace.” 55 Building a Cadre of Qualified Operators   The short‐  and long‐term impacts of training on utility performance could not be assessed independently, in part because the training was relatively recently conducted (with most FGD participants and KIIs having taken it within the last six months), and in part because many participants who had completed it more recently, had not begun to use what they had learned yet (e.g., by providing training in their area of expertise to staff at their companies). As with the ROCTP, when the CIS is fully operational, ACWUA will be able to better track the effectiveness and the objectives of the TOT program.   Buy‐in Among Utility Members Based on feedback from training participants, managers, and ACWUA, all indicators are moderately to highly positive, pointing to a well‐managed, high quality TOT training program, and suggesting that there is a good probability of positive outcomes, i.e., that knowledge gained from the TOT courses will be put to use. Buy‐in from utility companies appears to be good, based on interviews with utility company managers in Jordan and Egypt who spoke positively about the expected impacts of improved training capacity.  The MOU signed be the GoJ and ACWUA to train all WAJ operators is a clear indicator of this.   Certification of a Jordan organization (CAQA) Value for Member Countries A key element of the training process is the issuance of certificates to operators who have passed the test.   ACWUA is accredited by CAQA to deliver a water operators certification program in the Arab region. Under the Ministry of Labor, CAQA is Jordan’s government approved certification body, established in 2012. Along with Egypt’s HBRC, it is one of only two certification bodies in the MENA region for water operators. (CAQA covers all sectors except for higher education, accrediting a total of 125 organizations in Jordan). The Association has been endorsed by the GoJ and the Arab League, which approved it as a certification institution for all Arab countries. CAQA is considered the interim certification body for the MENA region until a regional certification body is created.   CAQA’s three main functions are: i) preparing standards for accreditation and quality assurance for all Jordanian training providers; ii) licensing and accreditation of both training providers and programs; and iii) conducting and supervising (occupational) tests.   Although CAQA’s role is seen as beneficial and valued, some FGD participants (who had taken the TOT course) did express frustration with its testing approach, as illustrated by the following quote: “The evaluation (which was administered by CAQA) was a weak point – it was all theory, some questions were very theoretical. Instead, the evaluation should include practical demonstration of what was learned.” ‐ Male TOT participant, Amman   Utility of the CAQA   ACWUA in their formal and informal communication with member countries that received certified trainings reported receiving positive feedback concerning CAQA’s certification. Through IWSMR, CAQA is 56 already providing certifications to 10 of ACWUA’s member countries. Water utilities in Egypt value and recognize CAQA as well, although they have their own certification and accreditation agency (Egyptian Water & Wastewater Regulatory Agency under the auspices of the Housing & Building National Research Center). In addition to post‐training evaluations, ACWUA is planning to implement a Training Impact Assessment (TIA) for a long‐term assessment review of the effectiveness of training, including the ROCTP and TOTs.   Conclusions Evaluation Question 4: How effective have ACWUA’s training and certification program been in building a cadre of qualified operators as well as buy‐in amongst the members? The positive feedback from both trainees and utility companies on the training organized by ACWUA has strengthened the organization’s reputation among members who have benefited from it. The certification process is seen as worthwhile and valuable to both utilities and individuals as a mark of achievement and as an indicator of capability. Clearly, the more operators who receive the training, and progress through the two or four levels relevant to their position, the more buy‐in for ACWUA there should be.   a. How have lessons learned from the pilot operator certification courses been incorporated into the ACWUA Training Roll‐Out Plan? Do they address needs in the respondent countries? While the Training Roll‐out Plan, taking into account lessons learned from the pilot courses offered, could not be reviewed at the time of the evaluation, ACWUA will need to be careful to avoid letting training quality lapse as it increases the scope, number of programs, and countries targeted. b. How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long‐term structure of the certification program? The impact of training on utility performance has yet to be assessed, but the fact that it rates highly increases the chances that its impact will be positive. Nonetheless, the impact is likely to depend on several factors beyond inherent quality, such as i) the degree to which internal training and knowledge sharing is valued and promoted within a company; ii) the amount of training which the TOT participant delivers and the number of staff who receive it; and iii) the added value of the training to the TOT participant (i.e., the more they benefited from the TOT, the greater the difference in their training capacity to train others should be). The broader impact of the TOT will depend in part on how many utility persons are trained and what the multiplier effect is, i.e., the secondary impact of how well they train others.   c. To what extent do member country utilities value the certification of a Jordan organization (CAQA)?  Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? 57 Based on feedback from Egypt, receiving certification from CAQA is clearly valued, its national mandate for Jordan (as opposed to for the entire region) notwithstanding. The fact that it was endorsed by the Arab League as an approved institution for all Arab countries carries considerable weight. This is in the context of a move to establish a regional body for certification.   Addendum Training programs represent an important revenue stream for ACWUA, which is placing a strong emphasis on scaling up the program, developing new training modules, and identifying new clients. The potential for training and certification of operators in MENA is potentially vast, with an estimated 100,000 water utility operators engaged in the sector region‐wide.   Recommendations ACWUA should build a larger and stronger cadre of experienced trainers to reduce the need for training participants to travel. The cost per trainee increases when travel and accommodations must be provided, which naturally limits the number of operators and other personnel which can be trained. Developing a cadre of capable trainers who can travel would increase the number of operators which could potentially be trained. ACWUA should consider revising the TOT program to make it less generic and incorporate more material or examples relevant to the water and wastewater sector to the extent feasible. While it was highly regarded by those who took the course, the link between TOT and the ROCTP program was not always clear and can be made more explicit.   UTILITY COMPANY TWINNING PROGRAM Issues related to the following evaluation questions are addressed in this section: Evaluation Question 5: What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs?   a. To what extent has information and peer support between water and wastewater utilities been established through the regional twinning program? b. What lessons have been learned from the two pilot virtual twinning/exchange activities run by IWSMR? c. What has been the nature of exchange and collaboration amongst the twinning partnerships? Has it helped in building staff capacity? d. How effective are the existing indicators in measuring effective management of twinning partnerships? 58 Findings Twinning Program Preparation As noted earlier, the Twinning Program had been prepared, and possible twinning pairs identified, but the program had not yet begun at the time of the evaluation. IWSMR provided support by reviewing literature on knowledge exchange and twinning programs, and then collecting information from member utilities on their needs to identify the focus for the twinning arrangements. As of the 6th quarter (January‐March 2015), IWSMR had produced three documents: a Draft Twinning Manual; a Twinning MOU template; and a Twinning Plan and Schedule.   Twinning Program Participants The Twinning program will initially focus on twinning Aqaba Water Company, considered the “most efficient and effective water utility in Jordan”41 with five water companies which are all part of the Water Authority of Jordan (WAJ) during the 7th and 8th quarters of IWSMR (covering the period April‐ September 2015). The relationship was described as a mentorship program, with Aqaba Water Company sharing its expertise with smaller utility companies who are affiliates of WAJ. It is noteworthy that Aqaba Water Company is a private operator while the mentees are publicly owned.   Table 9. Planned Utility Twinning Program (in Jordan) Mentor Utility Partner Utility Topic Date Aqaba Water   Salt   Non‐Revenue Water   Q7   Aqaba Water   Ramtha   Billing and Collection   Q7   Aqaba Water   Zarqa   Un‐accounted‐for Water   Q8   Aqaba Water   Karak   Data management   Q8   Aqaba Water   Talifah   Data management   Q8   Source: IWSMR Quarterly Report #6, January – March 2015 A regional Twinning Program would then draw on the experience from the Jordan program. Possible utilities and the issues of focus are outlined below:   Table 10. Regional Twinning Program Issues Participating utilities NRW/leak detection   Tunisia, Jordan (Aqaba Water)   Energy Efficiency   Egypt (Utility TBD)   Chemical Optimization   Egypt (Utility TBD)   Customer Support/Service   TBD   Billing & Collection   Miyahuna Jordan – Madaba   Asset management   Egypt (Cairo), Morocco (ONEP)   Source: IWSMR Quarterly Report #6, January – March 2015                                                              41 IWSMR Quarterly Report #6. 59 Some modifications were made to the Twinning Program to reduce costs, with plans to make it virtual, i.e., communication via electronic means rather than travel to onsite locations.  Whether or not this will impact its effectiveness remains to be seen.   Conclusions No conclusions regarding the impact of the Twinning Program could be drawn at the time of the evaluation. Recommendations No recommendations are made at time of the evaluation. 60 ANNEXES ANNEX 1. STATEMENT OF WORK Final Evaluation of the USAID ME/TS IWSMR Project I. Purpose and use of evaluation The purpose of this SOW is to request a final evaluation of Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR), a water activity managed by USAID’s Middle East Bureau/Technical Services (ME/TS), and implemented by Chemonics International, Inc. The evaluation will provide insight into the activity’s success in building the Arab Countries Water Utilities Association’s (ACWUA) capacity to sustainably carry out its mandate to promote certification and accreditation, capacity development and information exchange among water and wastewater utilities and professionals during and beyond project completion. Specifically, the evaluation should look into ACWUA’s ability to sustain itself as a regional entity and manage funds given its future role in the region and its capacity at end of activity. Findings and recommendations will be shared with ACWUA to help determine remaining needs and to address any gaps that may have been overlooked during life of activity. Additionally, the evaluation results will be used by ME/TS during the potential design of any follow-on activities’ ability to help improve management of water resources in the implementing countries. The findings and recommendations will also be shared with USAID Missions in the countries where the awards are implemented, with the IWSMR team, and with other relevant stakeholders, as needed. II. Identification of Award Award Title: Improving Water and Sanitation Services in the Middle East and North Africa Region Implementing Partner: Chemonics International, Inc. Mechanism: Contract Task Order; Award No. 263-TO-13-00003 Award Duration: Started on 09/30/2013 and is scheduled to end on 9/30/2015. Award Budget: $1,991,240 Areas of implementation: Egypt, Jordan, Lebanon, West Bank/Gaza, and Yemen III. Description of Award The Impr technical regionally ensure th founded members East and Lebanon Figure 1 The IWS including North Af expertise ● B ce p ● E ef Jo ● L m “K M These ob Phase I: roving Wate l and capaci y recognize hat there is a in 2006 with s include util d North Afri , West Bank . Timeline o SMR progra g GIZ, that a rica (MENA e and assista Building the ertification o rograms. Evaluating th ffectively ex ordan and/o everaging r management Knowledge MENA water bjectives are Year 1 er and Sanit ty building re ed water an a standard s h GIZ suppo lities, ministr ca. IWSM k/Gaza, and of ACWUA's m is implem aim to impro ) region. To ance around capacity o of water and he feasibilit xpand exist or Egypt to a regional US t and operat Exchange”) and wastew e being achie tation Servic esources to nd wastewa set of knowle ort; it official ries, and priv MR works w Yemen. s establishm mented in c ve water su o achieve th d the followin of ACWUA wastewater ty of, and ing water a regional sca SAID-funded tion by expl to encoura water industry eved in two d ces in the M ACWUA to ater operatio edge to incr ly began op vate sector c with the follo ment close coordi pply and sa is objective ng three activ to enable r utility profe developing and wastew ale. d improvem oring trainin age informat y. distinct phas MENA Regio support the on certificati rease operat perations in countries fro owing geogr nation with anitation serv , the IWSM vities: it to achiev essionals and g and imple water operat ments in wa ng opportuni tion and be ses: on (IWSMR) developmen ions. Opera tor efficienci January 200 om 18 count raphic areas other dono vices in the R activity p ve its mand d accreditat ementing a tor certificat ater and w ities (such a st practice activity pro nt and delive ator certifica ies. ACWUA 09 in Amma ries in the M s: Egypt, Jo ors in the se Middle Eas rovides tech date to pro ion of assoc a mechanis tion program wastewater as utility twin exchange in 61 vides ery of ations A was an. Its Middle ordan, ector, st and hnical omote ciated m to ms in utility nning n the 62 Stage 1: 1) Assessment 1: Surveys conducted 2) Planning for Implementation – ACWUA Capacity Building a. Revise Strategic Business Plan 2010-2014 Edits to include new ACWUA Operator Certification and Training Program b. Draft new Strategic Business Plan 2015-2019 c. Bridging Memo developed with ACWUA staff to link old and new strategic plan 3) Launch of Activities (Regional Certification Approach) a. Training of Trainers (goal: 109 according to APR1 PMP FO) b. Region-wide campaign to introduce operator certification (goal: 96 operators certified) c. CAQA testing and Certification 4) Pre-Award Survey conducted: ACWUA’s financial and management systems Phase II: 1) Assessment 2: Marketing; Surveys; ACWUA ROCTP Rollout Plan 2) Implementation – ACWUA Capacity Building a. 2015-2019 Business Plan Approved by ACWUA Board of Directors b. Help ACWUA obtain responsibility determination from USAID c. Develop and utilize the training and certification information system (CIS) d. Establish an Egypt-Jordanian task force to standardize curricula e. Establish a Certification Unit 3) Completion of Activities (Regional Certification Approach) a. Establishment of a regionally recognized certification scheme b. Training of Trainers c. ROCTP Pilot Trainings d. Jordan/Egypt ISD Course e. Certification Training in Jordan-Utility Management f. Twinning Plan finalized and Implemented Year 1 – All of Phase 1 and some components of Phase 2 were conducted during Year 1. Assessment surveys were given to member utilities. Information gathered through the assessments was used to a) review and update ACWUA’s 2015-2019 Strategic Business Plan, b) evaluate the feasibility of and identify approaches to scaling up regional certification programs, and c) explore opportunities to improve the service delivery of utilities (e.g. utility twinning program). The Year 2 stage of the assessment process began in October 2014 and will gather data from other interested ACWUA members for consideration in the ACWUA Training Roll-out Plan. Phase 1 is expected to end in January 2015 once all of the pilots have been completed. The assessments in Phase 1 were used to inform the implementation-planning phase and launch selected activities jointly with ACWUA from January to March 2014. This included revising the 2014 part of the Strategic Business Plan (2010 – 2014) to emphasize the new ACWUA Operator Certification and Training Program, and second, developing the ACWUA 63 Strategic Business Plan for 2015-2019, working closely with the ACWUA Board of Directors. The main output of the first stage was a “Bridging Memo,” which served as a link to the next five-year plan and increased the organization’s emphasis on certification. This memo was drafted in January 2014, in cooperation with ACWUA staff. Moreover, the Phase 1 assessments were used to gauge needs and ability to pay for the trainings from the training pilot countries. Raw data was also provided to ACWUA for eventual inclusion in the Training and Certification Information System (CIS). The CIS will include all data/information related to the regional certification program including records for operators/ trainees, certified trainers, training implemented, tests and results, and recertification requirements. The questionnaires collected data and information that confirmed that there were no existing in-country initiatives to establish certification schemes, and no constraints preventing the recognition of Jordan’s Center of Accreditation and Quality Assurance (CAQA)-issued certificates. These certificates serve as licenses to operators who have taken the training and passed the tests. Phase 1 of the assessment also provided preliminary recommendations for ACWUA knowledge exchange and twinning opportunities, in addition to plans for ongoing assessment, recommendations, and action planning. These results, and those forthcoming from Stage 2, will serve as the basis for developing a twinning program that ACWUA can undertake given the availability of required financial resources. IWSMR and ACWUA have collected data from member utilities on their needs and areas of focus for twinning arrangements, as inputs to a “match-making” process among utilities. In year 2, IWSMR will work with ACWUA to develop a plan for implementation of five virtual twinning activities and will directly support implementation of two of them on a multilateral basis while ACWUA takes the lead on the remaining three. Furthermore, IWSMR and ACWUA will jointly identify ten more twinning activities to be implemented in the future beyond the life of IWSMR. Finally, IWSMR will also support ACWUA in developing a twinning manual that can be used to carry out any twinning exercise in the future including suggested measureable indicators and impact evaluation techniques. To objectively evaluate ACWUA’s financial and management systems, IWSMR hired an auditing firm in Jordan to conduct a pre-award survey. Although it is called a pre-award survey, it is more of a pre-award audit. The survey looked at ACWUA’s legal structure, financial management and internal control systems, procurement systems, human resources systems, project performance management, and organization sustainability. The survey, completed on July 23, 2014, awarded ACWUA an overall score of 2.56, which falls within the “adequate” range, meaning that although a control weakness was noted, compensating controls and other factors exist to reduce the residual risk within the organization to acceptable levels. IWSMR used the findings from the survey to support ACWUA in addressing any deficiencies and submitting a responsibility determination to USAID. A responsibility determination is the pre-requisite to receive any direct USAID grants. Using a regional certification approach, IWSMR planned a series of pilot training initiatives to test the approach and generate demand for the program. The certification training pilots will be coupled with Training of Trainers courses, to build cadres of capable trainers in the pilot countries and other potential markets identified by ACWUA. To further support regional program development, Jordanian and Egyptian subject matter experts will participate in an Instructional 64 Systems Design (ISD) Course intended to facilitate further development of their existing programs per the recommendations of the Regional Certification Task Force. Equipped with the products generated by the Jordan and Egypt programs, ACWUA is capitalizing on operator certification advances to spearhead a regional certification program among its member utilities. Over the past five years, ACWUA has joined the American Association of Boards of Certification (ABC) as a regulatory member and has built a working relationship with its governing body and with Office of Water Programs at Sacramento State University, publishers of the definitive reference material in this domain. Over the course of Year 1, ACWUA’s board has demonstrated committed support to establishment of a regional certification program and believes the market is in need of region-wide standards for operators. Association members are ready to develop their human resources to help preserve utility infrastructure and improve service delivery. IWSMR began the process to fully exploit these achievements and, working through ACWUA’s board, is effectively assisting ACWUA in launching a region-wide campaign to introduce operator certification; the project close, 200 operators will be certified through the regional platform. These efforts included the solicitation of applicants for the various offerings under the program among the association’s members or through buy-ins from donor activities in the water and wastewater sector. CAQA in Jordan agreed to certify non-Jordanian operators and issue certification and licensing documents at the same rates charged for Jordanian operators providing that CAQA standards and requirements are met. However, the logistics and expenses of CAQA staff conducting tests on-site at ACWUA member utilities throughout the MENA region is clearly impractical and not cost-effective. CAQA, ACWUA, and IWSMR will pursue a solution whereby tests can be administered by a third party under CAQA control and standards. To date, IWSMR has worked closely with utilities in Egypt, Yemen, Lebanon, and West Bank to organize pilots and maximize attendance. Certification pilots have taken place in West Bank (Nablus), Yemen (Sanaa), Lebanon (Beirut), and Jordan (Amman). An Instructional Systems Design Course was held in Amman with 15 participants from Jordan and Egypt. The highly successful training of trainers (TOT) has trained 106 people from 10 countries in six separate training events (Jordan, West Bank, Yemen, two in Egypt, and Lebanon); the TOT is considered by ACWUA as the base for any successful regional certification and training program as it increases the regional pool of trainers. At least 25 percent of the training participants were women. Phase 2 – IWSMR will work with ACWUA and member countries and utilities to carry out recommended actions. As per the IWSMR implementation plan, planning, coordination, and training activities that were initiated in Year 1 and, as per the work plan, will continue throughout Year 2 (from October 1, 2014 to September 30, 2015). Throughout the project there have been minor obstacles that the project management unit has been able to overcome, thus not significantly deviating from the implementation and work plan schedules. There was a consensus building period between January and March 2014 to get buy-in from ACWUA on the implementation plan, which included detailed plans for certification training events, including the timing, location, and participating countries and utilities. This consensus-building period, though envisioned and planned in the IWSMR work plan, took 65 longer than anticipated. The result, however, was a robust set of activities that meet the approval of USAID, ACWUA, and IWSMR and aim to maximize the impact with the resources available. To avoid any similar delays in the future, the IWSMR team schedules weekly meetings with both USAID and ACWUA and documenting the agreements made in such meetings, thus improving the efficiency and transparency of communications. The West Bank TOT event was postponed at the request of the West Bank counterparts due to the deterioration in security in Nablus and throughout the West Bank in June 2014. Subsequently the visa applications were denied for all Jordanian travelers for the program, including the trainer and the ACWUA and IWSMR representatives. To avoid repeating this scenario, meetings and trainings scheduled in West Bank were relocated to Amman. As per the Implementation Plan/Year 2, during Quarter 5, the ACWUA/IWSMR team began the Stage 2 assessment process to support the ACWUA ROCTP Rollout Plan vital to program sustainability post-IWSMR. Survey questionnaires were distributed to ACWUA member countries along with a detailed ROCTP description including selection criteria and requirements. Two followup questionnaire forms were developed in Quarter 5 for distribution to utilities in Quarter 6: ● Enrollment Form to Request Additional ROCTP Courses – This form was developed for countries currently involved in ACWUA ROCTP: West Bank, Yemen, Lebanon, Jordan, Egypt ● Application Form for New ROCTP Pilot Courses – This form was developed for ACWUA member countries that are not yet involved in ROCTP. The IWSMR team experienced significant staff turnover during the third quarter. By the end of the quarter, however, all open positions had been filled. In the fifth quarter, however, the key position of COP and the DCOP resigned. This has slowed down project progress, but all targets have ultimately been met. IV. Existing Performance Information Sources There are a number of available relevant documents and performance information sources that the evaluation team can draw upon. Existing sources of information include: 1) Initial contract documentation 2) Annual work plans (Year 1 and Year 2) and Annual progress reports (Year 1); 3) Quarterly reports submitted by IWSMR to USAID covering the following periods: Oct￾Dec 2013; Jan-Mar 2014; Apr-Jun 2014; Jul-Sep 2014; Oct-Dec 2014; and Jan-Mar 2015. 4) Training materials, attendee lists, and completion reports, to the extent available. 5) Pre-Award survey, completed on July 23, 2014 V. Evaluation Questions The Evaluation Questions are, in priority order: 66 1) To what extent has the implementation of the recommendations coming from the pre-award survey resulted in a more sustainable and self-financing organization? a. How has ACWUA addressed deficiencies identified? What have been the constraints? b. How has ACWUA’s organizational and financial management changed pre and post IWSMR activity implementation? c. How has ACWUA developed its capacity and mandate to collect and manage member funds? d. How is the Certification Training Information System (CIS) being utilized? Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? 2) What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? a. What mechanisms are in place to ensure ACWUA’s financial sustainability? b. What mechanisms are in place to ensure ACWUA’s management sustainability? c. To what extent will ACWUA be able to maintain its current service provision? The same quality of service provision? d. What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? e. Is ACWUA’s management willing to accommodate stakeholder needs? 3) To what extent has the 2015-2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? a. How much success have the business planning exercises had in improving the reliability of ACWUA ‘s year to year revenue streams? b. Were lessons learned from implementing the Bridging Memo incorporated into the 2015-2019 Business Plan? c. Has ACWUA's operational capacity changed as a result of the Business Plan? 4) How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy-in amongst the members? a. How have lessons learned from the pilot operator certification courses been incorporated into the ACWUA Training Roll-Out Plan? Do they address needs in the respondent countries? b. How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long-term structure of the certification program? c. To what extent do member country utilities value the certification of a Jordan organization (CAQA)? Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? 5) What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs? a. To what extent has information and peer support between water and wastewater utilities been established through the regional twinning program? 67 b. What lessons have been learned from the two pilot virtual twinning/exchange activities run by IWSMR? c. What has been the nature of exchange and collaboration amongst the twinning partnerships? Has it helped in building staff capacity? d. How effective are the existing indicators in measuring effective management of twinning partnerships? Based on the answers to the above evaluation questions, the evaluation team should also address: 1) Lessons Learned: What are lessons learned, if any from implementation to date that ME/TS should take into consideration for design of future activities in this area? 2) Recommendations: a) What adjustments, if any, should be made to improve future activities' ability to help improve management of water resources in the implementing countries? VI. Evaluation Design and Methodology 1) Overall Evaluation Design Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to generate answers to the evaluation questions listed above. 2) The overall design of the evaluation should consider answering the evaluation questions on two levels. Suggested data collection methods include: 1. Review/analysis of project reports for USAID 2. Review of project documents prepared for government agencies and other stakeholders, including recent audits and pre-award surveys 3. Review of documentary evidence from utilities and other stakeholder groups 4. In-depth and open-ended interviews with key informants 5. Focus groups or group interviews with key informant groups 6. On-line surveys of key informants and training participants 7. Direct observations in the relevant organizations. 8. Process-tracing, contribution analysis, or other “case” based causal analysis Suggested sources to include: 1. Top and mid-level managers engaged in shaping regional water management strategies, and young professionals active in generating data and analysis; 2. Training and workshop participants (to be chosen from different countries); 3. ACWUA staff and training instructors; 4. ACWUA utility members; 5. Regulatory agencies 68 6. Jordanian (Center of Accreditation and Quality Assurance) and Egyptian (Egyptian Water Regulatory Authority) accreditation bodies 7. The Housing and Building National Research Center in Egypt *Other countries that ACWUA provides support to but are not beneficiaries of the USAID funded activity include: Iraq, Tunisia, Morocco, and Algeria. The evaluation team should interview and survey these utilities in the noted countries as to the quality of services being provided by ACWUA. The findings will help in evaluating ACWUA’s overall capacity. The list below includes contacts from participating agencies included in this evaluation. Participating Organization/Country Center of Accreditation and Quality Assurance (CAQA) Eng. Mohammad Khair Irshaid Center of Accreditation and Quality Assurance (CAQA) Mr. Emad Malkawi Egyptian Water and Waste water Regulatory Agency (EWRA) Dr. Eng. Sobhy Abd Al-Kader Housing & Building National Research Center (HBRC) Dr. Eng. Mohamed Nabeel Abdel Salam Arab Countries Water Utilities Association (ACWUA) Eng. Khaldon Khashman Palestinian Water Authority Mr. Ziyad Al-Fuqaha Jerusalem Water Undertaking, Ramallah/Al-Bireh Governorate Eng. Emil Abdo National Water and Sanitation Authority, Sana’a, Yemen Mr. Mohammed Al Mutawakel Holding Company for Water and Wastewater (HCWW) Eng. Mohammad Moawad Ministry of Water Resources, Baghdad- Iraq Eng. Ayed Hussein Qathem Al-Sa’aidi Jordan Water Company (MIYAHUNA) Eng. Majedah Ali Al-Zoubi Water Authority of Jordan (WAJ) Eng. Asma' Ahmad Al-Whadneh See an example of an illustrative design matrix for suggestions for linking the data collection methods and sources to evaluation questions. The evaluation team is required to develop a design matrix for the upcoming ACWUA evaluation. Fieldwork is expected to take place in Jordan and Egypt. The evaluation team will be accompanied by a staff member from ME/TS to observe interviews and fieldwork. The evaluation team will provide ME/TS with a list of interviewees and the schedule of meetings/interviews to take place based on discussion with ME/TS. 69 The evaluation team shall share the data collection instruments with ME/TS for review and feedback before carrying out the fieldwork. This will include key informant interview questionnaires, focus group discussion protocols, online survey questionnaires. Also, the evaluation team will develop a list with all agreed upon awards’ deliverables. The draft list will be reviewed and approved by ME/TS. The evaluation team will coordinate with ME/TS to ensure that the respective USAID missions and U.S. Embassies are informed well in advance of any fieldwork. VII. Deliverables and Reporting The Contractor must be familiar with the new USAID Evaluation Policy and ensure that the final deliverable is consistent with the concepts, terminology, and practices outlined therein. The final deliverables are as follows: 1. Evaluation Design and WorkPlan: The Contractor will submit a final evaluation design and workplan to Task Order Contracting Officer’s Representative (TOCOR) covering (a) the overall design strategy for the evaluation and evaluation design matrix, (b) the detailed methodology, including data collection and analysis plan for the evaluation including a fieldwork schedule, proposed interviewees, proposed questionnaires and other data collection tools, (c) a specific timeline (using Gantt chart) and (d) budget. The evaluation design and workplan will be submitted to the ME/TS TOCOR for approval and will be shared with the implementing partners prior to being finalized. The evaluation design and workplan will include two phases for the evaluation: Phase 1: Review of relevant documentation; and its analysis. Phase 2: Based on the results of Phase 1, conducting fieldwork to meet with the implementers and key stakeholders. This involves collecting data to answer the designed evaluation questions. 1. Draft Evaluation Report and Briefing: The Contractor will submit a draft report and conduct a detailed briefing to present preliminary findings and recommendations of the evaluation to the TOCOR (two) week before completion. Recommendations for future programming will be addressed; including any programmatic adjustments that are necessary. USAID will review the report and provide written comments on the draft within 14 days of receipt. 2. Final Evaluation Report: The Contractor will submit a final report that addresses USAID’s comments. Recommended length of no more than 30 pages, excluding the title page, annexes, and evaluation data. The final report is recommended to include: 1) Executive Summary of the purpose, background of the activities, main evaluation questions, methods, findings, conclusions, recommendations, and lessons learned, not to exceed three pages. 70 2) Introduction and Background (including evaluation purpose, background of the activities, award descriptions, and main evaluation questions). 3) Body of Report organized as follows for each component, not to exceed 30 pages: a) Methodology (including a discussion of limitations; b) Major Findings (Evidence); c) Conclusions; d) Recommendations; and e) Lessons Learned. 4) Annexes to include the following: statement of work (including any modifications), reference list of documents, list of persons contacted and affiliation, detailed methodology (including a discussion of the limitations of the methodology employed), data collection tools and protocols, evaluation team disclosures of real or potential conflicts of interest, any statements of difference submitted by activity implementers, team members or USAID. 5) Evaluation Data: raw data and records (e.g. interview summaries, survey responses, etc.) in electronic form collected by the evaluation team, along with relevant codebooks for the data. The data should be organized for use by those not fully familiar with the project or the evaluation. All reports must be submitted in English. The Contractor will submit four (4) hard copies of the Final Report and one (1) electronic copy in Adobe format to ME/TS. A copy of the final report will be submitted to the USAID’s Development Experience Clearinghouse (DEC) and will be shared with the implementing partner and beneficiary: IWSMR and ACWUA teams. The report will also be shared with the local ministries of the involved countries to enable them to make use of the findings and recommendations for their future programs. As per the USAID Evaluation Policy of January 2011(http://www.usaid.gov/evaluation), the following criteria must be followed to ensure the quality of the evaluation report: - The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the award, what did not and why. - Evaluation reports shall address all evaluation questions included in the scope of work. - The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. - Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report. - Evaluation findings will assess outcomes and impact of the program design and access to benefits and resources on males and females at the governance and community levels. 71 - Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.) - Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. - Sources of information need to be properly identified and listed in an annex. - Recommendations need to be supported by a specific set of findings. - Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. VIII. Illustrative Timeline and LOE: Stage Activities included Timeframe Planning Review of relevant award info/documents 1 week Evaluation Plan submitted (Proposal) and approved Evaluation Team meeting with ME/TS Preparations Selection of interviewees and interviewers 1 week Select and prepare sites Develop detailed data analysis plan ME/TS meeting with the ET to discuss work plan including list of interviewees and data collection tools. Field implementation Conducting Fieldwork (interviews and site visits) 2 weeks Data Analysis Analysis of data and findings 2 weeks Reporting Draft of report delivered 3 weeks Washington de-brief of findings, 72 conclusions and recommendations. Incorporated feedback and final draft submitted; dissemination of final report and findings to stakeholders via teleconference (i.e., ACWUA, implementor, IWSMR PMU, utilities interviewed, USAID/Jordan Water Resources Office, etc.) Total Duration 9 weeks *A week could be 6 working days. IX. Evaluation Team Composition The following three positions are considered key personnel. - Team Leader: The Team Leader will be a senior or mid-level international expert, with 5 years of experience in leading and conducting evaluations of development activities, preferably with experience in social science evaluation methods, especially program performance evaluations, rapid appraisal techniques, case studies and other relevant data collection/analysis techniques. The Team Leader will have technical water expertise with a minimum of 10 years of experience in leading and managing water projects, preferably in the MENA region. Experience working with USAID evaluations is required. The team leader should also have superior management, interpersonal relations and writing skills, and a solid technical understanding of issues related to water resources management, preferably in MENA region. A minimum of a graduate degree is preferred. - Team members (2): The Contractor will propose two team members (an international and local expert) who are required to complete the task. The Contractor’s proposal will include each team member’s name and key skills relevant to this evaluation, with a current Curriculum Vitae included as an annex to the Technical Proposal. Team members must include: One scientist or engineer with technical expertise (minimum 10 years’ experience) in water utility management; and at least one team member must be proficient in Arabic. X. Evaluation Management 1. Logistics 73 USAID will provide key documents to the evaluation team and provide introductions to the implementing partners prior to the initiation of field work. The evaluation team is responsible for arranging vehicle rental and drivers as needed for their site visits in the countries of fieldwork as well as scheduling meetings. They will also need to arrange their own hotel arrangements and procure their own work/office space, computers, internet access, printing and photocopying in line with relevant USAID regulations. Evaluation team members will be required to make their own payments. USAID/ME will be available to the team for consultations regarding sources and technical issues, before and during the evaluation process. 2. Period of Performance The expected duration of the evaluation is approximately 12 weeks from commencement to the delivery of the final report. ME/TS will respond to the content of the assessment with oral comments at the debriefing and will provide written comments within 14 working days of receipt of the draft. Annexes Attachment A: Results Framework for IWSMR. September 2013. Attachment A To achieve its objectives, the IWSMR program is: (a) customer- driven; (b) strategic, flexible and responsive to targets of opportunity; (c) results-oriented; and, (d) sustainable. Attachment B: Example of an  Illustrative and Abbreviated Design Matrix Evaluation Questions Suggested Methods and Data Sources for answering the question. 1. How are the relevant organizations involved adopting the tools and technology Macro‐level: On‐line survey of Top level management, mid‐level executives, young professionals in relevant organizations across all project countries regarding reactions to the tools and perceptions 74 provided by NASA to conduct their work?*   a) Which tools and technologies do these organizations perceive to be of value to for their own purposes? b) To what extent are appropriate staff of the organizations familiar with and proficient with the tools and technology? of value, and extent to which they are familiar with and proficient with particular tools and technologies they have been exposed to by the implementers.   Case studies (Jordan, Tunisia, Morocco): In person interviews of Top level management, mid‐level executives, young professionals in government ministries/agencies regarding reactions to the tools and perceptions of value, and extent to which they are familiar with and proficient with particular tools and technologies they have been exposed to by the implementers. Direct observation and documentary evidence (if possible) of staff regarding understanding of the tools and technology a plus.   c) To what extent have the tools been integrated into the work processes of the organizations? Macro‐level: On‐line survey of Top level management, mid‐level executives, young professionals in relevant organizations across all project countries regarding changes made within the organization to integrate particular tools and technologies they have been exposed to by the implementers.   Case studies (Jordan, Tunisia, Morocco): Interviews with ministry/agency staff (top‐level, mid‐level, young professionals) regarding changes made within the organization to integrate particular tools and technologies they have been exposed to by the implementers. Project and ministry documentary evidence and direct observation (if possible) regarding who is using and how a particular tool or technology is being used in government ministries/agencies.   2) What are the factors that lead to the adoption or non‐adoption of NASA tools and resources?   Case studies (Jordan, Tunisia, Morocco): Interviews with ministry/agency   (Top level management, mid‐level executives, young professionals) and project implementers   in relevant organizations regarding relevant organizational, bureaucratic, environmental, social, political factors that lead to adoption or non‐ adoption of tools and technologies.   a) What are the characteristics of the organizations that contribute to the adoption or prevention of these tools and technologies?   Case studies (Jordan, Tunisia, Morocco): Interviews with project partners and ministry/agency staff (top‐level, mid‐level, young professionals) regarding characteristics leading to or preventing adoption of the technology. May address organizational characteristics, staff capacity, leadership structure, etc. Documentary evidence to support project and ministry staff assertions. b) What are the internal factors that empower ministry staff to explore, adopt, and utilize NASA Case Studies: Interviews with ministry/agency staff (top‐level, mid‐ level, young professionals) regarding internal enabling factor such as motivation or buy‐in of leadership, staff motivation and buy‐in.   75 tools and technologies? Documentary evidence to support ministry staff assertions.   c) What are the external factors that lead to the integration of NASA tools and technologies into work processes for improved water resource management?    Case Studies: Interviews with ministry staff, project staff, stakeholders, beneficiaries regarding external demands for revision of current procedures or practices.   [1] USAID started the Jordan Operations and Maintenance Training (OMT), the Egypt Water Policy and Regulatory Reform (WPRR) and the Water and Wastewater Sector Support (WWSS) projects simultaneously in 2008 with the objective of initiating model operator certification and training programs in the MENA region. These are the first legally recognized operator certification and training programs in the region. 76 ANNEX 2. EVALUATION QUESTION TABLE – SUMMARY OF FINDINGS, CONCLUSIONS AND RECOMMENDATIONS No. Evaluation questions Synthesis of findings Conclusions Recommendations ACWUA’s Internal Capacity EQ1 To what extent has the implementation of the recommendations coming from the pre‐award survey resulted in a more sustainable and self‐financing organization? Since the recommendations are still being implemented, ACWUA’s sustainability cannot yet be determined. However, by improving its accounting and management systems, the impact is likely to be positive. Pre‐award survey seen by ACWUA as an effective tool to develop an understanding of the needed changes, to ensure ACWUA’s operational efficiency and effectiveness. Will serve to ensure ACWUA operates more transparently than it did before the survey. A follow‐up pre‐award survey should be considered to assess to what extent ACWUA has improved, prior to committing to additional funding. The evaluation did not have audit powers or an audit mandate, and therefore could only base its assessment of the pre‐award survey’s impact on circumstantial and secondary evidence ACWUA performance can be strengthened by introducing an internal monitoring and evaluation of performance results. According to the Strategic Business Plan 2015‐2019, an organizational unit (Financial and Administrative Affairs Unit) will be responsible for monitoring institutional performance compared to preset plans. In developing this unit’s functions, it should be ensured that they focus not just on outputs but on outcomes. EQ1a How has ACWUA addressed deficiencies identified? What have been the constraints? Recommendations in 6 areas covering 11 actions were made by the Pre‐award Survey and all have been or are in the process of being implemented by ACWUA. ACWUA took relatively quick action to address the recommendations, which suggests that ACWUA management has taken the recommendations seriously and sees benefits in improving its internal management and accounting practices. EQ1b How has ACWUA’s organizational and financial management changed pre and post IWSMR activity implementation? Processes appear to have contributed to more transparency and improved financial controls. Following IWSMR activity implementation management reports that it has become more aware of the need for instituting stronger internal management and accounting procedures.Following IWSMR activity implementation Because the evaluation did not have audit powers and could not review internal financial and management documents or spend time observing its accounting practices, it was not possible to assess changes in management. EQ1c How has ACWUA developed its capacity and mandate to collect and manage member funds? ACWUA does not appear to have taken action in this area, attributing non‐ payment and low contributions to the geopolitical situation. By adopting a more transparent accounting system, in line with the Pre‐ award Survey recommendations, which incorporates a breakdown of funding sources, ACWUA should be able to better manage member funds. However, it is not clear how and whether the collection of member funds has been affected by 77 No. Evaluation questions Synthesis of findings Conclusions Recommendations IWSMR activities. External factors seem to determine collection rates which appear to be beyond ACWUA’s control. EQ1d How is the Certification Training Information System (CIS) being utilized? Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? CIS has not been implemented at the time of writing. Still under development and available only to internal (ACWUA, CAQA) users. Too early to draw conclusions on the usage, effectiveness or impact of the CIS on member performance. ACWUA’s Sustainability EQ2 What is the level of evidence of ACWUA’s ability to adequately and effectively manage and grow a Water/Wastewater Management and Utilities Association post activity implementation? ACWUA’s potential impact is limited by several factors, including ACWUA’s resources relative to existing utility sector needs, the degree to which its activities and support address fundamental utility performance problems, and the fact that ACWUA is a young, expanding organization. Communication is an area where various key informants mentioned ACWUA can improve. ACWUA was not dependent on IWSMR support for the period from its start in 2009 through the end of 2013. There is reason to believe the organization will continue to grow and show prudent financial performance after the IWSMR activity ends. However, given ACWUA’s relatively recent establishment, significant changes in external conditions or management could yet have adverse impacts on its sustainability. It is recommended that USAID continue to provide support to ACWUA and associated institutions, through IWSMR‐type capacity building or similar activities, and possibly including direct budget support. Follow‐on assistance can best be used to focus on the following areas: Management capacity, Staff capacity, Communication to members, Cost of training, Expansion of individual training programs, and appropriately benchmarked performance indicators. EQ2a What mechanisms are in place to ensure ACWUA’s financial sustainability? Proactive in finding revenue‐generating activities, e.g. training programs conferences, etc. Membership fees present a challenge, but are a relatively small share of total revenue. Implementation of a new salary scale beginning with the new Business Plan (in 2015), with adjustments based on skills and qualifications, is expected to address this issue. EQ2b What mechanisms are in place to ensure ACWUA’s management sustainability? ACWUA fully embraced the Pre‐award Survey recommendations by implementing a series of financial and management controls. Suggests its performance has likely improved as a result, and signals that management takes internal management issues seriously. ACWUA’s reputation, name recognition, proactive orientation to identifying new revenue sources, and awareness in improving its internal management all point towards sustainability. EQ2c To what extent will ACWUA be able to maintain its current service provision? ACWUA is involved in many activities and staff is said to be overstretched. However, Will depend on ACWUA’s quality of the services provided, resources (how it 78 No. Evaluation questions Synthesis of findings Conclusions Recommendations The same quality of service provision? organizational restructuring and new hires is expected to address this issue. organizes staff, organizational capacity, etc.), demand for services, and reputation. Aside from internal capacity, which requires strengthening, indicators are positive. EQ2d What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? Strategic Business Plan, organizational restructuring, introduction of improved financial and management systems development of a new Financial Manual. Twinning Program has not been implemented yet. Strategic Business Plan 2015‐2019, the Pre‐Award Survey and development of Twinning Program material provide the foundation for sustainability. EQ2e Is ACWUA’s management willing to accommodate stakeholder needs? Generally receptive and accommodating but could be improved, specifically through basing training programs on assessment of utility company needs. ACWUA’s role in promoting awareness of challenges facing the sector is still evolving, but signs are encouraging that it will continue to grow in a positive direction. No key informant suggested that ACWUA was ineffective or that the concept of having such an organization was not worthwhile. External performance EQ3 To what extent has the 2015‐2019 Business Plan helped ACWUA in establishing itself as a regional water utilities power? Too early to determine. However, presentation of the Business Plan at Arab Water Week (2015) was well received and appears to have strengthened ACWUA’s credibility. Financial projections included in the Business Plan do not appear to play a strong role in guiding decisions. Greater attention to the financial tables in the Strategic Business Plan is recommended, along with the development of an Action Plan to accommodate potential fluctuations in revenue, as noted above. Lumpiness in revenue means that these projections may quickly become obsolete, with implications for ongoing expense items. ACWUA should develop a plan to reduce non‐payment of membership fees among utility companies, which have dropped significantly since the start of the Arab Spring. EQ3a How much success have the business planning exercises had in improving the reliability of ACWUA’s year to year revenue streams? ACWUA management do not appear overly familiar with the Business Plan’s cost and revenue projections. Revenue streams are likely to fluctuate for the foreseeable future, as they will depend, in part, on one‐off contracts and donor assistance, and less on steady membership fees. EQ3b Were lessons learned from implementing the Bridging Memo incorporated into the 2015‐2019 Business Plan? Bridging Memo reportedly did not have a direct influence on the Business Plan. NA EQ3c Has ACWUA's operational capacity changed as a result of the Business Plan? Too early to determine. However, past indicators and achievements suggest there should be positive impacts. ACWUA’s operational capacity has not yet changed, but is expected to increase in line with the organization restructuring envisaged in the Business Plan. 79 No. Evaluation questions Synthesis of findings Conclusions Recommendations Training and certification effectiveness EQ4 How effective have ACWUA’s training and certification programs been in building a cadre of qualified operators as well as buy‐in amongst the members? There is demand for TOT and ROCTP training and the courses organized to date have been positively rated by participants. Positive feedback from both trainees and utility companies on the training organized by ACWUA has strengthened the organization’s reputation among members who have benefited from it. The certification process is seen as worthwhile and valuable to both utilities and individuals. The task of assessing the impact of training and certification on utility company performance is likely to be complex. In the case of TOT it should take into account several key factors, including i) the degree to which internal training and knowledge sharing is valued and promoted within a company; ii) the amount of training which the TOT participant delivers and the number of staff who receive it; and iii) the added value of the training to the TOT participant (i.e. the more they benefited from the TOT, the greater the difference in their training ability should be). The broader impact of TOT will depend in part on how many utility persons are trained and what the multiplier effect is, i.e. the secondary impact of how well they train others. Costs for training increase when travel and accommodation for the trainees must be provided. This naturally limits the number of operators and other personnel which can be trained. Building a larger and stronger cadre of experienced trainers would reduce the need for participants to travel, bring down the total costs per trainee, and increase the number of operators which could potentially be trained. EQ4a How have lessons learned from the pilot operator certification courses been incorporated into the ACWUA Training Roll‐Out Plan? Do they address needs in the respondent countries? At the time of evaluation, ACWUA was in the process of finalizing a survey of the utilities in countries currently involved in ACWUA ROCTP (West Bank, Yemen, Lebanon, Jordan, Egypt) as part of IWSMR’s second assessment due to be delivered in June 2015. While the Training Roll‐out Plan, taking into account lessons learned from the pilot courses offered, could not be reviewed at the time of evaluation, ACWUA will need to be careful to avoid letting training quality lapse as it increases the scope, number of programs and countries targeted. EQ4b How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long‐term structure of the certification program? The certification program was already in place prior to the survey assessment. ACWUA then performed a survey of the utilities to identify their needs and priorities to incorporate into future certification programs. The impact of training on utility performance was not assessed, but the fact that it rates highly increases the chances that its impact will be positive. EQ4c To what extent do member country utilities value the certification of a Jordan organization (CAQA)? Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? Based on key informant interviews in Egypt and online responses, CAQA certification is valued outside Jordan. Its endorsement by the Arab League at Arab Water Week 2015 is a key indicator and contributor toward its acceptance. Based on feedback from Egypt, receiving certification from CAQA is clearly valued, its national mandate for Jordan notwithstanding. Twinning program effectiveness EQ5 What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been Too early to determine – program still in the planning stage. NA NA 80 No. Evaluation questions Synthesis of findings Conclusions Recommendations successful? OR What are the advantages and disadvantages of establishing twinning programs? EQ5a To what extent has information and peer support between water and wastewater utilities been established through the regional twinning program? Too early to determine – program still in the planning stage. NA NA EQ5b What lessons have been learned from the two pilot virtual twinning/exchange activities run by IWSMR? Too early to determine – program still in the planning stage. NA NA EQ5c What has been the nature of exchange and collaboration amongst the twinning partnerships? Has it helped in building staff capacity? Too early to determine – program still in the planning stage. NA NA EQ5d How effective are the existing indicators in measuring effective management of twinning partnerships? Too early to determine – program still in the planning stage. NA NA 81 ANNEX 3. ONLINE SURVEY FOR ROCTP PARTICIPANTS: QUESTIONS AND RESPONSES Total responses: 13 (out of 40 training participants contacted) Questions Response categories Yes No 1. Did you participate in Regional Operator Certification and Training Program (ROCTP) training organized by ACWUA? 8 5 Avg. 2. What is your age? 36.5 Male Female 3. What is your gender? 10 3 Avg. 4. How many years of experience working in the sector do you have? 8.0 Open‐ended responses: 5. What is your position? − Head of Design Directorates − Head of Water Distribution Section − Responsible for water distribution technicians − Engineer (2 responses) − Head of Hydrology Section" − CEO − Labs − Head of Section/Water Supply − Engineer in Charge of Operating Several Wastewater treatment Systems & Research assistant − Director of Specifications and Standards Department 2014 2015 6. When did you take the training? 8 5 Yes No 7. Did you complete the training? 13 Avg. 8. “Taking the ROCTP training was useful for my work” * 4.3 Yes No 9. Did you take the exam for the training? 13 Yes No 10. Did you pass the exam, which allows you to obtain the certificate? 13 Open‐ended responses: 11. What impact, if any, has the Certificate had on your work? − Unfortunately, I haven't received the certificate till now. The sponsoring body is responsible for this dereliction of 82 duty.   − Teaching colleagues the reason behind following work procedures   − It made me work harder. − Nothing for the time being since I moved to work under different section (Wells, springs and water resources) − Nothing new till now − Qualified to better level − Is an important recognition for the knowledge gained , and that classifies who has approved the exam as my particular case. Since without such program in the past I have been practicing this profession without an official specific certification. − It gives me good opportunities to enhance my work if I want to. − Not much yet Open‐ended responses: 12. What other type of training would benefit you most in your work? − Studies/design for water and wastewater networks − Practical part − Taking Level 2 in Water distribution. − GIS   − Hoping to participate in other training  courses and to visit plants with high efficiency − Environmental pollution impact − I am interested in the operators certification program level two, the one that follows the first received training   . − Any other training that concerns on conducting general specifications of the different types of treatment plants that PWA needs. Yes No No answer 13. Are you familiar with ACWUA activities? 5 1 7 Avg. 14. “I see ACWUA as an effective organization for strengthening the water sector in my country”* 4.5 Open‐ended responses: 15. Do you have any other comments? − Until today, we don't know what are our scores. We didn't receive the certificates although we've been told that we should have everything within one week − I would like to thank USAID & ACWUA for their continuous help & support. Looking forward for attending more training sessions related to the water sector and waste water sector.    − More effort should be invested in translation (to Arabic) to get along with the Lebanese standards − None − Thanks, for the big coordination and affords put to increase the capacity building of people working in the 83 sector . − The same staff that took the first level training should continue in the next levels so that to take the complete benefit. − Nope * On a scale of 1 to 5, where 1 means completely disagree and 5 means completely agree, please respond to the relevant statement.    84 ANNEX 4. LIST OF KEY INFORMANTS INTERVIEWED USAID 1. Ms. Alexandra Hadzi‐Vidanovic, Regional Environmental Advisor, Office of Middle East Programs 2. Amena Chenzaie, Monitoring and Evaluations Specialist, Office of Middle East Programs Chemonics/IWSMR 3. Mr. Kim Shelsby, Director, Middle East 4. Ms. Elyse Wesbey, Manager, Middle East 5. Mr. Othman Al‐Kurdi, Chief of Party, 6. Eng. Hisham Rawashdeh, Deputy Chief of Party (DCoP) ACWUA 7. Eng. Khaldon Khashman, Secretary General, ACWUA 8. Dr. Ahmad AlAzzam, Director of HR & Training, ACWUA 9. Eng. Mustafa Nasereddin, Director of Programs & Technical Services, ACWUA 10. Ms. Dana Daqqaq, Communications Director, ACWUA 11. Ms. Sarah Abu Salim, Training Coordinator, ACWUA CAQA 12. Mr. Mohammad Khair Irshaid, Director, CAQA 13. Mr. Emad Malkawi, CAQA Utility companies – Jordan 14. Mr. Mohammad Salim, Manager, Al Salt Water Co, WAJ 15. Ms. Asmaa Wahadneh, Head of Tendering Section, WAJ (Water Authority of Jordan) 16. Eng. Bassam Saleh, Assist. Secretary General, WAJ Disi Water Conveyance Project 17. Ms. Majeda Al Zoubi, Quality Control & Assurance Manager, Miyahuna 18. Mr. Zeyad Shawaqfeh, Manager, Zarqa Water Co/Miyahuna MC 19. Eng. Naem Saleh, CEO, AWC ‐Aqaba Water Company‐  20. Eng. Ahmad Abu Alsoud, IT Manager, AWC ‐Aqaba Water Company‐  21. Eng. Ahmad Abu Alsoud, Networks Engineer, AWC ‐Aqaba Water Company‐  22. Mr. Anas Al Khatib, Networks Engineer, AWC ‐Aqaba Water Company‐  23. Mr. Samir Ma'ateh, Utility Manager, Karak Water Company (WAJ) 24. Mr. Ibrahim Al Horah, O&M Manager, Karak Water Company (WAJ) 25. Mr. Ismaeel Assaf, O&M, Karak Water Company (WAJ) 26. Mr. Mohammad Al Basseni, O&M, Karak Water Company (WAJ) 27. Mr. Salman Al Gmar, Customer Service Manager, Karak Water Company (WAJ) 28. Mr. Ahmad Ma'ateh, Technical Service Manager, Karak Water Company (WAJ) 29. Mr. Nasir Ja'afreh, O&M, Karak Water Company (WAJ) 30. Mr. Mohammad Ja'afreh, Water Distribution, Karak Water Company (WAJ) 31. Mr. Mustafa Zannoun, Utility Manager, Tafila Water Company (WAJ) 32. Mr. Mousa, Customer Service Manager, Tafila Water Company (WAJ) 85 Development partners 33. Dr. Thomas Petermann, Regional Department MENA / Project Manager: ACWUA WANT, GIZ 34. Mr. Hani Kurdi, Deputy Chief Program Officer, JICA Jordan Office 35. Ms. Muna Bargouth, Regional Programme Manager, SIDA Other 36. Eng. Iyad Dahiyat, PMU Director, WAJ PMU    37. Mr. Udo Kachel, Chief Technical Advisor, WAJ PMU (formerly GIZ) 38. Mr. Tareq Zuriekat, CEO (Chief Executive Officer), Engicon, Amman 39. Mr. Firas Matar, CEO (Chief Executive Officer), O&M Engicon, Amman 40. Dr. AbdelKawi Khalifa, Lecturer, University of Ain Shams, Cairo Egypt stakeholders 41. Dr. Eng. Subhi Abd Al‐Kader, Egyptian Water and Waste water Regulatory Agency (EWRA),   42. Dr. Mohamed Abdel‐Salam, Professor of Structural Engineering, Housing & Building National Research Centre (HBRC) 43. Dr Omar Hassan, Housing & Building National Research Centre (HBRC) Utility companies – Egypt 44. Eng. Ahmed Gaber, Chairperson & Delegated Member, Alexandria Water Company 45. Ms. Yosr Salah El‐Din, Communications Manager, Alexandria Water Company 46. Eng.  Nasra Nour Eldin, Head of Technical Support Sector, Alexandria 47. Mr. AbdulRahman, Representative of Egyptian Water Utilities, Greater Cairo Water Company 48. Mr. Hamdi Shattah, Production and Distribution, Greater Cairo Water Company 49. Eng. Magdy Mabrouk Ali, Head of Development Sector, HR and Administration, Greater Cairo Water Company 50. Eng. Ibrahim Khalid, Chairman, Menia Company, Greater Cairo Water Company Other country stakeholders (telephone interviews) 51. Mr. Ziyad Al Fuqaha, Training Manager, Palestinian Water Authority 52. Eng. Emil Abdo, Jerusalem Water Enterprise, Ramallah/Al‐Bireh Governorate 53. Mr. Ayed Hussein Qathem, Ministry of  Water Resources Baghdad Iraq 54. Adel Salim Yasin, wastewater operator, West Bank (Palestine) 55. Hala Mohammad Hussein, wastewater operator, West Bank (Palestine) 86 ANNEX 5. KEY INFORMANT INTERVIEW QUESTIONNAIRES Note: All rating questions (signified by the word “RATING” preceding the statement) are on a scale of 1 to 5, where 1 = completely disagree and 5 = completely agree. Contractor (IWSMR) 1. What do you see as the key accomplishments to date? 2. What key challenges to meeting the IWSMR objectives (by component) have you faced / continue to face? How have they been addressed?   3. What are the weaknesses of the component activities, in terms of design / implementation?   4. What lessons have been learned to date? 5. What are the key factors contributing to activities success?   6. If the activities were extended, what would you suggest doing differently? (i.e. adjustments ME/TS should take into consideration for design of future activities in this area?) 7. What is your assessment of ACWUA performance?   8. In your estimation, to what degree has ACWUA implemented NUPAS recommendations?   9. (Review SOW Evaluation Questions EQ1 – EQ5) ACWUA members (institutional beneficiaries) General information 1. What is your role (as an individual) within the department/utility?   2. What are the main responsibilities of your department?   3. When did your organization join ACWUA?   4. What kind of services have you received from ACWUA?   5. RATING: We are completely satisfied with the services received from ACWUA.   6. How have those services affected your organization’s operations?   7. What are the greatest benefits your organization has seen since joining ACWUA?   8. How do you expect ACWUA to help your organization in the future?   87 9. RATING: ACWUA is responsive to my organization’s needs. (EQ2e). Please explain your answer. 10. RATING: ACWUA is an effective organization. (EQ2e). Please explain your answer. 11. What benefits would you like to see from ACWUA?   Training and certification 12. What type of training did your staff participate in? (TOT, ROPTC) 13. How were candidates for training selected?   14. RATING: ACWUA’s training and certification program has been effective in building a cadre of qualified operators at my organization? (EQ4). Please explain your answer. 15. RATING: ACWUA’s training and certification program has been useful to my organization’s operations? (EQ4). Please explain your answer. 16. How could the training and certification program become more useful to your organization?   17. Has the CIS improved performance or services to your organization? (EQ1d)   18. What are your organization’s impressions on using the CIS? Is it clear and easy to use? How comfortable are the relevant staff working with the CIS? (EQ1d) Twinning program [if applicable] Please describe your organization’s plans and expectations for engaging in the twinning program.   ACWUA staff Note – these questions will be addressed over multiple meetings with relevant ACWUA staff General questions 1. What are the main benefits you believe ACWUA has received from IWSMR?   2. What are ACWUA’s main accomplishments to date?   3. What are ACWUA’s main challenges?   4. What are ACWUA’s priorities?   5. What lessons have been learned from IWSMR support?   6. To what degree, if at all, does ACWUA depend on IWSMR support? Other donor support?   88 7. What other donors does ACWUA receive support from? What kind of support is it?   8. Should IWSMR support, what do you expect will change, if anything?   9. What are ACWUA’s biggest resource needs (financial, human, management, etc.)?   10. How effective was the involvement of external partners (if any) at different stages of the activities? Management effectiveness and financial sustainability 11. How has ACWUA’s organizational and financial management changed following IWSMR activity implementation? (EQ1) 12. To what extent has ACWUA implemented recommendations from the Pre‐Award Survey (NUPAS)? What specific areas? What specific institutional and procedural practices changed? What challenges and constraints has ACWUA faced in implementing those changes? 13. What timeframe does ACWUA envision for the IWSMR activity implementation to progressively and fully impact the organization toward growth and sustainability?   14. Between the pre‐award survey in July 2014 and when the ACWUA Financial Manual was updated in February 2015, what significant changes, if any, did ACWUA experience vis‐à‐vis financial and organizational management? 15. How does ACWUA monitor progress and performance with the new Financial Manual? 16. Although a short time has elapsed since the implementation of key tasks of the IWSMR, how does ACWUA view and characterize the contributions to and befits from IWSMR for ACWUA to achieve its strategic objectives?   17. How does ACWUA see IWSMR contributions for remaining the prevailing regional organization in terms of sustaining membership and providing an expanded role in the region?   18. The ACWUA Business Plan has utilized SWOT to capture potential risks that the organization may face in the next 5 years and beyond. Has a formal risk analysis been performed to further identify and understand the risks that the organization is facing vis‐à‐vis sustainability? How do activities such as ISWMR can help with mitigating the potentials risk? Which areas receive other funding and capacity building? 19. The 2015‐2019 Business Plan has identified 6 types of risks that can threaten the implementation of the Financial Plan, notably low demand for ACWUA’s training programs. To what extent these risks have been incorporated into the financial forecast mainly the revenues? 89 What mechanisms and tools are in place to ensure the effectiveness of implementation of the mitigation options?   20. To what extent does ACWUA’s new organizational structure help with ensuring its management sustainability, given the fact that the bulk of expenses of the organization go toward employee salaries? Business plan 21. Please describe the development of the 2015‐2019 Business Plan.   22. How useful has 2015‐2019 Business Plan been?   23. To what extent has the 2015‐2019 Business Plan helped ACWUA establish itself as a regional water utilities power? (EQ3) 24. How successful was the business planning exercises in improving the reliability of ACWUA’s year to year revenue streams? (EQ3a) 25. Were lessons learned from implementing the Bridging Memo incorporated into the 2015‐2019 Business Plan? (EQ3b) 26. Has ACWUA's operational capacity changed as a result of the Business Plan? (EQ3c) In what way?   27. Is it fair to assume that the February 2015 version of the ACWUA Financial Manual reflects most, if not all, recommendations from the July 2014 Pre‐Award Survey? If so, how does ACWUA envision these recommendations (e.g., Bank, Collections and Deposits, Petty Cash, and Disbursements policies) to help with a more financially sustainable ACWUS institution?     28. Did ACWUA go through significant institutional changes and re‐arrangements to incorporate recommendations from the pre‐award survey? 29. How often does ACWUA plan to update the Financial Manual to, for example, address emerging and unanticipated financial challenges that may or may not directly relate to external and internal challenges (as highlighted in the 2015‐2019 ACWUA Business Plan)? (Ref. the SWOT bank reconciliation procedure). 30. In what specific areas has the ACWUA Financial Plan influenced the 2015‐19 ACWUA Business plan? What specific policies and procedures, other than what has already been highlighted in the Financial Plan, guided the development of the Business Plan?    31. How has the manual incorporated the recommendations from the pre‐award survey to include, but not be limited to, formulation of policies and procedures and documents for finance and accounting?   90 32. What new and additional policies, procedures and standards were developed to address the deficiencies identified in the pre‐award survey? (EQ1a) 33. What if any logical framework has been used? 34. Is ACWUA utilizing all currently available resources? What additional resources does it plan to use? 35. Were the Financial Manual and follow‐up policies and procedures (if any) revised, updated and upgraded to cater to the needs of IWSMR financial and institutional audits?    36. Have there been any financial audits of how IWSMR funds been disbursed and managed? If so, what guided the audits? 37. Does ACWUA develop a monthly operational budget? What is the organizational procedure? What is guiding the budget development (e.g., policies and procedures) 38. The business plan (page 37) highlights projects, programs and services, including revenues from training programs, as the main source of revenue over the 5‐yaer planning horizon. What rational has been used in envisioning the demand for future trainings?   39. Is it fair to assume that trainings and capacity building will contribute to operational efficiency of utilities, leading to operational cost reduction and cost effectiveness, thus contributing to operational sustainability of the service providers? 40. The Business Plan (page 38, Table 6‐3) suggests that with the exception of year 2015, the total forecasted revenues barely exceeds the forecasted expenses and, in 2019, the forecasted expenses exceed forecasted revenues. How much confidence and reliability and risk analysis have been incorporated into these forecasts and what mitigation options should ACWUA take if it does not reach its targeted revenues? Service provision 41. To what extent will ACWUA be able to maintain its current service provision? The same quality of service provision? 42. What specifics measures does ACWUA have in place to ensure quality service? 43. Are there accountability measures and or monitoring and evaluation mechanisms other than what have been outlined in the Business Plan (ages 43‐44) 44. Does ACWUA plan or envision organizational reform to maintain and enhance the provision of its services? If so, in what direction? 91 45. What aspects of the IWSMR program are likely to be sustainable? Are there any gaps within the sustainability structure of the twinning program? 46. The IWSMR Activity rests upon 3 pillars: 1) Build ACWUA Capacity; 2) Operation Certification; and 3) Twining and Leveraging Best Practices. Which of these components and their respective sub‐components have had the most effect on the organization vis‐à‐vis sustainability? How has ACWUA measured the effectiveness of either or all of these components? 47. Is ACWUA’s management willing to accommodate stakeholder needs? (EQ2e) 48. What mechanisms are in place (and has ACWUA used them) to seek consultation and feedback from stakeholders in their short to long‐term operations and strategies?   49. Has Stakeholder consultations and participatory and consumer‐based decision been a challenge, constraint, enhancement, or need in organizational sustainability? Funding / Membership fees 50. What are ACWUA’s revenue sources?   51. How are membership fees calculated?   52. How do you encourage members to pay?   53. What are the reasons members give for non‐payment? Are they credible?   54. How has ACWUA developed its capacity and mandate to collect and manage member funds? (EQ1) 55. What has been the impact of reduced membership fee collection on ACWUA operations, if any?   56. Figure 2‐2 of the Strategic Business Plan suggests that the bulk of ACWUA’s revenue in 2013 drastically increased from that of the previous 4 years, attributed primarily to training and consulting fees. Is this a reflection of ACWUA’s recognized technical capacity? If so, will training and consulting activities remain as the main source of revenue for the organization? 57. Figure 2‐3 suggests that although revenues between 2009 and 2013 grew sustainably, however, at times expenses exceeded revenues, attributed primarily to salaries and conference? How has ISWMR helped capturing this issue in the 2015‐19 Strategic Plan to mitigate potential impacts? 58. Page 16 of the 2015 Strategic Plan suggests that “ACWUA depends on available revenue streams comprising mainly training courses and O&M certification and training programs and consulting assignments, in addition to membership fees and grants from donor agencies and Arab governments...” Did ISWMR activities help in identifying these sources as main sources of revenue to ensure sustainability? 92 59. How confident are you that the Strategic Plan is the right tool and right framework to guide the organization and ensure its sustainability?   Training 60. Please describe the design and implementation of the training and certification program. 61. How are trainees selected?   62. How are persons selected for training?   63. Was training was targeted at the right people? How can this be known?   64. What tools and methods if any and other than training assessments ACWUA has used to measure the effectives of the certification programs? 65. Where does ACWUA see the short to long‐term impacts and effects of the certification programs in operation efficiency? 66. In which areas of the utility operations, ACWUA believes in the benefits of the certification programs 67. What key measurable objectives have driven the ACWUA’s training certification programs? 68. How does ACWUA characterize the effectiveness of the Training of the Trainers (TOT) programs?   69. Did ACWUA adopt clear tools/indicators for measuring training effectiveness and measured these indicators? 70. Does ACWUA believe that the TOT objectives have all been achieved? 71. How has ACWUA incorporated findings from the survey assessments of the pilot training programs into the long‐term structure of the certification program? (EQ4b) 72. How have lessons learned from the pilot operator certification courses been incorporated into the ACWUA Training Roll‐Out Plan? Do they address needs in the respondent countries? (EQ4a) 73. To what extent do member country utilities value the certification of a Jordan organization (CAQA)?  Are they encouraging this certification for other staff? What has been the utility of the CAQA amongst member countries? (EQ4c) Certification Training Information System 74. How is the CIS being utilized? (EQ1) 93 75. Is CIS fully operational? 76. Has the CIS improved performance or services to members? What are members’ impressions on using the CIS? Is it clear and easy to use? (EQ1d) 77. How comfortable are the relevant staff working with the CIS? (EQ1d) 78. How has ACWUA verified/checked the accuracy of data entered into the CIS? QC? 79. What level of organizational management has so far been trained on CIS 80. What types of organizational decision making the CIS has facilitated and supported? 81. How has the CIS helped ACWUA to design future training activities? 82. Which of the attributes of the CIS have been used the most and found to be most useful and helpful: a) Sharing training and certification information b) Searchable repository for training records and certifications c) Has it served as a knowledge portal, allowing peer‐to‐peer interaction; and d) Has it served as a database of certified operators and trainers Twinning program (note – some questions may not be applicable since twinning has not begun yet) 83. What have been the advantages and disadvantages of establishing twinning programs among MENA utilities? To what extent have twinning programs been successful? What are the advantages and disadvantages of establishing twinning programs? (EQ5) 84. To what extent has information and peer support between water and wastewater utilities been established through the regional twinning program? 85. What tools and methods and means have   ACWUA and IWSMR used to collect data and information to support planning foe the Twinning Programs?   86. What lessons have been learned from the two pilot virtual twinning/exchange activities run by IWSMR? 87. What has been the nature of exchange and collaboration amongst the twinning partnerships? Has it helped in building staff capacity? 94 88. How effective are the existing indicators in measuring effective management of twinning partnerships? 89. How effective was the communication and knowledge transfer between the twinning counterparts? Others – government officials, experts, sector stakeholders (external stakeholders) ● Questions will be tailored to the specific individual/institution. They will relate primarily to how ACWUA is perceived from outside, its role, influence and importance for the sector.      95 ANNEX 6. FOCUS GROUP DISCUSSION GUIDE The following questions are to be used both for key informant interviews and focus group discussions. The target group is utility staff who received ROCTP and TOT training.   1. When did the training take place?   2. What was the duration of the training?   3. What topics were studied?   4. RATING: The amount of training sufficient/insufficient? Please explain your answer.   5. Was the mix of theory and practice well‐balanced?   6. RATING: The technical level of the training was too complicated. Please explain your answer. 7. RATING: The technical level of the training was too basic. Please explain your answer. 8. RATING: The training was useful? Please explain your answer. 9. Were the training methods appropriate?   10. RATING: The training has had a major impact on my work? Please explain your answer. 11. What has been the impact of the training on how the utility operates?   12. What would you recommend to improve training in the future?      96 ANNEX 7. BIBLIOGRAPHY OF DOCUMENTS AND SOURCES REVIEWED ACWUA Charter and Bylaws. (nd). ACWUA website. http://www.acwua.org/node/1    ACWUA. (2010). Business Plan 2010 – 2014. ACWUA. (2014). ACWUA Water Operators Certification Program. Third Edition. December 2014 ACWUA. (2014. 5 Years Achievements.   ACWUA. (2015). Strategic Business Plan (2015 – 2019). ACWUA. (nd). Certificate Information System. Presentation. Alexandria Water Company. Brochure.   Chemonics International. (2013). Work Plan ‐ Year 1. November 2013 IWSMR. (2013). Year 1 Workplan, November 2013 IWSMR. Quarterly Reports #1 ‐ #6.   PriceWaterhouseCoopers. (2014). The Improving Water and Sanitation Services in the MENA Region (IWSMR). Non‐U.S. Organization Pre‐award Survey (NUPAS) For Arab Countries Water Utilities Association (ACWUA). July 2014. PriceWaterhouseCoopers. (2015). The Improving Water and Sanitation Services in the MENA Region (IWSMR). Finance Manual updates. February 2015.USAID. (2011). Water and Development Strategy 2013‐2018. USAID. (2012). Operations and Maintenance Training Project. Final Report. October 2008‐November 2012. USAID. (2015). Task Order: AID‐263‐TO‐13‐00002 IQC EPP‐I‐00‐04‐00020; Integrated Water and Coastal Resources Management (Water II IQC). USAID. Water and Sanitation. USAID website. Available at: http://www.usaid.gov/what‐we‐do/water‐ and‐sanitation   World Bank. (2010). Water Brief. Available at: http://go.worldbank.org/0Z2C3PFS70    97 ANNEX 8. ANNEX PRE-AWARD SURVEY (NUPAS) RECOMMENDATIONS AND ACTIONS # NUPAS Recommendation Deliverable Complete In progress/ongoing 1 Bank Reconciliation Policies‐Document Policies and procedures to stipulate preparation and review of monthly bank reconciliations Create ACWUA Finance Manual ACWUA prepares bank reconciliations on a monthly basis 2 Payments‐Segregation of Duties‐formalize policy for cash and check disbursements and ensure duties are split Hire an Accountant for checks and balances of duties; create   ACWUA Finance Manual Duties were segregated in November, 2014 3 Accounting Cycle‐ Segregation of Duties‐  Document accounting delegation of authorities formally Hire an Accountant for checks and balances of duties: create ACWUA Finance Manual Done‐new accountant was hired in October 2014 4 Procurement policies and procedures‐update Tendering processes and other general policies to be more specific Update ACWUA's Procurement Policy Manual Updates to ACWUA’s procurement policies have been completed and needs to be integrated in the financial manual 5 Anti‐Terrorist Checking procedures‐check all suppliers on EPLS prior to procurement Update ACWUA's Procurement Policy Manual with requirement for EPLS/SAM checks OFAC and SAM checking has been added as a requirement in ACWUA’s procurement policy completed in November 2014. ACWUA’s Financial Director is completing SAM and OFAC checks on a regular basis, as needed. SAM Code for ACWUA is still not generated by the system due to  failure in CAGE Validation. We have contacted NSPA‐ NATO, DLA Center, and still following up. We may need help in this issue. 6 Staff Time Management‐  Create timesheets for ACWUA staff ACWUA Timesheet Template Timesheet has been created in November 2014 and starting 1 January 2015, all ACWUA employees are charging their time. 7 Payroll System‐Create Salary Scale for each ACWUA position ACWUA C14 The new salary scale was approved by the ACWUA Board of Directors in January, 2015 98 # NUPAS Recommendation Deliverable Complete In progress/ongoing 8 Budget vs. Actual Analysis‐  need policy on doing regular variance budget analysis of programs and operating financial data Create ACWUA Finance Manual with policy on budget analysis Done on annual basis as included in the business plan 9 Allowable and Unallowable Costs‐create written policy on spending functions Create ACWUA Finance Manual with clear policy on allowable/unallowable costs The finance manual now identifies cost principles including basis for allowability of costs. 10 Direct and Indirect Costs‐ are pooled in the same account. Create policy in place to classify and differentiate costs Update Accounting system to differentiate between indirects/directs; Develop ACWUA Finance Manual with new cost principles New cost centers created in the business plan to be applied in ACWUA’s accounting system 11 Chart of Accounts, GL and Financial Statements‐ Transactions should not be posted every 3 days, same person should not be entering data with no 2nd review; all revenue should not be recorded to one account Post transactions daily; create separate cost accounts for revenue; hire an accountant to support segregation of duties Done 12 Financial Records Management‐ Create emergency backup for electronic records Purchase C18 to backup ACWUA data All IT back‐up recommendations were completed through IWSMR/ACWUA IT supplier,“Tira for IT” Services. 13 Sources of Funding‐  Create written policies on funding and revenue; membership fee collection; create plan for maintaining funding Produce ACWUA Business Plan 2015‐2019 to document and plan sources of funding Done and included in the business plan 14 Audit and Review of Financial Statements‐ External annual auditors should issue management letter points (MPL), internal control systems should be reviewed Create ACWUA Finance Manual to include new policy of ensuring future audits include review of internal control systems and MLP Audit and review of financial system is under preparations to be released by end of June 2015 99 # NUPAS Recommendation Deliverable Complete In progress/ongoing 15 Project Management Capacity‐ACWUA should produce a Project Management Manual to guide project implementation Create Project Management Manual PMM is under preparation with the assistance of Chemonics PMM consultant approved by USAID. Development of PMM is ongoing 16 Projected Cash Flow Statements should be prepared to predict net cash flow Create cash flow statements on a quarterly basis, based on 5‐year projections. Achieve BOD approval. Quarterly cash flow projections have been prepared and approved by ACWUA’s BOD in January 2015    ANNE Name Title Organizat Evaluatio Evaluatio (contract USAID Pro project name(s) applicable I have rea interest t If yes an the follow Real or poten but are not lim 1. Close fami the USAID project(s) organizatio evaluated. 2. Financial significant implement are being e evaluation 3. Current or though ind being eval project des project. 4. Current or seeking em unit manag implement are being e 5. Current or organizatio competitor organizatio evaluated. 6. Preconceiv groups, org particular evaluated I certify (1) this disclos companies proprietary Signature Date   EX 9: DISC tion n Position? n Award or other instr oject(s) Evalu name(s), and award e) al or potentia o disclose. swered abov wing facts: ntial conflicts of in mited to: ily member who is D operating unit ma being evaluated or on(s) whose projec interest that is dire t though indirect, i ting organization(s evaluated or in the n. r previous direct o direct experience w uated, including in sign or previous it r previous work exp mployment with the ging the evaluation ting organization(s evaluated. r previous work exp on that may be see r with the impleme on(s) whose projec ved ideas toward i rganizations, or obj projects and orga that could bias the ) that I have co sure form prom s, then I agree t y and refrain fr e CLOSUR d Numbe rument) uated (Include implemente number(s), i al conflicts o ve, I disclose terest may include an employee of anaging the r the implementing ct(s) are being ect, or is in the s) whose projects e outcome of the r significant with the project(s) nvolvement in the terations of the xperience or e USAID operating n or the s) whose project(s) xperience with an en as an industry enting ct(s) are being individuals, bjectives of the anizations being e evaluation. ompleted this d mptly if relevan to protect the rom using the July 10, 2 RE OF AN Elly Preot Chief of P Internatio       Team er AID‐OAA‐ e er if Improving North Af Internatio 00003 of       Yes     e e, g g ) disclosure form nt circumstanc ir information information fo 015 NY CONF tle Party onal Developm m Leader         ‐I‐12‐00042/A g Water and frica Region onal, Inc., Co       No   m fully and to t ces change. If I from unautho or any purpose FLICTS O ment Group,   Team mem AID‐OAA‐TO‐1 Sanitation S n (IWSMR), ontract Task the best of my gain access to rized use or d e other than th OF INTER LLC (IDG) mber 14‐00046 ervices in the implemente Order; Awar y ability and (2) proprietary in isclosure for a at for which it REST e Middle Eas d by Chem rd No. 263‐T ) that I will upd nformation of o as long as it rem t was furnished 100 st and monics TO‐13‐ date other mains d. Name Nils Junge Title Team Leader Organization International Development Group, LLC (IDG) Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-12-00042/AID-OAA-TO-14-00046 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR), implemented by Chemonics International, Inc., Contract Task Order; Award No. 263-TO-13- 00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 10, 2015 Name Ehab Quran Title Water Specialist Organization Al Jidara Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-12-00042/AID-OAA-TO-14-00046 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Improving Water and Sanitation Services in the Middle East and North Africa Region (IWSMR), implemented by Chemonics International, Inc., Contract Task Order; Award No. 263-TO-13- 00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 28, 2015 Middle East Bureau/Technical Services U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523