USAID Pakistan February 2012 This is an internal report for the United States Agency for International Development. It was written by Heather Skilling (USAID/E3), Anthony Kolb (USAID/E3) and Raouf Youssef (Global Development Associates, LLC). EVALUATION Egypt Capacity Building and Policy Support in the Water and Wastewater Sector Performance Evaluation Report EGYPT CAPACITY BUILDING AND POLICY SUPPORT IN THE WATER AND WASTEWATER SECTOR PERFORMANCE EVALUATION REPORT WATER AND WASTEWATER SECTOR SUPPORT PROGRAM (WWSS) USAID CONTRACT # EPP-1-00-04-00020-00, ORDER NO. 3 AND WATER POLICY AND REGULATORY REFORM PROJECT (WPRR) USAID CONTRACT # EPP-1-00-04-00020-00, ORDER NO. 2 FEBRUARY 2012 This is an internal report for the United States Agency for International Development. It was written by Heather Skilling (USAID/E3), Anthony Kolb (USAID/E3) and Raouf Youssef (Global Development Associates, LLC). CAPACITY BUILDING AND POLICY SUPPORT IN THE WATER AND WASTEWATER SECTOR PERFORMANCE EVALUATION REPORT Global Development Associates, LLC 8702 Esquire Crossing Lane Vienna, VA 22180 Contracted under Task Order No. AID-263-O-12-00008 DISCLAIMER The author's views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE iii PERFORMANCE EVALUATION REPORT TABLE OF CONTENTS ACKNOWLEDGEMENTS..............................................................................................................vi ACRONYMS .................................................................................................................................. vii EXECUTIVE SUMMARY .................................................................................................................ix xi................................................................................................................................... الملخص التنفيذي 1. INTRODUCTION....................................................................................................................... 1 2. EVALUATION OVERVIEW....................................................................................................... 3 EVALUATION OBJECTIVES....................................................................................................... 3 EVALUATION METHODOLOGY............................................................................................. 4 EVALUATION LIMITATIONS.................................................................................................... 5 3. USAID/EGYPT WATER AND WASTEWATER SECTOR TECHNICAL ASSISTANCE ......... 6 PROJECT BACKGROUND........................................................................................................ 6 WATER AND WASTEWATER REFORM IN EGYPT ............................................................... 7 CURRENT SECTOR SUPPORT OBJECTIVES ........................................................................... 8 4. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM (WWSS) ......................... 11 TRACKING WWSS RESULTS.................................................................................................. 11 ACHIEVEMENT OF EXPECTED RESULTS BY COMPONENT.............................................. 14 ASSESSMENT OF WWSS RESULTS USING THE PMP ........................................................... 28 SUMMARY OF WWSS RELATED FINDINGS......................................................................... 34 5. WATER POLICY AND REGULATORY REFORM PROJECT (WPRR)................................... 35 TRACKING WPRR RESULTS................................................................................................... 35 ACHIEVEMENT OF EXPECTED RESULTS BY COMPONENTS............................................ 36 ASSESSMENT OF WPRR RESULTS USING THE PMP ............................................................ 47 SUMMARY OF WPRR RELATED FINDINGS.......................................................................... 52 6. CROSS-CUTTING ISSUES........................................................................................................ 54 SUMMARY OF FINDINGS RELATED TO CROSS-CUTTING ISSUES................................... 54 7. CONCLUSIONS AND RECOMMENDATIONS ..................................................................... 56 KEY CONCLUSIONS ON WWSS PROJECT ......................................................................... 56 EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE iv PERFORMANCE EVALUATION REPORT KEY CONCLUSIONS ON WPRR PROJECT .......................................................................... 57 CROSS-CUTTING CONCLUSIONS....................................................................................... 57 RECOMMENDATIONS FOR OPTION YEAR ........................................................................ 58 RECOMMENDATIONS FOR FUTURE PROGRAMMING...................................................... 59 ANNEX A: EVALUATION SCOPE OF WORK......................................................................... 62 ANNEX B: CAPABILITIES OF THE EVALUATION TEAM........................................................ 69 ANNEX C: LIST OF PEOPLE CONTACTED............................................................................. 70 ANNEX D: LIST OF DOCUMENTS REVIEWED....................................................................... 73 ANNEX E: LIST OF EVALUATION QUESTIONS...................................................................... 74 ANNEX F: EVALUATION TEAM SCHEDULE ........................................................................... 75 ANNEX G: LEVEL OF PERFORMANCE – COMPARATIVE ANALYSIS ................................... 76 ANNEX H: DISCLOSURE OF REAL OR POTENTIAL CONFLICTS OF INTEREST FOR USAID EVALUATIONS ................................................................................................................ 78 FIGURE 1: USAID/EGYPT WATER AND WASTEWATER PROGRAM RESULTS FRAMEWORK – MAY 2010…………………………………………………………………….....9 FIGURE 2: WWSS RESULTS FRAMEWORK – OCTOBER 2011……………………………....13 FIGURE 3: WPRR RESULTS FRAMEWORK – MID 2011………………………………………36 TABLE 1: EVALUATION QUESTIONS…………………………………………………………..3 TABLE 2: WWSS COMPONENT 1 RESULTS AGAINST SOW…………....…………………..15 TABLE 3: WWSS COMPONENT 2 RESULTS AGAINST SOW…………………………….…21 TABLE 4: WWSS COMPONENT 3 RESULTS AGAINST SOW…………………………….…23 TABLE 5: WWSS COMPONENT 4 RESULTS AGAINST SOW…………………………….…26 TABLE 6: WWSS PMP INDICATORS………………………………………………………..…29 TABLE 7: PIR1 INDICATORS (2007/8 OR LATER BASELINE TO 2009/10)………………..…30 TABLE 8: WWSS PIR1 INDICATORS (ACTUALS VS LIFE OF PROJECT TARGETS)……...…31 TABLE 9: WWSS PIR3 INDICATOR (STAFF TRAINED)………………………………………32 EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE v PERFORMANCE EVALUATION REPORT TABLE 10: WWSS PO INDICATOR (PERCENT OF COLLECTION)………………………....33 TABLE 11: WPRR COMPONENT 1 RESULTS AGAINST SOW………………………………37 TABLE 12: WPRR COMPONENT 2 RESULTS AGAINST SOW………………………………39 TABLE 13: WPRR COMPONENT 3 RESULTS AGAINST SOW………………………………42 TABLE 14: WPRR COMPONENT 4 RESULTS AGAINST SOW………………………………44 TABLE 15: WPRR COMPONENT 5 RESULTS AGAINST SOW………………………………45 TABLE 16: WPRR COMPONENT 6 RESULTS AGAINST SOW………………………………46 TABLE 17: WPRR PMP INDICATORS……………………………………………………….…47 TABLE 18: WPRR PMP RESULTS………………………………………………………………..48 EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE vi PERFORMANCE EVALUATION REPORT ACKNOWLEDGEMENTS This report was prepared by an Evaluation Team consisting of: ● Raouf Youssef, Consultant and Team Leader ● Heather Skilling, USAID/E3 ● Anthony Kolb, USAID/E3 ● Victoria Mitchell, USAID/Egypt Program and Project Development Officer ● Ariel Swan, USAID/Egypt Engineering Officer The Evaluation Team would like to thank USAID/Egypt PSD staff John Pasch, Noha El Maraghy, Atef Abdel Sayed, and Amani Selim from the Program Office for their guidance and support. The Team also wishes to thank key counterparts for their excellent briefings and for providing the Team with the relevant information that facilitated their work. Special thanks and appreciation to Mohamed El Alfy, the Assistant Minister of the Ministry of Housing, Utilities and Urban Development, and Mamdouh Raslan, the Deputy Chairman of the Holding Company for providing the Team with their outstanding and candid assessment of the status of the sector and prospective for the future. They shared with the Team their respective visions for the sector and are true champions for the institutional and policy reforms of the sector. EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE vii PERFORMANCE EVALUATION REPORT ACRONYMS AIR Annual Information Return CAPWO Cairo and Alexandria Potable Water and Wastewater Organization COP Chief of Party COR Contracting Officer‟s Representative CIP Capital Investment Planning CIP/PM Capital Investment Planning and Program Management E3 Bureau for Economic Growth, Education and Environment EWRA Egyptian Water Regulatory Agency FARA Fixed Amount Reimbursement Agreements GIS Geographic Information System GIZ Gesellschaft für Internationale Zusammenarbeit GOE Government of Egypt HCWW Holding Company for Water and Wastewater HR Human Resources IBRD International Bank for Reconstruction and Development IT Information Technology ITMP IT Master Plan JCESD Job Creation through Essential Service Delivery M&E Monitoring and Evaluation MASTER Maintenance through Systematic Tracking and Equipment Repair MARS Monitoring and Analysis Reporting System MHUUC Ministry of Housing, Utilities and Urban Communities MHUUD Ministry of Housing, Utilities and Urban Development MIC Ministry of International Cooperation MOF Ministry of Finance NOPWASD National Organization for Potable Water and Sanitary Drainage O&M Operations and Maintenance PEA Public Economic Authority PIR Program Intermediate Result PMEU Program Management and Evaluation Unit PMP Performance Monitoring Plan PO Program Objective PPP Public-Private partnerships PRiSM Project Management Information System QA/QC Quality Assurance and Quality Control QPR Quarterly Progress Report TOT Training of Trainers WWSS Water and Wastewater Sector Support PSD USAID Productive Sector Development Team UFW Unaccounted-for Water SOW Scope of Work USAID United States Agency for International Development WPRR Water Policy and Regulatory Reform Project WWC Water/Wastewater Company EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE viii PERFORMANCE EVALUATION REPORT WWSPRI Technical Assistance for Water and Wastewater Sector Policy Reform WWSS Water and Wastewater Sector Support Project Y3WP Year 3 Work Plan Y4WP Year 4 Work Plan EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE ix PERFORMANCE EVALUATION REPORT EXECUTIVE SUMMARY Since 1975, USAID has supported the development of Egypt‟s water and wastewater sector by providing more than $4 billion for infrastructure and associated institutional strengthening programs. The impact of this sizable investment is recognized not only for the direct benefit of improved water and wastewater services but for the related benefits of improved public health indicators and positive economic impacts. Over the past three years USAID/Egypt has funded two contracts providing technical assistance to the institutions responsible for overseeing and managing water and wastewater service provision in Egypt and sector investment planning and implementation. This report summarizes the findings of an evaluation of the performance of these two contracts. This performance evaluation aimed to: determine whether USAID/Egypt assistance is meeting stated objectives; summarize accomplishments/weaknesses of each contract; assess the performance monitoring plans for both contracts; and identify near term priority areas for possible future programming in the Egyptian water sector. In addition, the evaluation aimed to confirm the validity of the overall development hypotheses underlying the USAID/Egypt Mission's water and wastewater strategy. The Evaluation Team was able to determine that the Water and Wastewater Sector Support Project (WWSS) and the Water Policy and Regulatory Reform Project (WPRR) achieved many of their expected results and have contributed to progress toward ensuring the sustainability of infrastructure investments made by USAID and the GOE. WWSS assisted the Holding Company for Water and Wastewater (HCWW) and subsidiaries to adopt business planning, helped to prioritize and manage activities and investments, and provided extensive training. It also appears to have helped, on a small-scale, reduce subsidiary operational costs and broadly improve performance in at least one subsidiary. Most significantly, WPRR undertook additional tariff analysis, developed an operator classification system, began development of a model performance agreement, and continued to evolve service level indicators while building the capacity of the Egyptian Water Regulatory Agency (EWRA) staff. The Contractor‟s support to the Government of Egypt‟s (GOE) Public Private Partnership and the provision of technical assistance for transactions facilitated successful engagement of the private sector in the expansion of services. The project also finalized the drafting of a Water Law and is working with stakeholders to develop a water strategy. If ultimately implemented, these efforts have the potential to guide the water and wastewater companies in Egypt toward becoming commercially viable and sustainable institutions. However, there are a number of areas that require increased emphasis in the short term (option year) to promote systemic and sustained improvements in provision of water and wastewater services in Egypt. Under WWSS, the project should continue to improve the quality of performance data reported by the subsidiaries; implement a focused public outreach program based on messages that reinforce the need for demand management, financial sustainability and efficiency; and implement technical assistance that is more targeted toward achieving key performance related results. WPRR should strengthen the piloting of licensing/performance agreements and focus outreach efforts on helping the consumers and key decision makers at the national and selected governorate levels to understand the implications of tariff choices. EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE x PERFORMANCE EVALUATION REPORT The Evaluation Team recommends that USAID/Egypt take advantage of the available option year extension for both projects directing contractor efforts to emphasize the above and, in addition: 1. Target any future capital expenditure support to incentivize performance agreements, e.g., work with HCWW to make funding for capital replacement and piped network expansion contingent on achievement in improved asset management and billing and customer service measures respectively; 2. Begin before the end of the option year to explore developing Fixed Amount Reimbursement Agreement (FARA) type technical assistance through the holding company to replace the support currently being provided by WWSS; 3. Review and revise the USAID exit strategy for the sector to ensure local ability to sustain and deepen reform achieved to date with USAID support. This evaluation will be used by USAID/Egypt in making option year decisions and project corrections on the WWSS and WPRR contracts, and for providing a more informed basis to strategize, prioritize, and design future assistance for water and wastewater sector projects in Egypt. EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE xi PERFORMANCE EVALUATION REPORT الملخص التنفيذي ٍْزعبً ،1975 د عَ ج اىىمبىت ا ٍ.شينيتىيخ َْيت اىذوىيتفيٍصشح َْيتقطبع اىَيبٓواىصشف اىصحي ٍِ خ . هحىفيش أمثش ٍِ ٍ4ييبس دو.سىخَىيواىبْيتاىخحخيت وٍبيشحبظبهب ٍِبش اٍجاىذعٌ اىَؤسسي. وحٌإدساك أثش هزٓ االسخثَبس اث اىضخ َتىيس فقظفيَبيخعيق ببالس خفبدة اىَببششة ٍِححسي ِ خذٍبث اىَيبٓواىصشف اىصحي وىن ِفيَبيخصاىفىائذرث اىصيتبخحسي ِ ٍؤششاثاىصحت اىعبٍت واآلثبس االقخصبديت االيجببيت. وعيي ٍذي اىسْىاثاىثال د اىَبضيت ٍىى ج اىىمبىت األٍشينيتىيخ َْيت اىذوىيتفي ٍصشعقذي ِىخقذيٌ اىَسبعذاث اىفْيتإىياى َؤسسبثاىَسئىىت عِ اإلششافعييحقذيٌوإداسة خذٍبث اىَيبٓ واىصشفاىصحيفي ٍصش وحخطيظ وحْفيز االسخثَبساثفي هزاىقطبع .وييخص هزاىخقشيشاىْخبئجاىخيحىصوإىيهبحقييٌ أداء هزي ِاىعقذي .ِ يهذفحقييٌ األداء هزاإىيححذيذٍبإرامبّ جٍسبعذاثاىىمبىت األٍشينيتىيخ َْيت اىذوىيتفيٍصشقذ حقق ج األ هذاف اىَعيْت، وحيخيص اإلّجبصاث وّقبطاىضعففي مو ٍِاىعقذي ،ِ وحقييٌاىخطظوسصذ األداءىنوعقذوححذيذاىَجبالثراث األوىىيت قشيبت اىَذيىيبش اٍجاىَسخقبييتاىَحخَيتفيقطبع اىَيبٓبَصش. وببإلضبفتإىيرىل،يهذفاىخقييٌإىياىخأمذ ٍِصحت االفخشاضبثاىخَْىيتاىشبٍيتاىخيحضَْخهباسخشاحيجيتاىىمبىت األٍشينيتىيخ َْيت اىذوىيتفيٍصشىقطبع اىَيبٓواىصشف اىصحي. وحَن ِفشيقاىخقييٌ ٍِححذيذ أُ ٍششوع "دعٌقطبع اىَيبٓ واىصشف اىصحي (WWSS "(ٍٚششوع " إصالحسيبسبث وحْظيٌ اىَيبٓ "(WPRR(لذ حممب اٌىث١ش ِ ٓ ٴخبئدّٙب ا ٌّخٛلؼت ٚاٌخٟ أعّٙ ج فٟ إحشاص حمذَ ٴحٛ ظّب ْ اعخذ اِت ا.عخثّبساث فٟ اٌب١ٕت األعبع١ت اٌخٟ لذِخٙب اٌٛوبٌت األِش٠ى١ت ٌٍخ ١ّٕت اٌذ١ٌٚت فٟ ِصش ٚاٌحىٛ ِت اٌّصش٠ت. عبػذ ِششٚع "دػُ لطبع ا ١ٌّبٖ ٚا ٌصشف اٌصحٟ" اٌششوت اٌمبب عت ٌ ١ٍّبٖ ٚاٌصشف اٌصحٟ ٚا ٌششوبث اٌخببؼت ػٍٟ حبٕٟ حخط١ط األػّبي ٚعبػذ ػٍٟ ححذ٠ذ األ٠ٌٛٚبث ٚإداسة األٴشطت ٚاالعخثّبساث ٚحٛف١ش اٌخذس٠ببث ٚاعؼت إٌطبق. ٠ٚبذٚ أ٠عب أٴٗ عبػذ ػٍٟ ٴطبق ظ١ك، ػٍٟ خفط اٌخىب١ٌف اٌخشغ١ٍ١ت اٌفشػ١ت ٚػٍٟ ٴطبق أٚعغ، ححغ١ ٓ األداء ػٍٟ األل ً فٟ ششوت فشػ١ت ٚاحذة. ٚاألُ٘ ِ ٓ رٌ ه، لبَ ِششٚع "دػُ لطبع ا ١ٌّبٖ ٚاٌصشف اٌصحٟ" بؼًّ حح١ٍالث ٌٍخؼش٠فت ٚٚظغ ٴظبَ حص١ٕف حشغٍ١ٟ، وّب بذأ فٟ حط٠ٛش ّٛٔرج ال حفبل١ت األداء ٚاعخّش فٟ حط٠ٛش ِؤششاث ِغخٞٛ اٌ خذِت ٚفٟ ٴف ظ اٌٛل ج اٌؼّ ً ػٍٟ بٕبء لذساث ِٛظفٝ ا١ٌٙئت اٌّصش٠ت ٌخٕظُ١ ا ١ٌّبٖ (EWRA(. ٚلذ أدٜ دػُ اٌّمبٚي إٌٟ اٌششاوبث اٌخبصت ٚاٌؼبِت ببٌحىٛ ِت اٌّصش٠ت ٚحمذُ٠ اٌّغبػذاث اٌف١ٕت ٌٍّؼبِالث إٌٟ حغ١ٙ ً اٌّشبسوت إٌبخحت ٌٍمطبع اٌخبص فٟ اٌخٛعغ فٟ اٌخذِبث . وّب ػّ ً ا ٌّششٚع ػٍٟ ٚظغ اٌٍّغبث األخ١شة ٌّغٛدة لبٛٔ ْ ٌ ١ٍّبٖ ٚخبسٞ اٌؼّ ً ِغ اٌدٙبث اٌّؼ١ٕت ٌٛظغ اعخشاح١د١ت ٌ ١ٍّب.ٖ ٚإرا حُ حٕف١ز٘ب فٟ إٌٙب٠ت، فئ ْ ٘زٖ اٌدٛٙد عخىٛ ْ ٌذٙ٠ب اٌمذسة ػٍٟ حٛخٗ١ ششوبث ا ١ٌّبٖ ٚاٌصشف اٌصحٟ فٟ ِصش ٴحٛ اٌخحٛي إٌٟ ِؤعغبث ِدذ٠ت حدبس٠ب ِٚغخذ اِت. ِٓٚ ٴبح١ت ثبٴ١ت، ٕ٘بن ػذد ِ ٓ اٌّدبالث اٌخٟ حخطٍب حشو١ض أوثش ػٍٟ ا ٌّذٞاٌمص١ش )عٕت اٌخ١بس( ٌخشد١غ إدخبي ححغٕ١بث ِٕٙد١ت ِٚغخّشة فٟ حٛف١ش خذِبث ا ١ٌّبٖ ٚاٌصشف اٌصحٟ فٟ ِصش. ٚفٟ إطبس ِششٚع "دػُ لطبع ا ١ٌّبٖ ٚا ٌصشف اٌصحٟ" ٕ٠بغٟ ِٛاصٍت ححغٓ١ ٛٔػ١ت ب١بٴبث األداء اٌخٟ حبٍغ ػٕٙب اٌششوبث اٌخببؼت، ٚحٕف١ز بشاِح حٛػ١ت خّب١٘ش٠ت ِشوضة ِب١ٕت ػٍٟ اٌشعبئ ً اٌخٟ حؼضص اٌحبخت إٌٟ إداسة اٌطٍب ٚاالعخذ اِت ا ٌّب١ٌت ٚاٌىفبءة ٚاعخخذاَ اٌّغبػذاث اٌف١ٕت األوثش اعخٙذافب ٌخحم١ك ٴخبئح األداء اٌشئ١غ١ت راث اٌصٍت. وّب ٕ٠بغٟ أ ْ ٠ؼّ ً ِششٚع "إصالذ ع١بعبث ٚحٕظُ١ ا ١ٌّبٖ" ػٍٟ حؼض٠ض احفبلبث حدش٠ب١ت ٌألداء ٚاٌخشو١ض ػٍٟ خٛٙد اٌخٛػ١ت ٌّغبػذة اٌّغخٍٙى١ ٓ ٚصبٴؼٟ اٌمشاس اٌشئ١غ١١ ٓ ػٍٟ اٌصؼ١ذ اٌٛطٕٟ ِٚحبفظت ِخخبسة ػٍٟ فُٙ ا٢ثبس ا ٌّخشحبت ػٍٟ اٌخ١بساث اٌخبصت ببٌخؼش٠فت. ٚلذ أٚصٝ فش٠ك اٌخمُ١١ اٌٛوبٌت األِش٠ى١ت ٌٍخ ١ّٕت اٌذ١ٌٚت ببالعخفبدة ِ ٓ حّذ٠ذ "عٕت اٌخ١بس" ا ٌّخبحت ٌىال اٌ ّششٚػ١ ٓ ٌخٛخٗ١ خٛٙد اٌّمبٚي ٌٍخأو١ذ ػٍٟ ِب عبك، ببإلظبفت إٌٟ ِب ٍ٠ٟ: .1 اعخٙذاف أٞ ٴفمبث سأعّب١ٌت فٟ اٌّغخمب ً ٌذػُ احفبلبث ححف١ض األداء، ػٍٟ عب١ ً ا ٌّثبي، اٌؼّ ً ِغ اٌششوت اٌمببعت ٌ ١ٍّبٖ ٚاٌصشف اٌصحٟ ٌخٛف١ش اٌخ٠ّٛ ً اٌالص َ العخبذاي سأ ط ا ٌّبي ٚاٌخٛعغ فٟ شبىبث األٴبب١ب ِششٚطب بخحم١ك ححغٕ١بث فٟ إداسة األصٛي ٚإػذاد اٌفٛاح١ش ٚل١ب ط خذِت اٌؼّالء ػٍٟ اٌخٛاٌٟ، .2 اٌبذء لب ً ٴٙب٠ت عٕت اٌخ١بس فٟ اٌبحث ػ ٓ احفبل١ت ٌغذاد ا ٌّببٌغ اٌثببخت (FARA (ٌخٛف١ش اٌّغبػذاث اٌف١ٕت ِ ٓ خالي اٌششوت اٌمببعت ٌخح ً ِح ً اٌذػُ ا ٌّمذَ حب١ٌب ِ ٓ ِششٚع "دػُ لطبع ا ١ٌّبٖ ٚاٌصشف اٌصحٟ". .3 اعخؼشاض ِٚشاخؼت اعخشاح١د١ت اٌٛوبٌت األِش٠ى١ت ٌٍخ ١ّٕت اٌذ١ٌٚت ٌٍخشٚ ج ِ ٓ ٘زا اٌمطبع ٌعّب ْ حّى ٓ اٌمذساث اٌّح١ٍت ِ ٓ ححم١ك االعخذ اِت ٚحؼ١ّك اإلصالحبث اٌخٟ حُ إٴدبص٘ب حخٝ ا٢ ْ بذػُ ِ ٓ اٌٛوبٌت األِش٠ى١ت ٌٍخ ١ّٕت اٌذ١ٌٚت. ٚع١خُ اعخخذاَ ٘زا اٌخمُ١١ ِ ٓ لب ً اٌٛوبٌت األِش٠ى١ت ٌٍخ ١ّٕت اٌذ١ٌٚت فٟ ِصش فٟ احخبر اٌمشاساث اٌخبصت بغٕت اٌخ١بس ٚحص٠ٛببث ػمذٞ ِششٚع "دعٌقطبع اىَيبٓواىصشف اىصحي (WWSS "(ٚ ٍششوع " إصالحسيبسبث وحْظيٌ EGYPT - WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE xii PERFORMANCE EVALUATION REPORT اىَيبٓ )WPRR ")ٚحٛف١ش أعب ط أوثش اطالػب ٌٛظغ االعخشاح١ د١بث ٚححذ٠ذ األ٠ٌٛٚبث ٚحصُ١ّ اٌّغبػذاث اٌّغخمب١ٍت ٌّششٚػبث لطبع ا ١ٌّبٖ ٚاٌصشف اٌصحٟ فٟ ِصش. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 1 1. INTRODUCTION USAID has provided more than US $4 billion in assistance to the Egyptian water and wastewater sector over the last 35+ years. Over roughly the same period, the Egyptian Government has spent US$11 billion on water and wastewater plant construction. Despite investments, administrative and management problems persist in the sector including pervasive financial problems and limited accountability. In 2004, USAID supported the GOE in implementing important reforms in the water and wastewater sector. Former President Mubarak issued Decrees 135 and 136 that established a Holding Company for Water and Wastewater (HCWW) and EWRA. Fourteen public companies and public economic authorities were transformed into subsidiaries of the Holding Company. Since 2006 the Holding Company was given an additional mandate to take over water and wastewater operations in new communities and to corporatize operations in 10 governorates not served by subsidiary companies. In 2008, USAID signed contracts for two technical assistance contracts which were designed to support these nascent institutions – the regulator, the holding company and the operating subsidiaries. The two technical assistance projects were the Water and Wastewater Sector Support Project (WWSS) and the Water Policy and Regulatory Reform Project (WPRR). Through a competitive procurement, each contract was won by Chemonics International. Together, the two contracts are referred to as the Water and Wastewater Sector Support Program. WWSS focused its technical assistance on HCWW at the national level and 13 subsidiaries: Assiut, Aswan, Beni Suef, Cairo, Daqahliya, Giza, Luxor, Matrouh, Menufiya, Minya, Qena, Sinai and Sohag (Sinai Water and Wastewater Company, was added through a contract amendment in November, 2010). WPRR‟s primary recipients of assistance have been and the Egyptian Water Regulatory Agency (EWRA) and the Ministry of Housing, Utilities and Urban Development. However, recent WPRR work on tariff reform has engaged the non-WWSS supported subsidiary in Gharbia. The total USAID investment in the two contracts totaled $40M. USAID/Egypt requested this Performance Evaluation to independently: assess whether the assistance under these projects is meeting stated objectives; summarize the major accomplishments/weaknesses of each project; assess the performance monitoring plan for both projects to determine whether the plans with their performance indicators truly measure the project progress and achievements; and, identify immediate and near term priority areas that should be the focus of possible future programming in the Egyptian water sector particularly in light of the political changes in Egypt. Although the January 25, 2011 revolution presented Egypt with the opportunity to accelerate the pace of democratization and personal freedom, it has impacted the water sector in a negative way. In our discussion with many sector officials and observation it is unlikely that the Water Law will pass in the near future (18-24 months) and most reforms will be put on hold until the immediate priorities of the country are addressed. There is no political urgency, at present, to adjust water tariffs or to enforce collection policies, but salaries of workers in the sector are being increased at the rate of 20% a year and levels of staffing are also increasing. The net result is an increase in operating costs without a commensurate increase in revenues to the WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 2 subsidiaries. At the same time, the GOE has declining resources with which to subsidize the sector, which is already heavily subsidized. Support for both capital investment and operations and maintenance (O&M) are in decline. As a result, maintenance of meters, networks and pumps is being delayed which will lead to a need for more serious and costly system repairs later. The water and wastewater sector will face many challenges in the next few years. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 3 2. EVALUATION OVERVIEW Guided by the most recent USAID Evaluation Policy, USAID/Egypt developed a scope of work (SOW) designed to undertake a performance evaluation to evaluate the effectiveness of the achievements of the Water and Wastewater Sector Support Project (WWSS) and the Water Policy and Regulatory Reform Project (WPRR) against the stated program and project goals. The two contracts were awarded in 2008 for an initial four year period, with an option to extend for an additional year. To implement the Performance Evaluation, USAID/Egypt assembled an Evaluation Team (Team) consisting of Raouf Youssef, Consultant and Team Leader; Heather Skilling, USAID/Washington; Anthony Kolb, USAID/Washington; Victoria Mitchell, USAID/Egypt; and Ariel Swan, USAID/Egypt. The USAID budget ceiling for this evaluation activity was $47,000. EVALUATION OBJECTIVES The primary purpose of the evaluation is to determine whether the assistance provided by USAID/Egypt to the GOE is meeting the stated objectives of the project and contributing, as planned, to the development objectives of the program. The objectives of each project are set out in Sections 3 and 4 of this report. Additionally, the evaluation should provide a detailed picture of the major accomplishments and weaknesses of each project since inception, indicating as well what results are likely to be achieved by the completion of each project. The SOW directed the Team to assess the performance monitoring plan (PMP) of each project to determine whether the PMPs truly measure the project progress and achievements. Finally, the Team should discuss anticipated results and implications of options related to USAID/Egypt exercising the option year on both projects and to identify immediate and near term priority areas that should be the focus of possible future programming in the Egyptian water sector. The Evaluation SOW set out the following specific Evaluation Questions: TABLE 1: EVALUATION QUESTIONS Central Evaluation Questions Where addressed in this report Where did implementation in each project to date fall short of achieving expected (or desired) results? Sections 4,5,6 What constraints impeded achieving targeted results? Sections 4,5,6 Views of USAID, implementers, and beneficiaries, with respect to weaknesses/strengths in the design, implementation and management of each project? Sections 4,5,6 What are the anticipated results and implications of options related to USAID/Egypt exercising the option year on these two projects? Section 7 Based on the experience of the two projects evaluated and the status of the Egyptian water sector, what are the facts and implications of possible short and long term future USAID assistance? Section 7 WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 4 Additional Evaluation Questions Where addressed in this report Assess the PMP for each project to determine whether the Plan and performance indictors truly measure project progress and achievements. Sections 4,5 Identify implementation problems, unmet needs, or unintended consequences or impacts, taking into account the host country environment, especially following the events of January 25, 2011. Sections 4,5,6 and 7 Progress made on each project in terms of relevance, impact, sustainability, and cost-effectiveness. Sections 4,5,6 Confirm validity of the overall – and project-specific – development hypotheses or critical assumptions underlying the Mission‟s water and wastewater strategy and the projects that were designed, funded and implemented to make such strategy operational. Sections 4,5,6,7 EVALUATION METHODOLOGY This report was prepared by a five-person Evaluation Team consisting of an external consultant, two USAID/Washington staff members, and two USAID/Egypt staff members. Interviews and site visits were conducted over a roughly three-week period in Egypt in January 2012. The Team followed an original guiding SOW drafted by USAID/Egypt in late 2011 with some minor agreed changes in the schedule of site visits. The evaluation methodology included: Written design: The Team developed a set of evaluation questions (Annex E), building from the SOW. Data was collected through implementers, USAID staff, stakeholders and project beneficiaries. Project data analysis: The Team reviewed project documentation with particular attention to the Statements of Work, Annual and Quarterly Work Plans, and the PMP for each project. A list of reviewed documents is included as Annex D. Project analysis was performed of the sector results against project activities and the PMP stated results against project activities. Participation of national counterparts: The Evaluation Team conducted semi-structured interviews with a variety of stakeholders including the PPP unit at the Ministry of Finance; the Assistant Minister of the Ministry of Housing, Utilities and Urban Development; the Deputy Chairman of the Holding Company for Water and Wastewater (HCWW); the Deputy Chairman and Technical Staff of NOPWASD; the Monitoring and Planning Unit at MHUUR; and the EWRA key officers. In each of these interviews, discussions were guided by the Evaluation Questions listed in Annex E. Briefings and discussion with project implementers: The Team received project briefings from each of the project teams at the start of the evaluation. The briefings focused on the implementers‟ perspective on project progress, achievements and constraints. In each of these interviews, the Team guided the conversation according to the Evaluation questions. The Team subsequently held follow-up meetings with each team to obtain additional information and clarifications. Visits to project sites: As required by the Evaluation SOW, the Team visited WWSS project sites in Aswan and Sohag. An additional WWSS briefing was provided to the Team by the Minya WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 5 Governorate water and wastewater subsidiary. Due to logistical constraints, this briefing was conducted on site in Aswan. The Team attended part of a Tariff Adjustment Workshop held in Cairo under the auspices of the WPRR project and also visited the EWRA. A Gharbia water and wastewater company official traveled to Cairo to provide the Team with an extensive briefing on WPRR project activities related to tariff analysis. Those four companies presented a good balance of the water and wastewater companies receiving USAID assistance: two companies from Upper Egypt, one company from Middle Egypt, and one company from the heart of the Delta. Orientation and briefings with USAID/Egypt: The Team met with USAID/Egypt CORs responsible for the WWSS and WPRR projects in order to understand the project decisions that were taken, the COR understanding of project objectives, and issues encountered during project implementation. The team also met with the Program Officer responsible for the Evaluation Unit, Amani Selim, and was provided with a briefing on Mission expectations of the evaluation. Additional meetings were held with PSD staff to describe the Mission strategy for the water and wastewater sector and the history to the two projects. At the conclusion of each visit or interview, the Team members shared their notes and compared findings. EVALUATION LIMITATIONS The SOW for the evaluation is ambitious in scope for the time that the Team was able to provide, particularly in-country. That said, the Team felt that it was able to obtain the information necessary to conduct a thorough and meaningful evaluation. No significant limitations were encountered; however the following points can be recorded: ● All stakeholder meetings and site visits were conducted with USAID/Egypt personnel present for some or all of the discussion. This was logistically useful and helped facilitate discussion. There was no indication that interviewees were less than open in expressing any feedback on the projects. However, for the purposes of a purely objective evaluation, this may have been less than best practice. ● Due to security, cost, and time considerations, it wasn‟t feasible for the Team to visit all the water and wastewater companies cited in the SOW. However, a representative sample was selected and visited. ● While the SOW cites the need to analyze results against indicators identified in the PMP, the WWSS and the WPRR PMPs changed over the course of the Projects. The final revisions were made in 2011, leaving limited time to achieve and collect results. This data issue is discussed in greater depth elsewhere in the report. ● Although the evaluation policy encourages gender-sensitive indicators and sex￾disaggregated data, this data was not relevant to this project and was not collected through project activities and thus not part of the evaluation. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 6 3. USAID/EGYPT WATER AND WASTEWATER SECTOR TECHNICAL ASSISTANCE The more than 35-year history of USAID assistance to the Egyptian water and wastewater sector has focused heavily on infrastructure investment. However, over the decades USAID has also supported technical assistance to the sector which would lead to the sustainable use and maintenance of the infrastructure created. The WWSS and WPRR projects were consistent with USAID‟s efforts to support the sustainability of its investment. PROJECT BACKGROUND USAID water and wastewater infrastructure projects in the 1980s and 1990s, in addition to funding construction, contributed toward institutional development efforts to build the sustainability of the institutions tasked with operating and maintaining the infrastructure that USAID helped to expand and improve. These efforts were largely aimed at strengthening the ability of utility management to operate and maintain new infrastructure. During the 1990s, this institutional strengthening assistance was provided to the Cairo water and wastewater authorities, several secondary cities, Middle Egypt utilities, the Suez Canal Authority, the Canal Governorates (Suez, Ismailia and Port Said) and the Alexandria water and wastewater authorities. These projects brought training, equipment, spare parts, tariff analysis, public education and operations and maintenance improvement. However, these efforts had mixed success in producing sustained institutional improvement. One example is the USAID support provided through the Cairo I and II projects which ran from 1977 to 1998. As part of this largely construction-oriented effort, USAID contractors also provided technical assistance to the Public Authority for Water Utilities in Greater Cairo in developing eighteen performance indicators, establishing a Performance Management Office, and producing Performance Management Reports to empower managers. Based on report findings, fourteen Performance Improvement Programs (PIPs) were implemented that resulted in significant positive change: ● The improvement of skills and knowledge of the “empowered managers” through a management development training program. ● The reduction of costs in chemical inputs in five pilot water treatment plants, accounting to savings of up to 25 percent. ● The establishment of a Customer Services Center in Helwan that improved collection rates and cash flow. However, after completion of the project, and the departure of the USAID contractors in 1998, key elements of these programs including the Performance Management Office, the Customer Service Center, and a Computerized Mapping and Inventory Center all regressed into quasi￾functional arms of the utility. A 2006 USAID evaluation of its infrastructure portfolio concluded that this backsliding demonstrated that sector management had not internalized the value of performance management and customer care tools. The 2006 evaluation noted comparable backsliding against improvements achieved in other water and wastewater institutional strengthening efforts undertaken in Cairo and other cities. Based upon the mixed results of these previous institutional strengthening efforts, USAID concluded that improvements at the utility level could not be fully realized or sustained without equivalent attention to sector reform at the national level. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 7 WATER AND WASTEWATER REFORM IN EGYPT In the early 1990s, initial GOE sector reforms were triggered with the issuance of a series of presidential decrees that established seven water and wastewater Public Economic Authorities (PEA). The seven PEAs were utilities under governorate jurisdiction in Aswan, Minya, Beni Suef, Fayoum, Dakahleya, Gharbeya, and Sharqeya, and were intended to operate as business-oriented operations with more autonomy, prone to less political interference. To help further these reforms, and consistent with its intention to support national-level efforts, USAID awarded a 2-year contract in 1998 for the Legal Institutional Regulatory Reform (LIRR) project. That project, which was extended for an additional 2 years (LIRR II), worked on the following: ● A draft law creating a water sector regulatory authority ● Organizational and staffing plans for the new water regulator ● Analysis of tariff needs in a selected group of utilities ● Establishment of reform milestones for the sector ● Plans and studies for concessions and management contracts in a selected group of utilities. The only notable sector reform achievement during the life of LIRR I and II was the May 2000 Cabinet approval of draft presidential decrees calling for the creation of a water and wastewater regulatory agency and related institutional reforms. Under the third USAID water and wastewater sector reform program that began in July 2003 more significant reform progress was achieved. Under this 2-year project entitled “Technical Assistance for Water and Wastewater Sector Policy Reform” (WWSPR I) the contractor CH2M Hill provided technical assistance to the then Ministry of Housing, Utilities and Urban Communities (MHUUC) in four areas: utility regulation, privatization transactions, corporatization/ capital financing strategies and Management Information System (MIS). During WWSPR I the following notable sector reform milestones were achieved: 2003 - A draft law authorizing the creation of a regulatory agency for the water and wastewater sector is approved by the Cabinet and a draft law providing a legal framework for private sector participation in the water and wastewater sector is approved by the Cabinet. April 2004 - (1) Presidential Decree No. 135 is issued establishing a Holding Company for Water and Wastewater and Subsidiary Companies; and (2) Presidential Decree No. 136 is issued establishing the Regulatory Agency for Potable Water, Wastewater, and Consumer Protection. Together these agencies provided the basis for a modern water and wastewater sector, with a corporatized management structure and a sector regulator. September 2004 - The HCWW commenced operation and began actively managing 14 subsidiary companies established by Law 203/1991. The Laiha (implementing regulations) and the utility organizational structure were developed, approved, and implemented in the HCWW and subsidiary companies. The HCWW is actively managing the subsidiary companies and requiring performance monitoring reports and five-year financial plans. May 2005 - Minister of Housing appoints a board of directors for a regulatory agency for water and wastewater and the regulatory authority commenced operations and begins to review utility certification applications and utility performance indicators on a pilot basis. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 8 To continue the support provided under WWSPR I and the associated nascent sector reforms, USAID initiated a follow-on contract (WWSPR II) that ran from October 2005 until March 2009. WWSPR II aimed to: operationalize the new regulatory agency, HCWW and its subsidiary companies; provide program management assistance to MHUUD; and perform two Public￾Private Partnership feasibility studies. The total value of the WWSPR II contract was over US $13.5 Million. At the end of the WWSPR projects, USAID did not conduct an external evaluation of the extensive technical assistance they provided. However, the February 2007 Evaluation of USAID/Egypt Utility Projects and the September 2009 End of Contract Report from the WWSPR II contractor do offer some insights. However, the latter report fails to substantially quantify the impact of the project and instead, like much of the reporting under the subsequent WWSS and WPRR projects, focuses largely on description of activities completed and anecdotal results. The history of USAID‟s support to the sector reveals more than a decade of effort and up to US $300 million invested in helping to structure a national framework and institutions necessary to complement infrastructure investments and deliver improved sector performance. After such a prolonged effort, and with so many fundamental achievements in place, this WWSS and WPRR evaluation focused on examining whether the projects were delivering in terms of sector results. CURRENT SECTOR SUPPORT OBJECTIVES Under an Action Memorandum signed in late 2007, USAID/Egypt amended the Egyptian Utilities Management (EUM) Assistance Agreement, USAID No. 263-0270, to increase funding by just over $74.5 million and extend project assistance in the water and wastewater sector to September 30, 2013. In addition to authorizing funding for water and wastewater infrastructure construction activities, the Action Memorandum approved funding to: ● Support program management services for the GOE‟s water and wastewater capital investments ● Assist the HCWW in increasing private sector participation in the sector ● Encourage policy and regulatory reform ● Assist the Egyptian Water Regulatory Agency (EWRA) in regulating the sector ● Provide technical assistance to help establish new water/wastewater companies in select governorates. The Mission subsequently chose to support implementation of these non-construction related activities through the award of two new contracts in October 2008: (1) Water and Wastewater Sector Support (WWSS) Program, USAID Contract No. EPP-I-00- 04-00020-00 Order 3 with Chemonics International – a Task Order under the USAID WATER II indefinite quantity contract with subcontractors CH2M HILL and Chemonics Egypt (2) Water Policy and Regulatory Reform (WPRR) Project, USAID Contract No. EPP-I-00-04- 00020-00 Order 2 with Chemonics International – also a Task Order under the USAID WATER II indefinite quantity contract with subcontractors CH2M HILL and FinBi. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 9 Both contracts were designed to contribute toward one common program objective, namely “improved quality of and access to sustainable water and wastewater services”1, although each in a different way. Accordingly, USAID/Egypt worked together with the two contractors after contract award to develop an overall water sector program results framework (see Figure 1). In addition to the shared program objective, the results framework also specified three associated program intermediate results (PIRs) and seven key results areas (KRAs) to which the two contracts were meant to contribute as specified in the figure below. As stated in the results framework, the overarching program objective was: Improved quality of and access to sustainable water and wastewater. FIGURE 1: USAID/EGYPT WATER AND WASTEWATER PROGRAM RESULTS FRAMEWORK - MAY 2010 A Performance Monitoring Plan (PMP) is a tool to plan and manage the process of assessing and reporting progress towards achieving an assistance objective. The PMP should align with the results framework in terms of the indicators selected and the results reported. Initially WWSS and WPRR intended to develop separate PMPs. However given the inter-related results framework, USAID/Egypt decided that the two contracts should submit a joint PMP to assist in tracking expected results. The last draft of the joint PMP (dated May 23, 2010 – over 1.5 years into implementation of the two 4-year contracts) contained 37 indicators to track project 1 Similar to the goal of the earlier USAID/Egypt’s Special Object 18 )SpO18( that guided prior sector investments, i.e. increase access to sustainable utility services. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 10 results: 9 to be tracked by both contracts, 18 to be tracked solely by WWSS, and 10 to be tracked solely by WPRR. To simplify reporting, in 2010 USAID agreed to allow the two contracts to once again develop separate PMPs to replace the joint WWSS/WPRR PMP. In February 2011, the two Contractor teams submitted these separate PMPs taking into account changes in the projects over the course of years 1 and 2. The two PMPs recognized that the objectives and scope of work for each contract remained the same, however, the revised PMPs attempted to clarify each program‟s focus and better define measureable outputs and impacts. In the following two sections of the report, the Team describes the specific objectives and achievement of each project against: ● The components of each contract‟s scope of work; and ● The final approved February 2011 PMPs. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 11 4. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM (WWSS) According to the WWSS contract scope of work, the overall objective of the WWSS project is to support the GOE to develop and implement programs that: ● Increase the financial and commercial viability of existing water and wastewater companies; ● Establish new regional water and wastewater Subsidiary Companies; ● Develop and implement a capital investment planning and program/project monitoring and management mechanism; ● Build the capacity of staff; ● Increase managerial, technical and operational efficiency; ● Improve the quality of services; and, ● Expand access to water and sanitation. To achieve this broad objective, the WWSS SOW required the Contractor to provide services under four specified project components: ● Institutional support to the Holding Company and selected subsidiaries ● Establishment of new subsidiaries in selected governorates ● Capital investment planning and program/project management ● Staff development/professionalization The SOW described “specific results” to be achieved under each component and a minimum set of tasks and deliverables required to achieve these desired results. In an October 2011 contract modification, the SOW was modified to delete several deliverables and tasks. These tasks were either redundant between tasks in the WPRR scope of work or the HCWW requested deletion since the same activities were being supported by European Union funding. The Team notes these modifications in its component by component evaluation of results achievement. TRACKING WWSS RESULTS The SOW requires the Contractor to clearly demonstrate that it has completed all related deliverables and exerted “reasonable effort” to ensure that the specified results are achieved. The SOW further specifies that the Contractor‟s “performance shall be evaluated based on the completion of specific tasks as outlined in the Task Order, adherence to the work plan, and reports submitted to the Task Order Cognizant Technical Officer (TOCTO).” However, starting in the Year 2 work plan (December 9, 2009), the Contractor stopped organizing annual and quarterly reports based on original SOW-specified components and associated results and instead began reporting activities organized along four newly specified components: 1. Central activities – assistance to the Holding Company 2. Cross-cutting activities – assistance targeting two or more subsidiary companies in a particular functional area of operations 3. Subsidiary-specific activities – relating to tasks undertaken with a particular utility, among the 13 mandated in the contract scope of work WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 12 4. Project Development and Coordination Activities – activities that coordinate, monitor, and communicate progress of the implementation of technical activities The Contractor proposed this reorganization of the reporting system to develop and subsequently monitor counterpart-specific tasks and interventions, based on the needs of the Holding Company and the target subsidiaries. The Contractor identified these needs through: 1. Initial and periodic assessments conducted by the technical assistance team; 2. Interviews with senior and mid-level subsidiary officials, and observations by the team members in the course of their regular field trips; 3. Information obtained from Holding Company during the meeting of the joint HCWW￾WWSS Joint Working Group, as well as personal meetings with the Chairman and division heads of HCWW; 4. Starting year 3, priority programs identified in the business plans prepared by the operating subsidiaries; and, 5. Exchanges with members of the Local Working Groups, composed of local utility officials and TA team members, responsible for joint implementation and monitoring of programs outlined the subsidiary business plans. As a result of this needs assessment, the Contractor‟s newly appointed Chief of Party observed that the original scope of work and geographic coverage were too broad, given the resources of the project. Furthermore, the Contractor observed that the utilities were at various levels of development, with differing needs and priorities, and that certain expected results, such as tariff adjustment were difficult to attain in the then prevailing environment. Furthermore, efficiency improvement and cost saving measures (e.g., energy and chemical uses, unaccounted-for-water, O&M management) were thought to be essential and more urgent but difficult to sustain without additional allocation of technical assistance resources as the operation and maintenance organizations at the subsidiary HQ and the sub-regional levels required major restructuring, along with training, basic equipment procurement, and intensive follow up. Subsequently, the WWSS Board of Directors (composed of representatives from USAID, the Holding Company and the Program contractors) agreed to focus WWSS along the above redefined components and an agreed upon a list of priority activities, stressing high impact/high return interventions. They also endorsed a proposed technical assistance approach that tailored these interventions to the needs of specific counterparts, and tracked their performance and impact in relation to tasks and programs developed in cooperation with these counterparts. This new approach also allowed the Contractor to avoid having to report on duplicate or redundant tasks occasionally appearing in two of the original scope of work components (e.g., “business plans” appear in components 1 and 2). USAID/Egypt determined that this reorganization of the project did not abandon the original SOW or any of the expected results beyond those described in the October 2011 contact modification. In annexes to the Year 2, 3 and 4 work plans, upon USAID‟s request, the Contractor helpfully provided a matrix mapping the relationship between deliverables defined under each of the original SOW-specified components to specific work plan proposed activities under their redefined component headings. The Contractor updates this matrix quarterly, along with the list of deliverables and the quarterly evaluation report conducted usually in the last two week prior to the publication of the progress report. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 13 Regardless of these efforts, the Evaluation Team found these reporting changes made it difficult to track achievement of the expected results described in the original scope of work from the modified WWSS project reporting. In addition, mapping achievement of the originally stated expected results and WWSS contributions to broader program objectives is further complicated by significant changes in the Performance Monitoring Plan (PMP) for this contract following the shift from a joint PMP with WPRR to an individual PMP for both WPRR and WWSS. In the latest version of the WWSS PMP (dated October 30, 2011), a modified results framework is constructed with several notable changes.2 A new, lower level program objective is defined as “Financial and commercial viability of selected utilities increased,” the same as PIR 1 under the previous joint results framework. The PMP defines a new PIR as “Management and operational efficiency increased” –a redefinition of the original KRA 1.1. The PIR 2 on capital investment planning and management is retained and the KRA 4 from original results framework is renamed as PIR 3. USAID/Egypt viewed these changes as useful in redefining project results within the manageable interest of the WWSS contractor. FIGURE 2: WWSS RESULTS FRAMEWORK (OCTOBER 2011) Based on this revised results framework, USAID/Egypt agreed to track achievement of results using 11 indicators – fewer than half the 27 indicators originally proposed under the previous joint PMP. USAID/Egypt felt this reduction was reasonable given the increased focus of the project after Year 1. In other words, while the project was spread thin with numerous activities across 12 governorates in Year 1, the contractor refocused its efforts in a discrete number of activities starting in Year 2, justifying a more focused PMP. Other changes in the results framework and subsequent tracking from the previous joint PMP include the following: ● No indicators were proposed for tracking PIR 2 and only one indicator was proposed for tracking the overall project objective: “percent of collection.” 2 The overall sector Results Framework referenced in this final version of the PMP is slightly different from the one presented in the October 2010 joint PMP. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 14 ● Key results areas “1.2 Financial management improved” and “1.3 Performance management improved” were dropped. It seems that these two key results areas are assumed to continue to fall under the new PIR 1 as three indicators associated with the prior KRAs are retained. However, three indicators in the joint PMP tracking these KRAs are dropped: “percent of customers billed in target areas,” “number of project￾targeted subsidiaries producing quarterly MARS reports,” and “quality of information generated by MARS.” ● In line with dropping the higher level project objective from the USAID Water/Wastewater Results Framework, all three of the indicators on subsidiary performance that were previously jointly tracked were dropped with the caveat that the WPRR program will be tracking them. ● With the justification that “WWSS activities do not significantly impact the wastewater treatment process,” the indicator tracking the compliance of wastewater samples with quality standards was dropped. In addition to these quantitative indicators, the PMP also briefly describes collection of qualitative information on progress toward achieving behavior change on the part of the water company staff and management. While USAID and the Contractor see such change as a necessary condition to achieving program objectives, it was agreed that such change is best assessed qualitatively. This qualitative data has been reported since the middle of Year 2. The Contractors qualitative assessment has focused on: 1) monitoring of progress of program activities against the work plan, 2) application of program tools, approaches, and training courses, and 3) impact of such interventions. For each assessed activity (system, tool, or training course), the Contractor observed whether and how the tools/skills are being used, and what results are generated. The Contractor reports that this qualitative assessment has proven useful to ongoing planning and management of activities. For example, on a quarterly basis the Contractor asks each company if MARS is utilized, which particular indicators are being utilized, how often it is updated, and what problems utilities face with the system. USAID/Egypt feels this in-depth qualitative data has proven more useful for project performance management than a simple count of utilities producing MARS reports or a measure of the quality of information generated by MARS – the two relevant qualitative indicators that were dropped in the latest revision of the PMP. The Evaluation Team‟s analysis of WWSS results is presented in two sections. The first examines the level of achievement in reaching the “expected results” listed in the original scope of work. In the contract scope of work these results are listed with associated tasks and deliverables but are described in terms of the outcomes that these tasks and deliverables and others are meant to effect. These expected results are mostly expressed in relation to actions by the recipients of WWSS technical assistance that are seen as necessary to achieve the higher level results targeted by the program, e.g., subsidiaries implementing a long term Training Plan that is updated regularly with the assumption that this will lead to improved subsidiary performance. The second section presents an analysis of the PMP results associated with the WWSS results framework, which we view as a secondary source of information and perspective linking specific observations to higher level objectives. As such, these two sections provide two means of interpreting achievement of the overall project objectives. ACHIEVEMENT OF EXPECTED RESULTS BY COMPONENT The first evaluation question in the Team‟s scope of work was to evaluate where implementation to date fell short of achieving expected (or desired) results and what factors constrained implementation from achieving greater progress toward planned results. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 15 As such, we have organized findings around consideration of achievement of expected results listed under the original WWSS scope of work components. These findings are based on a review of quarterly and annual work plans and information provided during the interviews undertaken by the Evaluation Team. COMPONENT 1 – INSTITUTIONAL SUPPORT TO THE HOLDING COMPANY AND SELECTED SUBSIDIARIES 9 subsidiaries originally targeted under SOW – Cairo Water, Aswan, Beni Suef, Giza, Luxor, Matrouh, Menufiya, Minya, Qena – Sinai, Assiut and Sohag added later TABLE 2: WWSS COMPONENT 1 RESULTS AGAINST SOW Expected Results Findings HCWW and Subsidiaries performance improved through: ● Establishing progressive performance targets on an annual or multi-year basis that are linked to a bonus pool or pay packages; Level of achievement unclear – We found no reference to this result in any project reporting documentation. However, interviewees noted that the Holding Company is providing substantial bonuses to subsidiary staff (as much as doubling their salaries). However, we heard different things about performance incentives. We heard it was “all or nothing “ – that the whole company staff benefits if there was improvement to company performance – presumably what is meant by a “bonus pool.” Other interviewees said that there were management incentives. No one was specific about what level of performance earns such incentives or if there are progressive targets being set. USAID/Egypt staff explain that: “Initially, there was no interest by the Holding Company for such a system, as they have their own evaluation system linking subsidiary chairmen and staff compensation to the performance of the chairman and responsiveness to HCWW directions (25%) and the actual performance of the various units within the subsidiary companies (O&M, customer service, networks, etc…) (75%). There are indications now that the new chairman, who took over last year, is amenable to abandoning the previous “egalitarian” bonus system and switch to a performance-based compensation system. There appear to be a buy-in on the part of division heads, and willingness to start experimenting with a new performance evaluation and reward system.” So, while it seems that indeed a bonus system is in place, the extent to which HCWW use of this system is contributing toward performance improvements is difficult to discern. However, it is clear that WWSS support has not WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 16 yet contributed substantially toward achieving this result. ● Adopting a performance￾based incentive system based on the individual‟s performance and continuous education; Modified – Noting a lack of interest at HCWW in pursuing “market-based compensation plans,” the SOW was modified to drop this term and the Contractor adopted a focus on a “performance based incentive system” instead (referenced in the Contract‟s Deliverable Monitoring Matrix in the Year 4 Work Plan). Not yet achieved – The Contractor indicates progress toward this result via the support to establishment of an operator certification training program (activity HC 2 in Year 3) and development of a draft performance evaluation system for HCWW (activity HC 9 in Year 3). Both activities appear to be progressing well however rollout of the performance evaluation system is not yet complete. The last quarterly report indicates a planned rollout in early 2012. ● Implementing a long term Training Plan that is updated regularly; Not yet achieved – A host of activities are noted by the Contractor as having been pursued in relation to this expected result, including sponsoring of HCWW HR Management Diploma candidates (activity HC 12 in Year 2), training-of-trainers (TOT) program (CC 1 in Year 3), Advanced Management Seminar Series for Senior Utility Executives (CC 2 in Year 2), the AUC Utility Management Certificate Training program (CC 8), and a variety of other training efforts. Many of these activities are unlikely to have the long-lasting impact of an institutionally internalized commitment to training that is implied by this expected result. The training of trainer work and Certificate Training Program would appear to have the greatest potential for supporting such a commitment. The good progress in rolling out these activities is certainly encouraging but judging true achievement of this result is likely only to be possible after the end of WWSS support. USAID/Egypt staff explain that: “The Holding Company‟s HRD Division, co-located with the GIZ Water and Wastewater Program team, who is cooperating with them on a variety of tasks including the preparation of a system-wide training program, and assisting them in charting long-term career path for HCWW staff and subsidiary employees. It is our understanding that the division‟s plan is limited to a one-year training program. In agreement with the division, the assistance of WWSS was confined to the development of a strategy, and help in establishing a database for the system‟s employees.” WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 17 This information confirms our view that progress toward this “long-term” result has been limited to date even though what we judge to be productive support toward ultimately achieving this goal has been provided by the WWSS contractor. ● Carrying out Public Awareness Plans; Partially achieved – Several activities are noted by the Contractor as having been pursued in relation to this expected result, i.e., in Year 2 and 3 a corporate communication strategy was developed and the HCWW website was upgraded (under activity HC3), support to the HCWW public awareness unit and development of the HCWW press kit (HC 4), and development of a communications planning guide (CC6 in Year 2) in addition to other activities. The impact of some of this effort is noted in the end of Year 3 quarterly report where it is noted that HCWW and Cairo Water took into account the proactive approach laid out in the Corporate Communication Strategy in response to a major pipe burst in New Cairo during which “HCWW and Cairo Water formed a task force as soon as the pipe burst was reported, assigned a press manager on site around the clock, and dealt with the burst in a transparent and responsive manner. While press coverage undoubtedly presented the negative consequences of the burst, HCWW and Cairo Water were portrayed in the press as capable and trustworthy managers of water resources.” Such results notwithstanding, the Evaluation Team felt that from interviews with the Contractor and with utility staff that a clear, consistent, and proactive focus on customer communication on key issues appears to be lacking. HCWW and its subsidiaries do not yet appear to have prioritized consistent public outreach efforts on many important cost recovery issues such as timely payment of bills, cost of service sensitization, and importance of water conservations. This lack of focus appears to represent an important missed opportunity given the likely difficulty in launching such public outreach now in the current political environment. ● Issuing an official Service Disconnection Policy. Not yet achieved – During the third quarter of 2009 WWSS developed and delivered to the HCWW a draft Disconnection Policy to be implemented by subsidiaries upon HCWW approval. However, no such policy has been issued to date. It does not appear that the Contractor has taken further action in pursuit of this result. However, it should be noted that the draft Water Law (the development of which was supported by WRPP) does include some language regarding disconnection. It will likely be easier for HCWW to issue an official policy on disconnections when and if the Water Law is adopted. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 18 Improved performance of Subsidiary companies through: ● Having approved Business Plans; Partially achieved – Pursuit of this result has been a central focus of the Contractor‟s approach to implementation of WWSS. The 2011 WWSS PMP directly tracks this result by the milestone indicator 7 “Progress in developing and implementing subsidiary business plans.” The resulting positive achievements in development and updating of these plans demonstrates the Contractor‟s success in pursuing this focus. All but one (Sinai) of the 11 subsidiaries targeted for business plan support had completed a final draft by October 2011. Three of the 11 targeted have already completed an annual update of their plans and 37% of planned programs in the business plans have been completed (out of a target of 50%). However, it doesn‟t appear that the plans have developed specific performance targets that could be linked to performance agreements with the Holding Company or eventually EWRA. Thus, judging the impact of business plan implementation on the higher level WWSS objectives is difficult. ● Utilizing modern financial and management systems; Achieved – This result is one of the more vaguely defined in the scope of work. However, USAID and the contractor subsequently agreed to pursue this outcome by increasing the number of program-targeted subsidiaries (1) “producing quarterly financial and accounting statements” and (2) “using an automated [human resource] HR management system” – indicators 8 and 10 in the current PMP. All six subsidiaries receiving WWSS assistance in automating financial management (Assiut, Giza, Luxor, Matrouh, Sinai, and Sohag) are currently producing quarterly statements. However, it should be noted that all but one (Sinai) were already submitting such statements when WWSS support was initiated. In addition, USAID/Egypt staff note that: “…prior to the support of WWSS…, the [financial] reports prepared were for the most part manually prepared, often inaccurate and were not submitted in a timely fashion. The WWSS Program provided the utility financial management staff with in-depth training on the GOE‟s Unified Accounting System, helped standardize budget submissions, and helped automate the utility accounting systems using inexpensive proven off-the-shelf and user-friendly WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 19 accounting packages. This combined with regular technical assistance ensured a higher degree of accuracy and timely submission of financial statements.” Nine subsidiaries received WWSS assistance in automating HR management using the Advances Human Resources module; namely, Assiut, Aswan, Cairo, Giza, Luxor, Matrouh, Qena, Sinai, and Sohag. The other subsidiaries are not included because they already had automated HR management systems and were not supported by WWSS in this area. The Year 4 work plan indicates that all nine utilities are now using the module. The successful adoption of the automated systems by the targeted subsidiaries and the resulting quality improvement in financial reporting noted by USAID/Egypt indicate significant progress toward this expected result. However, the Evaluation Team finds it difficult to judge the exact scale of achievement and impact on higher level WWSS objectives given the lack of clear linkages between performance targets and use of such systems. ● Developing long term Tariff Plans for achieving cost recovery; Deleted from scope – Apparently because of the planned contributions by WPRR toward this result. ● Using compliance with Service Standards as a critical element in reviewing performance of companies. Level of achievement unclear - The Contractor‟s work to roll out a second version of the performance tracking software MARS indicates that HCWW has improved their capacity to utilize this tool for this purpose. Past weaknesses in the quality of the data reported under MARS also appears to be improving based on Evaluation Team interviews. Demonstrated application of this improved capacity to track compliance is unclear. There appears to have been some effort to link the WWSS efforts toward this result with the complementary WPRR work on Service Standard specification. However, given the still informal relationship between the Regulator and the utilities, responsibility for “enforcing” compliance with Service Standard criteria remains with HCWW. Given the unclear picture of service standard improvement, this arrangement does not appear to be effective in systematically improving service standards. A clear and well documented role and business relationship for the HCWW and its Subsidiaries Deleted from scope – The Contractor reported that HCWW was “not interested” in this result. Outsourcing of some functions options considered by Subsidiaries Partially achieved – The Evaluation Team found no project reporting indicating that this result was pursued by the Contractor. However, USAID/Egypt has informed us WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 20 that: “During the first year of the WWSS Program the technical team prepared, at the request of HCWW an inventory [of] potential functions and activities that could be outsourced, followed by requests to evaluate RFPs such as contracting out the management of wood forests reutilizing treated wastewater from select plants. Interest in [Public Private Partnerships] PPP, particularly the [Build Operate Transfer] BOT variety reappeared two years ago, when the team was asked to review and comment on a presentation presented to the former Minister of Housing. More recently, the Chairman of HCWW asked the WWSS Program for assistance in the contracting of O&M for wastewater treatment plants, and for outsourcing infrastructure rehabilitation. Working closely with HCWW and subsidiary procurement officers, the WWSS team prepared two manuals, one for contracting out O&M services and another one, advocating an IQC contracting mechanism to speed up the qualification, contract award and implementation of rehabilitation work. Both documents were well received, and presented to utility chairmen and their staff.” The Evaluation Team discussed the potential for outsourcing certain functions, such as bill collection, with subsidiary staff in Aswan and there appeared to be some interest in exploring such an approach. Thus, it appears that the goal of outsourcing options being “considered” by HCWW and subsidiaries has been achieved but no functions have actually been outsourced yet. It is unclear to the Evaluation Team why this result does not appear to have been pursued more rigorously. WWSS COMPONENT 1 FINDINGS The Contractor demonstrated a satisfactory effort in supporting HCWW and many of the subsidiaries in adopting tools envisioned as important to improving their performance under Component 1. All of the targeted subsidiaries are on track toward adopting the tools for which WWSS provided the most focused support, i.e., business planning and automated financial and human resource systems. However, the pursuit of some other expected results has been on a less consistent/focused basis or altogether abandoned, e.g., consumer orientated work such as establishment of a disconnection policy or targeted public awareness campaigns. As might be expected, there has been substantially less progress toward achieving the associated expected results. As the WWSS scope calls for the Contractor to be flexible in its technical support efforts, it is understandable that WWSS chose to prioritize pursuit of those results that were of WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 21 most interest to the utilities, and the Evaluation Team is hesitant to challenge the judgments used in making these prioritization choices. However, an overarching concern of the Team is the lack of evidence that WWSS efforts are collectively leading to the performance improvements that they were meant to support. Only a thorough quantitative analysis of key performance measures would allow a fair evaluation of the impact of the WWSS inputs. Unfortunately, well-validated and comprehensive data of this kind is lacking. Although MARS is tracking many key performance measures, only a limited subset of such data has been verified to date by the agreed auditing procedure. As such, a robust quantitative analysis of performance measures is not yet possible although we do present some preliminary analysis of available data in the following section of this report. Beyond the support provided in adopting the improved MARS program and their collection of six PMP performance related indicators, the WWSS Contractor seems to have placed too little emphasis on improving performance measurement to make robust performance analysis possible. COMPONENT 2 – ESTABLISHMENT OF NEW SUBSIDIARIES IN SELECTED GOVERNORATES Only 3 subsidiaries targeted as per SOW- Assiut, Luxor and Sohag TABLE 3: WWSS COMPONENT 2 RESULTS AGAINST SOW Expected Results Findings Subsidiary companies in selected governorates established and functioning using modern systems in all areas including: ● Financial planning; capital investment planning; staffing plans; accounting; management; organizational; technical; quality control and assurance; asset management; customer service; public awareness; tariff analyses and setting; program management; project and construction management; performance monitoring and evaluation; management information systems; management analyses and reporting systems; etc. Achieved – This very broad result is difficult to carefully assess especially as the WWSS PMP limits tracking of this result to indicator 8 “Number of program-targeted subsidiaries producing quarterly financial and accounting statements” and indicator 10 “Number of subsidiaries using an automated HR management system.” Good progress has been made according to these indicators. All three new subsidiaries are fully contributing to these indicators with the exception of Assiut that has yet to begin using the automated HR system. However, the last final report indicates that they too have adopted ADVAC HR. Other project reporting indicates that progress has been made in other areas as well. For example, the last Quarterly Report indicates that all three subsidiaries are using the asset management software MASTER and project management software PRiSM. The Team was very impressed by our visit to Sohag. This new subsidiary‟s articulate staff, impressive office arrangement, and customer service orientation would be the envy of many utilities around the world. The WWSS staff should be proud of their evident WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 22 contributions to this impressive startup. New subsidiary companies in selected governorates efficiently managed and have: ● Business plans; Achieved – The 2011 WWSS PMP directly tracks this result by the milestone indicator 7 “Progress in developing and implementing subsidiary business plans.” Each new subsidiary is noted to have developed and is updating regularly their business plan. ● Modern financial and management systems; Achieved – As noted above in the first result associated with this Component, such systems appear to be well in place. ● Long term Tariff Plans for achieving cost recovery; Achieved – It appears that subsidiary business plans have made efforts to include such planning. However, it is unclear what level of concrete efforts has been made by the subsidiaries to achieve the goal of cost recovery through tariff-based revenue generation. Given that the Contractor appears to have dropped any significant effort to support the subsidiaries in increasing revenue, any achievements under the Contracts appear to be superficial at best. ● Performance indicators based on meeting service standards; Level of achievement unclear – The same conclusions as the similar expected result in Component 1. The Contractor has supported roll out of a second version of the performance tracking software MARS with these new subsidiaries. However, demonstrated application of this improved capacity to track compliance is unclear. ● Progressive performance targets that are linked to a bonus pool or pay packages; Level of achievement unclear – Just as for Component 1, the Evaluation Team found no reference to this result in any project reporting documentation. However, use of bonuses was noted by staff from Sohag (one of the new subsidiaries receiving assistance under Component 2). ● Performance-based incentive system based on the individual‟s performance and continuous education; Modified – Noting a lack of interest at HCWW in pursuing “market-based compensation plans”, the SOW was modified to drop this term and the Contractor adopted a focus on a “performance based incentive system” instead. Not yet fully achieved – As noted in the Component 1 analysis efforts in this direction appear to be progressing well however rollout of a performance evaluation system is not yet complete. The last quarterly report indicates a planned rollout in early 2012. ● Long term Training Plans; Achieved – It appears that these subsidiaries‟ business plans have made efforts to include such planning. How effectively these plans are likely to be implemented however (as noted under Component 1) is a concern. The Contractor support to the Training Center in WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 23 Sohag is likely to be an important contribution to success in effective implementation if sufficient resources are allocated by the subsidiaries and/or HCWW. ● Public Awareness Plans; Achieved – It appears that the business plans for all three supported subsidiaries include such planning. ● Official service disconnection policy. Not yet achieved – During the third quarter of 2009 WWSS developed and delivered to the HCWW a draft disconnection policy to be implemented by subsidiaries upon HCWW approval. However, to-date no such policy has been issued. It does not appear that the Contractor has pursued this result further. Interviews in Sohag did not indicate progress in adopting such a policy. WWSS COMPONENT 2 FINDINGS The achievements of WWSS in support of the new subsidiaries are very impressive. Although the Evaluation Team was only able to visit one of the three subsidiaries, if Sohag is representative of their inputs into the other two, USAID should be very pleased with the result. Of course, just as with Component 1, it is difficult to objectively judge the impact of the contributions given the lack of quality performance data over time. Making such a judgment on the “functionality” and “effective management” among these three subsidiaries is even more difficult because of limitations on data availability, e.g., data for Sohag is only available for one year. However, it is notable that significant improvements in all six performance related PMP indicators were noted in Luxor between the first year reported (2008/09) and 2009/2010. Percent of fees collected increased by 5%, water losses decreased by 3%, percent of volume of water produced that is billed increased by 3%, percent in energy cost per cubic meter decreased by an impressive 71%, 100% of samples continued to meet standards, and the percent of functioning meters increased from 94 to 95%. All this occurred at a time when most other utilities were experiencing difficulties because of the political turmoil. These achievements in Luxor may be one of the clearest examples of WWSS impact on performance improvements. COMPONENT 3 – CAPITAL INVESTMENT PLANNING AND PROGRAM/PROJECT MANAGEMENT 12 subsidiaries originally targeted under SOW - Aswan, Assiut, Beni Suef, Cairo Water, Daqahliya, Giza, Luxor, Matrouh, Menufiya, Minya, Qena, Sohag – Sinai added in November 2010 TABLE 4: WWSS COMPONENT 3 – RESULTS AGAINST SOW Expected Results Findings The HCWW and selected subsidiaries have: ● A comprehensive and strategic approach to capital investment planning; program management; and project/construction management; strategic capital investments decision-making; Partially Achieved – Through activities such as “Support subsidiaries‟ capital investment planning efforts” (activity CC 7 – Support Subsidiaries' Capital Investment Planning Efforts - in Year 3), this result appears to have been partially achieved. Aided by a WWSS-developed capital investment planning manual WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 24 and regional seminars on the subject for utility decision-makers, nearly all participating subsidiaries completed a work plan for capital investments based on their master plans. The Contractor provided ongoing support to subsidiaries through training in hydraulic modeling, and on-the-job training on updating master plans, using the existing master plans. Critically, however, this result was not achieved within HCWW and was not a major focus of WWSS efforts. Although there has been some ongoing engagement with HCWW staff on related activities as noted by USAID/Egypt: “…the technical staff of the Holding Company reviews and comments on the [capital investment planning and program management] technical material prepared by the WWSS Program, and actively participates in most orientation programs and workshops targeting upper and mid-level managers at the HQ and subsidiaries. Mid-level and entry-level HCWW employees also participate in project-sponsored training activities and often participate in field visits.” ● More effective allocations of limited budgetary resources for water and wastewater infrastructure. Achieved – While not documented in program reporting, achievement of this result was noted by several sector stakeholders during interviews. In particular, it was noted that in the post-revolutionary, restricted budget situation the ability to use PRiSM in prioritizing allocation of limited resources to key projects was extremely valuable, at least at the Ministry level. HCWW and selected Subsidiaries‟ have: ● Annual Budgeting process that is based on priorities established in the Master Planning Process Partially Achieved – Given the continued reliance on subsidies from the Ministry of Finance to cover operating and maintenance budgets for all target subsidiaries, it would appear that achieving this result would entail allocating this subsidy between subsidiaries according to these established priorities. Given the Evaluation Team‟s understanding of how the subsidies are being distributed, i.e., seemingly without any correlation to subsidiary budgetary needs, it does not appear that HCWW has developed a coherent link between this annual budgeting process and the Master Plans. Meanwhile, it does appear that WWSS assistance has equipped subsidiaries with the tools they need to WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 25 effectively use the limited resources that they do receive. ● Long and medium term capital investment plans which address strategic development priorities, required financial resources, and cash flow requirements; Partially Achieved – As noted above, “planning” improvements were achieved under WWSS among targeted subsidiaries. However, it does not appear that HCWW in its role as distributor of the required “cash flow”/subsidy to fund these plans was supported in developing complementary plans to fund the subsidiary level plans. USAID/Egypt notes that HCWW distributes allocations from Ministry of Finance for rehabilitation work “based on an equitable „allocation formula,‟ which is often over-ridden by urgent needs.” ● The Program Management Monitoring and Evaluation System (PMMES) program well maintained and functional and PMU properly staffed, properly equipped, and functioning efficiently. Achieved – Program reporting and interviews suggest that this result has been achieved. However, it is difficult for the Evaluation Team to judge the quality of this achievement, i.e., to what extent the PMU is “properly staffed” or equipped. HCWW‟s institutional and technical capacity to effectively monitor and manage the water and wastewater capital investment planning strengthened, and project cycle (comprising of identification, appraisal, preparation, prioritization, implementation, management, monitoring, evaluation, startup. etc.) identified and properly managed. Partially Achieved – It would appear that capacity has largely been built in the area of project management rather than in investment planning. Training on PRiSM and support in the development of a PMU brought valuable improvements to the project management capacity. However, the project planning process at the HCWW level was secondary in focus and less successful. It should be noted that the Holding Company oversaw a European Union-funded, nation-wide master planning exercise conducted by consultants and finalized in 2009. With the exception of capital investment projects funded by the EU and the World Bank in the Delta region, HCWW plays little role in investment planning. The staff of the Holding Company reviewed and commented on the WWSS-produced manuals related to capital investment planning and program management, and requested WWSS to include all utilities in their training activities in these areas. WWSS COMPONENT 3 FINDINGS Like Component 1, it appears that WWSS has been successful in putting in place many of the tools necessary to achieve the implied result of more effective capital investment planning and management by HCWW and its subsidiaries. But also like Component 1, it is difficult to judge the extent to which these tools are being effectively used. Especially difficult is judging the extent to which the capital investment planning is “comprehensive and strategic” given the information available to the Team. Of even more concern is the fact that subsidiaries and HCWW have such limited direct access to funding for capital investments. As noted in the analysis of WPRR results, these decisions WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 26 remain highly centralized and are not particularly responsive to any planning done by HCWW or much less the subsidiaries. It seems appropriate that WWSS has focused its efforts under this component on “capital replacement,” a level of investment seemingly more within the manageable interests of both HCWW and the subsidiaries given their current revenues and the scale of their availability subsidy from the Ministry of Finance. One particular contribution in line with these efforts noted by USAID/Egypt includes: “At the request of the HCWW Chairman, the WWSS Program developed an approach merging the [Indefinite Quantity Contracting] IQC model with the GOE‟s annual contracting system to facilitate and speed up the outsourcing of rehabilitation and expansion work.” This contribution would appear helpful for increasingly timely contracting of capital improvements. However, it does seem that WWSS could have done more to build a systematic process around allocation of the available subsidy for capital replacement through increased attention to linking subsidiary Master Planning with HCWW subsidy distribution decisions. COMPONENT 4 – STAFF DEVELOPMENT/PROFESSIONALIZATION 11 subsidiaries originally targeted under SOW - Aswan, Assiut, Beni Suef, Cairo Water, Giza, Luxor, Matrouh, Menufiya, Minya, Qena, Sohag - Sinai added in November 2010 TABLE 5: WWSS COMPONENT 4 RESULTS AGAINST SOW Expected Results Findings The HCWW and selected subsidiaries have comprehensive human resource development plans that respond to the needs and present an integral part of the promotion system and compensation plan. Partially Achieved – The Team feels that the combination of extensive and successful support on HR planning and staff training goes a long way in achieving this result. However, it is unclear how the performance-based compensation is applied, thus confirming that more work is needed for full achievement. USAID/Egypt supports this conclusion by noting that: “…more work is needed. When the WWSS Program started in late 2008, HR divisions did not even exist at the utilities; they had only payroll departments (usually as part of their finance and administration divisions) and training departments (located elsewhere in the organization). The concept of Human Resources Development and Human Resources Management – i.e., having a system in place to attract, retain, and develop staff – is still new in the sector, so WWSS focused on the basics first: developing clear mandates for HR divisions along with step-by-step plans and long￾term staffing plans taking into account future needs and potential attrition. After developing mandates and job descriptions for all divisions at the companies, the next step, as the evaluators WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 27 note, is to link the departmental mandates and individual job descriptions to a performance-based compensation system.” The HCWW has a well-managed world class training institution that is tasked with developing technical and managerial capabilities of its staff to meet existing and future needs. Achieved – It appears that the strategy used to achieve this result shifted from developing an in-house institution to relying on a mix of in-house training facilities and outside relationships with established institutions to satisfy training/staff development needs, i.e., training via the subsidiary/regional training centers and senior management training through the American University of Cairo (AUC). USAID/Egypt highlights WWSS contributions to achievement of this result by noting the project‟s support for in-house training: “…the WWSS team…established a training center at the Sohag Water and Wastewater Company, and is equipping training facilities in North and South Sinai. Following equipment installation, WWSS is assisting these three facilities to develop and document their mandate, position descriptions, and training center procedures, and stock their library with WWSS and other training course materials.” HCWW needs to sustain both types of training and continue to develop the types of supportive relationships that they have with AUC. The HCWW and selected subsidiaries have clear plans for participation in international study tours and events. Achieved – Such plans were formulated and have largely been executed as part of the WWSS support. However, there is no indication that these plans will extend beyond the life of the WWSS project assistance. WWSS COMPONENT 4 FINDINGS There is no doubt that there have been great advances in the development of both HCWW and subsidiary staff as a result of ongoing USAID support under WWSS. However, for the time being, staff development is likely to only be plausibly supported via donor or direct government support rather than through individuals investing in their own professional development. Given the strained financial situation of the GOE, it is unclear to what extent staff development plans can be supported in the short term. Thus, the long-term impacts from contributions from WWSS under Component 4 are fragile. However, the Evaluation Team does appreciate the observation from USAID/Egypt that WWSS activities on human resource development “go beyond mere training” and have worked to institutionalize continued progress in this result area. The Mission helpfully points out that: WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 28 “Focusing first on the companies‟ new HR departments, the WWSS trained them in HR concepts, facilitated the development of HR plans, and continues to provide technical assistance in implementing the plans and monitoring the extent to which the companies are implementing the plans themselves. This technical assistance (on staffing plans, organizational charts, job descriptions, etc.) is showing signs of sustainability at the utilities, as documented in the quarterly assessment results. Nascent HR departments now understand their mandate, have a clear plan for its implementation, and have taken significant steps in that direction. Further, WWSS training programs are not stand-alone, but are performance-based and linked to activities in the utilities‟ business plans. The contractor worked with the companies to identify problems (e.g., high electricity bills), and pilot technical solutions to the problems. These technical solutions were linked to training programs – both technical training and TOT - to create local teams capable of rolling-out the programs within the companies. Following the training of trainers, WWSS is further facilitating roll￾out and institutionalization of activities by supporting the development of locally-led training plans, and monitoring ongoing training activities and implementation of technical activities.” Excerpt from WWSS assessment, related to subsidiary HR plans:  In Aswan: During the past quarter (Oct. – Dec. 2011), the Company worked with WWSS advisors to update their organizational structure, which was subsequently reviewed and approved by the Chairman, integrated with the ADVAC HR system, and applied to various departments. Further, with WWSS support, Company staff initiated development of division/department mandates according to the new structure....  In Sohag: The Company reports active implementation of its HR Plan, developed with WWSS assistance, through the development of an organizational structure and job descriptions, employee database, and improvement of their training center, among other activities.  In Sinai: The HR staff is actively implementing the WWSS-supported HR plan, the priorities of which are reflected in their business plan and joint WWSS action plan. They are currently focusing, with WWSS assistance, on developing their organizational structure, decentralizing some company functions, and establishing training centers in North and South Sinai. ASSESSMENT OF WWSS RESULTS USING THE PMP In addition to the above analysis of WWSS contributions to achievement of the contract scope of work specified expected results, we present below an analysis of the results as expressed by the agreed upon results framework for WWSS and measured by the associated quantitative indicators and qualitative analysis on which the Contractor reports according to the WWSS Performance Monitoring Plan (PMP). The WWSS PMP defines a results framework with the highest level program objective (PO) being “Financial and commercial viability of selected utilities increased.” Relying on the development hypothesis implicit in the design of the WWSS, the results framework further defines success in achieving this program objective as being dependent on progress toward three program intermediate results (PIRs): WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 29 PIR 1) Management and operational efficiency increased; PIR 2) Capital investment planning and program management improved; and, PIR 3) Capacity of staff in targeted institutions increased. The PMP further defines 11 indicators that were designed to track progress toward achievement of the program objective and these intermediate results. Each indicator is reported on according to the targeted subsidiary supported by WWSS. However, because WWSS type of support for each of the 13 targeted subsidiaries varies, only one of the indicators is reported on for all 13 subsidiaries, i.e., number of people trained. Reporting is most limited for Cairo (only indicator 9 and 11) because according to the PMP “due to its size and advanced state of development, [WWSS] activities there are limited to those requested specifically by the Chairman, which to date have not involved assistance in meter repair or other commercial functions” and Daqahliya (only indicator 11) because “per the WWSS contract scope of work, activities there are limited to the Capital Investment Planning and Program Management component.” TABLE 6: WWSS PMP INDICATORS # Indicator Measuring # Subsidiaries Reported On 1 Percent of collection PO 11 2 Percent increase/decrease in water loss PIR1 11 3 Percent of volume of water produced that is billed PIR1 11 4 Percent increase/decrease in energy cost per cubic meter (water) PIR1 11 5 Percent of samples meeting Egyptian water quality standards PIR1 11 6 Percent of [water] meters functioning in targeted areas PIR1 11 7 Progress in developing and implementing subsidiary business plans (milestone indicator) PIR1 11 8 Number of program-targeted subsidiaries producing quarterly financial and accounting statements PIR1 6 9 Number of subsidiary HR plans developed PIR1 12 10 Number of subsidiaries using an automated HR management system PIR1 9 11 Number of people trained PIR3 13 Given that the most recently approved PMP (October 30, 2011) for the WWSS contract substantially redefined the set of indicators used to track progress toward the framework results, it is currently difficult to interpret the majority of reported indicator levels against agreed expectations. The PMP lists targets for the final two years of the project (fiscal years 2010/11 and 2011/12) for all indicators. It also presents observed indicator levels (“actuals”) or initial observed indicator levels (“baselines”)3 for fiscal years (FY) 2007/2008, 2008/2009, 3 The baseline period for Aswan, Beni Suef, Menufiya, Minya, and Qena utilities is 2007/2008, which is the period directly before the WWSS project started. For Giza, Luxor, and Matrouh, which were established as WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 30 2009/2010. However, the PMP only presents comparisons between actuals and targets for indicators 7 through 11. Given these limitations, we use the available comparative numbers for the final five indicators and observed trends in the first seven indicators as indicative of WWSS achievements to date. PROGRESS TOWARD INCREASED MANAGEMENT AND OPERATIONAL EFFICIENCY (PIR1) This result is the one most extensively tracked by PMP indicators, i.e., nine of the eleven indicators are identified as tracking progress toward this result. Five of these indicators only report actuals over the past five years with baselines (initial values) varying between FY 2007/8 and FY 2009/10. WWSS reports indicator values for 11 subsidiaries where the most extensive technical support has been provided under the project. However, because values are only reported for one year (2009/10) for three subsidiaries, we can only discern trends for eight of the targeted subsidiaries. TABLE 7: TRENDS IN WWSS PIR I INDICATORS (2007/8 OR LATER BASELINE TO 2009/10) # Indicator Baseline Average Subsidiaries Improved Average Improvement Largest Improvement 2 Percent increase/decrease in water loss 34% 6 of 8 reporting 2% decrease Qena 13% decrease 3 Percent of volume of water produced that is billed 66% 4 of 8 reporting 1.5% higher Qena 12% higher 4 Percent increase/decrease in energy cost per cubic meter (water) 0.18 energy costs/m3 4 of 8 reporting Did not improve 6% increase Menufiya 30% decrease 5 Percent of samples meeting Egyptian water quality standards 90% 4 of 8 reporting 1.5% higher Qena 10% more 6 Percent of [water] meters functioning in targeted areas 80% 6 of 8 reporting 11% more Matrouh 48% more The most consistent progress has been achieved in Menufiya and Minya where improvements were achieved according to all five indicators. Qena also made notable advances in all indicator areas with the exception of its energy costs per volume of water produced. Giza, Luxor and Matrouh made less notable progress, i.e., improvements in only two of the five indicators were observed. Beni Suef did not see progress in any of the observed areas of indicator variables. It should be noted that the Contractor admits in the PMP that data quality associated with these indicators is unreliable. The PMP specifies several strategies that the Contractor is pursuing to verify this data but based on a review of similar performance data reported by WPRR with many obvious errors and implausible values, we can only speculate as to this data‟s accuracy. The other four indicators tracking PIR1 are measures of targeted subsidiaries‟ adoption of WWSS supported management tools (see Table 8). companies in FY 2007/2008, the first complete year of data available is 2008/2009 and therefore, this is their baseline year. Assiut and Sohag were established as companies in FY 2008/2009, and thus have incomplete data for that fiscal year; the baseline for those companies is accordingly 2009/2010. Sinai was added to the contract in November, 2010, so its baseline is also the 2009/2010 fiscal year. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 31 TABLE 8: WWSS PIR1 INDICATORS (ACTUALS VS. LIFE OF PROJECT TARGETS) # Indicator Targeted # Subsidiaries Actuals (as of FY2009/10) Slowest Progress 7 Progress in developing and implementing subsidiary business plans (milestone indicator) Assiut and Matrouh have yet to complete  All “development” milestones complete 11 3 any programs  Targeted # of programs complete 10 3 8 Number of program-targeted subsidiaries producing quarterly financial and accounting statements 6 6 None 9 Number of subsidiary HR plans developed 11 9 Assiut behind schedule 10 Number of subsidiaries using an automated HR management system 9 6 Aswan and Cairo behind schedule Significant progress has been made toward adoption of these tools in most of the targeted subsidiaries. While only three of eleven subsidiaries have achieved all the business plan-related milestones and completed the number of targeted programs under these business plans, it seems that most all the subsidiaries are on track to do so by the end of the project. The same can be said for the other three indicators. In addition to reporting on these quantitative indicators, WWSS quarterly reporting contains descriptive information on activities undertaken in cooperation with targeted subsidiaries and some of the achievements associated with them. The most concrete examples of management and operational efficiency improvements are associated with the pilot-scale energy and chemical conservation and leak detection work conducted in cooperation with a number of targeted utilities. Examples include: Luxor: The power factor correction panel installed at the Luxor water treatment plant and one additional feeder is reported to have resulted in a savings of LE 47,000 in penalties charged from 2009/2010 to 2010/2011. The Company has analyzed three additional plants using the WWSS-provided power factor analyzer, and is preparing the technical specifications for a request for proposals to purchase additional correction panels. Qena: Company staff continues to report reductions in chemical consumption as well as improvements in water quality from applying the concepts from the WWSS-provided chemical use optimization training. They have applied the concepts and adjusted dosing at plants in Salheya, Deshna, Naga‟a Hamady, and Qeft. In Salheya, they used 21.6 tons of alum in June 2011, compared with 40 tons in June 2010, by applying the jar test methodology for optimizing dosage (from 15 mgm/L to 8 mgm/L). This resulted in a savings of LE 33,000. Sohag: After the WWSS-supported pilot and training of trainer program in leak detection/reduction, the Sohag subsidiary has maintained an active leak detection program. They expanded the program in Akhmim City after the WWSS pilot, and, over a period of 1.5 months, reported a reduction in leakage there from 35% to 25% (17% commercial losses and 8% technical), and a savings of 110,460 m3 of water, valued at LE 27,615. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 32 Minya: The WWSS-supported leak detection pilot in the Sultana area was completed, and the Company reports a reduction of leaked water there from 61% to 32%. The Company‟s leak detection team then applied the same techniques in the New Minya area, and succeeded in reducing leakage there from 69% to 32%. These and other such improvements demonstrate that WWSS programs have not only had some direct impact on operational efficiency but are having some success in being institutionalized in the utilities, a critical step in achieving sustainable performance improvement. However, these notable results from the pilot activities are not significant enough to convince the Evaluation Team that overall achievements under PIR1 to date have been more than marginal. The first five non-direct program indicators either (1) do not yet provide insights on progress in targeted utilities, or (2) indicate that achievements have been both inconsistent between subsidiaries and generally small in scale. It is encouraging that the four indicators tracking uptake of program inputs are all making good progress. However, there is not yet good evidence that adoption of these tools is leading to significant improvement in subsidiary operational efficiency. PROGRESS TOWARD IMPROVED CAPITAL INVESTMENT PLANNING/PROGRAM MANAGEMENT (PIR2) There are no PMP indicators that directly relate to this result beyond the extent to which business planning achievements (Indicator 7) can be seen to have contributed to improved capital investment planning among targeted subsidiaries. Qualitative results reported in recent quarterly reports do not add any particular insights. PROGRESS TOWARD INCREASED CAPACITY OF STAFF IN TARGETED INSTITUTIONS (PIR3) The WWSS PMP does not use a direct measure of increased staff capacity, e.g., a test-based or performance-based metric. Instead the PMP tracks the number of persons trained by the Contractor as part of the program. TABLE 9: WWSS PIR3 INDICATOR (STAFF TRAINED) Fiscal Year Number Trained 2008/9 393 Actual 2009/10 1,119 Actual 2010/11 1,801 Actual 2011/12 890 Target TOTAL 4,203 An impressive number of targeted beneficiary staff has been trained by WWSS to date. Beyond the PMP 2010/11 target of 1,130, project reporting does not indicate an initially targeted life of project number of trainees. As such, beyond the substantially higher than targeted result in 2010/11, it is not possible to discern if the amount of training actually provided is in line with what was targeted. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 33 PROGRESS TOWARD INCREASED UTILITY FINANCIAL AND COMMERCIAL VIABILITY (PO) The sole indicator identified in the PMP to track the overarching WWSS program objective is “the proportion of the total outstanding billed amount that is collected during the reporting period,” i.e., the total value of billings collected during the reporting period divided by the value of all bills issued during the reporting period plus the value of outstanding balances. The data for this indicator is reported in the same way as for indicators two through six of the PMP, i.e., multiple year data only available for eight of the 11 targeted subsidiaries and no historic reporting of actuals compared to targets. Available historic data for this indicator are listed in the table below. TABLE 10: WWSS PO INDICATOR (PERCENT OF COLLECTION) Subsidiary FY 2007/08 FY 2008/09 FY 2009/10 Increase/decrease from baseline Assiut N/A N/A 26% N/A Aswan 34% 50% 51% 17% Beni Suef 88% 87% 87% -1% Giza N/A 60% 20% -40% Luxor N/A 54% 59% 5% Matrouh N/A 32% 30% -2% Menufiya 50% 75% 75% 25% Minya 55% 64% 74% 19% Qena 53% 44% 50% -3% Sinai N/A N/A N/A N/A Sohag N/A N/A 65% N/A Average 56% 58% 54% 2.5% The available data associated with this indicator is similar to that for the other PMP indicators that track non-WWSS program related aspects of subsidiary performance under PIR1. Specifically: 1. Mixed results observed among subsidiaries, i.e., half report improving the percent of collections and half report decreases; 2. Large difference in reported trends, e.g., Giza reports a massive decrease in collection (40% lower from 2008 to 2009) while Menufiya reports a somewhat implausible one year increase in collections (from 50% to 75%); 3. On average a slight improvement over the individual subsidiary baselines. Taken as a whole, this PO indicator and the other performance-related PIR1 indicators point to no clear trend in progress toward program results among the targeted subsidiaries. Instead, it appears likely that there is a subsidiary specific “story” behind changes in many of these indicators. Unfortunately, given that WWSS reporting focuses largely on outputs and outcomes directly related to Contractor inputs (just as PMP indicators seven to eleven largely do), it is difficult for the Evaluation Team to understand to what extent WWSS contributions have positively or negatively affected these individual “stories”, i.e., counteracted negative trends or promoted positive trends in the more objective performance measures in the PMP. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 34 The Evaluation Team attempted one additional method of objectively assessing WWSS contributions toward program results. We analyzed subsidiary “level of service” data reported by WPRR for each individual Egyptian water and wastewater subsidiary and compared the recent improvements in service between WWSS-targeted subsidiaries and those not benefiting from this assistance. Given the questionable quality of the data and the many assumptions that we needed to make in this analysis, we do not include it in the body of the report and instead include it as Annex G. SUMMARY OF WWSS RELATED FINDINGS WWSS has been successful in delivering on many of the expected results under the contract SOW components. Through flexible and responsive technical support, the Contractor has helped to directly contribute toward cost-effective service delivery improvements through energy efficiency improvement pilots, chemical optimization, and leak detection. However, there are two important weaknesses in WWSS implementation to date: 1. A lack of clear documentation of the impact of WWSS efforts on subsidiary-wide performance improvements. 2. No clear indication of how the multiple management “tools” (especially the multiple computer software tools) that have been adopted by targeted subsidiaries are resulting in performance improvement and sustained evolution of the utilities into more cost-effective organizations. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 35 5. WATER POLICY AND REGULATORY REFORM PROJECT (WPRR) According to the WPRR contract scope of work (SOW), the overall objectives of the program are to support the GOE to: ● Strengthen the policy, legal and regulatory framework for the water and wastewater sector in Egypt, and ● Improve the quality of water and wastewater services and extend them to new customers To achieve this broad objective, the WPRR contract SOW required the Contractor to provide a variety of services under six specified project components: 1. Policy and Legal Reform 2. Regulatory Reform 3. Water and Wastewater Operators Certification 4. Public-Private Partnerships (PPPs) 5. Capital Investment Planning 6. Program Management The SOW described “specific results” to be achieved under each component and a minimum set of tasks and deliverables required to achieve the expected results. The WPRR SOW was modified in December 2010 to increase the contract value from $12.9 to $17.4 million and to expand the scope to include additional support for PPP transactions, enhance the ability of MHUUD to manage capital investments through increased audit capacity, support the GOE Program Tracking System for Targeting Poverty project which aims to provide infrastructure to low income residents, and increase funding for certain WPRR project staff positions. TRACKING WPRR RESULTS The SOW requires the Contractor to clearly demonstrate that it has completed all related deliverables and exerted “reasonable effort” to ensure that the specified results are achieved. The SOW further specifies that the Contractor‟s “performance shall be evaluated based on the completion of specific tasks as outlined in the Task Order, adherence to the work plan, and reports submitted to the Task Order Cognizant Technical Officer (TOCTO).” The latest version of the WPRR PMP (approved June 2011) presents a new Results Framework (Figure 3) based on the WPRR assigned portions of the overall USAID Water Sector Results Framework (Figure 1). The only change in the WPRR Results Framework is a renumbering of the PIRs. Based on this revised Results Framework, the Contractor proposed tracking the achievement of the results using 13 output and outcome indicators–six less than the original 19 indicators proposed under the previous joint PMP. The Contractor notes that the “joint PMP contained indicators that were outside of the manageable interest of the WPRR project, and these indicators have therefore been removed. Other indicators have also been adjusted to better align with the modified approach to certain project activities as a result of the revolution and current political situation in Egypt.” One of the 13 indicators (percentage of Operation and Maintenance costs recovered by the targeted subsidiaries) is also cited as falling outside the WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 36 impact of the WPRR project; however, the data was collected for the purpose of overall reporting on USAID standard indicators. FIGURE 3: WPRR RESULTS FRAMEWORK (MID 2011) The Evaluation Team’s analysis of WPRR results is presented in two sections. The first examines the level of achievement of the project in achieving the “expected results” listed in the original scope of work. In the contract scope of work these results are linked to specific tasks and deliverables but are expressed in terms of the outcomes that these tasks and deliverables are meant to effect. The second section presents an analysis of achievements using the results framework and related PMP indicators as the basis for this assessment. These two sections provide two different perspectives on achievement of the overall project objectives. ACHIEVEMENT OF EXPECTED RESULTS BY COMPONENTS Our initial findings are organized by the expected results listed for each of the components of the original scope of work. These findings are based on a review of the most recent quarterly and annual reports and information provided during the interviews conducted by the Team. COMPONENT 1: POLICY AND LEGAL REFORM Under this component, the following tasks were undertaken: Assisting Ministry of Housing, Utilities, and Urban Communities (MHUUD) to develop appropriate policies, laws, and implementing regulations based on a well-defined, strategic plan for the sector. Specifically, to: ● Help MHUUD to develop a comprehensive strategy which addresses corporate governance and includes a more sustainable plan for financing operating and capital costs ● Help promote tariff reform, improve subsidy targeting to the poor, and strengthen organizational structure ● Help the GOE to draft the new Water Law and, upon passage of the new law, support MHUUD in implementing the law WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 37 TABLE 11: WPRR COMPONENT 1 RESULTS AGAINST SOW Expected Results for this Component Findings Clear roles and responsibilities for sector organizations defined. Partially achieved/hindered by external factors - Several WPRR activities were designed to contribute to this result including the drafting of the Water Policy, drafting of the Water Law and the current drafting of the Water and Wastewater Strategy. While the policy was approved in September 2010, passage of the Water Law has been further delayed by the change in Government. The sector strategy is under development. The package of documents – policy, law and strategy – will provide increased definition to the sector and help to articulate expectations and lines of accountability. It appears that the law would have passed if not for external events and the previous draft was greatly strengthened by the work of WPRR. Financing for the water and wastewater sector increased. Not yet achieved – Examining the tasks required under this component of WPRR shows that there are several tasks with the potential to develop a more sustainable, transparent and targeted financing plan for the sector, although not necessarily linked to “increased” financing. The relevant tasks are (1) development of a GOE sector policy, law and strategy that commit to sustainable financing, and (2) support of the promotion of tariff reform and improved subsidy targeting. The water sector strategy and the tariff analysis, in particular, will set out a path toward more transparent sector financing. It could also be argued that additional clarity in the sector roles and responsibilities would result in increased private investment allowing for additional or improved services. However, these tasks have not yet been completed. As a result, there is an unfinished dialogue regarding the political commitment to increased cost recovery through consumer charges versus the GOE‟s willingness to continue blanket subsidies to the sector. It cannot then be stated that the component has resulted yet in “increased financing” to the sector either in terms of additional capital or in terms of a more sustainable balance of financing. Interviews indicated that the amount of GOE subsidies to the sector has been flat while revenue collection has been declining. Neither WWSS nor WPRR focuses on revenue collection. The 2011 political events certainly diverted the GOE‟s attention from the sector agenda, but it does not appear that a clear consensus on cost recovery versus subsidy had emerged even prior to these events. Component 1, when fully accomplished, does, however, lay the groundwork for improved financial sustainability in the sector should the GOE chose to implement the law, strategy, and tariff reform. Further, the strategy and tariff analysis in particular offer the opportunity for a concrete discussion on the relative value of different WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 38 financing approaches. The draft strategy, which has undergone initial discussion with the GOE, will set out a range of financing scenarios for the sector. There is also a comment made that the increased salaries are not covered by a commensurate increase in revenues. It isn‟t clear whether this comment refers to salary increases post 2011 or what data supports this conclusion. Tariff reform to improve cost recovery. Not yet achieved - The WPRR is undertaking tariff reform through cost of service studies and analysis of tariff structure and adjustment procedures as described under Component 2. This task refers more specifically to support to the Policy Advisory Unit of the regulator, including helping to develop EWRA‟s capacity to perform analysis and to advise the Ministry – a task which is also included in Component 2. While tariff reform per se has not yet been achieved, WPRR has provided extensive support to EWRA, complementing the work of GIZ. The premise behind the MHUUD/EWRA support for the customer classification/tariff study is to be ready to make recommendations on tariff reform once the new government is in place. EWRA will have improved tools and information at its disposal as a result of the project, but it is less clear that EWRA is ready to advocate for tariff reform or to effectively set out the case for reform based on evidence. WPRR has, though, provided training to EWRA in such areas as Advanced Rate Setting, but the regulator so far has used the training only in pilot applications. WPRR did develop a paper on a Cost Recovery Policy with EWRA which provides a good overview of cost recovery and tariff approaches in Egypt and elsewhere. The paper should be valuable when steps are taken to implement reform. WPRR COMPONENT 1 FINDINGS WPRR has made satisfactory progress in developing the suite of policy, legal, regulatory and strategic documents which will be critical to implementing reform. When and if the next stage of sector reform is fully pursued (focusing on accountability, governance and cost recovery), the WPRR work will be valuable. However, it is less clear that the GOE has fully embraced the reform concepts embodied in the documents as there are no measurable results apart from acceptance of the Water Law. There are champions within the sector but not a widespread understanding of and commitment to the reform agenda. EWRA, meanwhile, has been operating in a largely theoretical context since the project began – it has not yet been able to fully test its knowledge and skills through practical application. Given the political events of 2011, the water reform agenda, stakeholders agree, will be shelved in favor of more pressing issues. WPRR should continue with the strategy process, drawing together the different strands of analysis which have been undertaken in WPRR (and WWSS) in order to leave a blueprint for sector reform when the GOE is able to re-focus. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 39 WPRR should examine the extent to which it continues to invest in EWRA training, support and skill development and focus on practical application where feasible. WPRR should continue to step back and encourage EWRA to test its decision-making and try out its skills as it has done in relation to the tariff application pilots. COMPONENT 2: REGULATORY REFORM Under this component, the following tasks were undertaken to support EWRA in improving economic and service regulation of the sector: • Develop a licensing system including institutional arrangements and supporting protocols/systems. • Draft a plan and schedule to adjust water and wastewater tariffs, and build capacity within EWRA to utilize tariff analysis and modeling tools. • Develop model performance agreements, and implement an information system to monitor the compliance of service providers with regulations. • Develop guidelines and systems for appropriate performance incentives for operations of water and wastewater facilities that encourage increased rates of revenue collection. • Review existing proposals for service standards and advise on refining them as appropriate. • Solicit and incorporate feedback on proposed standards from customers and the Holding Company. • Develop regulations which outline (a) key performance areas and minimum performance levels; (b) measurement, recordkeeping, and reporting systems and procedures to be applied by the companies to monitor their performance against the standards, and (c) EWRA enforcement measures. • Develop a strategy for enforcing service standards, and corresponding penalties for noncompliance. • Develop audit methods to monitor the quality of data provided by utilities in the monitoring system. TABLE 12: WPRR COMPONENT 2 RESULTS AGAINST SOW Expected Results per the SOW Findings Clearly defined responsibilities for regulating service providers. Not yet achieved – EWRA was established by Presidential Decree in 2004. However, it will not have full legal authority until the Water Law is passed. The first draft of the Water Law was completed in early 2007, during a previous USAID project, and extensively improved upon by WPRR. In 2009 a draft Water Law was approved by the High Water Committee which defines the regulatory responsibilities and duties of EWRA. Later in 2010, this draft was approved by the MHUUD and sent to the Cabinet for its approval. To become law, it must be passed by Parliament and signed by the President. To date this has not happened. EWRA has thus operated for at least 6 years without full definition and authority. The longer the sector functions and elements of reform continue without the contribution of the regulator, the greater the likelihood that questions will be raised regarding the WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 40 necessity of a sector regulator. In interviews, some stakeholders already questioned the value and role of EWRA. The body of work related to regulatory reform is somewhat diminished without this core legal stature. It is understandable that no action was taken post January 2011. The major progress from 2009-2011 was the presentation of the draft Water Law to the Minister of MHUUD and to the Prime Minister. Given the slowness of the legislative process, which was outside the control of WPRR, it isn‟t clear whether WPRR was able to identify other options for bringing this institutional clarity to the sector through other legal options or interim arrangements. Standard, uniform and equitable regulations established which improve the recovery of operating expenses and capital debt. Not yet achieved – EWRA is not yet in a legal position to issue regulations and is still collecting data and performing analysis to inform economic regulation. The Cost Recovery Policy Paper and the legal paper supporting the draft Water Law both provide an analytical framework for economic regulation, but it is not clear when such regulations might be issued. WPRR is working on draft Executive Regulations but there does not appear to be a regulation related to tariff reviews linked with cost recovery. Reliable and comprehensive data is captured and used in setting tariffs. Partially achieved – WPRR is conducting two pilot tariff applications in Beheira and Gharbeya with the Gharbeya process completed. The pilots are useful and help to establish the process for collecting, verifying, analyzing and using data in order to develop tariff review applications. The components of the tariff application, including the five year financial plan, cost of service model and tariff design scenarios were completed and the reports are currently being finalized for review by the company chairmen before they are submitted to EWRA in early 2012. WPRR has also undertaken, or plans to undertake, other studies to inform data reliability including a Willingness/Ability to Pay Study and a Customer Classification Study (which includes Beheira, Gharbeya and approximately sixteen other subsidiaries). WPRR has also invested in supporting EWRA with the AIR system which is used to collect and manage information from the subsidiaries, with the ultimate intention of economic and service regulation. At present the information is being used to inform the standard-setting process. The system was introduced in October 2009 and subsidiaries are gradually fulfilling the requests for information. EWRA is also beginning the process of auditing the information provided. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 41 It should be noted that the pilots would still need to be replicated across the full range of subsidiaries for the desired result to be achieved. In addition, the desired result refers to the reliability of data and implies a sustained effort to ensure reliability. Given that only pilot activities have been conducted, the sustainability has not yet been tested. Utilities are testing and measuring the impacts of cost recovery methods. Not yet achieved – There‟s no evidence that subsidiaries are testing cost recovery methods in a systematic way as a result of WPRR project activities. However, EWRA is working closely with the HCWW economic analysis unit and subsidiary representatives to test and measure scenarios for improved cost recovery as part of the tariff application pilots and customer classification/tariff study. Utilities are generating income streams that are reflective of the economic costs incurred in operation. Not yet achieved – Cost recovery data generated through the WPRR and WWSS projects is unclear. The Evaluation Team was informed by several interviewees that, as a whole, the sector is recovering O&M costs, but the results are uneven across the subsidiaries. Cost of service analysis would be required across the subsidiaries to have a true understanding of the income streams that are needed for current operation, new service standards that might be required by EWRA, and any necessary expansion. If, in fact, information is available which clearly defines cost recovery and measures each subsidiary against this benchmark, the information should be widely and consistently used. Service level standards are established, and a program for monitoring compliance is initiated. Partially achieved – EWRA has adopted a set of 20 Level of Service indicators and has set standards for 6 of these. The Levels of Service measures are calculated by EWRA from the AIR system. This process was hindered both by delays in receiving the final data and in the accuracy of the data provided. These issues are being resolved. WPRR provided guidance on setting and regulating levels of service and on auditing data provided by subsidiaries. WPRR reviewed and refined previous work on setting standards of service. WPRR is working with EWRA to link service standards with the tariff review and licensing processes to provide financial incentive for utilities to improve service delivery. Compliance monitoring has not yet been fully initiated because subsidiaries have not yet fully adopted service standards. This adoption will come through the performance agreement process whereby tariff rates and service standards will be linked. By early 2012, all remaining service standards are to be established. The PMP includes a milestone indicator for progress in developing, implementing, and assessing performance against WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 42 service standards. The PMP says that targets will be set against each service standard in the Annual Work Plan. The Year 4 Annual Work Plan does not include these targets but indicates that they will be set. The majority of the standards are established, but they are not fully agreed upon/implemented within the subsidiaries or used for monitoring purposes yet. WPRR COMPONENT 2 FINDINGS There is some disconnect between the tasks under Component 2, the expected results, and the indicators set out in the PMP. The expected results emphasize aspects of cost recovery while the Component 2 tasks cover more broadly EWRA‟s economic and technical regulatory responsibilities and the need for licenses, regulations and performance agreements to document regulatory arrangements. While WPRR has not accomplished many of the results set out in the contract SOW at the time of this Evaluation, the project has been active in all of the assigned tasks. The fact that neither the tariff studies nor the service standards are complete has meant that the WPRR project has not been able to fully implement a comprehensive regulatory regime even in a pilot area. Without this implementation, other critical aspects of the regulatory framework have yet to be tested including monitoring and enforcement of provisions in the licenses and agreements. COMPONENT 3: WATER AND WASTEWATER OPERATOR CERTIFICATION Component 3 tasks lead to the development of an Operators Certification Program, including: • Work with EWRA to determine the regulatory actions required to initiate the program. • Assist EWRA to work through a consultative process in developing the overall framework for the certification program, such as establishing the levels of certification and the training and/or on-the-job experience required for each level. • In collaboration with EWRA and MHUUD, identify and build the capacity of an unbiased, third party institution to be responsible for testing. • Provide assistance in designing testing materials. • Train staff at the institution to administer examinations. TABLE 13: WPRR COMPONENT 3 RESULTS AGAINST SOW Expected Results per the SOW Findings The program for certifying competencies of operators of water and wastewater facilities is established, and initiated. Partially achieved –WPRR has expended significant effort, including one international trip, on the certification program which was established by Ministerial Decree 204 in 2010. The procedure for certification has been identified and implemented, certification needs assessed and ranked, training programs established, and exams written. Specifically, WPRR:  Classified all treatment plants for size and complexity of process  Created the plant classification database  Developed core competencies for all certification classes for three sector positions  Created examination questions for all certification classes for two sector positions WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 43  Established examination question database  Developed the policy framework for the certification approach To date no tests have been conducted and no certifications have been issued. There were delays in the preparation of the training courses required for certification and in the issuance of the Ministerial Decree needed to officially establish the Certification Executive Committee which is to administer the program. The latter was not signed until September 2011. The system for monitoring compliance is developed. Partially achieved -Although testing has not yet started, systems for monitoring compliance have been developed. These include software systems for plant classification and operator credentials that will be used to identify deviations from regulatory requirements and a comprehensive policy framework for the program that was formally adopted by the certification board, which is made up of the chairmen of the Holding Company and the Building and Housing National Research Center, the Executive Director of EWRA, and other senior representatives of these organizations. These components contribute to the monitoring system which is still to be fully launched, allowing for an understanding of how well monitoring can be accomplished and sustained. WPRR COMPONENT 3 FINDINGS The certification program is largely complete in terms of preparatory work. However, tests have yet to be administered so there is no information on the success rate of applicants. Further, the Housing and Building Research Center is still relatively untested in administering the program although it administers another certification program. From a review of the WPRR materials, the certification program appears to have required significant effort and should lead to an improvement in sector performance through the establishment and monitoring of sector competencies. The link to improved sector performance cannot yet be demonstrated objectively given the time needed to develop the programs. COMPONENT 4: PUBLIC-PRIVATE PARTNERSHIPS (PPP) The Component 4 tasks include: ● Defining a policy framework for PPPs and operational guidance ● Providing recommendations on the types of PPPs that could support water and wastewater ● Assisting MHUUD to develop a strategy for pursuing PPP opportunities ● Drafting the contract language for PPP transactions, including terms for any future renegotiations ● Developing recommendations for the PPP Unit on how to address potential contract readjustments ● Capacity building to develop a PPP Unit at MHUUD including establishing the organizational structure and facilitating training ● Providing short-term engineering, environmental and financial technical assistance services to help execute up to three priority water and wastewater transactions WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 44 ● Assisting in pre-qualifying bidders, preparing tender documents, advising on technical aspects of bid evaluations, and assisting in post-bid negotiations ● Identifying legal issues and recommending legislative change, as necessary ● Pre-qualifying bidders, evaluating proposals, drafting contractual agreements, providing legal clarifications to bidders, and providing legal advice during negotiations TABLE 14: WPRR COMPONENT 4 RESULTS AGAINST SOW Expected Results per SOW Findings Legal and regulatory framework established for public-private contracting models for water and wastewater services Achieved - WPRR has been effective in supporting the PPP Unit including the organizational chart, departmental functions and job descriptions. In 2011 several additional PPP deliverables were completed including a PPP policy paper, draft language for PPP contracts and potential contract adjustments. WPRR has also advised on the establishment of a PPP unit within the MHUUD although formal establishment is on hold until passage of the Water Law. Increased private sector participation in financing and delivering water and wastewater services Not yet achieved – Component 4 of the WPRR has been significantly affected by the events of January 2011. All major PPP transactions in the sector were delayed until the political and security situation became more stable. Therefore, there has not been increased private finance attracted to the sector to date. However, the GOE commitment to PPPs has remained high and there has been no movement away from the approach. An IFC supported transaction is set to begin operations in May 2012. WPRR has delivered significant support to three major pending PPP projects in Abu Rawash, 6th of October, and Alex West. Contract language has been drafted for the Abu Rawash and 6th of October PPP transactions. The WPRR legal team contributed to the 6th of October data room, including the draft contract and annexes, Information Memorandum, Prequalification Document, and Invitation to Tender. Of the three transactions, the Abu Rawash Wastewater Treatement Plant upgrade is most likely to move forward in the short term. The project is already shortlisted among potential investors and there are no technical issues. If this does proceed with GOE approval, it would increase the private financing in the sector. The relevant PMP indicator relates to the number of PPP transactions for which environmental, technical and legal due diligence reports have been completed. The target was two transactions in 2011, which was met. Minimum of three PPP transactions completed Not yet achieved – fulfillment of PPP transactions is largely outside the control of WPRR which advises the PPP Unit but WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 45 does not serve as a transaction advisor per se. While several transactions have advanced, both under the auspices of IFC and USAID, the events of January 2011 slowed and altered the landscape for PPPs. One USAID-supported transaction is expected to move forward in the near term. WPRR COMPONENT 4 FINDINGS WPRR has made a tangible contribution to the PPP Unit in terms of overall capacity building and transaction support. The Unit appears well-functioning at this stage and may not require significant additional support from the project. Specific transactions would benefit from the due diligence inputs of the team but this support could equally be sourced elsewhere. COMPONENT 5: CAPITAL INVESTMENT AND PLANNING Component 5 includes supporting MHUUD to develop an integrated approach to medium-term planning and capital budgeting. The tasks include: • Incorporating recommendations from the National Strategic Master Plan for Water and Wastewater into capital investment planning • Strengthening the capital budgeting process to formulate a medium-term capital budget which balances development priorities with the state budget • Training MHUUD staff on a new approach to capital investment planning and budgeting TABLE 15: WPRR COMPONENT 5 RESULTS AGAINST SOW Expected Results Findings The GOE is carrying out integrated medium-term planning and capital budgeting for the water and wastewater sector. Not yet achieved - The GOE has 5-year capital investment plans for the water and wastewater sector based on an integrated planning process. The high level commitment to integrated planning and budgeting against transparent priorities and an open process for developing those plans against transparent criteria is still weak. Many capital projects are agreed upon in a non-transparent manner without a basis in planning. WPRR has provided support on a limited basis related to the integration of the pilot tariff studies with capital budgets and subsidy estimates. The models and approaches developed can be replicated on a larger scale. Further, WPRR is using the sector strategy development process as a mechanism to develop further coordination in planning. Strategic development priorities and programs are established. Partially achieved – The master planning process conducted through HCWW resulted in establishment of priority areas (rural sanitation and desalination) with associated strategies. It has not yet been demonstrated whether, in practice, these priorities will be respected and implemented. Improved strategic decision￾making and more effective allocations of limited budgets for water and wastewater infrastructure. Partially achieved – The prioritization model developed by WPRR has been piloted with data from three governorates and agreement reached to expand into other subsidiaries. The model has proven useful to the NOPWASD planning department, particularly in light of reduced resources for 2011/2012. NOPWASD has used the model to allocate WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 46 resources toward completion of projects under construction. WPRR COMPONENT 5 FINDINGS WPRR has provided tools to support better planning, prioritization and budgeting including the prioritization model, master planning support, and pilot activities to integrate capital planning with tariff scenarios. However, the coordination and commitment on the part of the GOE appears fragile. Political allocations of budget have been prevalent and the Holding Company/NOPWASD coordination is not consistent. WPRR plans to use the water strategy process as a way to highlight and address some of these issues, but whether that approach will yield tangible results during the project life is unknown. COMPONENT 6: PROGRAM MANAGEMENT Under Component 6, the tasks involve developing a comprehensive approach to program management, which moves beyond monitoring and evaluation functions. The tasks include expanded use of a database system to help inform decision-making at the program level. Other tasks include: • Assess the PMU‟s current capabilities and resources, and procure additional equipment • Evaluate the PRiSM database to identify and correct weaknesses • Identify objectives for utilizing data and identify how this will affect decision-making and lead to specific actions • Strengthen Program Management practices and improve Ministry ability to utilize data in the database to better manage construction activities • Develop protocols and actions to remedy problems identified in the portfolio of projects • Develop standardized project planning methodology, as well as supporting documents and procedures for use by all Ministry agencies that perform construction work related to water and wastewater • Train NOPWASD, CAPW, NUCA, Central Authority for Development (CAD), and the HCWW on program management and standard project planning. TABLE 16: WPRR COMPONENT 6 RESULTS AGAINST SOW Expected Result Findings Improve the GOE‟s institutional and technical capacity to manage the construction of capital investments. Achieved – The WPRR project has been largely successful in improving the GOE‟s capacity to manage at the project level through the use of improved databases and reporting. The changes have generated exitement among HCWW, agency and subsidiary staff. The availability of consolidated data has facilitated communication and rational decision-making and allowed for more accurate cost estimates and early identification of issues. The use of PRiSM allowed NOPWASD to identify that variations in procurement contracts were leading to project delays and problems. As a result, standard contract documents were developed with support from WPRR and WWSS. There were initial problems in data entry and data reliability. The burden lies with NOPWASD to audit and confirm data and to WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 47 verify changes made to the data. It isn‟t clear that this function can be sustained after WPRR project completion. It also isn‟t clear what incentives are realized at the project or agency level to provide the data needed for PRiSM. The prime motivator appears to be an ability to respond to management requests for information. WPRR COMPONENT 6 FINDINGS Component 6 has been successful in terms of providing NOPWASD with a practical tool and training that can be used to more effectively manage field results. There are questions of NOPWASD‟s ability to manage the volume of information and to sustain efforts to verify and audit data. ASSESSMENT OF WPRR RESULTS USING THE PMP In addition to the qualitative analysis of WPRR contributions to achievement of the expected results as specified in the contract scope of work analyzed in the previous section, we present below an analysis of the project results as measured by the quantitative indicators set out in the WPRR Performance Monitoring Plan (PMP). The PMP, in turn, is designed to help the contractor measure progress against the agreed upon results framework for WPRR (Figure 3). The WPRR PMP defines a results framework with the high level program objective (PO) being “Improved quality of and access to sustainable water and wastewater services.” Relying on the development hypothesis implicit in the design of the WPRR, the results framework further defines success in achieving this program objective as being dependent on progress toward three program intermediate results (PIRs), two of which (numbers 1 and 3) match PIRs for WWSS: PIR 1) Capital investment and program management improved; PIR 2) Enabling environment of water and wastewater services improved; and PIR 3) Capacity of staff in targeted institutions increased. The PMP further defines 13 indicators that were designed to track progress toward achievement of the specified program objective and intermediate results: TABLE 17: WPRR PMP INDICATORS Indicator Measuring 1 Progress in developing implementing and assessing performance against service standards PO 2 Percent of O&M costs covered by subsidiaries PO 3 Number of governorates in which the prioritization model is used to prioritize infrastructure projects PIR1 4 Percent of active projects entered into PRISM PIR1 5 Percent of projects entered into PRISM on a monthly basis PIR1 6 Number of standard contracting systems completed PIR1 7 Number of program management reports submitted to Chairman PIR1 WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 48 8 Number of utilities submitting Annual Information Returns within three months of request PIR2 9 Number of utilities submitting three year tariff studies PIR2 10 Number of certification examination questions developed PIR2 11 Number of plant operators and utility technical staff who take pilot certification exams PIR2 12 Number of PPP transactions for which…due diligence reports have been completed PIR2 13 Number of people trained PIR 3 The PMP approved for WPRR in 2011 is too recent to allow for useful progress monitoring. While targets and values are reported for 2010 and 2011 for the most part, this provides a limited data set. Two of the indicators measure results against the PO, Improved Quality of and Access to Water and Wastewater Services. Five of the PMP indicators measure progress against PIR1 related to capital investment planning and program management. Another five are stated as indicative of PIR 2, correlating to development of the enabling environment. The link between these indicators and the enabling environment is tenuous although this is an area of significant project activity. It is useful to look across the range of data from the Results Framework, PMP and Project Activities to get a full picture of project performance. In looking at this data in the table below, it becomes clear that the SOW activities were targeted toward results that were not consistently measured through the PMP. There is a particular mismatch between the actual legal, policy, regulatory and strategy work of WPRR, which has provided a valuable contribution to the sector, and the PMP indicators established for PIR2. TABLE 18: WPRR PMP RESULTS Activity Performance Indicator 2011 PMP PO/PIR Link PMP Result C.5.b.1 Policy and Legal Reform Expected Result: Clear roles and responsibilities for sector organizations defined No associated PMP indicator C.5.b.1 Policy and Legal Reform Expected Result: Financing for the water and wastewater sector increased No associated PMP indicator C.5.b.1 Policy and Legal Reform Expected Result: Tariff reform to improve cost recovery No associated PMP indicator C.5.b.2 Regulatory Reform Expected Result: Clearly defined responsibilities for regulating service providers. No associated PMP indicator C.5.b.2 Regulatory Reform Expected Result: Standard, uniform and equitable regulations established which improve the recovery of operating expenses and capital debt. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 49 No associated PMP indicator C.5.b.2 Regulatory Reform Expected Result: Reliable and comprehensive data is captured and used in setting tariffs Number of utilities submitting Annual Information Returns within three months of request PIR2 Linkage to PIR2 not obvious Achieved 19 in 2011 against a target of 23 Number of utilities submitting three year tariff studies PIR2 Linkage to PIR2 not obvious Target of 2 was not yet achieved. Result was 0. C.5.b.2 Regulatory Reform Expected Result: Utilities are testing and measuring the impacts of cost recovery methods No associated PMP indicator C.5.b.2 Regulatory Reform Expected Result: Utilities are generating income streams that are reflective of the economic costs incurred in operation. Percent of O&M costs covered by subsidiaries PO Link to PO not obvious Stated as an increase from 115% to 119% from 2009-2010, against a target of 100% coverage C.5.b.2 Regulatory Reform Expected Result: Service level standards are established, and a program for monitoring compliance is initiated. Progress in developing implementing and assessing performance against service standards PO Milestone targets are set in the annual work plans C.5.b.3 Water and Wastewater Operator’s Certification Expected Result: The program for certifying competencies of operators of water and wastewater facilities is established, and initiated. Number of certification examination questions developed PIR2 Linkage to PIR2 not obvious Only target was in 2010 when it was set for 2100 and exceeded at 3231 Number of plant operators and utility technical staff who take pilot certification exams PIR2 Linkage to PIR2 not obvious No target set until 2012 at 60. Not yet reached C.5.b.3 Water and Wastewater Operator’s Certification Expected Result: The system for monitoring compliance is developed. No associated PMP indicator WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 50 C.5.b.4 Public Private Partnerships Expected Result: Legal and regulatory framework established for public-private contracting models for water and wastewater services No associated PMP indicator C.5.b.4 Public Private Partnerships Expected Result: Increased private sector participation in financing and delivering water and wastewater services Number of PPP transactions for which…due diligence reports have been completed PIR2 Link to PIR2 not obvious Target of 2 in 2011 met C.5.b.4 Public Private Partnerships Expected Result: Minimum of three PPP transactions completed No associated PMP indicator C.5.b.5 Capital Investment Planning Expected Result: The GOE is carrying out integrated medium-term planning and capital budgeting for the water and wastewater sector. No associated PMP indicator C.5.b.5 Capital Investment Planning Expected Result: Strategic development priorities and programs are established. No associated PMP indicator C.5.b.5 Capital Investment Planning Expected Result: Improved strategic decision-making and more effective allocations of limited budgets for water and wastewater infrastructure. Number of governorates in which the prioritization model is used to prioritize infrastructure projects PIR1 3 in 2011. Target met C.5.b.6 Program Management Expected Result: Improve the GOE’s institutional and technical capacity to manage the construction of capital investments. Percent of active projects entered into PRISM PIR1 100% in 2011 Percent of projects entered into PRISM on a monthly basis PIR1 79% in 2011 below the target of 85% Number of standard contracting systems completed PIR1 20 in 2011 Number of program management reports submitted to Chairman PIR1 No target until 2012 (192) WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 51 Training Number of people trained PIR 3 Target of 800 in 2011. Result was 608 POLICY AND LEGAL REFORM While the qualitative analysis shows that substantial work has been done by WPRR to develop the Water Law and to advance a Water Strategy, there are no PMP indicators relevant to this component. REGULATORY REFORM There are four PMP indicators related to this Component. These include the ability of the utilities to generate and provide timely information to the regulator and to use that information for tariff studies. Other indicators measure the ability of the utilities to cover O&M costs and to perform against defined service standards. Nineteen utilities are submitting timely information against a target of 23 utilities set by the project. Targets related to timeliness have been in place since 2010 and issues remain related to the HCWW review (prior to EWRA receipt). Information received in 2012 will be the first on￾line submissions of data as inputs in the tariff studies. Targets (3 and 2 respectively) for three year tariff studies have also not yet been met in 2010 or 2011. Beheira and Gharbeya plan to submit the first such studies. The joint PMP included two indicators relevant to the sources of financing for the sector. Specifically, the indicators would have tracked the percentage of O&M costs being covered through customer revenue and through subsidies. Both indicators were removed in the 2011 WPRR PMP with the justification that “the WPRR scope of work does not impact this indicator.” Despite this statement, the WPRR project has been collecting and reporting some data against the indicator “percentage of O&M costs recovered by targeted subsidiaries.” The term “targeted subsidiary” is a holdover from the combined PMP and should have been eliminated in the WPRR PMP. In fact, data on O&M cost recovery is being collected for all subsidiaries. However, this data is ambiguous – baseline, 2009 and 2010 data show that more than 100% of O&M is collected within the “targeted subsidiary” during the “reporting period.” Yet interviews and other analysis indicated that, in fact, many of the subsidiaries are not meeting cost recovery in O&M. The final regulatory PMP indicator looks for progress in developing, implementing and assessing performance against service standards. The standards are discussed extensively in the Annual Work Plans but are still under development. CERTIFICATION PROGRAM The PMP contains two indicators related to this component. The first indicator, the number of certification questions developed, is inconsequential. Contrary to best practice in indicator development, it seemingly creates an incentive for additional questions whether needed or not. In fact, the number of questions developed exceeded the target by one third. Furthermore, it is a one-off indicator – it doesn‟t demonstrate progress against a goal. The second indicator measures the number of plant operators and technical staff who took the pilot certification exams. To date, none have taken the exam due to delays in rolling out the program. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 52 PUBLIC – PRIVATE PARTICIPATION The PMP assesses the number of transactions which have undergone due diligence by the WPRR team. The indicator is thus measuring an input by the team, but not assessing progress in the achievement of transactions. Although final transactions are listed as an expected result of the component, this would be largely out of the hands of the WPRR team especially under current circumstances. The target of 2 transactions with completed due diligence inputs was met in 2011. CAPITAL INVESTMENT PLANNING There is one PMP indicator for capital investment planning against three expected component results. The PMP includes an indicator for the number of governorates which have adopted the prioritization model. The 2011 target was three which has been met. PROGRAM MANAGEMENT There are four PMP indicators related to this component – two are related to PRiSM data, one relates to program management reporting and one measures usage of standard contracting. According to reported data, all projects are being entered into PRiSM and almost 80% are being updated monthly, narrowly missing the target of 85%. More than 20 standard contracts have been developed. No target had been set for this indicator. The target for submission of program management reports to the Chairman has only been established for 2012 at 192. This has not yet been reported on. TRAINING The WPRR PMP tracks the number of persons trained by the Contractor as part of the program which fell short of targets at 608 against a target of 800 in 2011. This PMP indicator does not relate to development of long term training plans, only absolute numbers trained. SUMMARY OF WPRR RELATED FINDINGS WPRR has a Program Objective to improve the quality of and access to sustainable water and wastewater services through three intermediate results: PIR 1 Capital investment and program management improved; PIR 2 Enabling environment of water and wastewater services improved; and PIR 3 Capacity of staff in targeted institutions improved. The WPRR project objective is to improve the quality of and access to water and wastewater services through these IRs. Through the evaluation process, the Team worked to identify the specific service issue that required attention through the WWSS and WPRR projects. Overall, water services appear to be acceptable with coverage at almost 100%. Stakeholders cited some coverage, quality, and service gaps, and there are efficiency and cost recovery problems which have not been well quantified. However, the major concern raised by most stakeholders was the lack of investment in wastewater. Given how broadly the PO is defined, and the difficulty in determining precisely where the service challenge lies, it is not surprising that the PIRs haven‟t had a demonstrable impact. It may be that there is too great a gap between the stated PO and the results that could be achieved through the IRs. However, the project activities have contributed to the desired results at the intermediate level, although not with full satisfaction. In particular, WPRR made good progress in developing the WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 53 suite of policy, legal, regulatory and strategic documents which will be critical to implementing reform. When and if the next stage of sector reform is fully pursued (focusing on accountability, governance and cost recovery), the WPRR work will be valuable. Potentially less successful is the work with EWRA. Despite extensive TA and training, the regulator is still working without full legal authority and engaged only in pilot activities. The work of WPRR is thus undermined by largely external factors which constrain the regulator from being fully empowered. Capital investment and management was addressed through the PPP activity, support to NOPWASD and the development of improved capital budgets and planning. WPRR has made a tangible contribution to the PPP Unit in terms of overall capacity building and transaction support; however progress has been slowed by political events. It is not clear that the PPP Unit requires significant additional support from the project. While WPRR has been able to develop tools and training that support capital programs and management, the impact has been relatively limited to date. The PRiSM tool has gained support within the sector, but again, the past and potential political interference in capital programming may undermine project efforts to expand transparence and coherence in the process. PIR 3 calls for capacity development within the sector. This was accomplished through the certification program and through the training provided under each activity. However, the training and capacity development has been broadly applied throughout the sector in areas ranging from technical certification to English language training. The approach may have been so diffuse as to limit the tangible results and WPRR‟s ability to drive any focus area through to completion. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 54 6. CROSS-CUTTING ISSUES The program was designed with the realization that certain issues and activities were cross￾cutting between the WWSS and the WPRR projects. Specifically, the project results framework specified that improved capital investment planning and program management would be a shared goal of the two projects. The WPRR project work plan further specifies that training is a cross￾cutting area of activity. Therefore, developing the tools and training to support this objective would likewise ideally be shared between the two projects. These overlapping activities would include the systems developed to support capital investment planning - a critical interest to the Holding Company, service providers and the regulator – and the training provided to support this function. The SOWs for each project, however, do not specify any particular tasks as being “cross￾cutting.” In fact, there is no mention in the respective SOWs of the need for the two project teams to coordinate on overlapping technical areas. This lack of clear directive was exacerbated by the fact that the two projects were awarded to the same contractor, Chemonics, albeit in different consortia compositions. Anecdotally, the contractors indicated that this dual award led to a decision to maintain clear boundaries of responsibilities for particular tasks and relationships with the key stakeholders, EWRA and the utilities, by the two contractor teams. On the USAID side, after being initially managed by one COR, USAID/Egypt decided to have each contract managed by a separate COR. The separation of the combined PMP further delinked the projects shared responsibility for sector outcomes. SUMMARY OF FINDINGS RELATED TO CROSS-CUTTING ISSUES The Team found instances where the lack of coordination in some areas between the two projects may have undermined the efficiency of USAID‟s support and may have resulted in lost opportunity for richer results. Examples include: ● We found no examples of significant collaboration between the projects on directed WWSS efforts to improve performance of particular subsidiaries with WPRR efforts to begin establishing performance baselines and targets for the envisioned regulatory regime that it is striving to establish. We feel this lack of focused, coordinated support of performance improvement to be a key missed opportunity. ● Upgrades to MARS have been developed and implemented through WWSS while AIR is the management information system supported by WPRR for EWRA. Both systems are designed to capture operating data and performance indicators for the sector which can be used for decision-making. It was unclear to the Team to what extent staff at EWRA had ever evaluated MARS while the Chairman of the HCWW appeared relatively unfamiliar with AIR. It is unclear what the joint efforts are of WWSS and WPRR to consolidate these systems or develop a common one. The Team feels that the sector would greatly benefit from being managed and regulated against one set of objectives and indicators which would link back to the Water Policy (and ultimately the Water Law). ● Each project has conducted limited outreach and public awareness activities and each has considered plans to conduct additional activities during the Option Year. Yet the planned messaging of each project is different and not necessarily mutually reinforcing – apart from a shared commitment to water conservation. For sector reform to be coherent and demonstrate a unified GOE position, it would be important for the WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 55 subsidiaries, HCWW and EWRA to develop a unified platform of messages. Ideally, these would be around the GOE and USAID objectives of improved and sustainable service delivery which might entail laying the groundwork for improved collection, more rational tariffs, and increased customer responsiveness. ● Finally, each project has invested heavily in training programs. It is not evident, however, that there has always been sufficient coordination between these efforts. However, the two projects did effectively collaborate in some areas. The following examples were noted by several key informants: ● Establishment of the Certification Program led by WPRR benefited from input by the WWSS team who are working closely with the subsidiaries on certification requirements. ● The PRiSM program is an area where the two projects have effectively worked together to improve the sector‟s ability to collect, track and prioritize capital projects. Working through subsidiaries, NOPWASD and the Ministry, the projects have been able to capture the full range of data within the sector and allow project managers to understand project status better than before. ● Another area of productive collaboration noted by key informants was contributions by the two projects in developing standard contracting documents for use in the sector. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 56 7. CONCLUSIONS AND RECOMMENDATIONS The mandate of the Team was to assess WWSS and WPRR project data in order to make conclusions about their effectiveness in achieving the results expected from them. The Team was also asked to identify whether these two investments as a whole are meeting the overall development objectives under the results framework guiding them both. Finally, based on this evaluation, the Team was asked to identify immediate and near term priority areas that should be the focus of possible future programming in the Egyptian water sector including anticipated results and implications related to USAID/Egypt exercising the option year on these two projects. KEY CONCLUSIONS ON WWSS PROJECT Overall the Team found that WWSS has been successful in delivering on the majority of expected results especially in support of establishing three new subsidiaries. In addition to the achievement of individual results, the Contractor also helped to directly contribute toward “cost-effective” service delivery improvements through piloting of energy efficiency improvements, chemical optimization, and leak detection. However, most of the direct performance improvement achievements seem surprisingly modest given the long history of similar support to the sector in Egypt. For example, the WWSPR II End of Contract Report notes that already in 2009 the program: “found ways to improve cost and performance of the companies. A case in point is the Cairo Water pilot study on solutions for chronic utility problems such as high unaccounted for- water, metering and low billing and collection efficiency (Component D.5). This study found ways to increase revenue by LE 106 million or reduce costs by LE 97 million. One component of this study is telling: The effort to reduce illegal connections in three suburbs took six weeks and resulted in additional annual revenue of LE 6.3 million. Efforts such as these, carried out in all companies, can make a major contribution to cost recovery and financial independence.” It is disappointing to see that such performance improvements have yet to be fully institutionalized and WWSS continued to “pilot” the implementation of such efforts rather than implement them on a broader scale. It is clear that USAID intended the WWSS project to promote at-scale performance improvement by promoting systemic institutional change. Such broad results are not apparent from indicator results or other qualitative descriptions of program results. As such, the Team notes two key weaknesses in WWSS: 1. A lack of clear documentation regarding WWSS impact toward significantly improving subsidiary performance improvements on a consistent basis. 2. No clear indication of how the multiple management “tools” (especially the multiple computer software tools) are resulting in performance improvement and sustained evolution of the utilities into more cost-effective organizations. The Team does recognize WWSS has worked to help ensure that their pilot programs secure the buy-in of counterpart subsidiaries, and provide key material support and training necessary for the subsidiaries, if they are convinced, to scale up the pilot efforts. However, given the current data available, there is limited evidence that they have been successful to date in this regard. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 57 The Team also feels that WWSS has largely provided utility support in areas of expertise that are relatively well established in Egypt. Many subsidiaries appear to continue to need such support, but the Team questions whether it is cost effective for USAID to hire an international contractor to arrange such support when most direct consultant inputs are provided using Egyptian or regional consultants with long-term experience in Egypt. KEY CONCLUSIONS ON WPRR PROJECT The nature and duration of the WPRR project make it difficult to have a direct and short term impact on the quality of and access to sustainable water and wastewater services. The project works to strengthen the framework for sustainable service – the capital investment process, the human resources, and the legal and regulatory environment. The Team believes that WPRR has delivered real results for the sector in its three primary areas of activity: ● PIR 1 Capital investment and program management improved; ● PIR 2 Enabling environment of water and wastewater services improved; and ● PIR 3 Capacity of staff in targeted institutions improved. More specifically, WPRR has delivered a draft Water Law that will help clarify GOE objectives and delineate roles; it is drafting a strategy that will help implement the objectives; WPRR helped bring transparency and monitoring capacity to capital management; it tackled the issue of tariff reform; and it introduced professional accreditation to the sector. These activities are all valuable for the long term viability and credibility of the sector. However, the events of January 2011 made it clear how dependent WPRR was, and is, on GOE focus and commitment. The project activities – potentially more than those of WWSS – are likely to falter without strong political will. The case of EWRA may demonstrate this. The regulator has been languishing for many years without a full mandate and without regulations to enforce. The Evaluation Team feels that it does not make sense for WPRR to provide English and management classes to staff rather than focus on establishing the case for EWRA‟s role. We feel a more consolidated focus on pilots which tangibly showed the value of a regulator would have been more useful. Potentially, a small number of representative subsidiaries could have been selected as pilots for all reform streams, including licensing, certification, tariff reform, and PRiSM. This concentration might have produced the data results that would have generated greater political will. As it is, WPRR struggles with an ambitious agenda and a challenging environment. A strict evaluation cannot assess the project as having met objectives. A more nuanced evaluation recognizes that the project adds value in meaningful and potentially enduring ways. When and if the political environment is clarified, the work of WPRR may be very valuable in forging a path forward. CROSS-CUTTING CONCLUSIONS The program would have benefitted from a more robust coordination between the two projects, USAID and the implementers to ensure that technical information was shared, resources were used most effectively, and pilots were conducted to the greatest effect. For instance, it would appear that the WPRR pilot licensing activity would be best conducted in a subsidiary where WWSS is active. This would allow for the effectiveness of the licensing to be assessed from both the regulatory perspective and the subsidiary perspective. Even more fruitful would be for a consolidation of pilot activities to force a simulation of conditions under WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 58 full reform as envisioned by the Water Law. This would involve conducting tariff application, efficiency improvements, standard setting, and tariff analyses in a small subset of subsidiaries, resulting in a full understanding of the linkages – and potential disconnects - among the different activities. RECOMMENDATIONS FOR OPTION YEAR The recommendations for the Option Year are based on the themes of focus and results. The program has not been effective at collecting, managing, and reporting on the data that would demonstrate project results and value. The program has also veered in too many directions, making it too difficult to achieve a meaningful impact in any one area. The Team, therefore, makes the following recommendations for the Option Year: PRIORITY AREAS FOR CONTINUED EMPHASIS IN THE SHORT TERM WWSS 1. Efficiency improvements (energy and chemical use, leak reduction) and maintenance management. These activities have support within the sector, can demonstrate real and dramatic cost savings, and are potentially sustainable after the project. The focus of the Option Year should be on ensuring sustainability. 2. Business planning. Support for business planning reinforces the message that the subsidiaries are meant to be corporate, autonomous entities. Increased ability to collect, assess and use company information will strengthen management and empower staff. 3. Development of public outreach with key messages. Public outreach is critical to establishing the credibility of the subsidiaries and building public awareness around demand management and the need for financial sustainability in the sector. 4. Targeted training (managers and operators). The training supports the ability of the subsidiary to achieve and sustain service improvements. WPRR 1. Sector strategy/legal and regulatory framework. WPRR should concentrate on the sector strategy. The strategy provides the unifying vision of the tariff, service improvement, investment and institutional work and sets out a blueprint that will endure after the USAID program. The strategy has to be supported by the full suite of legal and regulatory documents required. 2. Tariff studies and reform. The project should maintain focus on the tariff analysis and developing recommendations for improved structures and approval processes. The tariff reform provides the platform for other important conversations related to political commitment, financial goals for the sector, and the value associated with reliable and sustainable service. AREAS IN NEED OF INCREASED EMPHASIS IN THE SHORT TERM WWSS 1. Improve quality of performance data reported by subsidiaries. Performance data for the sector is unclear and uneven; it doesn‟t tell a story that demonstrates service improvement. The project needs to focus on its ability to help subsidiaries generate reliable sector data, monitor that data, and manage performance against that data. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 59 2. Focused performance related interventions (possibly limit focus to fewer utilities). The project interventions related to efficiency improvements have been important advancements., but the interventions are spread across a range of utilities, with different companies receiving different packages of support. This makes it difficult to assess the impact of the project activities and to see deep and thorough reform across a spectrum of subsidiary activities. WPRR 1. Piloting of licensing/performance agreements. The performance agreements and licenses are a critical part of the sector reform. There needs to be additional focus by the WPRR team on putting these in place – together – in selected subsidiaries in order to understand how this regime will work. 2. Operator certification program. The certification program is a sensible contribution to the professionalization of the sector. The program, however, has had some delays and no certifications have been issued yet. The program should be accelerated by WPRR to the extent possible. AREAS IN NEED OF INCREASED ALIGNMENT BETWEEN CONTRACTS 1. Public outreach on core messages related to financial sustainability of the sector. The projects should be working together to develop a unified message around the linkages among improved service, regulation and financial sustainability. These messages are potentially more valuable if aligned since each project has plans to conduct additional activities during the Option Year. The messages should be mutually reinforcing around the GOE and USAID objectives of improved and sustainable service delivery which might entail laying the groundwork for improved collection, more rational tariffs, and increased customer responsiveness 2. Piloting of licensing/performance agreements. This activity requires increased coordination between the two projects in terms of pilot selection and enactment. AREAS WHICH SHOULD BE DE-EMPHASIZED 1. Capital investment planning and project management support (PRiSM). This activity seems well-established. 2. Basic skills training for EWRA (English, MS Office etc). EWRA had enjoyed years of capacity building. However, this can be slowed or suspended in favor of more urgent regulatory issues. The English-language training does raise the question of why this was not set as a qualification for hiring, if critical. RECOMMENDATIONS FOR FUTURE PROGRAMMING In 2006, there was an Infrastructure Evaluation which recommended that the water and sanitation sector should continue as a high priority for USAID-GOE cooperation. However, the evaluation also suggested: ● An assessment should be conducted to prioritize the USAID-assisted water/wastewater facilities that need continued assistance and to make an estimate of the possible cost of a program of support. Although over time operational and institutional reform will and WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 60 must address many of these sustainability issues, an assessment of the need and costs of an interim program of continued assistance would be helpful. ● Efforts towards financial and operational sustainability should be continued and strengthened. At the direction and supervision of the Holding Company, a performance indicators system is an important step in this direction. All training, certification of technicians and managers, laboratory testing, and maintenance management systems done at the facility level should begin and end with the performance of the facilities in mind. This performance-based system has the potential to be used by the Holding Company as a compliance monitoring and utility comparison tool. The corporate leadership of the Holding Company and subsidiary companies must embrace optimal performance and customer service as their central tenets before such a system can be used effectively. This change will take time to be internalized and take hold. Top to bottom, all employees must believe that they are there to serve the consumer. If facilities are operated effectively, are efficient, and protect public investments; it is more likely that the consumer will be inclined to pay more for services. These recommendations still resonate. Six years later, it still cannot be determined that performance indicators are front and center of the sector management, operation and regulation. There are still questions regarding whether USAID support is contributing to sector performance in ways that are tangible and enduring. That said, the Team agrees that sector support should continue in recognition of the importance of water and sanitation to health, economic development, and – particularly important now – the legitimacy of government. USAID/Egypt is requesting US $11.0 million for a possible program in the water and wastewater sector. Moving forward with new programming, the Team suggests the following: 1. Improve targeting of capital expenditure to link with performance agreements. This can provide incentives for improvement and potential for competitive performance among subsidiaries. 2. Before the end of the option year, WWSS should identify ways to replace its technical assistance with FARA-based technical assistance enacted through the Holding Company. This will create a bridge to more country-driven and sustained reform. 3. Build the capacity of the holding company to promote Build/Own/Operate/Transfer contracts for future expansion of the sector. Assistance will focus on legal assistance, technical assistance, financial analysis, procurement, and outreach to major financial institutions and potential investors. 4. During the option year, the Mission should task the WWSS and WPRR contractors to draft an exit strategy for the water and wastewater sector. At the end of the optional year, September 30, 2013, the Mission will leave behind a plan for a functioning Regulatory Agency; a Holding Company operating on commercial basis; well-trained staff at critical institutions; and a system and tools for generating, assessing, and using performance data. We also suggest that future sector funding be used not only to support improved water and wastewater services but also effective decentralization by working with particular governorates that illustrate a firm willingness to operate efficiently and meet the needs of their residents. Advancing the above-mentioned recommendations could be done through an initial dialogue with the newly-elected members of the People‟s Assembly from the selected governorates, local WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 61 political and executive leaders in the selected governorates, and the respected Ministries of Local Administration, Health, and MHUUD. Management of such an intervention will require multi-office involvement from USAID and a single U.S. contractor that assembles the right mixture of skills in governance support (especially related to decentralization) and water/wastewater. The COR would be from the USAID/PSD office with close coordination with the Democracy and Governance office. The suggested program should have a component for robust public outreach. A number of success stories should emerge from this intervention regarding increased wastewater connections, additional revenue for the water and wastewater subsidiary companies, reduction in waterborne diseases, and decentralized decision making. These success stories should be used to advance U.S. foreign policy objectives in Egypt. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 62 ANNEX A: EVALUATION SCOPE OF WORK USAID/Egypt‟s Water and Wastewater Sector Support (WWSS) and Water Policy and Regulatory Reform (WPRR) Performance Evaluation I. BACKGROUND OF PROGRAM TO BE EVALUATED Building on decades of support to the water and wastewater sector primarily through capital investment projects, USAID/Egypt is supporting recent reform efforts by the Government of Egypt (GOE) to help promote the sustainability of the sector. With the assistance of the USAID financed Water and Wastewater Sector Reform Project, major reform of the sector was achieved in 2004 through the creation of the Egyptian Water Regulatory Agency (EWRA) and the Holding Company for Water and Wastewater (HCWW). In addition 14 water sector utilities were transformed into subsidiaries of the Holding Company. Since 2004 an additional 9 companies have become subsidiaries of the Holding Company. While significant progress has been achieved through the creation of EWRA and establishment of the HCWW, the need still exists to support these new organizations which are tasked with: water production and distribution; wastewater collection, treatment, and disposal; and water and wastewater capital investments; water sector regulation; capital investment planning and monitoring; and sector legal and regulatory reform. USAID is committed to the continued support to these organizations to assure the gains made by the 2004 USAID engineered reform of the sector are solidified and sustained. USAID acted upon this commitment in 2008 with the funding of two water sector technical assistance projects – the Water and Wastewater Sector Support (WWSS) program and the Water Policy and Regulatory Reform (WPRR) program. Given the natural (and healthy) tension which exists between any regulator and the agency being regulated, USAID determined that separate technical assistance contracts one assisting HCWW and one assisting EWRA – was the best way to provide the assistance required to assure the full maturation of these organizations. Under the WWSS program, USAID funds are being utilized to provide technical services and related resources to the Holding Company for Water and Wastewater (HCWW), and Water and Wastewater subsidiary companies (Subsidiaries) to strengthen their capability to: implement recent GOE initiatives aimed at improving the operational performance of the sector; attract private investments through aggressive policy, legal and regulatory reforms; operate more efficiently; and be responsive to the requirement of the new regulatory requirements. An important part of the program deals with capital investment planning and program management of the Government of Egypt‟s (GOE) water and wastewater capital investment program. The overall objective of the WWSS program is to support the GOE to develop and implement programs that: 1. Increase the financial and commercial viability of existing water and wastewater companies; 2. Establish new regional water and wastewater Subsidiary Companies; 3. Develop and implement a capital investment planning and program/project monitoring and management mechanism; and, WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 63 4. Build the capacity of staff; increase managerial, technical and operational efficiency; improve the quality of services; and expand access to water and sanitation. Under the WPRR program, USAID funds are being utilized to provide technical services and related resources to the Egyptian Water Regulatory Authority (EWRA) and the Ministry of Housing, Utilities and Urban Development (MHUUD). Assistance to EWRA has aimed at building the capacity of Agency staff through technical, management and English language training and US study tours. In addition WPRR staff are assisting EWRA staff to develop a utility licensing program, a utility level of service indicator system, a series of databases meant to organize data collected during the course of EWRA regulatory activities, a utility tariff application protocol and to carry out a tariff structure study in 18 utilities. WPRR assistance to the MHUUD is being directed at development and enhancement of a web based program management system for the monitoring of over 3,000 infrastructure projects being implemented by Ministry agencies, development and implementation of systems for prioritizing water sector infrastructure projects, development of a sector strategy based upon approved policies, assistance with the passage of a draft water law, assistance with the development and implementation of a sector certification program that will assure that critical functions within the sector are performed to international standards and provision of transaction support for water sector PPP projects. The overall objective of the WPRR program is to support the GOE to develop and implement programs that: 1. Strengthen the policy, legal and regulatory framework for the water and wastewater sector in Egypt; and 2. Improve the quality of water and wastewater services and extend them to new customers. 3. Build the capacity of EWRA staff; increase managerial, technical and operational efficiency; and improve the quality of services. 4. Improve the capacity of MHUUD staff to track and monitor infrastructure construction projects. II. PURPOSES AND USES OF THE EVALUATION The primary purpose of the evaluation is to determine whether the assistance provided by USAID/Egypt to the Government of Egypt is meeting the stated objectives. Additionally, the evaluation should provide a detailed picture of the major accomplishments/weaknesses of each project since inception, indicating as well what results are likely to be achieved by the completion of the program. The evaluation will assess the performance monitoring plan for both projects to determine whether the plans with their performance indicators truly measure the project progress and achievements. Finally, the evaluation will identify immediate and near term priority areas that should be the focus of possible future programming in the Egyptian water sector. The Evaluation will be reviewed by the Mission's Productive Sector Development team (PSD) as well as by Mission Management and the Program Office. The timing of this evaluation is propitious for making option year decisions, project corrections, as well as for providing a more informed base on which the Mission can begin to strategize, prioritize, and design future assistance for water and wastewater sector projects in Egypt. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 64 In addition, the Evaluation will serve: ● To identify if there are implementation problems, unmet needs, or unintended consequences or impacts, especially taking into account any unanticipated changes in the host country environment, particularly following the governmental upheaval after the events of January 25, 2011; ● To provide a better understanding of progress made by each project on such issues as relevance, impact, sustainability, and cost-effectiveness; and ● To confirm the validity of the overall - and project-specific - development hypotheses or critical assumptions underlying the Mission's water and wastewater strategy and the projects that were designed, funded, and implemented to make such strategy operational. III. PROPOSED EVALUATION QUESTIONS 1. Within each project, where did implementation to date fall short of achieving expected (or desired) results and what factors constrained implementation from achieving greater progress toward planned results? 2. What are the respective views of USAID, implementers, and beneficiaries with respect to weaknesses (as well as strengths) in the design, implementation, and management of each of the two projects? 3. What are the anticipated results and implications related to USAID/Egypt exercising the option year on these two projects? 4. Based on the experience of the two projects evaluated, what are the facts and implications of options concerning areas in which USAID assistance is most urgently needed, in both the short and long term? IV. EVALUATION METHODOLOGY The Evaluation Team will use the following methodology to conduct the evaluation. Document Review/Data Analysis: Team members will review the WWSS and WPRR Statements of Work; USAID/Egypt strategy document; quarterly and annual reports, and other relevant documents such as the PMPs. Key Informant Interviews: The team will conduct interviews and focus groups with a variety of stakeholders including Ministry staff (more specification needed here), program implementers, USAID staff, other donors, sub grantees and partners in Egypt (if applicable). Site Visits: Work on the WPRR project takes place largely in Cairo and visits will be expected to the appropriate contacts and offices. In addition, the WWSS project has a strong component located in Cairo; however, site visits or meetings will be required, at a minimum, with relevant staff from the following governorates: -Sohag -Aswan WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 65 -Gharbiya -Minya Staff from the USAID/Egypt Productive Sector Development Office will organize all site visits for the Evaluation Team and assist with logistics. USAID and Chemonics staff for the WWSS and WPRR projects will accompany the Evaluation Team as needed. V. COMPOSITION OF EVALUATION TEAM The Evaluation Team shall consist of at least five individuals, and possibly 6 if needed due to time constraints of the other Team members. This will include three professionals (two of whom are USAID/E3 Bureau staff: Heather Skilling and Anthony Kolb), with 10+ years of experience in water and sanitation service delivery in low-income countries with USAID and/or other donors, in addition to a team leader independent of USAID. The team shall also include a Program Officer from USAID/Egypt (Victoria Mitchell), as well as a translator/interpreter. Team members will be required to travel to pre-determined locations throughout Egypt, with a strong emphasis on activities in Cairo, to obtain an understanding of the projects‟ activities. A five-day work week is authorized for this activity. This activity is proposed to be conducted in Egypt and the U.S. for a total of seven work weeks, beginning on or about December 27, 2011 and continuing through February 26, 2012. The Team will provide 10 working days to USAID/Egypt for review and comments on the draft evaluation report and PowerPoint presentation, after which time the Team will be expected to spend another 3 days finalizing the report and presentation and then submitting the requested deliverables to USAID/Egypt as outlined in this SOW. The required areas of technical (subject matter) expertise that should be represented on the team correspond roughly to the technical foci of the program being evaluated: ● Water, sanitation and hygiene (WASH) programming ● Water and sanitation service delivery/utility management ● Water sector financing (including Public Private Partnership financing) ● Utility performance improvement planning ● Water and wastewater system operations and maintenance ● Outreach, communications and public awareness Team: 1. Water/Wastewater Service Delivery Specialist (Heather Skilling, USAID/E3): Heather Skilling has more than 20 years of experience in reforming water and sanitation services including corporate reform, regulatory/policy reform and sector restructuring. 2. Urban Health/Environment Specialist (Anthony Kolb, USAID/E3): Anthony Kolb has more than 20 years of experience sectors concerned with environmental infrastructure development and household and community demand assessment. 3. USAID/Egypt Program Officer: Victoria Mitchell will actively participate in at least one week, and possibly more, of the field-based data collection phase of this evaluation, to include work on data collection, analysis, and reporting. She will also contribute to and assist in editing the final report. 4. Interpreter/Administrative Assistant (TBD): Minimum 3 years of experience with direct interpretations from Arabic to English and English to Arabic. Experience relating to the water/wastewater sector and industry terminology is highly desired. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 66 5. Overall Team Leader – The team leader will serve as the primary point of contact between the USAID/Egypt Mission and Evaluation Team. The incumbent must: ● Be able to communicate effectively with senior U.S. and host country officials and other leaders; ● Have a proven track record in terms of leadership, coordination, and evaluation delivery for development projects and programs; ● Have excellent writing/organizational skills and proven ability to deliver a quality written product (Evaluation Report and PowerPoint). In addition the Team Leader may provide his/ her technical expertise in one or more areas to support this Evaluation. Proposals will be reviewed taking into account overall strengths of the assembled team's senior-level technical expertise, evaluation expertise, and the expertise and ability of the team's leadership to manage the team's budget and staff resources. USAID Egypt will designate staff from the Productive Sector Development (PSD) Office to provide logistical and administrative support to the team; however, the Team Leader will have the primary responsibility for ensuring the final deliverables are completed in a timely manner and are responsive to the scope of work and Mission comments. VI. USAID MANAGEMENT OF EVALUATION The USAID/Egypt point of contact for the evaluation will be John Pasch, Water and Engineering Team Leader. An Evaluation Committee comprised of the lead Mission staff (e.g., AOTR and/or COR4) responsible for each of the two projects being reviewed and other Mission team members will be formed to guide the Evaluation Team in their work by reviewing reports, responding to questions from the team and resolving administrative or logistical obstacles. DRAFT: Logistics & Time Frame (assumes Sunday to Thursday as workdays in Egypt) The following provides a notional presentation of a prospective allocation of level of effort for the Evaluation: Approximate Timing Activity Expected Duration Location December 27, 2011 to January 7, 2012 Preparation – In-brief with USAID/Egypt staff and evaluation team members. Document review. Finalization of evaluation methodological approach(s) and proposed field schedule. Development of questionnaires and/or other tools to be used in conducting surveys and fieldwork. 10 days U.S. or Egypt January 8-21, 2012 Field Work - In-depth interviewing of USAID staff and project implementers, partners, and beneficiaries 10 days Egypt 4 AOTR: Agreement Officer’s Technical Representative; the AOTR is the designated representative who provides technical and administrative oversight of an assistance award. COR: Contracting Officer’s Technical Representative; the COR performs a variety of duties including serving as the technical liaison between the Contracting Officer (CO) and the contractor. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 67 to confirm project results, identify constraints to the project being more successful in achieving desired results, and stakeholder views on ways in which assistance could be more effective in achieving expected/desired results. January 22-28, 2012 (*Jan. 25 is local holiday) Follow-up and synthesis – In-country team work culminating in delivery of draft report/findings to USAID/Egypt Mission and stakeholders. Additional meetings and interviews may also be scheduled to validate/confirm findings. 5 days Egypt February 13-19, 2012 Revision and refinement – In response to comments from USAID/Egypt staff and other stakeholders, team will incorporate feedback and other input into analysis, report and presentation. 5 days U.S. or Egypt February 20-26, 2012 Final report production and presentation – Completion and delivery of final evaluation report and summary power-point presentation. Presentation of results to the USAID/Egypt Mission (in Egypt) is preferred, although a video conference could be organized in lieu of an in-person presentation. 5 days U.S. or Egypt VII. DELIVERABLES The Evaluation team will be responsible for producing the following deliverables: ● Revised evaluation approach and draft schedule of field activities (prior to travel to the field) ● Draft and final questionnaire(s) to be used during interviews/stakeholder meetings (prior to travel to the field) ● Summary of document review and proposed outline of draft Evaluation Report (at end of first week of interviewing) ● Draft Evaluation Report and Summary PowerPoint Briefing (at the end of follow-up and synthesis effort and prior to Team‟s departure from Egypt) ● Final Evaluation Report and Stakeholder Presentation, following standard reporting format and branding guidelines (within 2 weeks of receiving Mission comments on draft report). An illustrative outline of the Evaluation Report is provided below: Executive Summary The Executive Summary will state the development objectives of the program/project evaluated; purpose of the evaluation; study method; findings; conclusions, lessons learned and future design implications. Table of Contents List of Acronyms and Glossary Introduction WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 68 The context of what is evaluated including the relevant history demography socioeconomic and basic political arrangements. Body of the Paper 1. Purpose and study questions of the evaluation. A brief description of the program. 2. Limitations to the study 3. Evidence, findings and analysis of the study questions. 4. Conclusions drawn from the analysis of findings stated succinctly. 5. Recommendations. Appendices shall include: 1. Follow-on program description (if appropriate) 2. Evaluation scope of work 3. List of relevant USAID targets and results (Operational Plan Program Elements) 4. List of documents consulted 5. List of individuals and agencies contacted 6. Technical topics including study methodology if necessary 7. Schedule of activities in an Excel format. 8. Questionnaires and other evaluation tools 9. Evaluation Team composition All reports are to be submitted in English in both electronic and hard copies. The Team will provide 5 printed copies of the Draft and Final Evaluation Reports and 5 printed copies of the PowerPoint presentation. USAID/Egypt‟s Business Center will assist with report production as requested. The Final Evaluation Report should not exceed 30 pages in length in its body, not including title page; Table of Contents; List of Acronyms; usage of space for tables, graphs, charts, or pictures; and/ or any material deemed important and included as Annexes. The executive summary with brief evaluation findings, conclusions and recommendations will be translated into Arabic and included in the final report. The translation will be the responsibility of USAID/Egypt. The Final Evaluation Report and PowerPoint addressing the Mission's comments should be submitted in both Word and PDF formats. Once the PDF format has been approved by the Mission, the Team will submit the Final Evaluation Report to the Development Experience Clearinghouse for archiving. VIII. ATTACHMENTS A. WPRR Scope of Work B. WWSS Scope of Work C. WPRR 2011 Work Plan D. WWSS 2011 Work Plan WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 69 ANNEX B: CAPABILITIES OF THE EVALUATION TEAM Raouf Youssef, Managing Partner of Global Development Associates, LLC with 34 years of USAID experience including USAID/Lebanon Mission Director and Director of USAID/Egypt Office of Power and Telecommunications that was responsible for the technical, institutional and policy reform for the two sectors which is relevant to the subject evaluation. Anthony Kolb, USAID/E3, Urban Health/Environment Specialist has more than 20 years of experience in the sector including environmental infrastructure development, household and community demand assessment. Heather Skilling, USAID/E3 Water/Wastewater Service Delivery Specialist has more than 20 years of experience in reforming water and sanitation services including corporate reform, regulatory/policy reform and sector restructuring. Victoria Mitchell, USAID/Egypt Program and Project Development Officer, was previously part of the Evaluation Team that performed a mid-term evaluation of the Integrated Water Resources Management II Project in Egypt. She is currently completing her training to become a USAID Evaluation Specialist. Ariel Swan, USAID/Egypt Engineering Officer WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 70 ANNEX C: LIST OF PEOPLE CONTACTED MONDAY, JANUARY 9, 2012 USAID/PSD: John Pasch, Water and Engineering Team Leader Ariel Swan, Engineer Atef Abdel Sayed, COR/WWSS Noha El Maraghy, COR/WPRR USAID/Program Office: Amani Selim, Senior Program Development Specialist/Evaluation Officer Jennifer Hansel, Private Enterprise Officer Victoria Mitchell, Project/Program Development Officer Overview of the WWSS Project (Chemonics Offices in Garden City): Ghassan Naked, Chief of Party Kathleen Sheridan, Director of Program Development Coordination Ibrahim Sabri, Senior IT Advisor Madiha Afifi, Senior HR/M Advisor Mohamed Hashem, D/COP and Engineer Atef Abdel Sayed, COR Noha El-Haddad, Publications Manager Ahmed Allam, Chief Engineer and South Egypt Team Leader TUESDAY, JANUARY 10, 2012 WPRR Project – Tariff and Customer Classification Study Workshop Preliminary Tariff Modeling Eric Rothstein, Galardi Rothstein Group Cost Function Celine Nauges, CVM David Fuente, CVM Racha Ramadan, CVM Classification Review Dennis Jackson, CH2M Hill Kurt Playstead, CH2M Hill WEDNESDAY, JANUARY 11, 2012 Meeting with the W/PPP Unit at the Ministry of Finance Atter Hannoura, Director of PPP Unit Bassel Shoirah, Project Manager for Wastewater Sector David Osgood, COP/WPRR Noha El Maraghy, COR/WPRR Overview of the WPRR Project: David Osgood, COP/WPRR Amani Abdel Wahab, Transactions Advisor Mamdouh Hassan, Engineer/Consultant WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 71 Lobna Abdel Latif, Economics and Strategy Manager Alan Bright, Reporting Monitoring and Evaluation THURSDAY, JANUARY 12, 2012 Aswan Water and Wastewater Subsidiary Ezzat Ibrahim El Sayed, Chairman and Managing Director Fatima El Zahraa, Director of Planning Minya Water and Wastewater Subsidiary Radwan Fathi, Chairman and Managing Director Saied Megahed, Director of Planning Gamal Shoukry, Engineering Department Mohamed El Houssani, Director of Water Losses SUNDAY, JANUARY 15, 2012 Ministry of Housing, Utilities, and Urban Development Project Monitoring and Evaluation Unit (PMEU) Randa El Menshawy, Director of PMEU Mohamed Shaker, Member PMEU Amr Khattab, Member of PMEU Ghada El Bedaiwy, Member PMEU Meeting with Mohamed El Alfy, Assistant Minister Briefing at WWSS (Chemonics Office is Garden City) Ghassan Naked, COP/WWSS Mohamed Hashem, D/COP Madiha Afifi, Senior HRD, Management Advisor Atef Abdel Sayed, COR MONDAY, JANUARY 16, 2012 Sohag Water and Wastewater Subsidiary Mohmoud M. Nafegh, Chairman and Managing Director Ansaf Abdel Rehim Mohamed, Director of Technical Office Hussam El Din El Zaghawi, Director of Wastewater Adel Moussa, Director of Potable Water Abu Dief Ahmed, Director of Projects TUESDAY, JANUARY 17, 2012 National Organization for Potable water and Sanitary Drainage (NAPWASD) Said Saied, Vice Chairman Somia Thaqeb, Director of APCU Wafa‟a Hassan, Cost and Schedule Engineer Wafa‟a Ahmed, Cost and Schedule Engineer Howaida Anani, Cost and Schedule Engineer Ahmed Morsi, Data Entry Clerk WEDNESDAY, JANUARY 18, 2012 Egyptian Water Regulatory Agency Mohamed Hassan, Head of Central Department of Technical Regulation WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 72 Hany Abdel Hakim, General Manager of MIS Department Mohamed Abdel Wahab, General Manager of Tariff and Pricing Department Khadiga Zaghlol, Human Resources and Training Consultant Sobhy Abdel Kader, General Manager of Standards and Compliance Department Farag Samhan, General Manager of Water Quality Department Hisham Gaafar, General Manager of International Cooperation Department Meeting with Mamdouh Raslan, Deputy Chairman, Holding Company for Water and Wastewater THURSDAY, JANUARY 19, 2012 USAID/Program Office Amani Selim, Senior Program Development Specialist Evaluation Officer USAID/PSD Team John Pasch, Water and Engineering Team Leader Ariel Swan, Engineer Noha El Maraghi, COR/WPRR Project Atef Abdel Sayed, COR/WWSS Project Briefing to USAID Mission Director Walter North, USAID Mission Director John Pasch, Water and Engineering Team Leader Ariel Swan, Engineer Noha El Maraghy, COR/WPRR Project Atef Abdel Sayed, COR/WWSS Jennifer Hansel, Private Enterprise Officer Amani Salim, Senior Program Development Specialist/Evaluation Officer MONDAY, JANUARY 23, 2012 Gharbia Water and Wastewater Subsidiary Abdel Rahman Abdel Aal, Director of Technical and Financial Affairs, Gharbia Subsidiary Amani Abdelwahab, Transaction Advisor WPRR David Osgood, COP/WPRR Mohamed Abdel Wahab, General Manager of Tariff and Pricing Department EWRA Victoria Mitchell, USAID/Egypt Program and Project Development Officer Jennifer Hansel, USAID/Egypt Private Enterprise Officer WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 73 ANNEX D: LIST OF DOCUMENTS REVIEWED USAID Evaluation Policy, Bureau for Policy, Planning and Learning, Washington, D.C. Arab Republic of Egypt, Ministry of Housing, Utilities and Urban Development, Development Policies, Water and Waste Water Sector in Egypt, September 2010 WATER POLICY AND REGULATORY REFORM PROJECT (WPRR) USAID/Egypt, WPRR Scope of Work USAID/Egypt, WPRR Project Work Plan, October 2008 – September 2012 USAID/Egypt, WPRR Project, Quarterly Report, July – September 2011, Egypt Water Policy and Regulatory Reform USAID/Egypt, Water Policy and Regulatory Reform, Fourth Year Work Plan, October 2011 – September 2012 WATER AND WASTEWATER SECTOR SUPPORT PROJECT (WWSS) USAID/Egypt, WWSS Statement of Work USAID/Egypt, Performance Monitoring Plan, May 23, 2010, Water and Wastewater Sector Support Program, Water Policy and Regulatory Reform Program USAID/Egypt, Performance Monitoring Plan: Water and Wastewater Sector Support Program, October 30, 2011 USAID/Egypt, Year 3 Work Plan, October 2010 – September 2011, Egypt Water and Wastewater Sector Support Program USAID/Egypt, Technical Assistance to the W/WW Sector – Water and Wastewater Sector Policy Reform Project II (WWSPRII), End of Contract Report, September 2009 (CH2M HILL) WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 74 ANNEX E: LIST OF EVALUATION QUESTIONS ● Where did implementation in each project to date fall short of achieving expected (or desired) results? ● What constraints impeded achieving targeted results? ● Views of USAID, implementers, and beneficiaries with respect to weaknesses/strengths in the design, implementation and management of each project? ● What are the anticipated results and implications of options related to USAID/Egypt exercising the option year on these two projects? ● Based on the experience of the two projects evaluated and the status of the Egyptian water sector, what are the facts and implications of possible short and long term future USAID assistance? ● Additional Evaluation Questions Where addressed in this report ● Assess the PMP for each project to determine whether the Plan and performance indictors truly measure project progress and achievements. ● Identify implementation problems, unmet needs, or unintended consequences or impacts, taking into account the host country environment, especially following the events of January 25, 2011. ● Progress made on each project in terms of relevance, impact, sustainability, and cost￾effectiveness. ● Confirm validity of the overall – and project-specific – development hypotheses or critical assumptions underlying the Mission‟s water and wastewater strategy and the projects that were designed, funded and implemented to make such strategy operational. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 75 ANNEX F: EVALUATION TEAM SCHEDULE WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 76 ANNEX G: LEVEL OF PERFORMANCE – COMPARATIVE ANALYSIS In its second year, WPRR worked with EWRA staff in coordination with an EU technical assistance team to select and adopt a set of 15 Level of Service indicators. These indicators include eight for water service, four for wastewater service and three for customer service. These indicators were chosen to provide information to decision makers that could guide improvements in the quality of services and maintaining such quality. In particular, chosen indicators are envisioned to eventually be used by ERWA in regulating subsidiaries through performance agreements. WPRR reports that the list of indicators is still evolving. The Evaluation Team agrees that the list should continue to evolve especially as we view four of the 15 indicators as having ambiguous comparative value. Specifically, Indicator Weakness in comparative value Indicator W3: % of population served by public taps A decrease would be positive if water service is moving from a standard of public tap provision to individual household connection, but an increase might also be positive if there are members of the population with no improved service. Indicator W4: % of population served by other methods including tankers Again, a decrease would be positive if the situation is one where service is moving from “other methods” to household connections, but on increase might also be positive if some percentage of the population has no improved service and there “other methods” are seen as improvements. Indicator WW2: % of population served with other local methods (decomposition tanks, manual collection, trenches) Again the same argument as above regarding ambiguity in the meaning of an increase or decrease in this indicator, however, in this case from a wastewater level service perspective. Indicator C3: Number of complaints per 1000 connections (total) Simply the sum of water and wastewater related complaints already proposed as separate indicators. As such, this indicator adds no additional comparative information. Eliminating these four indicators leaves the following eleven indicators:  W1 % of population served by continuous [water] supply 24 hours  W2 % of population served by intermittent [water] supply  W5 % of population not served [by improved water supply]  W6 % of planned [water supply] interruptions longer than 12 hours  W7 % of unplanned [water supply] interruptions longer than 12 hours  W8 % of [water] connections of low pressure- less than 2.5 bar  WW1 % of population connected directly to public sewerage network  WW3 % of population not served [by improved sanitation]  WW4 Number of flooding incidents per 100 km of sewerage system  C1 Number of complaints per 1000 water connections  C2 Number of complaints per 1000 wastewater connections WPRR reports on data for all eleven of these indicators only for fiscal years 2008/9 and 2009/10 for each of the 21 individual Egyptian water/wastewater subsidiaries, including each of the 13 utilities targeted by at least some of the WWSS assistance. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 77 To perform our comparative analysis, we replaced several obviously inaccurate data values flagged by WPRR and standardized5 the remaining differences in the indicator values between the two observed years. We then added these values together giving equal weight to each indicator. The resulting “level of service improvement indicator” was analyzed to discern if there was any significant difference in improvements between the 13 WWSS targeted subsidiaries and the eight non-targeted subsidiaries. Various analytical methods including a student t-test of the values of these two groups indicate no significant differences between them. It should be noted that for this simple comparative analysis in level of service indicators to be truly valid, the WWSS targeted subsidiaries should be considered to be comparable units of analysis and the “treatment” of WWSS assistance should have been assigned randomly. Clearly neither assumption is valid. Each subsidiary has a very distinct history and substantial operational differences, e.g., size of population served etc. Also, there were particular reasons that different types of WWSS assistance were targeted toward particular subsidiaries and were certainly not applied randomly 5 Raw difference minus the population average divided by the population standard deviation. WATER AND WASTEWATER SECTOR SUPPORT PROGRAM EVALUATION FEBRUARY 2012 PAGE 78 ANNEX H: DISCLOSURE OF REAL OR POTENTIAL CONFLICTS OF INTEREST FOR USAID EVALUATIONS Name Raouf Youssef Title Managing Partner Organization Global Development Associates, LLC Evaluation Position? x Team Leader Team member Evaluation Award Number (contract or other instrument) AID-263-0-12-00008 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID/Egypt Capacity Building and Policy Support in the Water and Wastewater Sector. Chemonics International, EPP-I-00-04- 00020, Task Orders No. 2 and 3 I have real or potential conflicts of interest to disclose. Yes x No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date December 26, 2011 X