FINAL PERFORMANCE EVALUATION OF THE FURTHER ADVANCING THE BLUE REVOLUTION INITIATIVE (FABRI) May 25, 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Development Group LLC (IDG). Learning Evaluation and Analysis Project (LEAP-II) Final Performance Evaluation of the Further Advancing the Blue Revolution Initiative (FABRI) May 25, 2017 Contract Number: Learning Evaluation and Analysis Project-II (LEAP-II) AID-OAA-I-12-00042/AID-OAA-TO-14-00046 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Executive Summary ...................................................................................................................... 3 Introduction ................................................................................................................................. 11 Evaluation Purpose ...................................................................................................................................................... 11 Evaluation Questions .................................................................................................................................................. 11 Project Background .................................................................................................................... 13 Overview of Water Resources Management in MENA .................................................................................... 13 FABRI Program Objectives ....................................................................................................................................... 14 FABRI Program Activities .......................................................................................................................................... 14 FABRI Reported Results ............................................................................................................................................ 14 Evaluation Methods and Limitations ......................................................................................... 16 Data Analysis Methods .............................................................................................................................................. 17 Limitations ..................................................................................................................................................................... 18 Findings and Conclusions ............................................................................................................ 19 Overall Program Findings .......................................................................................................................................... 19 Overall Program Conclusions .................................................................................................................................. 21 Evaluation Question 1: Sustainability and Potential ............................................................................................ 21 1a) Organizational structure ................................................................................................................................ 23 1b) Capacity to collect and manage member and donor funds .................................................................. 25 1c) Financial sustainability ..................................................................................................................................... 27 1d) Development and implementation of a long-term business plan ........................................................ 28 1e) Recognized convening authority among the NWC’s member base ................................................... 30 1f) Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC ......................................................................................... 31 Evaluation Question 2: Collaborations and Partnerships .................................................................................. 33 2a) How successful have small research grants and research pilots been in fostering research collaboration? .......................................................................................................................................................... 33 2b) To what extent has information around the results of pilot research projects been shared among members of the NWC? ........................................................................................................................... 36 2c) How, and to what extent, has the research generated by the NWC been used by policymakers and other stakeholders to inform water resources planning and management? .................................... 37 2d) What types of support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? ......................................................... 38 Evaluation Question 3: Incentives and Disincentives .......................................................................................... 39 3a) If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? Please consider value and benefits to date, as well as future expectations. ..................................................................................... 39 3b) Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? ............................................................................................................................................... 41 3c) Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? ...................................................................................................................................... 44 Evaluation question 4: Support to Managing Water Resources ...................................................................... 45 Recommendations ...................................................................................................................... 50 Short-term Recommendations ............................................................................................................................ 50 Long-term Recommendations ............................................................................................................................. 50 Annexes ........................................................................................................................................ 53 Annex 1: Statement of Work ................................................................................................................................... 53 Annex 2: Evaluation Methods Matrix ..................................................................................................................... 66 Annex 3: Water Resource Management in Morocco, Jordan, and Oman .................................................... 68 Annex 4: Performance Indicators and Evalaution Findings ............................................................................... 78 Annex 5: FABRI Final Report Lessons Learned and Evalution Findings ......................................................... 81 Annex 6: List of Interviewees ................................................................................................................................... 83 Annex 7 Keyword Analysis ....................................................................................................................................... 86 Annex 8: Organizational Capacity Assessment (OCA) Score Sheet .............................................................. 88 LIST OF TABLES Table 1: Hierarchy of Changes Analysis ..................................................................................................................... 21 Table 2: OCA Section 1 - Governance and Legal Structure ................................................................................. 24 Table 3: Social Return on Investment ......................................................................................................................... 25 Table 4: OCA Section 2 - Financial Management and Internal Control Systems ............................................. 26 Table 5: OCA Section 7 - Organizational Management and Sustainability ........................................................ 29 Table 6: OCA Section 5 - Program Management .................................................................................................... 31 Table 7: OCA Section 4 - Human Resource Systems ............................................................................................ 32 Table 8: Outcome Mapping Analysis ........................................................................................................................... 47 LIST OF FIGURES Figure 1: Total Renewable Water Resources per Capita, MENA Region ......................................................... 13 Figure 2: Improved water sources--MENA Region ................................................................................................. 13 Figure 3: Keyword analysis on program value, benefits and contributions ....................................................... 19 Figure 4: Gender of Key Informants and Survey Respondents ............................................................................. 20 Figure 5: Keywords Contexts and Sentiments ......................................................................................................... 22 Figure 6: Sustainability, Management, and Effectiveness ......................................................................................... 23 Figure 7: Constraints to Achieving Objectives ......................................................................................................... 24 Figure 8: Keyword analysis on funding/grant/budget comments .......................................................................... 28 Figure 9: Keyword analysis on business plan ............................................................................................................ 30 Figure 10: Survey responses on MENA-NWC convening authority ................................................................... 30 Figure 11: Survey Responses on MENA-NWC Management Expectations ...................................................... 32 Figure 12: With what frequency have you voluntarily contacted the following individuals associated with MENA-NWC during the past two years? .................................................................................................................. 34 Figure 13: Network of Contacts .................................................................................................................................. 35 Figure 14: Survey responses on collaboration .......................................................................................................... 35 Figure 15: Participation of women in grants programs ......................................................................................... 36 Figure 16: Publication of Research Results ................................................................................................................ 37 Figure 17: Survey responses on NWC’s engagement of policymakers .............................................................. 37 Figure 18: Support and initiatives that are most likely to increase or sustain research collaboration and information sharing among member institutions ..................................................................................................... 38 Figure 19: Keyword Analysis of Terms Relating to Incentives, Disincentives, and Value .............................. 39 Figure 20: Value of the MENA-NWC ......................................................................................................................... 40 Figure 21: Survey responses on MENA-NWC achievements .............................................................................. 40 Figure 22: Responses of men and women on the statement that the MENA-NWC has helped them advance in their career ................................................................................................................................................... 41 Figure 23: Organizations of people who have made the most contacts with other members ..................... 42 Figure 24: Roles of people who have made the most contacts with other members .................................... 43 Figure 25: Personal characteristics of people who have made the most contacts with other members ... 43 Figure 26: Countries of people who have made the most contacts with other members ............................ 44 Figure 27: Effectiveness of the MENA-NWC in addressing critical water issues ............................................ 47 i ACRONYMS ACWUA Arab Countries Water Utilities Association AfWA African Water Association COR Contracting Officer’s Representative DAI Development Alternatives, Inc. DEC Development Experience Clearinghouse ESCWA United Nations Economic and Social Commission of Western Asia FABRI Further Advancing the Blue Revolution Initiative GOJ Government of Jordan GWOP Global Water Operators Partnerships Alliance GYGA-MENA Global Yield Gap Atlas for the Middle East and North Africa IAV Institute of Agronomy and Veterinary Science Hassan II ICA Independent Consultant Agreement ICARDA International Centers for Agricultural Research in Dry Areas ICBA International Center for Biosaline Agriculture INRGREF National Institute for Research in Rural Engineering, Water, and Forestry IRESA Institution of Agricultural Research and Higher Education IWA International Water Association IWMI International Water Management Institute IWRM Integrated Water Resources Management KII Key Informant Interview MEDRC Middle East Desalination Research Center MENA NWC Middle East and North Africa Network of Water Centers of Excellence MPE Mekong Partnership for the Environment MRC Mekong River Commission NCARE National Center for Agricultural Research and Extension NGO Non-Governmental Organization NRW Non-Revenue Water NWRC National Water Research Center OCAT Organizational Capacity Assessment Tool OES State Department’s Oceans, Environment, and Science Bureau OMEP Office of Middle East Programs OSU Oregon State University PEER Partnerships for Enhanced Engagement in Research PIP Performance Improvement Plan PO Purchase Order PR&D Policy, Research and Development QEERI Qatar Environment and Energy Research Institute QNFSP Qatar National Food Security Program R&D Research and Development Regional Environment Office RFA Request for Applications RFP Request for Proposals RSS Royal Scientific Society RWS Reference Weather Stations SIDA Swedish International Development Agency SQU Sultan Qaboos University TAMU Texas A&M University TFDD Transboundary Freshwater Dispute Database ii TOR Terms of Reference TRG Training Resources Group, Inc. UAE United Arab Emirates UJ University of Jordan UNL University of Nebraska, Lincoln USAID United States Agency for International Development USG United States Government UTU Utah State University WASH Water, Sanitation, and Hygiene WEEC Water, Energy and Environment Center WWW World Water Week YWP Young Water Professionals YWSP Young Water Scientist Partnerships Small Grants Program 3 EXECUTIVE SUMMARY INTRODUCTION The purpose of the evaluation is to validate results achieved by the Further Advancing the Blue Revolution Initiative (FABRI) and assess the program’s progress in building an operational and sustainable network of Middle East and North Africa (MENA) water centers. Based on the answers to the evaluation questions discussed below, the evaluation team derived lessons learned from the implementation of FABRI for USAID to take into consideration when designing future activities in this area. PROJECT BACKGROUND The MENA region is beset with many different issues relating to the availability of freshwater, the resolution of trans-boundary water disputes, dwindling aquifers and, more recently, serious concerns about water supply to vulnerable communities living under unstable conditions. Increases in demand for water, as a result of growing urbanization and industrialization in the region, have exacerbated these issues. The majority of the water in the region is used – and much of it is wasted – in agriculture. Unsustainable agricultural practices contribute significantly to the shortage of water. FABRI is a five-year program implemented by Development Alternatives, Inc. (DAI) from September 2011 to June of 2016. The program’s primary objective was to promote sustainable management of water resources in MENA. Its theory of change states that sustainable management of water resources can be achieved through: 1) fostering collaboration and partnerships between major research institutions, 2) establishing a network of professionals and stakeholders in water management, 3) promoting dialogue among policy makers, and 4) encouraging continued communications between experts and policymakers. Program designers and implementers established the Middle East and North Africa Network of Water Centers of Excellence (MENA-NWC) as the main platform for the performance of these activities. EVALUATION METHODOLOGY The evaluation draws on four data sources: 1) secondary data and information contained in project documents (both qualitative and quantitative), 2) key informant interviews (KIIs), 3) group interviews, and 4) an online survey of NWC participants. The evaluation used qualitative data collection and analysis methods, supported by quantitative data and analysis where possible. The evaluation sought to capture perceptions, intents, and activities of the key stakeholders, including Government officials, research institutions, MENA-NWC staff and board members, and researchers (grantees). A number of analysis methods were used to analyze the data including USAID’s Organizational Capacity Assessment Tool (OCAT), frequency of response/trend pattern analysis, correlation and triangulation of data, and outcome mapping of qualitative data. 4 FINDINGS AND CONCLUSIONS Evaluation Question 1: To what extent has the Network of Water Centers (NWC) developed into a sustainable entity capable of addressing critical water issues in the MENA region? What more needs to be done to ensure the future viability of the Network? a) Organizational structure MENA-NWC is a legally registered non-profit organization in Jordan. The NWC was initially established as a non-profit association in the United States but shifted its registration to Jordan during the final 18 months of the FABRI program. The existing governance structures of the NWC are the NWC Secretariat and the Board of Directors. The network currently has 25 member centers in the region. MENA-NWC’s current organizational structure is not sufficient for it to conduct the activities necessary for achieving sustainability. Roles and responsibilities of the governance structure exist, but they are weak and do not fully align with the organization’s current vision and structure. External board members were initially selected by FABRI. During FABRI, the NWC Secretariat was required to check with FABRI and/or the board of directors to make any executive decisions. The NWC’s bylaws and policies are incomplete or outdated, and no structured settings exist within the NWC for its member centers to exchange ideas and discuss problems or opportunities for the network. The NWC should encourage engagement of its member centers to discuss and garner support for its future organizational strategy. b) Capacity to collect and manage member and donor funds MENA-NWC requires capacity building in the area of fundraising. The annual membership fee was initially set at US$ 4,000 for each water center. Many member centers expressed concerns that the fee was too high and did not pay the fees due to financial constraints. As a result, the NWC Secretariat had difficulty obtaining funds from the centers. Recently, the membership fees were reduced from US$ 4,000 per member to US$ 250. It is too early to determine the effect on membership fee collection. It is important for the MENA-NWC to focus on improving its value to members to encourage payment. The Secretariat of the NWC lacks the capacity to raise funds from other sources. To retain current funding levels including membership fees and donor contributions, MENA-NWC needs to develop a clearly defined purpose. The capacity to collect and manage member and/or donor funds is partly dependent upon members’/donors’ perception of the expected return on their investment. The social return on investment analysis indicated that the grants programs and the network itself had a moderately positive return. MENA-NWC should improve the members’ perception on how it is spending membership fees to ensure it is adding value for its members. MENA￾NWC’s financials were not transparent and as a result many interviewees felt that too much money was spent on non-essential matters, such as frequent travel by management and expenses for hosting large conferences. c) Financial sustainability Funding is the most critical constraint to MENA-NWC. MENA-NWC members and most survey respondents cited funding as the major constraint for the program achieving its objective. At present, MENA-NWC has a Low to Basic Capacity OCA score on the financial section of the assessment. After June of 2016, there were insufficient revenues to pay the Director and an additional staff member. The MENA-NWC Secretariat was struggling to survive without the support of a donor agency until recently when the Swedish International Development Cooperation Agency (SIDA) has offered to grant the network (US$ 450,000) to cover staff salaries and rent for the next two years 5 effective February 2017. The Government of Jordan confirmed that it will also provide a grant but solely for the maintenance of the organization. These funds will allow MENA-NWC to function only at a very basic level over the short-term making the organization unsustainable over the long-term. To achieve financial sustainability, MENA-NWC must develop and implement a fundraising strategy that includes non-donor income generation activities such as philanthropy, engagement of government counterparts, etc. as well as effectively engage members to increase membership fees. MENA-NWC has the necessary funds to maintain operations at the current level. With its current levels of funding and membership fees, MENA-NWC can maintain its activities and perhaps hire additional staff. However, in order to accomplish anything beyond maintenance of current level of activities, MENA-NWC will need additional funds. Without new initiatives, such as additional grant programs, MENA-NWC will have difficulty convincing its current members and potential new members to participate in the network and therefore struggle to attract additional funds from donors. d) Development and implementation of a long-term business plan The business plan is weak and requires significant revisions. The business plan was prepared with the active engagement of the MENA-NWC Founders Committee elected by FABRI. The business plan does not reflect MENA-NWC’s current vision, mission, and values. It is not based on an adequate analysis of strengths, weaknesses, opportunities, threats, and realistic resource requirements and availability. The business plan does not invest the board with any authority beyond technical assistance, and the revenues predicted in the business plan budget were not realized in 2016 or thereafter. The business plan is not perceived as a collaborative effort. Interviewees who were aware of the business plan did not see it as a collaborative effort in terms of engaging member centers or regional stakeholders, and spoke of the business plan in negative terms. It is expected that a new board will be elected by the membership and will develop a new business plan that more closely fits the objectives of the MENA-NWC moving forward. e) Recognized convening authority among the NWC’s member base MENA-NWC has been generally effective and influential in convening regional meetings. Due to financial constraints, MENA-NWC has not engaged in any activities to convene or engage with members except with respect to maintaining the website since the end of FABRI. The extent to which the MENA-NWC can engage in new training, workshops, and conferences is dependent upon obtaining additional funding for these activities. Interviews and survey responses strongly suggest that the MENA￾NWC has a good reputation among water professionals in the region. With the election of the new board and the injection of additional funds, the MENA-NWC could establish itself more firmly as a multi-lateral influential research entity in the region. Survey respondents and interviewees were most favorable about the potential for the MENA-NWC to sponsor fruitful regional meetings and workshops and disseminate valuable research to decisionmakers but questioned the MENA-NWC’s ability to increase influential policymakers’ participation. f) Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC While the NWC Secretariat has basic capacity in program management, its human resource system is weak. The OCAT score on human resource systems (which evaluates the adequacy of staffing and recruitment and retention of staff among others) is the lowest of NWC’s organizational capacity indicators. At the time of the evaluation, the NWC Secretariat consisted of one Director based in Jordan, and a co-director based in the US. MENA-NWC staff and current board members recognize that more staff is needed to for the network to continue. 6 Evaluation Question 2: To what extent has the NWC been successful in strengthening partnerships, collaboration and information sharing between research institutions in the region, as well as between research institutions and the private sector? a) How successful have small research grants and research pilots been in fostering research collaboration? The small research grants and research pilots have been generally successful in fostering research collaboration in the region. Grant recipients and participants in research pilot projects communicated regularly with others in the program. The research projects also introduced research techniques that employ accessible modern instrumentation, facilitated access to site-bound instruments, promoted collaboration with regional actors, and linked institutions with similar objectives. The pilot research projects also built the capacity of communities among universities and across institutions. The small research grants program was especially helpful in bringing women scientists into water research in the member countries. The small research grants program also fostered cooperation within countries. In addition to enhancing collaboration across countries in the region, several interviewees noted that the small grants program fostered cooperation within countries. Research institutions and government entities that previously had little interaction were brought together at FABRI workshops and were given the opportunity to interact and discuss of their research findings for the first time. Relationships were built in some cases, and a few research projects conducted under FABRI were scaled up in collaboration with government institutions, local NGOs, and communities. b) To what extent has information around the results of pilot research projects been shared among members of the NWC? Results of pilot research projects were sufficiently shared among members of the NWC. NWC organized workshops and conferences to present and share research results among the member centers. Most pilot project researchers reported that the results of their research were either already published or in-progress to be published. The pilot research projects helped to build learning communities within universities and across institutions by engaging post-graduate students in the projects and advancing their research work in universities. c) How, and to what extent, has the research generated by the NWC been used by policymakers and other stakeholders to inform water resources planning and management? The results generated by the research pilots do not seem to have been sufficiently used by policymakers, government agencies, and the private sector to inform water resources planning and management. Government and private sector representatives had limited engagement with the program. Interviewees stated that results generated by FABRI-sponsored research have not been used by policymakers, government agencies, or the private sector to inform water resources planning and management, with the exception of a few centers. There are conflicting accounts as to why this situation arose. Although the results and reports produced by FABRI are valuable in terms of non￾conventional water resources, the weak connection between research and industry was noted frequently by interviewees and survey respondents. d) What types of support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? Additional grants programs and regional workshops or meetings are the most likely initiatives to increase research collaborations. Researchers and research institutions were most interested in the possibility of additional support for projects, networking, and collaboration. In addition 7 to new funding, MENA-NWC will need to establish a systematic method of engaging partner centers and keeping researchers linked to each other via scientific groups or thematic interest groups. MENA￾NWC will also need to continue soliciting, managing, and funding grants for longer periods of time. Further engagement of the NWC Secretariat with governments and the private sector is also critical to bring governments and private institutions of the region into the picture. Evaluation Question 3: What are the key incentives (and disincentives) among NWC member institutions with regard to greater engagement in the network? a) If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? Please consider value and benefits to date, as well as future expectations. Key incentives that foster greater engagement are: 1) the opportunity to share knowledge about applied research, 2) availability of research funds, and 3) support for the training of a new generation of scientists. Participants believe the MENA-NWC is a valuable regional entity and are supportive of the idea of a regional network of research institutions. The primary benefits to participating in MENA-NWC is its ability to support collaboration and bring scientists together. Interviewees and survey respondents most often identified the benefits of MENA-NWC membership as collaborating and receiving funding for grants programs. KIIs also confirmed that the program was especially valuable in establishing lasting linkages. FABRI was successful in building the capacity of researchers and women scientists. FABRI generally raised the capacity of professionals, provided opportunities to young researchers, and helped them advance their careers. In particular, women grantees agreed that the MENA-NWC had helped them advance in their careers. The small grants programs were especially successful in incorporating women scientists into water research in the member countries. b) Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? Researchers in universities and academic research institutes are the most engaged in the MENA-NWC. Grantees maintained the highest level of communication which suggests that the grants program was one of the most important activities that encouraged active engagement. They maintained contacts and continued with projects that started under FABRI. While the MENA-NWC participants showed varying degree of engagement, no clear pattern emerged from their country of origin. According to the online survey, a higher percentage of female NWC participants (66 percent) engaged in communication with other participants than male participants. Overall, those who reported higher engagement perceived that the MENA-NWC generally exceeded expectations. Those reporting fewer engagements were not significantly more critical of the program results. c) Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? MENA-NWC will need to define clear objectives and issues to be addressed. The networking and collaboration under FABRI was excellent. However, ensuring sustainability of the network and securing future funding is essential for MENA-NWC to continue benefitting its members. Further work is needed to engage other actors, including the private and public sectors to leverage the full benefits of the regional network. Many interviewees recounted the network’s inability to engage people beyond the research sector. In order to be of benefit to its members in the future, MENA-NWC will have to move beyond simply providing a platform for information sharing on 8 research. MENA-NWC should engage a variety of stakeholders in order to make membership more attractive. Evaluation Question 4: To what extent has FABRI effectively supported the ability of countries to plan and manage their water resources in the following areas: watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture? FABRI did not substantially improve the host governments’ ability to manage water resources of the region. FABRI succeeded in enhancing collaboration and networking between regional actors on the region’s water resource management. However, FABRI failed to substantially engage government officials and thus the linkage between research and government priorities and/or interests was weak. Also, FABRI sponsored collaborative researches on water issues were not applied or scaled-up during the life of the program. While several research projects are continuing with funds from additional donors, there is still a large gap between academic research and application and scaling￾up. The capacity building through FABRI was mainly limited to individual grantees and therefore lacked the leverage to influence governments’ decision making process. Among the three areas (protection of freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture), FABRI had the largest effect on the improvement in the productivity of water used in agriculture. For example, a pilot research in Jordan and Tunisia resulted in a significant increase of crop yield while optimizing the productivity of water delivered for irrigation. However, FABRI’s influence on the countries’ watershed-based efforts to protect freshwater supplies was limited, and FABRI had the least effect on the regulation of groundwater development. RECOMMENDATIONS Short-term Recommendations for MENA-NWC MENA-NWC Organizational Structure  Adopt a clear vision and mission statement based upon member center defined needs.  Establish a proper system of accountability and performance measurement for the Secretariat to ensure effective use of authority.  Develop a new business plan that: 1) is developed with the participation of member centers, 2) sets realistic goals based upon current funding, and 3) contains a plan for obtaining funds from the other donors, the private sector, and participating governments.  Develop a plan for building on and extending linkages, including with government and the private sector.  Update and maintain the NWC database with the participation of member centers. Support in Research Collaboration and Information Sharing  Encourage more engagement and participation from the member centers by reaching out centers that have been inactive for the past year.  Continue to provide capacity building for universities involved in research including technical assistance and advice on preparing proposals to obtain funding.  Consider sponsoring network-wide small grants competitions for young scientists and women scientists to the extent permissible with the current funds.  Provide valuation services to help research institutions to identify opportunities for funding and connections with the private sector. 9  Conduct a baseline survey of current members to identify the most important water resource management issues in the region for MENA-NWC to engage in. Long-term Recommendations for MENA-NWC Hold annual planning workshops to engage members and solicit their feedback. An annual planning workshop should take place with the management and board of directors of the MENA-NWC, and heads of the water centers to suggest, vet, discuss, and agreed on the next year's activities and budget. Develop a flexible management structure for the MENA-NWC that promotes communication and collaborative between all stakeholders based upon:  A full capacity assessment after one year conducted by an independent contractor.  A communication plan that includes recommendations for expanding the MENA-NWC’s presence on social media and maintaining/upgrading the website to make it more interactive and up to date.  A network analysis after one year mapping the communications between centers and participants. Develop a public outreach program with the participation of member centers that focuses on:  Raising community awareness of effective uses for wastewater.  Fully engaging with the government and private sector.  Raising community awareness on important challenges to effective water resources management including water losses, the effects of climate change, and the interactions between food security, energy, and agricultural water use. Explore other non-donor income generation activities to achieve financial sustainability. MENA-NWC should explore other options of financing the network other than donor funding and membership fees. Channels of funding should be broadened to philanthropy, engagement of government counterparts, and the private sector. Diverse options should be considered such as involving the private sector by funding specific small grants competitions focused on a research area in which they have an interest. Facilitate the development of local networks at a country level to create the needed basic platforms for cross country collaborations and work. Promote the inclusion of women in water research throughout the region by:  Performing a full assessment of the challenges and opportunities to achieving gender balance in activities in the specific member countries.  Based on the results of the assessment, develop a plan in collaboration with member centers to address challenges and take advantage of opportunities. The plan should include specific steps that can be taken. Recommendations for Future USAID Water Programs Capitalize on the network of regional water centers established during FABRI. MENA-NWC and FABRI initiated valuable regional connections of water researchers. Future USAID water sector programs should leverage the network and its database to engage in further collaboration at the regional level. Ensure that all relevant key stakeholders, including government agencies, private sector and policymakers, are actively involved during program design. Engagement with private and public sector stakeholders proved to be a challenge for the MENA-NWC. Engaging with the key 10 stakeholders during program design and the early stages of project implementation will generate buy-in and increase the chances of successful engagement during implementation. Conduct a comprehensive assessment of past water projects implemented by other donor agencies in each respective country or region and build on lessons learned. Build the capacity of women scientists and technicians. FABRI’s impact on women scientists was one of the long-term successes of the program. It is vital that this aspect of the program continue to operate. USAID should enhance women scientists’ technical competencies in integrated water resources management by providing grants, scholarships, and/or training to women in engineering or sciences related to water conservation. Maximize the positive impact of FABRI’s pilot research projects on the lives of rural women. A number of new initiatives, such as hygiene enhancements through improved wastewater management and rainwater harvesting, were explored during FABRI. Future USAID projects should leverage on these research project initiatives and explore scaling up these activities to improve standards of living and access to water for women in rural areas. Recommendations for Future USAID Regional Network Programs Consider a longer implementation timeline for establishing regional networks. Establishing a fully function regional network requires extensive preparations and administrative procedures. For instance, the United Nations Economic and Social Commission of Western Asia (ESCWA) and GIZ took over three years to set up ACWUA as a regional entity (Spring 2006 – July 2009). Even in 2015, USAID provided support to ensure its sustainability. Future USAID programs aiming to establish a regional network should consider a flexible implementation timeline to set up a successful regional entity. Develop a strategic business plan with a clear roadmap for the financial sustainability of a network. MENA-NWC focused on donors as a sole primary source of revenue in addition to membership fees and failed to proactively pursue capital. For future USAID projects, a business plan should be developed in collaboration with the board members and other relevant stakeholders. The business plan should identify tools and risk mitigation options to reach financial sustainability. The plan must also be periodically updated and reviewed to accommodate potential fluctuations in revenue and regional needs. Carefully design the organizational governance framework and lines of communication to ensure effective management and transparency. The relationship between the management (Secretariat, Board of Directors), USAID project implementer, and network/association members should be reviewed closely to establish a robust and well-functioning governance framework. FABRI’s Board of Directors expressed concerns that they were not adequately consulted nor involved in the strategic planning of the MENA-NWC. Future USAID programs should refrain from the project implementer taking on a major managing role of the network to ensure effective management of the network post-USAID funding. 11 INTRODUCTION EVALUATION PURPOSE The purpose of the evaluation is to validate results achieved by the Further Advancing the Blue Revolution Initiative (FABRI) and assess the program’s progress in building an operational and sustainable network of Middle East and North Africa (MENA) water centers. Based on the answers to the evaluation questions discussed below, the evaluation team derived lessons learned from the implementation of FABRI for the United States Agency for International Development (USAID) to take into consideration when designing future activities in this area. Reflecting FABRI’s causal theory of change, and in accordance with the four evaluation questions (EQs) outlined in the Statement of Work (SOW) (see Annex 1), the evaluation focuses on the following key aspects of the Middle East and North Africa Network of Water Centers of Excellence (MENA-NWC): 1) capacity building of regional water resource professionals; 2) sustainability of networks and relationships established; 3) quality and quantity of the research produced; and 4) maintenance of research collaboration in the region. The data collection for the evaluation took place December 8, 2016 through March 20, 2017. EVALUATION QUESTIONS With the exception of EQ4, the evaluation questions are focused on the results related to the MENA￾NWC. The evaluation questions and sub-questions detailed in the SOW are as follows: 1) To what extent has the Network of Water Centers (NWC) developed into a sustainable entity capable of addressing critical water issues in the MENA region? What more needs to be done to ensure the future viability of the Network? In particular, consider the NWC’s strengths and liabilities in the following areas: a. Staffing and organizational structure b. Capacity to collect and manage member and donor funds c. Financial sustainability d. Development and implementation of a long-term business plan e. Recognized convening authority among the NWC’s member base f. Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC 2) To what extent has the NWC been successful in strengthening partnerships, collaboration and information sharing between research institutions in the region, as well as between research institutions and the private sector? a. How successful have small research grants and research pilots been in fostering research collaboration? b. To what extent has information around the results of pilot research projects been shared among members of the NWC? c. How, and to what extent, has the research generated by the NWC been used by policymakers and others stakeholders to inform water resources planning and management? d. What types of support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? 12 3) What are the key incentives (and disincentives) among NWC member institutions with regard to greater engagement in the network? a. If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? Please consider value and benefits to date, as well as future expectations. b. Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? c. Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? 4) To what extent has FABRI effectively supported the ability of countries to plan and manage their water resources in the following areas: watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture? 13 PROJECT BACKGROUND OVERVIEW OF WATER RESOURCES MANAGEMENT IN MENA The MENA region is beset with many different issues including the availability of freshwater, the resolution of trans-boundary water disputes, dwindling aquifers and, more recently, serious concerns about water supply to vulnerable communities living under unstable conditions. Increases in demand for water, as a result of growing urbanization and industrialization in the region have exacerbated these issues. Farming and grazing practices are a large part of the problem. In fact, 85 percent of water in this region is used by the agricultural sectors—and much of it is wasted. This is particularly striking considering the fact that innovation in the management of rivers and perennial springs developed in this part of the world earlier than anywhere else. The region now exhibits the most long lasting, and deleterious effects of unsustainable agricultural practices. For the past 25 years, renewable water resources in the region remained steady while the amount of water available per capita has steadily decreased: Figure 1: Total Renewable Water Resources per Capita, MENA Region A great deal of progress has been made in the past 25 years in improving water sources in the region as shown in Figure 2; however, without the development of greater renewable water sources, this improvement cannot be sustained. Figure 2: Improved water sources--MENA Region Please see Annex 3 for a discussion of water resource management in Jordan, Morocco, and Oman. 400 500 600 700 800 900 1000 1992 1997 2002 2007 2012 2014 Meters3 per capita Year 75 80 85 90 95 100 2007 2008 2009 2010 2011 2012 2013 2014 2015 Percentage Year Improved water source, urban (% of urban population with access) Improved water source, rural (% of rural population with access) Improved water source (% of population with access) 14 FABRI PROGRAM OBJECTIVES FABRI is a five-year program implemented by Development Alternatives, Inc. (DAI) from September 2011 to June of 2016. The FABRI program’s primary objective was to promote sustainable management of water resources in MENA. Its theory of change states that sustainable management of water resources can be achieved through: 1) fostering collaboration and partnerships between major research institutions, 2) establishing a network of professionals and stakeholders in water management, 3) promoting dialogue among policymakers, and 4) encouraging continued communications between experts and policymakers. Program designers and implementers established the MENA-NWC as the main platform for the performance of these activities. FABRI was originally designed to be a regional initiative covering Yemen, Tunisia, Morocco, Jordan and Oman. However, this did not fully take place due to regional conflicts and approximately one quarter of FABRI funds were reallocated to Sub-Saharan Africa. FABRI PROGRAM ACTIVITIES The main activities of FABRI can be summarized as follows: 1) MENA-NWC incorporated as a nonprofit association in Washington, D.C., in December 2012, and in Jordan during the final 18 months of the program. 2) MENA-NWC prepared an application to the U.S. Internal Revenue Service (IRS) for tax-exempt status in 2012, drafted the five-year strategic business plan and by-laws in 2014 (updated on June 1, 2015), and convened the Board of Directors in 2015. 3) Committed US$ 4.9 million in grants and subcontracts to support applied research in: water use efficiency and productivity, groundwater, nonconventional water, water/energy/food nexus, and water and sanitation. Grants programs initiated in December 2013. 4) In 2014, secured partnerships with four private sector companies in the Middle East and engaged with multi-country teams focused on applied research aligned with that companies’ corporate interests. 5) Launched small grants programs in the Middle East targeting young researchers and water practitioners to encourage their professional growth and advancement in late 2014. 6) Since October of 2014, raised the profile of MENA-NWC and AfWA through supporting participation in international events such as the Stockholm International Water Institute’s World Water Week, International Water Association’s World Water Congress and Exhibition, and regional conferences and meetings. 7) Launched interregional partnership between ACWUA and AfWA to reduce non-revenue water through technical and administrative audits of water utilities and development of performance improvement plans in January 2015. 8) In support of regional knowledge sharing, launched a website for MENA-NWC and an online Technical and Managerial Communities of Practice in December 2015. FABRI REPORTED RESULTS Although the evaluation questions do not directly address the extent to which FABRI met its performance goals, the final monitoring report on program results is relevant to the evaluation in that it: 1) provides a snapshot of the end of program evidence presented by program implementers for having met significant benchmarks; 2) provides a baseline against which to measure the sustainability of those results eight months after the program ended; 3) can be analyzed to determine whether the results articulated could actually be achieved by meeting the benchmarks established by the program; and 4) provides a framework for the discussion of the most significant changes reported by stakeholders in interviews and surveys and the extent to which those changes were intended or unintended. 15 At the outset of the contract, USAID identified seven results for FABRI: 1) Establish an operational Middle East and North Africa Network of Centers of Excellence (MENA NWC) 2) Strengthen Integrated Water Resources Management Programming. 3) Access to Clean Water and Sanitation Improved in Target African and Middle Eastern Countries 4) Strengthen Research and Development Capacities in Irrigation, Groundwater Management, and 5) Drought Risk Assessment and Mitigation 6) Strengthen Trans-boundary Water Cooperation in Key River Basins 7) Enhance Technical and Outreach Capacity of USAID Staff in Water and Sanitation Programming The chart in Annex 4 shows the results of the evaluation team’s findings with respect to each indicator. Summary findings with respect to these indicators during the evaluation are as follows (for detailed findings and evidence please refer to the Findings and Conclusions Sections): 1) The number of governance structures for the MENA-NWC listed at the end of the program was reported as nine, exceeding the target number of five (including the director, committees, board members and working groups). The number of existing governance structures is now two (the Director and the Board) 2) It was projected at the end of the program that $25 million would be raised from a combination of bilateral, multilateral, foundation, corporate, government, and individual donors. At the end of the program about $450,000 were reported as raised with unstated amounts pending from SIDA, the Government of Jordan, the World Bank, and the Government of Qatar. To date a total of $490,000 have been raised from the Government of Jordan and SIDA. 3) The number of private sector partners was reported as 30, exceeding the target of 15. Most of them were participating in the grants programs that are now ended. There are now, according to the Center’s Director, no private sector partners contributing to the MENA-NWC. 4) The target for public and private sector entities that apply FABRI researched technologies and practices was 12 but the actual number reported was one. There has been one new application of FABRI research in a community in Morocco, and two reported by NCARE and the Royal Scientific Society (RSS) in Jordan. 5) It was projected that 15 private sector sanitation interventions would be accomplished by program end. The actual cumulative number listed in the Annual Report was zero. The evaluation team’s interviews, however, have revealed that there were two in Jordan and one in Morocco. 6) It was estimated that 10 technologies or practices would be under research and 10 would be under field testing by the end of the program. The actual number was listed as 13 and six respectively. Research continues on a number of projects. However, to date, there are only three that have undergone field testing since the end of the program. FABRI’s Final Report prepared by the program implementers also listed a number of lessons learned and comments on the MENA-NWC’s finances, management, and sustainability. Annex 5 summarizes the findings relating to the lessons learned and indicate the extent to which these were confirmed, clarified, or refuted by evaluation data. 16 EVALUATION METHODS AND LIMITATIONS The evaluation used qualitative data collection and analysis methods, supported by quantitative data and analysis where possible. The evaluation sought to capture perceptions, intents, and activities of the key stakeholders, including Government officials, research institutions, MENA-NWC staff and board members, and researchers (grantees). This section describes the overall design, data collection tools, data collection methods, and data analysis methods. The first phase of data collection included the following:  Desk Review: The evaluation team reviewed FABRI program documents including program statements, annual reports, quarterly reports, and special studies. The desk review provided the team with a comprehensive understanding of the Moroccan, Jordanian and Omani contexts, focusing on water issues in each of the countries. Furthermore, the evaluation team was able to identify key stakeholders, work themes, gaps, and opportunities during the desk review. The documents served as a key source of information, and were used to shape the field work.  Key Informant Interviews (KIIs): In addition to consultation with USAID staff, the team interviewed the four primary FABRI program managers based in Arlington, VA. During the second phase of data collection, the evaluation team carried out fieldwork in Morocco, Jordan, and Oman. The countries were selected by USAID. The evaluation team performed:  KIIs: In the three countries visited by the evaluation team, 34 KIIs were conducted with FABRI sub-contractors, NWC-MENA leadership and board members, grantees, researchers, and Ministry representatives. In addition, three interviews with researchers in Tunisia were conducted via telephone by a French-speaking member of the team. KIIs were conducted to explore issues in-depth, to triangulate data collected in the desk review, and to solicit the input of grantees and NWC-MENA. Interviewees were selected based on participation and implementation in Morocco, Jordan, and Oman (see Annex 6 for the list of interviewees). The evaluation team used questionnaires to guide the KIIs.  Group Interviews: Group interviews in Jordan were conducted at three centers. The interviews were designed to investigate the capacities and priorities of researchers related to water management, as well as their views concerning research grants and projects and pertaining to the sustainability of the NWC-MENA network. To analyze the data collected during the group interviews, the evaluation team utilized the most significant change approach, which allowed the evaluation to touch on the significant change that occurred to the researchers’ professional careers due to their participation in the FABRI program. The third phase of data collection involved:  Online survey: The survey was designed to assess the sustainability and effectiveness of NWC￾MENA in achieving regional goals. The survey was sent to all interviewees and other identified participants (109 total) including research advisors, grantees or main researchers, private sector representatives, NWC-MENA representatives and board members, governmental officials, and university partners. The survey contained more close-ended specific questions than the semi￾structured KIIs and gave respondents the opportunity to express their opinions on scaled questions where their responses can be more easily quantified in descriptive statistics. There 17 were 49 responses to the online survey. Only 34 out of 49 survey respondents specified their gender (26 men and 8 women) and those who did not specify their gender did not provide any gender identifying characteristics such as their given names.  Additional KIIs in Tunisia and the US: Additional interviews were conducted with individuals identified during the fieldwork phase, and by one evaluation team member, with representatives of Institute of Agriculture Engineering and Forestry (INGREF) in Tunisia. DATA ANALYSIS METHODS Capacity/Performance Assessment The evaluation team used USAID’s Organizational Capacity Assessment Tool (OCAT) to assess the MENA-NWC’s organizational capacity.1 The OCAT has seven sections: 1) Governance and Legal Structure, 2) Financial Management and Internal Control System, 3) Administrative and Procurement Systems, 4) Human Resource Systems, 5) Program Management, 6) Project Performance Management, and 7) Organizational Management and Sustainability. While it is more commonly used where capacity building indicators have been established from the beginning of programming, the OCA can be applied to the results of programs that have been completed. For each of the seven sections, evaluators assessed the current stage of capacity using the steps outlined in the capacity building tool. Each area was given a score of 1 (low capacity) to 4 (strong capacity) as per the OCA methodology. Frequency Response/Trend Pattern Analysis The evaluation team conducted keyword analysis and sentiment analyses of aspects of the program frequently mentioned in interview statements. The analysis also included responses from open ended survey questions. The findings section presents bar charts illustrating the positive and negative instances of keywords relating to those aspects of FABRI and the MENA-NWC. The keywords were first reduced to word stems so that all tenses and forms of the words appearing in interviews and responses were recorded. Each sentence containing these words was read in context to determine whether the speaker was using them in a positive negative or neutral sense. Expected frequencies (Comp. Freq.) were assessed based upon the number of appearances of each term in the FABRI Final Report. The context and sentiment analyses only refer to the frequencies for the survey and interview responses, but not the expected frequency calculation. The expected frequency calculation is provided solely for comparison purposes since the sentiment analysis applies only to participants in the program and does not reflect the opinions of project implementers. Context for keywords were determined as follows: 1) “research” refers to keywords used in the context of the conduct and results of research; 2) “interact” refers to keywords used in the context of collaborations and partnerships and also includes keywords used with respect to interactions between institutions; and 3) “manage” refers to keywords used in the context of management of the FABRI program including management of grants programs. Correlation and Triangulation The evaluators triangulated and correlated key phrases from the desk review, interviews, and online survey data. This refers to validation of data through cross verification from two or more sources. In 1 For more information see: https://usaidlearninglab.org/sites/default/files/resource/files/ocawithfacilitatorsguidewithnupas7.10.2015.pdf 18 particular, it involved the application and combination of several research methods regarding the same theme. Outcome Mapping Data from the online survey was analyzed to improve the validity and rigor of the qualitative data analysis and construct, from both interview and survey data, a plausible hierarchy of short-term, medium-term and long-term changes along the knowledge, values, and action hierarchy of changes continuum. Evaluators outlined three spheres of results:  Sphere of Control: activities that are within the program implementer’s manageable interests;  Sphere of Influence: changes in behavior catalyzed among stakeholders; and  Sphere of Interest: higher-level changes that were catalyzed by FABRI. LIMITATIONS The online survey was an attempt to obtain more quantitative evidence but despite many different attempts to encourage responses, the sample represented by the number of respondents to the survey (n=49) was not statistically significant to allow for analyses beyond descriptive statistics and cross tabulations. Dependence upon qualitative data can affect the validity of findings in several ways. Bias in interpretation is a concern in collecting both quantitative and qualitative data but qualitative data collection introduces an element of bias in data reporting, analysis, and presentation that requires scrupulous attention to the data collection instruments. To address these concerns, the evaluation team made an effort to: 1) collect primary statements (i.e., direct quotes); 2) avoid leading questions; 3) elicit voluntary information in addition to answers to the questions on the semi-structured questionnaire form through probes; and 4) ask questions in the online survey designed to gauge the respondents’ level of knowledge of, interest in, and views of the program. The different analyses of data outlined above are also designed to provide a relevant and comprehensive mining of the qualitative data to obtain findings that reflect a complete picture of the program’s benefits and constraints to achieving its objectives. Because of the limited amount of text data the team did not employ CAQDAS software but did examine primary texts collected from interview and survey open-ended question responses to determine: 1) word repetitions (frequencies); 2) indigenous categories for terms/words; 3) keyword contexts; 4) how frequent words compared with statements of achievements and objectives from program reports; and 5) the existence of significant gaps (no terms/words and keyword synonyms referencing an important objective. 19 FINDINGS AND CONCLUSIONS This section first discusses overall findings and conclusions followed by findings and conclusions for each evaluation question and sub-question. OVERALL PROGRAM FINDINGS Contributions to overarching development objectives. The elements of the FABRI program’s capacity building of the MENA-NWC were in line with the national water sector development plans of the beneficiary countries. FABRI contributed to the integrated water resources management (IWRM) development needs including providing information, reports and issue papers through the member centers to support for decision making and for the development of the national water sectors. This was stated by the relevant ministries in Jordan and Oman, the water utility of Morocco (ONEE), regional desalination research center of Middle East Desalination Research Center (MEDRC), and USAID/Morocco suggesting that data, research reports and papers, and information supported the Millennium Development Goals (MDGs). FABRI was, most of all, an opportunity for those working in the MENA water sector to meet, share information with regional partners, and learn how to apply for research grants. The evaluation team created a word frequency table based on the themes, noted the context of each word or phrase (whether the discussion focused on research, human interactions or program management issues, and determined whether the use of word or phrase was framed in a positive, negative, or neutral manner. Respondents to the online survey and key informants were most positive about partnerships, collaborations, and pilot projects, and the most negative about issues related to funding, grants management, and government and private sector engagement as shown in Figure 3 below. (See Annex 7 for the full keyword analysis table.) Figure 3: Keyword analysis on program value, benefits and contributions Source: Online survey and KIIs Most informants and respondents see the future of the MENA-NWC in fostering research collaboration, information sharing, and supporting new projects. There is strong interest from the individual countries assessed to continue the collaboration that was established under FABRI, and at 0 20 40 60 80 100 120 Positive Neutral Negative 20 the MENA-NWC secretariat level, taking over the ownership of the delivered products and to develop them further. The engagement of government and private sector should have been more substantial. Respondents to the online survey and key informants both suggested that the MENA-NWC would have benefited from greater engagement of the governments and private sectors of the targeted countries. One key informant stated: “Private sector in MENA region is difficult to engage if the respective governments are not fully involved and supportive. There is no tradition of private sector R&D funding in the MENA region this needs to grow over time and with much more long term engagement from the governmental donor.” Other overall findings include:  FABRI was successful in giving researchers in the MENA region an opportunity for conducting research and the opportunity to work together collaboratively on a regional basis.  Respondents and informants are virtually all satisfied with the individual results they achieved from participation in the program according to interviews but believed that they did not have enough time to apply their results. Overall Findings on Gender Issues FABRI built capacity of women scientists and gave young women researchers opportunities to conduct research. The majority of FABRI grantees interviewed were women, with a high representation of women scientists especially in Morocco and Jordan. However, overall most key informants and survey respondents were men (see Figure 4). The Young Water Scientist Partnerships Small Grants Program was especially beneficial for young female scientists and provided the funding and opportunities for research participation. Figure 4: Gender of Key Informants and Survey Respondents Challenges remain for women scientists in the region. While women in Morocco and Jordan did not experience any restrictions, there were fewer women scientists in Oman not due to legal or policy restrictions, but due to cultural factors according to interviewees. Women are represented across technical and leadership positions in the water sector but the percentage of women in leadership positions is very low. Enforcing authority on research teams, gaining credibility, and obtaining cooperation from organizations to acquire tools are challenges for some women scientists in the region. Women community members are particularly supportive and open to water resource management projects. Dwindling water resources in the countries of the MENA-NWC region negatively impact women in rural areas. Women community members generally believe that water resource management could have a positive impact on their lives. Scientists who worked with these communities suggested that women were especially open to promoting water conservation to their children, household families, and adopting methods such as rainwater harvesting. 0 5 10 15 20 25 30 Key Informant Interview Survey Respondents Female Male 21 OVERALL PROGRAM CONCLUSIONS There is a strong interest from the individual countries assessed to continue the collaboration and research established under FABRI. Key informants and survey respondents would like to see the MENA￾NWC management (board members, director, and co-director) take ownership these activities and to develop them further. There is a question as to whether the positive results of the FABRI program and the MENA-NWC can be sustained, considering the latter’s current capacity level (low to basic) and its funding issues. In addition, because of the current instability in parts of the region there is a shortage of funds for these types of projects in general. EVALUATION QUESTION 1: SUSTAINABILITY AND POTENTIAL To what extent has the Network of Water Centers (NWC) developed into a sustainable entity capable of addressing critical water issues in the MENA region? What more needs to be done to ensure the future viability of the Network? The evaluation team addressed this question and sub-questions primarily by analyzing interview statements, survey data, and project documents. Analyses used include: hierarchy of changes, sustainability of changes and most significant changes, USAID OCA capacity building methodology for sub-questions a, b, d, and f, descriptive statistical analyses of survey data, social return on investment for sub-questions b and c, and content analysis. Overall Findings Respondents were interested in maintaining the program. Examining the hierarchy of changes and most significant changes (see Table 1 below), evaluators determined that there was substantial interest in continuing the MENA-NWC. Table 1: Hierarchy of Changes Analysis Knowledge—Broadening horizons Values—Expanding and implementing Action—Affecting policy and decision￾Makers Most Significant Change Researchers increased their knowledge about other water centers. Researchers used their increased knowledge to engage in collaborative research and partnerships. Collaborative research and partnerships produced valuable research which is gradually coming to the attention of policy￾makers. Researchers indicated that before FABRI they were unaware of the research being performed elsewhere in the region. Participants in the program believed that there are a number of incentives to continuing involvement with the MENA￾NWC. Respondents favor expanding the MENA￾NWC to a continuing grant program. Respondents offered suggestions for the further reengagement of policymakers. Respondents indicated that the incentive of a viable grant program made the network most appealing to them. 22 Knowledge—Broadening horizons Values—Expanding and implementing Action—Affecting policy and decision￾Makers Most Significant Change Participants in the program broadened their knowledge about watershed-based efforts to protect freshwater supplies, regulate groundwater development, and improve the productivity of water used in agriculture. Researchers conducted a number of pilot projects in these areas with viable results that could be scaled up. Without additional funds the replication or scaling up of these projects cannot take place. Respondents indicated that the most significant change brought about by the program in these areas was their ability to discuss the resolution of important issues with other researchers. Introducing new concepts to frame debates, putting ideas on the agenda, or stimulating debate. Developing new talent and institutional arrangements for research and analysis. Building networks that support delivery of change. Participants reported that the program changed their research institutions by influencing strategies. Participants reported that the program did not affect procedural change at domestic or regional level. Most significant change resulting from the program in this area was training on successful innovations developed in other countries. MENA-NWC has an overall Low to Basic Capacity score on the OCA. The evaluation used the USAID OCAT to assess MENA-NWC’s organizational capacity. The average OCA score is 1.7 out of 4 indicating low to basic capacity (see Annex 8 for scores and comments for all seven areas). Sustainability. The keyword analysis of terms related to sustainability and potential evaluators determined that the majority of interviewees and survey respondents focused their comments on the context of research with respect to collaboration, application, scaling up of research, and partnerships. Figure 5: Keywords Contexts and Sentiments There were more positive than negative or neutral sentiments associated with partnerships and scale (in the context of scaling up). The terms “sustain” and “continue” were expressed more often with negative 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Positive Neutral Negative 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Research Interaction Management 23 rather than with positive or neutral sentiments indicating some doubt among participants about the prospect for the MENA-NWC as an ongoing entity. Nevertheless, as illustrated by the chart in Figure 6 below, survey respondents showed the strongest agreement with respect to the need to strengthen and maintain the MENA-NWC. More respondents agreed and strongly agreed that the MENA-NWC has drawn attention to vital water issues in the region than those who disagreed or had no opinion. The fewest number of respondents agreed or strongly agreed with the proposition that MENA-NWC is well managed. Figure 6: Sustainability, Management, and Effectiveness Overall Conclusion A number of new projects have emerged from the program. The majority of the researchers whose work was funded under the program are still in contact with their partners. At the time of writing, the MENA-NWC’s presence as a regional entity is just over two years old and it needs more members, more engagement of existing members, and additional resources to operate efficiently in the future. In general, stakeholders in the MENA-NWC are unclear about its potential as a sustainable entity but believe that it should be sustained. The number of survey respondents who have been in contact with a MENA-NWC research center member in the last two years indicates that MENA-NWC helped water researchers to establish lasting and useful contacts outside of the research projects. The program also had lasting effects on providing opportunities for women scientists. Nevertheless, future sustainability will depend upon the organization outlining clear plans and objectives, and establishing an ongoing capability of raising additional funds. The MENA-NWC was not established as a regional presence early enough in the FABRI program to take its proper place in the region. Financial sustainability is the biggest concern to the overall sustainability of the MENA-NWC. Convening authority is strong among its member base but there is room for improvement. Organizational structure and the management of the MENA-NWC need to be revisited to ensure that the network is well organized and fully functional. 1a) Organizational structure The MENA-NWC is an existing membership organization and currently has 25 member centers. The MENA-NWC was originally registered in Washington, DC in 2012. During the final 18 months of the FABRI program, the MENA-NWC was re-registered as a non-profit in Jordan. MENA￾NWC staff at the time of the evaluation consist of one professional, the Director (who has a private sector background in manufacturing of water sensors), and seven board members. A new board, to be 0 5 10 15 20 25 30 MENA-NWC Should be strengthened and Maintained MENA-NWC has drawn attention to vital water issues in the region MENA-NWC is well Managed Number of Respondents Strongly Agree Agree Neutral Disagree 24 0 2 4 6 8 10 12 14 16 18 Organizational structure Funding Financial management Project implementers Time Issues Political Issues Communication Issues Number of Respondents elected from the membership, will convene in the summer of 2017. A current management framework (including structure, policies, and roles) exists for NWC. The MENA-NWC’s activities consist of board meetings, collecting membership fees from water centers, and maintaining its website. With the assistance of the existing board members, the Director contacts member centers for the payment of fees and encourages them to submit information on their current research to the website. The MENA-NWC is planning to hold its board meeting in a few months to elect a new board and review its organizational structure, revise its bylaws, and prepare a long-term business plan. The Founder’s Committee and Board Members of the MENA-NWC were selected by FABRI. During FABRI, the Secretariat was required to check with FABRI and/or the FABRI selected board of directors, and could not make decisions on its own. FABRI did not work on enhancing the MENA-NWC’s delegation of authority to function independently from FABRI. According to KIIs with program implementers (n=6), Qatar, Bahrain, Saudi Arabia, and the UAE dropped out of the network as a result of the decision to include Israel which was made by the U.S. Department of State. Figure 7: Constraints to Achieving Objectives Source: Online survey The constraints to the MENA-NWC achieving its objectives include: 1) funding, 2) project implementers, and 3) organizational structure. Eleven online survey respondents (25 percent) identified organizational structure as a major constraint. In addition to funding, political factors were cited by 50 percent of the participants in the program as a potential constraint to continuing the network. MENA-NWC received a score of 2 (Basic Capacity) for Governance and Legal Structure on the OCA as show below. Table 2: OCA Section 1 - Governance and Legal Structure Sub-Section Score Comments Section 1: Governance and Legal Structure 1.1 Vision and Mission 2 Vision and mission statements (strategic plan documents) but are only partly relevant to organization’s current purpose or aspirations 25 Sub-Section Score Comments 1.2 Legal requirements and status 3 The organization has legally registered, has by-laws that are incomplete or out of date 1.3 Organizational structure 1.3 The organizational structure is documented but out of date 1.4 Board composition and responsibility 2 An external board has been appointed but members are drawn from a narrow group and it is not independent of management 1.5 Succession planning 1.8 The organization has high dependence on its current leader Average section 1 score 2 Basic Capacity Conclusion The current organizational structure is not sufficient for the network to undertake the following activities which respondents stated were necessary to achieving sustainability: 1) sustain its relationship with the individual centers; 2) collect and raise funds; 3) create incentives for membership; and 4) initiate new programs. MENA-NWC staff and current board members recognize that additional staff is needed to enable the network to continue, and that member centers must be the main decision-makers for the organization in the future. Since responsibility was not delegated to MENA-NWC as a regional entity, MENA-NWC was not able to independently obtain adequate funding even though other donors had expressed an interest. The program did not work on enhancing the MENA-NWC’s authority and ability to function independently, either through a gradual process or otherwise. 1b) Capacity to collect and manage member and donor funds Social Return on Investment (SROI). The most tangible social results of the program that can be monetized are the advancement of water researchers in the region, the stimulation of new projects and donor funds stemming from association with the MENA-NWC, and the additional donor funds obtained by the MENA-NWC. Because most of FABRI funds were expended for conferences, travel, workshops, and other activities, which may have value but cannot be monetized for the purposes of SROI, only the ongoing projected salary increases, funds from additional grants (based on interviews), and funds directly attributable to MENA-NWC fundraising were taken into consideration as estimated benefits. This calculation took into account: 1) the costs for the MENA-NWC operation and for grants reported by the program implementer; 2) average increases for engineering faculty salaries calculated based on a survey of MENA countries performed in 2016; 3) the number of grantees (n=18) who reported on the survey that FABRI helped them advance and that they would not have had these opportunities without the program (this was 40 percent of the total grantees); and 4) the number and amounts of additional project funds estimated from interviews and survey data. (It should be noted that the team did not conduct a financial audit.) Table 3: Social Return on Investment Program Element Costs Current Monetized Social Benefit SROI (3 Year Period) Grants Programs US$ 1,125,000 (grant money attributable to 18 grantees) US$ 297,000 (value of career advancement and additional grants over a three-year period) 0.7 MENA-NWC US$ 747,501 (costs reported for 2016) US$ 508,750 (membership fees at current rate over three years plus US$ 490,000 in donor funds) 0.7 26 Many MENA-NWC member centers do not pay membership fees. The annual member fee was initially set at US$ 4,000. MENA-NWC had difficulty obtaining funds from member centers since many perceived the fee as too high and simply dropped out. A number of centers also dropped out due to regional wars and conflicts. As a result, MENA-NWC’s secretariat has been unable to stand on its own and be self-sufficient through membership fees. The membership fees were very recently reduced from US$ 4,000 per member to US$ 250 to encourage more membership centers to contribute fees, but at the time of writing of this report it is too early to determine the impact of the new fee structure. Several interviewees raised the concern that too much money was expended upon travel and other expenses, and not enough was spent on research. Interviewees in Jordan described a conference in Oman as “too lavish” and suggested the money spent on it would be better spent on research. However, a member center in Oman contributed US$ 80,000 to the conference and was able to secure reasonable rates at the hotel which was owned by the Sultanate of Oman. The perception that MENA￾NWC is not appropriately spending membership fees likely impacts members’ willingness to pay fees. At present, MENA-NWC has a Low to Basic Capacity OCAT score on the financial management section of the assessment. As shown in Table 4 below, MENA-NWC scored 1.5 out of 5 on the OCA conducted by the evaluation team. Table 4: OCA Section 2 - Financial Management and Internal Control Systems Sub-Section Score Comments Section 2: Financial Management and Internal Control Systems 2.1 Budgeting 1.3 The organization has an outdated budget based on the projections of funds made at the end of the project that did not materialize. 2.2 Accounting system 2 An accounting system was used in the past (up to 2016) but currently MENA-NWC is using a manual system because there is only one staff member. Future accounting will be determined by the donors (SIDA and the GOJ). 2.3 Internal controls 1.3 There has been and continues to be a weak to inadequate separation of duties among procurement staff and staff handling revenues. 2.4 Bank account management 1.3 Bank accounts are managed in accordance with donor requirements. 2.5 Financial documentation 1.2 Written financial documentation policies and procedures are weak and require significant changes. 2.6 Financial statements and reporting 1.8 Financial statements are included in end of project documents but at present, internal financial statements are not regularly produced. 2.8 Cost sharing 1.5 The organization has not had any donor cost-sharing requirements. Average section 2 score 1.5 Low to Basic Capacity Many interviewees hoped that the MENA-NWC was sustainable but pointed out, in order to be so, it would need to have a more defined purpose, and the capability to raise funds and get members to pay appropriate fees. One key informant stated: “What’s the network for at the end of the day?” 27 Another key informant noted that MENA-NWC is one of many networks in the region, highlighting the need for MENA-NWC to have a clear purpose and ability to add-value for members: “Is it a network for the sake of a network—we’re tripping over networks in this region.” MENA-NWC requires capacity building in area of fundraising. Following the Arab Spring, a shortage of funds in certain countries such as Jordan, and the FABRI program phasing out by the end of 2016, resulted in national-level organizations leaderships’ inability to continue with the elevation and extension of the research and human resources capacity building that comes with the regional linkages established under the FABRI program. To achieve financial sustainability, MENA-NWC must outline other non-donor income generation activities such as philanthropy, and engagement of government counterparts and the private sector. The MENA-NWC secretariat has been struggling to survive without the support of a donor agency until recently when the Swedish International Development Cooperation Agency (SIDA) offered to grant the network (US$ 450,000) to cover staff salaries and rent of the network for the coming two years effective February 2017. The Government of Jordan confirmed that it will also provide a grant but solely for the maintenance of the organization. These funds will allow MENA-NWC to function only at a very basic level over the short-term making the organization unsustainable over the long-term. Both MENA￾NWC members and other national water centers/organizations interviewed are aware of this fact and therefore suggested concentrating more intensively on the issue of sustainability in the event there is a forthcoming phase. Conclusion The capacity to collect and manage donor funds is partly dependent upon the donors’ perception of the perceived return on their investment. The SROI analysis indicated that the grants programs and MENA￾NWC itself had a moderately good return in terms of beneficial effects on certain participants (grantees) and in terms of the network if it is able to at least retain current funding levels including membership fees and donor contributions. Current levels of funding and membership fees are not enough, however, to enable the MENA-NWC to engage in new initiatives (such as additional grant programs). This may reduce its ability to convince current members and potential new members to participate and attract new funds from donors. 1c) Financial sustainability Funding is the most important constraint to the MENA-NWC. After June of 2016 there were insufficient revenues to pay the Executive Director and an additional staff member. Most survey respondents cited funding as the major constraint for the program achieving its objective. As shown in Table 4 above, the OCA assessment resulted in a score of 1.5 for the financial section in part because the MENA-NWC has not had sufficient financial resources to set up a viable accounting system. Interviewees and survey respondents not surprisingly mentioned the terms “grant,” “fund,” and “budget” mostly with respect to management. Interviewees and survey respondents who mentioned terms relating to financial matters were most negative about “budget” and most positive about “grant.” The discourse in which the term “grant” most often occurred was with respect to personal finance. The discourse in which the term “budget” most often occurred was with respect to program finances. 28 Figure 8: Keyword analysis on funding/grant/budget comments Based on a three-year estimate of monetized program benefits, the SROI for the grants programs and MENA-NWC only (excluding the major portion of the program funds) is 0.7 which is moderately good (see SROI analysis above). Conclusion MENA-NWC has the necessary funds to maintain operations at the current level, and perhaps to hire additional staff. In order to accomplish anything beyond maintenance of current activities they will need additional funds. Financial matters in the MENA-NWC were not made clear to any of the stakeholders and as a result many interviewees felt that too much money was expended on matters that were not essential to the program. 1d) Development and implementation of a long-term business plan The business plan is weak and requires significant revisions. The business plan was prepared with the active engagement of the MENA-NWC Founders Committee selected by FABRI. The business plan’s strategic objectives are similar to those for the overall program with respect to collaboration, partnerships, and increasing knowledge exchange. MENA-NWC’s organizational structure, according to the business plan, “must parallel its increasingly independent financial status, as it transitions from a program-driven creation to a financially sustainable enterprise.” The Council is invested by the plan with the authority to oversee and implement early fundraising campaign activities while acting on behalf of MENA-NWC’s board of directors. The business plan does not reflect MENA-NWC’s current vision, mission, and values and is not based on an adequate analysis of strengths, weaknesses, opportunities, threats, and realistic resource requirements and availability. Other weaknesses include:  The business plan does not invest the board with any authority beyond technical assistance.  The revenues predicted in the business plan budget were not realized in 2016 or thereafter. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% grant fund budget Research Interaction Management 0% 20% 40% 60% 80% 100% grant fund budget Positive Neutral Negative Source: Keyword analysis data 29 MENA-NWC has a Low to Basic Capacity OCAT score on the organizational management and sustainability section of the assessment. Table 5: OCA Section 7 - Organizational Management and Sustainability Sub-Section Score Comments Section 7: Organizational Management and Sustainability 7.1 Strategic (business) planning 2 The strategic plan (business plan) is weak and requires significant changes. It does not reflect MENA-NWC’s current vision, mission, and values and is not based on an adequate analysis of strengths, weaknesses, opportunities, threats, and realistic resource requirements and availability. 7.2 Annual workplans 1.5 Annual work plans have not been prepared. 7.3 Change management 1.3 Weak processes or structures for responding to changes in leadership, staffing, budgets, government policies, and donor funding levels and priorities. 7.4 Knowledge management and external linkages 2 Weak systems for documenting, storing, and disseminating program knowledge, occasionally analyzed and shared good practices and lessons learned internally, but did not actually apply them, established some formal networks, participated in discussions with donors, governments, and civil society organizations on approaches, lessons learned, and good practices and presented its approaches and results at external events. 7.5 Fundraising and new business development 2 The organization has had occasional cash flow problems, but positive net income last year, insignificant funding from cost recovery, sales, or membership fees, and limited absorptive capacity for additional projects. 7.6 Internal communications and decision making 1 Structured settings do not exist to exchange ideas and discuss problems or opportunities. In the past, management has not listened to staff and has inadequate space and infrastructure to facilitate internal communications. 7.7 External communications 2.3 Limited capacity for implementing an external communications strategy and overseeing written and oral products, weak website, a neutral reputation with key stakeholders but adequate capacity for overseeing written and oral products, and has a positive reputation with key stakeholders. 7.8 Advocacy and influence 1.3 Has not carried out significant advocacy activities over the past three years, lacks staffing and skills for effective advocacy, has not influenced the formulation or implementation of government policies at the national or local level, and has had little influence on the general public’s views. Average section 7 score 1.7 Low to Basic Capacity The business plan is not perceived as a collaborative effort. Interviewees who were aware of the business plan did not see it as a collaborative effort in terms of engaging regional stakeholders in the development of the plan, and spoke of the business plan in negative terms. 30 Figure 9: Keyword analysis on business plan Source: Keyword analysis data Conclusion While MENA-NWC has a business plan, a number of interviewees pointed out that it was not drafted with the participation of member centers or with the participation of people in the region. Those who were aware of the business plan had mostly negative sentiments about it. The business plan is not realistic with respect to staffing, revenues for MENA-NWC, or the duties of the board. It is expected that a new board will be elected by the membership and will develop a new business plan that more closely fits the objectives of the MENA-NWC moving forward. 1e) Recognized convening authority among the NWC’s member base The convening power of the MENA-NWC network is strong. Most the stakeholders believe that the MENA-NWC is effective in convening regional meetings. About one third of interviewees mentioned regional workshops and meetings favorably, and expressed the desire for more of them. When asked directly in the online survey if they believed that the MENA-NWC had been successful in convening regional meetings, 46 percent of the respondents indicated that the MENA-NWC was effective and 27 percent indicated that it was very effective. Twenty-three percent of respondents, however, indicated that the MENA-NWC was somewhat effective in convening regional meetings. Figure 10: Survey responses on MENA-NWC convening authority Source: Online survey 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% business plan board Research Interaction Management 0 2 4 6 8 10 12 14 16 18 Convening regional meetings Disseminating valuable research to decision makers Getting influential policy-makers to participate Number of Respondents Very effective Effective Somewhat effective Not at all effective 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% business plan board Positive Neutral Negative 31 Eighty percent of the respondents to the online survey agree that they would have not met or worked with certain institutions if it had not been for the MENA-NWC. The MENA-NWC brought stakeholders together that they would otherwise not have met. Conclusion Overall, the survey results suggest that the MENA-NWC has been generally effective and influential in convening regional meetings. However, the MENA-NWC has not engaged in any activities to convene or engage with members except with respect to the website and membership matters since the end of FABRI. The extent to which the MENA-NWC can engage in new training, workshops, and conferences is dependent upon obtaining additional funding for such activities. Survey responses and interviews strongly suggest that the MENA-NWC has a good reputation among water professionals in the region. With the election of the new board and the injection of additional funds, the MENA-NWC could establish itself more firmly as a multi-lateral influential research entity in the region. Survey respondents and interviewees were most favorable about the potential for the MENA-NWC to sponsor fruitful regional meetings and workshops, and disseminate valuable research to decision makers but more doubtful about the potential to get influential policy makers to participate. 1f) Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC As noted above, the OCA resulted in the MENA-NWC being assessed overall as 1.7 (Low to Basic Capacity). The MENA-NWC scored the highest (basic capacity) in Program Management (see Table 6). However, the weakest (low capacity) for the MENA-NWC is in Human Resource Systems as shown in Table 7 below. Table 6: OCA Section 5 - Program Management Sub-Section Score Comments Section 5: Program Management 5.1 Donor compliance requirements 3 MENA-NWC has received a direct award from other donors with complex compliance requirements. MENA-NWC has adequate systems and procedures to meet donor requirements, but the systems may need some updating. 5.2 Sub-award management 1.5 The organization has not provided formal sub-awards. 5.3 Technical reporting 2 Technical reports on projects have been prepared with significant assistance from a donor or technical assistance providers. 5.4 Stakeholder involvement 1 The organization has written policies and procedures for stakeholder involvement. It does not have regular meetings or communication with clients, and has inadequate physical space to meet with individual clients and groups. 5.5 Culture and gender issues 2.5 The organization has adequate tools and expertise for systematically assessing culture and gender issues. Average section 5 score 2 Basic capacity 32 Table 7: OCA Section 4 - Human Resource Systems Sub-Section Score Comments Section 4: Human Resource Systems 4.1 Adequacy of staffing and job descriptions 1 Policies and procedures on staffing and job descriptions are: are inadequate and require substantial changes, and are not followed. Many key management, technical, or finance positions have not been established, remain vacant, or are filled by people without appropriate skills. Some vacancies have significantly reduced efficiency or effectiveness for more than 6 months. 4.2 Recruitment and retention 1 Policies and procedures for staff recruitment and retention are inadequate or not followed. 4.3 Personnel policies 1 Personnel policies and procedures are inadequate and require substantial changes. 4.4 Staff time management and payrolls 1 Payroll policies and procedures were not updated after the end of FABRI. 4.5 Staff and consultant history 1 Staff and consultant resumes are not kept up to date for a roster for new proposals. 4.6 Staff salaries and benefits 1 Policies and procedures on staff salaries and benefits are not adequate and do not comply with labor laws. 4.7 Staff and contractor supervision and work planning 1 The organization has not defined and documented supervisory assignments. Average section 4 score 1 Low Capacity Survey respondents were not aware that the MENA-NWC was struggling financially as most believed that the organization met or exceeded their expectations with respect to funding, information provided, and management. Figure 11: Survey Responses on MENA-NWC Management Expectations Source: Online survey 0 5 10 15 20 25 30 35 Funding Information provided Management of the Program Number of Respondents Exceeded Expectations Met Expectations Below Expectations 33 Conclusion The MENA-NWC’s secretariat has been struggling to survive without the consistent support of a donor agency. Both the MENA-NWC and other national water center representatives interviewed were aware of this fact and suggested that the organization concentrate more intensively on the issue of sustainability in the event there would be a forthcoming phase. Continued support to the MENA-NWC for regional research linkages, and human resource capacity building in areas of fundraising was important to interviewees, and 26 out of 36 survey respondents agreed or strongly agreed that the MENA-NWC should be strengthened and maintained. EVALUATION QUESTION 2: COLLABORATIONS AND PARTNERSHIPS To what extent has the NWC been successful in strengthening partnerships, collaboration and information sharing between research institutions in the region, as well as between research institutions and the private sector? The evaluation team addressed Evaluation Question 2 and its sub-questions first by examining the research produced by regional experts in the water sector. In order to determine the MENA-NWC’s success in fostering research (sub-questions a. b., and d.), the evaluators used KIIs and survey response data to analyze the extent to which MENA-NWC has: 1) provided opportunities; 2) helped to inform researchers; 3) stimulated dialogue and affected decisions by policymakers; 4) helped to place water issues on policy agendas; 5) supported researchers in developing innovative solutions; 6) helped develop new research talent; and 7) enhanced collaboration. In addition to information taken from KIIs and the group interview, evaluators examined the networks that have been formed among researchers, member centers, and the public and private sectors to address sub-questions b and c. 2a) How successful have small research grants and research pilots been in fostering research collaboration? The research projects enhanced capacity, infrastructure, and culture by introducing research techniques that employ accessible modern instrumentation. They also facilitated access to site-bound instruments, promoted collaboration with regional actors, and linked institutions that have similar objectives. The capacity building measures of the program had considerable impact on the awareness of the importance of research and pilots to inform decision-making and the formulation of sound sector plans and policies. They also impacted stakeholders and entities using the water centers services. The hierarchy and most significant change analysis (see Table 1) indicates that the opportunity to perform collaborative research and engage in partnerships was the most important difference made by the program. Researchers indicated that before FABRI they were unaware of the research being performed elsewhere in the region. Researchers stated that FABRI ended before the results of these projects could be applied and the time allotted for the research (in most cases 18 months) was too short. No scientific committee was established to carry out the technical evaluation of the research work. A number of research pilots have fostered research collaboration in the region, especially in the fields of water use efficiency in agriculture, and rural communities’ wastewater treatment and reuse. The pilot research projects built the capacity of communities within universities and across institutions by engaging post-graduate students (MA and PHD students) in the research projects and advancing their research work in universities. Interviewees stated that they did not see the possibility of regional connection as possible before the network. Interviewees also noted that they not see the mutual benefits at the onset of FABRI but the MENA-NWC’s activities changed members’ minds. 34 However, while survey respondents were quite positive about the benefits of the program for strengthening partnerships, collaboration, and information sharing between research institutions in the region (see EQ 3a for details on the perceived benefits of the program), they were less so about the benefits of the program in strengthening collaboration between research institutions and the private sector and government. Also, the program had a weak element of full integration of the concept of experience exchange which was not an embedded element in the sustainability measures. The instrument would have contributed and shown a high potential to support the dissemination of best practices from exchanges that would have made the MENA-NWC a regional think tank. The small research grants program fostered cooperation within countries. Several interviewees noted that the small grants program fostered cooperation within countries as there had been little interaction within Morocco and Jordan between the research institutions and government entities. MENA-NWC held a workshop in Oman at the end of 2015 where the FABRI regional partners were invited to present their results. Some had the opportunity to meet for the first time with members of the network from the same country that they did not know. This was viewed by all as a valuable experience. Grant recipients and participants in research pilot projects were in touch regularly with others in the program. Analysis of networks based upon the survey responses (Figure 12) shows that most of the respondents to the question concerning contacts and frequency of contacts were regularly in touch with other participants in the program. Accessibility and expertise were the reasons most frequently given for contacts between participants. Figure 12: With what frequency have you voluntarily contacted the following individuals associated with MENA-NWC during the past two years? Source: Online survey 0 5 10 15 20 25 Researcher Advisors Grantees or main researchers Program implementers Representative of a MENA-NWC member center Government official MENA-NWC board member University research partner abroad MENA-NWC research center member Private Sector 6 or more times 1-5 times 0 times 35 Figure 13: Network of Contacts Source: Online survey Figure 13 demonstrates the frequency of contacts between participants who responded to the survey. Grantees were the most likely to contact others and their most frequent contacts were with other grantees. The smaller boxes represent less frequent contacts. Blue lines indicate a moderate level of communication and orange lines indicate the least amount of communication. Collaboration and information sharing was cited by most interviewees as the real successes of the program. The program implementation included opportunities for regional exchanges and annual study tours for research participants allowing information sharing. When asked about the effectiveness of the program, survey respondents found teamwork and collaboration, interfacing with colleagues, and workshops and meetings to be the most effective aspects of the program (see Figure 14). It should be noted that there was significant participation of women on these projects as shown by Figure 15 below. Figure 14: Survey responses on collaboration Source: Online survey Women scientists actively participated in the small grants programs. According to the survey, around 40 percent of the grant program participants were women scientists. For the Young Water Scientists Partnerships Small Grant Program, 50 percent of its grantees were women scientists. Given that the number of male survey respondents (26) was more than three times greater than the 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Workshops and meetings Interfacing with colleagues Team work and collaboration Engagement with the government Engagement with the private sector Not at all effective Somewhat effective Effective Very effective 36 number of women (8), participation of female sentiments in the small grant program is exceptionally high. Figure 15: Participation of women in grants programs Source: Online survey There is strong evidence that results of FABRI’s training on applying for research funding was successful. FABRI trained and actively engaged with research applicants on how to write a successful research proposal. The evidence is clear that participants advanced on research proposal writing. This improvement should eventually lead to advancement in their research careers. Many interviews stated that the training and their experience working with FABRI on research proposals were helpful in developing future applications of research and publishing of papers in international specialized research magazines and conferences. Conclusion The small research grants and research pilots have been generally quite successful in fostering research collaboration but there were some management and communication issues that constrained the program at its beginning. The collaborative activities established under the program are continuing as demonstrated by survey responses and interview statements. These programs were also the most successful in incorporating women into water research in the member countries. 2b) To what extent has information around the results of pilot research projects been shared among members of the NWC? Results of pilot research projects appear to have been sufficiently shared among members of the NWC (workshops and conferences were organized to present and share research results). Most pilot project researchers have reported in interviews and on the survey that the results of their research were being published. Considering the ratio of women to men, a greater percentage of women (75%) than men (62%) have been active in publishing the results of their work. 0% 20% 40% 60% 80% 100% Water Innovation Fellowship Grantee Policy Research and Development Grantee Young Water Scientists Partnerships Small Grants Program Other research fund through FABRI Female Male 37 Figure 16: Publication of Research Results Source: Online survey Conclusion The results of the pilot research projects have been shared among MENA-NWC members and several researchers report that they have published their results or the publications are in progress. The pilot research projects helped to build learning communities within universities and across institutions by engaging post-graduate students in the projects and advancing their research work in universities. 2c) How, and to what extent, has the research generated by the NWC been used by policymakers and other stakeholders to inform water resources planning and management? The results generated by the research pilots do not appear to have been sufficiently used by policymakers, government agencies, and the private sector to inform water resources planning and management. Interviewees stated that results generated by FABRI-sponsored research have not been used by policymakers, government agencies, or the private sector to inform water resources planning and management, with the exception of NCARE’s research (NCARE is a government entity). However, many survey respondents believed that the program was effective in engaging policymakers. Figure 17: Survey responses on NWC’s engagement of policymakers Source: Online survey Reserach Published, 58% Not Published, 3% Publication in Progress, 39% Yes No Publication in Progress 0 2 4 6 8 10 12 14 16 Very effective Effective Somewhat effective Not at all effective Number of Respondents 38 Interviewees and survey respondents indicated their belief that the program was not successful in engaging the private sector. Interviews with project implementers indicate that lack of engagement of the private sector was a decision by the donor, while participants in the project believe that it was decision by project implementers. Conclusion Government and private sector representatives had limited engagement with the program. There are conflicting accounts as to why this situation arose. Although the results and reports produced by FABRI are particularly valuable in terms of non-conventional water resources, the weak connection between research and industry was noted frequently by interviewees and by survey respondents. 2d) What types of support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? The majority of interviewees reported that grants programs was most effective at fostering research collaborations followed by regional workshops and meetings. Survey respondents presented with various alternatives were most interested in support for research projects and support for networking and collaboration as illustrated in Figure 18. Figure 18: Support and initiatives that are most likely to increase or sustain research collaboration and information sharing among member institutions Source: Online survey Conclusion Researchers and research institutions were most interested in the possibility of additional support for projects, networking, and collaboration. Most interviewees suggested that this was the most likely result that the MENA-NWC could achieve in the future. In order to do so, MENA-NWC will need additional funds and will also need to establish some systematic method of engaging partner centers, keeping the researchers linked to each other via scientific groups or thematic interest groups, soliciting, and ultimately managing and funding grants for longer time periods based on the MENA-NWC’s secretariat interacting with governments and the private sector to identify the problems they are facing, and fully bring governments of the region into the picture and engage the private sector. 28 39 17 26 20 30 0 5 10 15 20 25 30 35 40 45 Support for workshops and meetings Support for research projects Support for graduate education in the field Support for engaging policy￾makers Support for engaging the private sector Support for networking and collaboration Number of Respondents 39 EVALUATION QUESTION 3: INCENTIVES AND DISINCENTIVES What are the key incentives (and disincentives) among NWC member institutions with regard to greater engagement in the network? Evaluation Question 3 was examined by determining the subjective value that respondents place on the program as well as its perceived benefits. The evaluation team conducted sentiment analysis and discourse analysis of survey and KII qualitative data using keywords that relate to the value, benefits, and potential of the program. 3a) If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? Please consider value and benefits to date, as well as future expectations. Key incentives that foster greater engagement are: 1) the opportunity to share knowledge about applied research, 2) availability of research funds, and 3) support for the training of a new generation of scientists. Interviewees and survey respondents listed the opportunity to pursue and share knowledge about applied research as a major incentive to further engagement. The availability of additional research funds was also frequently mentioned during interviews in addition to training for new and existing water professionals. Figure 19: Keyword Analysis of Terms Relating to Incentives, Disincentives, and Value Source: Online survey Participants believe the MENA-NWC is a valuable regional entity and are supportive of a regional network of research institutions. All interviewees suggested that as a concept, a regional network of research institutions is positive. From the survey, an overwhelming number of respondents (78 percent) responded that they found the program to be valuable and 48 percent found it to be very valuable as shown in the chart below. 0 5 10 15 20 25 30 benefit value change exchange contribute capacity engage Number of Respondents Positive Neutral Negative 40 Figure 20: Value of the MENA-NWC Source: Online survey The following are responses referring to NWC’s value as a regional entity: “It makes a dream come true...It allowed me to be in touch with relevant international institutions and more.” “[NWC] has proven potential for bringing researchers from different MENA countries together which would be a good thing even if the research was marginal.” “The FABRI initiative is one of the most successful programs in water research I have witnessed in 20 years of research in the MENA area. The down side is that it needed to be a longer program so that it can establish a deeper impact to the water science community.” The primary benefit to participating in MENA-NWC is its ability to support collaboration and bring scientists together. Interviewees and survey respondents most often identified the benefits of MENA-NWC membership as the possibilities it could offer of reinforcing collaboration and providing funding for grants programs. KIIs also confirmed that the program was especially valuable in establishing lasting linkages. More than 20 percent of the survey respondents strongly agreed with the statement that the MENA-NWC has been the most successful in bringing scientists together and supporting collaboration. The majority of the respondents also feel strongly that the MENA-NWC assisted their research career (58 percent). Figure 21: Survey responses on MENA-NWC achievements Source: Online survey Not valuable at all, 5% Somewhat valuable, 10% Valuable, 30% Very valuable, 48% Other, 8% Not valuable at all Somewhat valuable Valuable Very valuable Other 0 5 10 15 20 25 MENA-NWC has been most successful in bringing scientists together and supporting… MENA-NWC has drawn attention to vital water issues in the region MENA-NWC has been successful in connecting researchers with policy-makers Number of Respondents Strongly agree Agree Neutral Disagree 41 The following are illustrative responses referring to the benefits of NWC: “Best part of the project was communication, technical collaboration, sharing a lot on both sides.” “Major difference made by the project was connecting with other people in the regions.” FABRI was successful in building the capacity of researchers and women scientists in particular. According to interviewees and survey respondents FABRI raised the capacity of professionals, provided opportunities to young researchers, and helped them advance their careers. Below are quotations from KIIs with women researchers who participated in the program: “FABRI had a positive impact on young engineers, enabled them to scale knowledge engage in research and development.” “We see a lot more women students in our classes—for a project you look for the best.” “I think I was treated very well at the same level as men.” All survey respondents believed that the program helped women scientists in some way. Over half of the survey respondents agreed that MENA-NWC encouraged women scientists to apply for research projects, that it supported the inclusion of women in research collaborations, and that it provided funding for women scientists. Women grantees in particular agreed that the MENA-NWC had helped them to advance and felt strongly about this result as shown in the following chart of grantees who responded affirmatively to this question. Figure 22: Responses of men and women on the statement that the MENA-NWC has helped them advance in their career Source: Online survey Conclusion Participants both in interviews and on the survey found the MENA-NWC to be very valuable and are supportive of the existence of a regional network of research institutions. The majority of the survey respondents believe that the MENA-NWC should be maintained (73 percent). The primary benefits to joining the MENA-NWC is its ability to bring scientists together and support collaboration. The convening authority is also recognized as an important benefit among the respondents. The majority of the respondents also feel strongly that the MENA-NWC assisted their research career. MENA-NWC’s was especially successful in building the capacity of researchers and women scientists in the region. 3b) Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? 0 2 4 6 8 10 Men Women Strongly Agree Agree 42 Individuals from regional universities and research institutes had the most frequent contact with other members. According to the survey respondents, 21 individuals have had greater than average contact with other participants/members in the past three years (n=39). Among these respondents, regional universities and institutes were the most likely to maintain frequent contact with other MENA-NWC participants (n=13/20). One interview stated the following: “The program shaped my career and future, opportunities and prospects, and provided a concrete base for my professional career path, and getting future funds for research. The project taught me how to communicate and engage with local and government partners.” Figure 23: Organizations of people who have made the most contacts with other members Source: Online survey Grantees and researchers actively communicated with other MENA-NWC members. Among the 21 individuals that had greater than average contact with other participants, grantees, and main researchers were the most likely to maintain frequent contact with other MENA-NWC participants (n=13/20). Below are quotations from KIIs regarding successful experiences by the grantees and researchers. “After project completion, we submitted to MENA-NWC and MENA and German funds post project completion. We published a couple of papers, one reviewed paper on the first project and received a request from the ‘Water for Food Magazine; to contribute on food security and were invited with full cost coverage by them to Seattle in 2013 as a sign of appreciation.” “The radar project in Oman and Morocco enabled us now to continue such work with the two US parties and Qatar foundation came on board.” “Now IAV has collaborations with a UAE contact at ECBA in Dubai working on Bio-saline agriculture and in Tunisia with the Institute of Agriculture Engineering and Forestry (INGREF), and the African Water Facility of the African Development Bank (ADB) who asked them to submit a proposal as a result of the FABRI networking.” “There is a planned USAID initiative to assist IAV/ONEE … where they will install equipment to the VTC center at IAS for improved water management, training materials using simulators… The project is in proposal state and if it goes ahead it can be an extension of the FABRI applied 2 1 1 6 7 2 1 Government Research Institute Regional Financial Institution Regional Project Implmenter Regional University Research Institute US University Water Utility 43 research. The award can enhance the sustainability component of FABRI and can bring in new regional partners.” Figure 24: Roles of people who have made the most contacts with other members Source: Online survey According to the survey, women participants maintained a high level of communication with other members of the NWC than men. Among the 21 individuals that had greater than average contact with other participants/members in the past three years, women and men were about equally likely to stay engaged (n=8 women, n=12 men, n=1 unidentified). When considering the gender balance of the survey respondents (n=12 women, n=26 men), around 66 percent of female respondents and 46 percent of male respondents had greater than average contact with other MENA-NWC participants in the past there years. Figure 25: Personal characteristics of people who have made the most contacts with other members Source: Online survey Varying level of engagement was observed for each participating MENA country. Among the survey respondents who had made 25 or more contacts in the last three years, the level of participation varied with the most responses from Oman (n=7), followed by Jordan and Tunisia (n=4 in each case). 1 13 1 4 1 1 0 2 4 6 8 10 12 14 American or European University partner Grantee or main researcher Implementing entity MENA-NWC board member Representative of a MENA-NWC member center Research Advisor Number of Respondents 2 6 22 1 6 1 0 0 2 4 6 8 10 12 14 30-39 40-49 50-59 Over 60 Number of Respondents Males Females 44 No clear theme emerged from the survey data that would inform common characteristics of entities that are most engaged and active in the MENA-NWC. Judging by survey responses and KIIs, the three countries with the most contacts had researchers working on continuing projects with partners. KIIs revealed that maintaining program contacts with Syria, Iraq, and Yemen were essentially impossible given the political turmoil. Figure 26: Countries of people who have made the most contacts with other members Source: Online survey Individuals who maintained a high level of communication with other MENA-NWC participants experienced high-value from MENA-NWC membership. All of the 21 individuals stated that the MENA-NWC was valuable or very valuable, and that their expectations were met or exceeded with respect to the MENA-NWC’s impact on watershed-based efforts, protecting freshwater supplies, and improvements in the productivity of water used in agriculture. These individuals also stated that their expectations with respect to the MENA-NWC were exceeded with respect to knowledge sharing, collaboration and networking, workshops and meetings, online networking, interfacing with colleagues, and supporting their work and helping to raise awareness of critical water issues. Those reporting fewer engagements were not significantly more critical of the program results. Conclusion Researchers in universities and academic research institutes are the most likely to remain engaged with the MENA-NWC both through maintaining continued contact and continuing with projects that started under FABRI. Grantees maintained the highest level of communication which suggests that the grants program was one of the most important activities that encouraged active engagement. While the MENA-NWC participants showed varying degree of engagement, no clear pattern emerged from their country of origin. According to the online survey, a higher percentage of female MENA-NWC participants (66 percent) engaged in communication with other participants than male participants. Overall, those who reported higher engagement perceived that the MENA-NWC generally exceeded expectations. Those reporting fewer engagements were not significantly more critical of the program results. 3c) Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? A number of concerns were raised on the sustainability of the MENA-NWC. While the participants generally recognized the value of MENA-NWC as a regional entity, interviewees suggested that a clear vision and realistic planning are important for assuring sustainability of the MENA-NWC. There were noticeable concerns about the management of MENA-NWC. Sustainability is also closely 4 7 4 3 111 0 1 2 3 4 5 6 7 8 Jordan Oman Tunisia USA Israel Kuwait Syria Country Number of Respondents 45 linked with the availability of future funding. Interviewees and respondents believe that the MENA-NWC should be more proactive in seeking funding for current research proposals and ensuring regional collaboration in projects. One key informant stated: “The program played an important role in setting up the network of centers and in bringing together research institutions in the MENA region. However, the sustainability of the Network, whose operation depends on donor funding, is not guaranteed.” A primary disincentive is the membership fees which have not been commensurate with benefits of membership. While the membership fees have been reduced, the formerly high membership fee was cited, especially by university representatives, as a major disincentive. MENA-NWC should take a more focused approach to its objectives and pursue new and more timely objectives. Clearer objectives were cited by interviewees as an important area for the MENA-NWC to consider in order to engage more members. Investigating the nexus between water scarcity and climate change was identified as an example. Other suggestions included: 1) choosing one area each year in order to solicit research proposals; 2) investing in training of new water scientists and water resource managers; and 3) investing in and encouraging on-the-ground and scaled-up applications. Another disincentive has been the inability of the MENA-NWC to engage people beyond the research sector. In order to be of benefit to its members, MENA-NWC will have to move beyond simply providing a platform for information sharing. MENA-NWC should engage with a variety of stakeholders to make membership more attractive. Keyword analysis of interview and open-ended survey responses indicate that respondents used the term “engage” more frequently in a negative context particularly in reference to the inclusion of other stakeholders (see Figure 19). Conclusion The networking and collaboration under FABRI was excellent. However, ensuring sustainability of the network and securing future funding is essential for MENA-NWC to continue benefitting its members. Going forward, MENA-NWC will need to define its objectives and the water issues to be addressed. A focused approach will allow the members to concentrate its efforts or maximize the benefit from the network. As mentioned above, further work is needed to engage other actors, including the private and public sectors to leverage the full benefits of the regional network. EVALUATION QUESTION 4: SUPPORT TO MANAGING WATER RESOURCES To what extent has FABRI effectively supported the ability of countries to plan and manage their water resources in the following areas: watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture? The evaluation team addressed question 4 by analyzing interview statements and the survey data for outcome mapping. A plausible hierarchy of short, medium, and long-term changes are analyzed along the knowledge, values, and action hierarchy of changes continuum. FABRI enhanced collaboration and networking between regional actors on MENA’s water resource management. As noted above, MENA-NWC provided a forum for scientists of the region who previously had no regional connection. The transfer of knowledge between the regional actors on 46 non-conventional water resources and water governance is valuable and a major success of the MENA￾NWC. FABRI did not substantially improve the host governments’ ability to manage the scare water resources of the region. The outcome mapping results (details in Table 8) show that FABRI did not achieve its expected ‘as-planned’ results in the sphere of interest (impact) in particular. It scores an average of 1.7 out of 4 on its success in achieving its long-term effects at the national and regional level. A number of factors contribute to this. First, FABRI failed to substantially engage government officials and thus the linkage between research and government priorities and/or interests was weak. Second, the FABRI sponsored collaborative research on water issues was not applied or scaled-up during the life of the program. While several research projects are continuing with funds from additional donors, there is still a large gap between its academic research its application and scaling-up. Third, capacity building through FABRI was mainly limited to individual grantees and therefore lacked the leverage to influence governments’ decision-making process. MENA-NWC members are uncertain of FABRI’s contribution to planning and managing water resources in the region. The survey data below indicate that the majority of survey respondents had no opinion on how the program supported the three water resource management areas - watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture. Among the three areas listed in EQ4, FABRI had the largest effect on the improvement in the productivity of water used in agriculture. Most of those who did respond with an opinion from the online survey stated that the program was most effective in encouraging improvements in water productivity for agriculture. For example, several of the research projects in Jordan and Morocco had the capability of increasing water productivity in rural communities, especially for women. A pilot project in Oman using treated wastewater for groundwater recharge providing significant protection of the aquifer from seawater intrusion. IAV in Morocco is continuing its collaborations with a UAE center and a Tunisian center (Institute of Agriculture Engineering and Forestry, INGREF) on further research in water productivity and bio-saline agriculture. Pilot projects implemented under FABRI using real-time monitoring of irrigation systems have improved irrigation efficiency and productivity in Jordan, improving its crop yields. Similar pilot projects have been conducted in Tunisia (INGREF) and in UAE (ICBA). GYGA (Global Yield Gap Atlas) implemented in Morocco and Jordan under FABRI in collaboration of the University of Nebraska made solid progress to improve crop yield in rain-fed and irrigated agriculture. 47 Figure 27: Effectiveness of the MENA-NWC in addressing critical water issues Source: Online survey FABRI’s influence on the countries’ watershed-based efforts to protect freshwater supplies was limited. FABRI provided tools (watershed model) to help improve water resources management at river basin level in Morocco, Tunisia, and Jordan. Training on the tool was organized but was not sufficient to induce change in the government’s ability to manage water resources. Research projects focusing on river basins in Jordan could potentially impact the use of freshwater resources for agriculture in those areas. FABRI had the least effect on the regulation of groundwater development. However, one notable success strong in groundwater regulation is in Jordan where the groundwater monitoring shows significant groundwater replenishment due to research stemming from the FABRI program. Table 8: Outcome Mapping Analysis Spheres As Planned As Evaluated Scores Impact (Sphere of Interest, long term effects— national and regional) 1) Contribution to the host Government’s ability to use objective criteria as opposed to a subjective criteria, personal influence or individual preferences in water resources management decisions. 2) Increased ability of the government to better legitimize its water resources management decisions. 3) Alignment of the FABRI established MENA-NWC with international norms for similar information and decision centers around the world, becoming a model for future replication in the region. 4) Increase in effective watershed￾based efforts to protect freshwater supplies, regulation of groundwater 1) FABRI did not substantially engage government officials according to the majority of interviewees (n=20) and survey responses (n=40). 2) Interviewees and survey respondents believed the program was less than fully effective in improving water resources management decision making. 3) The MENA-NWC is in a nascent stage demonstrating low capacity (see OCA scores) but all member center representatives surveyed and interviewed believe that it has the potential to be influential. 4) FABRI was rated by interviewees and survey respondents as somewhat effective in improving watershed-based efforts to protect freshwater supplies, regulation of groundwater 1) Not effective=1 2) Somewhat effective=2 3) Somewhat effective=2 4) Somewhat effective=2 Average Score=1.7 (Somewhat effective) 0 2 4 6 8 10 12 14 16 18 20 Watershed-based efforts to protect freshwater supplies Regulation of groundwater development Improvements in the productivity of water used in agriculture Number of Respondents Exceeded expectations Met Expectations Below Expectations 48 Spheres As Planned As Evaluated Scores development, and improvements in the productivity of water used in agriculture. development, and improvements in the productivity of water used in agriculture. Outcome (Sphere of Interest, Sustainability and continued use of services— participants and communities) Continuing research grants and training courses and workshops (different groups, different courses). Partners monitoring and evaluation (M&E) system continuing, effective and in use. Business and development plans continuing and in use. 1) A few (n=5) new research projects and proposals have been generated from FABRI sponsored research but there have been no funds for training courses and workshops although some member centers (i.e., MEDRC and ONEE) engage in continuing. 2) In Jordan and Morocco some projects continue to monitor results (n=4). 3) There is, as yet, no viable development plan. There is a business plan but it is outdated and requires revision. 1) Somewhat effective=2 2) Effective=3 3) Not effective=1 Average Score=2 Somewhat effective) Outcome (Sphere of Influence, participants and partners) FABRI partners have acquired means, tools and methods of strategic management (planning, M&E, human resources development and planning, gender studies and integration, etc.) and can better respond to the demands of their government agencies. The FABRI network of water centers has structures with which its own further education requirements can be identified and met in the future. 1) FABRI advanced the careers of young and women water professionals but did not substantially engage government agencies. 2) The organization has legally registered, and has by-laws and a business plan but they are incomplete and outdated. The business plan was not accepted or used by FABRI partners. An external board has been appointed but has exercised little authority and no new planning or programs have taken place. 1) Effective=3 2) Somewhat effective=2 Average Score=2.5 (somewhat effective to effective) Output (Sphere of Control, NWC and partners) 1) Institutional needs analysis and reports available in order to respond efficiently and with flexibility to new demands. 2) Structures and capacity development modules are in place and human resources capacity building requirements identified and developed. 3) A human resources development plan is in place that is reliable and reviewed on a continuous basis. 4) Opportunities to get hands-on experience (practical application) in real life and concrete situations with success beyond theoretical and academic knowledge. 1) There is an outdated needs analysis for the FABRI program but the MENA￾NWC's ability to engage in needs analyses and generate reports is limited. 2) FABRI had a plan for future MENA￾NWC activities but the organization has not had the funds to develop this capacity. 3) FABRI had a plan for MENA-NWC staffing but funds were not available to implement them. 4) FABRI has conducted useful training in the past but the MENA-NWC does not currently have the resources to provide training for partners. 1) Not effective=1 2) Somewhat effective=2 3) Effective=3 4) Somewhat effective=2 Average Score=2 (Somewhat effective) Conclusion The FABRI sponsored collaborative research on all of the three issues, but for most of the program stakeholders, it was not applied or scaled-up beyond case studies and/or brought to the attention of decision-makers. Also, capacity building was limited to support for individual grantees. The evidence shows that training efforts concentrated on all three areas, but most particularly in the area of 49 agricultural productivity. Training efforts have the potential to positively impact water resources management in the future. FABRI’s training in how to write research proposals was a valuable asset gained by inexperienced researchers. Thus, it is unlikely that the behavioral and professional changes, described in previous sections will disappear. Other training measures are already showing a high degree of sustainability. This is clear from statements of participants about their acquired knowledge and skills, the marked rise in the number of scientific researchers looking toward application of their research results, papers published in international specialized research magazines and presented at conferences, and the horizons that were expanded for them working with research partners in other countries. The program’s capacity building efforts are in line with the national water sector development plans of the beneficiary countries. Regional expansion of the MENA-NWC came late into the program which limited the authority of the network on national and international levels. Further, the constant monitoring of the results of the implementation of the research activities during program implementation was not linked to governmental development plans as governments were either uninformed about FABRI activities, as in the case of Jordan, or were informed post-completion as in the case of Oman and Morocco. FABRI opened doors for women and young professionals to conduct their research and publish. Overall, the program created favorable conditions for learning processes—leading at the individual level to tangible results (research skills, reports and papers, manuals tools) achieving some results (applications for funding procedures and tools) at the institutional level and paving the way for a future more sustainable regional linkages among the MENA water research centers. 50 RECOMMENDATIONS Short-term Recommendations for MENA-NWC MENA-NWC Organizational Structure  Adopt a clear vision and mission statement based upon member center defined needs.  Establish a proper system of accountability and performance measurement for the Secretariat to ensure effective use of authority.  Develop a new business plan that: 1) is developed with the participation of member centers, 2) sets realistic goals based upon current funding, and 3) contains a plan for obtaining funds from the other donors, the private sector and participating governments.  Develop a plan for building on and extending linkages, including with government and the private sector.  Update and maintain the NWC database with the participation of member centers. Support in Research Collaboration and Information Sharing  Encourage more engagement and participation from the member centers by reaching out centers that have been inactive for the past year.  Continue to provide capacity building for universities involved in research including technical assistance and advice on preparing proposals to obtain funding.  Consider sponsoring network-wide small grants competitions for young scientists and women scientists to the extent permissible with the current funds.  Provide valuation services to help research institutions to identify opportunities for funding and connections with the private sector.  Conduct a baseline survey of current members to identify the most important water resource management issues in the region for MENA-NWC to engage in. Long-term Recommendations for MENA-NWC Hold annual planning workshops to engage members and solicit their feedback. An annual planning workshop should take place with the management and board of directors of the MENA-NWC, and heads of the water centers to suggest, vet, discuss, and agreed on the next year's activities and budget. Develop a flexible management structure for the MENA-NWC that promotes communication and collaborative between all stakeholders based upon:  A full capacity assessment after one year conducted by an independent contractor.  A communication plan that includes recommendations for expanding the MENA-NWC’s presence on social media and maintaining/upgrading the website to make it more interactive and up to date.  A network analysis after one year mapping the communications between centers and participants. Develop a public outreach program with the participation of member centers that focuses on:  Raising community awareness of effective uses for wastewater.  Fully engaging with the government and private sector. 51  Raising community awareness on important challenges to effective water resources management including water losses, the effects of climate change, and the interactions between food security, energy, and agricultural water use. Explore other non-donor income generation activities to achieve financial sustainability. MENA-NWC should explore other options of financing the network other than donor funding and membership fees. Channels of funding should be broadened to philanthropy, engagement of government counterparts, and the private sector. Diverse options should be considered such as involving the private sector by funding specific small grants competitions focused on a research area in which they have an interest. Facilitate the development of local networks at a country level to create the needed basic platforms for cross country collaborations and work. Promote the inclusion of women in water research throughout the region by:  Performing a full assessment of the challenges and opportunities to achieving gender balance in activities in the specific member countries.  Based on the results of the assessment, develop a plan in collaboration with member centers to address challenges and take advantage of opportunities. The plan should include specific steps that can be taken. Recommendations for Future USAID Water Programs Capitalize on the network of regional water centers established during FABRI. MENA-NWC and FABRI initiated valuable regional connections of water researchers. Future USAID water sector programs should leverage the network and its database to engage in further collaboration at the regional level. Ensure that all relevant key stakeholders, including government agencies, private sector and policymakers, are actively involved during program design. Engagement with private and public sector stakeholders proved to be a challenge for the MENA-NWC. Engaging with the key stakeholders during program design and the early stages of project implementation will generate buy-in and increase the chances of successful engagement during implementation. Conduct a comprehensive assessment of past water projects implemented by other donor agencies in each respective country or region and build on lessons learned. Build the capacity of women scientists and technicians. FABRI’s impact on women scientists was one of the long-term successes of the program. It is vital that this aspect of the program continue to operate. USAID should enhance women scientists’ technical competencies in integrated water resources management by providing grants, scholarships, and/or training to women in engineering or sciences related to water conservation. Maximize the positive impact of FABRI’s pilot research projects on the lives of rural women. A number of new initiatives, such as hygiene enhancements through improved wastewater management and rainwater harvesting, were explored during FABRI. Future USAID projects should leverage on these research project initiatives and explore scaling up these activities to improve standards of living and access to water for women in rural areas. Recommendations for Future USAID Regional Network Programs Consider a longer implementation timeline for establishing regional networks. Establishing a fully function regional network requires extensive preparations and administrative procedures. For instance, the United Nations Economic and Social Commission of Western Asia (ESCWA) and GIZ took over three years to set up ACWUA as a regional entity (Spring 2006 – July 2009). Even in 2015, 52 USAID provided support to ensure its sustainability. Future USAID programs aiming to establish a regional network should consider a flexible implementation timeline to set up a successful regional entity. Develop a strategic business plan with a clear roadmap for the financial sustainability of a network. MENA-NWC focused on donors as a sole primary source of revenue in addition to membership fees and failed to proactively pursue capital. For future USAID projects, a business plan should be developed in collaboration with the board members and other relevant stakeholders. The business plan should identify tools and risk mitigation options to reach financial sustainability. The plan must also be periodically updated and reviewed to accommodate potential fluctuations in revenue and regional needs. Carefully design the organizational governance framework and lines of communication to ensure effective management and transparency. The relationship between the management (Secretariat, Board of Directors), USAID project implementer, and network/association members should be reviewed closely to establish a robust and well-functioning governance framework. FABRI’s Board of Directors expressed concerns that they were not adequately consulted nor involved in the strategic planning of the MENA-NWC. Future USAID programs should refrain from the project implementer taking on a major managing role of the network to ensure effective management of the network post-USAID funding. 53 ANNEXES ANNEX 1: STATEMENT OF WORK Statement of Work Final Performance Evaluation of “Further Advancing the Blue Revolution Initiative (FABRI)” I. Purpose and use of evaluation The purpose of this statement of work is to conduct a final performance evaluation of the “Further Advancing the Blue Revolution Initiative (FABRI)” activity, which is implemented by Development Alternatives, Inc., in partnership with local and international research organizations and university partners. Implemented in the Middle East, North Africa, and sub-Saharan Africa, FABRI aims to strengthen regional networks of water centers and build sustainable water management capacity throughout the Middle East and Africa. The focus of the final performance evaluation will be specifically on FABRI’s programming in the Middle East and North Africa (MENA) region (i.e., the evaluation will not include FABRI’s programming in sub-Saharan Africa). The purpose of this evaluation is to validate results achieved and assess FABRI’s progress in building an operational and sustainable network of MENA water centers. Lessons learned from the evaluation will be used to inform USAID’s future programming in the water sector. In addition to USAID staff, expected users of the evaluation include other donors working in the water sector and local stakeholders. II. Identification of Award Award Title: Further Advancing the Blue Revolution Initiative (FABRI) Implementing Partner: Development Alternatives, Inc. (DAI) Mechanism: Contract Activity Number: AID-EPP-00-04-00023 Award Duration: September 8, 2011 to March 31, 2016 (No Cost Extension until June 30, 2016) Award Budget: $20,000,000.00 USD (of which $15,000,000 for programming in the Middle East and North Africa) Implementation Locations: Middle East and North Africa (Jordan, Tunisia, Oman, Morocco, Bahrain, Qatar, West Bank); additional countries of implementation in sub￾Saharan Africa III. Background 54 Many countries in the Middle East and North Africa (MENA) face growing challenges related to water resource management and distribution. These challenges include drought, inefficient agricultural water use, groundwater management, inadequate or poor quality water supply, and weak or absent mechanisms for sharing transboundary water, among other issues. Such challenges can lead to conflict within communities and among countries, as well as have negative health, financial, and social impacts on populations. Governments in the region need support in developing the capacity to address these growing problems, including sound water sector management and access to needed science and technology to expand supply, manage demand, and increase efficient and productive use of water resources. To avert conflict and meet basic water needs, diverse stakeholders—including governments, water users, research institutions, donors, and the private sector— must work in partnership to define and implement bold, coordinated actions. Responding to these challenges, USAID’s bilateral investments in the MENA water sector are strategic and substantial, and support implementation of the joint USAID/STATE water strategy and the Water for the World Act of 2014. Complementing these robust bilateral efforts, USAID also supports regional approaches that bring together water resource managers and experts from neighboring countries to build partnerships and networks that promote trust, confidence, and understanding of one another’s problems, and provide opportunities for countries to work together to solve water-related problems. Embodying this approach, in his June 2009 speech in Cairo, President Obama announced his intention to intensify engagement with Muslim-majority countries, including the establishment of centers of scientific excellence in areas such as energy, green jobs, and clean water. To realize this presidential commitment, USAID led an interagency design team, working in collaboration with the Department of State and other Federal agencies, to establish a Middle East North Africa Network of Water Centers of Excellence (MENA NWC). The purpose of MENA NWC is to stem the increasing water scarcity in the Middle East to achieve greater water security. Member centers of the NWC are MENA research and educational institutions joined through recognition that they are “centers of excellence” in the water sector. While the focus is on participation of Arab institutions, an Israeli center is also included in the network. A center of excellence acts as a hub of innovation and integrates efforts from multiple disciplines. It also provides training opportunities that increase the pool of professional scientists and engineers with skills in water and land management. MENA NWC aims to address shared challenges in the water sector by using locally appropriate scientific, technological, and management solutions. MENA NWC links technical and research institutions across the region, encouraging them to work together and with outside counterpart institutions on critical water challenges. The NWC strives to foster partnerships that build and exchange regional science and technology capacity to improve water resources planning and management; and develop and disseminate policy tools and technical and management interventions that expand water supply, manage demand, and dramatically increase its efficient and productive use. In addition to developing and disseminating applied research and policy tools and promoting evidence-based technical and management interventions, the NWC facilitates collaborative activities, including competitive grants programs, capacity building, institutional exchanges, fellowships, and knowledge sharing. IV. Description of Activity USAID established the “Further Advancing a Blue Revolution Initiative” (FABRI) activity as its focal activity to stand-up the MENA NWC and provide additional technical support in addressing regional water challenges. FABRI was awarded in 2011 with a mandate to accomplish six key results: 55 1. Middle East North African Network of Water Centers of Excellence established and operational; This result has six requirements: 1) support establishment of the network; 2) design a trust fund or endowment to help ensure the sustainability of the network; 3) support the operations of the MENA NWC secretariat; 4) support the operations of a Founders Committee; 5) support launch of technical, demonstration activities of the Network; 6) identify additional private sector stakeholders to join the Network 2. Integrated Water Resources Management programming strengthened; 3. Access to Clean Water and Sanitation Improved in Target Countries; 4. Research and development capacities in the areas of irrigation, groundwater management, and drought risk assessment and mitigation are strengthened; 5. Trans-boundary water cooperation strengthened in key river basins; and 6. Technical and outreach capacity of USAID staff in water and sanitation programming enhanced. FABRI’s mandate includes staying on the cutting edge of analysis, information, strategy and program design in the water sector in order to provide technical support and guidance to USAID’s Middle East and Africa bureaus. In support of the sustainability of the MENA NWC, FABRI fosters the development of partnerships between and among the scientific community, governments, donors, universities and research institutions, the private sector, and civil society. Additionally, FABRI supports innovative WASH activities in Africa, focusing on providing institutional support to two regional associations to test and share successful approaches in reducing loss of non-revenue water and on setting national policies to enhance sanitation programs throughout Africa. MENA NWC participating centers include:  Arabian Gulf University (Bahrain)  Institute of Agronomy and Veterinary Science Hassan II (Morocco)  International Center for Agricultural Research in Dry Areas  International Center for Biosaline Agriculture (United Arab Emirates)  International Water Management Institute  Jordan University of Science and Technology  Middle East Desalination Research Center  National Center for Agricultural Research and Extension (Jordan)  National Institute for Research in Rural Engineering, Water, and Forestry (Tunisia)  National Office for Potable Water (Morocco)  Qatar National Food Security Program  Qatar Environmental and Energy Research Institute  Royal Scientific Society (Jordan)  Sultan Qaboos University (Oman)  Technion – Israel Institute of Technology  University of Jordan  Water and Environment Studies Institute, An Najah University (West Bank/Gaza) U.S. university partners include:  California Institute of Technology  Georgia Institute of Technology  Purdue University  Texas A&M University 56  University of California at Davis  University of Florida  University of Nebraska  University of Rhode Island  University of Toledo  Utah State University V. Relationship to other USAID/USG, Donor, and MENA Activities/Initiatives The World Bank, Food and Agriculture Organization (FAO), and USAID all have water security initiatives in the Middle East. The World Bank is negotiating with SIDA and other donors to establish a special trust fund to fund the World Bank initiative on water security. A special component of the fund would be dedicated to the MENA Region. Presenting another opportunity, the Global Institute for Water and Health in Geneva is currently expecting support for a special trust fund from the EU and other donors including the Swedish International Development Agency (SIDA) to finance partnerships with MENA NWC to monitor the UN Sustainable Development Goal #6 (Water). SIDA has also advised that it would support select regional partnership activities with MENA NWC focusing on capacity development that are responsive to the recently approved SIDA five year Strategy for MENA. V. Results to Date Results achieved to date by result area include: Result 1: Establish an operational Middle East and North Africa Network of Centers of Excellence (MENA NWC).  Following the launch of the MENA NWC, FABRI developed the Network’s governance and research foundations for implementation and sustainability. FABRI worked with the Network’s 18 participating Centers to design and refine their research proposals for grants and for unsolicited research funds. FABRI assisted with the launching of the MENA NWC Founders Committee, which met three times during the year, and actively set a direction for the Network in governance, business planning, research, and knowledge exchange. FABRI acted as the Interim Management Unit of the MENA NWC until a Directorate was established.  After careful consideration, the MENA NWC Founders Committee agreed to register the NWC in United States. Through a competitive procurement process, FABRI selected the Washington, DC-based law firm of Harmon, Curran, Spielberg + Eisenberg LLP (HCS+E) to prepare the documentation for the incorporation of the Network and to claim Federal tax￾exempt status. HCS+E then worked collaboratively with the Founders Committee to prepare and file the Network’s documents for incorporation in an appropriate jurisdiction, draft the Network’s bylaws, and prepare and submit to the Internal Revenue Service (IRS) an application for Federal tax exempt status as a 501(c)3.  In December 2012, FABRI helped incorporate MENA NWC as a nonprofit association in Washington, D.C. FABRI wrote MENA NWC’s five-year strategic business plan and bylaws, convened the Board of Directors, and prepared application to the U.S. Internal Revenue Service (IRS) for tax-exempt status. FABRI, in consultation with the Board of Directors, decided to postpone the design of a Trust Fund or endowment to help ensure the sustainability of the NWC until 2017.  At this time, MENA NWC has three staff: a CEO, research director, and project manager.  FABRI initiated knowledge exchange activities, including the creation of a website to serve as a clearinghouse for publications and training materials relevant to researchers and practitioners 57 active in the water sector. The website introduces the Network to a wide audience which includes the staff of Network Centers, government institutions, private sector partners and potential funders/donors. It is also an important means to encourage interaction and knowledge exchange between the Centers through its searchable database of Network members and, potentially, through moderated online discussion forums. Result 2: Strengthen Integrated Water Resources Management Programming. FABRI strengthened the legal and regulatory framework for IWRM and implementing methodologies to improve and promote the efficient use of water resources through the MENA NWC IWRM Thematic Partnership. Result 3: Access to Clean Water and Sanitation Improved in Target African and Middle Eastern Countries. FABRI strengthened the performance of water and sanitation service providers by working with regional “platforms” and associations. The goal is to provide them with innovative and tangible support to improve financial, economic, and operating efficiency, including development of association Business Plans, Non-Revenue Water (NRW) initiatives, capacity building, and communications and knowledge sharing programs. Result 4: Strengthen Research and Development Capacities in Irrigation, Groundwater Management, and Drought Risk Assessment and Mitigation. FABRI worked with one of the MENA NWC Thematic Partnerships to identify the most promising and regionally-relevant technologies and techniques for efficient, productive use of water in agriculture. FABRI engaged with the leading experts in these technologies to transfer knowledge to MENA NWC institutions; and developed and presented outreach activities in target countries to encourage government engagement and private sector partnerships around the most successful approaches. Result 5: Strengthen Transboundary Water Cooperation in Key River Basins. Through Oregon State University, FABRI conducted an analysis of transboundary cooperation approaches and applied study findings to the Tigris-Euphrates basin. In addition, FABRI hired a Senior Mekong Affairs Advisor based in Bangkok who provided technical support to the Mekong River Commission. Result 6: Enhance Technical and Outreach Capacity of USAID Staff in Water and Sanitation Programming. FABRI strengthened USAID staff capacity in water and sanitation programming by providing materials and insights based on non-revenue water and sanitation programs in Africa. Other Result 6 accomplishments included:  FABRI received 16 proposals for the Policy, Research, and Development (PR&D) Grants Program.  FABRI supported the development of four U.S.-MENA NWC research project partnership which complemented the PR&D Grants Program.  In conjunction with USAID Partnerships for Enhanced Engagement in Research (PEER), FABRI held a research project design and proposal writing workshop at MERDC in Muscat, Oman on 58 October 7-9, 2012. The workshop included 35 participants from 29 institutions in 12 countries. The goals for the workshop were to increase scientists’ planning and grant writing skills, particularly in water and environmental research; help scientists become more competitive for local and international sources of research funding; and facilitate networking among scientists.  In May 2013, FABRI released the Annual Program Statement (APS) for the Young Water Scientist Partnerships (YWSP) Small Grants Program. Research projects must address a policy, operational, technical, or managerial issue that is identified by stakeholders in government, business, NGOs, or civil society. They must also address one or more of MENA NWC’s priority research areas: water use efficiency and productivity, groundwater, non-conventional water, water/energy/food nexus, and water supply and sanitation. Program is open to full-time or part-time employees of MENA NWC Centers between the ages of 21 and 40.  In September 2013, FABRI released the APS for the Water Innovation Fellowships (WIF) small grants program. The fellowships seek to stimulate partnerships between researchers and practitioners and to reward innovative thinking to solve water problems. As with the PR&D and YWSP small grants program, research projects must address one of MENA NWC’s priority research areas: water use efficiency and productivity, groundwater, non-conventional water, water/energy/food nexus, and water supply and sanitation. Applicants must be citizens and residents of a Middle Eastern and North African country and a full-time employee of an institution or entity based in the Middle East and North Africa. Ten research teams funded by the Policy, Research and Development (PR&D) grants and four research teams funded by the Young Water Scientist Partnership (YWSP) programs are actively collaborating to achieve their research objectives. Four fellows are being supported through the Water Innovation Fellowships program. VI. Award Modifications Award modifications to date include: Date of Mod Amount Origina l 1,395,000.00 Mod 01 1,315,000.00 2,005,000.00 3,320,000.00 Mod 02 4,112,100.00 1,173,600.00 2,367.05 5,288,067.05 Mod 03 7,307,000.00 1,000,000.00 100,000.00 8,407,000.00 59 Mod 04 64,769.72 Mod 05 6/28/2013 850,000.00 incl. budget realignment and no-cost extension Mod 06 7/9/2013 correct completion date to 9/29/2015 Mod 07 8/7/2014 675,162.95 add AFR funds and extend completion date by 6 months to March 31, 2016 Mod 08 12/9/2014 Admin Mod -- Change of COP to Rob Ryan Silva Mod 09 2/3/2015 Admin Mod -- Change of Key Personnel Mod 10 3/3/2016 Change of completion date to 6/30/2016 VII. Evaluation Questions The evaluation should answer the following questions: 5) To what extent has the Network of Water Centers (NWC) developed into a sustainable entity capable of addressing critical water issues in the MENA region? What more needs to be done to ensure the future viability of the Network? In particular, consider the NWC’s strengths and liabilities in the following areas: a. Staffing and organizational structure b. Capacity to collect and manage member and donor funds c. Financial sustainability d. Development and implementation of a long-term business plan e. Recognized convening authority among the NWC’s member base f. Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC 6) To what extent has the NWC been successful in strengthening partnerships, collaboration and information sharing between research institutions in the region, as well as between research institutions and the private sector? a. How successful have small research grants and research pilots been in fostering research collaboration? b. To what extent has information around the results of pilot research projects been shared among members of the NWC? c. How, and to what extent, has the research generated by the NWC been used by policymakers and others stakeholders to inform water resources planning and management? d. What types of support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? 7) What are the key incentives (and disincentives) among NWC member institutions with regard to greater engagement in the network? 60 a. If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? Please consider value and benefits to date, as well as future expectations. b. Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? c. Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? 8) To what extent has FABRI effectively supported the ability of countries to plan and manage their water resources in the following areas: watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture? Based on evaluation findings, the evaluation team should identify actionable recommendations to help USAID to improve design and implement effective and sustainable water programming in the MENA Region. To the extent feasible, recommendations also should speak to long-term NWC sustainability and management considerations. VIII. Existing Activity Data/Information USAID will provide the evaluation team with the following activity documents: o Concept paper o SOW (Task order and modifications) o Activity quarterly and annual reports o Annual work plans o Activity M&E Plan o Other documents, as available and requested by the evaluation team The evaluation team should complete the document review prior to beginning field work. The evaluation team may also request and review additional resources to the extent necessary to perform its work, and USAID will do its best to provide these resources to the extent they are available. IX. Evaluation Design and Methodology 1) Overall Evaluation Design Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to generate answers to the evaluation questions listed above. The evaluation must follow the principles and guidelines for high quality evaluations outlined in the USAID Evaluation Policy (January 2011) and ADS 203. 2) Data Collection Methods The evaluation team must develop data collection tools that will provide high quality data to answer the evaluation questions. Data collection methods may include, but are not limited to, a combination of the following: ● Review of relevant documentation (e.g., activity reports, etc.); 61 ● Key informant interviews (KIIs) (which should include, but are not limited to, KIIs with USAID ME/TS Water and Environment Team staff, including the COR; FABRI/NWC staff; activity beneficiaries, and key donors); ● Focus group discussions or group interviews with beneficiaries and partners (e.g., research grant recipients, scientific community, governments, donors, universities and research institutions, the private sector) and other counterparts and stakeholders; and ● On-line surveys of key informants and training participants. Fieldwork is expected to take place in Jordan, Morocco, and Oman. The evaluation team must propose a methodology that identifies as representative of a cross-section of activity beneficiaries as possible. All beneficiary data should disaggregated by gender and geographic location. USAID will facilitate the introduction of the evaluation team to DAI’s Bethesda, MD office and NWC’s Amman office and the DAI project managers and NWC Secretariat. The evaluation team will be responsible for all logistics for field work. The NWC Secretariat may be able to offer some assistance. Prior to beginning field work, the team is required to share data collection tools with USAID for review and feedback before they are used in the field (e.g., key informant interview questionnaires, focus group discussion protocols, online survey questionnaires, etc.). During this discussion, the evaluation team should also provide USAID with the proposed list of beneficiaries and stakeholders that the team intends to interview during the evaluation and the schedule of meetings. 3) Evaluation Design Matrix and Analysis Plan Prior to beginning field work, the evaluation team must submit for USAID review and approval an evaluation design matrix that details the proposed methodology for data collection and analysis for each of the evaluation questions. The analysis plan should detail what procedures will be used to analyze qualitative and quantitative data; how data from focus group discussions and key informant interviews will be transcribed and analyzed; and how the evaluation will weigh and integrate qualitative data with quantitative survey and performance monitoring data to reach conclusions. 4) Data Quality and Data Limitations The evaluation team must ensure that the data collected clearly and adequately answer the evaluation questions, is sufficiently precise to present a fair picture of performance, and is at an appropriate level of detail to inform conclusions and actionable recommendations. USAID requires that any issues potentially affecting the quality of evaluation data (including the data validity, integrity, timeliness, precision, and reliability) be discussed and documented in the evaluation planning stage and assessed on an ongoing basis during evaluation implementation, including during data collection and analysis. All limitations and measures to address or overcome limitations should be discussed by the evaluation team and USAID in the implementation phase and detailed in the final report. The final evaluation report should be clear and transparent about any notable limitations and if and how they may affect the evaluation’s findings, conclusions, and recommendations. X. Evaluation Deliverables 1. Evaluation Design Matrix, Analysis Plan, and Evaluation Work Plan: The Contractor will submit a final evaluation design matrix, including analysis plan, and evaluation workplan to the Task Order Contracting Officer’s Representative (TOCOR) covering (a) the overall design strategy for the evaluation, (b) the detailed evaluation methodology, including data collection and analysis plan for the evaluation, proposed interviewees, proposed questionnaires, and other data 62 collection tools, and (c) a specific evaluation implementation timeline and fieldwork schedule (using Gantt chart). The evaluation design matrix, analysis plan, data collection tools, evaluation work plan, and budget must receive USAID approval prior to the evaluation team commencing field work. 2. Evaluation Team Planning Meeting. Prior to the onset of the evaluation, the evaluation team should expect to have at least two in-person or phone meetings with USAID, including, but not limited to, the evaluation TOCOR, the activity’s COR, and the Middle East Bureau’s Technical Services (ME/TS) office to: a. Review the evaluation questions; b. Review the evaluation work plan; c. Finalize team members’ roles and responsibilities; d. Discuss the draft evaluation matrix; analysis plan; data collection methods, and instruments and tools; e. Review and clarify any logistical and administrative procedures for the assignment; and f. Discuss other issues, as needed. The evaluation team should review the evaluation questions with USAID for relevance and feasibility prior to the design matrix being finalized. If any adjustments are made to the evaluation questions, these changes, along with detailed rationale for the changes, should be clearly documented in an Annex to the evaluation. Any changes must receive the approval of the evaluation TOCOR. 3. Draft Evaluation Report and Briefing: The Contractor will submit a draft report and conduct a detailed briefing of preliminary findings and recommendations of the evaluation to USAID, including but not limited to the evaluation TOCOR and activity AOR, within three weeks of field work completion. USAID will review the report and provide written comments on the draft within 10 days of receipt. 4. Final Evaluation Report: The final evaluation report must be submitted within seven working days from of receiving USAID’s comments. The evaluation report should not exceed 30 pages, excluding the title page, annexes, and evaluation data. The report must be submitted in English, electronically. At the time of submission of the final report, the survey instruments and data sets, per ADS 2032 and ADS 579, must be submitted on a flash drive to the evaluation TOCOR. The final evaluation report should include: 1. Executive Summary (including activity background, activity purpose, evaluation questions, methodology, findings, conclusions, recommendations, and lessons learned, not to exceed four pages). 2. Introduction and Background (including evaluation purpose, background of the activity, award description). 2 Please note that ADS 203 is currently being revised and will become part of a consolidated ADS 201 chapter. If ADS 201 is approved during the implementation of this evaluation, ADS 201 and its related mandatory references will supersede ADS 203 as the basis of reference for USAID evaluation standards. 63 3. Body of Report organized as follows: a. Evaluation Questions; b. Methodology; c. Limitations; d. Key Findings (Evidence); e. Conclusions; and f. Recommendations. 4. Annexes to include the following: statement of work (including any modifications), reference list of documents, list of persons contacted and affiliation (the evaluator should notify USAID if there are any concerns from a privacy or confidentiality perspective), evaluation design matrix, data collection tools and protocols, evaluation team disclosures of real or potential conflicts of interest, references, short bios of each evaluation team member, and any statements of difference submitted by activity implementers, team members, or USAID. 5. Evaluation Data: per guidelines in ADS 203 and ADS 579, any raw quantitative data (e.g., survey responses, etc.) in electronic form collected by the evaluation team, along with relevant codebooks for the data. The data should be organized for use by those not fully familiar with the project or the evaluation. The Contractor will submit four (4) hard copies of the Final Report and one (1) electronic copy in Adobe format to the evaluation TOCOR. The contractor will submit the final English language version of the evaluation report to USAID’s Development Experience Clearinghouse (DEC) within 30 days of USAID’s approval of the report. The final evaluation report must conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy (http://www.usaid.gov/evaluation). This evaluation will not be considered “final” until the evaluation TOCOR has confirmed, in writing, that the report has met all of the quality criteria. General evaluation report guidelines include: ● The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not, and why; ● The report must include the evaluation Scope of Work as an annex. All modifications, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline must be agreed upon in writing by the COR; ● Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an Annex in the final report; ● Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.) and what is being done to mitigate the threats to validity; ● Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; ● Sources of information must be properly identified and listed in an annex; ● Recommendations must be supported by a specific set of findings; and 64 ● Recommendations must be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. XI. Illustrative Timeline and LOE: Stage Activities included Timeframe Planning Review of relevant award info/documents 1.5 weeks Evaluation Plan submitted (Proposal) and approved Evaluation Team meeting with ME/TS Water Team and Evaluation Activity Manager Preparations Selection of interviewees and interviewers 2 weeks Select and prepare sites Develop detailed data analysis plan ME/TS meeting with the ET to discuss work plan including list of interviewees and data collection tools. Field implementation Conducting Fieldwork (interviews and site visits) 4 weeks Data Analysis Analysis of data and findings 2.5 weeks Reporting Washington de-brief of findings, conclusions and recommendations. 4 weeks Draft of report delivered Incorporated feedback and final draft submitted; dissemination of final report and findings to stakeholders via in-person meeting or teleconference. Total Duration 14 weeks *A week could be 6 working days. XII. Evaluation Team Composition The following three positions are considered key personnel. - Team Leader : The Team Leader will be a senior international expert, with at least seven years of experience in leading and conducting evaluations of development activities, preferably with experience in social science evaluation methods, especially program performance evaluations, rapid appraisal techniques, case studies and other relevant data collection/analysis techniques. The team leader should also have superior management, interpersonal relations and writing skills. It is strongly preferred that the team leader have 65 a solid technical understanding of issues related to water resources management, preferably in MENA region. A minimum of a graduate degree is required. - Team members (2): The Contractor will propose two team members (at least one of which should be a local expert) who are required to complete the task. The Contractor’s proposal will include each team member’s name and key skills relevant to this evaluation, with a current Curriculum Vitae included as an annex to the Technical Proposal. Team members must include: One scientist or engineer with technical expertise (minimum 10 years’ experience) in the water sector, specifically in research and development; and at least one team member must be proficient in Arabic. - Local M&E Specialist The local M&E specialist should have strong experience in social science methods and at least five years of relevant experience, including experience conducting performance evaluations. Relevant economic growth technical experience strongly preferred. All members of the evaluation team should have superior writing, English language, interpersonal, and analytical skills. The names of the evaluation team leader and supporting team members should be included in the Contractor’s proposal, along with key skills relevant to this evaluation. Current Curriculum Vitae for the evaluation team leader and team members should be included as an annex to the Technical Proposal. A curriculum vitae for the local M&E specialist should be provided to USAID during the initial evaluation team planning meeting. XIII. Evaluation Management 1. Logistics The evaluation team is responsible for managing fieldwork logistics. 2. Period of Performance The evaluation is expected to commence in August 2016. The expected duration of the evaluation is approximately14 weeks from commencement to the delivery of the final report. 66 ANNEX 2: EVALUATION METHODS MATRIX Evaluation Question Sub-question Data Collection Methods Data Analysis 1) To what extent has the Network of Water Centers (NWC) developed into a sustainable entity capable of addressing critical water issues in the MENA region? What more needs to be done to ensure the future viability of the Network? In particular, consider the NWC’s strengths and liabilities in the following areas: Qualitative: Hierarchy of Changes, Sustainability of Changes and Most Significant changes Capacity building assessment (using USAID OCA methodologies for sub-questions a, b, d and f Quantitative: descriptive and inferential statistical analyses of survey data Qualitative-Quantitative: Social Return on Investment for sub￾questions b and c. Content analysis of text from KII and FGD, a. Staffing and organizational structure KIIs, MENA NWC documentation b. Capacity to collect and manage member and donor funds KIIs, MENA NWC financial records c. Financial sustainability MENA NWC financial records, KIIs d. Development and implementation of a long-term business plan KIIs, MENA NWC business plan e. Recognized convening authority among the NWC’s member base KIIs, e-survey of NWC members f. Capacity of the NWC Secretariat as an independent facilitation, coordination, and management body able to support the operation of the NWC KIIs, e-survey of NWC members 2) To what extent has the NWC been successful in strengthening partnerships, collaboration and information sharing between research institutions in the region, as well as between research institutions and the private sector? a. How successful have small research grants and research pilots been in fostering research collaboration? KIIs, FDGs, FABRI quarterly/annual reports Qualitative: Hierarchy of Changes, Sustainability of Changes and Most Significant changes Quantitative: Some quantitative analysis of scaled survey responses, mean rating scores, analysis and correlation￾regression analysis of sub-question response rankings b. To what extent has information around the results of pilot research projects been shared among members of the NWC? KIIs, FDGs, MENA NWC documentation, e￾survey of NWC members c. How, and to what extent, has the research generated by the NWC been used by policymakers and others stakeholders to inform water resources planning and management? KIIs, FDGs, MENA NWC documentation, FABRI quarterly/annual reports d. What types of KIIs, FDGs, MENA 67 Evaluation Question Sub-question Data Collection Methods Data Analysis support and initiatives are the most likely to increase or sustain research collaboration and information sharing among member institutions? NWC documentation, e￾survey of NWC members 3) What are the key incentives (and disincentives) among NWC member institutions with regard to greater engagement in the network? a. If and what do NWC members see as the 1) overall value of the NWC as a regional entity; and 2) the primary benefits to their institution of participation in the network? KIIs, e-survey of NWC members Qualitative: Hierarchy of Changes, Sustainability of Changes and Most Significant changes for sub-question a Qualitative-Quantitative: Sentiment analysis and discourse analysis of survey and interview data code weighting of statements and keywords that relate to value of the program, benefits of the program and potential of the program. Quantitative: some quantitative analysis of scaled survey responses, mean rating scores, analysis correlation-regression analysis of sub-questions a and b and response rankings. b. Are there any common characteristics of the entities that are most (and least) engaged and active in the NWC? KIIs, MENA NWC documentation, FABRI quarterly/annual reports c. Based on the perspective of NWC members, if and how can NWC create more value for its members going forward? KIIs, e-survey of NWC members 4) To what extent has FABRI effectively supported the ability of countries to plan and manage their water resources in the following areas: watershed-based efforts to protect freshwater supplies, regulation of groundwater development, and improvements in the productivity of water used in agriculture? KIIs, FGDs, FABRI quarterly/annual reports/ PMP/ Online survey to retrieve a robust sample of individuals not reached through KIIs and FDGs—primarily researchers and grantees who are outside of Jordan, Morocco and Oman Qualitative: Hierarchy of Changes, Sustainability of Changes and Most Significant changes Qualitative data analysis using: Primarily outcome mapping based upon hierarchy of changes model (Annex 2) for research oriented programs Qualitative-Quantitative: Analysis of workshop reports and assessments 68 ANNEX 3: WATER RESOURCE MANAGEMENT IN MOROCCO, JORDAN, AND OMAN Water Resources Management in Morocco Morocco is located in a semi-arid to arid zone and is a water-scarce country confronted with a serious depletion of surface water and groundwater, and a strong dependence on rain-fed agriculture. Only 15% of its total agricultural land is irrigated, resulting in inefficient water use and management. Rainfall is highly variable as to both location and time. The annual average precipitation varies from about 800 mm in the northern region, to 400-600 mm in the center, to 50-200 mm in the southern region of the Atlas Mountains. The northern region, which represents about 7% de the total area of the country, generates about 50% of its water resources, as illustrated in the figure below: Figure 1: Regional Water Resources in Morocco. Dam reservoirs mobilize most of these resources. Nearly 90% of the available surface water resources are already regularized by the existing dam reservoirs, leaving little margin for the mobilization of additional water. Available water resources are more limited and more expensive to mobilize in most of the river basins as illustrated by the figure below. 69 Figure 2: Available water resources and the potential resources per capita per year. As the figure shows, the available water resources and the potential resources per capita per year that could be mobilized are significantly declining below the water stress threshold (1000 m³/capita/year). The figure also shows that the resources mobilized per capita per year have stopped growing, since much of the available resources have been already mobilized. Water resources management in Morocco is decentralized, under the auspices of the river basins agencies (ABH) and the regional irrigation agencies (ORMVA). At the national level the water sector policies are largely the responsibility of the Ministry of Water and the Ministry of Agriculture. There is increasing commitment on the part of the government to seek innovative approaches to expand water supply for all users in the country. One area of particular interest is promoting expanded use of treated wastewater for a variety of applications, especially in agriculture. Support for increased wastewater reuse is a key element in the National Water Plan. Improving water resources management in Morocco faces several major physical, social and economic challenges. The most significant of these are: 1) Lack of clear policies, laws or regulations: Among the most fundamental impediments at this stage is the fact that Morocco still does not have complete or clear policies, laws, or regulations to guide effective and safe reuse of treated wastewater for agriculture or other purposes. 2) Groundwater depletion: Morocco’s large irrigation perimeters are primarily supplied by gravity-fed surface water delivered by dam reservoirs. However, a significant and growing amount of agricultural production relies on groundwater resources that are under growing stress. Groundwater is utilized both as an occasional supplemental source extracted and managed directly by the irrigation agencies (ORMVAs) within the irrigated perimeters, or by individual farms. 70 3) Water use efficiency in agriculture: Under the national plan for agriculture development (PMV), widespread adoption of technologies and practices to optimize water use in agriculture is encouraged but is not occurring due to the following reasons: a. Weak farmer organization around issues of water resources management, including collective investment and management of drip irrigation systems which is at a nascent stage. b. Financial constraints on adopting improved water management: An accelerated adoption of more water efficient irrigation technologies will only be possible if the financial constraints facing smallholder farmers are resolved. c. Smallholder access to credit: Even when this subsidy can be accessed, smallholder farmers are often still not able to raise the remaining funds necessary to make investments in irrigation system conversion. In order to address these problems, it will be necessary for the national and regional governments and international donors to continue to work on programs and policies that: 1) Improve institutional capacity: A critical element of success for sustainable water management is the existence of strengthened and capable institutions at the national, regional and local levels. At the regional and local levels, inefficient capacities are still impeding the implementation of adequate water resources management. 2) Halt the Increasing degradation of water resources: Water resources are under the threat of increasing pollution generated by urban waste water disposal and by agricultural activities using fertilizers and pesticides 5) Address the problem of irrigation water pricing: This is perhaps the most difficult policy issue encountered in the water sector in Morocco. Water is scarce to meet irrigated agriculture needs, yet farmers are not charged close to a market-clearing price for the water that they use. There is no inclination on the part of the government to significantly increase water-use charges. Rather, the government continue to rely on a combination of subsidies and command-and-control mechanisms to achieve its objectives. Still, this approach has not so far produced the expected results. 6) Involve the private sector in water resource management: One approach adopted by the irrigation Authorities to improve water use efficiency in agriculture consists in delegating the management functions for the country’s large irrigated perimeters to private sector management, under specific contracts. This new approach is expected to increase the efficiency and sustainability of irrigation management, increase cost recovery, and improve the quality of service. 7) Mitigate the Impact of climate change: The projections generated by the regional climate change for Morocco, based on the rainfall series, show that by 2050, rainfall reduction could reach 26% in certain regions of the country. Water Resources Management in Jordan Jordan is at the top of the list of the driest countries in the world with water demand greatly exceeding available water resources. Access to a safe water supplies is an essential requirement for all sectors; however, some sectors have made excessive claims on the available water resources. The country can be divided into four physiographic regions: 1) The Jordan Rift Valley (JRV) along the western border of the country; 2) The Highlands to the east of JRV which consist of ranges of mountains and plains; 3) The 71 plains extending from north to south along the western borders of the desert region. and 4) Al-Badia, the desert region in the east which is an extension of the vast Arabian Desert. Figure 3: Jordan Geographical Map The climate of Jordan is semitropical in the JRV, Mediterranean in the Highlands and with continental influence in the eastern desert and plains region. Winter is the rainy season and is warm in the JRV, moderate to cool in the Highlands and extremely cold and dry in the desert land, whereas the summer is hot in the JRV, moderate in the Highlands and hot in the plains and the desert. Groundwater abstraction takes place at more than twice its recharge rate. Developed surface water in Jordan is estimated at 295 MCM at approximately 37% of Jordan’s total water supply. The contribution of the groundwater is estimated at 54% of the water supply. Other water sources include treated wastewater which is used for irrigation in addition to desalinated water from some springs. The Kingdom has been facing a chronic imbalance in the water resources equation and according to the national water strategy, irrigated agriculture covered around 33% of the cultivated area. Rainfall varies considerably with location, mainly due to the country’s topography. It usually occurs between October and May. Annual rainfall ranges between 50 mm in the eastern and southern desert regions to 650 mm in the northern Highlands. Over 91% of the country receives less than 200 mm of rainfall per year. Improving water resources management in Jordan faces several major physical, social and economic challenges. The most significant of these are: 1) Natural Aridity: As one of the driest countries in the world, Jordan’s natural aridity is forecast to get worse with climate change in the next few decades. 2) Climate Change: The Levant appears to have been disproportionately affected by climate change already, and models agree that by 2030 to 2040, temperatures in Jordan will be one to two Celsius degrees warmer, precipitation will be 10 to 15 mm less (13 to 20%), and droughts will be more pronounced. 72 3) Scarcity and Rising Costs: USAID assessments identify absolute scarcity, increasing costs of infrastructure and operations, inefficient use of water in the agriculture sector, and declining water quality as the major threats to water security in Jordan. At present, Jordan provides high quality and reasonable volumes of water for human and commercial needs but at the cost of rapidly depleting its groundwater reserves. 4) Depletion of Groundwater: Research conducted by the U.S. Geological Survey estimates that Jordan will deplete over a third of its groundwater reserves in the next two decades. Overuse of water is inextricable from misuse, especially contamination from industrial, municipal, and agricultural disposal. 5) Spring and Well Depletion: As groundwater levels fall, springs and agricultural wells are going dry, and pumping costs are increasing to the point that many poor farms are unprofitable. Jordan’s most profitable crops are winter vegetables from the Jordan Valley. 6) Water Intensive Agriculture: Water-intensive highland crops, such as maize, barley, wheat, and irrigated olives produce modest or even negative returns. 7) Water Loss: Water scarcity and leakage issues also cause fiscal strains on the GOJ. The water sector already carries a debt of $1.4 billion, and as of 2013 the GOJ is paying additional obligations of up to $120 million per year to a private consortium for water pumped to Amman from the Disi area in southern Jordan. In order to address these problems, it will be necessary for the national and regional governments and international donors to continue to work on programs and policies that: 1) Capping irrigation demand: Setting a limit on irrigation water use is necessary to satisfy municipal, industrial and tourism water demands. Jordan’s water is derived from surface and underground sources. Support more desalination: Recent USAID assessments concur with GOJ opinion that desalination will be essential in meeting human needs within the next few decades. 2) Moderately increase urban water costs: Less than 1% of expenditures in the average Jordanian household go to water and wastewater services, so a moderate increase in water costs would not greatly affect most households. This is emphatically not the case for the rural poor who pay more for water, depend on fewer and more vulnerable water sources, and earn much of their income from water-dependent agriculture. 3) Increase capital investments in water resources: Although water utilities in Amman and Aqaba more than cover their operating costs through tariffs and user fees, and water and wastewater treatment plants are generally well maintained, utilities across the board do not have the funds for major capital investments (as are required to cover the operating costs for the new water originating from the Disi water aquifer). Additionally, most utilities outside of Amman and Aqaba cannot properly maintain their networks. 4) Decrease non-revenue water: In fact, over 40% of Jordan’s municipal water is lost and not billed, either through leakage, poor quality equipment or pipes, malfunctioning meters, illegal connections, or poor administrative practices. Although Amman and Aqaba are much better in this regard, and better than most utilities in the world, overall losses are unaffordable. 73 5) Maximize groundwater and minimize costs in agriculture: The challenge in the water sector is to maximize the value of Jordan’s groundwater (which, in the agriculture sector, means getting “more crop per drop”) while minimizing the human and economic costs of adapting to increasing scarcity. 6) Support adaptation measures for Climate Change in the Jordan Valley: The limited fresh water resources in Jordan are used in municipal, tourist, industry, and agriculture sectors. Increasing demand in domestic water use, tourism and intensification in agriculture requires more water in the future. Agriculture consumes about 67% of the available water resources while 30% is for domestic use. In order to protect the groundwater aquifers, new water resources, such as but not limited to treated wastewater, must be explored that will augment the available water supplies. Water Resources Management in Oman and the Gulf Oman lies in an arid region; rainfall is limited and irregular over much of the country. Mean annual rainfall in the coastal plains and desert areas is relatively low, less than 40mm. In mountain areas, however, where rainfall is greater, up to 350mm, it provides a source of natural recharge to a number of aquifers including those in the interior and coastal areas. There are two main types of water resources in the Sultanate; conventional water resources (natural) including surface and ground water that represent about 87% of the water supply and non-conventional water resources including desalinated water and treated wastewater that represent about 13%. Figure 4: Oman and the Gulf Region Oman still uses a traditional water system called Aflaj (the singular is Falaj) originally developed by the Persians in 500 AD and used throughout the Middle East. The Aflaj System is officially a UNESCO World Heritage site as well as an operating water system. The Aflaj System consists of three primary sources of water—surface water represented by the Ghaily Falaj (springs and a few permanent wadi flows); the Aini Falaj (groundwater) and the Daudi Falaj (a system of wells (qantats). Together, these sources cover 70% of the total water usage. Some areas have shown more severe water depletion due to over-abstraction, in Al Batinah and Salalah coastal areas saline intrusion has caused irreparable damage to the aquifer. Desalination of sea-water which become an important contributor to water supply where natural water resources are unavailable or inadequate. Natural water resources comprise two main resources “renewable” and “non-renewable.” The “renewable” resources are replenished from rainfall and stored in under groundwater aquifers. These 74 resources are generally suitable for domestic and agricultural purposes. Groundwater becomes brackish and saline toward the coast, and there is similar deterioration of water quality in the interior particularly in the lower reaches of the wadis. In the Najd (Southern Oman), groundwater occurs within four distinct limestone formations and the water quality within them varies from potable to brackish. Although Oman has made great strides in meeting its water challenges, improving water resources management in the Gulf Region in general faces several major physical, social and economic challenges. The most significant of these are: 1) Huge water deficits: GCC countries are suffering from a huge deficit in their water resources reaching more than 20 billion cubic meter. These deficits are being met mainly by an intensive over-drafting of renewable and non-renewable groundwater resources for the agricultural sector, and by the extensive installation of highly expensive desalination plants for the municipal sector, and by reusing a small percentage of treated wastewater in the agricultural and municipal sector. 2) The conflict between the agricultural and domestic sectors on the limited water resources in the region are rising, and as a result, groundwater over-exploitation and mining is expected to continue in order to meet growing demand in these two sectors. 2) Population growth: If current population growth rates, water management approaches, water use practices and patterns continue, annual water demand may reach more than 50 billion cubic meter (Bcm) by the year 2030. 3) Future limited desalination capacity and wastewater reuse, means the demand will have to be met mainly by further mining of groundwater reserves, with its negative impacts of fast depletion and loss of aquifer reserves and the deterioration of water quality and salinization of agricultural lands, of which these resources usefulness is questionable with the expected deterioration of their quality. 4) Water scarcity adverse effects: Under these circumstances, water will become an increasingly scarce commodity, and will become a limiting factor for further social, agricultural and industrial development. 5) Inefficient water use in the agriculture: Traditional irrigation practices and high per capita water use and growing amounts of non-revenue water all contribute to the growing scarcity. 6) Rising internal water allocation conflicts: Conflicts between the agricultural and municipal sector for water are likely to increase. 7) Rapid depletion of groundwater and groundwater quality; Deterioration due to its over-exploitation, with multiple impacts on agricultural productivity and ecosystems, is likely to occur under present policies and practices. 8) Urban water deterioration: Inferior quality of water services in large cities due to fast pace of urbanization; and weak water institutions due to fragmentation of water authorities and lack of coordination and inadequate capacity development will particularly affect urban areas. A major review and shifts in the current policies are required to address these challenges including: 75 1) Encouraging food self-sufficiency and undertake sweeping measures in water conservation: Currently, there two main challenges of water resources management in the GCC countries. are the unsustainable use of groundwater resources with its ramification for socio-economic development, and the escalating urban water demands and its heavy burden on their national budget and negative impacts on the environment. 2) Interpreting the sustainability of these resources in a socio-economic rather than a physical context: This implies that full consideration must be given not only to the immediate benefits and gains, but also to the “negative impacts” of development and to the question of “what comes after?” 3) Identifying exit strategies: Strategies need to be identified, developed, and implemented if and when the aquifer becomes seriously depleted. An exit strategy scenario must include balanced socio￾economic choices on the use of aquifer storage reserves and on the transition to a subsequent less water-dependent economy, and the replacement water resource. 4) Focusing on the demand side of management as well as the supply side: GCC countries are going ahead with desalination plant construction and expansion in order to meet the spiraling domestic water demands – a function of population and urbanization growth. This is due mainly to the reliance on the supply side of management with little attention given to the demand management and the non￾existence of price-signaling mechanism to consumers. 5) Evaluate desalination plants for the future: Despite their relatively enormous cost and heavy burden on the national budget, limited operational life (15-25 years), dependence on depleting fossil fuel, and negative environmental impacts on the surrounding air and marine environment, desalination remains an important technology for the GCC countries with limited directed R&D directed towards these technologies. 6) Mitigate the Impact of climate change: According to a recent study, climate change in much of the area occupied by GCC countries will cause a decrease in precipitation in combination with a projected temperature rise and will render even wider areas of the Gulf States unfit for agriculture and inhabitable for a non-nomad population. This will raise the stakes for existing water supplies in the region, accelerating the depletion of non-renewable saline aquifers. Most fossil water resources on the Arabian Peninsula are between 10,000 and 30,000 years old. Already, over-exploitation has resulted in an increase of salinity levels in groundwater from saltwater intrusion and made desert springs disappear on the Peninsula. WATER RESOURCES MANAGEMENT IN TUNISIA Tunisia is located in North Africa, in semi-arid to arid zone. Rainfall is highly variable in space and time and drought episodes are more frequent. The country has a total population of about 10.9 million, 65% of which is urban, residing primarily on the coastal region. Water resources in Tunisia are characterized by scarcity and a pronounced seasonal and spatial irregularity. Non-Conventional water resources, mainly treated waste, are also used. The northern region provides about 82% of the total available surface water resources, although this region covers only 16% of the total area of the country. The center region, covering 22% of the area, is characterized by irregular resources, and provides about 12% of the total surface water resources. The southern part of the country which accounts for approximately 62% of the total area, provides about 6% of the country’s total potential water resources. The current global rate of water resources exploitation is estimated to be about 90% of the total available resources, with surface water exploitation representing about 85% and groundwater 76 exploitation 95%. About 80% of mobilized water resources are delivered to irrigation, and 20% of the mobilized resources are delivered to municipal, industrial and tourism sectors. Groundwater, representing 44% of total water resources, provides two-thirds of the national water supply and 82% of water needed for irrigated agriculture. As a result, one-third of the country’s aquifers are currently severely over-exploited. Figure 5. Map of Tunisia During the last decades, water has been a high priority for the government, with heavy investment in water resource management and water services. Major infrastructure investments developed to date (dams, deep and shallow wells as well as complex systems of inter-basin transfers) allow the country to capture and use about 80 percent of its total available water resources. Although Tunisia has made great strides in meeting its water challenges, improving water resources management in the country in general faces several major physical, social and economic challenges. The most significant of these are: 1) Agricultural over-use: Agriculture is the largest water consumptive center of use. More attention needs to be paid to the local soil and plant characteristics. Understanding the vulnerability of soils and plant to salinity and to hydraulic stress could translate into improvement in soil moisture conditions, adequate drainage systems design, reduction in water intake and optimal farm environment. 2) Increasing demands, decreasing supply: While Tunisia is able to meet more than 90 percent of its current demand for water, the country has few options for finding new sources of water. 3) Close to the limits of water use: The use of conventional water resources in Tunisia will reach its limits in the near future. Simulation, using river basins simulation model, of future scenarios of 77 projected water supply and demand indicate that most of the regions in the center and the south, will face significant water shortages by the year 2030. To supplement traditional sources of water for irrigated agriculture, Tunisia has actively pursued non￾conventional sources of water, such as re-use of treated wastewater, and aquifer artificial recharge. In addition, a set of measures has been adopted in order to achieve the strategy objectives. These include: 1) Mobilization of all water resources which could be mobilized at reasonable costs. 2) Water transfer from river basins in the northern region, having excess of water, to the other river basins having water shortages. 3) Rational use of groundwater resources and promotion of aquifers artificial recharge. 4) Exploration of new groundwater resources in deep confined aquifers. 5) Conjunctive use of surface water and groundwater. 6) Promotion of techniques and practices aiming at water saving, especially in irrigation. as this sector water consumption represents about 80% of the mobilized resources. 7) Protection of dam reservoirs against siltation. 8) Promotion of participatory water management, by involving farmers in irrigation infrastructure rehabilitation in the large irrigation schemes. 9) Promote further use of treated waste water in agriculture and other sectors. 78 ANNEX 4: PERFORMANCE INDICATORS AND EVALAUTION FINDINGS Result Sub-Result Indicators Target Final Report Sustained 1: MENA-NWC Network established and Operating 1.1 MENA￾NWC Governance structure established 1.1.1 MENA NWC legally registered 1.1.2 Number of essential governance structures established and operative (Founders Committee, Steering Committee, Board of Directors, Secretariat) Yes 5 Yes 9-Founders Committee, interim board, board of directors, 4 board committees, Center Director working groups, technical advisory committee Yes 2-Center Director and Board of Directors to be replaced by elected board (from member centers) 1.2 MENA￾NWC financially sustainable 1.2.1 Business plan developed and approved 1.2.2 Financial support pledged from donors, private sector, and other sources 1.2.3 Number of types of pledged donors (bilateral, multilateral, foundations, corporate, government, individual) demonstrates diversity of funding sources Yes 25 Million 4 Business plan developed and approved by Board of Directors 72k from Oman 75K from GOJ World Bank, SIDA and Qatar funds 3-Government, multilateral donor, bilateral donor New Business plan will be submitted to new board 140K from SIDA and 40K from GOJ 2-Government and multi-lateral donors 1.3 MENA￾NWC technical program addressing critical regional water needs 1.3.1 Number of technologies or management practices under research as a result of USG assistance 1.3.2 Number of technologies or management practices under field testing as a result of USG assistance 1.3.3 Number of technologies or management practices made available for transfer as result of USG asst. 1.3.4 Number of government agencies, private sector firms, and/or civil society 10 10 4 20 13 as part of P, R and D projects and 3 U.S.-Network Research Partnerships 6 as part of P, R and D projects and 3 U.S.-Network Research Partnerships 4 Red Tides Manal, River Basin Modeling, NRW Toolkit, Radar Probing 106 engaged in submitted proposals Yes Yes Yes No 79 Result Sub-Result Indicators Target Final Report Sustained organizations partnering with Network Centers on research and development projects 1.3.5 Number of young and women researchers engaged in MENA NWC supported research activities 1.3.6 Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations and community-based organizations that applied new technologies or management practices as a result of USG assistance 1.3.7 Number of visits to the MENA NWC website 1.3.8 Number of fans on the MENA NWC Facebook Page 40 12 25,000 5000 Young=53 Women =84 (of which 15 are working at universities and research centers in the U.S. 1-Government of Oman, Radar Probing 7655 283 Yes 1.4 MENA￾NWC strengthened through private sector participation 15 30 2: Integrated Water Resources Management Programming Strengthened; 2.1 IWRM Legal and Regulatory Frameworks Strengthened 2.1.1 Number of government agencies, utilities, and service providers involved in IWRM research activities 8 10 2.2 IWRM Methodologies Implemented 2.2.1 Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations and community-based 6 0 80 Result Sub-Result Indicators Target Final Report Sustained organizations (CBOs) that applied new technologies or management practices as a result of USG assistance 3: Access to Clean Water and Sanitation Improved 4: Research and Development Capacities in Irrigation, Groundwater Management, Drought Risk Assessment and Mitigation Strengthened 5: Trans￾boundary Water Cooperation Strengthened 6: Technical and Outreach Capacity of USAID Staff in Water and Sanitation Programming Enhanced. 81 ANNEX 5: FABRI FINAL REPORT LESSONS LEARNED AND EVALUTION FINDINGS The Final Report from program implementers listed a number of lessons learned and comment on NWC’s finances, management, and sustainability. These are summarized below with the evaluation findings relating to those lessons indicate the extent to which these were confirmed, clarified or refuted by evaluation data. FABRI Final Report Evaluation findings Lessons Learned 1 When creating a new regional institution, time is more important than funding. In general interviews and survey comments confirmed this as it was mentioned frequently that there was not enough time to complete and launch projects. 2 Too much available project funding can undermine the sense of buy-in and ownership as partners fall into dependency relationships. From the point of view of the researchers interviewed, not enough funding was allocated to apply the research they had done and too much was allocated for things such as travel, meetings, workshops, etc. 3 One-way dependency relationships are unlikely to become more balanced at project speed. In the end, the three grants programs, all fully funded by FABRI, created an imbalance between the Network and the Centers. Interviewees and survey respondents attributed this to the fact that the MENA-NWC was not fully established as a regional entity until the last phase of the program. 4 The Network cannot solely do research if it is to be viable in the future. Discussions with donors revealed that they cannot use their funds for research. If the Network is to be useful, it must respond to donor priorities related, particularly to policy and implementation priorities. Interviews and survey responses indicate that this is indeed the case but also suggested that the MENA-NWC needed to look beyond the donor community. Further, other donors will fund the application of research which is something that researchers noted was a missing component of the program. 5 The Board of Directors discouraged the possible policy implications of the research by not requiring government approval of and involvement in research projects. There are differing views on this question. While most respondents and interviewees believed that there should have been a greater role for the government some suggested that this could result in long delays for the projects. 6 Scientists are conservative. Research centers are not immediately prepared to work with the private sector, as they harbor both mistrust and arrogance. FABRI underestimated the antipathy that many of the Centers held for the private sector, often disdainful of their profit seeking. No interviewees and respondents indicated any antipathy for the private sector—in fact, most of them faulted the program for not engaging more with it. 7 Participating scientists and researchers were largely engineers. It will take very deliberate actions to get social scientists, legal experts, land tenure specialists, and business management experts involved and playing substantive roles in design. The evaluation findings do not confirm or refute this lesson other than the observation that most participants were engineers is true. 8 Fundraising should be carried out in-house. Outsourcing was unsuccessful as fundraisers were more committed to their own companies than the Network. Those who addressed this issue agreed that this was indeed the case. 82 FABRI Final Report Evaluation findings 9 USAID can play an outsized role among donors because of their high regard, even when USAID is reluctant. USAID needs to exploit that generally high regard and leverage it into attracting other donor funds. Those who addressed the question of leveraging other resources suggested that both USAID and the program implementers did not pursue other donors for the program. 10 Regional partners should design and fund their own websites rather than use project funds. FABRI managed the design of two websites for MENA NWC and AfWA. Both required time and funds, and by project end, it is still uncertain if either one is being used or will be used. At the time of the evaluation the MENA-NWC did not have any funds and the Director was working on a volunteer basis with assistance from a former program expert from the region who was also working as a volunteer. Since the MENA￾NWC had not begun as a regional entity until late in the project, informants stated that this hampered their ability to raise money or even collect membership fees. 11 Success comes from using local experts rather than relying on foreign experts. The informants who addressed this question concurred with this observation. 12 Regional grants programs need a well-trained on the ground implementer to help with the day to day management, monitoring and payments, especially when dealing with multiple countries and currencies. The informants who addressed this question concurred with this observation. 13 There is more to be gained by scaling up rather than piloting projects. The informants who addressed this question concurred with this observation. Finances, Management, and Sustainability 1 Appropriate MENA NWC Staffing. FABRI staffed MENA NWC with full time Executive Director, Director of Policy, Research and Development and Communications and Knowledge Sharing Specialist, as well as several short term part-time positions. The evaluation confirmed that only one staff member, the Director, with the technical assistance of a co-director based in Washington, DC is currently working for the MENA-NWC 2 Diligent and Responsible Budget Management. Cost￾effectively managed a USD 20 million budget, spending USD 19,272,132 as of May 31, 2016. With the contract closing on June 30, FABRI projects spending an additional $500,000, including future Negotiated Indirect Cost Rate Agreements adjustments, spending for a total of roughly 99% of the contract ceiling over the four and a half-year project life. According to some interviewees there were $180,000 remaining from the FABRI plus the $80,000 donated by Oman which was supposed to be used to fund the site of MENA NWC and office rent in ACWUA, a server, a screen and a printer. 3 Smooth Project Management Transitions. FABRI effectively managed staff transitions on the project, quickly replacing important staff. According to interviewees the effects of these transitions were more problematic in the region. 4 Successfully Awarded and Managed Complex Research Projects. FABRI successfully managed five research programs across the Middle East and Africa. The PR&D, WIF, YWSP, U.S. –Middle East Research Partnerships Program in the Middle East included 28 scientific research projects, implemented by 46 grantees and 11 subcontractors. The grants were generally viewed positively by researchers although there were some issues with respect to the management of the program cited by some. Project implementation was cited as the second most serious constraint to the program’s achieving its goals by survey respondents (n=12). 5 Assisting MENA NWC to Transition to Institutional Independence. FABRI negotiated effective partnerships with two Jordanian entities. According to interviews the MENA-NWC was not funded at the end of the program. 83 ANNEX 6: LIST OF INTERVIEWEES Institution Country Type Gender Name Title Email Sultan Qaboos University Oman Other Stakeholders Male Osman Abdalla Director of Water Research Center osman@squ.edu.om Royal Scientific Society Jordan Other Stakeholders Male Dr. Muhamma d Saidam muhammad.saidam@rss.jo Royal Scientific Society Jordan Other Stakeholders Female Eng. Rana Ardah rana.arda@rss.jo Arab Fund for Economic and Social Developmen t, Kuwait Kuwait Board of Directors Male Muwaffaq Saqqar Chairman; Water and Environment Consultant Engineer msaqqar@yahoo.com Hassan II. Institute of Agronomy and Veterinary Medicine Morocco Conference and Workshop Participants Male El Houssine Bartali Researcher bartali.h@gmail.com; bartali_elhoussine@yahoo.fr Hassan II. Institute of Agronomy and Veterinary Medicine Morocco Conference and Workshop Participants Female Ouiam Lahlou ouiamlahlou@gmail.com ECO Consult Jordan Implementing partner Female Razan Quossous Grants manager rquossous@siyaha.org Hassan II. Institute of Agronomy and Veterinary Medicine Morocco Conference and Workshop Participant Female Yasmina Imani yasmina.imani@gmail.com Sultan Qaboos University Oman Other Stakeholders Male Dr. Ali Khamis Al￾Maktoumi Assistant Dean for Postgraduate Studies and Research; Assistant Dean for Postgraduate Studies and Research; Assistant Professor, Environmental Eng. (Water Resources) ali.almaktoumi10@gmail.com 84 Institution Country Type Gender Name Title Email Sultan Qaboos University Oman Other Stakeholders Male Ahmed Al-Busaidi Department of Soils, Water and Agricultural Engineering ahmed99@squ.edu.om; albusaidiahmed@yahoo.com; albusaidiahmed1@gmail.com Sultan Qaboos University Oman Other Stakeholders Male Yaseen Al-Mulla Associate Professor yalmula@squ.edu.om Executive Director MRNA￾NWC Jordan Other Stakeholders Male Dr. Ghazi Abu Rumman g.aburumman@menanwc.org AQABA Water Jordan Other Stakeholders Male Mohamm ad Alshafey Head of Research and Planning m.shafei@aw.com.jo Eng Mohammed Hossam Sallam Jordan Other Stakeholders Male Eng Mohamm ed Hossam Sallam sallam@rocketmail.com Ministry of Water Jordan Other Stakeholders Male Eng Ali Subah ali_suboh@mwi.gov.jo University of Jordan Jordan Other Stakeholders Male Dr. Sayyed Khattari khattari@ju.edu.jo NCARE Jordan Other Stakeholders Female Dr. Sireen Naoum naoum@ncare.gov.jo; sireen_2004@yahoo.com NCARE Jordan Other Stakeholders Male Dr. Muien Qaryouti qaryouti@ncare.gov.jo; muienqaryouti@yahoo.com Royal Scientific Society Jordan Other Stakeholders Male Dr. Almoayie d Assayed almoayied.assayed@rss.jo JUST Jordan Other Stakeholders Male Dr. Jamal Abu Ashour jamals@just.edu.jo JUST Jordan Other Stakeholders Female Dr. Muna Abu Dalo maabudalo@just.edu.jo Royal Scientific Society Jordan Other Stakeholders Female Eng. Tharwh Qutaish tharwh.qutaish@rss.jo Royal Scientific Society Jordan Other Stakeholders Male Rafar Assi rafat.assi@rss.jo ECO￾Consult Jordan Other Stakeholders Male Ra'ed Daoud raed.daoud@ecoconsult.jo ACWUA Jordan Other Stakeholders Male Eng. Khaldon Khashman khaldoun_khashman@acwua.org Former FABRI staff Morocco Other Stakeholders Female Yosr Sibari yosr.sibari@gmail.com University Hassan II – Faculty of Sciences Morocco Stakeholder/Researc her Female Dalila Loudyi loudyi.d@gmail.com 85 Institution Country Type Gender Name Title Email ONEE￾International Institute of Water & Sanitation Morocco Other Stakeholders Male Mahmoud Hafsi Mahmoud mhafsi@onee.ma International Institute of Water & Sanitation Morocco MENA-NWC board member- Male Samir Bensaid Samir.bensaid@gmail.com International Institute of Water & Sanitation Morocco Stakeholder Female Khaoula Lamzouri khaoulalamzouri@gmail.com University Cadi Ayyad￾Marrakech￾Center of Studies & Research on Water & Energy Morocco MENA-NWC board member Female Laila Mandi mandi@uca.ma University Cadi Ayyad￾Marrakech￾Center of Studies & Research on Water & Energy Morocco Researcher Male Abdessam ad Hajjaj dasHaouz@yahoo.fr University Cadi Ayyad￾Marrakech￾Center of Studies & Research on Water & Energy Morocco Researcher Male Lahbib Latrach latrachlahbib@gmail.com University Ibn Toufail￾Kenitra￾Faculty of Sciences Morocco Stakeholder/Researc her Male Mohamed Taky takymohamed@gmail.com INGREF Tunisia MENA-NWC board member Male Thameur Chaibi chaibithameur@yahoo.fr chaibi.medthameur@iresa.agrine t.tn INFREF Tunisia Researcher Male Bahri Haithem Haithem.bahri@gmail.com INGREF Tunisia Researcher Female Olfa Mahjoub olfama@gmail.com 86 ANNEX 7: KEYWORD ANALYSIS Theme Word/Phrase Male Female Freq. Positive Negative Freq. Positive Negative Sustain￾ability partner[s, ed, ing, ship] 38 12 12 57 22 12 continu[e, es, ed, al, ing] 14 3 7 11 4 4 propos[e, es, ed, al] 19 3 8 14 5 0 sustain[ed, s, able, ability] 21 0 13 7 1 5 collaborat[e, es, ion, ed] 22 8 15 42 11 3 appl[y, ied, ing] 11 2 12 23 4 4 business plan 6 1 9 9 0 5 scal[e, es, ed, ing] 5 6 3 12 2 1 board 31 0 27 26 2 12 conference 4 1 5 9 3 1 travel[s, ed, ing] 7 0 10 7 0 5 Effective￾ness weak 14 0 15 4 0 4 train[s, ing, ed} 13 0 4 29 10 11 [e, a]ffect[s, ive] 11 2 4 4 3 0 staff[ed, ing] 11 4 7 6 1 1 program[s, me, ed, ing] 14 3 12 28 10 8 grant[s, ed ing, ee] 45 4 20 25 5 5 fund[s, ed ing] 120 3 46 78 9 28 budget[s, ed ing] 11 0 10 10 0 6 USAID 29 5 21 20 0 8 Mission 5 0 2 0 0 0 DAI 41 0 35 22 2 12 Outcomes benefi[ts, ted, icial, iciary] 13 11 7 14 2 4 valu[e, es, ed, ing] 13 5 10 9 2 2 chang[e, es, ed, ing] 3 1 4 18 4 4 exchang[e,es, ed, ing] 6 6 0 6 0 5 contribut[e, es, ed, ing] 6 2 5 9 5 1 capacit[y, ies] 14 8 11 14 1 6 differen[ce, ces, t] 20 5 8 12 0 0 pilot[s, ed, ing] 12 6 1 18 9 4 Impact government 12 6 17 17 1 7 private sector 19 3 15 21 2 16 institution[s, al] 8 5 15 32 6 8 communit[y, ies] 17 8 2 22 6 0 women 9 1 1 9 8 0 87 Theme Word/Phrase Male Female Freq. Positive Negative Freq. Positive Negative young 2 0 0 12 9 0 Challenges Washington 4 0 4 3 0 2 communicat[e, es, ed, ing] 5 0 5 8 4 4 Israel 6 0 9 8 0 7 politic[s, al] 7 0 7 7 0 5 time 52 3 39 41 0 37 88 ANNEX 8: ORGANIZATIONAL CAPACITY ASSESSMENT (OCA) SCORE SHEET Sub-Section Score Comments Section 1: Governance and Legal Structure 1.1 Vision and Mission 2 Vision and mission statements (strategic plan documents) but are only partly relevant to organization’s current purpose or aspirations 1.2 Legal requirements and status 3 The organization has legally registered, its by-laws are incomplete or outdated 1.3 Organizational structure 1.3 The organizational structure is documented but outdated 1.4 Board composition and responsibility 2 An external board has been appointed but members are drawn from a narrow group and it is not independent of management 1.5 Succession planning 1.8 The organization has high dependence on its current leader Average section 1 score 2 Basic Capacity Section 2: Financial Management and Internal Control Systems 2.1 Budgeting 1.3 The organization has an outdated budget based on projections of funds made at the end of the project that did not materialize 2.2 Accounting system 2 An accounting system was used in the past (up to 2016) but currently has a manual system because there is only one staff member. Future accounting will be determined by the current donors (SIDA and the GOJ). 2.3 Internal controls 1.3 There has been and continues to be a weak to inadequate separation of duties among procurement staff and staff handling revenues 2.4 Bank account management 1.3 Bank accounts are managed in accordance with donor requirements 2.5 Financial documentation 1.2 Written financial documentation policies and procedures are weak and require significant changes 2.6 Financial statements and reporting 1.8 Financial statements are included in end-of-project documents but internal financial statements are not regularly produced 2.8 Cost sharing 1.5 The organization has not had any donor cost-sharing requirements Average section 2 score 1.5 Low to Basic Capacity Section 3: Administrative and Procurement Systems 3.1 Operating policies, procedures, and systems 1.7 Operating policies, procedures, and systems are inadequate and require substantial changes 3.2 Information technology 2 Use of IT is basic and there are no IT systems 3.3 Travel policies and procedures 1 Travel policies and procedures followed USAID requirements until the end of FABRI in 2016. At present, there are no funds for travel. 89 Sub-Section Score Comments 3.4 Procurement 1.3 Procurement policies and procedures are written and weak and require significant changes 3.5 Fixed assets management 1.7 Asset management policies and procedures are not written 3.6 Branding and marking 2 The organization had prior donor branding and marking requirements that were not fully met (from USAID) and is now in the process of complying with its first donor branding and marking requirements for other donors Average section 3 score 1.6 Low to Basic Capacity Section 4: Human Resource Systems 4.1 Adequacy of staffing and job descriptions 1 Policies and procedures on staffing and job descriptions are: are inadequate, require substantial changes, and are not followed. Many key management, technical, or finance positions have not been established, remain vacant, or are filled by people without appropriate skills. Some vacancies have significantly reduced efficiency or effectiveness for more than 6 months. 4.2 Recruitment and retention 1 Policies and procedures for staff recruitment and retention are inadequate or not followed 4.3 Personnel policies 1 Personnel policies and procedures are inadequate and require substantial changes 4.4 Staff time management and payrolls 1 Payroll policies and procedures were not updated after the end of FABRI 4.5 Staff and consultant history 1 Staff and consultant resumes are not kept up to date for a roster for new proposals 4.6 Staff salaries and benefits 1 Policies and procedures on staff salaries and benefits are not adequate and not in compliance with labor laws 4.7 Staff and contractor supervision and work planning 1 The organization has not defined and documented supervisory assignments Average section 4 score 1 Low Capacity Section 5: Program Management 5.1 Donor compliance requirements 3 The organization has received a direct award from other donors with complex compliance requirements. MENA-NWC has adequate systems and procedures to meet complex donor requirements, but they may need some updating. 5.2 Sub-award management 1.5 The organization has not provided formal sub-awards 5.3 Technical reporting 2 Technical reports on projects have been prepared with significant assistance from a donor or technical assistance providers 5.4 Stakeholder involvement 1 The organization has now written policies and procedures for stakeholder involvement no regular meetings or communication with clients and inadequate physical space to meet with individual clients and groups 5.5 Culture and gender issues 2.5 The organization has adequate tools and expertise for systematically assessing culture and gender issues 90 Sub-Section Score Comments Average section 5 score 2 Basic capacity Section 6: Project Performance Management 6.1 Monitoring and quality assurance 1.8 Policies and procedures for monitoring are not written and the organization has had significant difficulty setting realistic targets and meaningful performance indicators 6.2 Project and program evaluation 1.5 Written policies and procedures for evaluation are weak and require significant changes and Inadequate ability to review evaluation data quality and reports or address evaluation findings and recommendations in existing and new projects 6.3 Service delivery standards 1.8 The organization has adopted recognized, international, national, or sectoral standards for service delivery 6.5 Project performance 2 Over the past 3 years, the organization has received average performance ratings or references from donors and government funders (3 on a scale of 5) Average section 6 score 1.8 Close to Basic Capacity Section 7: Organizational Management and Sustainability 7.1 Strategic (business) planning 2 The strategic plan (business plan) is written and weak and requires significant changes, does not reflect its current vision, mission, and values and is not based on an adequate analysis of strengths, weaknesses, opportunities, threats, and realistic resource requirements and availability 7.2 Annual workplans 1.5 Annual work plans have not been prepared 7.3 Change management 1.3 Weak processes or structures for responding to changes in leadership, staffing, budgets, government policies, and donor funding levels and priorities 7.4 Knowledge management and external linkages 2 Weak systems for documenting, storing, and disseminating program knowledge, occasionally analyzed and shared good practices and lessons learned internally, but not actually applied them, established some formal networks, participated in discussions with donors, governments, and civil society organizations on approaches, lessons learned, and good practices and presented its approaches and results at external events 7.5 Fundraising and new business development 2 The organization has had occasional cash flow problems, but positive net income last year, insignificant funding from cost recovery, sales, or membership fees, limited absorptive capacity for additional projects 7.6 Internal communications and decision making 1 Structured settings do not exist to exchange ideas and discuss problems or opportunities management has not in the past listened to staff and inadequate space and infrastructure to facilitate internal communications 91 Sub-Section Score Comments 7.7 External communications 2.3 Little capacity for implementing an external communications strategy and overseeing written and oral products, weak website, a neutral reputation with key stakeholders but adequate capacity for implementing the external communications strategy and overseeing written and oral products and a positive reputation with key stakeholders 7.8 Advocacy and influence 1.3 Has not carried out significant advocacy activities over the past three years, lacks staffing or skills for effective advocacy, has not influenced the formulation or implementation of government policies at the national or local level, has had little influence on the general public’s views Average section 7 score 1.7 Low to Basic Capacity Average OCA score (average of the seven section scores) 1.7 Low to Basic Capacity