USAID/EGYPT ECONOMIC POLICY INITIATIVE CONSORTIUM (EPIC) ADMINISTERED BY THE INTERNATIONAL CENTER FOR ECONOMIC GROWTH (ICEG) Cooperative Agreement # 263-0233-A-00-7001-00 EVALUATION REPORT This report was prepared by Richard Lindsey Wellons for Chemonics International Inc., prime contractor to the United States Agency for International Development (USAID) for the TAPR Program in Egypt Submitted to: USAID Submitted by: Chemonics International Inc. November 29, 2000 EPIC/ICEG Evaluation i LW Draft, 04/30/04 ACTIVITY EVALUATION - DRAFT – November 29, 2000 - Economic Policy Initiative Consortium (EPIC) administered by the International Center for Economic Growth (ICEG) Table of Contents Executive Summary I. BACKGROUND AND PURPOSE OF THE EVALUATION A. Background B. Purpose and Methodology II. PROJECT GOALS, ACTIVITIES AND IMPLEMENTATION EXPERIENCE A. Overall Project Goals B. Research Program and Policy Papers C. Research Dissemination D. Growth Fellows Program E. Staff Training for Ministries F. Institutional Capacity Building, Technical Assistance, Special Projects III. EVALUATION CONCLUSIONS, LESSONS LEARNED APPENDIX Major Research Studies Policy Papers Grantees / Experts Newsletter Topics Policy Briefs Topics EPIC/ICEG Evaluation i LW Draft, 04/30/04 Activity Evaluation Economic Policy Initiative Consortium (EPIC) administered by the International Center for Economic Growth (ICEG) EXECUTIVE SUMMARY - Draft, November 29, 2000 - I. BACKGROUND AND PURPOSE OF THE EVALUATION A. Background The U.S. Agency for International Development (USAID/Egypt) entered into Cooperative Agreement # 263-0233-A-00-7001-00 with the International Center for Economic Growth (ICEG) effective October 20, 1996 for a four-year period. The purpose of the agreement was for ICEG to implement a set of economic research activities through the establishment of an informal consortium of research institutions in Egypt to be called the Economic Policy Initiative Consortium (EPIC). EPIC’s main activities included support of policy oriented research projects, dissemination of policy research, technical advice to ministries, institutional development, training for economic ministries' staff: specialized training on economic reporting, and overseas scholarships in the field of economics.. B. Purpose and Methodology The purpose of this evaluation is to describe, critically assess and document the effectiveness of sponsored activities. It describes in general terms the contribution of those activities to the Egyptian economic reform program. Given that this evaluation is descriptive and intended for purposes of documentation, this evaluation is not intended to provide a set of recommendations pertaining to prospective USAID support for this cooperative agreement. This report provides a descriptive evaluation of the project as distinct from an operational assessment of how the project has been implemented, or an assessment of its wider impact on beneficiaries. For purposes of this evaluation, questions addressed include: What have been the main activities and outputs of the project? Have the original project goals and targets been met? What have been the main constraints on meeting project goals, and, what are the main lessons to be learned for implementing similar future projects? II. SUMMARY OF FINDINGS: PROJECT GOALS, ACTIVITIES AND IMPLEMENTATION EXPERIENCE A. Overall Project Goals 1. EPIC services have contributed toward strengthening the capabilities of Egyptian economic research institutions, but many institutions continue to have strong needs despite project achievements. As EPIC performance was generally positive, this indicates that the initial project document probably set overly ambitious goals. The original rationale for EPIC was appropriate, and remains so, given the continuing strong needs also on the part of Egyptian government EPIC/ICEG Evaluation ii LW Draft, 04/30/04 policy-makers for improved economic policy research and analysis to help support Egypt’s policy reform program. 2. EPIC has had mixed success in meeting the original project document’s first overall goal “to create a community of Egyptian scholars and policy analysts… committed to support the reform process.” The implicit vision of creating a strong, vibrant consortium whose members cooperate closely and effectively to coordinate research priorities supporting reform, has not been achieved. However, ICEG does appear to have had more success in supporting certain consortium member institutions, individual Egyptian researchers, provincial universities and most recently the Ministry of Economy in more direct, one-to-one relations, rather than through a consortium (see Conclusion No. 4) 3. EPIC has been partly successful in meeting the second and third overall goals, providing Egyptian policy makers with quality advice on economic reforms and increasing awareness of and support for policy reform by larger circles of influence, such as business leaders, university professors, and the media, through dissemination of research and training. 4. EPIC appears to have gone far toward meeting the fourth goal of strengthening policy research by working more on a unilateral, one-to-one, basis with most consortium members and individual researchers rather than through the unified “consortium” approach, through providing research grants, dissemination support and in some cases technical assistance. This was due to a shift in management staff and roles about mid-way through the EPIC project. 5. Confusion over management roles had an adverse impact on EPIC operations over the start-up period and into the second year of operations, in terms of delayed implementation of research projects, policy papers published and related services. 6. Many targets for output were met relatively recently, closer to the end of the project period. This appears to have been partly due to delays in completing some research studies, although the “gestation” period required for major studies means that completion naturally would come later rather than earlier in the project. Improved project administration in the second period may also have contributed to overcoming delays and speeding completion of activities. B. Research Program and Policy Papers 7. ICEG and EPIC have conducted a substantive research policy program over the course of the project period and have met the initial project targets for commissioned research and for policy papers and other publications in terms of the number of research projects and the range of subjects originally suggested. However, there has been mixed success in meeting certain goals, such as completion targets that were not always met on time, and the resulting lack of adequate public outreach, especially in the first management period. 8. EPIC research subject areas have generally matched the major policy priorities of USAID, and, to a lesser extent, those of the government of Egypt. However, the direct impact of EPIC sponsored policy research on Egyptian policy reform is not possible to measure credibly in a quantitative manner. 9. EPIC deserves credit for focusing on the subject of Labor Market Studies, while other policy research institutions in Egypt (such as ECES or DEPRA) have focused more EPIC/ICEG Evaluation iii LW Draft, 04/30/04 elsewhere, particularly finance, liquidity crises, capital markets and trade policies, partly leaving a gap in the area of labor studies which EPIC has helped fill. 10. In summary, EPIC research activities have had mixed success, due partly to design flaws in the original project document which set ambitious goals and resulted in activities that were spread too thinly, and which did not set clear, detailed definitions of roles and responsibilities for EPIC/ICEG management. C. Research Dissemination 11. EPIC made strong efforts to disseminate research activities upon completion of research, but results were mixed. A considerable number of studies were delayed until the third and fourth years of the project period, delaying public perception of EPIC and allowing little time to gain dissemination experience. Moreover, despite its best efforts, EPIC had mixed effectiveness in influencing policy-makers, as some of its output was reportedly considered to be overly academic or not sufficiently concise and practical for some policy-makers. D. Growth Fellows Program 12. There was a failure to meet the goal of providing post-doctoral training to Growth Fellows, and subsequently placing them in Government ministries and other institutions to support senior policy makers. EPIC’s role was limited to helping select seven candidates, of whom five were sent. Four were placed upon their return, but none in government ministries as advisors. E. Staff Training for Ministries 13. EPIC provided practical training to Ministerial staff generally in short-term, courses in English and technical language training, computer training and specific technical training on National Accounts implementation and on Capital Markets. Training appeared useful to participants and well received. But almost all training was offered only during 1998, involved mainly one-time short courses, and was provided to between 150 and about 200 participants, indicating that more could have been accomplished under this component with available funding. F. Institutional Capacity Building, Technical Assistance, Special Projects 14. EPIC’s Doctoral Training support has been conducted relatively smoothly and successfully. More candidates than originally proposed have been supported. The return rate appears to be quite low and more than acceptable. A number of observers note that given the relative success of the program, it is highly unfortunate that the Doctoral program budget was cut. No additional candidates have been sent since late 1999. Given this, it is particularly important that monitoring and follow-up of current candidates be continued in an effective manner. 15. EPIC grants for TA to help build institutional capacity appear to have been generally successful in providing practical support, particularly to small provincial universities, for improved library resources, office equipment, journal subscriptions, translations and computer/internet use. Most observers indicate that it would be unfortunate if a practical, useful program of this nature were not to continue in some form in the future, particularly as many basic needs for TA to provincial educational institutions remain largely unmet. EPIC/ICEG Evaluation iv LW Draft, 04/30/04 III. EVALUATION CONCLUSIONS, LESSONS LEARNED 16. Overall Performance: The EPIC program has made a positive contribution toward strengthening Egyptian economic research institutions, but due partly to the extent of needs, has not yet achieved the initial project vision of creating a strong, vibrant consortium whose members cooperate closely and effectively to coordinate research priorities in support of Egypt’s economic reform program. 17. Quantitative Target Performance: In meeting goals and targets for output set out in the original project document, ICEG and EPIC have had a generally successful record, but one that includes both project successes and areas where performance fell short of targets. 18. Organizational Structure and Relationships: Achievement of goals and targeted output which started slowly and improved during the second half of the project, were adversely affected by several organizational constraints: Project design tended to be ambiguous regarding organizational and managerial roles which were not adequately defined in the initial project design. This contributed toward organizational and institutional relationships that had an adverse impact on performance mainly during the first phase of the project. Partly as a result, EPIC shifted its focus from encouraging cooperation among consortium members, toward more unilateral, direct support to EPIC members and other institutions. Lessons Learned In retrospect, EPIC’s areas of achievement, and activities where it fell short, point to a number of areas in which it might have done more, such as: • Better coordinate research subjects with other institutions, particularly in “popular” fields such as finance, where several other research institutions are highly active • Liaise more with Government agencies to determine their priority research needs • Minimize costs of major long-term research projects, partly by making grants to fewer full￾time, already established professors, and more to junior researchers • Focus more on short-term, concise policy studies directly supporting topical government research priorities, and less on large scale research studies which can often be conducted more effectively by the major established institutions such as ECES • Support future MOE activities especially in exchange rate issues and trade enhancement, Continue support for the Monthly Economic Digest, the website and similar research • Translate more publications into Arabic and vice-versa • Minimize lengthy studies or delayed reports, some are still “in progress” and may be disseminated too late to be used as relevant input for policy-makers (e.g., mortgage reform) • Focus on less theoretical and more practical studies and functional to policymakers, (e.g., DEPRA studies on port reform) • Ensure that statistical data supported are consistent with other government agencies (eg, Monthly Economic Digest of the Ministry of Economy may not be consistent with CAPMAS • Provide more short-term training to Ministries • Organize more practical smaller seminars rather than large academic conferences • Ensure more effective, hands-on monitoring and evaluation by USAID, and encourage more supportive relationships between contractors and technical officers EPIC / ICEG Evaluation 1 LW Draft, 04/30/04 ACTIVITY EVALUATION - November 29, 2000 - Economic Policy Initiative Consortium (EPIC) administered by the International Center for Economic Growth (ICEG) I. BACKGROUND AND PURPOSE OF THE EVALUATION A. Background The U.S. Agency for International Development (USAID/Egypt) entered into Cooperative Agreement # 263-0233-A-00-7001-00 with the International Center for Economic Growth (ICEG) effective October 20, 1996 for a four-year period. The purpose of the agreement was for ICEG to implement a set of economic research activities mainly through the establishment of an informal consortium of research institutions in Egypt to be called the Economic Policy Initiative Consortium (EPIC). EPIC's main objective was to support indigenous research to assist the government in its reform initiative To achieve its objectives, EPIC has conducted a wide range of technical support and capacity￾building programs. They include support of policy oriented research projects, dissemination of policy research, technical advice to ministries, institutional development, training for economic ministries' staff: specialized training on economic reporting, and overseas scholarships in the field of economics. EPIC was established under the First Subcommittee of the US-Egypt Partnership for Economic Growth and Development, and operates under the sponsorship of the Ministry of International Cooperation, Government of Egypt. The program has been funded through the Cooperative Grant Agreement with USAID and administered by the ICEG, a U.S.-based non-profit organization active in supporting international development worldwide The Cooperative Agreement between USAID and ICEG was extended through January 2001 for administrative purposes. B. Purpose and Methodology The purpose of this evaluation is to describe, critically assess and document the effectiveness of sponsored activities. It describes in general terms the contribution of those activities to the Egyptian economic reform program. Given that this evaluation is descriptive and intended for purposes of documentation, this evaluation is not intended to provide a set of recommendations pertaining to prospective USAID support for this cooperative agreement. This report provides a descriptive evaluation of the project as distinct from an operational assessment of how the project has been implemented, or an assessment of its wider impact on beneficiaries. During the course of the project, monitoring and evaluation was limited to liaison by the USAID cognizant technical officer and others as part of normal duties, and included reviews of output based on quarterly and annual reports submitted to USAID. No formal mid-term evaluation, or other formal assessments were required in the original project document nor were any conducted prior to this evaluation. EPIC / ICEG Evaluation 2 LW Draft, 04/30/04 For purposes of this evaluation, questions addressed include: What have been the main activities and outputs of the project? Have the original project goals and targets been met? What have been the main constraints on meeting project goals, and, what are the main lessons to be learned for implementing similar future projects? Data for the impact assessment team was collected in three main ways: 1) through review of background literature related to ICEG/EPIC, (original project document and Scope of Work, Quarterly Progress Reports, Annual Reports, Work Plans, ICEG/USAID records, project output); 2) through com￾parisons of project output and policy studies to economic policy issues of top priority to USAID and to the GOE; and, 3) through personal interviews with ICEG/EPIC staff, USAID personnel, EPIC consortium members, Egyptian government officials and similar EPIC beneficiaries. The annex presents background material on activities and goals met for each component. II. PROJECT GOALS, ACTIVITIES AND IMPLEMENTATION EXPERIENCE A. Overall Project Goals The main goals of the EPIC activity were to strengthen the capacity of Egyptian economic policy institutes and scholars to provide research and technical support to the Egyptian government's economic reform initiative; and to expand public awareness of and support for economic policy reform in Egypt. As set out in the initial EPIC project document,1 its four main specific purposes were: 1. To create a community of Egyptian scholars and policy analysts committed to research and educational activities that will support the economic reform process in Egypt within the context of the Partnership; 2. To provide Egyptian policy makers with quality advice on issues and problems relating to economic reform in Egypt; 3. To increase awareness of and support for policy reform by all Egyptians, including policy makers and their staffs, and especially by larger circles of influence, such as business leaders, university professors, and the media; and 4. To strengthen policy research at non-governmental policy institutes (such as CEFRS) and at selected Egyptian universities, which will increase the value of the research and policy advice made available to the Egyptian government by Egyptian institutions and scholars. The Consortium’s initial members, as suggested in the original project document, included: the Center for Economic and Financial Research and Studies (CEFRS) at Cairo University; the Economics Department of Cairo University; the Economics Department of the American University in Cairo; the Economic Research Forum (ERF); and the Egyptian Center for Economic Studies (ECES). The project document stated that EPIC “will probably expand over the life of the project.” Although no other institutions formally joined the consortium, a wide range of additional educational institutions, mainly provincial Egyptian universities, received EPIC support through grants and technical assistance. 1 “Statement of Work”, Project to Mobilize and Strengthen Egyptian Institutions in Support of the Egyptian Government’s Economic Reform Initiative, ICEG, 1996. EPIC / ICEG Evaluation 3 LW Draft, 04/30/04 To help meet its goals, the initial Scope of Work described specific activities and outputs to be developed, (described in detail in the following sections), including: A. Research Program and Policy Papers (Selection of topics, EPIC establishment, governance, monitoring progress, workshops; Policy Advice/Briefing Papers (similar to commissioned research but more informal) B. Dissemination of policy research and policy dialogue (mapping principal audiences, Publications, seminars and special briefings, presentation of final research results, doing market research, training) and newsletters and related publications C. Growth Fellows (mapping the Ministries, conducting workshops) D. Staff Training for Ministries (language, computer, technical training) E. Institutional Capacity Building (Ph.D. Training, special institutional capacity building grants, and technical assistance for strategic planning, communication) In meeting the overall goals outlined above through implementing these services, ICEG and EPIC have had a successful record generally, with strong performance in some activities but mixed results in others. Assessing project performance in relation to EPIC’s initial broad objectives, beginning with the main rationale for undertaking EPIC, results in the following overall conclusions: 1. EPIC services have contributed toward strengthening the capabilities of Egyptian economic research institutions, but many institutions continue to have strong needs despite project achievements. As EPIC performance was generally positive, this indicates that the initial project document probably set overly ambitious goals. The original rationale for EPIC was appropriate, and remains so, given the continuing strong needs also on the part of Egyptian government policy-makers for improved economic policy research and analysis to help support Egypt’s policy reform program The rationale for EPIC was that many Egyptian institutions were inadequately equipped to produce relevant policy research, both qualitative and quantitative. The need for well-trained technocrats, giving internal technical support to the government's reform initiative, was strong. Developing the institutional structure of economic institutions and strengthening the professional skills of Egyptians involved in the policy debate, through research, teaching, and communication forums, were considered to be essential to help support policy reforms. EPIC's research, technical support, training, and institution￾building programs were intended to help meet such needs. But observers agree that strong needs for similar services continue, despite EPIC achievements, and perhaps more strongly now than before. This is true particularly as Egypt progresses in its economic reform program toward further, more specific trade and investment regime reforms as well as toward implementation of recent reforms. Research support will likely become increasingly necessary to help policy-makers analyze and respond to more rapidly changing economic conditions as Egypt’s evolving market-oriented policies move it further along the path toward greater integration into the multilateral trading system and global economic environment. 2. EPIC has had mixed success in meeting the original project document’s first overall goal “to create a community of Egyptian scholars and policy analysts… committed to support the reform process.” The implicit vision of creating a strong, vibrant consortium whose members cooperate closely and effectively to coordinate research priorities supporting reform, has not been achieved. However, ICEG does appear to have had more success in supporting certain consortium EPIC / ICEG Evaluation 4 LW Draft, 04/30/04 member institutions, individual Egyptian researchers, provincial universities and most recently the Ministry of Economy, in more direct, one-to-one relations, rather than through a consortium (see Conclusion No. 4) This original goal, in particular, may have been overly ambitious. The difficulty of creating a highly cooperative consortium is due partly to the inherent difficulty of an external, informal, institution attempting to help set research priorities and procedures for well￾established, independent-minded academic institutions. This would be a highly challenging goal in many if not most countries. The difficulty is also partly due to certain design flaws in EPIC’s management structure (discussed below) that did not clearly assign roles and authority for setting research priorities and for implementing activities. 3. EPIC has been partly successful in meeting the second and third overall goals, providing Egyptian policy makers with quality advice on economic reforms and increasing awareness of and support for policy reform by larger circles of influence, such as business leaders, university professors, and the media, through dissemination of research and training. Performance appears to have been mixed, in the sense that EPIC activities have gone farther toward meeting these goals during the second half of the project period than in the first part. First, there was less research output actually completed in the earlier period. Early research tended to be more academic and theoretical than practical and oriented toward policy-makers. At the start of the second period (late 1998), EPIC set more specific policy research priorities that more closely reflected USAID’s own revised priorities under its Development Support Program, including studies on finance and capital market, labor markets, and pension reform studies. Since early 1999, EPIC research priorities have in fact been more demand driven directly by government policy-makers, including officials at the Ministry of Economy. ICEG has cooperated increasingly with the Ministry of Economy, particularly to support statistical data analysis and presentation. The original project document specified that EPIC activities were to support “non￾governmental institutions”, (partly because other USAID projects did so). But extending additional support to MOE at its request seems highly valid, especially as it meets the other main goals of supporting policy-makers. ICEG now helps prepare the Ministry’s Monthly Economic Digest, and funds five staff members at the Ministry to work part time, about three days a month, to update monthly economic data (MOE maintains the related website). ICEG now also prepares Policy Briefs partly based on issues suggested by MOE , which checks them before distribution. ICEG also prepares the Ministry’s small Data Handbook, and annual larger Digest. EPIC has also tended to fulfill more of the goal of raising awareness of and support for policy reform during the second half of the project period. During this time, most research studies and policy papers were completed, released, and publicized through seminars and conferences, as well as published and distributed. As for the degree of public awareness actually raised, any quantitative estimate would depend on more intensive monitoring including conducting public opinion surveys, which is beyond the scope of this activity evaluation. 4. EPIC appears to have gone far toward meeting the fourth goal of strengthening policy research by working more on a unilateral, one-to-one, basis with most consortium members and individual researchers rather than through the unified “consortium” approach, through providing research grants, dissemination support and in some cases technical assistance. This was due largely to a shift in management staff and roles about mid-way through the EPIC project. EPIC / ICEG Evaluation 5 LW Draft, 04/30/04 In brief, EPIC implementation was divided into two phases in terms of management activities and priorities, covering more or less the first two years and the second two years. The first phase lasted generally from start-up to about end-1998. Most observers interviewed tend to agree that some confusion existed over the role and authority of the Board of Directors (consisting mainly of the consortium institutional members) on one hand, and of ICEG and EPIC management on the other. This appears to have been due partly to a relatively vague definition of management roles in EPIC’s initial Scope of Work. Apparently, some members of the Board representing consortium member institutions viewed the role of EPIC as one in which it should fund research that each consortium member wanted to conduct, or was already planning to conduct, with minimal interference. ICEG managers of EPIC, on the other hand, viewed EPIC’s role as being more involved in setting priority topics for research and ensuring quality. Board members apparently acted less as an advisory board dealing mainly with overall policies and more as an executive board, actively making decisions on daily operating issues. 5. Confusion over management roles had an adverse impact on EPIC operations over the start-up period and into the second year of operations, in terms of delayed implementation of research projects, policy papers published and related services. Contentious issues included questions of: whether to staff EPIC with local economists or expatriates; how much major research was to be conducted in-house; whether and how the consortium should expand; funding questions over whether members should receive grants directly from USAID or through EPIC; and generally, the role and authority of ICEG management as opposed to Board representatives of the consortium member institutions. Consortium members varied in how actively they participated in EPIC activities and cooperated with other EPIC members. CEFRS reportedly was among the most cooperative members with ICEG, and participated actively in building its institutional capabilities through grants for technical assistance rather than for research. Grants were made for training, hiring staff, and for seminars, and CEFRS also was successful in raising considerable matching funds. Initially the Economic Research Foundation was an active participant in conducting research projects, but less so after the first two years. A.U.C. after the first year was relatively inactive, along with ECES, which had little active involvement after the first period. Before the transition there was a relatively limited amount of research conducted, including two major research studies (Egyptian Capital Markets, Poverty Profile). About 90% of output was in policy papers. Publication of the quarterly “Policy Briefs” were delayed beyond the first year, and did not start until January 1999, produced in-house by EPIC staff rather than by consortium members. 6. Many targets for output were met relatively recently, closer to the end of the project period. This appears to have been partly due to delays in completing some research studies, although the “gestation” period required for major studies means that completion naturally would come later rather than earlier in the project. Improved project administration in the second period may also have contributed to overcoming delays and speeding completion of activities. Prior to the transition in late 1998, only two major research studies had been completed, one on Egyptian capital markets, and one covering on labor markets and poverty issues. Since then, completed research studies included eight dealing with capital and financial markets; nine dealing with labor markets and three on pension reform. Policy papers, which tended to be more concise and less comprehensive than the major research studies were, however, completed mainly during the first phase (15 of 22). EPIC / ICEG Evaluation 6 LW Draft, 04/30/04 The second management phase was marked by a shift away from full and active participation of consortium members, as ICEG has restructured its research to work more with a few of the consortium members on an individual basis, primarily CEFRS and to an extent, ERF and other individual grantees. Also, EPIC has developed a “new” client in terms of the Ministry of Economy which has shown a strong demand for EPIC services in support particularly of its economic statistics and data analysis presentation Since early 1999, EPIC has conducted only a few activities with the original consortium members, including only one major research project, a Labor Force Sample Survey with ERF. The original Board of Directors has, primarily on its own accord, stopped meeting, in early to mid-1999. Instead, EPIC staff since about 1998/99 has included two part-time research “directors” who as professional economists, have helped direct research grants. Grants have been focused on research in the three major functional subject areas which EPIC adopted as priorities following the management transition: Financial and Capital Markets; Labor Markets; and Pension Reform studies. Much of EPIC’s research priorities are now set with the direct input of the Ministry of Economics, reflecting their major interests and needs. The main focus on issues now is: Banking reform, privatization, bond markets, mortgage reform; exchange rate flexibility; pension reform, privately funded systems, and notional accounts; capital markets development including training seminars for CASE staff and journalists. These areas also tend to reflect major economic reform priorities of USAID, particularly since the shift toward its more recent Development Support Program. B. Research Program and Policy Papers One of the four main objectives of EPIC as described in the project document was: “To strengthen policy research at non-governmental Egyptian policy institutes such as the Center for Economic and Financial Research and Studies and at selected Egyptian universities, which will increase the value of the research and policy advice made available to the Egyptian government by Egyptian institutions and scholars. This objective will be implemented by both short-term measures that increase current outputs and also…long-term measures that will increase the talent pool of highly trained Egyptian economists.”2 This objective was the basis for achieving the other main objectives, which included creating a community of scholars and researchers; providing policy-makers with advice on economic reform; and increasing public awareness of and support for policy reform. EPIC made grants mainly to individuals rather than to institutions for research projects (EPIC’s main substantive research program) and policy papers (involving less primary research). The only institution to receive a grant was the ERF for a Labor Market study. EPIC’s selection system for grants to individuals tended to be relatively informal. It did not formally solicit or commission research publicly except for a few large projects. Rather, it tended to rely on a network of professionals and their contacts among researchers in generating lists of candidates and sought suggestions from Ministries, other institutions and individuals. In the context of a relatively small and restricted number of universities and researchers in Egypt, as in many developing countries, this may have been as effective a system as possible for identifying candidates, given the environment. 2 “Statement of Work” , Project to Mobilize and Strengthen Egyptian Institutions in Support of the Egyptian Government’s Economic Reform Initiative, ICEG, 1996. EPIC / ICEG Evaluation 7 LW Draft, 04/30/04 EPIC major research output included major research studies , and policy papers, (see complete list by title in Appendix; see the section below on Dissemination for quarterly and monthly periodicals, monthly statistical data updates and related publications) EPIC research output included the following through November 2000: Research Studies: 22 research studies commissioned, of which 15 are completed • Financial and Capital Markets: 9 research studies commissioned, of which 3 are ongoing • Labor Markets: 10 studies commissioned, 2 ongoing • Pension Reform: 3 studies commissioned, 2 ongoing • Institutions associated with individual authors: - Cairo University - American University in Cairo (A.U.C.) - E.R.F. - Ain Shams University - Tanta University - National Planning Institute - Ministry of Economy - Central Bank of Egypt - Kent University (Egyptian researcher) - U.C.L.A. - Toronto University - University of Minnesota (Egyptian researcher) Policy Papers: 22 papers commissioned, of which 17 are completed • Financial and Capital Markets, 8 papers commissioned, 1 ongoing • Trade and general economic issues: 14 papers commissioned, 4 ongoing • Policy papers were presented and discussed in a series of monthly seminars on each paper, organized with Cairo University • Institutions associated with individual authors: - Cairo University - E.R.F. - Alexandria University - Suez Canal University - Al-Ahram Center for Strategic Studies - Zagazig University - G.A.F.I. - Helwan University - Sadat Academy - CAPMAS 7. ICEG and EPIC have conducted a substantive research policy program over the course of the project period and have met the initial project targets for commissioned research and for policy papers and other publications in terms of the number of research projects and the range of subjects originally suggested. However, there has been mixed success in meeting certain goals, such as completion targets that were not always met on time, and the resulting lack of adequate public outreach, especially in the first management period. EPIC / ICEG Evaluation 8 LW Draft, 04/30/04 Over the project period a total of 22 research studies have been commissioned under EPIC’s research program, (complete List in Appendix) of which 15 have been completed and 7 remain ongoing. The original project document suggested that a total of 16 research projects be initiated, four each year, and that a total of 8 be completed by the end of the fourth project year. Both goals have been considerably exceeded. In the project’s first two years, only two studies were completed, one each in 1997 and in 1998. Eleven studies were completed in 1999, and 3 in 2000 through October, (most ongoing studies are expected to be completed by late 2000 or early 2001). The relatively slow start has been criticized by some observers, but may be understandable given the inherently lengthy process of commissioning, conducting, reviewing and publishing research studies. In fact, EPIC studies were completed more quickly than initially envisioned in the project document, which expected none to be completed in the first year, two the second year, and three each in the third and fourth years. In terms of subjects covered, research commissioned has focused on three major subject areas: a. Finance and capital markets (9 studies); b. Labor markets and poverty studies (10) and c. Pension reform studies (3). Within each general area a diverse range of sub-topics have been covered. This compares favorably both with the many of the suggested subject areas of the project document (financial system, exchange rate regime, labor laws and practices, pension funds). EPIC Policy Papers have focused both on financial subjects, as well as on additional topics suggested initially, covering general trade and economic issues. Originally suggested topics were trade liberalization, privatization, and regulatory topics including environmental issues, health and safety requirements, and procedures for business establishment and operation. Main topics in Policy Papers have included finance and capital markets, particularly stock exchange experience, exchange-rate issues, and to a lesser extent, trade policies and foreign direct investment issues. 8. EPIC research subject areas have generally matched the major policy priorities of USAID, and, to a lesser extent, those of the government of Egypt. However, the direct impact of EPIC sponsored policy research on Egyptian policy reform is not possible to measure credibly in a quantitative manner USAID’s policy reform initiative tended to be relatively broad up until the late 1990s, covering a wide range of trade, investment, financial and other regulatory issues. Some were tied to USAID’s cash transfer program while others were not. Due in part to the ambitious nature of USAID goals, implementation of reforms tended to be uneven with considerable delays. Since then, USAID has revised its policy priorities, under its Development Support Program. Many now tend to be more focused, involving mainly cash transfer programs related to policy reform areas such as financial and capital markets development, trade issues, and pension reform, which generally match the main subjects on which EPIC has focused. The extent of credit for actual policy reform that any Egyptian policy advocacy institution can be granted is highly debatable, given the lack of an adequate system to monitor and evaluate AID-supported and other policy advocacy initiatives. A credible and rigorous system would need to include both quantitative and qualitative measurement, preferably an index comparing specific policy changes with advocacy group areas of focus. It would also require input from panels of functional experts meeting regularly to rate the extent of policy change and the general level of credit with which advocacy institution could credibly be granted. Without a substantive monitoring system, it will remain difficult to measure credibly the impact of policy research and advocacy on policy reform even in general terms. EPIC / ICEG Evaluation 9 LW Draft, 04/30/04 On the positive side, EPIC research has been conducted in several key policy areas that are priorities to the Government, such as financial development and pension systems. EPIC also conducted a targeted program of disseminating research through conferences, seminars, press conferences and distribution (see below). Less positively, EPIC’s research program has tended to focus on more academic analysis conducted over relatively long-time periods, more so than other institutions (ECES) or private sector business associations (AmCham, FEI). EPIC has focused less on practical, concise policy analysis aimed directly at supporting key government policy-makers. More recently (since early 1999) EPIC has moved more toward this role with its quarterly Policy Briefs and particularly support for statistical analysis to the Ministry of Economy’s and its Monthly Economic Digest. Regarding qualitative measurement of EPIC research, it is not the scope of this review to conduct an in-depth analysis of the functional content of all EPIC studies. In general, EPIC research studies appear to be on the whole adequate. However, in terms of quality, they cover a wide range, with some better researched, presented more professionally, and with recommendations made more effectively than others. 9. EPIC deserves credit for focusing on the subject of Labor Market Studies, while other policy research institutions in Egypt (such as ECES or DEPRA) have focused more elsewhere, particularly, finance, liquidity crises, capital markets and trade policies, partly leaving a gap in the area of labor studies which EPIC has helped fill. Labor, employment, household income and poverty studies, are subjects that have become recognized increasingly both by USAID and by the Government of Egypt as particularly critical areas for research and policy reform. Some observers have been critical of EPIC and other AID-supported institutions for failing to coordinate research adequately and for duplication of effort, On the contrary, this is an example of a productive division of labor, and is a subject area in which EPIC has made, and could continue to make, a valuable contribution. Of course, coordination could also have been improved in the other subject areas as well. 10. In summary, EPIC research activities have had mixed success, due partly to design flaws in the original project document which set ambitious goals and resulted in activities that were spread too thinly, and which did not set clear, detailed definitions of roles and responsibilities for EPIC/ICEG management. EPIC’s originally intended focus was on helping to build a broad foundation for developing an effective community of economic researchers and academic institutions in Egypt over the long-term. EPIC was not designed to focus on one or two key activities such as direct policy reform support of policy advocacy. Its broad approach tended to involve it in perhaps too many activities with too few resources, and with uneven results. Overall performance, particularly meeting start-up goals, also suffered in part from confusion over management roles, described above. C. Research Dissemination The purpose for disseminating research studies set out in the project document was: “To increase public awareness of and support for policy reform by all Egyptians, including policymakers and their staffs, and especially by larger circles of influence, such as business leaders, university professors, and the media;”3 3 “Statement of Work , p. 4, ICEG, 1996. EPIC / ICEG Evaluation 10 LW Draft, 04/30/04 The original project document clearly stressed the importance of research dissemination, stating that “the project’s major outputs will be achieved through communication and education… (even) high quality research activities will be useless if they are not disseminated to people who can act on them”4 . EPIC undertook solid efforts at dissemination, and hired a full-time Communications Director as suggested in the project document, although this was delayed during the confusion over management roles in the first phase. The Director focused on two main activities, research dissemination, and networking among provincial universities to identify needs for technical assistance (see section on T.A. below). Activities included dissemination of research to all major audiences, including relevant government ministries, academic institutions and the wider community of researchers, to the private sector mainly through business associations, and to the Egyptian media. 11. EPIC made strong efforts to disseminate research activities upon completion of research, but results were mixed. A considerable number of studies were delayed until the third and fourth years of the project period, delaying public perception of EPIC and allowing little time to gain dissemination experience. Moreover, despite its best efforts, EPIC had mixed effectiveness in influencing policy-makers, as some of its output was reportedly considered to be overly academic or not sufficiently concise and practical for some policy-makers. EPIC developed useful ties with key media groups, organized generally effective seminars and conferences (particularly on Labor and Finance topics), and adequately distributed research to relevant government ministries. However, perhaps more efforts could have been made to reach policy-makers by providing more concise summaries, translating more studies, or even providing brief, graphic presentations to Ministries. On the other hand, delays in output made for a relatively short time period during which EPIC has been active in PR. EPIC did not have the chance to accumulate experience in dissemination that larger, well-established think-tanks have developed. Also, EPIC’s original goals were ambitious in aiming for strong direct impact on policy-makers and economic reforms. EPIC held half- or full-day conferences to announce newly released research publications. Most were held at Cairo University and were attended by about 40-50 invited professors, journalists, and interested ministry staff and some private sector representatives. Attendance varied depending on topics of interest, with the most popular being studies on financial topics judging by attendance. Government officials tended to show stronger interest in financial issues including capital markets development, central bank issues, exchange rates, domestic debt, while private firms were apparently attracted more by trade issues and sectoral studies. EPIC also networked with newspapers; held press conferences, and conducted a limited amount of short-term training courses for journalists on economic issues. Copies of all studies were sent to relevant government ministries. Dissemination activities varied among other research institutions. ECES generally sends all its publications to all major journalists. ERF does publicize some studies, but not as much as ECES. DEPRA did not do as much in terms of public relations or have close press relations, partly because of less translation into Arabic (most EPIC publications were in Arabic). 4 Ibid. p. 10. EPIC / ICEG Evaluation 11 LW Draft, 04/30/04 EPIC organized two major conferences, but not until well into its second management phase in late 1999 and early 2000. In October 1999 it held a two-day conference on “Labor Markets and Human Resource Development in Egypt”, using nine research stiudies in the field as the basis for individual sessions, with papers presented by authors followed by critiques led by discussants. This conference was relatively well attended, with over 150 participants, and was generally considered to be successful. The second major EPIC conference was held over two days in May 2000 on “Financial Development and Conpetition Policies in Egypt” and was co-hosted by EFG-Hermes. EPIC financial studies were the basis for presentations. While well-attended by some 200 participatns, and generally well-received, some dissatifaction was reported that the program was overly academic. Judging by press reports, the conference received relatively wide and favorable media attention. Additional activities related to dissemination included: • EPIC Newsletters, quarterly, first issue January 1998, English and Arabic • EPIC Policy Briefs, every six weeks, first issue January 1999, English and Arabic, about 1,500 copies each issue are distributed • Monthly Economic Digest, Ministry of Economy and Trade, monthly, first issue March 1999 and electronic edition on the website of the Ministry, updated monthly • Egypt Annual Statistical Book, 1999, 2000 • Euromoney Egypt Book, for distribution at the Euromoney Conference, September 2000 • Distinguished Speakers Series”: Public lectures delivered by five foreign specialists, from universities and research institutions, including UCLA, Infosafe Systems Inc. based in the U.S., the former Foreign Affairs Minister of Poland, the former Minister of Commerce of Thailand, and a representative of U.S.-based Basis, Inc, specialized in internet/web design. • Linking Egyptian Institutions on the Web: A program to help Egyptian institutes develop web sites and to provide training on their use and development D. Growth Fellows Program EPIC supported two major kinds of training programs. The first was the “Growth Fellows” or Overseas Long-Term Degree Scholarship Program, under which candidates were to be selected for post￾doctoral training of up to one-year in the U.S. and upon completion were to be placed as special special advisors to Egyptian economic Ministers and other policymakers who agreed to take them. The second was a program under which candidates were to be selected for PhD. Programs in economic/business fields of up to four or five years in the U.S. As this program was initially placed under EPIC’s Institutional Development component, it is described in the later section below. The purpose of the Growth Fellows Program as stated in the project document was: “To increase exposure to modern economic ideas and advanced research methods (by selecting candidates)…for a year of training on economics and policy reform in the U.S. When they return, some will serve as special consultants for 2-3 years to Ministers and other top officials, as assistants to key policy researchers, or as special consultants to businesses. Those who do not serve in these special capacities will return to their home research institutes or universities.”5 5 “Statement of Work” , p. 4 ICEG, 1996. EPIC / ICEG Evaluation 12 LW Draft, 04/30/04 This program was considered to be one of the most important activities of the project. It was listed first among all components in the original project document, in fact, before the description of EPIC itself, and described in considerable detail. It suggested careful preparation by “Mapping the Ministries” to determine needs and interests to market the advisors to Ministers, and holding workshops to prepare candidates before and after training. The concept was key to the project’s major purposes of directly supporting Egypt’s economic policy reform program. The initial work plan called for selecting, preparing and sending the first group of Fellows to the U.S. in mid-1997. The number of candidates was not specified and was left up to available funding. Additional groups were to depart in each of the subsequent project years. Actual experience in selecting and sending Growth Fellows was as follows: • In 1997 – two candidates were selected and sent - one returned, and was placed in GAFI; the other subsequently left Egypt and so could not be placed; • In 1998 – five candidates were selected, and three were sent - all three returned; - one was placed at Al Azhar University, one at El Giza Research and Development Center, a private business, and one at “Central Auditing Organization” • In 1999 – None were selected nor sent as program funding was cut by USAID as part of a Mission-wide cutback program in early 1999 12. There was a failure to meet the goal of providing post-doctoral training to Growth Fellows, and subsequently placing them in Government ministries and other institutions to support senior policy makers. EPIC’s role was limited to helping select seven candidates, of whom five were sent. Four were placed upon their return, but none was placed in government ministries to be advisors to policy-makers. There are several reasons for the failure of this program. Initial project design was partly flawed. The groundwork for selecting appropriate candidates who would be acceptable for placement in Ministries was neglected by EPIC. And there was insufficient follow-up and monitoring of candidates by DTII/IIE, whose main responsibility was funding the training. Design flaws included insufficient attention given to defining specific requirements for both selecting and preparing candidates, for identifying Ministers’ needs and desires for assistants, and for monitoring and follow-up of candidates on their return. One year training may be too long for post￾graduates in full-time employment and it may have been more realistic to support shorter training of not more than three or four months. Technically, EPIC was responsible only for selecting candidates. DTII/IIE was responsible for funding and general follow-up (usually limited to a re-entry visit, and one additional visit to measure impact.) However, both EPIC and DTII appear to have carried out their responsibilities more according to the letter, rather than the full spirit, of the project. If both had been required to devote more effort to the program than was initially proposed, the chances of success would probably have been greater. E. Staff Training for Ministries This component was initially described in the project document as providing “short - term, in-country training for key staff of ministries on computers, software, and similar kinds of things.” EPIC / ICEG Evaluation 13 LW Draft, 04/30/04 As such, its scope was smaller than most other project components in terms of level of effort and funding. No quantitative targets or indicators were specified in the initial Scope of Work, other than stating that training should begin in the first year and continue to be provided each year. EPIC funded nine training courses mainly in basic and advance English language training, English training in economics terminology and in finance terminology. In computer skills, in language and technical training for Capital Markets Authority staff, and a training program in implementation techniques of the National Accounts System. Most training course were short-term, with a duration of from a few weeks up to two or three months, and were provided generally for one time only (some advanced language training was requested, and four consecutive training programs were provided in capital markets technical training). Training institutions included mainly A.U.C. (6 out of 9 training courses) as well as the Information and Decision Support Centre (IDSC) and individual specialists of capital markets training. Only one training course was provided in 1997; all others were provided in 1998, and none have been offered since. Ministries receiving training, specific courses, and number of participants were: 1. Ministry of Planning (language training, 40 participants) 2. Ministry of Planning (advanced language training, 19 participants) 3. Ministry of Planning, of Economy, the Central Bank and CAPMAS (Implementation of the 1993 National Accounts System, 40 participants) 4. Ministry of Economy, (language training, 21 participants) 5. Ministry of Economy (computer skills training, 21 participants 6. Ministries of Planning, of Commerce and Supply, the Central Bank, CMA (Engligh economics terminology, 19 participants) 7. Various Ministries, Central Bank (English finance terminology) 8. Capital Markets Authority (English financial terminology training) 9. Capital markets Authority (four consecutive programs) 13. EPIC provided practical training to Ministerial staff generally in short-term, courses in English and technical language training, computer training and specific technical training on National Accounts implementation and on Capital Markets. Training was apparently useful to participants and well received. But almost all training was offered only during 1998, involved mainly one-time short courses, and was provided to between 150 and about 200 participants, indicating that more could have been accomplished under this component with available funding. F. Institutional Capacity Building, Technical Assistance, Special Projects The first of three activities proposed under the project’s component for institutional capacity building was the Doctoral Training Program. The rationale for this activity was that “…Egypt needs well-trained PhDs who can provide a base of technical support to sustain and deepen the commitment to economic reform.” 6 6 Statement of Work” , p. 13, ICEG, 1996. EPIC / ICEG Evaluation 14 LW Draft, 04/30/04 The initial scope of work called for five groups of students to be sent to the U.S. - four students in the first group, and three in each of the next four years, for a total of 16 Ph.D candidates. Selection would be done mainly by EPIC members and monitoring and funding by DTII/IIE (the average cost of a doctoral program is $145,000). Actual numbers of candidates selected and sent exceeded initial proposals, amounting to 26 over the period 1997 – 1999: Year No. Sent Returned early Completion date ’97 8 - ’02 ’98 11 2 ’03 ’99 7 - ‘03 No PhD. candidates have had time yet to complete their program. Of the two candidates that returned without completing the program, one apparently did not adjust well to a university in Washington D.C., and returned for personal reasons. The other failed qualifying exams. In IIE’s experience, it is not uncommon for candidates to return early due to the lack of adequate academic background and technical preparation. Overall, however, selection criteria appear to have been applied effectively, given the low return rate and the fact that other programs conducted by DTII (e.g., in public health) have had a higher return rate. 14. EPIC’s Doctoral Training support has been conducted relatively smoothly and successfully. More candidates than originally proposed have been supported. The return rate appears to be quite low and more than acceptable. A number of observers note that given the relative success of the program, it is highly unfortunate that the Doctoral program budget was cut. No additional candidates have been sent since late 1999. Given this, it is particularly important that monitoring and follow-up of current candidates be continued in an effective manner. The second activity under Institutional Development was the program of “Special Institutional Capacity Building Grants” mainly technical assistance. A third activity was grants for “special projects” such as strategic planning and fundraising training which were also administered under the grants below. Most TA under these grants was provided for library support (maintaining journal subscriptions) supporting basic staff at research institutions, and support for computer use. Grants have been made to the following institutions: 1. Cairo University, Faculty of Economics and Political Science 2. Cairo University, Center for Economics and Financial Research and Studies (CEFRS), This was a major multi-year grant proposed in the initial project document, representing a major contribution of EPIC under this component. CEFRS priority issues include poverty, labor and unemployment. CEFRS appears to have made practical use of the grants (of $150,000 for two years) to build up library resources, staff and computer capabilities. 3. The Economic Research Forum 4. Institutional Support for Departments of Economic in Egyptian Universities - This program was successfully extended to support over ten Egyptian universities, mainly provincial and smaller institutions, including the Universities of Zagazig, Ismailia, Assiut, Suez Canal, Tantra, and Alexandria. Funding is not for actual research, as capabilities are limited, but to provide a foundation for later research. Grants of up to $15,000 provide TA for library improvement, on-line connections, computer equipment, office equipment; staff salaries, training of researchers who are grad students, translations, web/ distance education and publishing of research. EPIC / ICEG Evaluation 15 LW Draft, 04/30/04 5. ECES technical Assistance, in 1996 supporting a proposal for developing an endowment. (later grants remain unused, and were eventually returned) 6. CEFRS Technical Assistance, to develop a strategic and fund-raising plan, interrupted and then resumed. CEFRS has since pursued and won matching grants from other sources. 7. Fundraising Strategies Workshops, in 1999, provided to participants including directors of ECES, ERF, CEFRS, IDSC, PARC and others Most grants for technical assistance appear to have been put to practical use. Not all grants, however, were utilized, including a grant to ECES, and a $150,000 grant to AUC for setting up a center which was not used. 15. EPIC grants for TA to help build institutional capacity appear to have been generally successful in providing practical support, particularly to small provincial universities, for improved library resources, office equipment, journal subscriptions, translations and computer/internet use. Most observers indicate that it would be unfortunate if a practical, useful program of this nature were not to continue in some form in the future, particularly as many basic needs for TA to provincial educational institutions remain largely unmet. III. EVALUATION CONCLUSIONS, LESSONS LEARNED 16. Overall Performance: The EPIC program has made a positive contribution toward strengthening Egyptian economic research institutions, but due partly to the extent of needs, has not yet achieved the initial project vision of creating a strong, vibrant consortium whose members cooperate closely and effectively to coordinate research priorities in support of Egypt’s economic reform program. It is unlikely that this vision could be achieved to a significant degree during the relatively short four-year project period. Many institutions continue to have strong needs despite project achievements. EPIC’s generally positive performance indicates that the initial project document probably set overly ambitious goals. Overall achievements were constrained mainly by two additional factors. First, EPIC research activities had mixed success due partly to design flaws in the original project document which set a very broad range of activities and resulted in efforts that were at times spread too thinly. Secondly, detailed definitions of roles and responsibilities for EPIC/ICEG management were not defined as clearly as they should have been at the beginning of the project, resulting in a relatively slow start, with most output and goals met during the second half of the project. 17. Quantitative Target Performance: In meeting goals and targets for output set out in the original project document, ICEG and EPIC have had a generally successful record, but one that includes both project successes and areas where performance fell short of targets. Activities where performance was generally successful include setting appropriate priorities for research topics, moving toward direct support of government policy-makers especially in the Ministry of Economy, and selecting candidates for the long-term Ph.D. program that have been successful to date. EPIC / ICEG Evaluation 16 LW Draft, 04/30/04 Areas where performance has been mixed include delays in completing research studies, more so in the first phase of the project period, and more importantly, the failure to adequately implement the Growth Fellows program and place returning researchers in key positions to support policy-makers. 18. Organizational Structure and Relationships: Achievement of goals and targeted output which started slowly and improved during the second half of the project, were adversely affected by several organizational constraints: Project design tended to be ambiguous regarding organizational and managerial roles which were not adequately defined in the initial project design. This contributed toward organizational and institutional relationships that had an adverse impact on performance mainly during the first phase of the project. Partly as a result, EPIC shifted its focus from encouraging cooperation among consortium members, toward more unilateral, direct support to EPIC members and other institutions. Lessons Learned In retrospect, EPIC’s areas of achievement, and activities where it fell short, point to a number of areas in which it might have done more, such as: • Better coordinate research subjects with other institutions, particularly in “popular” fields such as finance, where several other research institutions are highly active • Liaise more with Government agencies to determine their priority research needs • Minimize costs of major long-term research projects, partly by making grants to fewer full￾time, already established professors, and more to junior researchers • Focus more on short-term, concise policy studies directly supporting topical government research priorities, and less on large scale research studies which can often be conducted more effectively by the major established institutions such as ECES • Support future MOE activities especially in exchange rate issues and trade enhancement, Continue support for the Monthly Economic Digest, the website and similar research • Translate more publications into Arabic and vice-versa • Minimize lengthy studies or delayed reports (e.g., many are still “in progress”) which may be disseminated too late to be used as relevant input for policy-makers (e.g., mortgage reform) • Focus on less theoretical and more practical studies and functional to policymakers, (e.g., DEPRA studies on port reform) • Ensure that statistical data supported are consistent with other government agencies (eg, Monthly Economic Digest of the Ministry of Economy data are not always consistent with CAPMAS data) • Provide more short-term training to Ministries • Organize more practical smaller seminars rather than large academic conferences • Ensure more effective, hands-on monitoring and evaluation by USAID, and encourage more supportive relationships between contractors and technical officers