TABLE OF CONTENTS TABLE OF CONTENTS...........................................................................................................................2 ACRONYMS & ABBREVIATIONS...........................................................................................................3 1. EXECUTIVE SUMMARY .................................................................................................................1 2. EVALUATION AND ASSESSMENT QUESTIONS .........................................................................5 3. KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS: SUMMARY ..............20 ANNEXES..............................................................................................................................................22 Annex 1: Sources of Information from Field Research ....................................................................22 Annex 2: Data Collection Instruments for KIIs and FGDs.................................................................25 Annex 3: Survey Results ...................................................................................................................43 ACRONYMS & ABBREVIATIONS CYPRESS Capacity-Performance-Results-Sustainability SG Secretary General DLS Department of Lands and Survey EU European Union FGD Focus Group Discussion FRPFM GBD USAID/Jordan Fiscal Reform and Public Financial Management Project General Budget Department GCC Gulf Cooperation Council Countries GFMIS Government Financial Management Information System GOJ Government of Jordan GRB Gender-Responsive Budgeting GST General Sales Tax IMF International Monetary Fund IPSAS International Public-Sector Accounting Standards ISTD Income and Sales Tax Department KII Key Informant Interview KOICA Korean International Cooperation Agency MELA USAID/Jordan Monitoring, Evaluation, and Learning Activity MOF Ministry of Finance MTFF Medium-Term Financial Framework PAYE Pay-as-you-earn tax payments PEFA Public Expenditure and Financial Accountability PFM Public Financial Management PPP Public-private Partnership RMS Revenue Management System ROB Result-Oriented Budgeting SEPD Studies and Economic Policy Directorate TOT Training of Trainers TPU Tax Policy Unit USAID United States Agency for International Development 1 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV JORDAN MONITORING, EVALUATION, AND LEARNING ACTIVITY (MELA) Phase 2 Evaluation Fieldwork: Assessment of Current Context and the Government of Jordan’s Capacity1 1. EXECUTIVE SUMMARY This is the Phase 2 Report for USAID Jordan’s external evaluation of its Fiscal Reform and Public Financial Management (FRPFM) project. This evaluation was requested by the USAID’s Economic Development and Energy Office through the mission’s Monitoring, Evaluation, and Learning Activity (MELA) conducted by The Kaizen Company. FRPFM was implemented by Deloitte Consulting over the period April 2016 to September 2020 with a ceiling price of almost $36 million. It is a time￾and-materials task order contracted through the agency’s PFM II IDIQ. The project had a tall order. It was designed to end Jordan’s cycle of donor dependence in public financial management (PFM) and to pave the way to an “exit strategy” for USAID. This project was to advance PFM beyond the Ministry of Finance, to more holistically encompass the Government of Jordan, including its line ministries, state-owned enterprises, and governorates. The project operated in four very broad areas: tax policy and administration; budget; fiscal sustainability; and an exit strategy. This external evaluation is the first evaluation of FRPFM. There was no midterm evaluation. Our work is both an evaluation and an assessment. Phase 1 is a desk review, understanding the project’s experience through its own and other documents and using a lens of international best practice. Phase 2 is an evaluation that incorporates fieldwork through key informant interviews and focus group discussions. These were to be done on-site, but that was not fully possible due to the COVID 19 pandemic. The team did them through a mix of remote and on-site participation of the evaluation teams. Phase 3 is to provide an assessment and planning for a new fiscal reform project. The evaluation team has been given a list of evaluation and assessment questions, and our reports address these. This report addresses the seven Phase 2 evaluation questions, and a summary of the major findings follows. 1.1 WHICH INTERVENTIONS UNDER FRPFM CAN BE EXPECTED TO SUSTAIN OVER TIME? WHY AND HOW? This question was first addressed during the desk review for Phase 1. (A revision in the evaluation team’s statement of work later shifted this question to Phase 2.) During Phase 1 the evaluation team found sustainable results in the following areas: e-services, including electronic filing, electronic payments, and electronic refunds; gender-responsive budgeting; and macro-fiscal databases and modeling. Phase 2 fieldwork confirmed these findings. Several tentative conclusions from Phase 1 regarding positive sustainability were not confirmed in the Phase 2 fieldwork and those are explained in the following sections. These concern PPP project development; reform at the governorate level; and institutional capacity building 1 The evaluation team is comprised of the following members: The team was contracted by The Kaizen Company to conduct this external final examination of USAID Jordan’s Fiscal Reform and Public Financial Management (FRPFM) project as part of the mission’s Monitoring, Evaluation, and Learning Activity (MELA). 2 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV through the contractor’s proprietary Capacity-Performance-Results-Sustainability (CYPRESS) “maturity state” framework.2 1.2 WHAT ARE THE PROS AND CONS OF ASSIGNING EMBEDDED ADVISORS TO WORK AT THE MINISTRY OF FINANCE? Of the many questions that the evaluation team posed, none elicited more responses—and viewpoints—than that of the “embedded advisor.” Depending on the specific requirements of the follow-on project and its level of funding, there could be a role for one or more embedded advisors that have well-specified terms of reference with highly needed technical skills for tasks that would be completed during the period of the contract. 1.3 WHAT ARE THE MAIN FISCAL REFORM PRIORITIES FROM THE GOJ’S PERSPECTIVE, TAKING INTO CONSIDERATION THE READINESS OF GOVERNMENT INSTITUTIONS, AND THE IMPORTANCE, AND THE URGENCY TO WORK ON THE REQUIRED ACTIVITIES? The desk review in Phase 1 and field research in Phase 2 confirm that Jordan’s need for public financial management reform continues. Jordan has not completed its “journey to self-reliance” in its fiscal and public financial management responsibilities, which was an aspiration of the implementor’s 2016 contract. The broad dimensions of the mission’s new PFM project to address the GOJ’s fiscal reform priorities should include domestic revenue mobilization, with assistance in tax administration and tax policy, and budget formulation and execution. It was generally recognized by respondents that one of the major accomplishments of FRPFM was assistance to the Income and Sales Tax Department (ISTD) in developing the new income tax law. However, attention should now turn to other aspects of tax policy, a renewed focus on tax administration, and a more focused effort on budgeting. 1.4A WHAT ARE THE MAIN CHALLENGES THAT THE NEW PROJECT IS EXPECTING TO FACE, INCLUDING CHANGE RESISTANCE, CURRENT GOVERNMENT CAPACITY, ETC.? The major challenges that a new project faces will be correctly designing approaches and solutions to the project’s technical requirements, developing methods to accurately measure and target progress, and to systematically measure progress against these benchmarks or indicators. Among other possible challenges, we heard voiced the following in interviews:3 Frequent turnover of the Minister and the Secretary General (SG). Resistance of incumbent staff to technological and other changes. Ensuring ministry “buy in” to the development program. Retaining highly skilled staff trained by the project, or high-skilled staff who had been paid higher salaries by the project. Addressing carefully the public’s acceptance of project activities. 2 Upon reviewing this sentence (with reference to PPPs and the governorates), FRPFM officials noted that “legislative frameworks and decisions to reorganize government in these areas were outside of the span of control of the project.” 3 In reviewing the draft report, FRPFM officials noted that one might also include “continued performance under IMF EFF program, access to international capital markets, and external economic and political developments.” 3 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV 1.4B DOES THE MINISTRY OF FINANCE HAVE THE ADEQUATE CAPACITY AND COMMITMENT TO EFFECTIVELY IMPLEMENT THE FOLLOWING? Result-oriented budgeting (ROB) and gender-responsive budgeting (GRB) The ministry is committed to implementing ROB and GRB. FRPFM implemented activities for these, especially for GRB. Much remains to be done, such as for the budget law and regulations; development of the government financial management information system (GFMIS); capacity building in budget planning, project analysis, and monitoring and evaluation of budget results; development of operations manuals for ROB; and stakeholder awareness about the importance of ROB. IPSAS and Accrual Accounting The ministry is committed to implementing international public-sector accounting standards (IPSAS) and to move to accrual accounting, but even after almost five years of FRPFM’s effort, the MOF is not ready to do this. Automate the budget planning process The General Budget Department has made progress, but automation of budget planning and reporting is incomplete. IT system for ISTD The current IT system is more than 10 years old, does not perform up to modern standards, and should be replaced, based on an assessment by FRPFM. Roll out the e-procurement system The challenge is either to build an interface between GFMIS and the new South Korean supported e-procurement system, or to use the existing e-procurement system.4 1.4C DOES MOF (INCLUDING ISTD, GBD, AND SEPD) HAVE THE INTERNAL CAPACITY TO MANAGE AND PRODUCE FISCAL ANALYSES FOR OUTSIDE STAKEHOLDERS, SUCH AS THE INTERNATIONAL MONETARY FUND (IMF)? MOF officials and project staff stated that the MOF and the three departments (ISTD, GBD, SEPD) need more time and support to increase staff capacity to produce reports, including fiscal analysis for the use of outside stakeholders, such as the IMF. 1.5 HOW FEASIBLE IS IT TO REORGANIZE TAX ADMINISTRATION ALONG FUNCTIONAL LINES AND DEVELOP A REVENUE MANAGEMENT SYSTEM? The evaluation team learned in interviews with senior project officials that the Ministry of Finance, and its Income and Sales Tax Department (ISTD) in particular, are organized along functional lines. What this means is that the important functional 4 In reviewing the draft report, FRPFM officials provided the following additional input: “The new procurement system provided by the Koreans is based on a Chart of Accounts that does not match the GFMIS. Same for the Inventory System. Aligning the three chart of accounts is a massive undertaking. Technically the Korean system is proprietary and closed so specialized interfaces will need to be built. GBD and MOF do not have the technical resources to solve the systems problems. Moreover, linkages between commitment controls, allotments and procurement will require appropriate bridges between the systems. This is not just a GFMIS problem.” The evaluation team believes these are highly important questions that the follow-on project must consider if it provides assistance with e-procurement. 4 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV dimensions of tax administration are organized at headquarters by audit, compliance, collections, registration, and taxpayer services, each directed by an assistant director general. District offices are organized in the same way. This current structure would be in contrast to, for example, an organizational structure based on tax type (e.g., personal income tax, corporate income tax, general sales tax). An RFP for a new IT Tax system was developed by FRPFM and would be managed by ISTD. 1.6 ASSESS THE OVERALL EFFECTIVENESS OF FRPFM’S APPROACH, SPECIFICALLY, AND HOW DID THE PROJECT ACHIEVE ITS INTENDED OUTCOMES TILL NOW? WHAT ARE THE MAIN CHALLENGES? HAS FRPFM BEEN REACTING TO THESE CHALLENGES AS REQUIRED? ARE THERE ANY RECOMMENDATIONS BASED ON BEST PRACTICES IMPLEMENTED BY FRPFM? Question 6 was also answered as question 2 in Phase 1, which appeared in an earlier version of our statement of work. The discussion here is based on new information collected during field research. As part of Phase 1, the evaluation team reviewed the contractor’s approach and design, and the results and accomplishments of the contractor’s efforts. The contractor’s design itself is reasonable and consistent with the international standards that are found in USAID’s Guide to Public Financial Management; however, the contractor did not log and track its results in a way that would allow an outside observer (or even the project itself) to monitor progress against project targets and program goals consistent with this design. In particular, the Phase 1 desk review found that there were no targets and goals for most aspects of the project’s design. In order to dig deeper into what the project actually accomplished, the evaluation team compiled a stock-taking of project results. (See Box 2 in Annex 2 of the Phase 1 report.) The stock-taking showed that important work in public financial management was implemented across all four of the project’s components. However, for most of the functional areas of PFM that are in the contractor’s design, there were no benchmarks to measure progress, few quantitative measures of impact, and few if any assessments of sustainability of results achieved. Findings from the key informant interviews (KIIs) and focus group discussions (FGDs) confirm these conclusions. 1.7 HOW WILL THE CURRENT COVID-19 CRISIS AFFECT THE ABILITY OF THE GOVERNMENT TO DO PFM REFORMS AND HOW DOES THIS AFFECT THE FISCAL SUSTAINABILITY OF JORDAN? WHAT ARE THE ECONOMIC, SOCIAL AND POLITICAL FACTORS IN THE MEDIUM TERM THAT WILL AFFECT THE ABILITY OF THE GOVERNMENT TO DO FISCAL REFORMS IN THE MEDIUM TERM? During the Phase 1 desk review, the evaluation team learned that the short-term effects of COVID-19 in Jordan would be heavy and damaging. Nonetheless, most interviewees believe the project and the government are coping reasonably well, given the pandemic. There remain many unknowns, however, and project and government officials alike anticipate heavy costs that may disproportionately hurt the poor and certain sectors (education was mentioned), and possibly affecting women more than men. ّم مشروع المتابعة والتقييم والتعل المرحلة الثانية من التقييم لمشروع اإلصالح المالي واإلدارة المالية العامة (FRPFM ( العمل الميداني: تقييم السياق الحالي 1 كفاءةو ال ية حكومة األردن .١ ملخص تنفيذي يغطي هذا التقرير المرحلة الثانية من التقييم الخارجي للوكالة األمريكية للتنمية الدولية في األردن لمشروع اإلصالح المالي واإلدارة المالية العامة (FRPFM(. تم طلب هذا التقييم من قبل مكتب التنمية االقتصادية والطاقة التابع للوكالة األمريكية للتنمية ّ شركة كايزن. وقد تم تنفيذ مشروع اإلصالح المالي الدولية من خالل مشروع المتابعة والتقييم والتعلم (MELA (ذ من قبل المنف واإلدارة المالية العامة (FRPFM (من قبل شركة ديلويت لالستشارات خالل الفترة من أبريل نيسان/ ٢٠١٦ سبتمبر أيلول/ إلى مرتبط ببعد زمني ومادي تم ٢٠٢٠ بميزانية قارب ت ٣٦ ومليون دوالر. قد تم إرساء مشروع اإلصالح المالي ب أمر تشغيلي التعاقد عليه من خالل عقد اإلدارة المالية العامة التابع للوكالة األمريكية والذي يتصف بأنه عقد تسليم إلى أجل غير مسمى كمية و غير محددة. ً لقد واجه مشروع حيث اإلصالح المالي تحديا هتم تصميم إدوامة إلنهاء عتماد األردن على المانحين فيما يخص اإلدارة ً كبيرا المالية العامة ول تمهيد الطريق لـ "استراتيجية خروج" للوكالة األمريكية للتنمية الدولية. فقد عمل على المشروع دفع اإلدارة المالية العامة ألبعد من ،وزارة المالية ليغطي الحكومة األردنية بشكل أكثر شمولية، بما في ذلك الوزارات التنفيذية والمؤسسات المملوكة للدولة والمحافظات. حيث ارتكز عمل المشروع على أربعة مجاالت واسعة: السياسة واإلدارة الضريبية؛ وال وازنةم ؛ واإلستدامة المالية؛ وإستراتيجية الخروج . ّذ ةخارجيجهة ته هو التقييم األول لمشروع اإلصالح المالي وإدارة المالية العامة (FRPFM (حيث ويعتبر هذا التقييم الذي نف لم يكن هناك تقييم منتصف مدةل المشروع. وألن عملنا يجمع بين التقييم والتقدير ، فقد تضمنت المرحلة األولى المراجعة المكتبية بهدف فهم تجربة المشروع من خالل المستندات الخاصة به وغيرها من المستندات باإلضافة إلى مقارنة ممارسات المشروع بأفضل الممارسات الدولية. وكانت الثانيةالمرحلة عبارة عن تقييم يتضمن العمل الميداني من خالل مقابالت مع المخبرين الرئيسيين ومناقشات مجموعة التركيز. وفي حين كان من المقرر أن تتم أنشطة المرحلة الثانية على أرض الواقع، إال أن ً حيث . إلى التقييم فريق لجأ جمع المعلومات من لظروف ا المرتبطة بجائحة كوفيد ١٩ لم ن ّ تمك الفريق من تنفيذ الدراسة ميدانيا خالل مزيج من المشاركة عن بعد والمشاركة الوجاهية حسب الظروف الراهنة. أما المرحلة الثالثة قد تضمنت ف تقديم دراسة يةوتخطيط ةيتقييم لمشروع إصالح مالي جديد . تم لقد مشاركة فريق التقييم قائمة بأسئلة تقييمية وتقديرية سيتم ذكرها بالتفصيل في هذا التقرير . بحيث س يتناول هذا التقرير األسئلة السبعة المستخدمة في ه، ويتبعالثانية المرحلة ملخص للنتائج الرئيسية . 1.1 في إطار مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM (ما هي التدخالت التي من المتوقع أن تتصف بالديمومة مع مرور الوقت؟ لماذا وكيف؟ يم. ، أخصائية محلية؛ أفنان الحديدي، م خض أخصائ ي يتألف فريق التقييم من األعضاء التالية أسماؤهم: د. جون كري هفيلد ، قائد الفريق ؛ د. فادي عل حمد ، تق 1 ف لإلصالح المال كايزن إلجراء هذا الفحص النهائ الوكالة األمريكية للتنمية الدولية االردن ر الخارج لمش . وع ر تم التعاقد مع الفريق من قبل ش أخصائية محلية كة والتقييم والتعلم (MELA (الممول من الو كالة األمريكية للتنمية الدولية. رمش وإدارة المالية العامة (FRPFM (كجزء من وع المتابعة ّ لقد تم التطرق لهذا السؤال في بادىء األمر أثناء المراجعة المكتبية للمرحلة األولى )حيث أسفرت مراجعة بيان ال الخاص عمل بفريق التقييم إلى تحويل هذا السؤال إلى المرحلة الثانية .(في وقت الحق خالل المرحلة األولى ، وجد فريق التقييم نتائج مستدامة في المجاالت التالية: الخدمات اإللكترونية؛ بما في ذلك اإليداع اإللكتروني والمدفوعات اإللكترونية والمبالغ المستردة اإللكترونية؛ وال وازنةم المراعية لل جتماعينوع اإل ؛ وقواعد البيانات المالية الكلية والنمذجة. وقد أكدت المرحلة الثانية من العمل الميداني هذه النتائج. إال أن العديد من االستنتاجات المؤقتة المتعلقة باالستدامة اإليجابية والمستقاة من المرحلة األولى لم يتم تأكيدها عند القيام بالعمل الميداني الخاص بالمرحلة الثانية كما هو موضح في األقسام التالية. أكثرهاتتعلق و بتطوير مشروع الشراكة بين القطاعين العام والخاص؛ واإلصالح على مستوى المحافظة؛ وبناء القدرات المؤسسية من خالل إطار العمل "حالة 2 االستدامة -النتائج -األداء -المقاول كفاءةلالنضج" (CYPRESS(. 1.2ما هي مزايا وعيوب تعيين مستشارين مدمجين للعمل في وزارة المالية؟ لقد القت مسألة "المستشار المدمج" العديد من اإلجابات ووجهات النظر على عكس غيرها من األسئلة التي طرحها فريق التقييم. قد يكون هناك دور لواحد أو أكثر من المستشارين المدمجين الذين لديهم اختصاصات محددة جيدًا ويتمتعون بالمهارات التقنية المطلوبة لتنفيذ المهام التي سيتم إكمالها خالل فترة العقد، إال أن ذلك يتوقف على المتطلبات المحددة ل قادم مشروع الل ومستوى تمويله. 1.3 ام هي األولويات الرئيسية لإلصالح المالي من وجهة نظر الحكومة األردنية، مع األخذ بعين االعتبار جاهزية المؤسسات الحكومية، والحاجة الملحة للعمل على األنشطة المطلوبة؟ تؤكد المراجعة المكتبية في المرحلة األولى والبحث الميداني في المرحلة الثانية أن حاجة األردن إلصالح اإلدارة المالية العامة مستمرة. حيث أن األردن لم يكمل "رحلته نحو االعتماد على الذات" فيما يتعلق بمسؤوليات اإلدارة المالية العامة و المالية، السنة ّذ العقد تحقيقها في العام .٢٠١٦ يجب أن تش تمل األبعاد العريضة لمشروع اإلدارة المالية العامة على الجديد والتي كان يأمل منف معالجة أولويات اإلصالح المالي للحكومة األردنية وحشد اإليرادات المحلية، مع المساعدة في اإلدارة الضريبية والسياسة ًوم لمشروع اإلصالح الضريبية، وصياغة الموازنة وتنفيذها. فقد أقر المستجيبون لألسئلة عم ا بأن أحد اإلنجازات الرئيسية المالي واإلدارة المالية العامة (FRPFM (هو مساعدة دائرة ضريبة الدخل والمبيعات (ISTD (على تطوير قانون ضريبة الدخل الجديد. ومع ذلك، يجب أن يتحول االهتمام اآلن إلى جوانب أخرى من السياسة الضريبية، والتركيز المتجدد على اإلدارة ً الضريبية، وبذل جهد أكثر تركيزا على إعداد الموازنة. 1.4أ. ما هي التحديات الرئيسية التي يتوقع أن يواجهها المشروع الجديد، بما في ذلك مقاومة التغيير، والكفاءة الحكومية الحالية، وما إلى ذلك؟ ُهج والحلول للمتطلبات الفنية للمشروع، تتمثل التحديات الرئيسية التي س يواجهها المشروع الجديد في التصميم الصحيح للن الدقيق للقياس ل وتطوير طرق لتقدم في تحقيق األهداف، وقياس التقدم بشكل منهجي مقابل هذه المعايير أو المؤشرات. ومن بين 3 التحديات المحتملة األخرى، فقد سمعنا ما يلي المقابالت أثناء : عند مراجعة هذه الجملة )باإلشارة إلى الشراكات بين القطاعين العام والخاص والمحافظات(، الحظ مسؤولو مشروع اإلصالح المالي وإدارة المالية العامة 2 (FRPFM (أن "األطر التشريعية والقرارات إلعادة تنظيم الحكومة في هذه المناطق كانت خارج نطاق سيطرة المشروع". عند مراجعة مسودة التقرير، أشار مسؤولو مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM (ً إلى أنه يمكن أيضا تضمين "األداء المستمر في إطار برنامج 3 تمديد تسهيل الصندوق - صندوق النقد الدولي )EFF IMF )، والوصول إلى أسواق رأس المال الدولية، والتطورات االقتصادية والسياسية الخارجية." • التغيير المتكرر للوزير واألمين العام . • مقاومة الموظفين الحاليين للتغيرات التكنولوجية وغيرها . • ضمان مشاركة الوزارة في برنامج التنمية . • االحتفاظ بالموظفين ذوي المهارات العالية المدربين من قبل المشروع، أو الموظفين ذوي المهارات العالية الذين حصلوا على رواتب أعلى من قبل المشروع . • ال الدقيقة لمعالجة قبول الجمهور ألنشطة المشروع. 1.4ب. هل تمتلك وزارة المالية الكفاءة الكافية وااللتزام الكافي بتنفيذ ما يلي بشكل فعال؟ الموازنة الموجهة نحو النتائج (ROB (والموازنة المراعية للنوع اإلجتماعي (GRB ( تلتزم الوزارة بتنفيذ الموازنة الموجهة نحو النتائج (ROB (والم المراعية لل وازنة نوع اإلجتماعي (GRB(. بالنسبة لألنشطة المنفذة لمشروع اإلصالح المالي وإدارة المالية العامة (FRPFM(، خاصة بالنسبة إلى الموازنة المراعية للنوع اإلجتماعي (GRB (نه إف ال يزال هناك لقيام به،لالكثير مثل قانون الموازنة ولوائحها؛ وتطوير نظام معلومات اإلدارة المالية الحكومية (GFMIS(؛ وبناء القدرات في تخطيط الموازنة، وتحليل المشاريع، ورصد وتقييم نتائج الموازنة؛ تطوير و عملياتكتيبات الموازنة الموجهة نحو النتائج (ROB(؛ وتوعية أصحاب المصلحة حول أهمية الموازنة الموجهة نحو النتائج . (ROB) المعايير المحاسبية الدولية للقطاع العام (IPSAS (والمحاسبة على أساس االستحقاق تلتزم الوزارة بتنفيذ المعايير المحاسبية الدولية للقطاع العام (IPSAS (واإلنتقال إلى المحاسبة على أساس االستحقاق، ولكن حتى بعد مرور ما يقرب من خمس سنوات من جهود مشروع اإلصالح المالي واإلدارة المالية العامة (FRPFM(، فإن وزارة المالية ليست مستعدة للقيام بذلكبعد . أتمتة عملية وازنة تخطيط الم واضحا، عملية لكن و أتمتة تخطيط الموازنة وإعداد التقارير ال تزال غير مكتملة . ً لقد أحرزت دائرة الموازنة العامة تقدما ً نظام تكنولوجيا المعلومات لدائرة ضريبة الدخل والمبيعات (ISTD ( يبلغ عمر نظام تكنولوجيا المعلومات الحالي أكثر من ١٠ ً سنوات، وال يرقى إلى مستوى المعايير الحديثة، ويجب استبداله بناء نتائج على تقييم مشروع اإلصالح المالي واإلدارة المالية العامة (FRPFM(. طرح نظام الشراء اإللكتروني يتمثل التحدي إما في بناء واجهة بين نظام معلومات اإلدارة المالية الحكومية (GFMIS (ونظام المشتريات اإللكترونية . 4 الجديد المدعوم من كوريا الجنوبية، أو استخدام نظام الشراء اإللكتروني الحالي عند مراجعة مسودة التقرير، قدم مسؤولو مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM (المدخالت اإلضافية التالية: يعتمد " نظام المشتريات الجديد الذي 4 قدمه الكوريون على مخطط حسابات ال يتطابق مع نظام معلومات اإلدارة المالية الحكومية (GFMIS(. ينطبق عنفس الشيء لى نظام الجرد. تعد محاذاة المخطط الثالثي لحسابات مهمل ة ضخمة. من الناحية الفنية، فإن النظام الكوري هو ملكية خاصة ومغلقة، لذا يجب بناء واجهات متخصصة. ال تمتلك دائرة الموازنة العامة )GBD ) ووزارة المالية )MOF )الموارد التقنية لحل مشاكل األنظمة. عالوة على ذلك، ً ستتطلب الروابط بين ضوابط االلتزام والتخصيصات والمشتريات جسورا مناسبة بين واألنظمة. هذه ليست مشكلة نظام معلومات اإلدارة المالية الحكومية (GFMIS (فقط ". يعتقد فريق التقييم أن هذه أسئلة مهمة للغاية يجب أن يأخذها مشروع المتابعة بعين االعتبار إذا كان يقدم المساعدة في المشتريات اإللكترونية . 1.4ج. هل تمتلك وزارة ةالمالي بما) في ذلك دائرة ضريبة الدخل والمبيعات (ISTD(، دائرةو الموازنة العامة )GBD ، ) وقسم الدراسات والسياسات االقتصادية )SEPD ))الكفاءة الداخلية على إدارة وإنتاج التحليالت المالية لألطراف الخارجية ،المعنية مثل صندوق النقد الدولي (IMF(؟ صرح مسؤولو وزارة المالية وموظفو المشروع بأن وزارة المالية واإلدارات الثالث )دائرة ضريبة الدخل والمبيعات(ISTD(، ودائرة الموازنة العامة )GBD)، وقسم الدراسات والسياسات االقتصادية )SEPD ) بحاجة إلى ( المزيد من الوقت والدعم لزيادة كفاءة الموظفين على إصدار التقارير، بما في ذلك التحليل المالي إلستخدام األطراف ةالخارجي المعنية، مثل صندوق النقد الدولي. 1.5ما مدى جدوى إعادة تنظيم إدارة الضرائب عبر الخطوط الوظيفية وتطوير نظام إلدارة اإليرادات؟ علم فريق التقييم في مقابالت مع كبار مسؤولي المشروع أن وزارة المالية، وإدارة ضريبة الدخل والمبيعات (ISTD (على وجه الخصوص، منظمة على أسس وظيفية. وذلك يعني أن األبعاد الوظيفية الهامة لإلدارة الضريبية يتم تنظيمها في المقر الرئيسي عن طريق التدقيق واإلمتثال والتحصيل والتسجيل وخدمات دافعي الضرائب، يترأس كل منها مساعد المدير العام. ويتم تنظيم مكاتب المحافظات وريقة. بنفس الط هذا الهيكل الحالي يتناقض مع، على سبيل المثال، الهيكل التنظيمي القائم على نوع الضريبة )على سبيل المثال، ضريبة الدخل الشخصي، وضريبة دخل الشركات، وضريبة المبيعات العامة(. تم تطوير طلب تقديم العروض لنظام ضر يبة جديد خاص بتكنولوجيا المعلومات من قبل مشروع اإلصالح المالي واإلدارة المالية العامة (FRPFM (وستتم إدارته بواسطة إدارة ضريبة الدخل والمبيعات. 1.6تقييم الفعالية الكلية لنهج مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM(، على وجه التحديد، وكيف حقق المشروع نتائجه المقصودة حتى اآلن؟ ما هي التحديات الرئيسية؟ هل قام مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM (مواجهةب هذه التحديات كما هو مطلوب؟ هل توجد أية توصيات تستند إلى أفضل الممارسات التيو تم تنفيذها بواسطة مشروع اإلصالح المالي وإدارة المالية العامة (FRPFM(؟ تمت اإلجابة على السؤال السادس ً أيضا باعتباره السؤال الثاني في المرحلة األولى، والذي ظهر في نسخة سابقة من بيان عملنا. تستند المناقشة هنا إلى المعلومات الجديدة التي تم جمعها أثناء البحث الميداني. وكجزء من المرحلة ألولىا ، استعرض فريق التقييم نهج الشركة المنفذة للمشروع، وتصميم المشروع، ونتائج وإنجازات جهود الشركة المنفذة للمشروع. وبالرغم من أن تصميم الشركة المنفذة للمشروع بحد ذاته معقول ويتوافق مع المعايير الدولية الموجودة في دليل الوكالة األمريكية للتنمية الدولية لإلدارة المالية العامة؛ إال أن الشركة المنفذة للمشروع لم تقم بتسجيل وتتبع نتائجه بطريقة تسمح لمراقب خارجي )أو حتى المشروع نفسه( بمراقبة التقدم المحرز مقابل أهداف المشروع وأهداف البرنامج المتوافقة مع هذا التصميم. على وجه ف الخصوص، وجدت المراجعة المكتبية للمرحلة األولى أنه ال توجد أهداف وغايات لمعظم جوانب تصميم المشروع . وبهدف التعمق في ما أنجزه المشروع بالفعل، قام فريق التقييم بعمل قائمة ب نتائج المشروع. )انظر المربع ٢ ٢في الملحق من تقرير األولىالمرحلة (. حيث أظهر ت قائمة النتائج أن العمل المهم في اإلدارة المالية العامة قد تم تنفيذه عبر جميع مكونات المشروع األربعة. ومع ذلك، بالنسبة لمعظم المجاالت الوظيفية لإلدارة المالية العامة الموجودة في تصميم الشركة، لم تكن هناك معايير لقياس التقدم، كما وأن هناك عدد قليل من المقاييس الكمية للتأثير، وال ،، إن وجدتقليل من تقييمات استدامة النتائج المحققة. وقد جاءت نتائج مقابالت المخبرين الرئيسيين (KIIs (ومناقشات مجموعات التركيز (FGDs (لتؤكد هذه االستنتاجات. 1.7كيف ستؤثر أزمة فيروس كورونا الحالية على كفاءة الحكومة في تنفيذ إصالحات إدارة الشؤون المالية العامة وكيف سيؤثر ذلك على االستدامة المالية لألردن؟ ما هي العوامل االقتصادية واالجتماعية والسياسية في المدى المتوسط والتي ستؤثر على كفاءة الحكومة على تنفيذ اإلصالحات المالية على المدى المتوسط؟ خالل المراجعة المكتبية للمرحلة ،األولى علم فريق التقييم أن اآلثار قصيرة المدى ل -كوفيد ١٩ في األردن ستكون شديدة ومؤذية. يعتقد معظم ،ومع ذلك من تمت مقابلتهم أن المشروع والحكومة يتعامالن بشكل جيد مع الوباء . وال يزال هناك العديد من األشياء المجهولة، حيث يتوقع مسؤولو المشروع والحكومة على حد سواء تكاليف بأن ال الباهظة قد تضر بشكل غير متناسب بالفقراء وبعض القطاعات ) ،التعليم( مثل قطاع وربما تؤثر على الن ساء أكثر من الرجال . 5 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV 2. EVALUATION AND ASSESSMENT QUESTIONS Introduction This Phase 2 Report addresses the seven questions identified in section 5 of the evaluation team’s statement of work.5 Two of these questions (1 and 6) were addressed in the Phase 1 report (the evaluation team was using an earlier outline of the evaluation questions at that time). In light of the substantial information that the team has learned during its field research, we provide additional observations for question 6 here. The field research consists of more than 25 key informant interviews and five focus group discussions. Many of these were conducted on-site at the Jordanian Ministry of Finance and incorporated internet conferencing capabilities.6 Interviewees include FRPFM project staff, government of Jordan officials from the Ministry of Finance (MOF), the General Budget Department (GBD), the Income and Sales Tax Department (ISTD), the Studies and Economic Policy Directorate (SEPD), the Customs Department, the Department of Lands and Survey (DLS), governorates, other ministries, and representatives of USAID and the donor community. Annex 1 provides a list of officials (by position) and organizations included and dates of the interviews. No one in our report is identified by name to protect individual confidentiality. Annex 2 presents the survey instruments used for key informant interviews (KIIs) and focus group discussions (FGDs), in both English and Arabic. Annex 3 presents the results of our questionnaire that was given to participants in the FGDs. 2.1 WHICH INTERVENTIONS UNDER FRPFM CAN BE EXPECTED TO SUSTAIN OVER TIME? WHY AND HOW? See answer to question 4 in the Phase 1 report.7 At that time we reported the following tentative recommendations: “The evaluation team’s stock-taking exercise does suggest that numerous activities under FRPFM appear to be productive, are likely to be sustainable, and should be encouraged and supported in the follow-on project. The following is an illustrative list of activities that were making notable progress and would be good candidates for continuation. It is not a comprehensive list, and is one the evaluation team will add to during in-depth field research in Phase 2. E-services, such for electronic filing, electronic payments, and electronic refunds (1.1.4 in the stock-taking exercise). The increased use of e-services is impressive, appears to be sustainable, and should be supported in the follow-on project. Gender responsive budgeting (2.1.5). FRPFM has initiated GRB across GOJ ministries. To date the impact of GRB appears minimal (e.g., so far there are no apparent programmatic or budgetary implications), but minimal reporting across ministries has been initiated. 5 See Monitoring, Evaluation, and Learning Activity (MELA), USAID Jordan Fiscal Reform and Public Financial Management (PFM) Evaluation and Assessment, Section 5: Evaluation and Assessment Questions, June 2020, pp. 5-6. 6 All meetings were conducted practicing social distancing, a limited number of participants, and the use of masks and gloves in accordance with the Jordanian prime minister’s Defense Order #11. 7 An earlier version of our statement of work included this as question 4 in the Phase 1 report. 6 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV PPP certification and training (2.3.1). The large gains in PPP pipeline growth and PPP project development and approval can likely be sustained through the PPP unit established by the project and the PPP training and certification achieved. The follow-on project could continue to support this work.8 The substantial work on the macro-fiscal database (3.1.3), macro models (3.1.4), and other capacity building in the SEPD could and should continue through the staff training initiated by FRPFM. Although training-level results were substantially under the targets (see MEL indicators 4.3 – 4.7), the concept is valid and should continue. More general capacity building across the MOF and into other ministries as pertains to PFM should continue. This includes at the governorate level. FRPFM placed considerable emphasis on the contractor’s propriety CYPRESS approach to institutional capacity building. On the basis of project reporting, good progress appears to have been achieved as to the “maturity state” of the SEPD, PPP unit, and ISTD. The follow-on project should carefully review these results and assess whether CYPRESS or some other approach would be relevant to the follow-on project.” Now that the evaluation team has completed most of the field work, we would hold to most of these, with some exceptions. First, work on public-private partnerships, however valid, is not at the core of public financial management. The mission may wish to consider a program of this type as part of a separate project. Second, there does not appear to be sufficient political will to make substantial progress at the governorate level. Third, before proceeding with CYPRESS, which is a proprietary approach of the contractor, or something similar, it should be more clearly shown that it is providing SEPD, ISTD, and the PPP unit measurable impact on performance. 2.2 WHAT ARE THE PROS AND CONS OF ASSIGNING EMBEDDED ADVISORS* TO WORK AT THE MINISTRY OF FINANCE? *FRPII used to have embedded advisers but the current FRPFM did not have any. From the Mission’s perspective, it may be worth comparing both approaches and asking GOJ which was more helpful/effective. Of the many questions that the evaluation team posed, none elicited more responses—and viewpoints--than that of the “embedded advisor.” Indeed, several of our interviewees had served in the role of embedded advisor under earlier USAID fiscal reform projects contracted to Bearing Point (now Deloitte Consulting) or DAI. There was even discussion as to what, exactly, an embedded advisor is. We use the term to mean a full-time position, funded by the project, where the employee works primarily at the ministry and reports to someone at the ministry as well as to the project (the chief of party or a component lead). Several respondents took an equivocal position, stating that the value of the role “depends,” or is neither black nor white. The evaluation team believes that it is important to view the question in the broader perspective of the developmental￾diplomatic goals of the mission. In most cases the project (and USAID) and the ministry can be well served by a long-term advisor (not necessarily expatriate) who possesses highly needed technical skills as clearly defined in written terms-of￾reference and who serves a finite period of time that is no greater than the period of performance for the project itself. Such advisors are likely to earn more than government employees, due to their skills and to competitive salaries possible 8 The evaluation team has learned that no one was fully certified for the CPEP. Many took the foundation course, but few did part 2. 7 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV outside of government. The mission must be careful regarding supporting seconded government employees already funded by the government, and especially at higher than regular government salaries. The advisor would report to the project but would require a peer counterpart within the ministry to coordinate the work defined in the terms of reference. An example could be a senior expert in budgeting whose role is to train and mentor GBD staff in the practice of monitoring and evaluation required for sound budget execution and performance evaluation. Most of the project and government officials we interviewed would support this approach. A few felt that in general embedded advisors should be avoided, especially if they do not report back to the project or have clear terms-of-reference. Most government officials are comfortable with the idea of embedded advisors, and some confided that they are desperately needed. But in almost all cases, the government officials we interviewed stated that their goal is to raise the capabilities of their own staff sufficiently so as to replace the embedded advisors at the end of their assignments.9 Depending on the specific requirements of the follow-on project and its level of funding, there could be a role for one or more embedded advisors that have well￾specified terms of reference with highly needed technical skills for tasks that would be completed during the period of the contract. A possibility could be a budget expert who could provide hands-on guidance and on-the-job training to the GBD as it implements gender-responsive budgeting and results-oriented budgeting, IPSAS and accrual accounting, and implements automated budgeting through the Hyperion budget module to the GFMIS. Another possibility could be an information technology expert who could guide ISTD in the implementation of a new revenue management system. 2.3 WHAT ARE THE MAIN FISCAL REFORM PRIORITIES FROM THE GOJ PERSPECTIVE, TAKING INTO CONSIDERATION THE READINESS OF GOVERNMENT INSTITUTIONS, AND THE IMPORTANCE, AND THE URGENCY TO WORK ON THE REQUIRED ACTIVITIES? The desk review in Phase 1 and field research in Phase 2 confirm that Jordan’s need for continued public financial management reform continues. If there is one point on which all interviewees could agree, it is that all work areas contained in FRPFM—and especially under components 1 and 2—require more attention. Jordan has not completed its “journey to self reliance” in its fiscal and public financial management responsibilities, which was an aspiration of the implementor’s 2016 contract.10 Continuing to do everything, however, is not a strategic plan for PFM reform. The evaluation team puts forth here a prioritized list of potential interventions that represent the major fiscal reform priorities of the GOJ. These are also consistent with the developmental and diplomatic objectives of USAID and the U.S. government. Jordan is a strategic political ally of the United States in the Middle East. The United States provides roughly $840 million in budget support to Jordan each year—an amount that represents about eight percent of Jordan’s total annual revenues and grants.11 It is in U.S. and Jordanian interests that support to PFM continues, and in a manner consistent with building long-run sustainability. This is also consistent with a 9 The evaluation team observed that not all donors use embedded advisors. The EU, for example, only employs short-term advisors to address well specified, short-term technical requirements. 10 USAID’s self-reliance roadmaps incorporate indicators for a country’s commitment and capacity for development, covering many aspects of sound public financial management. These include tax system effectiveness, government effectiveness, information and communication technology adoption, open government, economic gender gap, business environment, and risk of external debt distress. 11 This is based on IMF, 2020 Article IV Consultation, press release, April 2020, p. 4, and interview data. 8 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV memorandum of understanding signed between the United States and Jordan in 2018, where the United States provides $1.275 billion per year in bilateral foreign assistance over a five-year period for a total of $6.375 billion (FY 2018-FY 2022).12 The broad dimensions of this PFM strategy include domestic revenue mobilization, with assistance in tax administration and tax policy, and budget formulation and execution. It was generally recognized by respondents that one of the major accomplishments of FRPFM was assistance to ISTD in developing the new income tax law. However, attention should now turn to other aspects of tax policy, a renewed focus on tax administration, and a more focused effort on budgeting, as discussed below. MOF officials appear to have a relatively short time horizon given the evolving political landscape. When asked about the GOJ’s major priorities, one ministry official summarized these as revenue collection, strengthening macro-fiscal analysis so as not to be so dependent on the IMF and other international players, and to build an economic environment conducive to investment and growth. It was emphasized that a new project should be structured but also flexible; have strong outreach capabilities; and transfer knowledge. On the one hand, results-oriented budgeting and gender-responsive budgeting are viewed as important, but not immediate priorities. On the other hand, pursuing automated budgeting and a new revenue management system are high priorities, as is strengthening capacity in international public-sector accounting standards (IPSAS). In addition, institutionalizing training from transfer of practical skills and knowledge from technical advisors (to include embedded advisors) has been weak and should be improved. This came out in the written responses to the questionnaire from mid-tier government officials. These are the front-line ministry officials who lead the day-to-day work in the ministries and departments.13 The basic outline for the new program could be as follows: 2.3.1 Domestic Revenue Mobilization A. Tax Administration Assisting in the installation and maintenance of the new IT Tax system , based on procurement documents developed under FRPFM. The evaluation team would prefer to see an off-the-shelf system (as opposed to a custom￾made system), so as to “benefit from international experience and compatibilities,” as a senior GOJ official told the team. The GOJ would purchase the system and the new project would assist in installing and in maintaining it. Details would be worked out in discussions with the ISTD in establishing the new project’s year 1 work-plan. Assistance with pay-as-you-earn (PAYE) tax payments. Improving information exchanges between ISTD and other government units, such as the Social Security Administration, building on FRPFM. Improved risk-based auditing, building on FRPFM. Implementing a new system of e-invoicing. Continued assistance in reducing arrears. 12 See Jordan: Background and U.S. Relations (June 18, 2020), p. 12, https://fas.org/sgp/crs/mideast/RL33546.pdf 13 See Annex 3 to this report, Survey Results. 9 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV For the Department of Lands and Survey (DLS): roll-out one-stop shops to some of the 34 DLS branches; implement a communications plan; build a land administration database; and improve cybersecurity. Provide on-the-job training and capacity development for tax issues. Many of these interventions would address aspects of tax evasion and tax fraud, such as through improvements in e-invoicing and better auditing. B. Tax Policy A thorough review of the general sales tax (GST), special tax exemptions, such as at tax-free zones, and tax holidays; some interviewees stated that indirect taxation in Jordan is very high; Develop an improved GST; For SEPD: Assist in building economic models and in restructuring the directorate, as required, and conduct training needs assessment; build capacity through on-the-job training, and establish links to Jordanian universities and research institutes for fiscal and economic analysis. C. Customs A customs component would implement the single window for custom (following the procurement documents developed by FRPFM). There could also be focus on preclearance, risk assessments, electronic inspections, integrating systems through on-line interfaces, assisting with tariff classification of good and fee structures (e.g., introduction of a flat-rate system), and in establishing a training center. Other requirements could involve assistance in establishing a single export entity to consolidate current law that allows for the Aqaba Special Economic Zone. In general, it would be preferable to implement all customs-related interventions together, either as a customs component in the new public financial management activity, or as a separate project. 2.3.2 Budgeting There is a general view within the MOF that FRPFM did too little in budgeting and that a follow-on project should address budget formulation and execution. The following areas were addressed in FRPFM and could benefit from a more focused development effort, with greater attention to long-term expert assistance provided to the GBD: Continued development of automated budgeting and GFMIS. Whether there should be continued development of the budget module Hyperion should be reviewed together by the GBD and the new project.14 Continued development of IPSAS, recognizing that full accrual accounting is a long-term goal and that in some instances, cash accounting may be a better shorter-term option. A new GFMIS based on accrual accounting would ultimately be required. Consideration should be given to preparing procurement documents for such a new system. Results-oriented budgeting and gender-responsive budgeting can continue, but with clear and limited targets. Effort could focus on assistance towards 14 The GOJ chose Oracle as the foundation for this budget module. Oracle’s website says the following about Hyperion: “Hyperion Financial Management is a comprehensive, web-based application that delivers global financial consolidation, reporting and analysis in a single, highly scalable software solution. Oracle Hyperion Financial Management utilizes today’s most advanced technology, yet is built to be owned and maintained by the enterprise’s finance team.” 10 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV providing more disaggregated GRB data from line ministries, building on the experience gained from the nine pilot ministries from FRPFM. Other elements of a potential reform agenda that may be lesser priorities include the following: Deconcentrated budgeting and fiscal decentralization. There does not appear to be the political will at this time to pursue deconcentrated budgeting as attempted under FRPFM. In addition, a more thorough effort towards decentralization of revenue and expenditure decision-making does not appear to be a high GOJ priority at the present time.15 Debt management. The evaluation team did not see a strong need for further debt management assistance, beyond what a U.S. Treasury program might continue to offer. The priority given to e-procurement should be reviewed. It may be an important intervention in the medium term (three to five years), but not immediately. One MOF official stated that the ministry’s “current procurement system is good, even very good.” Potential suppliers submit all required documents, receive a password, and provide bank guarantees. There remain issues with cybersecurity, monitoring and evaluation, computer storage capacity, and tools such as a procurement manual. The current system must be connected to the budget through GFMIS to identify procurement allocations for capital expenditures for each government unit’s budget. Establishment of a tax policy unit. The evaluation team observed that building capacity—and maintaining it—at the MOF has been a challenge for over 15 years. It may not be any different now, and a less ambitious approach would be to limit new support to assisting the macro-fiscal unit in the SEPD. Institutionalizing training. The evaluation team recognizes the long-term value of training programs, such as supporting small numbers of the MOF staff in specialized training and economic students through graduate degree programs, including grant assistance for graduate degree programs in the United States. There would also be value in building closer ties between research institutions within Jordanian universities and the Ministry of Finance. However, it appears premature to support more sophisticated internal training capacity within the MOF and its Financial Institute and departments in light of weak sustainability of previous efforts. Similarly, unless the capabilities provided by CYPRESS can be clearly demonstrated to show improved performance (and “maturity”) and value-for-money, this intervention could be given lower priority. 2.4A WHAT ARE THE MAIN CHALLENGES THAT THE NEW PROJECT IS EXPECTING TO FACE, INCLUDING CHANGE RESISTANCE, CURRENT GOVERNMENT CAPACITY, ETC.? The major challenges that a new project faces will be correctly designing approaches and solutions to the project’s technical requirements, developing methods to accurately measure and target progress, and to systematically measure progress against these bench-marks or indicators (see question 3 above). In addition to these, 15 Several GOJ officials from directorates and ministries that have local responsibilities (e.g., the Ministry of Interior) were enthusiastic about FRPFM’s support to governorates’ budget allocation decisions and would like to see this effort continue. 11 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV there are numerous more generic challenges that the new project will face. We identify some of the most important of these below: Frequent turnover of the minister and the secretary general (SG). A solution could be to ensure that capacity building and hands-on assistance include deputies and lower-tier managers in the civil service that do not turn over so frequently. Resistance of incumbent staff to technological and other changes. A solution could be to introduce changes gradually, encourage staff input, and to provide sufficient capacity building and training to reduce perceived threats and disadvantages from change. Ensuring ministry “buy in” to the development program. As related under question 6 below, ministry officials feel cut-off from planning and implementing project activities in their own departments. A solution could be to include ministry officials in project planning and implementation, including sign-offs by SGs for activities within their departments. A focal person in departments and directorates could facilitate coordination between the project and the SG. Retaining highly skilled staff trained by the project, or high-skilled staff who had been paid higher salaries by the project. The GOJ and USAID must approach these issues smartly. If a project’s interventions create long-term jobs that require higher salaries to be competitive in the marketplace (e.g., for high skilled IT employees), then the GOJ must be willing to fund these positions at competitive wages when the project ends. If not, lower technology solutions should be used. USAID understands that paying “embedded” staff higher salaries to complete well-defined and finite technical tasks completed before project ending can be appropriate. However, it is much less appropriate to fund such positions that cannot be completed before the project’s end unless the ministry has a well-defined plan to accept these responsibilities at that time. The new project must address carefully the public’s acceptance of project activities. Public outreach, including taxpayer awareness campaigns, will be essential for the project’s ultimate success. The evaluation team notes that FRPFM did not consult representatives of taxpayers, and in particular, accountants and auditors, of project activities. This was an unfortunate omission in the project’s public outreach and resulted in lack of taxpayer feedback. 2.4B DOES THE MINISTRY OF FINANCE HAVE THE ADEQUATE CAPACITY AND COMMITMENT TO EFFECTIVELY IMPLEMENT THE FOLLOWING? Result-oriented budgeting (ROB) and gender-responsive budgeting (GRB) The ministry is committed to implementing ROB and GRB. Many actions were implemented with project support in these areas in the ending project. KPI indicator 2.8, “percentage of GOJ institutions implementing GRB,” showed good progress in GRB.16 Based on project staff expectations, PEFA is expected to conduct an approach to evaluate Jordan’s GRB in public financial management. This approach 16 See Annex 3 in the Phase 1 report, which is the indicator tracking table. 12 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV has nine indicators including working on policy, budget execution, accounting, and reporting in addition to external scrutiny and audit. Continued work on ROB is required, and government officials mentioned in the KIIs that more must be done in order to implement ROB. These include: The budget law and regulations: modifications are required in order to connect results with budgets. System development: to enable GFMIS to cover more detail about each government unit’s plan and accomplishments, and also more detail about each account and the chart of accounts. MOF officials mentioned that the ministry needs more assistance in budget topics, and also help in the development of the IT system. This will help in the connection between the budget numbers with their expected results. Capacity building is required for staff developing and implementing the budget, and is also required in project analysis and in selection of indicators. This could be through trainings and workshops, TOT, and on-the-job trainings. Supporting documents and material: The GBD needs further specialized operations procedures manuals in ROB, explanation forms, and flow charts for the budget implementation structure and procedures for budgeting, starting from the development, approval, implementing and evaluation of ROB. Awareness about the importance of ROB: stakeholders are not sufficiently aware of ROB and about the need and the importance for implementing it. These suggest that the next project could work on numerous tasks related to ROB. A detailed action plan is required to cover these topics to support ROB activities in: Legal review GFMIS systems development Capacity development Conducting awareness campaigns Developing ROB manuals that include GBD operating procedures. IPSAS and Accrual Accounting The ministry is committed to implementing international public-sector accounting standards (IPSAS) and to move to accrual accounting, but even after almost five years of FRPFM’s effort, the MOF is not ready to do this. Indicator 2.1 from the indicator tracking table shows that no progress was achieved during the five years. MOF officials in the KIIs mentioned that the ministry wants to continue work on IPSAS in the next project, and project staff said that the ministry must take more actions, especially the general accounts department, in order to implement accrual accounting. The new project must be sure that activities are well targeted and have ministry approval and commitment of the project’s work-plan. The ministry must designate appropriate staff to work with the project towards the implementation and completion of IPSAS and a transition to accrual accounting. Automate the budget planning process The General Budget Department has made progress, but automation of budget planning and reporting is incomplete. In interviews government officials identified topics that create difficulty in the automation of budgeting. Some of the important issues are as follows: 13 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV The current GFMIS requires improvements in efficiency and security. Some issues are internet connections, cybersecurity, and firewalls. Reporting should be improved to reflect monitoring and evaluation of budget outcomes using the chart of accounts at the subministry level (departments, governorates). Not all ministries are aware of the importance of the budget planning process, and how this process will be used for monitoring and evaluation. IT system for ISTD The current system does not cover required needs, and based on government officials, the main concerns are these: The current IT system is more than 10 years old, does not perform up to modern standards, and should be replaced, based on an assessment by FRPFM; Integrate the system with other tax systems (e.g., GST); Enhance reporting capabilities of the system; Focus on tax awareness for implementing the IT system; Increase cybersecurity in this system; Raise ISTD - IT staff capacity in developing the tax system; ISTD needs to do process reengineering before implementing the new IT system Implement the national invoicing system to control tax evasion. Roll out the e-procurement system Government procurement is the responsibility of the General Supplies Department at the MOF. FRPFM did not work on e-procurement, which was not part of its statement-of-work. Through the KIIs, some interviewees mentioned that a new system for e-procurement is in development, donated by the South Korean government (KOICA). It should be confirmed that the best option now is to build an interface between GFMIS and the new South Korean supported e-procurement system, as opposed to using the old procurement system. 2.4C DOES MOF (INCLUDING ISTD, GBD, AND SEPD) HAVE THE INTERNAL CAPACITY TO MANAGE AND PRODUCE FISCAL ANALYSES FOR OUTSIDE STAKEHOLDERS, SUCH AS THE INTERNATIONAL MONETARY FUND (IMF)? MOF officials and project staff stated that the MOF in general and the three departments (ISTD, GBD, SEPD) need more time and support to increase staff capacities to produce reports, including fiscal analysis for the use of outside stakeholders, such as the IMF. The main difficulties for reporting are these: The systems used do not provide all the required data needed or requested, and data are not updated on-line, including from other ministries. Ministry staff capacities are not capable of analyzing and reporting data as required by some stakeholders, such as the IMF. Reporting in English is a major difficulty faced by ministry staff. Some reports require technical economic models that are beyond current ministry capabilities. 14 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV An analysis of reporting capacities of each department is as follows: ISTD: The department is working on issuing a newsletter about collections, reforms, and achievements happening within ISTD. This can be considered as a base for reporting in the future. The newsletter will be used internally initially. In the future the newsletter audience will become the public. A goal is to increase ISTD staff capacity in reporting and analyzing data, especially in collections and about taxpayers. This will assist forecasting and getting more accurate data about public finance revenues for predictions. GBD: The department issues financial statements for the general budget based on IPSAS in Arabic only. Not all items of the budget law are in English. Issuing reports for GBD needs enhancements in order to make them ready to be used by external stakeholders. SEPD: SEPD’s structure was recently changed. SEPD staff capacities are increasing, but reporting skills must be improved. SEPD’s goal is to help other departments in issuing reports at the level required by outside stakeholders. However, SEPD needs assistance in preparing and updating new economic models. In general, the evaluation team recognizes that there are limits to improving ministry capacity without more general civil service reform. This is due to the GOJ’s difficulty in retaining high-quality staff in the face of competing offers from the private sector, NGOs, donors, and more widely in the region. 2.5 HOW FEASIBLE IS IT TO REORGANIZE TAX ADMINISTRATION ALONG FUNCTIONAL LINES AND DEVELOP A REVENUE MANAGEMENT SYSTEM? The evaluation team learned in interviews with senior project officials that the Ministry of Finance, and its Income and Sales Tax Department (ISTD) in particular, are organized along functional lines. What this means is that the important functional dimensions of tax administration are organized at headquarters by audit, compliance, collections, registration, and taxpayer services, each directed by an assistant director general. District offices are organized in the same way. This current structure would be in contrast to, for example, an organizational structure based on tax type (e.g., personal income tax, corporate income tax, general sales tax). The SEPD structure was recently changed, and it includes the following divisions: macroeconomics division, fiscal policy (modelling and analysis) division, revenue policy division, including tax policy, and statistics and international development. There was discussion as to whether tax policy should be included in SEPD’s revenue divisions or should be established within ISTD as a tax policy unit (TPU). There were also questions about whether a new TPU in ISTD should be in addition to or possibly to replace the SEPD. As mentioned, SEPD contains as one of its units a revenue division that provides some analysis into tax-related questions, such as tax expenditures. The evaluation team did not hear a consensus view within the government on the establishment of a tax policy unit (favored by the IMF). This is a political question that should be decided by the GOJ. The evaluation team cautions that establishing a TPU could face sustainability issues due to the difficulty in hiring or training appropriate staff, and maintaining the unit’s capacity, issues faced in earlier fiscal reform projects. 2.6 ASSESS THE OVERALL EFFECTIVENESS OF FRPFM’S APPROACH, SPECIFICALLY, AND HOW DID THE PROJECT ACHIEVE ITS INTENDED OUTCOMES TILL NOW? WHAT ARE THE 15 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV MAIN CHALLENGES? HAS FRPFM BEEN REACTING TO THESE CHALLENGES AS REQUIRED? ARE THERE ANY RECOMMENDATIONS BASED ON BEST PRACTICES IMPLEMENTED BY FRPFM? As part of Phase 1, the evaluation team reviewed the contractor’s approach and design, and the results and accomplishments of the contractor’s efforts.17 The contractor’s design itself is reasonable and consistent with the international standards that are found in USAID’s Guide to Public Financial Management; however, the contractor did not log and track its results in a way that would allow an outside observer (or even the project itself) to monitor progress against project targets and program goals consistent with this design. In particular, the Phase 1 desk review found that there were no targets and goals for most aspects of the project’s design.18 In order to dig deeper into what the project actually accomplished, the evaluation team compiled a stock-taking of project results. (See Box 2 in Annex 2 of the Phase 1 report.) The stock-taking showed that important work in public financial management was implemented across all four of the project’s components. However, for most of the functional areas of PFM that are in the contractor’s design, there were no bench-marks to measure progress, few quantitative measures of impact, and few if any assessments of sustainability of results achieved.19 Findings from the key informant interviews (KIIs) and focus group discussions (FGDs) confirm these conclusions. Most experts and government and donor officials that we interviewed could refer to general impacts. But few referred to bench-marks or targets attained (exceptions are noted below), and few results were viewed as sustainable. An overall assessment by one project official was that if USAID were to end PFM assistance, the Ministry of Finance and its departments could continue with basic budgeting, GFMIS, and tax administration; answer questions and provide basic information requested by the IMF and other international financial institutions; but could not do more than the basics and could provide little if no fiscal policy analysis. Some government officials believed that ministry capabilities had advanced little if at all since 2010. In the absence of sound PFM monitoring, it is difficult to counter these claims. What follows is a summary of the most important views (both cross-cutting and for individual components) that the team learned through its many interviews. Cross-cutting findings (across all components) There is a recurring theme across many KIIs and FGDs that there was insufficient engagement between the project and the MOF and its departments. This took many forms, such as the following:20 17 Question 6 was also answered as question 2 in Phase 1, which appeared in an earlier version of our statement of work. The discussion here for question 6 is based on new information collected during field research. 18 In the Phase 1 report the evaluation team refers to these as the project’s three-digit functional design elements, which are the building blocks of a PFM system. 19 We are referring here to potential substantive impacts of FRPFM that were not captured in the MEL plan. The Phase 1 report has already shown that many of the targets in the MEL plan were not achieved, and that many anticipated results in the contract were not attained. 20 The bulleted points in the main text are essentially verbatim comments from high-level ministry officials. In reviewing the draft report FRPFM officials noted the following, “FRPFM collaborated with counterparts on project priorities, workplans and timelines; Ministerial oversite and regular meetings with SG/DGs effectively functioned as steering committee; FRPFM provided secretariat support for decentralization; significant on the job training was provided at all counterpart agencies.” The evaluation 16 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV The project should provide its plans in advance to ministry counterparts, and these plans should have clear objectives and time-lines; The project should provide more options, such as offering more than one CV for potential technical advisors, and that the CVs presented should accurately match the skills of the experts; There should be a steering committee that includes representatives of the project, the MOF and other concerned ministries, and USAID; Proposed consultants should be better informed about the Jordanian issues they are to address; Training provided should be less theoretical, more practical, and focus on on￾the-job training; training of trainers should be emphasized. Component 1 The following identifies accomplishments and shortcoming for component 1 (tax policy and tax administration) brought to the team’s attention with project representatives, government officials, and donor representatives. In each interview the interviewee was asked to identify the project’s most important accomplishments and challenges. Many of these were also identified in our stock-taking exercise in Phase 1 (Box 2 in Annex 2). The project assisted the GOJ in implementing the new income tax law. Some government officials said this was the project’s most important accomplishment. Tax registrations have increased through new registrations and improved interfaces with other ministries. However, project documentation does not report targets, total potential coverage, or progress made towards targets. The project made excellent progress for on-line services. (Note: This is one of the exceptions where the project monitored and achieved quantifiable targets. The other is for CYPRESS, discussed below.) E-services increased from four percent to 100 percent; e-payments from two percent to 94 percent; and e￾refunds from 0 percent to 94 percent. One project representative cautioned that these increases in themselves do not mean more efficient operations since manual checks of documents still must occur. A government official said that a reason behind this is that electronic transactions have not replaced paper transactions as being legally binding.21 The project has introduced risk-based audits, with greater focus on large and medium taxpayers, smaller samples from small taxpayers, and a shift in ministry manpower from small to larger taxpayer units. However, it is not clear how much of this program has been implemented and how much new revenue has been garnered because of it. The project has speeded up payment of arrears by using private mail and by working with the Ministry of Trade to block delinquent accounts. It is not clear, however, how much has actually been implemented, how sustainable it is, and how much revenue has been gained. team believes that one of its important roles is to help identify such discrepancies in points-of-view that might not otherwise be voiced by the beneficiaries. 21 A better measure of the value of on-line services would be net revenues gained due to these on-line services. 17 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Several interviewees said little or no progress was made with tax policy analysis, and that the ministry must rely on the IMF, World Bank, and donors for help in this area. The ministry itself can only produce basic data and monthly data reports. Other accomplishments identified during the interviews: the case management system and the system for nonfilers were automated; a pay-as￾you-earn (PAYE) system was set up; improvements were made in using third￾party information from other government units; and tax ID numbers were standardized (using national ID numbers). In all of these cases, however, it was not possible for the evaluation team to discern how much progress was actually made because there are no quantitative measures of targets and results achieved. For almost all areas the interviewees said that more work could be done and that the sustainability of results achieved was uncertain. Component 2 In the absence of benchmarks and targets for most functional areas in component 2 (see Box 1 of Annex 2 in the Phase 1 report), the evaluation team compiled a stock￾taking of reported accomplishments (Box 2 in Annex 2 in the Phase 1 report). Unfortunately, these lists of accomplishments from the project’s first three years do not tell a coherent story of progress, and this tentative conclusion from Phase 1 is born out in the interviews. One high ranking GOJ official noted little (or no) progress in the three key reform areas of results-oriented budgeting, charts of accounts, and decentralization. Several other summary statements expressed to the evaluation team were these: There was little progress with GFMIS during the project’s four-year duration; There was a lack of progress with accrual accounting and IPSAS during the project’s duration; There was no clear vision for results-oriented budgeting, and little or no progress was made in ROB. These observations are generally consistent with the finding in the Phase 1 report. Additional observations were made in other interviews: 93 GBD staff were trained in IPSAS, a project management office was established for IPSAS, and an IPSAS roadmap developed; The project completed a policies and procedures manual for IPSAS (an activity handed over from the EU), which was to be (but has not been) piloted in six ministries; During the project’s last year, the design for component 2 was revised to focus on budget preparation, IPSAS, reporting, arrears, and cash management. How these revisions carry on from the original design is not clear. (This would be difficult to show in the absence of targets.) It is not clear why these changes were made, nor how these new areas would be monitored. Little was said by any interviewee (GOJ or donor) about public-sector partnerships, despite the attention given to PPP in the project’s reporting and the MEL plan.22 22 This is subcomponent 2.3 in the project design: “increased GOJ adoption of public-private partnerships.” See Box 1 in Annex 2 to the Phase 1 report. 18 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Interviewees expressed reasons why project advisors were ineffective in MOF’s departments. It was mentioned that advisors did not have the right experience in technical areas (e.g., did not have hands-on experience, or lacked knowledge about Jordan’s work environment), or that only one CV was presented for consideration for an advisor. It was also stated that short-term training is of limited value and is impractical, and does not provide follow-up. Several officials expressed their frustration that the MOF was insufficiently involved in establishing project plans and activities. Component 3 Interviewees touched on the accomplishments for component 3 discerned in our stock-taking exercise (Box 2 in Annex 2 in the Phase 1 report). Most items referred to as “successes” were qualified by their limitations and none was portrayed as having attained predetermined targets that would define successful project achievement. The interviews identified the following: The project helped SEPD develop analytical models, including a macro model and the restoring of a medium-term financial framework (MTFF) that had been abandoned in about 2010. However, the sustainability of these new models remains a question. In particular, it is not evident why new modelling capabilities developed during the current project (e.g., for the MTFF) will not diminish as it did for the earlier MTFF developed in the prior fiscal reform projects. There are concerns that the models developed are too complicated for SEPD staff, that the models cannot be maintained or up-graded, and that their results are unreliable. It was mentioned that the project provided training in the use of these new models. However, due to staff turnover, SEPD capacity did not increase. There is little evidence that the SEPD retained an institutional capacity to continue training without project advisors.23 Regarding grants, one interviewee noted that FRPFM had been designed to provide $3.75 million in grant assistance; however, in the end, less than $1 million was spent as funding was reallocated to other activities. The reason, the evaluation team was told, was that technical assistance for other components of the project, were considered to be of greater importance.24 Another interviewee mentioned that five grants were made. The project provided no way, however, to assess the quality of the work (such as economic research) supported through these grants, nor whether the local research institution that received these funds became more sustainable and no longer required USAID funding (as stipulated as an anticipated result in the contract). Components 4 and 5 There were few comments or reactions to component 4 (the exit strategy), even though, as one interviewee pointed out, this was supposed to be USAID’s last PFM project in Jordan. Another official commented that the overriding goal in the contract—that “the GOJ will be able to carry on its fiscal and public financial management responsibilities independently (contract, p. 19)”—was perhaps naïve. 23 On this, MEL indicator 4.6 (number of person hours of training for component 3) does not give much optimism, given the significant underperformance of this indicator. 24 It is not clear what other potential grants were not funded, if any had been specifically identified. There were five grants, including two to research organizations. 19 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV To whatever extent this is a valid assessment, the project did not in general monitor the gaps between the anticipated results in the contract and current project status. An exception is in training, where the MEL plan records the project’s shortcoming (see indicators 4.4, 4.5, 4.6, and 4.7 in Annex 3 of the Phase 1 report.) Only a few interviewees referred to component 4’s activities regarding CYPRESS, which is the contractor’s proprietary institutional maturity modeling framework, or could articulate what this meant. The evaluation team observed that the annual reports show increasing “maturity state” values for all three government units represented (SEPD, PPP, and ISTD). The year three annual report states that these measures “show progress made in advancing the performance of the three entities from basic to nearly advanced performance maturity level in 3 years (Annual Progress Report, Oct. 15, 2019, p. 7),” and indicate advanced maturity performance in MOF capacities, including “strategic planning, manage learning, promote human resource development and to refine leadership capacities as well as improving core PFM functions.”25 Nonetheless, there appears to be a disparity between these measures and other observations referred to above that point to limited capacity development (e.g., ability to maintain technical competencies within ministry units). The project added new activities after its debut in 2016. These included assistance to the Department of Lands and Survey (DLS) at the south Amman branch in year 2 and writing procurement documents for a new revenue management system and a customs single window.26 These tasks were represented either under component 3 (3.4 or 3.4.1) or at other times referred to as a new component 5. These activities are not included in the MEL plan nor are they tracked systematically. Observations collected during the interviews revealed the following: DLS is the second largest revenue generator for the government. There are 34 DLS branches throughout the country, including one in south Amman, where the project helped to set-up a one-stop shop for all DLS services, and established better procedures for risk management and cybersecurity. Although the evaluation team saw no systematic monitoring of results at the south Amman branch, the results achieved there appear to be having their intended effects and can be applied elsewhere, such as at the West Amman branch. The project provided other services that do not fit clearly under the four components. For example, it developed procurement documents including requests for information, for a new revenue management system, a single window for customs, and electronic invoicing. These are documents that the MOF would have had difficulty producing itself and could be the basis for future project activity (see question 3). 2.7 HOW WILL THE CURRENT COVID-19 CRISIS AFFECT THE ABILITY OF THE GOVERNMENT TO DO PFM REFORMS AND HOW DOES THIS AFFECT THE FISCAL SUSTAINABILITY OF JORDAN? WHAT ARE THE ECONOMIC, SOCIAL AND POLITICAL FACTORS IN THE MEDIUM TERM THAT WILL AFFECT THE ABILITY OF THE GOVERNMENT TO DO FISCAL REFORMS IN THE MEDIUM TERM? 25 As stated in the Annual Progress Report, Oct. 1, 2016—Sept. 30, 2017, p. 9. The evaluation team notes that the contractor’s systematic measuring and tracking of performance—in this case for “maturity state”—is a good example of sound monitoring since it tracks performance over time in terms of predetermined bench-marks (basic, developing, advanced, and leading). 26 During the field research we also learned that assistance was provided to the Ministry of Economy and Entrepreneurship in developing an econometric model and in working with the private sector. 20 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV During the Phase 1 desk review, the evaluation team knew that the short-term effects of COVID-19 in Jordan would be heavy and damaging. Tourism represents 12 percent of GDP, and Jordan relies on remittances, especially from the Gulf Cooperation Council (GCC) countries. The Central Bank of Jordan reduced policy interest rates by 50 basis points and the government temporarily reduced general sales tax rates. However, at that time the IMF stated it was not possible to quantify the economic effects. We learned through our interviews that GDP would likely decline by at least six percent, that the budget deficit would almost double, and that debt-to-GDP could reach or exceed 110 percent. An MOF official remarked that of 1,700 MOF employees, only about 70 now go to the office (this number changes based on emergency announcements and procedures), including those providing emergency related work plus those who make payments. Nonetheless, most interviewees believe the project and the government are coping reasonably well, given the pandemic. Several pointed out that better use of the internet is reducing “red tape” from unnecessary interventions (and interference) at government offices and has promoted “one-stop shops” and much quicker transactions, such as for transferring ownership of a car or in updating residency information. The value of e-services, developed by the project prior to COVID, such as e-filing and e-paying, highlights the benefits of on-line services. The project has adjusted in other ways as well since March, such as through increased use of virtual training. Several months into the crisis, government officials are demonstrating an ability to better plan for new contingencies and a renewed confidence that they can work with the limitations created by the pandemic. MOF officials mentioned that reduced public revenue caused by COVID has generated political support to raise revenue in other ways, such as by tackling tax evasion and by reforming tariff structures. The project’s final report noted that the financial commitment and contingent liabilities program (the FCCL), supported by the project, will be evaluating the potential net effects of COVID-19 as a force majeure event on the government of Jordan. There remain many unknowns, however, and project and government officials alike anticipate heavy costs that may disproportionately hurt the poor and certain sectors (education was mentioned), and possibly affecting women more than men.27 3. KEY INFORMANT INTERVIEWS AND FOCUS GROUP DISCUSSIONS: SUMMARY As part of the end-of-project evaluation for the Fiscal Reform and Public Financial Management (FRPFM) project, the evaluation team held 28 key informant interviews. Interviewees included 14 FRPFM staff, three representatives of donor agencies (including USAID), and government of Jordan officials from the Ministry of Finance, the General Budget Department, the Income and Sales Tax Department, the Department of Lands and Survey, the Customs Department, and the Studies and Economic Policy Directorate. (See Annex 1 for a complete list of interviews.) An interview typically included the entire evaluation team, led by the team leader, and the interviewee. Due to the COVID pandemic, these were mainly on-line. Each interviewee was asked a series of questions from the general to the specific regarding FRPFM, which allowed the team to hone in on the interviewee’s field of expertise. (See Annex 2 for the interview guide.) The phase 2 report draws on answers to these interview questions without identifying respondents by name or position to protect confidentiality. 27 The evaluation team noted in its review of the project’s final report that the financial commitment and contingent labilities program (FCCL), supported by the project, will be evaluating the potential net effects of COVID-19 as a force majeure event on the government of Jordan. 21 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV The evaluation team also held five focus group discussion with 22 participants (18 male and four female), representing several governmental ministries that received FRPFM support. Most were from the Ministry of Finance, one was from the Ministry of Digital Economy and Entrepreneurship, and one was from the Ministry of Interior. (See Annex 1 for the complete list of participants by position.) The participants joined the sessions in small groups or on-line. All focus group participants were requested to complete a survey before the discussions. The survey consisted of nine questions. (See Annex 2 for the discussion guide and for the questionnaire, and Annex 3 for the full survey results.) Conclusions from the focus group questionnaire: All focus group participants were government employees who had direct experience with FRPFM. Most were involved in workshops and training courses conducted by the project, and more than half felt that institutional capacity building was the most important issue addressed by the project. Many achievements were cited, but the main achievements of FRPFM from the perspective of these participants concerned institutionalized training. The most important challenge mentioned, as viewed by most, was limited project funding. Many areas are considered important for achieving sustainable results in a follow-on project. Most important of these is continued capacity building, but many mentioned continued development of e-services and continued work in automating the financial management system. Written comments regarding lessons learned and suggestions for a new project stressed better monitoring of project results, more practical and hands-on training, and closer coordination between the project and the ministry, to include more engagement with the ministry concerning project implementation. 23 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Risk Management Director MOF - DLS 19-Aug KII Director - Economic Development & Energy Office USAID 25-Aug KII DG of Customs Department MOF - Customs Department 15-Sep KII DG Assistant of Customs Department MOF - Customs Department 15-Sep KII Programme Manager - PFM European External Action Service 16-Sep KII Minister of Finance Assistant - Policy Analyst MOF 17-Sep KII Tax Institute Director MOF - ISTD 31-Aug FGD Public Administration Sector Lead MOF - GBD 31-Aug FGD Head of National Single Window MOF - Customs Department 31-Aug FGD Inspector - GFMIS MOF 31-Aug FGD Director of Planning MOF - DLS 31-Aug FGD Director of Policy and Strategy Department MODEE 31-Aug FGD Information Technology Director MOF - ISTD 2-Sep FGD DG Assistant of ISTD Advisor MOF - ISTD 2-Sep FGD Public Revenue Director MOF 2-Sep FGD Public Accounts Director MOF 2-Sep FGD Budgets Sector Lead MOF - GBD 2-Sep FGD Internal Control Director MOF 6-Sep FGD Taxpayer Services and Tax Culture Director MOF - ISTD 6-Sep FGD North Amman Tax Office Director MOF - ISTD 6-Sep FGD Head of Risk Management Division MOF - DLS 6-Sep FGD Assistant Director of the Fiscal Decentralization FDU 6-Sep FGD IPSAS PMO Head MOF 8-Sep FGD GFMIS Project Director MOF 8-Sep FGD 24 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Fiscal Decentralization Director - Local Development Department MOI 8-Sep FGD Public Expenditures Director - Treasury Department MOF 8-Sep FGD Director of National Invoicing Project MOF - ISTD 8-Sep FGD Head of Studies Department MOF - GBD 13-Sep FGD Advisor to SG for FCCL/Former Head of PPP Unit MOF 13-Sep FGD 25 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV ANNEX 2: DATA COLLECTION INSTRUMENTS FOR KIIs AND FGDs Key Informant Interviews (KIIs): Guide Interviewee`s name and position: Telephone: Email: Gender: Age: Institution: Work Address: Interviewer: Date of interview: Location of interview: Time interview started: Time interview ended: Note taker: Introduction USAID/Jordan has requested the Jordan Monitoring, Evaluation, and Learning Activity (MELA) to conduct an end-of-project performance evaluation of the Fiscal Reform and Public Financial Management (FRPFM) activity. The evaluation`s purpose is to identify the extent to which FRPFM achieved intended results as well as to assess the relevance, effectiveness, and efficiency of approaches to be used in USAID/Jordan`s new public financial management (PFM) activity. The evaluation will identify successes, best practices, lessons learned, and challenges that affected the implementation of FRPFM. The evaluation will also help the new Jordan Public Financial Management Activity (PFMA) plan its interventions. The primary audience for this evaluation is USAID/Jordan, particularly the Economic Development and Growth Office. This evaluation team is Dr. John Crihfield, Mr. Fadi Ali Hamad, Ms. Mai Khader, and Ms. Afnan al Hadidi. We are carrying out data collection in Amman and elsewhere in Jordan through key informant interviews and focus group discussions. We have questions (and follow-up questions, as needed) to pose to you and very much appreciate your frank and candid responses. Nondisclosure: All comments made in key informant interviews and in focus group discussions shall remain unattributed. Findings will be disaggregated by gender, age, geographical location, and institutions, where feasible. 26 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV We have a set of questions that we would like to ask you. Please take your time in answering. Please ask us to clarify, if necessary. This is an opportunity for the evaluation team to listen and to note your candid views. 1. Please tell us briefly about your responsibilities, and the type of involvement that you have had with USAID’s most recent fiscal reform project? We would like to go from the more general to the more specific, so we will begin with some fairly general questions: 2. What are the most important lessons learned from USAID’s Fiscal Reform and Financial Management project (FRPFM)? What were its major challenges? 3. As this project ends, and from the perspective of the government of Jordan (GOJ), what are the government’s most important fiscal reform priorities? 4. What are the most important priorities that USAID’s new public financial management project should address? What are the major challenges the new project will likely face? 5.. Does the MOF, and especially the ISTD (Income and Sales Tax Department), the GBD (General Budget Department), and the SEPD (Studies and Economic Policy Directorate) have the capacity to produce fiscal analyses for outside stakeholders, such as the IMF? (Please give examples) 6.. To what extent has the current project strengthened gender equality by promoting opportunities for advancing women in the MOF and in society more generally? (Please give examples) 7. Did the project support other USAID and USG objectives in Jordan (e.g., through U.S. government conditions precedent and the IMF program for Jordan?) 8. The current project did not have embedded advisors in the MOF. Prior USAID PFM projects did have embedded advisors (such as Fiscal Reform Projects I and II from 2006 to 2016). From your perspective, which approach was more effective? How does the embedded staff affect the sustainability of activities? 9a. How will the COVID-19 pandemic affect the government of Jordan`s ability to pursue PFM reforms in the short term? How about somewhat longer term over the next two or three years? 9b. How does the pandemic affect the fiscal sustainability of Jordan? Let us turn to some more specific questions. If a question does not apply to your responsibilities or expertise, just pass on it. 27 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV 10. Does the MOF (and its related directorates) have the capacity and commitment to work in the following areas? As you answer, please consider progress made during the current project, and work that remains to be done. Results oriented budgeting? Gender responsive budgeting? IPSAS (international public-sector accounting standard) and accrual accounting? An automated budget planning process? (This refers to the budget module Hyperion in GFMIS.) An IT system for the ISTD (Income and Sales Tax Department? (This refers to a new Revenue Management System.) An e-procurement system? Decentralized budgeting for the governorates? Fiscal policy formulation? Management of general government debt? Increased public-private dialogue and outreach? Institutionalized training? How feasible would it be to reorganize tax administration along function lines? Is this already being done? 11. Do you have any other comments about the ending project? Thank you. We greatly appreciate your time and interest in USAID programming in Jordan. 28 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Focus Group Discussion (FGD): Guide Sign-in sheet: Name of participant and contact information FGD # Date: Facilitator: Other evaluation team members: Note taker: Location of meeting: Time FGD started: Time FGD ended: Introduction USAID/Jordan has requested the Jordan Monitoring, Evaluation, and Learning Activity (MELA) to conduct an end-of-project performance evaluation of the Fiscal Reform and Public Financial Management (FRPFM) activity. The evaluation`s purpose is to identify the extent to which FRPFM achieved intended results as well as to assess the relevance, effectiveness, and efficiency of approaches to be used in USAID/Jordan`s new public financial management activity. The evaluation will identify successes, best practices, lessons learned, and challenges that affected the implementation of FRPFM. The evaluation will also help the new Jordan Public Financial Management Activity (PFMA) plan its interventions. The primary audience for this evaluation is USAID/Jordan, particularly the Economic Growth Office. This evaluation team is Dr. John Crihfield, Mr. Fadi Hamad, Ms. Mai Khader, and Ms. Afnan al Hadidi. We are carrying out data collection in Amman and elsewhere in Jordan through key informant interviews and focus group discussions. We have questions (and follow-up questions, as needed) to pose to you and very much appreciate your frank and candid responses. Nondisclosure: All comments made in key informant interviews and in focus group discussions shall remain unattributed. Findings will be disaggregated by gender, age, geographical location, and institutions, where feasible. 29 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Questionnaire for the Focus Groups Name: Position: Gender: Age: Work Address: Questions Approx. time (minutes) 1. Introductions, and explain your involvement with FRPFM 10 □ Worked for FRPFM contractor □ Government of Jordan employee □ Donor organization (for example, USAID, EU) □ Grant participant, or participant in grant activity □ Other (please specify) 2. How did FRPFM build capacity in your organization? 15 (Please check all that apply.) □ Training courses □ On-the-job training □ Workshops □ Study visits □ Assisting in preparing manuals or forms □ Assisting with studies, assessments, and data □ Other (please specify) 3. Did FRPFM address the most important fiscal reform issues in your organization in the areas below? (Please check all areas that apply.) 10 □ Tax reform □ Budget reform □ Fiscal sustainability □ Institutional capacity building 30 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV □ Other (please specify) 4. What were the main achievements of FRPFM in your organization? (Please check all that apply.) 25 □ Results-oriented budgeting □ Gender-responsive budgeting □ IPSAS and accrual accounting □ An automated budget planning process □ An IT system for ISTD □ An e-procurement system □ Decentralized budgeting at governorate offices □ Fiscal policy formulation □ Increased public-private dialogue and outreach □ Institutionalized training □ Other (please specify) 5. Were there challenges with project implementation? (Please check all that apply; circle the most important.) 15 □ Legal barriers □ Technical difficulty of project activities □ Coordination between the project and the MOF □ Time to implement project activities □ FRPFM project leadership □ Changes in leadership at the MOF □ Limitations in project funding □ Other (please specify) 31 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV 6. USAID intends to put in place a follow-on public financial management project. What should be done in the new project to ensure sustainability of results? (Please check all that apply.) 15 □ Extend the duration of the project □ Better coordination with MOF counterparts □ Place embedded advisors in the MOF □ Continue automating the financial management system □ Continue developing e-services □ More community outreach □ Continue building capacity within the MOF □ Continue providing grants to the nongovernmental sector □ Other (please specify) 7. How did FRPFM address gender aspects of public financial management in Jordan? (Please check all that apply.) 10 □ Gender responsive budgeting □ Female participation in the workforce □ Greater female participation in leadership positions □ Other (please specify) 8. What were the most important lessons learned from FRPFM? Please specify the most important. 9. What are your most important suggestions for improving the new project? Please specify the most important. 32 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV Key Informant Interviews (KIIs): Guide Interviewee`s name and position: Telephone: Email: Gender: Age: Institution: Work Address: اسم ومنصب الشخص الذي تتم مقابلته: هاتف او جوال: البريد اإللكتروني: الجنس: العمر: المؤسسة: عنوان العمل: Interviewer: Date of interview: Location of interview: Time interview started: Time interview ended: Note taker: المحاور: تاريخ المقابلة: مكان المقابلة: توقيت بداية المقابلة: توقيت نهاية المقابلة: آخذ المالحظات: Introduction USAID/Jordan has requested the Jordan Monitoring, Evaluation, and Learning Activity (MELA) to conduct an end-of-project performance evaluation of the Fiscal Reform and Public Financial Management (FRPFM) activity. The evaluation`s purpose is to identify the extent to which FRPFM achieved intended results as well as to assess the relevance, effectiveness, and efficiency of approaches to be used in USAID/Jordan`s new public financial management (PFM) activity. طلبت الوكالة األمريكية للتنمية الدولية / األردن من مشروع ,Evaluation ,Monitoring Jordan the MELA (Activity Learning and )الرصد والتقييم والتعلم في األردن )MELA )إجراء تقييم ألداء نهاية المشروع اإلصالح المالي وإدارة المالية العامة ) FRPFM). الغرض من التقييم هو تحديد مدى تحقيق FRPFM للنتائج المرجوة باإلضافة إلى تقييم مدى مالءمة وفعالية وكفاءة األساليب التي سيتم استخدامها في المشروع الجديد لإلدارة المالية العامة الممول من الوكالة األمريكية للتنمية الدولية / األردن )PFM). 33 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV The evaluation will identify successes, best practices, lessons learned, and challenges that affected the implementation of FRPFM. The evaluation will also help the new Jordan Public Financial Management Activity (PFMA) plan its interventions. The primary audience for this evaluation is USAID/Jordan, particularly the Economic Development and Growth Office. سيحدد التقييم النجاحات ، وأفضل الممارسات ، والدروس المستفادة ، والتحديات التي أثرت على تنفيذ مشروع FRPFM ً . سيساعد التقييم أيضا مشروع اصالح اإلدارة المالية العامة األردني الجديد )PFMA )على تخطيط انشطته. الجهة الطالبة لهذا التقييم هي الوكالة األمريكية للتنمية الدولية / األردن ، وال سيما مكتب التنمية والنمو االقتصادي. This evaluation team is Dr. John Crihfield, Mr. Fadi Ali Hamad, Ms. Mai Khader, and Ms. Afnan al Hadidi. We are carrying out data collection in Amman and elsewhere in Jordan through key informant interviews and focus group discussions. We have questions (and follow-up questions, as needed) to pose to you and very much appreciate your frank and candid responses. فريق التقييم هذا هو الدكتور جون كريفيلد ، والسيد فادي حمد ، والسيدة مي خضر ، واآلنسة أفنان الحديدي. نقوم بجمع البيانات في عمان وأماكن أخرى في األردن من خالل مقابالت مع الخبراء الرئيسيين ومناقشات مجموعات ً التركيز. لدينا مجموعة من األسئلة نطرحها عليك ونقدر كثيرا ردودك الصريحة. Nondisclosure: All comments made in key informant interviews and in focus group discussions shall remain unattributed. Findings will be disaggregated by gender, age, geographical location, and institutions, where feasible. عدم اإلفصاح: يجب أن تظل جميع التعليقات التي تم اإلدالء بها في المقابالت مع الخبراء الرئيسيين وفي مناقشات مجموعة التر كيز دون إسناد. سيتم تصنيف النتائج حسب الجنس والعمر والموقع الجغرافي والمؤسسات ، حيثما أمكن ذلك. تعريف على أعضاء الفريق: ثم تعريف من قبل الشخص الذي تتم مقابلته: We have a set of questions that we would like to ask you. Please take your time in answering. Please ask us to clarify, if necessary. This is an opportunity for the evaluation team to listen and to note your candid views. لدينا مجموعة من األسئلة التي نود طرحها عليك. من فضلك خذ وقتك في الرد يرجى طلب التوضيح ، إذا لزم األمر. هذه فرصة لفر يق التقييم لالستماع ومالحظة آرائك الصريحة. 1. Please tell us briefly about your responsibilities, and the type of involvement that you have had with USAID’s most recent fiscal reform project? .1 من فضلك أخبرنا بإيجاز عن مسؤولياتك ، ونوع مشاركتك مع أحدث مشروع إصالح مالي للوكالة األمريكية للتنمية الدولية؟ 34 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV We would like to go from the more general to the more specific, so we will begin with some fairly general questions: نود أن ننتقل من العام إلى األكثر تحديدًا ، لذلك سنبدأ ببعض األسئلة العامة إلى حد ما: يرجى االخذ باالعتبار عند اإلجابة اإلشارة إلى: - ما تم إنجازه - ما هو المطلوب عمله في المشروع القادم - ما هي المعيقات 2. What are the most important lessons learned from USAID’s Fiscal Reform and Financial Management project (FRPFM)? What were its major challenges? .2 ما هي أهم الدروس المستفادة من مشروع اإلصالح المالي واإلدارة المالية )FRPFM )التابع للوكالة األمريكية للتنمية الدولية؟ ما هي التحديات الرئيسية؟ 3. As this project ends, and from the perspective of the government of Jordan (GOJ), what are the government’s most important fiscal reform priorities? .3 مع انتهاء هذا المشروع ، ومن وجهة نظر الحكومة األردنية ، ما هي أهم أولويات اإلصالح المالي للحكومة؟ 4. What are the most important priorities that USAID’s new public financial management project should address? What are the major challenges the new project will likely face? .4 ما هي أهم األولويات التي يجب أن يتناولها مشروع إدارة المالية العامة الجديد التابع للوكالة األمريكية للتنمية الدولية؟ ما هي التحديات الرئيسية التي من المحتمل أن يواجهها المشروع الجديد؟ 5.. Does the MOF, and especially the ISTD (Income and Sales Tax Department), the GBD (General Budget Department), and the SEPD (Studies and Economic Policy Directorate) have the capacity to produce fiscal analyses for outside stakeholders, such as the IMF? (Please give examples) 5 .. هل تمتلك وزارة المالية ، وال سيما دائرة ضريبة الدخل والمبيعات )ISTD )، ودائرة الموازنة العامة )GD )، وإدارة الدراسات والسياسات االقتصادية )SEPD )القدرة على إنتاج التحليالت المالية ألصحاب المصلحة الخارجيين ، مثل صندوق النقد الدولي؟ )يرجى إعطاء أمثلة( 6.. To what extent has the current project strengthened gender equality by promoting opportunities for advancing women in the MOF and in society more generally? (Please give examples) .6 إلى أي مدى عزز المشروع الحالي المساواة بين الجنسين من خالل تعزيز فرص النهوض بالمرأة في وزارة المالية وفي المجتمع بشكل عام؟ )يرجى إعطاء أمثلة( 7. Did the project support other USAID and USG objectives in Jordan (e.g., through U.S. government conditions precedent and the IMF program for Jordan?) .7 هل دعم المشروع أهداف الوكالة األمريكية للتنمية الدولية وحكومة الواليات المتحدة األخرى في األردن )على سبيل المثال ، من خالل الظروف السابقة للحكومة األمريكية وبرنامج صندوق النقد الدولي لألردن؟( 8. The current project did not have embedded advisors in the MOF. Prior USAID PFM projects did have embedded advisors (such as Fiscal Reform Projects I and II from 2006 to 2016). From your perspective, which approach was more effective? How does the embedded staff affect the sustainability of activities? 35 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV .8 لم يكن لدى المشروع الحالي مستشارون في وزارة المالية. كان لدى مشاريع اإلدارة المالية العامة السابقة للوكالة األمريكية للتنمية الدولية مستشارون )مثل مشروعي اإلصالح المالي األول والثاني من 2006 إلى 2016(. من وجهة نظرك ، أي نهج كان أكثر فعالية؟ كيف يؤثر طاقم العمل على استدامة األنشطة؟ 9a. How will the COVID-19 pandemic affect the government of Jordan`s ability to pursue PFM reforms in the short term? How about somewhat longer term over the next two or three years? 9 أ. كيف سيؤثر جائحة COVID19- على قدرة الحكومة األردنية على متابعة إصالحات إدارة الشؤون المالية العامة على المدى القصير؟ ماذا عن المدى الطويل إلى حد ما خالل العامين أو الثالثة أعوام القادمة؟ 9b. How does the pandemic affect the fiscal sustainability of Jordan? 9 ب. كيف يؤثر الوباء على االستدامة المالية لألردن؟ Let us turn to some more specific questions. If a question does not apply to your responsibilities or expertise, just pass on it. دعونا ننتقل إلى بعض األسئلة األكثر تحديدًا. إذا كان السؤال ال ينطبق على مسؤولياتك أو خبرتك ، فما عليك سوى تمريره. 10. Does the MOF (and its related directorates) have the capacity and commitment to work in the following areas? As you answer, please consider progress made during the current project, and work that remains to be done. Results oriented budgeting? Gender responsive budgeting? IPSAS (international public-sector accounting standard) and accrual accounting? An automated budget planning process? (This refers to the budget module Hyperion in GFMIS.) An IT system for the ISTD (Income and Sales Tax Department? (This refers to a new Revenue Management System.) An e-procurement system? Decentralized budgeting for the governorates? Fiscal policy formulation? Management of general government debt? Increased public-private dialogue and outreach? Institutionalized training? How feasible would it be to reorganize tax administration along function lines? Is this already being done? .10 هل تمتلك وزارة المالية )والمديريات التابعة لها( القدرة وااللتزام بالعمل في المجاالت التالية؟ كما تجيب ، يرجى النظر في التقدم المحرز خالل المشروع الحالي ، والعمل الذي ال يزال يتعين القيام به. • الموازنة الموجهة نحو النتائج؟ • الموازنة المستجيبة للنوع االجتماعي؟ • المعايير المحاسبية الدولية للقطاع العام )المعايير المحاسبية الدولية للقطاع العام( والمحاسبة على أساس االستحقاق؟ • عملية تخطيط الموازنة المحوسبة؟ )يشير هذا إلى وحدة الموازنة Hyperion في GFMIS. ) • نظام تكنولوجيا المعلومات لـ ISTD( دائرة ضريبة الدخل والمبيعات؟ )يشير هذا إلى نظام إدارة اإليرادات الجديد.( • نظام الشراء إاللكتروني؟ 36 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV • الموازنة الالمركزية للمحافظات؟ • صياغة سياسة المالية العامة؟ • إدارة الدين الحكومي العام؟ • زيادة الحوار والتوعية بين القطاعين العام والخاص؟ • التدريب المؤسسي؟ ًا للخطوط الوظيفية؟ هل هذا تم بالفعل؟ • ما مدى جدوى إعادة تنظيم اإلدارة الضريبية وفق 11. Do you have any other comments about the ending project? .11 هل لديك أية مالحظات أخرى حول انتهاء المشروع؟ Thank you. We greatly appreciate your time and interest in USAID programming in Jordan. شكرا لك. نحن نقدر بشدة وقتك واهتمامك ببرامج الوكالة األمريكية للتنمية الدولية في األردن. 37 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV يرجى التأكد من تسجيل االسم على نموذج أسماء المشاركين باالجتماع. مالحظة: "الخانات المعلمة باالصفر سيتم تعبئتها من قبل فريق التقييم" رقم االجتماع __________: تاريخ االجتماع: /8/2020 إسم ميسر الجلسة___________ : أسماء أعضاء فريق التقييم اآلخرون _______________: إسم مسجل المالحظات_________________: مكان االجتماع ____________________: وقت بداية االجتماع: ___________ وقت نهاية االجتماع: ___________ المقدمة: طلبت الوكالة األمريكية للتنمية الدولية في األردن من مشروع ,Monitoring Jordan (MELA (Activity Learning and ,Evaluation ، إجراء تقييم الداء مشروع إصالح وإدارة المالية العامة )FRPFM )والغرض من التقييم هو تحديد مدى تحقيق مشروع FRPFM للنتائج المرجوة منه وكذلك تقييم مدى صلة وفعالية وكفاءة المناهج التي سيتم استخدامها في األنشطة )المشاريع( الجديدة للوكالة األمريكية للتنمية الدولية / األردن. سيحدد التقييم النجاحات وأفضل الممارسات والدروس المستفادة والتحديات التي أثرت على تنفيذ مشروع FRPFM ً كما وسيساعد التقييم أيضا مشروع اإلدارة المالية العامة األردنية الجديد )PFMA )على تخطيط تدخالته. المهتم الرئيس بهذا التقييم هي الوكالة األمريكية للتنمية الدولية في األردن وبشكل خاص مكتب التنمية االقتصادية في الوكالة. فريق التقييم مكون من الدكتور جون كريفيلد، والسيد فادي حمد، والسيدة مي خضر، واآلنسة أفنان الحديدي. يقوم الفريق بجمع البيانات من عمان ومن أماكن أخرى في المملكة من خالل المقابالت مع الخبراء الرئيسيين واجتماعات مع مجموعات التركيز )مناقشات جماعية(. من خالل طرح مجموعة من األسئلة )ويمكن أن تكون هناك أسئلة الحقة إذا اقتضت الحاجة لذلك( تتطرح عليكم ونقدر لكم ردودكم الصريحة. سرية المعلومات: جميع االفادات التي يتم إجراؤها في المقابالت مع الخبراء الرئيسيين وفي المناقشات الجماعية المركزة هي سرية، ويمكن أن يتم تصنيف النتائج حسب الجنس والعمر والموقع الجغرافي والمؤسسات إذا أمكن ذلك . 38 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV استبيان المناقشات الخاصة بمجموعات التركيز االسم: الوظيفة: الجنس: العمر: عنوان العمل: األسئلة الوقت التقريبي المقدر لإلجابة )بالدقائق( .1 تقديم، وشرح عن دورك في مشروع إصالح وإدارة المالية العامة: 10 □عملت/ي لدى المشروع. □موظف/ة في الحكومة األردنية □أعمل في منظمة مانحة )مثل الوكالة األمريكية للتنمية الدولية أو االتحاد األوروبي( □مشارك/ة من خالل المنح، أو في نشاط المنح مع المشروع ___________________________________________ □ أخرى )يرجى التحديد( .2 كيف قام مشروع إصالح وإدارة المالية العامة ببناء القدرات في مؤسستك؟ 15 )يرجى اختيار كل ما ينطبق(. □دورات تدريبية □التدريب أثناء العمل □ورش عمل □زيارات عمل )لالطالع على تجارب الدول( □المساعدة في إعداد ادلة العمل أو النماذج □المساعدة في إعداد الدراسات والتقييمات والبيانات ___________________________________________ □أخرى )يرجى التحديد( 39 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV .3 هل عالج المشروع أهم قضايا اإلصالح المالي في مؤسستك في العناوين المبينة ادناه؟ )يرجى التحقق من جميع المجاالت التي تنطبق(. 10 □اإلصالح الضريبي □إصالح الموازنة □االستدامة المالية □بناء القدرات المؤسساتية __________________________________________ □أخرى )يرجى التحديد( .4 ما هي اإلنجازات الرئيسية للمشروع FRPFM في مؤسستك؟ )يرجى تحديد كل ما ينطبق(. 25 □الموازنة الموجهة نحو النتائج □الموازنة المستجيبة للنوع االجتماعي □المعايير المحاسبية الدول ية للقطاع العام والمحاسبة على أساس االستحقاق □حوسبة عملية التخطيط وإدارة الموازنة □نظام ضريبي محوسب لإلدارة العامة لضريبة الدخل والمبيعات □نظام مشتريات إلكتروني □الموازنة الالمركزية في مكاتب المحافظات □ صياغة السياسة المالية □دعم الحوار والتواصل بين القطاعين العام والخاص □تدريب مؤسسي _______________________________________________ □أخرى )يرجى التحديد( 40 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV .5 هل كانت هناك تحديات في تنفيذ المشروع؟ )يرجى تحديد كل ما ينطبق ؛ ضع دائرة حول أهمها.( 15 □الحواجز القانونية □الصعوبة الفنية ألنشطة المشروع □التن سيق بين المشروع ووزارة المالية □محدودية المدة لتنفيذ أنشطة المشروع □قيادة مشروع FRPFM □التغييرات في القيادة بوزارة المالية □محدودية تمويل المشروع ________________________________________ □أخرى )يرجى التحديد( .6 تعتزم الوكالة األمريكية للتنمية الدولية تنفيذ مشروع جديد الستكمال أنشطة إصالح اإلدارة المالية العامة، ما الذي يجب فعله في المشروع الجديد لضمان استدامة النتائج؟ )يرجى تحديد كل ما ينطبق(. 15 □تمديد مدة المشروع لفترة أطول. □تنسيق أفضل مع الشركاء في وزارة المالية □تعيين المستشارين المضمنين Advisors Embedded في وزارة المالية □االستمرار في أتمتة نظام اإلدارة المالية □مواصلة تطوير الخدمات اإللكترونية □المزيد من التواصل مع المجتمع □مواصلة بناء القدرات داخل وزارة المالية □االستمرار في تقديم المنح للقطاع غير الحكومي ________________________________________________ □أخرى )يرجى التحديد( _ 41 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV .7 كيف عالج مشروع FRPFM الجوانب الجنسانية )النوع االجتماعي( في اإلدارة المالية العامة في االردن؟ )يرجى تحديد كل ما ينطبق(. 10 □الموازنة المستجيبة للنوع االجتماعي □مشاركة المرأة في القوى العاملة □زيادة مشاركة المرأة في المناصب القيادية بالوزارة ________________________________________ □أخرى )يرجى التحديد( .8 ما هي أهم الدروس والعبر المستفادة من مشروع إصالح وإدارة المالية العامة؟ يرجى تحديد األهم. 10 .9 ما هي أهم اقتراحاتكم لتحسين المشروع الجديد؟ 10 42 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV أسماء المشاركين في اجتماع مجموعة مناقشة وتقييم أداء مشروع إصالح وإدارة المالية العامة تاريخ االجتماع: ___2020/8/ مكان عقد االجتماع: __________ رقم االجتماع: _____ رقم الهاتف االيميل المتنقل االسم الوظيفة اسم المؤسسة 43 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV ANNEX 3: SURVEY RESULTS Introduction As part of the end-of-project performance evaluation for the Fiscal Reform and Public Financial Management (FRPFM) project, the evaluation team held five focus group discussion during the period August – September 2020, with 22 participants (18 male and four female), representing several governmental ministries that received FRPFM support. Most were from the Ministry of Finance, one was from the Ministry of Digital Economy and Entrepreneurship, and one was from the Ministry of Interior. The participants joined the sessions in small groups or on-line. All participants were requested to complete a survey before the discussions. The survey consisted of nine questions. (See Annex 2 for the questions.) All participants were government of Jordan employees, as indicated in responses to the first question. Of the remaining eight questions, six were multiple-choice, where the respondent could choose more than one answer or write down comments. Questions eight and nine were open-ended. Below is an analysis of questions two through nine using the numbering on the survey:28 Q#2: How did FRPFM build capacity in your organization? 91% identified workshops and 77% identified training courses as FRPFM’s main capacity-building approaches; about half indicated assisting with studies or on-job training, and 41% indicated assisting in preparing manuals or forms. Only three reported study visits as part of the capacity-building approach used by FRPFM. One respondent proposed periodic meetings and another proposed that participation in the implementation of plans and actions could be other capacity building options. Q#3: Did FRPFM address the most important fiscal reform issues in your organization in the areas below? About half of the respondents reported institutional capacity building as the most important fiscal reform issue addressed, while 24% 28 Most survey questions were answered. There were only a few questions in some of the surveys that were unanswered. □ Tax reform □ Budget reform □ Fiscal sustainability □ Institutional capacity building 24% 19% 14% 52% Most Important Fiscal Reform Issues □ Training courses □ On-the-job training □ Workshops □ Study visits □ Assisting in preparing manuals or forms □ Assisting with studies, assessments, and data 77% 50% 91% 14% 41% 55% Capacity Building 44 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV reported tax reform. The least important in the list was fiscal sustainability, with only 14%. In their written comments, the respondents identified the following as important issues addressed by FRPFM: - The application of International Public Sector Accounting Standards (IPSAS) to accrual accounting, and helping to bring the Internal Control Directorate closer to undertaking an internal audit function. - The project addressed fiscal gaps that affect decentralization by issuing regulations; drafting laws, manuals, and guidebooks; and regulating financial matters. - Fiscal management and general government. - The project assisted the Ministry of Digital Economy and Entrepreneurship in preparing a study to restructure the telecommunications sector, reviewing the tax structure and refunds, and assisting the ministry in developing recommendations. Q#4: What were the main achievements of FRPFM in your organization? 43% of respondents identified institutional training as one of the project's main achievements, while about a quarter reported ROB, fiscal policy formulation, and increased public￾private dialogue and outreach. Smaller percentages identified IPSAS and accrual accounting, and IT system for ISTD, decentralized budgeting, and e-procurement as achievements. Only 5% listed an automated budget planning process, while none listed gender￾responsive budgeting as one of FRPFM achievements. In their written comments, respondents identified the following as notable achievements of the project: - The GFMIS activity improved the dashboard system that helps the decision-makers get financial developments in real-time, and developed a methodology for preparing government financial reports at the general government level. - Developing the PPP guide, conducting feasibility studies for PPP projects, and supporting contracting authorities (line ministries, local governments, and other government institutions) through the PPP unit in providing technical support on PPP projects. - The transition from internal control to internal audit function. 45 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV - Assisting the Ministry of Digital Economy and Entrepreneurship in preparing a study to restructure the telecommunications sector. - Conducting a study to assess the risks and requirements of computerized systems. Q#5: Were there challenges with project implementation? Limitations in project funding was indicated as the main challenge according to half of the respondents; legal barriers, technical difficulty of project activities, and time to implement project activities were indicated by about 30% of the participants. 10% perceived the FRPFM project leadership as one of the challenges. Through written comments the following were mentioned by the respondents as challenges that project implementation faced: - Project support was limited to mostly technical support. This limited the ability of the project to reach departmental goals, where gaps and needs were not addressed. - There were weaknesses in dealing with fiscal decentralization related to the tension between the Ministry of Finance, the reform project, and the governorates councils. Q#6: USAID intends to put in place a follow-on public financial management project. What should be done in the new project to ensure sustainability of results? The majority agreed that continued capacity building within the MOF is needed to sustain results. More than half indicated the need to extend the project's duration, continue automating the GFMIS, and developing e￾services. Community outreach was only indicated by 15% of respondents. In written comments respondents stated that projects should be implemented in an integrated manner with clear goals, performance indicators, and allocations of funds, and should adopt a participatory approach in developing □ Extend the duration of the project □ Better coordination with MOF counterparts □ Place embedded advisors in the MOF □ Continue automating the financial management system □ Continue developing e-services □ More community outreach □ Continue building capacity within the MOF □ Continue providing grants to the nongovernmental … 55% 30% 25% 50% 60% 15% 80% 35% Results Sustainability □ Legal barriers □ Technical difficulty of project activities □ Coordination between the project and the MOF □ Time to implement project activities □ FRPFM project leadership □ Changes in leadership at the MOF □ Limitations in project funding 38% 38% 19% 33% 10% 24% 52% Challenges 46 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV plans and activity monitoring; quarterly assessments should address deviations from plans. Q#7: How did FRPFM address gender aspects of public financial management in Jordan? Only 13 of the 22 respondents answered this question, out of which 54% indicated gender-responsive budgeting and female participation in the workforce as to how the activity addressed gender. 46% reported FRPFM addressed female participation in leadership positions. One respondent identified increasing female participation in capacity building programs had been important, while another mentioned that there was no addressing of gender. Q#8: What were the most important lessons learned from FRPFM? 15 participants answered this open-ended question. The following points summarize their responses as to lessons learned. - Technical support and reporting are not useful if not accompanied by implementation on the ground. Technical support should be institutionalized. - The project successfully contributed to building capacity of MOF staff on project development; how to benefit from funding opportunities; how to positively benefit from international experiences and best practices; contributing to stabilizing the financial situation in general and responding to the government requirements under national priorities. - The cost of interventions must consider the value of intended results; high costs must be linked to results with high added values. Recommendations should not remain on paper only. - Coordination between project management and relevant stakeholders needs to be not limited to the minister and secretary general level; it also needs to involve department directors. Some of the answers indicated the need to focus on strategic and operational planning based on clear and measurable performance indicators to reach institutional goals successfully. - FRPFM succeeded in focused support, for example, the governance and institutionalization of tax work, automating the tax work, results￾oriented budgeting, assisting in the electronic transfer of procedures, and preparing the unified chart of accounts to follow up the government's financial and operations within the general budget law. That was only done through close follow-up by the project, considering all points of view, and understanding the project's role relates it to the need to develop the public sector. Q#9: What are your most important suggestions for improving the new project? Please specify the most important. □ Gender responsive budgeting □ Female participation in the workforce □ Greater female participation in leadership positions 54% 54% 46% Gender Aspects of Public Financial Management 47 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV - The need to have projects with a comprehensive approach with achievable results responding to needs and recommendations. Adopt a joint management approach, create a base for the administrative units to manage or coordinate projects within the ministry, sharing the project's operational plan to the higher administrators of the relevant departments, including the performance indicators of the project in their annual plans. - Implementation should be accompanied by continuous evaluation, quarterly evaluation in coordination with the project implementing party, USAID, and the MOF. - Study the current situation before starting implementation of any new project through direct contact with those concerned for any project expected to be implemented through the upcoming financial reform program. - Advisors and project staff should adopt an active and practical role, not limited to just giving instructions and suggesting programs. Advisors should be oriented on the work culture in Jordan and the government work environment to work effectively and make full use of the available resources to achieve the goals. - Consider allocating part of the support to improve the work environment in the ministry. - Focus more on developing institutional capacity within the MOF, granting senior management “buy in” to develop skills and focus on the importance of automation and the use of international standards at work. Training should be more practical and include learning experiences from other similar countries, granting smooth transition from local to international standards, and providing adequate support for this process. - Respondents provided specific comments for technical areas that need to be included in any future support: o Continuing to develop mechanisms for collecting local revenues electronically. o Develop and implement all requirements for the transition to the accrual basis, support with experts in IPSAS and accrual accounting, in addition to building an electronic system that works on the accrual basis. o Work to support women and gender-related issues. o Automate all financial operations, starting from preparing the budget to issuing financial reports. o Develop management supplies and inventory systems, connected with the e-procurement. o Establishing a department or agency concentrating in identifying and risk management. Conclusions All focus group participants were government employees who had direct experience with FRPFM. Most were involved in workshops and training courses conducted by the project, and more than half felt that institutional capacity building was the most important issue addressed by the project. Many achievements were cited, but the main achievements of FRPFM from the perspective of these participants concerned institutionalized training. The most important challenge mentioned, as viewed by 48 | FRPFM EVALUATION AND ASSESSMENT PHASE 2 USAID.GOV most, was limited project funding. Many areas are considered important for achieving sustainable results in a follow-on project. Most important of these is continued capacity building, but many mentioned continued development of e￾services and continued work in automating the financial management system. Written comments regarding lessons learned and suggestions for a new project stressed better monitoring of project results, more practical and hands-on training, and closer coordination between the project and the ministry, to include more engagement with the ministry concerning project implementation.