February 17, 2022 This report was produced for review by the United States Agency for International Development (USAID). It was prepared by International Business & Technical Consultants, Inc. (IBTCI), under the Monitoring and Evaluation in Tunisia and Libya (METAL) Activity (AID-280-TO-17-00001). EVALUATION INTERIM EVALUATION OF LIBYA PUBLIC FINANCIAL MANAGEMENT (LPFM) INTERIM EVALUATION OF LIBYA PUBLIC FINANCIAL MANAGEMENT (LPFM) February 17, 2022 International Business & Technical Consultants, Inc. (IBTCI) In the U.S.: 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1.703.749.0100 In Tunisia: 1er Étage, Blue Bloc Vital Cube Building 4 L’Ile de Tasmanie Les Berges du Lac 2 1053 Tunis, Tunisia AID-280-TO-17-00001 Cover photo: GECOL Revenue Enhancement Workshop, November 2019 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of IBTCI and do not necessarily reflect the views of USAID or the United States Government (USG). 1 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 ABSTRACT The Libya Public Financial Management (LPFM) Activity is a five-year initiative with $18.6 million allocated for the first two base years, from September 2019 to September 2021. It addresses critical macro-fiscal, electricity sector, and private sector constraints facing the Libyan economy. The USAID/Libya Economic Growth (EG) team requested an interim evaluation of LPFM to assess how effective the activity has been to date, to inform USAID efforts to advance Libya’s self-reliance and stability. LPFM appears to be largely successful because it will have completed most planned tasks by year-end and made significant gains in its three component areas of strengthening public financial management, modernizing the electricity sector, and improving the business enabling environment. The range of achievements is impressive considering its operating environment: working in a fragile context it has had to cope with a civil war, a pandemic, and a new government. Despite being off-site, USAID stands out as a reliable, credible, and trusted partner. A few LPFM performance indicators are likely to be missed due to pandemic restrictions and political instability. The quality of the LPFM team is critical for driving overall performance. LPFM has a robust, reliable, and responsive Monitoring, Evaluation, and Learning (MEL) system that was adjusted to account for the turmoil in Fiscal Year (FY) 2020. Training was effective in transferring new skills and knowledge to beneficiaries, but many recipients were not yet applying them mostly due to external factors. LPFM’s approach to ensure that training is used and adopted by partners has been affected by COVID-19 and the change of government. There is scope to improve training. Although the conflict and COVID-19 made some technical work and management more difficult, LPFM found “work-arounds” so that results were minimally affected, and it continued to operate when other donors’ activities were suspended. 2 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 TABLE OF CONTENTS ABSTRACT 1 ACRONYMS 4 I. EXECUTIVE SUMMARY 7 A. BACKGROUND 7 B. EVALUATION PURPOSE 7 C. EVALUATION DESIGN 7 D. KEY FINDINGS AND CONCLUSIONS (SUMMARY) 7 E. KEY RECOMMENDATIONS 13 F. INFOGRAPHICS ON FINDINGS AND RECOMMENDATIONS 15 II. INTRODUCTION 18 III. BACKGROUND 18 IV. EVALUATION PURPOSE AND EVALUATION QUESTIONS 19 V. EVALUATION DESIGN, METHODS, AND LIMITATIONS 20 VI. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 22 EVALUATION QUESTION 1 23 EVALUATION QUESTION 2 33 EVALUATION QUESTION 3 35 EVALUATION QUESTION 4 37 EVALUATION QUESTION 5 39 EVALUATION QUESTION 6 41 Annex A: LPFM Logic Model 44 Annex B: Scope of Work 45 Annex C: Data Collection Instruments 53 Annex D: LPFM Work streams and Planned Results (FY 2021 WP) 56 Annex E: Success Stories (Highlights) 61 Annex F: Data Collection and Analysis Plan 63 Annex G: Sources of Information 65 Annex H: LPFM Municipality Locations 71 Annex I: LPFM Training Overview 72 Annex J: Performance Indicator Tracking Table 75 Annex K: Findings-Conclusions-Recommendations 77 Annex L: LPFM Performance Data Flow 89 Annex M: Evaluation Team Member Qualifications 90 3 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 LIST OF FIGURES Figure 1. LPFM Purpose and Results Statement............................................................................................... 18 Figure 2. Interview Categories and Distribution.............................................................................................. 20 Figure 3. An Approach to Improve Training Relevance and Utility............................................................. 41 LIST OF TABLES Table 1. Activity Summary Table ......................................................................................................................... 19 Table 2. LPFM Incomplete Deliverables............................................................................................................. 23 4 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 ACRONYMS ACS Account Code Structure ADS Automated Directives System AmCham American Chamber of Commerce in Libya AMELP Activity Monitoring, Evaluation, and Learning Plan BEE Business Enabling Environment BPMU Budget Planning and Monitoring Unit C&C Conflict and COVID-19 CBL Central Bank of Libya COP Chief of Party COR Contracting Officer’s Representative CSO Civil Society Organization DCOP Deputy Chief of Party DO Development Objective DQA Data Quality Assessment EG Economic Governance EQ Evaluation Question ES Electricity Sector ET Evaluation Team FGD Focus Group Discussions FY Fiscal Year GAM Government Accounting Manual GDSC Government Decision Support Center GECOL General Electricity Company of Libya GFMIS Government-wide Integrated MIS GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Agency for Technical Cooperation) GNA Government National Accord GNU Government of National Unity GOL Government of Libya GUC Grants Under Contracts HCLA High Council for Local Administration HOR House of Representatives IGF Inter-Governmental Finance IMF International Monetary Fund IP Implementing Partner IPSAS International Public Sector Accounting Standards IR Independent Regulator IT Information Technology KII Key Informant Interview KWh Kilowatt Hour LAB Libyan Audit Bureau LBC Libyan Business Council LCA Libyan Customs Authority LED Light-Emitting Diode LEGS Libyan Emergency Grid Stabilization Program LELSA Libya Elections and Legislative Strengthening Activity LESP Libya Economic Stabilization Program (pilot of MEG) 5 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 LGCS Local Governance and Civil Society (Activity) LOC Letter of Cooperation LOP Life of Project LPFM USAID’s Libya Public Financial Management (Activity) LSO Legal Support Office M&E Monitoring and Evaluation MEG Middle East Economic Growth (Best Practice Project) MEL Monitoring, Evaluation, and Learning (Plan) MERP Middle East Regional Platform METAL Monitoring and Evaluation for Tunisia and Libya (Activity) MFTR Memorandum on Foreign Trade Regime MFU Macro Fiscal Unit (MOF) MIS Management Information System MOE Ministry of Environment MOED Ministry of Education MOET Ministry of Economy and Trade MOF Ministry of Finance MOH Ministry of Health MOLG Ministry of Local Government MOP Ministry of Planning MTB Medium-Term Budget MTEF Medium-Term Expenditure Framework NGO Nongovernmental Organization OSR Own Source Revenue PC Presidential Council PE Primary Education PEFA Public Expenditure and Financial Accountability (framework) PFM Public Financial Management PHC Primary Health Care PIP Public Investment Program PITT Performance Indicator Tracking Table PMO Prime Minister’s Office PPD Public–Private Dialogue PPP Public–Private Partnership PSAP Private Sector Advocacy Platform PSD Public Sanitation Department QPR Quarterly Performance Report REAOL Renewable Energy Authority of Libya SOE State-Owned Enterprises SOP Standard Operating Procedures SOW Scope of Work STTA Short-Term Technical Assistance SWM Solid Waste Management TA Technical Assistance TFESR Task Force for Electricity Sector Reforms TL Team Leader TPM Third-party Monitoring TWG Technical Working Group 6 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 UN United Nations UNDP United Nations Development Programme UNSMIL United Nations Support Mission in Libya USAID United States Agency for International Development USG United States Government WB World Bank WP Work Plan WTO World Trade Organization 7 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 I. EXECUTIVE SUMMARY A. BACKGROUND On July 26, 2021, USAID/Libya submitted a scope of work (SOW) to the Monitoring and Evaluation for Tunisia and Libya Activity (METAL) to conduct an interim evaluation of the Libya Public Financial Management Activity (LPFM). Implemented by the Pragma Corporation, LPFM is a five-year initiative, with $18 million allocated for a two-year base period, from September 2019 to September 2021. The purpose of LPFM is to advance Libya’s national stability by strengthening the Government of Libya’s (GOL) public financial management (PFM) policies and processes to strengthen the fiscal foundations for sustainable and inclusive growth, providing technical assistance (TA) and capacity-building for Libya’s electricity sector (ES), and improving Libya’s business enabling environment (BEE) to stimulate growth of the private sector. B. EVALUATION PURPOSE The evaluation SOW (Annex B) states that the purpose of the LPFM interim evaluation is to inform the USAID/Libya Economic Growth (EG) team on how effective the activity has been to date in supporting USAID’s efforts to advance Libya’s self-reliance and stability. Evaluation recommendations will inform USAID to make changes to the existing activity, ensuring that USAID’s efforts are as effective as possible given the challenging operational environment. The main audience is the USAID/Libya EG team and the broader USAID/Libya team, though the reports may also be shared in full or in part with the Libya External Office (LEO), USAID implementing partners (IPs), and the Middle East Bureau. C. EVALUATION DESIGN The evaluation team (ET) gathered data and information from a desk review of activity and other pertinent documents; conducted 70 key informant interviews (KIIs) with LPFM activity staff, counterparts, and other stakeholders; and held five focus group discussions (FGDs) with activity beneficiaries. D. KEY FINDINGS AND CONCLUSIONS (SUMMARY) Evaluation Question 1: 1a) How effective has LPFM been to date in achieving the award’s three objectives? 1b) What challenges did LPFM face in achieving the objectives? 1c) In which areas was LPFM the most successful? 1a. Effectiveness ▪ Many activities and tasks were achieved, but some fell short of work plan targets, largely due to GOL slowness in reviewing and approving draft documents. ▪ Reasons why some interventions were not fully completed or initiated are shown in Table 2. 1b. Challenges ▪ Libya, as a fragile state, has serious shortcomings with respect to peace, justice, and institutional framework. ▪ Major challenges were conflict/security, the COVID-19 pandemic, and change of government. 8 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 ▪ Shuffles within the General Electricity Company of Libya (GECOL) board meant a loss of continuity and experience. 1c. Most successful areas Through its PFM national sub-component, LPFM: ▪ Created a Medium-Term Expenditure Framework for sector budgeting in Solid Waste Management (SWM), Primary Health Care (PHC), and Primary Education (PE) to use resources more effectively. ▪ Drafted Medium-Term Budgets (MTBs) at the Ministry of Planning (MOP) for public investments to repair infrastructure, in collaboration with the Ministry of Finance (MOF), for public investments to repair infrastructure. ▪ Supported the Macro Fiscal Unit (MFU) of the MOF in the preparation of macro-fiscal reports and budget analysis for the Ministry. Through its PFM subnational sub-component, LPFM: ▪ Assisted 46 municipalities in the preparation of Own Source Revenue (OSR) budgets to provide extra resources for public services. The recent Cabinet Regulation authorizing the decentralization of funding to municipalities and the collection of OSRs is a big win for LPFM. ▪ Supported pilot municipalities in the development of MTBs for operations and capital spending for SWM, PHC, and PE. ▪ Developed an inter-governmental finance grants (IGF) formula for calculating municipality grants allocation. ▪ Trained 60 civil society organizations (CSOs) in “citizen budgeting” for greater financial transparency and accountability. Through its Electricity sector component, LPFM: ▪ Revised a law on the electricity sector to restructure the General Electricity Company of Libya (GECOL), to set up an independent energy sector regulator, to encourage private sector investments in generating capacity, and to allow leasing of GECOL’s 4,000-km fiber optic network to a telecom operator. ▪ Developed GECOL’s financial unbundling/restructuring proposal/strategy as well as financial asset valuation guidelines. ▪ Assisted GECOL to identify the first wave of 4,000 positions to be shifted to non-core businesses. ▪ Developed a web-based “dashboard” system for GECOL in billing and collections and in procurement and maintenance. ▪ Provided a grant to local nongovernmental organization (NGO) Moomken, for an energy conservation public awareness campaign. ▪ Secured the agreement of the National Standards Bureau regarding high-efficiency specifications for appliances. ▪ Conducted a Light Emitting Diode (LED) replacement program involving the private sector. ▪ Led a multi-donor effort to implement the Libyan Emergency Grid Stabilization (LEGS) program to reduce outages. Through its Business Enabling Environment component (BEE), LPFM: ▪ Developed the Libyan Business Council (LBC) e-legislation database on business laws to better inform foreign investors. ▪ Created a Private Sector Advocacy Platform at LBC to help identify private sector issues. 9 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 ▪ Crafted a Memo on Foreign Trade Regime (MFTR), a first step in World Trade Organization (WTO) accession. ▪ Drafted 30 legislative, regulatory, and institutional proposals on commercial law reform. ▪ Trained 60 public lawyers on legislative drafting to improve legal writing skills. ▪ Trained the Government Decision Support Centre (GDSC) in organizing three high-level public–private dialogue (PPD) events to help identify BEE reform priorities ▪ Conducted legal and regulatory assessment of more than 65 laws in trade and investment and compared them to international standards and best practices. LPFM appears to be largely successful because it will have completed 17 of 26 planned tasks by year￾end and the nine incomplete tasks are mostly due to external factors. LPFM aims for fundamental transformation in many initiatives but the shift in government priorities and counterparts undermined this effort. Evaluation Question 2: 2a) Compared to donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? 2b) In which areas is USAID’s assistance addressing gaps not met by other donors? 2a) USAID comparative advantages ▪ USAID has a strong track record in Libya and through LPFM is addressing top priorities. ▪ USAID is seen as a leader: other donors follow LPFM strategy on SWM and the UNDP calls on USAID for advice on renewable energy. ▪ USAID is seen as stable and reliable with extensive counterparts in many ministries. 2b) Where USAID is addressing gaps ▪ LPFM is working in areas not targeted by other donors: SWM, PE, and PHC at the local level. ▪ A Letter of Cooperation (LOC) between USAID and MOF/GOL was signed to fill “gaps” such as Libyan Customs Authority (LCA), Central Bank of Libya (CBL), and Libyan Audit Bureau (LAB). ▪ The LOC also helped USAID/LPFM to coordinate with the World Bank and “sort out mutual interests.” ▪ LPFM is complementary to another USAID intervention, the Local Governance and Civil Society (LGCS) Activity: LPFM assists municipalities to budget and LGCS helps them spend the money on community priorities. ▪ LPFM collaborates with French and German donor agencies on BEE and SWM, respectively. ▪ USAID helps raise Libyan government credibility and legitimacy through LPFM and other initiatives such as the Libya Elections and Legislative Strengthening Activity (LELSA) and LGCS that are working to strengthen civil society and the capacity of national and subnational governments. USAID has enabled LPFM to be flexible and demand-driven to address GOL priority areas (e.g., a contracting and procurements dashboard was not in the initial work plan). Despite not being on the ground in Libya, USAID can leverage LPFM’s extensive network of contacts and focus on areas neglected by other donors such as the business side of electricity, municipal financial management, and linking national to subnational PFM. Other donors look to USAID for leadership in SWM and in renewable energy. 10 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 Evaluation Question 3: 3a) To what extent is LPFM on track to meet its targets? 3b) What factors have led to its success in meeting targets? 3c) What factors led to missing targets? 3a) Extent on track to meet targets ▪ The LPFM Monitoring, Evaluation, and Learning (MEL) team is tracking 20 performance indicators, of which 19 have targets. ▪ Performance on 11 indicators are on track (two indicator targets were achieved and nine were exceeded), while five will likely be achieved in Q4. On the other hand, two indicator targets were not achieved, and one has a data lag. The lone context indicator is not expected to have a target. 3b) Success factors ▪ Targets are for the most part “realistic and therefore achievable.” ▪ Flexible training, such as switching to virtual training when COVID restrictions precluded in￾person training. ▪ Local advisors/LPFM staff provide the follow-up on training, e.g., on-the-job training. ▪ LPFM senior staff are highly experienced and knowledgeable experts in their field. ▪ Many local staff worked on the Libyan Economic Stabilization Program (LESP) that proceeded LPFM, allowing for continuity. ▪ Local staff are technically competent and coached by LPFM international experts. ▪ Selected experts (and local advisors) are embedded in partner institutions. ▪ Counterpart relations are established with reform-minded elements. ▪ Technical Working Groups (TWGs) are the “working level” for coordination purposes. ▪ The implementing partner mastered the requisite technology to manage virtual platforms for meetings and training. ▪ LPFM’s small grants facilitate planned deliverables. 3c) Factors leading to missed targets ▪ The pandemic slowed and delayed activities, as LPFM advisors were unable to travel to Libya, in country advisors were unable to meet in person with members of GECOL and other national, and local government entities, and on-site technical assistance could not be delivered. ▪ International experts could not travel to Libya, nor could Libyans travel to Tunis. ▪ Many BEE activities were considered “non-training” events and were not captured by MEL because of the MEL training indicator’s definition. ▪ During conflict, the power grid was damaged so a new workstream was added for the electricity sector. The expertise, commitment, and diligence of the LPFM team is considered critical for driving performance. Pandemic restrictions combined with political instability and uncertainty were the main factors that contributed to LPFM missing a few targets. 11 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 Evaluation Question 4: 4a) LPFM has a detailed activity monitoring, evaluation, and learning plan (AMELP). To what extent did the activity successfully collect data and monitor activities as described in the AMELP? 4b) What challenges did LPFM face in activity monitoring, both the monitoring it managed and in working with USAID/Libya’s monitoring and evaluation contractor on third-party monitoring? 4a) Monitoring, Evaluation, and Learning (MEL) System ▪ The Monitoring and Evaluation for Tunisia and Libya Activity’s (METAL) November 2020 Data Quality Assessment (DQA) Report stated that “LPFM’s MEL protocols and Standard Operating Procedures (SOP) constitute a good practice that should be shared with other IPs.” ▪ The MEL system features a manual, quality assurance control, and weekly data updates shared with LPFM management for informed decisions. ▪ There have been constant improvements to the system which are evidence of adaptive management. ▪ The system is secure, and access is limited only to key personnel. ▪ MEL team is responsive to both USAID and METAL requests for data. ▪ MEL team is doing perception surveys on views of electricity tariff increases and BEE. 4b) Internal monitoring and third-party monitoring (TPM) ▪ After the FY 2020 revision to the work plan (WP), indicators had to be modified. ▪ Remote processes and procedures were added to the MEL system. ▪ Setting up multiple access points was an Information Technology (IT) challenge. ▪ There is effective coordination between LPFM and METAL. ▪ Changing events and times is a challenge for the planning of monitoring. ▪ In general, the monitoring reports and feedback are positive. ▪ 150 TPM reports have been sent from METAL to the Contracting Officer’s Representative (COR) to confirm that LPFM events took place. The MEL system is thorough, robust, and responsive, and performance data is reliable. Due to the turbulence in FY 2020, indicators were modified, and then remote processes and procedures were developed. While frequent changes to event scheduling creates challenges, TPM runs smoothly, in part due to the close collaboration between LPFM MEL and METAL. Evaluation Question 5: 5a) To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? 5b) What has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? 5c) How can training be improved going forward to improve relevance and utility? 12 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 5a) Individual training ▪ LPFM is focused on reforms supported through capacity development. ▪ Feedback from counterparts and beneficiaries on training delivery and content was positive, but a recent report on capacity development found only half of trainees had applied acquired know-how on the job, due largely to external factors related to their current work. ▪ Training recipients who said they had not yet had an opportunity to apply the know-how expected to be able to in the future. ▪ COVID-19 was a significant barrier to applying acquired know-how on the job, as trained personnel were precluded from working in the office where the equipment and systems they were trained on were located. ▪ LPFM’s training approach is aligned with best practice approaches as recommended by EuropeAID, Swedish Development Cooperation, and the Organization for Economic Co￾operation and Development (OECD). 5b) Training adopted by government entities ▪ LPFM provided on-the-job training/coaching to trainees as a follow-up to formal training. ▪ LPFM also set up TWGs to facilitate the development and implementation of capacity development plans. ▪ LPFM engaged with counterparts on its 28 workstreams but a change of government in March resulted in many new counterparts who had to be educated about LPFM to obtain their “buy￾in.” ▪ Despite the impediments resulting from COVID-19 and the change in government, many interventions went well, and some training skills and knowledge were applied so that institutions benefited. ▪ LPFM does not capture or report data on the nature and scope of critical training follow-up. 5c) Improving training ▪ Training relevance and utility were enhanced through host government lead, written agreements, deep analysis of context and needs, fully engaging reform-minded counterparts, custom solutions (tools, TA, and training), delivery by suitable experts, Training of Trainers, targeting relevant recipients, follow-up (on-the-job training), expanded MEL, and digital archiving. Figure 3 elaborates this approach. Although trainees stated that training content and delivery was good, about half the trainees indicated that they are not yet using the newly acquired know-how because COVID-19 restrictions kept them home, limiting their ability to apply new skills. To ensure that training is used and adopted, LPFM has provided follow-up coaching to training recipients, embedded experts in partner institutions, set up TWGs, and engaged extensively with counterparts. Evaluation Question 6: 6a) How have the conflict and the COVID-19 pandemic impacted implementation of the project? 6b) What adjustments did LPFM make in response to the conflict and the pandemic? 6c) How did the conflict and the pandemic impact the achievement of the project’s objectives? 6d) How did the project demonstrate resilience/flexibility in response to these challenges? 6a) Conflict and COVID-19 (C&C) impact on implementation ▪ LPFM implementation was slowed, and technical work became more difficult. 13 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 ▪ Half of the LPFM staff and many counterparts suffered from the COVID-19 virus. ▪ Government officials avoided in-person meetings, offices were closed, and people were not answering phones. ▪ Staff had to work from home and carry out meetings and interviews virtually. ▪ This made it harder to get data and do research. ▪ Expats were evacuated and could not meet face-to-face with counterparts. 6b) Adjustments to C&C ▪ Virtual meetings and training and off-site meetings were held; staff worked from home. ▪ Virtual MEL team adjusted data collection method and evidence. ▪ Staff recorded sessions and made them available to participants. ▪ Meetings and training were done late at night when electricity and internet was better. ▪ Internal travel restrictions were mitigated through use of family and clan ties to cross into the East. ▪ LPFM enhanced security protocols, security bulletins, close protection, etc. 6c) C&C effect on results ▪ LPFM was able to achieve most results, but technical work was made more difficult. ▪ International experts could not provide TA or training on-site. ▪ New Government of National Unity (GNU) and associated political instability are a challenge to achieving results. 6d) Resilience/flexibility ▪ Staff remained committed and diligent and drew on their networks. ▪ LPFM’s good relationship with counterparts allowed them to continue activities during the pandemic. ▪ Effective workarounds (e.g., shift to virtual platforms and remote management) were used. ▪ Local staff/advisors could step in for international experts. ▪ LPFM continued to operate while other projects suspended activities. C&C slowed LPFM implementation by making technical work and activity management more difficult. But LPFM found effective workarounds to mitigate the worst effects. Effects of COVID-19 and the conflict therefore became manageable and minimally affected LPFM overall results. E. KEY RECOMMENDATIONS LPFM should do the following: 1. Report on deliverables status in Quarterly Performance Reports using annual work plan tables and reference numbers. (EQ1a) 2. Continue to engage with reformist elements in Libyan partner institutions in a meaningful and consistent manner. (EQ1b) 3. LPFM senior leadership should continue to ensure that field staff have the authority and responsibility to make decisions relevant to their scope and work plan engagement. (EQ1b) 14 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 4. Continue to demonstrate flexibility by accommodating new requests for assistance from counterparts that respond to GOL’s changing priorities, as appropriate and within LPFM scope of work. (EQ1b) 5. LPFM leadership and activity managers should continue coordination and communication efforts with donors and donor projects in Libya that are active in related fields to avoid duplication of effort. (EQ1b) 6. Continue to deliver anti-corruption training in close collaboration with public financial institutions in Libya such as the MOF and CBL. (EQ1b) 7. Continue to promote successes using all media sources to reach target audiences. (EQ1c) 8. Develop a gender strategy, and with USAID formulate a social inclusion indicator with targets. (EQ3a) 9. Continue to put concerted effort into training up local staff to take on the roles of experts denied access to Libya. (EQ3c) 10. Capture data in a systematic manner to report on the nature and scope of critical training follow-up (e.g., on-the-job coaching) as well as other non-training events that have learning value (e.g., working sessions and technical meetings). (EQ4a) 11. Carry out more focus group discussions with training recipients to determine the extent that training know-how is used and adopted in the workplace. (EQ4a) 12. There are no specific recommendations for this question as the LPFM training approach is already aligned with best practice approaches as recommended by EuropeAID, Swedish Development Cooperation, and the Organization for Economic Co-operation and Development (OECD). (EQ5c) 13. Continue to be adaptable and flexible to mitigate the effects of conflict and COVID-19. (EQ6b) 14. Maintain a dynamic, innovative team with strong counterpart relationships and home office support to overcome the debilitating effects of the conflict and COVID-19. (EQ6d) USAID should do the following: 1. Allow LPFM to continue incomplete tasks into FY 2022 so that it can bring all its reform initiatives to fruition. (EQ1a) 2. Continue to focus on priority needs that are neglected by other donors. (EQ2a) 3. Organize a donor forum in Libya on a regular basis for enhanced donor coordination. (EQ2b) 4. Encourage LPFM to respond to host government requests for assistance within. (EQ2b) 5. Allow an Activity to adjust indicators and targets when the situation deteriorates and is warranted. (EQ3a) Pragma should do the following: 1. Continue to maintain the priority of bringing senior field leadership back into Libya when the situation on the ground permits, for hands-on management and better counterpart relations. (EQ1b) 2. Continue to aim for a stable staff complement, especially at the management level, to maintain continuity in counterpart relations, teamwork, and Activity institutional memory. (Q6a) 15 | P a g e LPFM Interim Evaluation Report – METAL AID-280-TO-17-00001 F. INFOGRAPHICS ON FINDINGS AND RECOMMENDATIONS 16 | P a g e LPFM Interim Evaluation Report – METAL AID -280 -TO -17 -00001 17 | P a g e LPFM Interim Evaluation Report – METAL AID -280 -TO -17 -00001 18 II. INTRODUCTION On July 26, 2021, USAID/Libya submitted a scope of work (SOW) to the Monitoring and Evaluation for Tunisia and Libya Activity (METAL) to conduct an interim evaluation of the Libya Public Financial Management Activity (LPFM). Implemented by the Pragma Corporation, LPFM is a five-year initiative, with $18 million allocated for a two-year base period, from September 2019 to September 2021. The Libya Public Financial Management (LPFM) Activity is designed to address critical macro-fiscal, electricity sector, and private sector constraints facing the Libyan economy.1 This document is an interim evaluation report of LPFM as described in the USAID/Libya scope of work (Annex B). III. BACKGROUND Libya’s political instability is exacerbated by its inability to manage its oil wealth efficiently and fairly, in a transparent and accountable manner, and for the benefit of all its citizens. Even before the recent challenges of the escalated conflict and the coronavirus pandemic, modern public financial management (PFM) was almost completely lacking in Libya, as demonstrated by Libya’s weak ability to budget for basic services, including reliable electricity. Heavily subsidized electricity and gasoline, combined with a massive public sector payroll, further stress the country’s fiscal health. Implemented by the Pragma Corporation, LPFM supports USAID’s development hypothesis2 that if the Government of Libya (GOL) can achieve greater efficiency and effectiveness in the management of public finances, including more efficient management of the electricity sector, then macroeconomic stability will be restored, creating the fiscal space for greater public investment in critical services and infrastructure, thereby increasing Libya’s political and social stability. It further hypothesizes that if Libya can create a business environment that is conducive to economic growth and job creation, then this will help to legitimize the state and increase political and economic stability. The LPFM Logic Model is in Annex A, and the results framework is in Figure 1, below. Figure 1. LPFM Purpose and Results Statement LPFM builds upon the PFM and Electricity Sector reform assistance provided previously by USAID’s Libya Economic Stabilization Program (LESP).3 1 FY 2020 QPR-4 p. 1. 2 FY 2020 Work Plan 1st Revision January 2020 p. 1 and FY 2021 AMELP p. 7. 3 FY 2020 Work Plan 1st Revision January 2020 p. 1–4. 19 Table 1. Activity Summary Table Activity Name Libya Public Financial Management (LPFM) USAID Office USAID/Libya Implementer The Pragma Corporation Contract # [Task Order] 7200AA18D00016 / 72067019F00001 Total Estimated Ceiling (TEC) of the Evaluated Activity $14.9 million, total estimated cost up to $55 million over five years Life of Activity September 16, 2019–September 15, 2021 (end date of September 15, 2024, if all option years are exercised) Active Geographic Regions Nationwide Development Objectives (DOs) DO1: Effectiveness of key government institutions improved. DO2: Libyans empowered to mitigate drivers of instability and conflict. IV. EVALUATION PURPOSE AND EVALUATION QUESTIONS Evaluation Purpose On July 26, 2021, USAID/Libya submitted a scope of work (SOW) to the Monitoring and Evaluation for Tunisia and Libya Activity (METAL) to conduct an interim evaluation of the Libya Public Financial Management Activity (LPFM). The SOW states that the purpose of the LPFM interim evaluation is to inform the USAID/Libya Economic Governance (EG) team on how effective the activity has been to date in supporting USAID’s efforts to advance Libya’s self-reliance and stability. Specifically, the interim performance evaluation will identify the extent to which LPFM is achieving intended results over the base period (effectiveness), as well as assess the relevance and efficiency of the approaches used to achieve USAID/Libya objectives. Evaluation recommendations will inform USAID to make changes to the existing activity, ensuring that USAID’s efforts are as effective as possible given the challenging operational environment, and inform future PFM activities. The main audience is the USAID/Libya EG team and the broader USAID/Libya team. Evaluation Questions The SOW for the LPFM interim evaluation specifies six evaluation questions (EQs): EQ 1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? EQ 2: Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? In which areas is USAID’s assistance addressing gaps not met by other donors? EQ 3: To what extent is LPFM on track to meet its targets? What factors have led to its success in meeting targets? What factors led to missing targets? EQ 4: LPFM has a detailed activity monitoring, evaluation, and learning plan (AMELP). To what extent did the activity successfully collect data and monitor activities as described in the AMELP? What challenges did LPFM face in activity monitoring, both the monitoring it managed and in working with USAID/Libya’s monitoring and evaluation contractor on third-party monitoring? EQ 5: To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? If so, what has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? How can training be improved going forward to improve relevance and utility? EQ 6: How have the conflict and the COVID-19 pandemic impacted implementation of the project? What adjustments did LPFM make in response to the conflict and the pandemic? How did the conflict 20 and the pandemic impact the achievement of the pr4oject’s objectives? How did the project demonstrate resilience/flexibility in response to these challenges? V. EVALUATION DESIGN, METHODS, AND LIMITATIONS The aim of this interim evaluation is to produce coherent answers to key questions posed in the evaluation SOW based on sound evidence and analysis. This section describes the methodology that was followed, i.e., what was done to develop evidence-based findings to deduce conclusions (interpretation of facts) and to make recommendations (judgment). Data Collection Methods Data and information were collected from three main sources: desk review of primarily LPFM documentation, focus group discussions (FGDs), and key informant interviews (KIIs). The Evaluation Team (ET) engaged with USAID at each step of the process: during a kickoff meeting held July 30, a mid-briefing during fieldwork, and an exit-brief presentation on preliminary findings held September 17, 2021. All evaluation activities were consistent with USAID’s Evaluation Policy (January 2011; updated October 2016) and Automated Directives System (ADS) 201. Secondary Data Sources Desk review data and information were drawn primarily from LPFM activity files. Specific LPFM documents provided by USAID and the IP for this desk review included the LPFM contract, Annual Work Plans, AMELPs, Quarterly Reports, and the Annual Report. The ET also reviewed technical papers, articles, documents, and reports provided by other agencies and derived from internet research, as deemed appropriate (Annex G: Sources of Information). Primary Data Sources Evaluation data and information were drawn from two primary sources: ▪ Key Informant Interviews were conducted with LPFM staff (Chief of Party [COP], Deputy Chief of Party [DCOP], Component Team Leads, MEL Manager),4 relevant Pragma staff at headquarters, relevant METAL monitoring staff, USAID representatives such as LPFM COR, senior host government officials (i.e., ministry and municipal government counterparts), Libyan sector experts, GECOL board members, other stakeholders such as chamber and business community, research institutes, civil society, and donor representatives (i.e., the European Union, World Bank, GIZ, and United Nations). The KIIs have the following distribution as shown in Figure 2. Figure 2. Interview Categories and Distribution 4 LPFM leadership interviewed more than once due to follow-up questions. 21 • Focus Group Discussions with selected direct beneficiaries in five FGDs corresponding to LPFM training output indicators: senior government officials, municipal staff, persons in the “PFM cycle,” General Electricity Company of Libya (GECOL), and Business Enabling Environment (BEE). A list of interviewees FGD participants can be found in Annex G: Sources of Information. Research Methods (Data Collection) Key Informant Interviews (KIIs) A total of 70 KIIs were conducted. A KII Guide (qualitative instrument) was drafted with open-ended questions to steer the conversation. The ET was prepared to “drill down” on specific matters (data collection instrument in Annex C). During each KII, the interviewee comments were captured with careful note taking but not recorded as this tends to limit responses. Responses were manually coded to facilitate analysis. Focus Group Discussions (FGDs) The ET conducted five FGDs with direct beneficiaries for the purpose of capturing their perceptions of LPFM activities and outputs. Ten beneficiaries were selected for each FGD from the LPFM dataset of training participants. Purposeful sampling was used to target women, multiple users, different organizations, and different locations. FGDs were conducted virtually (hosted online) and carried out in Arabic. A Focus Group Guide (qualitative instrument) was drafted containing a slightly different introduction to the KIIs; the facilitator was instructed to focus discussions primarily on EQ5 (Annex C). Data Analysis and Synthesis The methodology behind the desk review (analysis of documentation) was to gather all pertinent literature and then review, discuss, and synthesize understanding to develop a comprehensive context for the activity evaluation. Project and other relevant documents were reviewed during the inception phase and continued into the fieldwork as new documents were obtained. Data analysis for KIIs and FGDs commenced at the beginning of the fieldwork. The fieldwork commenced August 23, 2021, and was scheduled for four weeks to allow for a mid-brief and an exit brief. Data from interviews (FGDs and KIIs) were abstracted and shared among team members. A thematic analysis was run against evaluation questions throughout the fieldwork. Analysts looked for emergent themes, contextual factors, and trends or patterns in the responses. Results of the analysis informed findings, conclusions, and recommendations (Annex K). Data Quality and Limitations This section highlights possible evaluation limitations, their implications for the evaluation, and how the ET mitigated them. There are several noteworthy bias risks in data collection and analysis: ▪ Recall bias occurs when interviewees and FGD participants may not recall events and experiences clearly, if at all, so their responses may not be meaningful. To mitigate this bias, the ET conducted as many KIIs and FGDs as possible to triangulate responses to increase the validity of the evaluation findings, and probed respondents’ answers to ensure accurate recollections. ▪ Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable. ET mitigated this risk by explaining evaluation requirements clearly at the beginning of each KII or FGD. ▪ Selection bias is an inherent risk when the activity implementer refers the ET to more sympathetic and supportive counterparts or recipients. To mitigate this risk, the ET identified persons having some involvement with LPFM and arranged meetings directly without LPFM 22 assistance. The ET sought LPFM assistance to reach targeted informants only when they were hard to reach. Due to the security situation in Libya and the COVID-19 pandemic, the ET was forced to conduct interviews remotely using an online meeting platform (Google Meet), secure communications technology (WhatsApp), and occasionally, due to connectivity disruption, cell phone calls to continue interviews. The Team Leader (TL) and one ET member carried out interviews in English with LPFM staff, USAID officials, donors, and other English-speaking counterparts. Other members of the team conducted interviews and focus groups with Libyan officials and stakeholders in Arabic. Remote interviews and focus groups using video conferencing software posed unique challenges because the responses of key informants and FGD participants may be misconstrued due to the virtual nature of the interaction. For example, a lack of interpersonal interaction may mean missing out on key body language. The ET asked the IP for names of short-term technical assistance (STTA) experts as well as a description of their outputs, link to work streams, timing/duration, and delivery mode so that their specific contributions could be analyzed. Because a list of only names was received without any additional information, the ET was not able to assess the different roles played by local staff/advisors and local and international experts in the delivery of TA and training. Gender Considerations Efforts were made to ensure full participation by women in interviews and focus groups. In addition to gender and age, geographic location, and role, disaggregation was included in the data analysis, where applicable. Although social inclusion was mentioned in the Task Order, there were no indicators or targets for gender or marginalized populations. As such, there was no programmatic focus on social inclusion, only reporting on disaggregated data for gender and age in training. There were very few females as beneficiaries (in training events), despite LPFM encouraging females to attend municipal training and legislative drafting training. However, some females participated in the technical working groups, and LPFM actively promoted female leadership in CSOs, highlighted International Women’s Day, and prepared special profiles on women. In the subnational area, LPFM worked in remote (marginalized) areas such as Sabha and the East. The Data Collection and Analysis Plan in Annex F shows the ET’s overall approach to data collection (sources, tools, and methods) and data analysis (tools and methods). VI. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Data and information collected were analyzed to determine fact-based qualitative evidence, written up as concise findings in this section of the report. Conclusions are a synthesis or interpretation of the findings, while recommendations are proposed actions drawn from one or more specific findings. According to the LPFM AMELP, component outcome-level results will be achieved if lower-level outputs are achieved. Because EQ3a asks whether LPFM will meet its (mostly outcome) indicator targets, it was agreed with USAID that the ET response to EQ1 should focus on achievement of planned interventions. To help answer EQ 1, the ET prepared a table template listing over 90 tasks as per work plans and asked LPFM to populate the tables with the deliverables produced so far and to note the difference between actuals and planned. 23 EVALUATION QUESTION 1 EQ 1a. How effective has LPFM been to date in achieving the award’s three objectives? Unfortunately, the progress of deliverables was difficult to determine in the Quarterly Progress Reports (QPRs). The reporting of interventions did not differentiate activities/tasks from deliverables produced. To add to the confusion, interventions were not linked to the work plan, i.e., there were no reference numbers, nor was there an indication of how much progress had been achieved on any outstanding deliverables. The task versus work plan table was reviewed together with component leads and the COP and DCOP to determine what had been achieved under each of LPFM’s 28 work streams (Annex D). They were then validated with counterparts in interviews and confirmed in QPRs. Findings: ▪ Based upon the evidence in the task table, QPRs, and interviews with counterparts, most tasks have been achieved, which demonstrates progress in meeting LPFM’s three objectives. ▪ However, there are areas that fall short of expectations in the agreed work plans. Based on discussions with LPFM leadership, the evaluation team was able to ascertain the reasons why a number of tasks were not completed or even initiated. The table below lists the incomplete tasks and explains their status as of June 30, 2021 (FY 2021 QPR-35): Table 2. LPFM Incomplete Deliverables Work Stream # as per Work Plan Task Achieved Remaining Tasks PFM C1.1 A.1, 2, and 3 MTEF: 2022 Op. plans and Public Investment Programs (PIPs) in SWM, PHC (Ministry of Health [MOH]), and PE (Ministry of Education [MOED]) in selected municipalities Anticipate FY 2021 Q4 completion A.4 Sectoral PIPs as model for MTB for multiyear capital budgeting by MOP and MOF PIPs to be applied FY 2020 PHC or SWM Pipelined PIPs will be completed in FY 2021 Q4 B.1 State-Owned Enterprises (SOE) Budget Planning & Monitoring Unit (SOE-BPMU) established Postponed to FY 2021 Proposal counterparts endorsed Awaiting Minister signature B.2 Assist SOE-BPMU to establish a detailed budget planning and monitoring framework and Framework and process paper [game plan] Waiting for SOE Unit to be established [political decision – change of Minister delay] 5 ET asked for info and data from Q4, but LPFM said they could not provide it because activities were ongoing, and data was being gathered and needed to be validated. 24 Work Stream # as per Work Plan Task Achieved Remaining Tasks process for two major SOEs – GECOL and National Oil Co. C.2 MOH and MOED adopt, and initiate strategies and implementation plans to identify ghost workers and reduce their wage bill Change in government delayed progress – waiting for decision by GNU on downsizing pilot; head of General Information Authority (main counterpart) pressing MOF for support to carry out the task D.1 Best practice public procurement procedures in collaboration with BEE Public–Private Partnership (PPP) framework for infrastructure financing Training technical support Waiting for MOF, MOP, GDSC, and Presidential Council Legal Office (PCLO) to adopt best practices and PPP framework D.2 Harmonize Public Accounting – International Public Sector Accounting Standards (IPSAS), Government Accounting Manual (GAM), Account Code Structure (ACS) [Chart of Accounts also under MFU – see 1.3 A.2] TWG set up planning discussions gap analysis Contentious items transferred to World Bank (WB) July 20216 Based upon a Letter of Cooperation signed December 2020 D.4 Single Treasury Account Framework Best practice report training modules Transferred to WB in July 2021 D.5 Government-wide integrated management information system (GFMIS) Planning Draft tender Transferred to WB in July 2021 PFM C1.2 A.1 2021 OSR Budget Preparation in Legacy-LESP Municipalities (21) It was not possible to work on 2022 OSR budgets because MOLG stalled on issuing the 2022 budget circular 6 The formal explanation is in coordination with USAID, a determination was made during Q3 to forgo any further focus on implementation activities in these areas, pending further clarification from the World Bank and the MOF regarding future World Bank assistance in these areas. FY 2021 QPR-3 p. 9. 25 Work Stream # as per Work Plan Task Achieved Remaining Tasks - 2022 OSR budgets to be approved by Ministry of Local Government (MOLG) - Certified by MOLG/High Council for Local Administration (HCLA) and municipalities graduate from LPFM Certifying 21 legacy municipalities is not currently feasible in Libya A.8 Assist in preparing OSR budgets FY2022 for 25 second-generation municipalities LPFM pressing MOLG to issue the budget circular; 7 the no-confidence vote in the GNU of September 20, 2021, is delaying the circular issue A.9 Select 15 third-generation municipalities and initiate Public Expenditure and Financial Accountability (PEFA) assessment and compliance training Selection made PEFA assessment not done as no budget circular issued B.3 Municipal property tax policy, strategy, and implementation plan to be developed and adopted TWG First step decentralizes state budget financing to municipalities [Reg. 330 issued September 10, 2021, allows municipalities to collect and spend OSR] This item is delayed C.3h PE funding mechanism developed in pilot municipalities, and donor funding identified and leveraged The delay in approving the PE plans (OP and PIP) has put on hold any decisions on funding D With Ministry of Water Resources– develop a medium-term water and sanitation strategy, and implement in five pilot municipalities Due to the change in government and GNU opposition to developing a full Water and Sanitation strategy, a “strategic outline” has been delayed until FY 2021 Q4 7 A Budget Circular is necessary for municipalities to kick-start the budgeting process (e.g., establish budget committees, put together teams to work on expenditures and revenues, etc.). 26 Work Stream # as per Work Plan Task Achieved Remaining Tasks PFM C1.3 C.2 Fiscal report on FY 2021– 2022 developed and submitted to MOF Anticipate FY 2021 Q4 completion ES C2.1 A.2 PC adopts electricity law/decree Technical work is done; the ES law and decree were adopted by Prime Minister’s Office (PMO) The law was passed to House of Representatives (HOR) for approval and is under designated committee’s reviews; ES regulator decree has not yet been issued by PMO to the HOR for approval B.4 and 5 1 st and 2nd tariff increases Technical work is completed Proposal is with PMO, but GOL is unlikely to approve currently due to summer outages and upcoming election C.1 PC appoints key personnel of independent electricity regulator Regulator decree revised and presented to Council of Ministers Waiting for PM signature to appoint Commissioners C 3.4 Implement capacity￾development of regulator, establish twinning with best practice regulator Technical work is complete Twinning arrangement initiated Waiting for PM approval of regulator decree D.1 Restructuring/unbundling proposal finalized and submitted to the PC Strategy/plan workshops on proposal Draft law To be presented to GECOL Board in Q4, then to Presidential Council (PC)/PMO for approval D.2 Financial asset valuation guidelines developed and submitted to the PC Unbundling model training modules proposal to upgrade GECOL financial system included in LEGS Guidelines presented to Seeking PC approval8 8 Fiscal valuation work requires longer project time and specialized STTA. 27 Work Stream # as per Work Plan Task Achieved Remaining Tasks GECOL in workshops E.1 Assist GECOL workforce realignment technical committee implement the action plan 15 non-core functions identified Human Resources plan to transition workforce from 41,000 to 14,000 presented to GECOL LPFM to assist with plan implementation9 ES C2.2 G.1 Energy conservation public awareness campaign by CSO grantee Moomken delivered media training To turn over campaign material (app, billboards, eight videos, six cameras, and radio messaging) to GECOL in FY 2021 Q4 G.4 Energy Audit Service Company formed and entered a client relationship with a private sector business Energy audit group identified at GECOL performed two energy audits Spin-off to revenue- generating energy company requires PMO to approve the strategy to establish a PPP firm BEE C3 E.3 Seven critical legal/regulatory reform proposals adopted and institutional implementation Given political uncertainties and transition from GNA to GNU, finalization of drafts was delayed; however, many proposals are at the final stages of moving toward adoption F.1 Assist GOL to develop and submit Libya’s MFTR to WTO All technical work done on MFTR The last step requires a joint reading of the MFTR with LPFM, which was to be held in Tunis in August 2021; given COVID-19 travel restrictions from Libya to Tunis, the meeting did not occur Conclusion: LPFM is likely to complete most of its planned tasks by year-end. Reasons for incomplete tasks are that GOL approvals are pending, or activities have stalled due to a lack of GOL support to move forward. Five tasks are scheduled to be completed in Q4 FY21. Recommendations: The following recommendations will bring greater clarity to LPFM planning and reporting and help carry tasks forward to completion: ▪ USAID should instruct LPFM to report on deliverables in QPRs as per annual work plan tables and reference numbers – include progress, status, % complete, and explanation of delays. 9 Human resources realignment also requires longer project duration to complete the task, e.g., skills mapping. 28 ▪ USAID should instruct LPFM to break down all complex tasks into an action plan with milestone outputs. ▪ LPFM should define each deliverable clearly so that it can be measured and validated. ▪ USAID should allow LPFM to continue incomplete tasks into FY 2022 so that LPFM can bring all its reform initiatives to fruition. EQ 1b. What challenges did LPFM face in achieving the objectives? Findings: Libya is considered a fragile state10 because there are serious shortcomings with respect to peace, justice, and strong institutions.11 In addition to the weak context, LPFM has experienced a “perfect storm” of major challenges: conflict/insecurity, the COVID-19 pandemic, and a change of government. ▪ Security: Shortly after LPFM commenced operations in 2019, conflict between the East and West made it hard to access the East hard throughout FY 2020 and therefore it was difficult for LPFM Advisors and experts to engage officials there. Since the ceasefire in October 2020, movement has been much better. In October 2021, the Department of State issued a level 4 travel advisory: It advises American citizens not to travel to Libya due to crime, terrorism, civil unrest, kidnapping, and armed conflict, and to reconsider travel to Libya due to COVID-19. Visas for foreigners are difficult to obtain, which affects the ability of LPFM’s American technical consultants to travel to Libya. ▪ Pandemic: In March 2020, there was a global COVID-19 outbreak. Many LPFM staff and counterparts got sick, borders were closed, and expatriates evacuated. Events were canceled or postponed due to lockdowns and curfews, and travel/flight and visa restrictions meant international experts could not go to Libya nor Libyans to Tunis, where events are frequently held. In-person meetings with government officials were also prohibited, and many officials did not answer their phones. ▪ New Government: In March 2021, following the UN-facilitated Libyan Political Dialogue Forum, the Government of National Unity (GNU) was established as an interim government. 12 The change in government from the Government of National Accord (GNA) to the GNU resulted in a complete turnover of ministers and counterparts who did not understand and/or support LPFM reform efforts, although this was primarily an issue at the national level and did not significantly impact LPFM’s work at the subnational level with municipalities. There were also several shuffles of the GECOL board and General Assembly, so much continuity, knowledge, and experience were lost with the departure of previous board members. LPFM must build new relationships and gain the buy-in of the new government officials. In addition, the new Prime Minister advised public officials to be wary of NGOs, including donors. Public officials became reluctant to accept the increased scrutiny of engaging in donor-sponsored activities. The Ministers of Finance and Planning initially refused to meet LPFM staff. In addition, LPFM has experienced lesser challenges, as follows: ▪ Disconnect between national and subnational: As one LPFM staff person said, “There [is] political and bureaucratic resistance to devolving authority and money to municipalities.” Due to Law 59, municipalities have no legal mandate to spend money on health clinics, and the MOED does not support delegating to the local level (Previous GNA strongly supported delegation). ▪ Engaging with partners: The dynamic dialogue with counterparts on interventions is time￾consuming but necessary to discuss what is needed and what is a priority (easier at subnational 10 The Fund for Peace Fragile State Index ranks Libya 20 out of 174 countries in 2021. https://fragilestatesindex.org/2020/05/10/libya-continues-path-as-the-decades-most-worsened-country/ 11 The Center for Strategic and International Studies states there is no single definition of “fragility” so instead promotes UN Sustainable Development Goal 16 on peace, justice, and strong institutions. 12 National elections that were planned for December 2021 have been postponed. 29 level). At the national level, senior official approval of project proposals is needed. Municipal officials overall were satisfied with their engagement with LPFM staff. However, one municipal official commented that not all municipal councils/local politicians were enthusiastic about LPFM and “some did not take their proposals seriously.” One national-level ministry official stated, “Pragma staff came for his approval of several projects, but he was shocked as he had no idea about them.” The ET was also informed by Moomken, a Pragma subcontractor working with GECOL that “the SOW was not discussed and shared with GECOL, and they (GECOL) were kept in the dark13 ▪ Staffing and experts: LPFM experienced some departures in management. The initial COP left in the first quarter, the PFM TL left after six months, the BEE TL left in August 2021, and the long￾term COP departed in September 2021. Though not a significant turnover considering the challenging operating environment, it was somewhat disruptive to the Activity operations. Also, bringing foreign experts into Libya is currently problematic now due to visa issues. ▪ Getting data: Obtaining information from public institutions regarding laws, regulations, and performance data related to business and electricity, school enrollment, and other relevant areas to help inform LPFM activities was problematic. As one researcher said, “It is hard to find [data], and it’s of low quality.” The BEE component found many public institutions unresponsive to their requests for information and, especially during the pandemic, were not to be found in government offices. In sub-national SWM, one municipal official stated, “Data collection was a lot of work.” The Primary Education TWG informed the ET that they could not collect data from schools when they were shut down because of the pandemic, or due to holidays or because schools were busy with exams. ▪ Electricity blackouts: Most informants stated that power outages and unstable internet connections made remote trainings and meetings difficult. ▪ Shift to virtual platforms: Due to COVID-19 and the unstable security environment, training migrated online but many counterparts and beneficiaries were not enthusiastic. One stated, “[online training] is no substitute for the real thing” and as another pointed out, “any follow-up [training] has to be on the ground.” One counterpart commented, “TA is almost impossible to do over Zoom or Skype,” due, in part, to periodic electricity and internet outages. LPFM indicated that initially it was not easy to convince counterparts and beneficiaries to use Zoom for online meetings and training, and for some it was a technological challenge. Then, after counterparts and beneficiaries got used to it, some reported “Zoom fatigue.” 14 The evaluation team was informed by several informants that “counterparts are tired of online meetings.” In response, LPFM introduced a hybrid delivery model in 2020 combining online with in-person interaction. For meetings, workshops, and trainings, LPFM brought Libyan staff together in local venues (hotels, stakeholder offices, LPFM offices, etc.) with STTA or expat LPFM staff logging in remotely, often supported by LPFM Libyan staff to co-facilitate and conduct capacity-building efforts as mentored by LPFM expat staff. When LPFM stakeholders and counterparts could not attend in person, due to safety concerns or GOL imposed limitations on travel, the use of online platforms enabled the program to continue without interruption. ▪ Management: LPFM was providing highly technical assistance in three sectors across 28 workstreams in a very challenging environment. It was challenging to successfully ensure flexible and impactful implementation of such a broad and complex program. As Pragma informed the ET, “It is not easy to manage an omnibus project with lots of moving parts.” The need to communicate and coordinate with many ministries, counterparts, beneficiaries, and donors, as well as deal with the shifting priorities of the GOL is challenging. This became more difficult after Activity 13 LPFM indicates that the SOW for grants was not discussed in detail with counterparts to respect the integrity of the procurement process, but that the overall campaign goals were discussed with GECOL. 14 Zoom fatigue is tiredness, worry, or burnout associated with the overuse of virtual platforms of communication, particularly videoconferencing. 30 management shifted remotely in April 2020 (apart from the DCOP, who remained in Tripoli). A Pragma interviewee stated, “It is always better to be on the ground and interfacing with people.” ▪ Changing scope: Because USAID required LPFM to expand its national PFM sub-component to more agencies in FY 2020 Q1, 15 LPFM prepared a revised work plan (WP) in January 2020. That WP had to be further revised in June 2020 to reflect the changed context posed by COVID-19 and the ongoing conflict. Many activities were postponed to FY 2021. The June 2020 WP contains a 35-page table showing the shifting deliverables across the three WPs in FY 2020 (i.e., an initial WP, and two revised WPs). ▪ Budget: LPFM’s assistance consists of training and technical assistance and not the provision of equipment and supplies. Yet, several counterparts and municipal officials indicated a need for material support (e.g., GECOL Enterprise Resource Planning is outdated and uses servers instead of cloud storage). ▪ World Bank: The WB tends to focus on its strengths in national PFM and high-level dialogue. LPFM had to “go slow” in FY 2020 on certain activities in its national PFM sub-component (IPSAS, GAM, Charter of Accounts, GFMIS, Internal Review Office etc.), because, according to Pragma and WB, they were essentially “doing the same thing.” In FY 2021 LPFM ceased to implement work lines that WB sought to manage, and a Letter of Cooperation was signed in December 2020 to address duplication of effort. After a series of discussions, all duplicated activities were transferred to the WB in July 2021. ▪ Systemic reform: Due to its complexity, major reform (e.g., GECOL restructuring/ unbundling) takes time.” LPFM informed the ET that, “Reform of the energy sector can take a decade,” and “the WTO accession process usually takes five years. “One counterpart said, “Major reforms are not possible in an election year” while another said, “GNU has appetite for reform and wants tangible results as an election is due.” Conclusion: Libya is a difficult operating environment for international development assistance. On top of fragility, LPFM operated during a civil war, a pandemic, and a change of government. The shift in government priorities and counterparts was especially challenging for LPFM. Recommendations: The following recommendations aim to facilitate Activity implementation: ▪ LPFM should continue to engage with reformist elements within partner institutions in a meaningful and consistent manner. LPFM management should be prepared to “educate” new counterparts to gain their buy-in, i.e., explain the rationale for the intervention so that they are more likely to support and ideally champion the initiative. ▪ LPFM should offer more training on field-level research and data collection methods to GOL and other Libyan stakeholders. ▪ LPFM senior leadership should continue to ensure that field staff have the authority and responsibility to make decisions relevant to their scope and work plan engagement. ▪ Pragma should continue to maintain the priority of bringing senior field leadership back into Libya when the situation on the ground permits, for hands-on management and better counterpart relations. ▪ LPFM should continue to demonstrate flexibility by accommodating new requests for assistance from counterparts that respond to GOL’s changing priorities, as appropriate and within LPFM’s scope of work. ▪ LPFM leadership and activity managers should continue coordination and communication efforts with donors and donor projects in Libya that are active in related fields to avoid duplication of effort. . 31 EQ 1c. In which areas was LPFM the most successful? Findings: Despite LPFM’s relatively short duration, there are many successes confirmed by informants. See Annex E for highlighted LPFM success stories.16 Key LPFM accomplishments in the various components are as follows: PFM national: ▪ Helped set up the Medium-Term Expenditure Framework (MTEF) for GOL to use resources more effectively. It involves comprehensive sector budgeting (operational plans and investment plans) in Solid Waste Management with selected municipalities, Primary Health Care with MOH, and Primary Education with MOED (added FY 2021). ▪ Developed the budget planning and monitoring unit for State-Owned Enterprises (SOEs) for MOF, which is intended to improve SOE governance for greater transparency and disclosure. ▪ Set up a multiyear budget planning process at MOP in collaboration with MOF for public investments in infrastructure such as roads and bridges. ▪ Supported the Macro-Fiscal Unit (MFU) in preparing reports on a regular basis that were published on Facebook for the public. MFU’s budget analysis supports policy makers and the MOF to make informed decisions. The MFU prepared for the first time a fiscal framework that was used by the GOL. The Minister of Finance recently sent a letter of appreciation to the MFU.17 However, not all parties are enthusiastic about the MFU’s work: one donor remarked, “MFU reports are basic, and they need to contract economists for policy options and position papers.” ▪ In FY 2020, LPFM trained 17 government officials from the Ministry of Finance, Macro Fiscal Unit of the MOF, and Central Bank in anti-corruption areas relating to transparency, procurement, and system safeguards. PFM subnational: ▪ Assisted 46 municipalities with preparing own-source revenue (OSR) budgets. Each municipality took part in a one-week course to build their capacity in budgeting.18 OSR budgets are decentralized and transparent and according to one municipal official, “OSR opened up municipal council eyes about new sources of funding.” OSR represents extra resources for public services— for landfills, health clinics, and schools. ▪ Assisted in the development and promotion of Regulation No. 330, which was issued by the GOL Cabinet on August 23, 2021, authorizing the decentralization of funding to municipalities and the collection of OSRs; this legal mandate is widely regarded as a major win for LPFM. ▪ Supported pilot municipalities with preparing medium-term budgets (MTB) for operations and capital budgets covering Solid Waste Management, Primary Health Care, and Primary Education— so-called budgeting from the bottom up. ▪ Developed the general purpose inter-governmental finance (IGF) grants formula for allocating grants to municipalities for Solid Waste Management and Primary Health Care. The IGF not only links national to subnational governments but encouraged cooperation between MOP and MOF. ▪ Trained 60 CSOs in “citizen budgeting” for greater financial transparency and accountability. 16 LPFM success stories were promoted rigorously through “communications” – some featured on USAID/Libya and Libya External Office social media platforms while others were picked up by local media. 17 “We have reviewed the studies and reports prepared by the MFU in collaboration with USAID and within the framework of the cooperation and support agreement with the Ministry of Finance. We congratulate you for the brilliant work and efforts, I convey my thanks and gratitude to you and your collaborators and encourage you to greater success.” 18 Informant stated, “Municipalities are interested in learning OSR budgeting even if they have no budget as they consider it capacity-building. . . . Municipal staff are willing to obtain new skills and knowledge.” 32 Electricity Sector: ▪ Revised the law on the electricity sector that was approved by the GECOL board and adopted by the Presidential Council (PC). The law will restructure GECOL for greater efficiency19 and subsidy reduction, establish an independent energy sector regulator to improve oversight and mitigate corruption, allow private sector participation through PPPs to invest in additional generating capacity, and lease GECOL’s 4,000-km fiber optic network to the telecom operator to generate non-electric revenues. ▪ Developed GECOL’s financial unbundling/restructuring proposal/strategy (to spin off 15 non-core businesses) as well as financial asset valuation guidelines. 20 These were presented to the GECOL board in workshops during FY 2021 Q4, and the first wave of 4,000 positions to be shifted to non￾core businesses were identified and their training needs analyzed. ▪ Prepared a web-based billing and collection dashboard system for GECOL executive management and regional offices to enhance revenue (via increased billings and collections] and to reduce commercial losses. Also developed a procurement and maintenance dashboard for GECOL contracting department to help tackle corruption. 21 ▪ Convinced the National Standards Bureau to accept high-efficiency specifications for appliances, and these standards were adopted by Ministry of Environment (MOE). LPFM’s LED replacement program that involves the private sector has also been adopted by MOE. ▪ Led a multi-donor22 effort to implement the Libyan Emergency Grid Stabilization (LEGS) program aimed at reducing outages by bringing damaged generation capacity back online with repair and maintenance, holding workshops for GECOL on practical load shedding management to manage blackouts, and agreements with large industries to shift 500MW from peak to non-peak. ▪ In FY 2020, LPFM trained 57 GECOL officials in anti-corruption areas relating to transparency, procurement, and system safeguards. Business Enabling Environment: ▪ Developed the Libya Business Council (LBC) e-legislation database on business laws in English and Arabic to better inform foreign investors. It has been officially launched on the LBC website and will be maintained by LBC. ▪ Established a Private Sector Advocacy Platform (PSAP) at LBC to help identify private sector issues and lobby for solutions. ▪ Crafted a Memorandum on Foreign Trade Regime (MFTR)—the first serious step in WTO accession. 23 This application is a significant milestone for Libya to boost trade and investment. LPFM’s research and analysis supporting this effort required the review of 150 laws and regulations, including revising customs law compliance. ▪ Developed and submitted over 30 draft legislative, regulatory, and institutional proposals, and submitted them to the Government Decision Support Centre Reform Secretariat and to the Ministry of Economy and Trade (MOET).24 19 LPFM TA to GECOL aimed at addressing inefficiencies and corruption has somehow validated GECOL requests to GOL for subsidies and payments or subsidies were “unlocked.” 20 IP engaged the services of an international consultant to carry out financial valuations in FY2021 Q4, but due to travel restrictions, the expert was only able to provide book value valuations. 21 Anti-corruption training was offered across all components – see Annex I table column 1.1c 22 So-called “crisis” group of United Nations Support Mission in Libya (UNSMIL), UNDP, WB, and EU. 23 ET informed that the process can be fast-tracked if the host government is committed and focused. 24 Such as Law on Secured Transactions; Foreign Investment Registration Regulations; Law on Public Private Partnerships; Law on Public Procurement; OMNIBUS Decision on trade liberalization (18 reforms); Decision on Establishing Inter￾Ministerial Commission on WTO Accession; Decision for Establishing Formal Consultation Mechanism; Decision for Establishing National Committee on Trade Facilitation; Amendments to the Customs Law; Risk Management Guidelines; Recommendations for Enhancing LCA Website; Comprehensive Integrated Tariff System; and Proposal for New Organizational Structure for the LCA. 33 ▪ Conducted a systematic legal and regulatory assessment of more than 65 laws and sub-legal acts governing the trade, investment, and business environment and compared them to best practice and international standards.25 ▪ Arranged legislative drafting training for 60 public lawyers—legal writing skills are essential for quality laws and junior female lawyers were targeted. ▪ Trained the GDSC in high-level events management through organizing three PPD events,26 which will help identify reform priorities in BEE. Conclusion: LPFM successes are impressive considering it works across three significant sectors in a highly challenging environment. Despite structural reform taking considerable time, LPFM has done significant foundational work in several areas. It aims for fundamental transformation in many initiatives—for instance, work on regulatory frameworks is a good entry point and constitutes a “pathway” for the next wave of reforms. Assuming a stable political economy, especially post-election, LPFM’s significant momentum should bring systemic reforms to fruition in FY 2022. Recommendations: ▪ LPFM should continue to promote successes using all media sources to reach target audiences. ▪ LPFM should continue to expand anti-corruption training to other municipalities and additional national government institutions. EVALUATION QUESTION 2 EQ 2a. Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? Findings: ▪ Because there is no U.S. Embassy in Libya,27 USAID is forced to work “off-site” (like the WB), and as such is not able to closely monitor the situation on the ground and build close relations with senior officials in the GOL. ▪ Despite this handicap, through its various economic governance support activities in Libya since 2011, USAID has a strong track record and is regarded as stable and reliable while other donors come and go. ▪ USAID/LPFM is seen as addressing Libya’s top priorities: macro-fiscal, the business side of electricity, and decentralization. As one municipal official said, “They focus on the municipalities and no other donor focuses on improving Solid Waste Management, Primary Education, and Primary Health Care at [the] local level.” Another stated, “USAID is way ahead of other donors.” 28 ▪ USAID is seen as a leader in the donor community in certain priority areas. For example, other donors follow LPFM’s strategy developed with MOLG to improve municipalities’ management of solid waste, and the UN calls on USAID/LPFM for advice and guidance on nascent renewable energy. ▪ The GOL wanted deeper involvement with USAID/LPFM on national PFM because their assistance is seen as actionable and impactful. In the words of one municipal official, “They (Pragma) focus on practical topics such as Own Source Revenue,” whereas other donors “just talk.” 25 The BEE team developed a detailed action plan with 176 reform recommendations; a 197-page report was submitted to GDSC and the MOET. 26 Public–private dialogue (PPD) has been picked up by the Organization for Economic Co-operation and Development (OECD), who wants to establish a permanent platform for identifying private sector issues. 27 Germany and France have embassies in Tripoli. The EU has a delegation. 28 This official is likely referring to LPFM support in financial planning, budgeting and procurement/outsourcing at the local level in key sectors, and not GIZ’s material support to SWM in selected municipalities). 34 ▪ USAID is also known for promoting private sector approaches (e.g., in SWM LPFM promotes outsourcing garbage collection through public tenders compared to GIZ buying trucks for municipalities). ▪ LPFM introduces best practices and quality outputs through international experts. One municipal official stated that the OSR budget they prepared with LPFM support was a “great report.” ▪ Through LPFM, USAID has strong links to numerous counterparts in many ministries such as MOLG, MOP, MOF, MOET, and Ministry of Health (MOH). One counterpart stated, “USAID is seen as having no hidden agenda, unlike Turkey.” Conclusion: Despite not being on the ground in Libya, through LPFM USAID can leverage an extensive network of contacts and focus on areas neglected by other donors such as the business side of electricity, municipal financial management, business enabling environment, and linking national to subnational PFM. Other donors look to USAID for leadership in SWM and renewable energy. Recommendations: The following recommendations aim to safeguard and enhance USAID’s reputation in Libya and build on its solid track record. ▪ USAID should continue to focus on priority needs that are neglected by other donors. ▪ USAID should consider increasing its budget allocation to Libya to address high-priority areas. EQ 2b. In which areas is USAID’s assistance addressing gaps not met by other donors? Findings: ▪ Following the transfer of some of LPFM’s work streams to the World Bank, USAID is working in areas where other donor activity is nonexistent or minimal. It is addressing municipal financial management, helping to modernize and commercialize the electricity sector, and is improving the business enabling environment. ▪ Through LPFM, USAID is the only donor working on PFM at both the national and subnational levels. ▪ When USAID/LPFM was invited by GOL to take up interventions in budget execution and monitoring, the World Bank was not pleased. GOL was seen as “taking advantage of a disjointed donor network.” ▪ In January 2021 a Letter of Cooperation (LOC)29 was signed by MOF on behalf of the GNA and USAID. It aimed to formalize existing collaboration and proposed new areas for cooperation— “gaps” such as Libyan Customs Authority (LCA), Central Bank of Libya (CBL), and Libyan Audit Bureau (LAB). ▪ The LOC also aimed to coordinate with WB to avoid duplication on chart of accounts, IPSAS, GFMIS, auditing etc. Pragma stated that since the signing there were many conversations to “sort out mutual interests.” ▪ USAID/LPFM’s interventions aim to be complementary. For example, at the subnational level LPFM is helping municipalities develop budgetary systems to manage financial flows while LGCS helps them to allocate the money effectively, in response to community priorities. ▪ USAID/LPFM is also collaborative as it shares information with Expertise France on BEE, and with GIZ on SWM. LPFM works on Solid Waste Management in all major municipalities (helping with strategies, plans, and budgets), while GIZ’s approach is to pilot in certain localities and test technical and financial instruments. 29 The Letter of Cooperation is non-legal and non-binding. USAID does not allow an activity to sign a memorandum of understanding. 35 ▪ In the Electricity Sector, LPFM is assisting with implementation of the LEGS program to reduce outages. It is also helping to develop a contracting and procurement dashboard linked to regional/international market prices benchmarking in response to a request from GECOL’s Contracts Department. These important initiatives were not in the FY 2020 WP and thereby demonstrate USAID/LPFM flexibility and responsiveness. Conclusion: USAID is the only donor addressing certain urgent economic priorities in Libya— subnational PFM, electricity, and business reforms—in a substantive manner. The LOC formalizes collaboration, addresses duplication, and, more importantly, proposes new areas for cooperation that address gaps not met by other donors. Recommendations: These recommendations are aimed at strengthening USAID’s presence and effectiveness in Libya. ▪ The U. S. Government (USG) should try to establish an Embassy in Tripoli to allow USAID staffers to gain a much deeper understanding of the political and economic context and to facilitate regular face-to-face high-level meetings to build strong relations with senior officials and politicians. ▪ Given its positive reputation among other international donors and experience in promoting complementarity and preventing technical assistance duplication, USAID should attend and/or host a donor forum in Libya on a regular basis for enhanced donor coordination. ▪ USAID should encourage Activities to be demand- driven and allow them to respond to host government requests for assistance within their SOW. EVALUATION QUESTION 3 EQ 3a. To what extent is LPFM on track to meet its targets? Findings: ▪ Illustrative indicators from Pragma’s proposal and some additional USAID-required indicators were included in the Task Order. After the Task Order was signed, the AMELP and the work plan were developed to be fully aligned. USAID/COR subsequently instructed LPFM that the Life-of￾Project targets for the indicators could not be adjusted despite revised approved work plan shifts, resulting in LPFM programming that still had to achieve the original pre-COVID, pre-transitional government targets. ▪ The initial targets assigned to various indicators were generated through iterative discussions at the outset between USAID and LPFM leadership. ▪ Over two years LPFM has prepared four work plans. In the June 2020 second revision WP, activities were significantly scaled down/rescheduled/deferred to FY 2021 (see WP table pp. 51– 89) but there were no changes made to the FY 2020 MEL plan. ▪ According to MEL’s change log in the FY 2021 AMELP, five indicators were slightly modified as the program was rolled out to make them more practical, one was removed, and monitoring and data collection methods were adjusted to add remote procedures. ▪ Despite LPFM being overcome by events (e.g., conflict, COVID-19, and change of government), targets were not changed in FY 2021. Rather than allow the IP to modify targets, USAID suggested the IP provide justification for any shortfall. ▪ The LPFM MEL team is tracking 20 performance indicators, of which 19 have targets. One does not because it is a context indicator. ▪ Although social inclusion was mentioned in the Task Order, there were no indicators or targets for gender or marginalized populations. As such, there was no programmatic focus on social inclusion, only reporting on disaggregated data for gender and age in training. 36 ▪ According to the FY 2021 QPR-3 ending June 30, 11 indicator targets were on-track (two performance indicator targets were achieved while nine were exceeded), five will likely be achieved in Q4, two targets were not achieved, and one has a data lag. See the Performance Indicator Tracking Table (PITT) in Annex J for a fuller explanation of the status of each indicator target. LPFM’s MEL system does not track deliverables unless they are part of an indicator.30 Conclusion: As of FY 2021 Q3, LPFM has met or exceeded 11 of 19 indicator targets. As activities are ongoing in FY 2021 Q4 and many targets are annual or end-of-program, by year-end LPFM expects to meet 16 of the 19 indicator targets. Recommendations: These recommendations aim to ensure LPFM can adjust to shifting realities, capture all relevant outcomes, and incorporate important aspects of development. ▪ USAID should permit an activity like LPFM to adjust indicators and targets when the situation on the ground changes significantly. ▪ LPFM’s MEL system should capture data in a systematic manner to report on the nature and scope of critical training follow-up (e.g., on-the-job coaching) as well as other non-training events that have learning value (e.g., working sessions and technical meetings). ▪ LPFM should develop a gender strategy, and, with USAID, formulate a social inclusion indicator with targets. EQ 3b. What factors have led to its success in meeting targets? Findings: LPFM success factors are numerous. They include the following: ▪ Training: LPFM is focused on reform primarily enacted through capacity-development activities, of which training is paramount. LPFM has reached or exceeded targets because of prudent design and planning with relevant stakeholders, as well as appropriate and flexible training delivery methods. ▪ Staffing: Senior staff and STTAs are highly experienced and knowledgeable and are considered experts in their field and good project managers. The local staff are also technically competent. Local staff participate in training events and are coached by international experts so LPFM leadership can substitute local staff for delivery when experts are not available. For subnational work, there is greater reliance on municipal coordinators under close supervision. ▪ Head Office: Pragma is seen as “a professional and efficient organization.” They provide strong technical, strategic, and management support to the field. ▪ Counterparts: The ET was informed by many sources that counterpart and stakeholder relations are excellent. LPFM staff manage these relationships well to retain strong counterpart support.31 ▪ Plans: LPFM operates under a wide variety of strategies depending on the work stream and intervention, and work streams serve to focus activities. ▪ TWGs: Technical working groups serve to coordinate with different ministries and are relatively stable so if there is a change of government and ministers, TWG members remain fairly constant. ▪ Grants: LPFM’s small grant-making facility32 provides targeted, measurable scopes of work to support and facilitate planned deliverables. For example, nongovernmental organization (NGO) 30 There is a strong clan/tribal system in Libya, but data is not disaggregated on that basis due to sensitivities. 31 Many donors instruct their project implementers to use a Steering Committee model for project oversight and governance. With USAID concurrence, LPFM is using an informal approach to governance though there are some more formal component-level structures involving counterparts such as the GECOL Inter-Ministerial Task Force. 32 The average grant is $52,551 for Grants under Contract (GUC). 37 Moomken was selected to develop an energy conservation publicity program for GECOL33 and designed a phone app to provide citizens real-time data on load shedding. Other success factors include targets that are realistic and achievable, and flexibility to adjust WPs and activities depending on conditions on the ground as well as changing GOL and USAID needs. Conclusion: LPFM success factors are numerous, not the least of which is the expertise, commitment, and diligence of the IP/LPFM team that is considered critical for driving overall performance, strong relations with counterparts and local officials, and development of capacity of partner organizations through training. Recommendations: There are no recommendations proposed to improve upon success factors. EQ 3c. What factors led to missing targets? Findings: The two indicator targets falling short are both training output indicators: the executive branch (which crosses all three components) and trade and investment or BEE. The shortfall is explained by the following: ▪ The GNU instructed senior government officials to avoid donor community interaction starting March 2021. ▪ The pandemic generally slowed and delayed activities, as LPFM advisors were unable to travel to Libya, in-country advisors were unable to meet in person with members of GECOL, and on-site technical assistance could not be delivered. ▪ Travel restrictions were imposed, and borders were closed to foreigners so LPFM could not bring international experts to Libya or take Libyans to Tunis for training and meetings. ▪ A change in government meant new ministers, counterparts, and mayors, who needed to be informed of interventions and convinced of their vital importance. ▪ The ongoing power struggle between East and West, Prime Minister’s Office (PMO) and House of Representatives (HOR), and central government and municipalities led to further training delays. ▪ For BEE training, a contributing factor is that LPFM MEL uses USAID’s ADS 201 definition of “training,” which excludes many BEE activities that have considerable learning value, such as working sessions, technical meetings, and coaching. ▪ For ES, during the conflict, the power grid was damaged. However, in FY 2021 a new work stream was added to focus on restoring generating capacity through repair and maintenance. Conclusion: Pandemic restrictions combined with political instability and uncertainty were the main factors that contributed to LPFM missing some training indicator targets. Recommendation: LPFM should continue to put concerted effort into training up local staff to take on the roles of experts who are denied access to Libya. EVALUATION QUESTION 4 EQ 4a. To what extent did the activity successfully collect data and monitor activities as described in the AMELP? Findings: METAL’s Data Quality Assessment (DQA) Report from November 2020 describes the MEL information system in some detail. It reached the following conclusions: 33 GECOL wants the campaign to run without reference to donor or IP due to negative political perception. 38 ▪ The IP maintains a good level of MEL function independence not only in the staffing structure but also in having full control over primary data collection at the field/beneficiary level. ▪ Overall, the IP’s MEL system is designed to ensure consistency of data collection procedures over time and provides clear guidance not only on collecting data, but also on data collation and analysis, data verification procedures, and reporting. ▪ The IP’s multi-tiered data verification procedures and cross-check methods typically follow the “data chain.” The roles and responsibilities are adequately documented in the MEL System Processes Manual and the Standard Operating Procedures (SOPs). ▪ The documentation of Pragma’s data management procedures in indicator-specific SOPs is comprehensive and underpins the reporting of good quality data. ▪ The IP has documented its remote monitoring protocol to inform MEL specialists of how to undertake remote monitoring and collect verifiable evidence for all online events and engagements with stakeholders. The ET notes that the MEL system features the following attributes: ▪ There is a manual, periodic quality assurance/control, and backup archiving. ▪ Weekly data updates are shared with LPFM management for informed decision-making. ▪ Constant updates and improvements are made to the system (adaptive management). ▪ Data collected is related only to indicators, not on “non-training” events. ▪ The system is password-protected and only those who have access can view the data freely. ▪ A manual specifies all the standard operations procedures for each indicator. The system also has an Excel indicator tracking table that is part of the quarterly reports. ▪ Occasional surveys are conducted of trainees on training and utilization of know-how (see under EQ5), and perception surveys on electricity tariff increases and BEE. The LPFM performance data flow contained in the AMELP is shown graphically in Annex L. Conclusion: The MEL system as outlined in the AMELP is thorough, robust, and responsive, and performance data captured and reported is reliable. Pragma said, “We have over-designed the LPFM MEL system” and USAID indicated that LPFM has set a high bar for other IPs in Libya. Recommendation: LPFM MEL should carry out more focus groups of trainees and counterparts to determine to what extent training know-how is used and adopted in the workplace. EQ 4b. What challenges did LPFM face in activity monitoring, both the monitoring it managed and in working with USAID/Libya’s monitoring and evaluation contractor on third-party monitoring? Findings: ▪ There were two main challenges for LPFM in activity monitoring. First, after the FY 2020 revision to the work plan, LPFM MEL had to adjust the indicators to comport with the new work plan. But USAID told MEL it could not alter them. Second, due to growing insecurity, the MEL team added remote processes and procedures for data collection to the MEL system but indicated that adding multiple access points had been an IT challenge. ▪ LPFM related that “the system could be more efficient if Pragma were an IT company so that a system could be created that allows for different sites that had been given permissions to access the data and upload and review the evidence in real time, but this is not the case.” ▪ As for third-party monitoring (TPM), there is good coordination between LPFM and METAL in carrying out TPM. It runs smoothly and communication is close and effective. 39 ▪ The METAL web tool captures all the information needed from the IP, and METAL gives access to the MEL team to make inputs to the monitoring site visit reports. ▪ The frequent changes to the dates of trainings and other events poses a challenge but LPFM notifies METAL of these changes immediately. Conclusion: Due to the contextual challenges in FY 2020, indicators were modified, and then remote processes and procedures were developed and incorporated into the FY 2021 AMELP. Apart from challenges with changes in event schedules, TPM appears to run smoothly, in part due to the close collaboration between LPFM MEL and METAL. Recommendation: There are no recommendations because the MEL system is functioning well. EVALUATION QUESTION 5 EQ 5a. To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? Findings: ▪ LPFM is focused on reforms that are facilitated through meetings and capacity development of which training is paramount. ▪ LPFM has carried out a large volume of training, especially for municipalities, CSOs, and public lawyers. Training is reported under five training output indicators and the volume and variety of training is diverse, as reflected in an overview table in Annex I. ▪ The delivery of training was generally good. Feedback from counterparts and beneficiaries was positive. Some reported the following: “Training was good, useful, effective, well organized,” and “Training content, delivery, organization and follow-up were all good,” and “We learned a lot.” Another beneficiary said, “The training was useful to my job.” A post-training survey carried out in April 2021 confirmed that training was of good quality and training at the subnational level was rated more positively than other training. ▪ Training content was also relevant to the trainee’s job. LPFM has indicators to measure the number of people trained and/or person hours (i.e., the volume of training). Indicator 1.1d refers to the percent of officials inside municipalities, MOLG, MOF, and MFU with improved skills to perform their functions. This is assessed through surveys and focus group participants’ perceptions of their skills and knowledge after training. The FY 2021 QPR3 reported 71 percent had improved skills (the target was 80 percent). ▪ However, a report on capacity development carried out in April 2021 found only half of trainees had applied acquired know-how on the job. A sizeable number of respondents in the survey reported that they had not yet had an opportunity to apply the knowledge and skills gained through training, mainly due to restrictions on work due to the pandemic but expected to do so in the future. They also commented that they had not yet advanced to the point where they were able to utilize their new skills. ▪ The survey shows that COVID-19 was a significant barrier to applying acquired know-how on the job as many of those trained were instructed to stay home, got sick from the virus, or their offices were closed. Conclusion: LPFM has conducted a lot of training to develop the capacity of partner institutions. Though training content and delivery was good, about half the trainees indicate they are not yet applying the newly acquired know-how. They either have not yet had an opportunity to apply it or COVID-19 restrictions limited their ability to be in the office applying their skills. However, they do expect to use them in the future. Recommendation: Figure 3 outlines a best practice approach to training design and delivery that will maximize training relevance and utility. Because the LPFM approach is already aligned with this 40 process, there are no specific recommendations, only that LPFM should follow the steps systematically. (EQ5c) EQ 5b. If so, what has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? Findings: ▪ LPFM wants training to be used and adopted so that partner institutions benefit. To ensure this LPFM provides on-the-job training/coaching to trainees as a follow-up to formal training. Often this involves embedding the expert inside the institution to work with training recipients (e.g., a local expert was inside GECOL to work on the dashboard). Experts are typically backed up by local advisors who also work on-the-job to facilitate effective utilization of trained skills by recipients. This is especially critical because in the past six months foreigners have been denied entry into Libya due to COVID-19 and government restrictions. To strengthen their technical expertise, experts have coached local advisors to become subject matter specialists, although COVID-19 restrictions make training follow-up more challenging. The feedback on follow-up was generally positive: one informant stated, “There was regular follow-up with us” while another said, “They visited us frequently,” and another also remarked, “The local (municipal) coordinator visited weekly to check on our work” and “We regularly call the local coordinator for advice.” But the feedback from municipal staff was not unanimous; some said the follow-up assistance was lacking. ▪ LPFM set up many TWGs to facilitate the development and implementation of capacity development plans. This “working level” arrangement is mostly stable even when a change of government occurs. ▪ LPFM engages with many counterparts on its 28 work streams but a change of government in March resulted in many new counterparts who, according to LPFM, “had to be educated to obtain their buy-in.” Without proper understanding and support, the adoption of training know-how became tenuous. For example, technical work is completed on raising electricity tariffs and establishing the SOE Unit, but these initiatives are stalled because there is no political will to approve and implement at this time. ET was informed that with an election due in December, the GOL is reluctant to move forward with major reforms. ▪ Despite the impediments resulting from COVID-19 and the change in government, many interventions went well, and training was used and adopted so that institutions benefit. Examples include the preparation of OSR budgets that relied on extensive training of 46 municipalities, and a billing and collection dashboard that is being used by GECOL senior management. ▪ Although LPFM conducts post-training focus groups and surveys, it does not capture or report data on the nature and scope of critical training follow-up. Conclusion: To ensure that training is used and adopted to benefit government entities, LPFM has provided follow-up coaching to training recipients, embedded experts in partner institutions, set up TWGs, and engaged extensively with counterparts. However, COVID-19 restrictions and a change of government have served to undermine the efficacy of this approach. Recommendation: It is vital that intensive follow-up with training participants occurs to ensure new know-how is applied on the job and that reform measures are fully adopted by partner institutions and thereby take root. EQ 5c. How can training be improved going forward to improve relevance and utility? Recommendation: Figure 3 below outlines a best practice approach34 to training design and delivery that will maximize training relevance and utility. Because the LPFM approach is already aligned with 34 Synthesized from Institutional Assessment and Capacity Development: Why, what and how? EuropeAid. September 2005, Overview p. 7; Support to Capacity Development-Identifying Good Practice in Swedish Development Cooperation. SIDA. 2015; the Challenge of 41 this process, there are no specific recommendations. LPFM should continue to follow the steps systematically. (EQ5c) Figure 3. An Approach to Improve Training Relevance and Utility ▪ Champions: It is important to identify reform-minded persons who are willing to be drivers of change and committed to the initiative. ▪ Agreement: A signed agreement between the donor and the host government will ensure there is clear understanding of expectations, roles, and responsibilities, and foster true partnership. ▪ Understanding context: It is essential that LPFM develop a deep understanding of the institutional situation when designing training programs. ▪ Engagement: There should be a dynamic dialogue with counterparts on priority needs and interventions, and the partner should actively be involved in preparing the capacity development plan. ▪ Imported solutions: LPFM should rely on existing training programs and materials where appropriate to save money and time. ▪ Experts: Involve counterparts in identifying suitable Arab-speaking international experts to introduce and adapt materials or deliver training. ▪ Training of Trainers: Focus on training counterparts to lead the training because this is critical to sustain the learning. ▪ Trainees: Identify suitable people for training, 35 including targeting women and women-only events in the capacity-development plan and encourage chaperones, if necessary. ▪ Follow-up: Ensure intensive on-the-job training for optimal utilization of skills and knowledge. ▪ Embed experts: Follow-up is much improved when LPFM experts are located inside a ministry. ▪ Upgrade local staff: Experts should upgrade project staff to help them become subject matter experts and play a more effective role in delivering training and follow-on support. ▪ MEL: Ensure data on delivery of follow-on activities are captured and reported to management and USAID. ▪ Digital: Record and store events digitally for remote access. EVALUATION QUESTION 6 EQ 6a. How have the conflict and the COVID-19 pandemic impacted implementation of the project? Findings: ▪ The conflict and COVID-19 slowed LPFM implementation and made technical work more difficult. ▪ Libya suffered a severe COVID-19 outbreak. Many LPFM staff and counterparts got sick, borders were closed, and expatriates were evacuated in April 2020 so staff could not meet face-to-face with key counterparts), training events were canceled or postponed, activities got delayed, lockdown and curfews occurred, and travel/flight and visa restrictions meant LPFM could not bring international experts to Libya, nor Libyans to Tunis for meetings and training. Capacity Development: Working Towards Good Practice. DAC Network on Governance. OECD. 2006; and Supporting Capacity Development in PFM – A Practitioner’s Guide. 4th High Level Forum on Aid Effectiveness. 29 Nov-1 Dec 2011. Busan, Korea. 35 For citizen budgeting initiative, CSO applicants were vetted (e.g., core staff, organic funding), before being invited to training, and then joining citizens committees. 42 ▪ It was also illegal for government officials to attend in-person meetings (except subnational), offices were closed, and officials stopped answering their phones. As one counterpart said, “We experienced three months of shutdown due to the pandemic.” ▪ During the pandemic, officials had to work from home and carry out meetings and interviews virtually, and it became harder to get data and do research. ▪ Many informants supported the notion that the virtual delivery of training is lower quality than in￾person. Conclusion: COVID-19, much more than the conflict, slowed LPFM implementation by making technical work and activity management more difficult. Recommendation: Pragma should continue to aim for a stable staff complement, especially at the management level, to maintain continuity in counterpart relations, teamwork, and Activity institutional memory. EQ 6b. What adjustments did LPFM make in response to the conflict and the pandemic? Findings: ▪ LPFM rapidly adjusted to the twin challenges of the conflict and COVID-19. They switched to virtual meetings and training, off-site meetings, and staff worked from home. ▪ In Libya, frequent power outages interrupted virtual events. In response, LPFM records the sessions and makes them available to participants. Also, LPFM moved meetings and training to the evening hours when electricity and internet were better. ▪ LPFM sought to mitigate the disruptions in movement from the conflict by relying on family and clan ties to travel to the East and continued local-level relations. LPFM also enhanced its security through protocols, security bulletins, and use of close protection. Conclusion: LPFM found effective workarounds to mitigate the worst effects of the conflict and COVID-19. Recommendation: LPFM should continue to be flexible and adaptable, as needed, to mitigate the effects of the conflict and COVID-19. EQ 6c. How did the conflict and the pandemic impact the achievement of the project’s objectives? Findings: LPFM was able to achieve most results, despite the difficult conditions. COVID-19 remains somewhat challenging but is now manageable. The conflict is less of a hindrance now, but the associated political instability and uncertainty related to the December 2021 elections are still of concern. Conclusion: Both COVID-19 and the conflict became manageable and did not significantly affect LPFM’s overall results. However, the new government and the associated change of ministers and counterparts and the upcoming election may make it harder for LPFM to obtain the approval and implementation of the laws, policies, systems, processes, and reforms that it has been helping to develop for the national and subnational governments. Recommendation: See EQ6d. EQ 6d. How did the project demonstrate resilience/flexibility in response to these challenges? Findings: LPFM demonstrated resilience and flexibility by: ▪ Having staff who are persistent, committed, and diligent, and drew on their networks for support. 43 ▪ Establishing good relationships with counterparts prior to COVID-19 made it possible to continue activities during the pandemic. ▪ Being a flexible organization that significantly shifted its activities to address the changing situation.36 ▪ Shifting to virtual platforms for meetings, training, and remote management. ▪ Utilizing local staff and experts for delivery as a substitute for international experts who could not get into Libya. Conclusion: LPFM has coped well with the dual challenges of conflict and COVID-10; Pragma said, “Libyans are great at adapting.” Through workarounds LPFM has continued to operate while other donors suspended projects or delayed launching new ones. Recommendation: LPFM should maintain a dynamic, innovative field team with strong counterpart relationships and Head Office support to overcome the debilitating effects of the conflict and COVID￾19. 36 FY 2020 Work Plan 2nd revision. June 2020. Work Plan table pp. 51–89. 44 ANNEX A: LPFM LOGIC MODEL37 US Mission Libya: (MO 1.1) Libya successfully advances its democratic transition w/national and local institutions of governance becoming more legitimate and accountable to its citizens; (MG 3) An improved security, governance and regulatory environment creates new opportunities for investment, trade, and economic diversification LPFM Goal: Libya’s Macroeconomic and Fiscal Foundations for Sustainable and Inclusive Growth Strengthened Cross-Cutting: Strategic Communications with Libyan Policy Makers and General Public Improved ES Component Result: Libya’s electricity sector modernized and commercialized Sub-Result: GECOL efficiency strengthened Illustrative Activities: Continue TA and training of GECOL Technical Committees; assist GECOL to develop and roll out new Customer Relationship Management￾Billing platform; pilot Automated Meter Reading program; support GECOL’s workforce realignment technical committee implements action plan; etc. Sub-Result: Demand-side management improved Illustrative Activities: Support energy conservation public awareness campaign; establish appliance standards, develop energy-efficient action plan; LED change-out program PFM Component Result: Capacity for Libya’s public financial management at the national and subnational levels strengthened Skills & knowledge of PFM staff / officials improved Sub-Result: Macro-Fiscal Unit (MFU) capacity to produce analysis and studies strengthened Illustrative Activities: MFU continues to produce monthly economic reports, fiscal reports for 2020/2021, fiscal reports for 2021/2022 budget; seminars conducted for MFU staff; produce studies (equilibrium exchange rate, financial stability, fiscal rules and sustainability, key structural reforms, countercyclical fiscal policy, macrostructural constraints, inflation, etc.) Sub-Result: Fiscal decentralization and subnational PFM is promoted Illustrative Activities: TA and train municipalities on OSR execution; continue PE implementation in targeted municipalities; implement SWM and PHC plans in all target municipalities; adopt IGF grants formula; begin MDF operations; train CSOs on citizen budgeting; train municipalities on conducting public hearings; Community Budget Committees operating Sub-Result: Threats to fiscal sustainability and Libya’s economic transition are mitigated BEE Component Result: Economic governance and business enabling environment improved Illustrative Activities: Assess BEE and EG frameworks; continue building capacity of PSAP; complete operationalization of online e-legislation database at LBC and establish link on AmCham website; assist GDSC reform secretariat to organize and hold high-level PPD meetings; adopt critical legislative regulatory reform proposals and initiate institutional implementation; etc. Sub-Result: National-level PFM reforms supported Illustrative Activities: Develop comprehensive Sectoral Operational Plans and PIPs; assist SOE Budget Planning and Monitoring Unit to establish a budget transfer planning and monitoring framework and process for two major SOEs; targeted TA to MOF to help downsize the public sector payroll; TA and training to develop and adopt better practice public procurement; develop and train officials on new Municipal Auditing procedures 37 AMELP FY 2020. In FY 2021 ES sub-result 2.1 was changed. initiated; etc. 45 ANNEX B: SCOPE OF WORK SCOPE OF WORK Interim Evaluation of Libya Public Financial Management (LPFM) 46 I. PURPOSE OF THE EVALUATION This scope of work (SOW) for the Monitoring and Evaluation for Tunisia and Libya contract (METAL) outlines USAID/Libya’s need for an interim evaluation of the Libya Public Financial Management (LPFM) to inform the USAID Economic Growth (EG) team on how to advance Libya’s self-reliance and stability. Recommendations from the evaluation will inform the team to make changes to the existing activity, ensuring that USAID’s efforts are as effective as possible given the challenging operational environment. The main audience is the USAID/Libya EG team and the broader USAID/Libya team, though the reports may also be shared in full or in part with the Libya External Office (LEO), USAID implementing partners, and the Middle East Bureau. II. SUMMARY INFORMATION Strategy/Project/Activity Name Libya Public Financial Management (LPFM) USAID Office USAID/LIBYA Implementer(s) The Pragma Corporation Contract # 7200AA18D00016 / 72067019F00001 Total Estimated Ceiling of the Evaluated project/Activity (TEC) $23.3M, total estimated cost is up to $55M over five years Life of Strategy/Project/Activity September 16, 2019–September 15, 2021 [base period], and to July 2, 2023 [options period] Active Geographic Regions Nationwide Development Objective(s) (DOs) DO1: Institutions of Governance more accountable to and better represent the interest of citizens DO2: Increased Opportunity for Licit Economic Growth and Participation Required evaluation? Yes External or internal evaluation? External III. BACKGROUND The Libya Public Financial Management (LPFM) activity advances Libya’s national stability by (1) strengthening the Government of Libya’s (GOL) public financial management (PFM) policies and processes to strengthen the fiscal foundations for sustainable and inclusive growth, (2) providing technical assistance and capacity-building for Libya’s electricity sector, and (3) seeking to improve Libya’s business enabling environment to stimulate growth of the private sector. A. Description of the Problem and Context USAID’s Libya Public Financial Management (LPFM) Program—also known as “Bab Al Tamkeen”—is a five-year initiative designed to address critical macro-fiscal, electricity sector, and private sector enabling environment constraints facing the post-Arab Spring Libyan economy. Libya’s political instability is exacerbated by its inability to manage its oil wealth fairly, in a transparent and accountable manner, and for the benefit of all its citizens, including provision of basic services. Even before the recent challenges of the escalated conflict and the coronavirus pandemic, modern Public Financial Management (PFM) was almost completely lacking in Libya, as demonstrated by Libya’s weak ability to budget for and provide basic services, including reliable electricity. Heavily subsidized electricity and gasoline combined with a massive public sector payroll to further stress the country’s fiscal health. LPFM is designed to address these challenges. LPFM’s goal is to help the Government of Libya manage its substantial oil wealth in a more equitable manner and deliver resources to provide improved basic services to the public. LPFM is formulated to help Libya improve its transparency and accountability using international best 47 practices in budget planning, budget execution, and extractives transparency, as well as an improved decentralized fiscal system that allows for greater participation by subnational governments in budget decisions. These improvements will enable Libya to provide basic services to its citizens, thereby increasing stability. B. Description of the Intervention to be Evaluated and Theory of Change The Theory of Change is that improved governance and service delivery will increase citizen support for, and stability of, national and local institutions. Also, if economic integration improves, economic reforms advance and key institutions will have more revenue and Libyan self-reliance will improve, as will national stability. IV. EVALUATION QUESTIONS Evaluators will review and finalize questions in collaboration with USAID prior to finalizing the evaluation design. Results and Impact 1. How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? 2. Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? In which areas is USAID’s assistance addressing gaps not met by other donors? 3. To what extent is LPFM on track to meet its targets? What factors have led to its success in meeting targets? What factors led to missing targets? Monitoring 4. LPFM has a detailed activity monitoring, evaluation, and learning plan (AMELP). To what extent did the activity successfully collect data and monitor activities as described in the AMELP? What challenges did LPFM face in activity monitoring, both the monitoring it managed and in working with USAID/Libya’s monitoring and evaluation contractor on third-party monitoring? Measuring Sustainability 5. To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? If so, what has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? How can training be improved going forward to improve relevance and utility? COVID-19 Response 6. How have the conflict and the COVID-19 pandemic impacted implementation of the project? What adjustments did LPFM make in response to the conflict and the pandemic? How did the conflict and the pandemic impact the achievement of the project’s objectives? How did the project demonstrate resilience/flexibility in response to these challenges? V. EVALUATION DESIGN AND METHODOLOGY USAID expects that, at a minimum, the evaluation team will: ▪ Upon award, familiarize themselves with documentation about the LPFM activity and USAID’s current assistance in the Economic Growth area in Libya and the region. ▪ Review and assess the activity’s performance reports and performance data. ▪ Test survey instruments for clarity prior to use. ▪ Meet and interview USAID project beneficiaries, partners, and host government counterparts at appropriate levels. ▪ Interview USAID staff and a representative number of experts working in the sector. Due to COVID-19, it is expected that interviews will be conducted remotely. The evaluation team must address this limitation and appropriate mitigation measures in its evaluation design. The desk review includes at a minimum: ▪ USAID Libya Public Financial Management SOW. 48 ▪ LPFM materials: Annual and Quarterly Reports, Annual Work Plan, MEL Plans, sector assessments, trip reports, performance reports, gender analyses, relevant sections of the Project Appraisal Document, and miscellaneous thematic reports from other sources. The contractor will submit the preliminary evaluation design for review by USAID. The evaluation Contracting Officer’s Representative (COR) must approve the evaluation design prior to fieldwork commencing. VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. In-briefing Given travel restrictions related to the COVID-19 pandemic, the evaluation team will conduct a virtual in-briefing with the EG Officers and local staff for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the SOW, if necessary. 2. Evaluation Inception Report The evaluation team must prepare an evaluation inception report that includes the following: ▪ Draft evaluation team schedule, including key milestones and logistical arrangements. ▪ Members of the evaluation team, delineated by roles and responsibilities. ▪ Description of the proposed evaluation methodology. This section should include a detailed evaluation design matrix that links the evaluation questions from the SOW (in their finalized form) to data sources, methods, and the data analysis plan. ▪ List of potential key informants and proposed selection criteria and/or sampling plan, as applicable. ▪ Draft data collection tools. ▪ Limitations to the evaluation design. ▪ Dissemination plan (designed in collaboration with USAID). The data analysis plan should clearly describe the evaluation team’s approach for analyzing quantitative and qualitative data (as applicable), including proposed sample sizes, specific data analysis tools, and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans should be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to help communicate evaluation [findings/conclusions/recommendations]. See the Evaluation Toolkit for guidance on Developing an Evaluation Dissemination Plan. If applicable, based on the Disclosure of Conflict of Interest Forms submitted with the awardee’s proposal, the evaluation design will include a conflict of interest mitigation plan. 3. Midterm Briefing and Interim Meetings The evaluation team is expected to hold a midterm briefing with the USAID/Libya EG team, Middle East Regional Platform MEL specialist, and other relevant USAID/Libya team members on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/AOR with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 4. Preliminary Presentation for Recommendations Development The evaluation team is expected to hold a preliminary presentation by virtual conferencing software to discuss the summary of findings, conclusions and recommendations with USAID. Any presentations or workshops will be scheduled as agreed upon during the in-briefing. 5. Final Presentation 49 The evaluation team is expected to hold a final presentation by virtual conferencing software to discuss the summary of findings, conclusions, and recommendations with USAID. 6. Draft Evaluation Report The draft evaluation report should be consistent with the guidance provided in Section IX, Final Report Format and must meet all evaluation report and quality requirements detailed in USAID Automated Directives (ADS) Chapter 201. The report will address each of the questions identified in the SOW. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the USAID/Libya team will have ten working days in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report within ten working days, and again the USAID/Libya team will review and send comments on this final draft report within ten working days of its submission. 7. Final Evaluation Report The evaluation team will be asked to take no more than ten working days (or as agreed upon in the work plan) to respond to and incorporate the final draft evaluation report and presentation comments from the USAID/Libya team. 8. Submission of Final Evaluation Report to the Development Experience Clearinghouse Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. 9. Submission of Dataset(s) to the Development Data Library Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non￾proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. Please review ADS 579.3.2.2 Types of Data to Be Submitted to the DDL to determine applicability. 10. Other Deliverables Two-Page Summary Brief. Following final approval of the evaluation report, the contractor must produce a two-page brief of the evaluation that includes evaluation purpose and summarizes evaluation findings, conclusions, and recommendations. This brief should be visually appealing and, if appropriate, incorporate photos or other visuals. This brief may be shared with stakeholders. Post-Evaluation Action Plan. Within 30 days of the approval of the evaluation report, the contractor must work with the USAID technical team to develop a “post-evaluation action plan” per the guidelines in ADS 201. VII. EVALUATION TEAM COMPOSITION The Contractor must propose an appropriate team for this assignment and provide information about evaluation team members, including their curricula vitae. The Contractor must explain how each evaluation team member’s qualification and expertise meets the requirements in the evaluation SOW and will contribute to the achievement of a technically sound, objective, and high-quality evaluation. Submissions of writing samples or links to past evaluation reports and related deliverables composed by proposed team members are highly desirable. The evaluation team must be approved by the METAL COR. Any substitutes to the proposed key personnel must be vetted and approved by the COR before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means. 50 Per ADS 201.3.5.14, all team members must provide to USAID a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project or activity being evaluated (i.e., a conflict of interest form). VIII. DELIVERABLES AND TIMELINE The Contractor must provide the following deliverables. All written documentation for submission by the Contractor to USAID/Libya must be in English. This schedule is illustrative and will be updated (as necessary) in collaboration with USAID. Deliverables Due Date (Time frame) 1 Inception report containing work plan and evaluation design Prior to the start of the document review and meetings, the Contractor must submit to USAID an Inception Report for USAID approval. The report must include: ● Complete list of documents planned for review ● Complete list of planned interviews and proposed individuals ● Proposed list of questions for the interviews ● Detailed description of the data collection plan and analysis methodology ● Literature review USAID will have one week to review and provide comments/ feedback. Work plan and evaluation design period: August 13, 2021 (August 2–13) USAID Review Period: August 20, 2021 (August 16–20) 2 PowerPoint presentation of LPFM Interim Evaluation preliminary findings and recommendations The Contractor must develop and deliver a PowerPoint presentation of the preliminary findings and recommendations of the LPFM Interim Evaluation to USAID/Libya staff and other relevant stakeholders to be delivered to the USAID/Libya team. September 17, 2021 (August 23–September 17) 3 Draft Interim Evaluation Report Refer to section “Final Report Format and Supporting Data” below. Should be submitted as a MS Word document. Draft report will include feedback from PowerPoint presentation of preliminary findings and recommendations. USAID will have two weeks to review and provide comments/ feedback. Draft Report Prep Period: October 1, 2021 USAID Review Period: October 15, 2021 (October 1–October 15) 4 Final Interim Evaluation Report Refer to section “Final Report Format” below. All comments provided by USAID should be addressed in the Final Report. The report should be submitted in MS Word and PDF formats. October 22, 2021 (October 15–October 22) IX. FINAL REPORT FORMAT 1. Abstract 2. Executive Summary 3. Evaluation Purpose 4. Background on the Context and Activity Being Evaluated 5. Evaluation Questions 6. Methodology 7. Limitations to the Evaluation 8. Findings, Conclusions, and Recommendations 51 9. Annexes See the Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be two pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. Annexes to the report must include: ▪ Evaluation SOW (updated, not the original, if there were any modifications). ▪ All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides. ▪ All sources of information or data identified and listed. ▪ Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable. ▪ Signed disclosure of conflict-of-interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest. X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ▪ Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. ▪ Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ▪ The Executive Summary should present a concise and accurate statement of the most critical elements of the report. ▪ Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ▪ Evaluation methodology should be explained in detail and sources of information or data properly identified. ▪ Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ▪ Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ▪ Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence. ▪ If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. ▪ If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements for additional guidance. XI. OTHER REQUIREMENTS 52 All modifications to the required elements of the SOW of the contract/agreement, whether in evaluation questions, design and methodology, deliverables and reporting, evaluation team composition, schedule, and/or other requirements will be agreed upon in writing by the COR. Any revisions made will be noted in the SOW annexed to the final Evaluation Report XII. LIST OF ANNEXES 1. Activity Monitoring, Evaluation, and Learning (MEL) Plan 2. Year 1 and Year 2 Work Plans 3. Task Order and executed modifications 4. Data Quality Assessments (DQAs) 53 ANNEX C: DATA COLLECTION INSTRUMENTS Key Informant Interviews (KIIs): Guide for Interviewers This data collection instrument, though framed as a questionnaire, is a guide for field researchers. It aims to steer in￾person and online conversations with stakeholders. It is not to be shared or handed out. Interviewee’s name, title, and organization: Tel: Email: Interviewer[s]: Date of the interview: Location: Time interview started: Means of recording: a- Note taking b- Voice recording Time interview ended: Inform interviewee that the recording function is on, but the recording will be deleted after notes are made and/or validated. Read out the consent form [annex N] and get an oral agreement from the interviewee – it explains how we will use this information. Is the consent explained and consent form agreed? Yes No Ask the following ET questions – and follow-up questions as needed to probe. The 1st column in the table below shows the evaluation 6 core research questions that contain 17 sub-questions. The 2nd column shows the ET main questions. Note that some questions are more relevant to some e.g., Q2 is primarily for donors. 1. Tell us briefly about the nature of your involvement or interaction with LPFM. EQ1 Results & Impact ET Questions [for all] 1.1 How effective has LPFM been to date in achieving the award’s three objectives? 2. What has LPFM achieved since September 2019 [a] in PFM (national & subnational) [b] in electricity sector (ES) and [c] in private sector BEE? S1.2 What challenges did LPFM face in achieving the objectives? 3. Post-conflict environments present unique challenges – what were the main obstacles or difficulties for LPFM in achieving results [a] in PFM, [b] in electricity, and [c] in BEE? 4. Is LPFM scope too broad and complex? If yes, please elaborate. 1.3 In which areas was LPFM most successful? 5. Which components were more successful? 6. Within C1, was PFM support more effective at national or subnational? Please explain your answer. 7. In which areas [28 workstreams] has LPFM excelled? 8. What training, technical assistance or tools do you consider a success? EQ2 Results & Impact Q primarily for donors, Pragma/LPFM, and USAID 2.1 Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? 9. Compared to other donors in Libya, what advantage[s] does USAID have in [a] developing Libya’s PFM at national and subnational levels, [b] in modernizing and commercializing the electricity sector [ES], and [c] in enhancing the BEE? 10. Is there a forum for the donor community to share plans, information, and experiences? 11. What is the forum and is it effectiveness? 12. Who coordinates donor meetings? [need name and contact details] 13. Do they circulate the minutes of donor meetings? 2.2 In which areas is USAID’s assistance addressing gaps not met by other donors? 14. Which donors are active in Libya PFM, ES, and BEE sectors? 15. What are their interventions? 16. How are they different from LPFM? 54 17. Does USAID have a strategy to support development in Libya? If so, what is it? 18. Is USAID addressing specific gaps in Libya not met by other donors? 19. If so, what gaps are being addressed by USAID through LPFM? 20. Are there advantages to combining three sectors [PFM, ES, and BEE] in one project/activity? EQ3 Results & Impact Q. 3.1 and 3.3 for Pragma/LPFM and USAID, 3.2 for all 3.1 To what extent is LPFM on track to meet its targets? 21. Are LPFM indicator targets [as per AMELP] reasonable, realistic and appropriate in the Libyan context? If no, why not? 22. Will LPFM fulfill all its targets by mid-September 2021? 23. Which targets do you think will not be met by mid-September? 24. Why do you think these targets will not be met? 25. Why are there no gender-related targets? 3.2 What factors have led to its success in meeting targets? 26. How would you describe LPFM “success factors”? 27. Did ‘grants under contract’ or GUC contribute to LPFM success? In which ways? 3.3 What factors led to missing targets? 28. In LPFM areas [workstreams] that are incomplete, or lacking results, what factors have impeded LPFM implementation? EQ4 Monitoring For LPFM/IP, USAID 4.1 To what extent did the activity successfully collect data and monitor activities as described in the AMELP? 29. Summarize the MEL system for LPFM. 30. Are there any shortcomings in LPFM performance of its MEL duties (i.e., collect data and monitor activities)? 31. If yes, please describe them (so the system can be improved). 4.2 What challenges did LPFM face in activity monitoring, both [a] the monitoring it managed and [b] in working with USAID/Libya’s Monitoring and Evaluation (M&E) contractor on third-party monitoring? 32. What were LPFM challenges in activity monitoring? 33. What were challenges of third-party monitoring (i.e., working with METAL)? EQ5 Measuring Sustainability Primarily for beneficiaries in FGDs and counterparts from benefiting institutions 5.1 To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities 34. Why do you think training [skills and knowledge] was selected for support by USAID through LPFM? 35. In the current Libya context, is training of civil servants and technocrats (on good practices) combined with TA/advisory and tools [e.g., software] better than core PFM reforms (improvement to systems, structures, processes, and procedures)? Please explain your response. 36. How were partner institutions/organizations selected? 37. How were training topics determined (e.g., gap analysis)? 38. How were trainees selected? 39. Why did so few women participate in training? 40. What could be done to encourage more female trainees? 41. How was the training delivered? 42. Were short-term consultant-trainers effective? In which ways? 43. Was the training successful? 44. If not, why not? 45. Which training was most useful to your job (as trainee), and which was least effective? 46. Did persons trained by LPFM use/apply this new know-how? 47. If not, why not? 55 48. How did newly acquired knowledge and skills improve trainees’ job performance? 5.2 What has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? 49. In which ways did government entities benefit from staff training provided by LPFM? 50. What did LPFM do to ensure institutions receive meaningful benefit from the training provided to its employees? 5.3 How can training be improved going forward to improve relevance and utility? 51. Should LPFM focus on supporting PFM, ES, or BEE? Please elaborate you answer. 52. Should LPFM focus its efforts on top-down national PFM reform [e.g., MFU support] or bottom-up subnational PFM [e.g., CSO engagement for civil oversight]? Please explain your response. 53. Were there shortcomings in LPFM’s approach to implementing training? What were they? 54. How was LPFM capacity-building efforts integrated or embedded into partner institutions? 55. How can LPFM training be improved to make it more useful and relevant, both to trainees and institutions? EQ6 COVID-19 Response Q for all, but especially for LPFM 6.1 How have the conflict and the COVID-19 pandemic impacted implementation of the project? 56. How has the conflict and COVID-19 in Libya impacted implementation of LPFM? 57. Were other donor programs similarly impacted by the pandemic and political strife? 6.2 What adjustments did LPFM make in response to the conflict and the pandemic? 58. In which ways did LPFM respond to the conflict and COVID-19? 6.3 How did the conflict and the pandemic impact the achievement of the project’s objectives? 59. How did the conflict and the pandemic affect LPFM results? 6.4 How did the project demonstrate resilience/ flexibility in response to these challenges? 60. In which ways did LPFM respond to COVID-19 and the conflict? 61. How could the LPFM response be more flexible and/or resilient? 62. Were there any unanticipated consequences or outcomes from LPFM? 63. What did LPFM do to promote social inclusion? 64. What are the main lessons learned from LPFM? 65. Have you any suggestions for improving LPFM? 66. Any final comments? 56 ANNEX D: LPFM WORK STREAMS AND PLANNED RESULTS (FY 2021 WP) COMPONENT 1: NATIONAL AND SUBNATIONAL PFM REFORMS COMPONENT 1.1 NATIONAL-LEVEL PFM REFORMS A. Develop Medium-Term Expenditure Framework (MTEF) 1. Comprehensive Sectoral Operational Plans and Public Investment Plans (PIPs) – SWM – in collaboration with the selected Municipalities 2. Comprehensive Sectoral Operational Plans and Public Investment Plans (PIPs) – PHC – in collaboration with the Ministry of Health 3. Comprehensive Sectoral Operational Plans and Public Investment Plans (PIPs) – PE – in collaboration with the Ministry of Education 4. Integrated multiyear budget planning process jointly established by MOF budget unit and MOP capital budgeting unit for the SWM, PHC, and PE sectors B. Establish SOE Budget Planning and Monitoring Unit 1. Establish SOE Unit 2. Budget Transfer Planning & Monitoring Framework and Process for GECOL and one other SOE to be selected in Q.1 C. Rationalize Wage Bill Expenditures across Public Sector 1. Systematic game plan for estimating true size of public sector workforce and ‘ghost’ workers 2. The MOH and MOED adopt, and initiate strategies and implementation plans to identify ghost workers and reduce their wage bill D. Improve Budget Execution and Monitoring Processes 1. Procurement a- Provide training and technical support to MOF, MOP, the Government Decision Support Center (GDSC), and the Presidential Council Legal Office (PCLO) on better practice procurement procedures b- MOF, MOP, GDSC, and the PCLO adopt better practice procurement procedures 2. Harmonization a- Public Accounting TWG completes harmonization of public accounting standards with IPSAS b- Develop and adopt new GAM c- Harmonize ACS 3. Internal Auditing a- Report assessing compliance of internal audit framework and guidelines with international standards b- Create new Municipal Audit Manual in collaboration with MOLG and the Audit Bureau c- Create training material for new the Municipal Audit Manual d- Pilot new manual in 10 municipal audit offices; revise and finalize new manual 4. Single Treasury Account a- Develop and deliver at least two formal training modules to MOF officials b- MOF issues better practice budgetary allocation, commitment control, and payment authorization procedures c- Public Accounting TWG develops related material and incorporates into GAM 5. To the extent that may be feasible and practicable: a- Integrate budget execution procedures in new GFMIS system b- MOF initiates rollout and training on the new system at two-line ministries/agencies c- Two-line ministries/agencies utilize budget tracking and control reports generated d- Update system codes COMPONENT 1.2 FISCAL DECENTRALIZATION & SUBNATIONAL PFM A. Municipal OSR Budgets – Legacy and New Partner Municipalities 1. Prepare 2021 OSR Budget (and revenue estimates) in Legacy Municipalities (21) a- All 21 OSR budgets for 2021 have Mayoral approval b- 2022 OSR budgets to be approved by MOLG c- Certified by MOLG / HCLA and municipalities graduate from LPFM 2. Prepare 2021 OSR Budget in New (second-generation) Partner Municipalities (25) – PEFA assessment 3. Prepare 2021 OSR Budget in New (second-generation) Partner Municipalities – capacity development 4. Identify 2 CSOs for citizen budgeting in each of 25 second-generation municipalities 5. Train CSOs in citizen budgeting 6. Establish, train, and operationalize Municipal Budget Committees and Citizen Budget Committees (in 25 second-generation municipalities) 57 7. Train 25 second-generation municipalities on budget development, execution, and monitoring 8. Assist 25 second-generation municipalities to develop 2022 OSR budgets 9. Select 10–15 third-generation municipalities and initiate PFEA assessment and compliance training B. Promote Inter-Governmental Finance (IGF) Reform 1. Develop new general purpose IGF grants formula 2. Develop new categorical grants formulae for PHC and PE sectors 3. Develop and adopt property tax policy and strategy, and implementation plan C. Medium-Term Budget Reform Pilot at Municipal Level 1. Solid Waste Management (SWM) – Operational and Public Investment Plan a. Nine original pilot legacy LESP municipalities implement SWM strategy and operational plans b. Nine additional legacy LESP municipalities implement SWM strategy and operational plans c. Develop, revise, and finalize SWM strategy, pilot operational plan, and PIP d. Hold four workshops on SWM procurement and management e. Select five second-generation municipalities for new SWM pilots, hold three workshops on SWM procurement and management, and initiate procurement f. Select additional five second-generation municipalities for new SWM pilots and hold workshop on procurement and management g. Ongoing dialogue with UNDP on SWM funding in Libya 2. Public Health Care (PHC) “Pilots” – Operational and Public Investment Plan a. Six original PHC pilot municipalities fully implement PHC strategy and operational plan b. Twelve additional municipalities implement PHC strategy and operational plan c. Revise and finalize PHC strategy, pilot operational plan, and PIP d. Hold six PHC training workshops for all pilot municipalities e. Establish effective PHC databases at all PHC pilot municipalities f. Establish regular reporting systems at all PHC clinics involved in pilots g. Develop funding guidelines for PHC services that are adopted by MOH and MOF h. Select five second-generation municipalities for new PHC pilots, hold workshop, and establish and staff health offices i. Select five additional second-generation municipalities for new PHC pilots and hold workshop j. Ongoing dialogue with UNDP on PHC funding for Libya 3. Primary Education (PE) – Operational and Public Investment Plan a. PE TWG adopts and proposes to MOED and MOF improved budget processes to better support primary education; MOED and MOF approve and implement proposed processes b. Assess technical capacity of PE pilot municipalities c. Provide four workshops and regular on-the-job training to PE municipalities to improve their PE financial planning and administrative oversight capacity; municipalities develop and implement fee-based service payment mechanisms d. Ten PE pilot municipalities fully implement PE strategy and operational plan e. Identify two PE CSOs in each pilot municipality that agree to participate in PE-related aspects of the citizen budgeting f. Assess capacity of PE CSOs; hold two workshops and provide regular on-the-job training to PE CSOs g. Ten PE municipalities fully implement pilots h. Develop more efficient and effective PE funding mechanisms in pilot municipalities; identify and leverage available PE funding from donors D. Water and Sanitation Sector Strategy Developed ▪ Select five W&S pilot municipalities and train on strategy and implementation plan E. Standards, Policies, and Procedures on Municipal OSR Generation, Tracking, Budgeting, and Management 1. Measures activating elements of Local Revenue Regulation devolving PFM authority to municipalities developed and implemented by MOF, MOLG, and HCLA 2. MOLG and HCLA develop and implement OSR standards 3. Develop and implement training program covering four workshops on OSR for MOLG and municipal staff 4. LPFM partner municipalities develop and implement OSR policy and administrative guidelines and budget planning and management procedures COMPONENT 1.3 MFU CAPACITY DEVELOPMENT, STUDIES/ANALYSIS FOR DECISION￾MAKERS A. Improved MFU Monthly Macro-Fiscal and Quarterly Budget Execution Reports 58 1. MFU produces improved monthly macro-fiscal reports that focus on the key macro-fiscal information, identify trends, and deduce the relevant policy implications 2. MFU produces substantially improved quarterly budget execution reports MFU develops and provides recommendations to MOF on proposed reforms to the ACS B. Increased MFU Macro-Fiscal Technical Capacity 1. Financial Programming a- MFU trained on use of a refined financial programming model to develop and provide to the MOF projections for use in the development of 2021 and subsequent annual budgets b- MFU develops a detailed Medium-Term Fiscal Framework that includes projections under at least three alternatives and scenarios; MFU develops and applies a Gross Domestic Product estimation model to inform annual and multiyear budget formulation discussions (published as a Technical Note) 2. MFU is trained on financial programming of the real sector and inflation and develops a long-term inflation model 3. MFU is trained to understand the importance of private sector development to economic diversification and the negative economic impact of SOEs; MFU develops technical note diagnosing the economic impact of Libya’s SOEs, assessing the rationale for privatization, and reviewing best practices in the management and governance of SOEs 4. MFU has the capacity to assess the output gap and assist with development of countercyclical fiscal policies 5. MFU is trained on financial programming that focuses on the monetary and financial sector and can assist MOF in carrying out financial programming and the design of fiscal policy 6. MFU is trained on the external sector aspects of financial programming and can assess more accurately the impact of fiscal policy on the external sector 7. MFU constructs a nominal and real effective exchange rate series; MFU develops technical note assessing the exchange rate regime’s efficiency and impact on competitiveness and estimating the equilibrium exchange rate 8. MFU is trained to assess the capacity of the banking sector to provide financing to the private sector and to absorb shocks 9. MFU is trained on the development of fiscal rules and develops technical note on fiscal rules and sustainability 10. MFU is trained on assessing current subsidies regime and impact of reforms; technical note on assessing the key structural reforms that can be implemented in the short term 11. MFU develops a budget strategy paper and submits it to MOF C. Assist MFU to Prepare and Present Fiscal Reports to MOF 1. Fiscal report supporting preparation of the 2021 budget developed and submitted to MOF 2. Fiscal report on FY 2021–2022 developed and submitted to MOF D. Formal Trainings for MFU 1. Deliver six training modules to the MFU 2. Deliver six training modules through the MOF Financial Institute COMPONENT 2: MODERNIZE AND COMMERCIALIZE ELECTRICITY SECTOR COMPONENT 2.1: GECOL EFFICIENCY STRENGTHENED [electricity regulator in AWP-1] A. Assist TFESR to Develop and Advocate Electricity Law/Decree 1. Task Force for Electricity Sector Reforms (TFESR) submits and advocates PC adoption of decree 2. PC adopts decree B. Reduce Commercial Losses and Enhance Revenue at GECOL 1. GECOL rolls out new Customer Relationship Management billing platform 2. GECOL pilots Automated Meter Reading and Prepay programs 3. Most GECOL regions reduce commercial losses to 19% or less 4. First tariff increase implemented in March 2021, raising the average tariff from 2.7 to 3.9 cents/KWh 5. Second tariff increase implemented in September 2021, raising the average tariff to 5.1 cents/KWh 6. Assist GECOL to develop policies, procedures, and forms to establish business arrangements with the private sector on fiber optic network utilization 7. Initiate first business arrangement for network utilization C. Independent Electricity Regulator Established 1. The PC appoints key personnel of Regulator 2. Assess capacity needs of the Regulator and develop capacity-building program 3. Implement capacity-building support program to make the Regulator operational 4. Establish twinning arrangement between the Regulator and a best practice foreign regulator D. GECOL Restructuring/Unbundling 1. Finalize restructuring/unbundling proposal and submit to GECOL and TFESR 59 2. Develop financial asset valuation guidelines and submit to GECOL and TFESR 3. PC approves restructuring/unbundling proposal and financial guidelines 4. Initiate financial valuation of GECOL assets E. GECOL Workforce Realignment 1. Assist GECOL’s workforce realignment technical committee to implement the action plan by establishing performance objectives at the corporate, functional, team, and individual levels 2. Identify 2,000–3,000 specific positions to realign to non-GECOL companies to reduce reliance on government subsidies 3. Conduct training needs assessment and develop recommendations F. Improve Private Sector Participation in Electricity Sector 1. Assist GECOL and two municipalities to initiate pilot programs involving partnerships with private sector companies to provide electricity services at the distribution level utilizing smart and/or prepaid meters and collection points COMPONENT 2.2: DEMAND-SIDE MANAGEMENT G. Energy Conservation and Demand Side Management 1. Monitor and ensure the proper implementation of the energy conservation public awareness campaign being implemented by CSO grantee 2. PC, GECOL, and Renewable Energy Authority of Libya (REAOL) establish an independent Energy Conservation and Efficiency Authority 3. Energy Efficiency Authority develops high-efficiency appliance standards 4. Form Energy Audit Service Company and enter into client relationship with a private sector business 5. Initiate GECOL LED change-out program COMPONENT 3: ECONOMIC GOVERNANCE AND BUSINESS ENVIRONMENT A. Complete Institutionalization of Private Sector Advocacy Platform (PSAP) at LBC 1. PSAP secretariat fully institutionalized at LBC and trained on focus group organization, conduct, and reporting 2. Hold three PSAP focus groups (one in each of the first three quarters), consolidate and prioritize reforms identified by focus groups 3. Update priority reform recommendations based on PSAP focus groups and submit to GDSC reform secretariat and other GOL stakeholders semiannually B. Finalize the Operationalization of Online E-Legislation Database at LBC 1. Establish publicly accessible online e-legislation database at LBC; link established on AmCham website 2. Maintain e-database of legislative and regulatory instruments organized by subject matter at LPFM office C. GDSC Reform Secretariat Trained on PPD and Holds Two PPD Meetings 1. GDSC reform secretariat trained to organize PPD meetings between key GOL officials and PSAP participants 2. GDSC reform secretariat organizes and holds two high-level PPD meetings 3. Developed and distribute minutes of PPD meetings D. Complete Guillotine Assessment and Recommendations for Eliminating Requirements that Are Unnecessary or Inconsistent with Best Practice 1. Complete comprehensive guillotine assessment 2. Compile actionable reform recommendations for addressing the issues revealed by the guillotine assessment and submit to GDSC reform secretariat E. Achieve Adoption of Seven Key Legislative–Regulatory Reform Proposals 1. Develop five new legislative and regulatory reform proposals and submit to the GDSC reform secretariat 2. GDSC reform secretariat to establish and hold working groups of public sector lawyers and officials to review and further develop the eight draft instruments developed in FY2020 and the eight developed in FY2021 3. Seven critical legislative regulatory reform proposals adopted, and institutional implementation initiated 4. Hold training program on adopted reforms institutionalized at GDSC and four trainings for civil servants on newly adopted reforms F. Assist GOL to Develop and Submit Libya’s MFTR to WTO 1. MOE and Ministry of Economy and Industry develop GOL’s MFTR, the key document that must be submitted to the WTO to initiate the accession process 2. MFTR reviewed and approved by the PC and submitted to the WTO G. Complete Institutionalization of Legal Support Office (LSO) at PC to Provide Legal Support to Reform Effort 60 1. LSO institutionalized at PC to support the reform effort by participating in GDSC legislative-regulatory working groups and assisting GDSC with training civil servants on newly adopted reforms 2. Four training sessions for LSO and other public sector lawyers held on the drafting of legislation, explanatory notes, and legal opinions; the legislative process; and the rules of legislative interpretation 61 ANNEX E: SUCCESS STORIES (HIGHLIGHTS) The individual highlights are success stories produced and disseminated by LPFM Communications staff: ▪ Energy Sector – Promoting Energy Conservation in Libya ▪ BEE: Easing the Way for Business Development in Libya ▪ BEE: Trading Up – Connecting Libya to the World Economy ▪ Drilling Down: Not for Oil but for Greater Electricity Sector Efficiency ▪ Snapshot Highlights of LPFM January 2020 and Snapshot Highlights LPFM March 2020 ▪ Solid Waste Management: Making Inroads to Cleaner Streets in Libyan Cities ▪ Going Local: Strengthening Municipal Public Service Management and Delivery ▪ Fearless and Female: A Young Woman Entrepreneur Breaking Gender Barriers in Libya ▪ The Woman Trying to Flip the Switch to Energy Conservation in Libya ▪ Climate Links: Blog Submission (Can Libya Flip the Switch to Energy Conservation?) ▪ LPFM Newsletter Highlights December 2020: PFM o Letter of Cooperation Highlights Commitment to Financial Transparency in Libya o Municipal Budgeting Training Helps Strengthen the “Backbone of the State” ▪ LPFM Stories picked up by the Media – January–March 2021 o Two virtual workshops on Operational Plans for PHC for two municipalities o Workshop on cost efficiency of basic primary education (PE) o Workshop on cost efficiency of delivery of basic health services package (PHC) ▪ LPFM Newsletter Highlights January 2021 o Electricity sector: High Power Meeting on Power Sector Reforms in Libya o Municipal PFM: Independent Citizen Budget Committee – A Public Voice for Public Services o Municipal PFM: Prioritizing Primary Healthcare Services in Eastern Municipalities o International Women’s Day March 2021 Independent ▪ LPFM Newsletter Highlights March 2021 o Eliciting Wise Words and Wide Reach for Strengthened Municipal PFM o Improving Solid Waste Management with Libyan Municipalities o Submission for Climate Links – Can Libya Flip the Switch to Energy Conservation? ▪ LPFM Stories picked up by the Media – October–December 2020 o Two workshops on drafting municipal Own Source Revenue (OSR) budgets for 2021 o Virtual workshop on operational plan for primary healthcare (PHC) for Gharyan o Three workshops on drafting municipal Own Source Revenue (OSR) budgets for 2021 o Four training workshops on legislative drafting for lawyers and policy specialists ▪ BEE: Raising the Bar to Improve Legislative Drafting in Libya ▪ Macro-Fiscal Unit Serving a Central Role in Guiding Libya through COVID-19 and Other Crises ▪ LPFM Newsletter Highlights November 2020 o PFM: New Decrees Strengthens Macro-Fiscal Unit’s Role and Sustainability o Building the Momentum for Municipal Control of Local Revenues and Expenditures ▪ LPFM Newsletter Highlights October 2020 o PFM: Building Capacity for Fiscal Planning and Sustainability in Libya o BEE: Building Public–Private Dialogue for More Effective BEE Reform ▪ LPFM Newsletter Highlights September 2020 o Electricity Sector: Higher Efficiency Standards Set for Electrical Appliances in Libya o PHC: Better Primary Healthcare Services, Closer to Home o PE: Defining Quality Primary Education and Key Steps to Providing It ▪ Electricity Sector Reliance and Reform in a Time of Pandemic ▪ LPFM Newsletter Highlights April 2021 o Building Dialog Around Business Sector Concerns ▪ LPFM Newsletter Highlights May 2021 o A Multi-donor Effort Joins USAID Work to Stabilize Libya’s Electricity Grid o Flipping the Switch on Energy Management and Conservation in Libya ▪ LPFM Newsletter Highlights: June 2021 o Reducing Commercial Losses and Leaving a Legacy of Improved Management at GECOL 62 o Solid Waste Management and a Solid Footing in Government Decentralization o Internal Audit Capacity and Accountability within Libyan Municipalities 63 ANNEX F: DATA COLLECTION AND ANALYSIS PLAN The table below provides data sources, data collection methods, and analysis approaches for each evaluation question. EQ1 Data Sources Data Collection Method Data Analysis 1.1 How effective has LPFM been to date in achieving the award’s three objectives? 1.2 What challenges did LPFM face in achieving the objectives? 1.3 In which areas was LPFM most successful? ▪ Documents: Annual Work Plans (AWPs), QPRs, AMELPs, Success Stories/Case Studies ▪ Stakeholders & Partners esp. LPFM and USAID ▪ Activity beneficiaries ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Focus Group Discussions (using FGD Guide) ▪ Descriptive and content analysis ▪ Trend analysis ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings EQ2 Data Sources Data Collection Method Data Analysis 2.1 Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? 2.2 In which areas is USAID’s assistance addressing gaps not met by other donors? ▪ Documents: Task Order, AWPs, QPRs, Donor mapping surveys, USAID docs, donor meeting minutes ▪ Counterparts, Stakeholders & Partners ▪ Activity beneficiaries ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Focus Group Discussions (using FGD Guide) ▪ Descriptive and content analysis ▪ Trend analysis ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings EQ3 Data Sources Data Collection Method Data Analysis 3.1 To what extent is LPFM on track to meet its targets? 3.2 What factors have led to its success in meeting targets? 3.3 What factors led to missing targets? ▪ Documents: QPRs, AMELPs, AWPs, Task Order ▪ Counterparts, Stakeholders & Partners ▪ Activity beneficiaries ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Focus Group Discussions (using FGD Guide) ▪ Descriptive and content analysis ▪ Trend analysis ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings EQ4 Data Sources Data Collection Method Data Analysis ▪ Documents: FY1/2 AMELPs, AWPs, QPRs, third-party monitoring reports ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Descriptive and content analysis ▪ Trend analysis 64 4.1 To what extent did the activity successfully collect data and monitor activities as described in the AMELP? 4.2 What challenges did LPFM face in activity monitoring, both [a] the monitoring it managed and [b] in working with USAID/ Libya’s monitoring and evaluation contractor on third-party monitoring? ▪ LPFM management and MEL staff, USAID staff ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings EQ5 Data Sources Data Collection Method Data Analysis 5.1 To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? 5.2 What has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? 5.3 How can training be improved going forward to improve relevance and utility? ▪ Documents: AWPs, QPRs, training reports, training surveys, focus groups ▪ Counterparts ▪ Activity beneficiaries ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Focus Group Discussions (using FGD Guide ▪ Descriptive and content analysis ▪ Trend analysis ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings EQ6 Data Sources Data Collection Method Data Analysis 6.1 How have the conflict and the COVID-19 pandemic impacted implementation of the project? 6.2 What adjustments did LPFM make in response to the conflict and the pandemic? 6.3 How did the conflict and the pandemic impact the achievement of the project’s objectives? 6.4 How did the project demonstrate resilience/ flexibility in response to these challenges? ▪ Documents: AWPs, QPRs, AMELPs ▪ Stakeholders & Partners ▪ Activity beneficiaries ▪ Desk Review/Research ▪ Key Informant Interviews (using KII Guide) ▪ Focus Group Discussions (using FGD Guide} ▪ Descriptive and content analysis ▪ Trend analysis ▪ Gap analysis ▪ Comparative analysis ▪ Gender analysis ▪ Triangulation of findings 65 ANNEX G: SOURCES OF INFORMATION Key Informants Interviews (KIIs) Informants Title & Entities Name Interview Date LPFM Sr. Field Leadership COP Dr. Ross Worthington Aug 23, 27 Sr. Field Leadership DCOP Ammar Jarrar Aug 19, 24, 30 Sr. Field Leadership Electricity Sector (ES) Team Leader Abdelkader “Ed” El Rahal Aug 18 Sept 8 Fin. Restructuring & Unbundling Advisor, ES Milad Zaed Aug 24 Policy & Governance Advisor, ES Dr. Suliman Fituri Aug 26 Revenue Enhancement, Billing & Collection Advisor, ES Abdallah El Kamel Aug 27 Sr. Field Leadership PFM Team Leader George Dimos Aug 20 Sept 9 Ex PFM TL Dr. Hashim Al Ali Sept 5 Int Consultant, PFM Mark Minford Aug 25, 30 Tripoli-Based Field Leadership Sr. National Advisor, PFM Galal El Kharraz Tripoli-Based Field Leadership Lead Municipal Coordinator, PFM Abdulbaset Tushani Aug 24 MFU Specialist, PFM Jean-Etienne Carlotti Aug 30 MFU Specialist, PFM Zouhair El Kadhi Sr. Field Leadership Sr. Legal Specialist (BEE) Baylasan Gudgud Aug 18 Sept 8 Ex BEE TL Enas Aribi Aug 26 Legal Specialist, BEE Ahmeda Almagri Sept 7 BEE Subcontractor – WTO Expert F. Youwakim Farhat Sept 16 Technical Advisor, BEE William Klawonn Aug 25, 27 MEL Team Lead Mohamed Ali Kannou Aug 19, 23, Sept 13 Sr. MEL Support Specialist, Pragma Heather McHugh Communications Advisor Valeria Gwinner Aug 25 Tripoli-Based Field Leadership Grants Manager Mohammed Abdaljlel Aug 26 Pragma President Dr. Paul Davis Sept 1, 29 Exec. V-P, CFO Jim Defay Sept 10 LGCS COP Keith Hunter Sept 1 LESP Ex-Chief of Party (COP) Anthony Mataitini Sept 2 66 Informants Title & Entities Name Interview Date Ex PFM Team Leader Lejla Catic Sept 6 USAID COR (ret.) Russell Bauer Sept 2 EG Specialist, Tunis Khalil Zouari Sept 6 Regional Economic Officer, Middle East Regional Platform (MERP) Kail Padgitt Aug 31 Program Specialist, M&E, MERP Jessica Pearson Sept 10 COUNTERPA RT BEE Executive Director, Business Supervision Council Al Aref Hassan Sept 1 Dr. Abubaker Abuelkasem Sept 2 Ministry of Economy, Undersecretary Rajeh Khalil Sept 1 Director, Public Sanitation Department (PSD) and Investment Unit, MOE Mousa El Zobik Sept 3 Technical Blogger Amin Salah Sept 4 Project Manager Abuder El Forjani Sept 1 Civil Activist Reem Belkhair Sep 3 ES/GECOL Board & Sr. Consultant Mukhtar Al-Kilani Sept 7 Deputy Board Chair Dr. Abdul Sallam Ansari Sept 5 New CEO Dr. Ibrahim Fallah Sept 7 General Director, Contracts & Development Mohamed Nemri Sept 1 NGO , Head Abdul Bari Banoon Sept 6 G.M. IT & Communications Ali Ayad Sept 14 Sr. IT Consultant A. Benitaif Sept 14 Director, Decision Support Centre Dr. Nuri Elabbar Sept 9 NGO Moomken Shahab Fouzi Musba Sept 8 SUBNATIONA L Head Finance Dept. Tobruk Municipality Dareen Bader Mohamed Sept 1 Finance Dept. Bengazi Municipality Mounir Al Badri Geidalla Sept 5 Ghadamis Youth Org Ali Suliman Aug 31 Employee, Zawiya Khair Org. Samar Mohamed Wallani Sept 12 Zawiya Municipality Zawiya ElKhair Sept 15 City Manager Tripoli Municipality Nasser Al-Krewei Sept 11 Awbari Social Partnership Mohamed Abdulrahman Sept 4 Head Env. Sabrata Mun. Abdulnasser Bashir Sept 6 Director, Economic Affairs & Investment Misurata Municipality Abd Al-Hafeth Manitta Sept 5 67 Informants Title & Entities Name Interview Date MOLG ex Provincial & Municipal Affairs Director Salah Fattah Sept 13 Head Env. Protection, Assaba Mun. Ahmed Emhamed Sept 13 Ministry Environment, Head of QC Dept. Hanan Ahmed Sept 5 DONORS Program Mgr. Economy & PSD, EU Antonis Tsamoulis Sept 16 Ex- Libya Country Rep., World Bank Michael Schaeffer Aug 26 Libya Decentralization & Local Governance Coordinator, GIZ Rani Daoud Sept 19 COP, Private Sector Development Program Libya, Expertise France Sophiane Mehiaoui Sept 13 Focus Group Discussions [FGDs] FGD Sep 7 – Executive branch training indicator 1 Marwa Al-Mahdi Al￾Tajuri Head and Founder and Head of Legal dept. of CSO, Qudwa Al-Baqiyat 2 Omran Osman National Accountant, Ministry of Planning and National Youth Administration 3 Aymen Elshrea Head of the Communication Unit at Government Decision Support Center of the Prime Minister’s Cabinet 4 Mohamed-Saad Shadi Member of the Economy Unit team and Accountant, Ministry of Finance 5 Abed-ElMonem Al-Sadi Head of Technical Cooperation, National Centre for Decision Support FGD Sep 8 – Municipal training indicator 1.2c 6 Dr. Seddik Chaibi Manager of the financial and accounting department, and head of the Macroeconomic and Financial Analysis Unit, MOF 7 Faisal Youssef Al￾Shantah Billing and Accountant Officer, Central Bank of Libya, and Manager at the Macroeconomic Unit, responsible for establishing the Planning and Budget Control Unit for the State-owned companies, MOF FGD Sep 9 – Anti-corruption training indicator 1.1c 8 Mustafa Al-Kabeer Engineer and Head of the Design Studies Department, Benghazi Municipality 68 9 Sarah Al Naeimi Office Manager, Tripoli Municipality Member of the Municipal Council and Women’s Institute 10 Omar Alsudani Project Management Engineer, Sirte Municipality 11 Ahmed Faraj Adviser, Sahat Municipality FGD Sep 12 – Trade & investment training indicator 3.c 12 Abdelmenem Youssef Legal Dept., Government Decision Support Centre (GDSC) 13 Rabie Youssef Legal Consultant 14 Abdulaaziz Estuka Employee, Libyan Business Council 15 Alaa Aldenn Belal Researcher, GDSC 16 Abduladim Benlatif Information Technology Consultant, GECOL FGD Sep 15 – Technical energy (GECOL) training indicator 2.1b 17 Abdulhafeed Shaheen Member of the Board of Directors of the Renewable Energy Agency and an Engineer (REAOL) 18 Hisham Abu Grad Head of Operations Dept, Regional Control Room, GECOL 19 Mohamed Ben koura Senior Advisor, Office of International Cooperation, National Oil Corporation (NOC) 20 Musbah Shehab Project Coordinator, NGO 21 Abdulsadik Saoud Director of Consumer Services Systems Dept., GECOL 69 BIBLIOGRAPHY Project Documents 1. Annual Work Plans for FY 2020 (Nov. 2019), FY 2020 1st revised (February 2020), FY 2020 2nd revised (June 2020), and for FY 2021 (September 2020).38 2. Quarterly Reports for FY 2020 (1, 2, 3, and 4) and FY 2021 (1, 2, and 3).39 3. Contract Award [Activity Task Order] for Pragma 16.09.2019, and modification. 4. METAL Data Quality Assessment Report (DQA) November 2020. 5. Activity Monitoring, Evaluation and Learning Plan FY 2020 (AMELP) October 2019 and finalized/approved December 2019, and FY 2021 AMELP September 2020. 6. Communications [public outreach] Plan, November 2019. 7. Highlights or ‘success stories’ found in QPR annexes. 8. An Instructional Field Guide for Grant-Making, November 2019. 9. Letter of Cooperation between MOF/GNA and USAID/Libya 4/12/20. 10. 150 METAL third-party monitoring reports. 11. MEL System Processes, PowerPoint slides, August 2021. 12. Report for Capacity Development Activities, LPFM April 2021. USAID Documents 1. Libya Economic Stabilization Program [LESP] Middle East Economic Growth (MEG) Final/ Completion Report.40 2. Scenario-based Strategic Plan 2018. 41 3. LPFM Fact Sheet, USAID February 2020. 4. USAID Evaluation Policy: Learning from Experience, USAID updated October 2016. 5. Libya Economic Stabilization Program: Best Practices for a Post-Arab Spring Economy, October 2019. 6. USAID (2015). Qualitative Methods – Module 15. Retrieved from https://usaidlearninglab.org/sites/default/files/resource/files/mod15_qualitativemethods.pdf Other Documents 1. Project Completion review – Top Sheet, Libya PFM Program, Department for International Development (DFID), January 2016. 2. IMF Country Report No. 13/150, IMF, May 2013. 3. The Decentralization Process in Libya, National Municipal Councils Association, EU undated. 4. Country Engagement Note for the State of Libya, International Bank for Reconstruction and Development (IBRD) and International Finance Corporation (IFC), February 2019. 5. Best Practices for Designing and Developing Training, DOD, April 2015. 6. Supporting Capacity Development in PFM – A Practitioner’s Guide, vol. 1, 4th High Level Forum on Ad Effectiveness, OECD, Korea, November 2011. 7. Local Authorities Roadmap for 2021–2027, EU, December 2020. 38 There is no life-of-project work plan. 39 There is no annual report – quarterly reports are cumulative with respect to indicator targets. 40 MEG includes LESP pilot – precursor to LPFM 41 There is no USAID Country Development Cooperation Strategy (CDCS) for Libya due to its non-presence operating unit status. 70 8. Supporting Peace and Stability in Libya: A Compilation of Existing Analysis on Challenges and Needs, World Bank, 2018. 9. Libya Electricity Sector Reform Roadmap, 2017. 10. Libya’s government regains control of western Libya: Taking the East will be harder. The Economist, June 2020. 71 ANNEX H: LPFM MUNICIPALITY LOCATIONS Western Area Details 72 ANNEX I: LPFM TRAINING OVERVIEW LPFM: TRAINING OVERVIEW (derived from output indicators cumulative data in FY2021 QPR-3) 1.1b # Executive Branch Personnel Trained with USG Assistance 1.1c # Government Officials receiving USG supported anti-corruption training 1.2c # Subnational government entity personnel trained with USG assistance 2.1b # People trained in technical energy fields supported by USG assistance 3.c Person hours of USG-supported training completed in trade and investment Life of Project (LOP) target 200 75 175 150 1,200 LOP % achieved as of June 30, 2021 66% 195% 263% 217% 25% Actuals 131 146 460 410 305 Female % 12% 0 4% Under 1% 34% (21 F and 41 M trained for 305 hours completed, so 21 is 34% of 62) Age under 30 0 0 8 0 53 person hours Participant “employer” MOLG, MOF, CBL, MOH, Supreme Council of Local Administration, MOP, GDSC, NCDC, High Council GECOL, MFU, Municipalities (23), Customer Services administration, Green Mountain, MFU, CBL, MOP, MOF Municipalities Tripoli University, GECOL, Tripoli Municipality, GOL, Government Decision Support Center, Civil Society, Private Sector, Customer Services, Administration (Green Mountain) Libyan Business Council, Government Decision Support Center (GDSC) Training Topic  Assessing municipal PFM  MFU – report preparation for macro projections & wage bill Current Economic Challenges and Proposed Solutions  Inter-governmental finance framework  Medium-Term budget workshop  Launching Primary Health Care op plan  Municipal Budget improvements  Fiscal decentralization  Intro to legislative drafting ▪ Independent Citizens Budget committees matters (24 sessions) ▪ Commercial losses workshop – regional billing & collection target performance (14 sessions) ▪ Design a framework with clear objectives & priorities that delivers long-run fiscal sustainability & addresses commodity price volatility ▪ Budget Preparation, assumptions, challenges & objectives (2 sessions) ▪ Subsidy reform in the context of devaluation ▪ SWM Operational Plan including solid waste collection & public investment (4 sessions) ▪ PHC Task force on revising assessment report & working on pilot budget ▪ Tripoli Center PHC task force operational plan training ▪ Task force on piloting of operational plan including COVID-19 measures ▪ Piloting of PHC operational plans including COVID-19 measures (6 sessions) ▪ Electricity Law (33 sessions) ▪ Independent Electricity Regulator (13 sessions) ▪ GECOL financial unbundling (5 sessions) ▪ Commercial losses Workshop (2 sessions) ▪ GECOL Load Shedding Strategy (3 sessions) ▪ GECOL workforce realignment (3 sessions) ▪ Distribution Concessions (2 sessions) ▪ Energy Efficiency Action Plan (4 sessions) ▪ Capacity-building training – Event Management & Facilitation skills (3 sessions) ▪ Public–Private Dialogue – Training for Libyan Business Council Staff (6 sessions) ▪ Public–Private Dialogue – Focus groups with Government Decision Support Centre (4 sessions) ▪ Training on Modern Techniques of Legislation Drafting (7 sessions) 73 LPFM: TRAINING OVERVIEW (derived from output indicators cumulative data in FY2021 QPR-3) 1.1b # Executive Branch Personnel Trained with USG Assistance 1.1c # Government Officials receiving USG supported anti-corruption training 1.2c # Subnational government entity personnel trained with USG assistance 2.1b # People trained in technical energy fields supported by USG assistance 3.c Person hours of USG-supported training completed in trade and investment  Drafting electricity law (3 sessions)  Budget preparation  Subsidy reform in the context of devaluation  Capacity-building, training event management, and facilitation skills  Current economic challenges and proposed solutions ▪ Independent Citizens budget committees matters ▪ Excel basic skills training (29 sessions) ▪ SWM operational plan including zoning of localities and costing of solid waste collection ▪ Introduction to SWM public investment planning ▪ Launching the PHC operational plan in Libya ▪ Municipal Budget OSR Planning & Budgeting Handbook ▪ Municipal budget improvement workshop ▪ Launching fiscal decentralization & subnational PFM program ▪ Municipal budget validation & recommendation program ▪ Municipal Internal Auditors training ▪ Misurata’s task force on revising assessment report, & working on PHC pilot budget ▪ First-term progress workshop (2 sessions for 32 municipalities) ▪ Commercial Losses – review regional billing & collection (12 sessions) ▪ Regulatory Setup ▪ GECOL Emergency Restoration Training (2 sessions) ▪ GECOL Financial Restructuring ▪ Finalizing electricity regulatory decree (4 sessions) ▪ Non-Electric Revenue (3 sessions) ▪ Task Force for Electricity Sector Reforms ▪ Revenue enhancement Training Location Virtual Skype calls (5) Virtual Teams calls (1) Tripoli (4) Albayda (1) Benghazi (4), Tripoli (8), Misurata (3), Albayda (2), Yefren (1), Sabratha (1), Virtual Skype Call (5) Virtual Zoom Call (2) Virtual Teams call (2) 74 LPFM: TRAINING OVERVIEW (derived from output indicators cumulative data in FY2021 QPR-3) 1.1b # Executive Branch Personnel Trained with USG Assistance 1.1c # Government Officials receiving USG supported anti-corruption training 1.2c # Subnational government entity personnel trained with USG assistance 2.1b # People trained in technical energy fields supported by USG assistance 3.c Person hours of USG-supported training completed in trade and investment Virtual Zoom call (1) Misurata (2) Tripoli (6) Benghazi (1) Misurata (1) Virtual Skype call (15) Tobruk (1), Gharyan (1), Alghorifa (1), Sabha (1), Ghadames (1), Sirte (1), Ajdabiya (1), Alriyayna (1), Oujla (1), Virtual Zoom call (9), Virtual Skype call (12) Tripoli (12) Regional Control Center (1) GECOL (1) GECOL Data Center (2) Pragma Office (2) Recorded Sessions 75 ANNEX J: PERFORMANCE INDICATOR TRACKING TABLE The LPFM MEL System maintains a performance indicator tracking table (PITT) to support regular updating of the indicator values and for analysis and reporting. The LPFM PITT includes baseline values and dates, targets, and actual indicator values for all performance indicators collected and reported for the life of the project. For FY 21 Q3, the PITT includes 20 performance indicators (including one context indicator). LPFM Performance Indicator Baseline Cum. Total Progress to LOP Target LOP Date Value Target Status as of June 30, 2021 O3 # of laws, policies, or regulations formally proposed, adopted, or implemented as supported by USG assistance Oct. 19 0 960 points 96% 10 (1,000 points) The LOP target for this indicator has been met, and good progress continues. 1.a % of USG-assisted organizations with improved performance (CBLD-9) Oct. 19 0 79% N/A 80% The FY 2020 annual target for this indicator was met. Data for FY 21 will be collected and reported in the QPR-4. 1.1b # of executive branch personnel trained with USG assistance (Archived Standard DR 2.2) Oct. 19 0 131 66% 200 Good progress toward the LOP target despite restrictions on Executive Branch officials in response to COVID-19 during 2020, and to uncertainties due to the change of government. 1.1c # of government officials receiving USG-supported anti-corruption training (Standard DR 2.4-1) Oct. 19 0 146 195% 75 The LOP target for this indicator continues to be exceeded, indicating strong demand for anti-corruption training. 1.1d % of officials in key institutions with improved skills to perform their functions (Linked to UNDP 1.5.1) Oct. 19 0 71% 89% 80% Additional data will be collected and reported in the Q4 performance report. 1.2a # of municipal budgets formulated with citizen input Oct. 19 0 32 139% 23 LOP target for this indicator has been exceeded. 1.2b # of subnational government entities receiving USG assistance to improve their performance (Archived Standard DR 2.3, Linked to UNDP/Libya 3.1.1) Oct. 19 0 46 164% 28 LOP target for this indicator continues to be exceeded, demonstrating the high demand for LPFM support. 1.2c # of subnational government entity personnel trained with USG assistance Oct. 19 0 460 263% 175 LOP target for this indicator continues to be exceeded, signaling massive rollout of training to municipal officials. 1.3a # of MFU analytical products produced Oct. 19 0 40 143% 28 LOP target continues to be exceeded. 2.a Commercial losses as a % of net production Oct. 19 35% 33% (avg.) 75.5% 22% There is a data lag for this indicator. GECOL’s Q1 2021 commercial losses rose again from the previous quarter; however, collected revenue as a % of the current quarter sales were 154.2%, marking the first time in the activity’s history that collected revenues were greater than billed 76 LPFM Performance Indicator Baseline Cum. Total Progress to LOP Target LOP Date Value Target Status as of June 30, 2021 revenue. This is primarily because of improved bill payment from government-related entities. 2.b % of Libyan private & public sector entities who are aware of the need for higher electricity tariffs Apr. 21 52% N/A N/A 75% Baseline value was collected during the period of performance. 2.c % of GECOL total operating expenses collected as revenues because of additional electricity tariffs Oct. 19 <13% 43% (avg.) 207.3% 30% The increase in collections relative to billed revenue occurred primarily because of the annual payment settlement process between GECOL and other government entities. The LOP target has been met. 2.1.b # of people trained in technical energy fields supported by USG assistance (EG 7.3-1) Oct. 19 0 410 273% 150 LOP target for this indicator continues to be exceeded because the activity continues to reach out to engage other key players in the electricity sector. 2.2a # of KWh saved as a result energy efficiency measures supported by USG assistance Oct. 19 0 N/A N/A 50,000 The ES Team is working with the BEE Team to secure estimates of KWh savings and will be able to report actuals at end of the fiscal year. 2.2b # of requests for information on energy conservation Oct. 19 0 0 0% 300 The grantee began operations in FY 2021 Q2 and will be collecting and reporting actuals at the end of the fiscal year. 2.2c # of new Energy Conservation Programs being implemented by GECOL Oct. 19 1 1 25% 4 Three additional programs are in the planning stage and are expected to be implemented by the end of FY 2021. 3.b Number of sustainable institutions/programs established through USG support to drive the BEE reform process Oct. 19 0 290 145% 2 (200 points) The remaining program (the E-Legislation Website) is expected to be fully operational by FY 2021 Q4. LOP target continues to be exceeded. 3.c Person hours of USG-supported training completed in trade and investment (EG 2-1) Oct. 19 0 305 25% 1,200 Performance for this indicator is mirroring approved revisions of the LPFM/BEE work plan for FY 2021; additional training events are planned for the next reporting period. 3.d # of WTO Trade Facilitation Agreement provisions supported by USG assistance (EG 2.1-1) Oct. 19 0 8 200% 4 The LOP target continues to be exceeded due to adjustments made to the approved LPFM work plan, and to GOL and other Libyan stakeholder commitment to improving the enabling environment for trade. C5 Average hours of electricity service per week over the last three months Sept 19 136 Hours 148 Hours (Avg.) N/A N/A No targets are set for context indicators. There is a data lag for this indicator; data reported in the LPFM quarter represents the data from January to March 2021. 77 ANNEX K: FINDINGS-CONCLUSIONS-RECOMMENDATIONS Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations 1a Effective ▪ Component results are tied to lower-level outcomes and output results. ▪ Deliverables progress is not clear from QPRs. ▪ Populated tables showed status of deliverables produced. ▪ For some tasks, several deliverables were necessary to fully complete the task. ▪ Determining clear outputs from interventions was not easy. ▪ Several tasks to be “initiated” – not very meaningful. ▪ Majority of tasks have been achieved but a few fell short of expectations. ▪ Table 2: mixed reasons why several tasks were not fully completed or initiated. 1b Challenges ▪ Libya is a fragile state – shortcomings in peace, justice, and strong institutions. ▪ Major challenges: conflict/security, COVID-19/pandemic, change of government. ▪ Security: early in 2020 LPFM found itself in the middle of a civil war. ▪ Pandemic: in March 2020 there was a severe COVID-19 outbreak. ▪ The ceasefire precipitated a change in government – serious setback for LPFM. ▪ GECOL board shuffle – a lot of continuity, knowledge, and experience left. ▪ Friction between the PMO/GNU and HOR persists. ▪ Lesser challenges – disconnect between national and subnational. ▪ Engaging with partners – dynamic dialogue on interventions. ▪ Getting through the last mile – to “institutionalize” or adopt deliverables. ▪ Staffing and experts – turnover in LPFM field leadership is disruptive. ▪ Bringing foreign experts into Libya is problematic. ▪ Getting data from public institutions for research – many unresponsive. ▪ Power outages and unstable internet connection make remote work difficult. ▪ Shift to virtual platforms – meetings and training migrated online but “TA is almost impossible to do over Zoom or Skype.” ▪ “Zoom fatigue” – “tired from online meetings.” ▪ Lower-quality training due to online delivery. ▪ Not easy to manage an omnibus project with lots of moving parts (28 workstreams). ▪ Hard to coordinate with multiple counterparts, and beneficiaries and donors. ▪ Shifting priorities of GOL and new counterparts who needed to be “educated” for their “buy￾in.” ▪ Remote management – not being in-country is very difficult. ▪ Changing scope: expand national PFM and revise work plan January 2020. ❖ LPFM is largely successful with respect to achieving its objectives. It is likely to complete most of its planned tasks by year-end. Reasons for incomplete tasks are that GOL approvals are pending or activities have stalled due to a lack of GOL support to move forward. ❖ Libya is a tough operating environment for international development assistance. On top of fragility, LPFM operated during a civil war, a pandemic, and a change of government. ⮚ EQ1a LPFM should report on deliverables in QPRs as per annual work plan tables and reference numbers – include progress, status, and % complete and explain any delays. ⮚ EQ1a LPFM should break down all complex tasks into an action plan with milestone outputs. ⮚ EQ1a LPFM should define the deliverable clearly so that it can be measured and validated. ⮚ EQ1a USAID should allow LPFM to continue incomplete tasks into FY 2022 so that it can bring all its reform initiatives to fruition. ⮚ EQ1b LPFM should continue to engage with reformist elements within partner institutions in a meaningful and consistent manner. ⮚ EQ1b LPFM management should be prepared to “educate” new counterparts to gain their buy-in (i.e., explain the rationale for the intervention so that they are more likely to support and ideally champion the initiative). ⮚ EQ1b LPFM should offer more training on field-level research and data collection methods to GOL and other Libyan stakeholders. ⮚ EQ1b In light of the remote management of the Activity, senior leadership should continue to 78 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ Revise work plan again in June 2020 due to conflict and COVID-19. ▪ No budget for material to support TA – 20% of LGCS budget. ▪ World Bank: focus on national PFM LPFM had to “go slow” due to duplication. ▪ In FY 21 LPFM abandoned work lines WB felt strongly about – see LOC. ▪ Systemic reform is “long game” not possible in election year. ▪ “GNU has appetite for reform and wants tangible results in election year.” 1c Most Successful ▪ Training was quite successful, especially municipal and CSO training. ▪ Cross-cutting: training was done by all components; see EQ5. ▪ Despite only 2 years, many outstanding areas. PFM national ▪ MTEF sector budgeting in SWM, PHC, and PE to use resources more effectively. ▪ MTBs at MOP with MOF for public investments to repair roads, bridges, etc. ▪ Budget unit for SOEs within MOF to improve transparency and disclosure. ▪ MFU – macro-fiscal reports, budget analysis support policy makers, MOF. PFM subnational ▪ OSR budgets were prepared for 46 municipalities, including one-week course for capacity development. ▪ Decentralized and transparent – new sources of funding for public services. ▪ New Regulation No. 330 issued by the GOL Cabinet – big win for LPFM. ▪ Authorizing the municipalities to collect Own Source Revenues. ▪ Medium-term budget (MTB) – pilot municipality ops and capital budgets for SWM, PHC, and PE from the bottom up. ▪ Inter-governmental finance (IGF) grants formula (for municipality grants allocation) links national to subnational, link MOP and MOF. ▪ Trained 60 CSOs in “citizen budgeting” for greater financial transparency and accountability. ES ▪ Revised law on electricity sector – restructured GECOL for greater efficiency and subsidy reduction, independent energy sector regulator to improve oversight, private sector participation through PPP to invest in generating capacity, and leasing GECOL’s 4,000-km fiber optic network to telecom operator to generate non-electric revenues. ▪ GECOL financial unbundling/restructuring proposal/strategy (to spin off 15 non-core businesses) and financial asset valuation guidelines – presented to GECOL Board in workshops during Q4. ▪ Identified first wave of 4,000 positions to be shifted to non-core businesses and Training Needs Analysis (TNA) done for new expertise requirements. ❖ LPFM successes are fairly impressive considering it works in three significant sectors in a very troubling environment. Despite structural reform being a “long game,” LPFM has done considerable foundational work in several areas and aims for fundamental transformation in many initiatives – work on regulatory frameworks is a good “entry point” and constitutes a “pathway” for the next wave of reforms. Assuming a stable political economy, especially post-election, LPFM’s significant momentum should bring systemic reforms to fruition in FY 2022. ensure that field staff has the authority and responsibility to make decisions relevant to their scope and work plan engagement. ⮚ EQ1b Pragma should continue to maintain the priority of bringing senior field leadership back into Libya when the situation on the ground permits, for hands-on management and better counterpart relations. ⮚ EQ1b LPFM should continue to demonstrate flexibility by accommodating new requests for assistance from counterparts that respond to GOL’s changing priorities, as appropriate and within LPFM scope of work. ⮚ EQ1b LPFM leadership and activity managers should continue coordination and communication efforts with donors and donor projects in Libya that are active in related fields to avoid reoccurrence of overlap. ⮚ EQ1b LPFM should set up a Ministerial Steering Committees for each of its components for enhanced accountability and transparency (i.e., governance). ⮚ EQ1b LPFM should continue activities to tackle corruption across all components and in particular work with public financial institutions in Libya such as MOF and CBL. 79 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ Developed web-based billing and collection dashboard system for GECOL to enhance revenue and to reduce commercial losses. ▪ Developed procurement and maintenance dashboard for GECOL contracting dept. to help tackle corruption. ▪ GECOL has taken ownership of the National Energy Efficiency Action Plan (NEEAP). ▪ Some energy audits were completed by a new energy audit group inside GECOL. ▪ LPFM gave a grant to a local NGO Moomken to prepare GECOL for carrying out energy conservation public awareness campaign. ▪ National Standards Bureau accepted the high-efficiency specifications for appliances. ▪ MOE adopted LED replacement program that involves the private sector. ▪ LPFM led a multi-donor effort to implement the LEGS to reduce outages by bringing damaged generation capacity back online through repair/maintenance; held workshops for GECOL on practical load shedding/management; and reached agreements with large industries to shift 500MW from peak to non-peak. BEE ▪ LBC e-legislation database on business laws to better inform foreign investors. ▪ Private Sector Advocacy Platform (PSAP) at LBC to help identify PS issues. ▪ Memo on Foreign Trade Regime (MFTR) – first step in WTO accession. ▪ Commercial law reform drafted 30 legislative, regulatory, and institution proposals. ▪ Legislative drafting training 60 public lawyers. ▪ GDSC trained in high-level events management [3 PPD events]. ▪ Conducted “guillotine” quality assessment of more than 65 laws in trade and investment. ⮚ EQ1c LPFM should continue to promote its outstanding accomplishments using all media sources to reach out to target audiences. EQ.2: Compared to other donors working in Libya, to what extent does USAID have a comparative advantage in the nature of assistance it provides under LPFM? In which areas is USAID’s assistance addressing gaps not met by other donors? Findings Conclusions Recommendations 2a USAID comparative advantages ▪ Because there is no U.S. Embassy in Libya, USAID is forced to work off-site. ▪ Despite lacking a physical presence, since 2016, USAID has a strong track record. ▪ Through its “flagship project”, USAID is addressing Libya top priorities – macro-fiscal, the business side of electricity, and decentralization. ▪ No other donor focuses on SWM, PE, and PHC at the local level. ▪ Through LPFM, USAID understands many sectors, has extensive networks. ▪ Drives reform through capacity development (TA, training, technology, and tools). ▪ USAID is seen as a leader – other donors follow LPFM strategy on SWM and UN calls on USAID for advice and guidance on nascent renewable energy. ❖ Despite having no Embassy in Tripoli and working remotely, USAID has nonetheless cultivated a positive image in Libya. It is seen as having a clear commitment toward Libyans, as being stable, reliable, and practical relative to other donors, and stands out as a credible and trusted partner. ⮚ EQ2a USAID should safeguard its reputation in Libya by continuing to focus on priority needs that are neglected by other donors. ⮚ EQ2a USAID should increase its budget allocation to Libya to address high-priority areas. 80 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ $23 million p.a. is not an insignificant sum. ▪ USAID seen as stable and reliable. ▪ It is “on the ground” while other donors “come and go.” ▪ GOL wanted “deeper involvement” with USAID/LPFM on national PFM. ▪ Some elements inside GOL were not enthusiastic about WB assistance. ▪ LPFM assistance is seen as actionable and impactful, other donors “just talk.” ▪ Private sector approaches are promoted, e.g., in SWM outsourcing. ▪ LPFM assistance introduces best practices through international experts. ▪ LPFM has strong links to numerous stakeholders – it partners with reformers (counterparts) in many Ministries – MOLG, MOP, MOF, MOET, MOH, and MOEPE. ▪ USAID seen as “having no hidden agenda, unlike Turkey.” 2b Addressing Gaps ▪ USAID is working in areas neglected by other donors – decentralization (municipal financial management), ES, and BEE. ▪ USAID is the only donor working both national and subnational. ▪ USAID/LPFM invited by GOL to take up interventions in budget execution and monitoring, WB domain – caused resentment as unnecessary duplication. ▪ LOC proposes to fill gaps such as LCA, CBL, and LAB. ▪ LOC helps USAID coordinates with WB to “sort out mutual interests.” ▪ USAID seeking synergies – complementary between LPFM getting municipalities budget ready and facilitate fund flows, and LGCS helps them to spend the money on community priorities. ▪ USAID/LPFM collaborative – it shares information with Expertise France on BEE, and with GIZ on SWMUSAID/LPFM work on LEGS to reduce outages – raise GOL credibility and legitimacy. ▪ LEGS and contracting and procurement dashboard not in FY 2020 work plan but added ES in response to GOL request. ▪ Expand into renewables – UNDP has done feasibility on solar power generating stations and water pumps, GIZ is also interested. ▪ International Cooperation Office at MOP not effective at donor coordination. ▪ LPFM conducts “informal coordination” with other donors and projects. ▪ LPFM has grown organically, i.e., is demand-driven. ❖ USAID is the only donor addressing Libyan urgent priorities (subnational, electricity and business) – other donors do not have a strong presence in these areas. The LOC formalizes collaboration, addresses duplication, and, more importantly, proposes new areas for cooperation or gaps. ⮚ EQ2b USAID should set aside 10%– 20% of an Activity budget for material/physical assistance to support TA in Libya. ⮚ EQ2b USAID should lobby USG to establish an Embassy in Tripoli to allow USAID staffers to gain a deeper understanding of the context and for regular face-to-face high-level meetings. ⮚ EQ2b USAID should attend and/or host a donor forum in Libya on a regular basis for enhanced donor coordination. ⮚ EQ2b USAID should continue to encourage Activities to be demand￾driven and allow them to respond to host government requests for assistance within their SOW. EQ.3: To what extent is LPFM on track to meet its targets? What factors have led to its success in meeting targets? What factors led to missing targets? Findings Conclusions Recommendations 3a on track ❖ LPFM success in meeting indicator targets is mixed. According to FY 2021 QPR-3, of ⮚ EQ3a USAID should permit an Activity to adjust indicators and 81 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ Illustrative indicators in the Task Order were inserted into the Pragma contract without any discussion and “they had to work with them.” ▪ No effort to align them with the subsequent work plans in FY 2020. ▪ Initial targets were generated through discussions between COR and LPFM. ▪ In first year, LPFM has prepared three work plans. ▪ In the June 2020 2nd revision work plan: activities were significantly scaled down/ rescheduled/deferred to FY 2021, but no changes were made to FY 20 MEL plan. ▪ AMELP FY 2021, 5 indicators were slightly modified, 1 was removed, and monitoring and data collection methods adjusted to add remote procedures. ▪ Despite overtaken-by-events targets were not changed in FY 2021. ▪ MEL is tracking 20 performance indicators; 19 have targets – one is context indicator so no target. ▪ Two indicators targets are met, nine have been exceeded, two will fall short (i.e., below target), five will likely be achieved in Q4, one has a data lag, and one has no target – see PITT in Annex J for a fuller explanation of the status of each target. ▪ MEL does not track deliverables unless they are part of an indicator. ▪ Social inclusion in the Task Order but no indicators or targets related to integration. ▪ No programmatic focus on social inclusion – report only gender and age in training. ▪ Some females on TWGs and LPFM promote female leadership in CSOs and have highlighted International Women Day and special profiles on women. ▪ Very few females are on LPFM staff, few females in training events. ▪ Subnational area, LPFM works in remote (marginalized) areas such as Sabha and the East, promotes females in municipal training and legislative drafting training. 3b Success Factors Targets are for the most part “realistic and therefore achievable” Training ▪ LPFM is focused on reform supported by capacity development, a means to an end in which training is key element – LPFM has a heavy focus on training, and it was designed and delivered effectively and well received. ▪ Flexible training – best practice tools in workshops; skills training in short courses. ▪ Local Advisors/LPFM staff provide the follow-up on training, e.g., on-the-job training. ▪ Coaching in the workplace to ensure adoption by local staff and institutions. Staffing ▪ Sr. staff highly experienced and knowledgeable – experts in their field so credible – COP, DCOP, and component leads good project management (local staff also technically 20 performance indicators tracked by MEL, two targets are met, nine have been exceeded, two will fall short, five will likely be achieved in Q4, and one has a data lag.. There is no target for gender. ❖ LPFM success factors are numerous not least of which is the expertise, commitment, and diligence of the IP/LPFM team that is considered critical for driving overall performance, strong relations with counterparts and local officials and development of capacity of partner organizations through training. targets when this is warranted by changing conditions on the ground. ⮚ EQ3a LPFM should capture data in a systematic manner to report on the nature and scope of critical training follow-up (e.g., on-the-job coaching) as well as other non-training events that have learning value (e.g., working sessions and technical meetings). (EQ4a) ⮚ EQ3a LPFM should develop a gender strategy, and with USAID formulate a social inclusion indicator with targets. 82 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations strong and highly qualified) – all are Libyans so very familiar with the context and are very well connected. ▪ LPFM coordinators work with clusters of municipalities. ▪ “Pragma has a top-quality technical team” and “committed and serious.” ▪ Many local staff/advisors worked on LESP so there is continuity. ▪ Local staff: have good rapport with each other, i.e., teamwork. ▪ Local staff participate in training events, and coached by international expert, so LPFM leadership can use local staff for delivery of follow-up. ▪ Subnational work is fairly basic “greenfield,” so can rely on municipal coordinators with minimal supervision. Experts ▪ STTAs are international experts – Arabic speakers, familiar with best practice tools, introduce the best practice tool, and help to adapt or customize. ▪ Backed up by local advisors on-the-job for effective utilization of tools. ▪ Some experts (and local advisors) embedded in partner institutions. IP ▪ Pragma is a professional and efficient organization that provides strong technical and strategic support to the field, i.e., backstopping. Counterparts ▪ Counterpart and stakeholder relations are excellent – well-managed relationships to continue strong support. ▪ In ES component there is GECOL Task Force, subnational key relationship is with MOLG and individual municipal Mayors, national PFM is MOF, sectoral work coordinate with line Ministries. ▪ Essential to forge alliances with reform-minded counterparts. ▪ Champions – the “drivers of change.” ▪ Engage with the counterpart at every step – get them on-side and informed. ▪ In addition to WP, LPFM looks for “targets of opportunity.” ▪ Approach – expert introduces best practice tools (models, systems, etc.) and works with TWG to customize and adapt and get ministerial or senior management endorsements of interventions. Engagement ▪ Share overall work plan with key counterparts. ▪ Jointly prepare plans for a reform initiative. ▪ Cooperation is better under GNU than with previous government GNA. 83 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ Buy-in or ownership from counterparts, to help integrate deliverables into partner institutions – to adopt into practice. Plans ▪ Clear coherent strategy and plans – detailed approaches, e.g., municipal OSR. ▪ Workstreams serve to focus activities. ▪ High-quality/best practice actionable deliverables, not just papers, e.g., OSR. TWGs ▪ TWGs is the working level and key to coordination. ▪ Useful for coordinating with different Ministries, are relatively stable. ▪ USAID support important. Technology ▪ Mastered virtual platforms for meetings and training. ▪ MEL Tight feedback loop – data provided to management for informed decisions. ▪ DCOP based in Tripoli – on the ground. ▪ Workarounds on conflict, COVID-19, and change in government (see EQ6.2). ▪ Creative, e.g., helping municipalities work around Law 59 on PHC budget. ▪ Constant flux, LPFM needs to be agile and flexible – LOC sorted out duplication and overlapping WB and identified emerging needs. ▪ LPFM’s small grants facilitate planned deliverables such as [a] Moomken to run an energy conservation publicity program on behalf of GECOL; [b] three CSOs to arrange citizen awareness and social accountability sessions; and [c] Libyan Business Council for the establishment of an e-legislation web portal. 3c Factors Missing Targets ▪ Two indicator targets falling short are training output indicators – executive branch (that crosses all three components) and trade and investment BEE because GNU told senior government officials to avoid donors, plus COVID-19 restrictions. ▪ Pandemic slowed and delayed activities, virtual meetings/trainings lower quality than in￾person, “Zoom fatigue,” travel restrictions, could not bring international experts to Libya or take Libyans to Tunis. ▪ Change in government means new counterparts need to be educated for their support. ▪ Politic instability – power struggle between East and West, PMO and HOR, and central and local. ▪ For BEE training, a factor is that MEL uses a narrow definition of “training” – many BEE activities are considered “non-training” events and as such are not counted but have learning value. ❖ Pandemic restrictions combined with political instability and uncertainty were the main factors that contributed to LPFM missing some targets. ⮚ EQ3c LPFM should continue to put concerted effort into training up local staff to take on the roles of experts. 84 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ Remote management suboptimal – best to be on the ground, real interface. ▪ For ES, during conflict, power grid damaged – new workstream was added. EQ.4: LPFM has a detailed activity monitoring, evaluation, and learning plan (AMELP). To what extent did the activity successfully collect data and monitor activities as described in the AMELP? What challenges did LPFM face in activity monitoring, both the monitoring it managed and in working with USAID/Libya’s monitoring and evaluation contractor on third-party monitoring? Findings Conclusions Recommendations 4a MEL System ▪ A Data Quality Assessment Report (DQA) Nov. 2020 details MEL system o The IP maintains a good level of MEL function independence o IP’s MEL system is designed to ensure consistency of data collection procedures o The IPs’ multi-tiered data verification procedures and cross-check methods and data management procedures in indicator-specific SOPs are comprehensive and the IP has documented its remote monitoring protocol ▪ The MEL system features: o A manual, periodic quality assurance/control and back up archiving o Weekly data updates shared with LPFM management for informed decisions o Constant updates and improvements to the system – adaptive management o Collects data only related to indicators, not on non-training events o System is independent of individual views and is available anywhere or anytime ▪ MEL is responsive to both USAID and METAL requests for data. ▪ Handled MEL changes in January 2020 exceptionally well. ▪ “Set the standard for other activities in Libya.” ▪ MEL did a survey to follow up on training, whether they utilize the trainings. ▪ MEL doing perception surveys on electricity tariff increases and BEE. ▪ Performance data flow contained in the AMELP is shown graphically in Annex L. 4b Internal Monitoring & TPM Internal Monitoring ▪ After FY 2020 revision to WP, MEL had to modify indicators (but not targets). ▪ MEL also added remote processes and procedures to the MEL system. ▪ Adding multiple access points to the MEL system was an IT challenge. Third-party Monitoring (TPM) ▪ Good coordination LPFM and METAL since LESP. ▪ Changing events and times is challenge for TPM. ▪ METAL web tool captures all the information needed from the IP. ❖ The MEL system as outlined in the AMELP is thorough, robust, and responsive, and performance data captured and reported is reliable. Pragma admits that “LPFM has over-designed its MEL system” and USAID stated LPFM has “set a high bar” for other IPs in Libya. ❖ Modified indicators and remote processes and procedures incorporated by LPFM MEL into the FY 2021 AMELP was an adaptation to the turmoil on the ground. Apart from changing events and times due to the pandemic, LPFM MEL and METAL had a good working relationship vis-à-vis TPM. ⮚ EQ4a LPFM MEL should carry out more focus groups to determine to what extent training know-how is used and adopted in the workplace. (EQ4a) 85 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ METAL gives access to the MEL team to make inputs into the forms. ▪ System sends a web notification with a link to give a timeline to enter all inputs. ▪ The constant changes of programs and events is a challenge, but LPFM notifies METAL of these changes immediately. ▪ In general, the reports and feedback are positive. ▪ The 150 TPM reports so far are useful in terms of action. ▪ TPM reports go from METAL to COR “confirming the LPFM events.” EQ.5: To what extent have the individuals who were trained by LPFM been able to use the training to better perform their roles and responsibilities? If so, what has LPFM done to ensure that the knowledge gained from training is adopted and used to benefit government entities? How can training be improved going forward to improve relevance and utility? Findings Conclusions Recommendations 5a Individual Use Training ▪ LPFM has a focus on reform that is supported by capacity development activities. ▪ Capacity development is a means to an end, in which training is the key element. ▪ A typical intervention process starts with agreeing with the counterpart then LPFM brings in an expert who introduces a best practice tool and works with TWG to develop/adapt tools. ▪ Afterwards, relevant staff are selected, a training program is developed, sometimes Training of Trainers. ▪ Training occurs to transfer know-how, then follow-up with individuals on-the-job to ensure they use and apply the tool. ▪ Training design and delivery varies, e.g., workshop format for knowledge sharing. ▪ BEE public lawyers legislative drafting is skills training. ▪ Usually, training is a mix of knowledge and skills. ▪ Feedback from counterparts and beneficiaries – training was “good, useful, effective, well organized,” “We learned a lot,” “The training was useful to my job,” “There was regular follow-up with us,” “They visited us frequently,” and “We regularly call the local coordinator for advice.” ▪ But one informant stated, “Some of the knowledge was not entirely new,” “Training was short,” and “It was a problem via Zoom.” ▪ Delivered a lot of training especially for municipalities, public lawyers, and CSOs. ▪ Municipal training was basic (“greenfield” as 46 municipalities had no experience). ▪ Training was extensive (training program was one week). ▪ Legislative drafting training 60 public lawyers. ▪ 60 CSOs trained in “citizen budgeting” and “a public voice for public services.” ▪ Training was reported in QPRs under five training output indicators. ❖ Training is key to capacity development and LPFM has conducted much training. Though training content and delivery was good about half the trainees indicate they are not yet using the newly acquired know-how. They either have not yet had an opportunity to apply it or COVID-19 restrictions limited their ability to be in the office applying their skills. However, they do expect to use this in future. 86 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ The volume and variety of training is reflected in an overview table Annex I. ▪ Initially many civil servants did not like virtual training but out of necessity got used to it – whereas some now see “Zoom fatigue.” ▪ Ongoing debate about efficacy – the best approach is “blended” or hybrid. ▪ Note the low number of female participants in training, except for CSO training. ▪ Training follow-up after events was mixed. ▪ CSOs said LPFM followed up with them while some municipal staff said not. ▪ Without intensive follow-up, there is weak application and integration. ▪ Recent survey on capacity development found only half training recipients had applied know￾how on the job. ▪ Survey shows that COVID-19 was a barrier to applying acquired know-how. ▪ Those trained were instructed to stay home and could not apply new know-how. 5b Training Adopted by Government Entities ▪ LPFM provided on-the-job training/coaching to trainees as a follow-up to formal training. ▪ It also set up many TWGs to facilitate the development and implementation of capital development plans. ▪ LPFM engaged with many counterparts on its 28 workstreams but a change of government in March resulted in many new counterparts who did not understand the intervention rationale. ▪ Despite the impediments resulting from COVID-19 and the change in government, many interventions went well, and training was used and adopted so that institutions benefit. ▪ LPFM does not capture or report data on the nature and scope of critical training follow-up. 5c Improving Training [Recommendations] The following approach (Figure 3 is suggested to improve training relevance and utility: o Libya-led: Aid recipients should lead with donors playing a supporting role. o A participatory approach will build trust and commitment. o Champions: identify reform-minded persons to be drivers of change. o Engage counterparts at all stages for “ownership.” o Agreement: Memorandum of understanding will ensure there is clear understanding of roles. o TWG: facilitate and coordinate the initiative – the working level. o Understanding context: deep understanding of institutional situation. o Engagement: dynamic dialogue with the counterpart on priority needs. o Imported solutions: do not insist on specific training. o Genuine commitment at the highest level within an organization. o Involvement: Ensure the partner is actively involved in, and agrees to, the reform strategy and the capacity development plan. ❖ To ensure that training is used and adopted to benefit government entities, LPFM has provided follow-up coaching to training recipients, embedded experts in partner institutions, set up Technical Working Groups, and engaged extensively with counterparts. However, COVID-19 restrictions and a change of government have served to undermine this approach. ❖ Enhancing the performance of the public sector is not easy due to technical complexity, political sensitivities, and ⮚ EQ5c Figure 3 outlines a best practice approach to training design 87 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations o Expert: involve counterpart in identifying suitable international experts. o Training of trainers: to be considered as this is better for sustainability. o Trainees: Identify suitable people for training and target women. o Delivery mechanism: in-person, virtual, or mix/blended. o Workshop better for interactive, “classroom” for lecture style. o Training plan: Develop agenda/curriculum, materials, instructor/facilitator SOW. o Deliver training in relevant know-how and gauge their satisfaction. o Follow-up: Ensure on-the-job training for optimal integration of know-how. o Embedded: follow-up is improved when staff or experts are located inside organization. o Upgrade local staff: Experts to help them become “subject matter experts.” o MEL: Ensure to capture data on TA, training (and non-training events). o Digital: Record and store events digitally in a library for remote access. resistance to change but a sound process will enhance training relevance and utility. It is vital that intensive follow-up with training participants occurs to ensure new know-how is applied on the job and that reform measures [models, systems, procedures] are adopted by partner institutions and thereby take root. and delivery that will maximize training relevance and utility. Because the LPFM approach is already aligned with this process there are no specific recommendations only that LPFM should follow the steps systematically. (EQ5c) EQ.6: How have the conflict and the COVID-19 pandemic impacted implementation of the project? What adjustments did LPFM make in response to the conflict and the pandemic? How did the conflict and the pandemic impact the achievement of the project’s objectives? How did the project demonstrate resilience/flexibility in response to these challenges? Findings Conclusions Recommendations 6a Conflict and COVID-19 Affected Implementation ▪ Conflict and COVID-19 slowed LPFM implementation and made technical work more difficult. ▪ Since ceasefire in Oct 2020 there have been only “small flare-ups,” and rising criminality. ▪ Severe COVID-19 outbreak, half LPFM staff and many counterparts got sick. ▪ Border closed, expats evacuated April 2020 (but DCOP stayed in Tripoli), events canceled or postponed, delays, lockdowns, curfews, travel, visa restrictions. ▪ Could not bring international experts to Libya or Libyans to Tunis. ▪ Government officials avoid in-person meetings, offices closed, not answering phones. ▪ Staff had to work from home, carry out meetings and interviews virtually – harder to get data and do research. ▪ Expats evacuated so cannot meet face-to-face with counterparts. 6b Response to C&C ▪ Virtual meetings and training, off-site meetings, staff work from home. ▪ MEL work – the only issue was the postponement of the events. ▪ Virtual MEL – Adjusted data collection method. ▪ In Libya the frequent power outages means internet is unreliable and participants log off, so sessions were recorded and made available to participants. ▪ Meetings and training done late at night when electricity and internet is better. ▪ Initially LPFM leadership was on-site in Libya but due to C&C had to leave. ❖ Conflict and COVID-19 slowed LPFM implementation by making technical work and activity management more difficult. Since the ceasefire, movement has returned to normal apart from small flare￾ups, but the pandemic continues to hinder LPFM implementation today to a lesser degree. ❖ LPFM found effective workarounds to mitigate the worst effects of the conflict and COVID-19 so that they are no more than a “minor hiccup.” After some creative adjustments, technical work progressed and LPFM was soon back on track. ⮚ EQ6a LPFM should continue to be adaptable and flexible to mitigate the effects of the conflict and COVID-19. ⮚ EQ6a Pragma should continue to aim for a stable staff complement, especially at the management level, in order to maintain continuity in counterpart relations, teamwork, and Activity “memory.” 88 Findings-Conclusions-Recommendations Matrix EQ.1: How effective has LPFM been to date in achieving the award’s three objectives? What challenges did LPFM face in achieving the objectives, and in which areas was it the most successful? Findings Conclusions Recommendations ▪ DCOP stayed on-site. ▪ Initial resistance to virtual training, but now accepted; mix of virtual and in-person. ▪ LPFM was directly involved in COVID-19 response via revision of PHC budgets. ▪ Conflict mitigated through use of family and clan ties, to cross into the East. ▪ LPFM enhanced security protocols, security bulletins, close protection, etc. 6c C&C Affected Results ▪ LPFM able to achieve most results, despite the onerous conditions, but harder to work and to achieve results. ▪ International experts could not provide TA on-site or in-country training. ▪ COVID-19 remains somewhat challenging but is manageable. ▪ Conflict less of hindrance now, but the associated political instability is a concern. ▪ New GNU and ongoing political instability are challenging for achieving results. 6d Resilience/Flexibility ▪ Staff remained committed, diligent, drew on their networks, were resourceful. ▪ Good relationships with counterparts enabled continuity during the pandemic. ▪ Work planning was flexible, produced three work plans in the first year. ▪ Came up with workarounds, e.g., shift to virtual platforms, remote management. ▪ Local staff/advisors could step in for international experts – twinning arrangement. ▪ When COVID-19 hit, many donors suspended projects, or delayed projects, but LPFM continued to work. ▪ Even with fighting, local LPFM staff continued their dealings at local level. ▪ LPFM senior leadership managing remotely – adaptive management. ❖ Both COVID-19 and the conflict became manageable and hardly affected LPFM overall results. However, the new government and the associated change of ministers and counterparts etc. together with ongoing political uncertainties, made the “final mile” adoption of reform initiatives more tenuous. ❖ LPFM has coped well with the dual challenges of COVID-19 and the conflict – LPFM said “Libyans are great at adapting.” Through workarounds LPFM has continued to operate while other donors suspended projects or delayed launching new ones. ⮚ EQ6d LPFM should maintain a dynamic, innovative field team with strong counterpart relationships and Head Office support to overcome the debilitating effects of the conflict and COVID-19. 89 ANNEX L: LPFM PERFORMANCE DATA FLOW42 42 AMELP FY 2020 v. 3 December 2019. Quarterly Reports Annual reports, Final Reports** Data & other evidence filed by MEL Tunis – both hard & soft copies** Completed data collection forms reviewed by MEL in Tunis LPFM activity takes place* Beneficiaries Stakeholders Municipalities, Ministries, GECOL, firms, etc.* Reports w/ indicator data* LPFM data collection forms filled out by LPFM staff in Libya & Tunis* Questionnaires, Surveys, Polls, FGDs, etc.** LPFM data collection forms filled out by LPFM MEL staff in Tunis & Libya LPFM Staff in Tunis & Libya, LPFM Consultants & STTA Completed data collection forms reviewed by component teams METAL & USAID site visits Data entered into LPFM MEL MIS** *QA/QC &/or site visits conducted by LPFM MEL **QA/QC conducted by Pragma HQ MEL 90 ANNEX M: EVALUATION TEAM MEMBER QUALIFICATIONS The evaluation team brought a wealth of relevant experience and expertise to this evaluation. ▪ Team Leader Denis Dunn has 38 years of professional experience, 25 of which were working for major donors in emerging and conflict-affected countries. He has extensive experience as a team leader in PSD/BEE, as well as public sector reform, M&E, and capacity-building, and has worked previously on evaluations for USAID in Afghanistan and Tunisia. ▪ Medhat El Ghdamsi has 35 years of professional experience in senior Libya public managerial positions (administration, anti-corruption, social policy, and PFM), as well as extensive experience in preparing, designing, and delivering professional training. He has considerable knowledge of project cycle management, programming, implementation, evaluation, and international assistance coordination mechanisms. ▪ Mohamed El Magbri is a Libyan local governance expert with 10 years of experience. He has consulted for the World Bank, USAID, UNSMIL, and the EU Delegation to Libya. He provides research and policy advice on local governance issues. He has a master’s degree from the Walsh School of Foreign Service at Georgetown University and is a recipient of the Open Society Foundation’s Civil Society Leadership Award. ▪ Tamim Rashed is British-educated Libyan consultant experienced in the monitoring and evaluation field in Libya. As a researcher and political analyst, he is committed to supporting the Libyan democratic transition, in particular the constitution building process. ▪ Ali Khatib, a staff member of METAL, has 12 years of experience in Monitoring, Evaluation, and Learning in development projects for USAID, DFID, EU, IBRD, and the UN in Palestine, Tunisia, and Libya. He has expertise in building M&E systems focusing on the development and design of performance monitoring systems and methodology, providing performance management, third-party verification, data quality assessments, and impact evaluation. 91 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523